[
https://issues.apache.org/jira/browse/ARTEMIS-3045?focusedWorklogId=549078&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-549078
]
ASF GitHub Bot logged work on ARTEMIS-3045:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 06/Feb/21 13:54
Start Date: 06/Feb/21 13:54
Worklog Time Spent: 10m
Work Description: franz1981 edited a comment on pull request #3392:
URL: https://github.com/apache/activemq-artemis/pull/3392#issuecomment-774405162
@clebertsuconic @michaelandrepearce @jbertram @brusdev @gtully
This change seems to perform best, but I cannot say I am satisfied, because
I see a problem both on this and the original implementation ie we are not
handling back-pressure back to the replicated journal caller threads.
This remind me of
https://cassandra.apache.org/blog/2020/09/03/improving-resiliency.html for who
is interested..
TLDR:
- in the master implementation we can make the broker to go OOM by adding
too many Runnables to the replication stream because it can block awaiting
Netty writability for 30 seconds
- in this PR first implementation we can have a JCTools q of the outstanding
packet requests that can grow unbounded for the same reason and makes the
broker to go OOM again
- in this PR last implementation we can have the Netty internal outbound
(off-heap) buffer that can grow unbounded
Despite the first 2 solutions seems better at first look, because they wait
for enough room in Netty buffer, we can still get OOM under the same
circumstances ie while awaiting the backup to catch-up.
We should:
- track the amount of pending work for monitoring
- propagate back-pressure
Similarly to Cassandra, we could setAutoread false on client connections,
but that means:
- clients should handle themselves from OOM
- they're stopped to make progress in whatever operation they're performing
vs the broker (not just ones that involves replication)
Sadly this doesn't seem to be a viable solution, for the latter effect.
Another solution would be to start diminishing the credits for the addresses
that involves large msg/paging and journalling, but it's probably more feasible
per-protocol, not a unified solution...
I cannot say what's the best option here and I don't see if we are already
using some form of end-to-end protection.
I see we just let the journal replication request to be enqueued, leaving
the `OperationalContextImpl` in charge of sending back acks to clients only
when both the local durable write and the replication has succeeded: this seem
to prevent the same "user" to ask for a new request while awaiting the previous
one (thinking about `MessageProducer::produce` with a durable write), but just
increasing the amount of clients can increase the amount of enqueued requests
to cause OOM, under load.
Any thought?
IMO solving this correctly can bring a huge performance increase with an
improved stability too.
Note:
An "easy" solution to this problem would be to pause the replication
request(s) (it should happen on an I/O thread) unblocking it when needed (or if
the slow replication connection is closed) if the amount of pending requests
get too high, but the risk is to have a stealthy performance killer due to the
cost of blocking/unblocking in case of misconfigured Netty watermarks or TCP
buffers...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 549078)
Time Spent: 5h 40m (was: 5.5h)
> ReplicationManager can batch sent replicated packets
> ----------------------------------------------------
>
> Key: ARTEMIS-3045
> URL: https://issues.apache.org/jira/browse/ARTEMIS-3045
> Project: ActiveMQ Artemis
> Issue Type: Improvement
> Reporter: Francesco Nigro
> Assignee: Francesco Nigro
> Priority: Major
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
