mck commented on CASSANDRA-13480:

I can see immediate benefit to users of Reaper with this, so would like to see 
it back ported to 3.11.x

[~jjirsa], what are your thoughts?

Regarding the patch:
 - the only public signature added are:
 -- the method {{List<String> StorageServiceMBean.getParentRepairStatus(int 
 -- the enum {{ActiveRepairService.ParentRepairStatus}}
 -- the method {{ActiveRepairService.recordRepairStatus(…)}}
 - the addition of the 100k entry cache for ParentRepairStatus events

While it seems to be reasonably safe, if it is back ported I wonder if the 
{{getParentRepairStatus(..)}} method should be marked as {{@Beta}}?

> nodetool repair can hang forever if we lose the notification for the repair 
> completing/failing
> ----------------------------------------------------------------------------------------------
>                 Key: CASSANDRA-13480
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13480
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Tools
>            Reporter: Matt Byrd
>            Assignee: Matt Byrd
>            Priority: Minor
>              Labels: repair
>             Fix For: 4.0
> When a Jmx lost notification occurs, sometimes the lost notification in 
> question is the notification which let's RepairRunner know that the repair is 
> finished (ProgressEventType.COMPLETE or even ERROR for that matter).
> This results in nodetool process running the repair hanging forever. 
> I have a test which reproduces the issue here:
> https://github.com/Jollyplum/cassandra-dtest/tree/repair_hang_test
> To fix this, If on receiving a notification that notifications have been lost 
> (JMXConnectionNotification.NOTIFS_LOST), we instead query a new endpoint via 
> Jmx to receive all the relevant notifications we're interested in, we can 
> replay those we missed and avoid this scenario.
> It's possible also that the JMXConnectionNotification.NOTIFS_LOST itself 
> might be lost and so for good measure I have made RepairRunner poll 
> periodically to see if there were any notifications that had been sent but we 
> didn't receive (scoped just to the particular tag for the given repair).
> Users who don't use nodetool but go via jmx directly, can still use this new 
> endpoint and implement similar behaviour in their clients as desired.
> I'm also expiring the notifications which have been kept on the server side.
> Please let me know if you've any questions or can think of a different 
> approach, I also tried setting:
>  JVM_OPTS="$JVM_OPTS -Djmx.remote.x.notification.buffer.size=5000"
> but this didn't fix the test. I suppose it might help under certain scenarios 
> but in this test we don't even send that many notifications so I'm not 
> surprised it doesn't fix it.
> It seems like getting lost notifications is always a potential problem with 
> jmx as far as I can tell.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

Reply via email to