[jira] [Updated] (CASSANDRA-12966) Gossip thread slows down when using batch commit log
[ https://issues.apache.org/jira/browse/CASSANDRA-12966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Brown updated CASSANDRA-12966: Fix Version/s: 4.0 3.11.1 3.0.15 > Gossip thread slows down when using batch commit log > > > Key: CASSANDRA-12966 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12966 > Project: Cassandra > Issue Type: Improvement >Reporter: Jason Brown >Assignee: Jason Brown >Priority: Minor > Fix For: 3.0.15, 3.11.1, 4.0 > > > When using batch commit log mode, the Gossip thread slows down when peers > after a node bounces. This is because we perform a bunch of updates to the > peers table via {{SystemKeyspace.updatePeerInfo}}, which is a synchronized > method. How quickly each one of those individual updates takes depends on how > busy the system is at the time wrt write traffic. If the system is largely > quiescent, each update will be relatively quick (just waiting for the fsync). > If the system is getting a lot of writes, and depending on the > commitlog_sync_batch_window_in_ms, each of the Gossip thread's updates can > get stuck in the backlog, which causes the Gossip thread to stop processing. > We have observed in large clusters that a rolling restart causes triggers and > exacerbates this behavior. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-12966) Gossip thread slows down when using batch commit log
[ https://issues.apache.org/jira/browse/CASSANDRA-12966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Brown updated CASSANDRA-12966: Resolution: Fixed Status: Resolved (was: Patch Available) > Gossip thread slows down when using batch commit log > > > Key: CASSANDRA-12966 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12966 > Project: Cassandra > Issue Type: Improvement >Reporter: Jason Brown >Assignee: Jason Brown >Priority: Minor > > When using batch commit log mode, the Gossip thread slows down when peers > after a node bounces. This is because we perform a bunch of updates to the > peers table via {{SystemKeyspace.updatePeerInfo}}, which is a synchronized > method. How quickly each one of those individual updates takes depends on how > busy the system is at the time wrt write traffic. If the system is largely > quiescent, each update will be relatively quick (just waiting for the fsync). > If the system is getting a lot of writes, and depending on the > commitlog_sync_batch_window_in_ms, each of the Gossip thread's updates can > get stuck in the backlog, which causes the Gossip thread to stop processing. > We have observed in large clusters that a rolling restart causes triggers and > exacerbates this behavior. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-12966) Gossip thread slows down when using batch commit log
[ https://issues.apache.org/jira/browse/CASSANDRA-12966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paulo Motta updated CASSANDRA-12966: Reviewer: Stefan Podkowinski > Gossip thread slows down when using batch commit log > > > Key: CASSANDRA-12966 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12966 > Project: Cassandra > Issue Type: Improvement >Reporter: Jason Brown >Assignee: Jason Brown >Priority: Minor > > When using batch commit log mode, the Gossip thread slows down when peers > after a node bounces. This is because we perform a bunch of updates to the > peers table via {{SystemKeyspace.updatePeerInfo}}, which is a synchronized > method. How quickly each one of those individual updates takes depends on how > busy the system is at the time wrt write traffic. If the system is largely > quiescent, each update will be relatively quick (just waiting for the fsync). > If the system is getting a lot of writes, and depending on the > commitlog_sync_batch_window_in_ms, each of the Gossip thread's updates can > get stuck in the backlog, which causes the Gossip thread to stop processing. > We have observed in large clusters that a rolling restart causes triggers and > exacerbates this behavior. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CASSANDRA-12966) Gossip thread slows down when using batch commit log
[ https://issues.apache.org/jira/browse/CASSANDRA-12966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Brown updated CASSANDRA-12966: Status: Patch Available (was: Open) > Gossip thread slows down when using batch commit log > > > Key: CASSANDRA-12966 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12966 > Project: Cassandra > Issue Type: Bug >Reporter: Jason Brown >Assignee: Jason Brown >Priority: Minor > > When using batch commit log mode, the Gossip thread slows down when peers > after a node bounces. This is because we perform a bunch of updates to the > peers table via {{SystemKeyspace.updatePeerInfo}}, which is a synchronized > method. How quickly each one of those individual updates takes depends on how > busy the system is at the time wrt write traffic. If the system is largely > quiescent, each update will be relatively quick (just waiting for the fsync). > If the system is getting a lot of writes, and depending on the > commitlog_sync_batch_window_in_ms, each of the Gossip thread's updates can > get stuck in the backlog, which causes the Gossip thread to stop processing. > We have observed in large clusters that a rolling restart causes triggers and > exacerbates this behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)