[jira] [Commented] (CASSANDRA-13265) Expiration in OutboundTcpConnection can block the reader Thread
[ https://issues.apache.org/jira/browse/CASSANDRA-13265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15986426#comment-15986426 ] Christian Esken commented on CASSANDRA-13265: - I am fixing the branches, while you work on the dtests. I will continue updating this comment as long as I work on it. || branch || sqaushed? || Unit Tests OK? || comment || | cassandra-13265-3.0 | no | (CircleCI currently running) | | | cassandra-13265-3.11 | no | CircleCI (/) | | | cassandra-13265-2.2 | yes | ant test (/) | CicrleCI hasn't kicked off tests for the branch | | cassandra-13265-trunk | no | CircleCI (?) | My unit test works. Bu there is a strange unrelated unit test failure: ClassNotFoundException: org.apache.cassandra.stress.CompactionStress | > Expiration in OutboundTcpConnection can block the reader Thread > --- > > Key: CASSANDRA-13265 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13265 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 3.0.9 > Java HotSpot(TM) 64-Bit Server VM version 25.112-b15 (Java version > 1.8.0_112-b15) > Linux 3.16 >Reporter: Christian Esken >Assignee: Christian Esken > Fix For: 3.0.x > > Attachments: cassandra.pb-cache4-dus.2017-02-17-19-36-26.chist.xz, > cassandra.pb-cache4-dus.2017-02-17-19-36-26.td.xz > > > I observed that sometimes a single node in a Cassandra cluster fails to > communicate to the other nodes. This can happen at any time, during peak load > or low load. Restarting that single node from the cluster fixes the issue. > Before going in to details, I want to state that I have analyzed the > situation and am already developing a possible fix. Here is the analysis so > far: > - A Threaddump in this situation showed 324 Threads in the > OutboundTcpConnection class that want to lock the backlog queue for doing > expiration. > - A class histogram shows 262508 instances of > OutboundTcpConnection$QueuedMessage. > What is the effect of it? As soon as the Cassandra node has reached a certain > amount of queued messages, it starts thrashing itself to death. Each of the > Thread fully locks the Queue for reading and writing by calling > iterator.next(), making the situation worse and worse. > - Writing: Only after 262508 locking operation it can progress with actually > writing to the Queue. > - Reading: Is also blocked, as 324 Threads try to do iterator.next(), and > fully lock the Queue > This means: Writing blocks the Queue for reading, and readers might even be > starved which makes the situation even worse. > - > The setup is: > - 3-node cluster > - replication factor 2 > - Consistency LOCAL_ONE > - No remote DC's > - high write throughput (10 INSERT statements per second and more during > peak times). > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13265) Expiration in OutboundTcpConnection can block the reader Thread
[ https://issues.apache.org/jira/browse/CASSANDRA-13265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15986426#comment-15986426 ] Christian Esken edited comment on CASSANDRA-13265 at 4/27/17 11:53 AM: --- I am fixing the branches, while you work on the dtests. I will continue updating this comment as long as I work on it. || branch || sqaushed? || Unit Tests OK? || comment || | cassandra-13265-3.0 | no | (CircleCI currently running) | No stress-test in build.xml. I patched circle.yml to match that: https://github.com/christian-esken/cassandra/commit/1a776e299c76093eb3edf20e0d9054e14549a667 | | cassandra-13265-3.11 | no | CircleCI (/) | | | cassandra-13265-2.2 | yes | ant test (/) | CicrleCI hasn't kicked off tests for the branch | | cassandra-13265-trunk | no | CircleCI (?) | My unit test works. Bu there is a strange unrelated unit test failure: ClassNotFoundException: org.apache.cassandra.stress.CompactionStress | was (Author: cesken): I am fixing the branches, while you work on the dtests. I will continue updating this comment as long as I work on it. || branch || sqaushed? || Unit Tests OK? || comment || | cassandra-13265-3.0 | no | (CircleCI currently running) | | | cassandra-13265-3.11 | no | CircleCI (/) | | | cassandra-13265-2.2 | yes | ant test (/) | CicrleCI hasn't kicked off tests for the branch | | cassandra-13265-trunk | no | CircleCI (?) | My unit test works. Bu there is a strange unrelated unit test failure: ClassNotFoundException: org.apache.cassandra.stress.CompactionStress | > Expiration in OutboundTcpConnection can block the reader Thread > --- > > Key: CASSANDRA-13265 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13265 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 3.0.9 > Java HotSpot(TM) 64-Bit Server VM version 25.112-b15 (Java version > 1.8.0_112-b15) > Linux 3.16 >Reporter: Christian Esken >Assignee: Christian Esken > Fix For: 3.0.x > > Attachments: cassandra.pb-cache4-dus.2017-02-17-19-36-26.chist.xz, > cassandra.pb-cache4-dus.2017-02-17-19-36-26.td.xz > > > I observed that sometimes a single node in a Cassandra cluster fails to > communicate to the other nodes. This can happen at any time, during peak load > or low load. Restarting that single node from the cluster fixes the issue. > Before going in to details, I want to state that I have analyzed the > situation and am already developing a possible fix. Here is the analysis so > far: > - A Threaddump in this situation showed 324 Threads in the > OutboundTcpConnection class that want to lock the backlog queue for doing > expiration. > - A class histogram shows 262508 instances of > OutboundTcpConnection$QueuedMessage. > What is the effect of it? As soon as the Cassandra node has reached a certain > amount of queued messages, it starts thrashing itself to death. Each of the > Thread fully locks the Queue for reading and writing by calling > iterator.next(), making the situation worse and worse. > - Writing: Only after 262508 locking operation it can progress with actually > writing to the Queue. > - Reading: Is also blocked, as 324 Threads try to do iterator.next(), and > fully lock the Queue > This means: Writing blocks the Queue for reading, and readers might even be > starved which makes the situation even worse. > - > The setup is: > - 3-node cluster > - replication factor 2 > - Consistency LOCAL_ONE > - No remote DC's > - high write throughput (10 INSERT statements per second and more during > peak times). > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13265) Expiration in OutboundTcpConnection can block the reader Thread
[ https://issues.apache.org/jira/browse/CASSANDRA-13265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15986426#comment-15986426 ] Christian Esken edited comment on CASSANDRA-13265 at 4/27/17 1:24 PM: -- I am fixing the branches, while you work on the dtests. I will continue updating this comment as long as I work on it. || branch || sqaushed? || Unit Tests OK? || comment || | cassandra-13265-3.0 | no | (/) / (?) | No stress-test in build.xml. I patched circle.yml to match that: https://github.com/christian-esken/cassandra/commit/1a776e299c76093eb3edf20e0d9054e14549a667 . CircleCI still kicks off a 4th test, which fails but can likely be ignored for now. | | cassandra-13265-3.11 | no | CircleCI (/) | | | cassandra-13265-2.2 | yes | ant test (/) | CicrleCI hasn't kicked off tests for the branch | | cassandra-13265-trunk | no | CircleCI (?) | My unit test works. Bu there is a strange unrelated unit test failure: ClassNotFoundException: org.apache.cassandra.stress.CompactionStress | was (Author: cesken): I am fixing the branches, while you work on the dtests. I will continue updating this comment as long as I work on it. || branch || sqaushed? || Unit Tests OK? || comment || | cassandra-13265-3.0 | no | (CircleCI currently running) | No stress-test in build.xml. I patched circle.yml to match that: https://github.com/christian-esken/cassandra/commit/1a776e299c76093eb3edf20e0d9054e14549a667 | | cassandra-13265-3.11 | no | CircleCI (/) | | | cassandra-13265-2.2 | yes | ant test (/) | CicrleCI hasn't kicked off tests for the branch | | cassandra-13265-trunk | no | CircleCI (?) | My unit test works. Bu there is a strange unrelated unit test failure: ClassNotFoundException: org.apache.cassandra.stress.CompactionStress | > Expiration in OutboundTcpConnection can block the reader Thread > --- > > Key: CASSANDRA-13265 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13265 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 3.0.9 > Java HotSpot(TM) 64-Bit Server VM version 25.112-b15 (Java version > 1.8.0_112-b15) > Linux 3.16 >Reporter: Christian Esken >Assignee: Christian Esken > Fix For: 3.0.x > > Attachments: cassandra.pb-cache4-dus.2017-02-17-19-36-26.chist.xz, > cassandra.pb-cache4-dus.2017-02-17-19-36-26.td.xz > > > I observed that sometimes a single node in a Cassandra cluster fails to > communicate to the other nodes. This can happen at any time, during peak load > or low load. Restarting that single node from the cluster fixes the issue. > Before going in to details, I want to state that I have analyzed the > situation and am already developing a possible fix. Here is the analysis so > far: > - A Threaddump in this situation showed 324 Threads in the > OutboundTcpConnection class that want to lock the backlog queue for doing > expiration. > - A class histogram shows 262508 instances of > OutboundTcpConnection$QueuedMessage. > What is the effect of it? As soon as the Cassandra node has reached a certain > amount of queued messages, it starts thrashing itself to death. Each of the > Thread fully locks the Queue for reading and writing by calling > iterator.next(), making the situation worse and worse. > - Writing: Only after 262508 locking operation it can progress with actually > writing to the Queue. > - Reading: Is also blocked, as 324 Threads try to do iterator.next(), and > fully lock the Queue > This means: Writing blocks the Queue for reading, and readers might even be > starved which makes the situation even worse. > - > The setup is: > - 3-node cluster > - replication factor 2 > - Consistency LOCAL_ONE > - No remote DC's > - high write throughput (10 INSERT statements per second and more during > peak times). > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13265) Expiration in OutboundTcpConnection can block the reader Thread
[ https://issues.apache.org/jira/browse/CASSANDRA-13265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15986786#comment-15986786 ] Ariel Weisberg commented on CASSANDRA-13265: You don't need to get all the tests passing. Only the ones broken specifically by your changes. So you have to run the tests on the base branch (cassandra-2.2, cassandra-3.0, cassandra-3.11, cassandra-trunk). Compare the results to https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-2.2-test-all/ or the one specific to the version of your branch. You also need to be aware that a test might just be unreliable in CircleCI. The unit tests seem to be reliable to me when I was running them with my circle.yml, but I am not sure about the one on trunk. Certainly it runs additional tests which weren't running all the time before Circle and I don't think they were reliable when we started running them. > Expiration in OutboundTcpConnection can block the reader Thread > --- > > Key: CASSANDRA-13265 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13265 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 3.0.9 > Java HotSpot(TM) 64-Bit Server VM version 25.112-b15 (Java version > 1.8.0_112-b15) > Linux 3.16 >Reporter: Christian Esken >Assignee: Christian Esken > Fix For: 3.0.x > > Attachments: cassandra.pb-cache4-dus.2017-02-17-19-36-26.chist.xz, > cassandra.pb-cache4-dus.2017-02-17-19-36-26.td.xz > > > I observed that sometimes a single node in a Cassandra cluster fails to > communicate to the other nodes. This can happen at any time, during peak load > or low load. Restarting that single node from the cluster fixes the issue. > Before going in to details, I want to state that I have analyzed the > situation and am already developing a possible fix. Here is the analysis so > far: > - A Threaddump in this situation showed 324 Threads in the > OutboundTcpConnection class that want to lock the backlog queue for doing > expiration. > - A class histogram shows 262508 instances of > OutboundTcpConnection$QueuedMessage. > What is the effect of it? As soon as the Cassandra node has reached a certain > amount of queued messages, it starts thrashing itself to death. Each of the > Thread fully locks the Queue for reading and writing by calling > iterator.next(), making the situation worse and worse. > - Writing: Only after 262508 locking operation it can progress with actually > writing to the Queue. > - Reading: Is also blocked, as 324 Threads try to do iterator.next(), and > fully lock the Queue > This means: Writing blocks the Queue for reading, and readers might even be > starved which makes the situation even worse. > - > The setup is: > - 3-node cluster > - replication factor 2 > - Consistency LOCAL_ONE > - No remote DC's > - high write throughput (10 INSERT statements per second and more during > peak times). > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13265) Expiration in OutboundTcpConnection can block the reader Thread
[ https://issues.apache.org/jira/browse/CASSANDRA-13265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15986786#comment-15986786 ] Ariel Weisberg edited comment on CASSANDRA-13265 at 4/27/17 3:33 PM: - You don't need to get all the tests passing. Only the ones broken specifically by your changes. So you have to run the tests on the base branch (cassandra-2.2, cassandra-3.0, cassandra-3.11, cassandra-trunk) or compare the results to https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-2.2-test-all/ or the one specific to the version of your branch. You also need to be aware that a test might just be unreliable in CircleCI. The unit tests seem to be reliable to me when I was running them with my circle.yml, but I am not sure about the one on trunk. Certainly it runs additional tests which weren't running all the time before Circle and I don't think they were reliable when we started running them. was (Author: aweisberg): You don't need to get all the tests passing. Only the ones broken specifically by your changes. So you have to run the tests on the base branch (cassandra-2.2, cassandra-3.0, cassandra-3.11, cassandra-trunk). Compare the results to https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-2.2-test-all/ or the one specific to the version of your branch. You also need to be aware that a test might just be unreliable in CircleCI. The unit tests seem to be reliable to me when I was running them with my circle.yml, but I am not sure about the one on trunk. Certainly it runs additional tests which weren't running all the time before Circle and I don't think they were reliable when we started running them. > Expiration in OutboundTcpConnection can block the reader Thread > --- > > Key: CASSANDRA-13265 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13265 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 3.0.9 > Java HotSpot(TM) 64-Bit Server VM version 25.112-b15 (Java version > 1.8.0_112-b15) > Linux 3.16 >Reporter: Christian Esken >Assignee: Christian Esken > Fix For: 3.0.x > > Attachments: cassandra.pb-cache4-dus.2017-02-17-19-36-26.chist.xz, > cassandra.pb-cache4-dus.2017-02-17-19-36-26.td.xz > > > I observed that sometimes a single node in a Cassandra cluster fails to > communicate to the other nodes. This can happen at any time, during peak load > or low load. Restarting that single node from the cluster fixes the issue. > Before going in to details, I want to state that I have analyzed the > situation and am already developing a possible fix. Here is the analysis so > far: > - A Threaddump in this situation showed 324 Threads in the > OutboundTcpConnection class that want to lock the backlog queue for doing > expiration. > - A class histogram shows 262508 instances of > OutboundTcpConnection$QueuedMessage. > What is the effect of it? As soon as the Cassandra node has reached a certain > amount of queued messages, it starts thrashing itself to death. Each of the > Thread fully locks the Queue for reading and writing by calling > iterator.next(), making the situation worse and worse. > - Writing: Only after 262508 locking operation it can progress with actually > writing to the Queue. > - Reading: Is also blocked, as 324 Threads try to do iterator.next(), and > fully lock the Queue > This means: Writing blocks the Queue for reading, and readers might even be > starved which makes the situation even worse. > - > The setup is: > - 3-node cluster > - replication factor 2 > - Consistency LOCAL_ONE > - No remote DC's > - high write throughput (10 INSERT statements per second and more during > peak times). > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13418) Allow TWCS to ignore overlaps when dropping fully expired sstables
[ https://issues.apache.org/jira/browse/CASSANDRA-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15986853#comment-15986853 ] Jon Haddad commented on CASSANDRA-13418: I get the desire to guard users to not shoot themselves in the foot, but this is a pretty reasonable approach to solving the problem. The tradeoff is a bit of risk. There's been a lot of proposed features like these that are in the "great if you know what you're doing but crazy dangerous otherwise" category. I feel like these features should be thoroughly documented and guarded behind a config flag that's disabled by default. > Allow TWCS to ignore overlaps when dropping fully expired sstables > -- > > Key: CASSANDRA-13418 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13418 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Corentin Chary > Labels: twcs > > http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html explains it well. If > you really want read-repairs you're going to have sstables blocking the > expiration of other fully expired SSTables because they overlap. > You can set unchecked_tombstone_compaction = true or tombstone_threshold to a > very low value and that will purge the blockers of old data that should > already have expired, thus removing the overlaps and allowing the other > SSTables to expire. > The thing is that this is rather CPU intensive and not optimal. If you have > time series, you might not care if all your data doesn't exactly expire at > the right time, or if data re-appears for some time, as long as it gets > deleted as soon as it can. And in this situation I believe it would be really > beneficial to allow users to simply ignore overlapping SSTables when looking > for fully expired ones. > To the question: why would you need read-repairs ? > - Full repairs basically take longer than the TTL of the data on my dataset, > so this isn't really effective. > - Even with a 10% chances of doing a repair, we found out that this would be > enough to greatly reduce entropy of the most used data (and if you have > timeseries, you're likely to have a dashboard doing the same important > queries over and over again). > - LOCAL_QUORUM is too expensive (need >3 replicas), QUORUM is too slow. > I'll try to come up with a patch demonstrating how this would work, try it on > our system and report the effects. > cc: [~adejanovski], [~rgerard] as I know you worked on similar issues already. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13418) Allow TWCS to ignore overlaps when dropping fully expired sstables
[ https://issues.apache.org/jira/browse/CASSANDRA-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987267#comment-15987267 ] Jeff Jirsa commented on CASSANDRA-13418: I think Marcus' concern is valid, but having run TWCS in production for a long time, I really wish we just had a dangerous-sounding option that defaulted into a safe state that would let append-only users ignore overlaps when they want to drop sstables. Adding code to flush read repaired data to different sstables is a lot more invasive, and would require follow-up changes to TWCS (so as not to try to immediately recompact those sstables with the larger post-window-major large final sstables). We talked about doing this in 9666 (in fact, we committed to it as a condition of merging TWCS), and I think it's probably a perfectly reasonable thing to do, but it's a lot more effort than simply telling cassandra "this table has no deletes, we don't care about overlaps". Maybe the right thing is to get 9779 done so we can block deletes, and then this is a much-less-scary option? > Allow TWCS to ignore overlaps when dropping fully expired sstables > -- > > Key: CASSANDRA-13418 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13418 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Corentin Chary > Labels: twcs > > http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html explains it well. If > you really want read-repairs you're going to have sstables blocking the > expiration of other fully expired SSTables because they overlap. > You can set unchecked_tombstone_compaction = true or tombstone_threshold to a > very low value and that will purge the blockers of old data that should > already have expired, thus removing the overlaps and allowing the other > SSTables to expire. > The thing is that this is rather CPU intensive and not optimal. If you have > time series, you might not care if all your data doesn't exactly expire at > the right time, or if data re-appears for some time, as long as it gets > deleted as soon as it can. And in this situation I believe it would be really > beneficial to allow users to simply ignore overlapping SSTables when looking > for fully expired ones. > To the question: why would you need read-repairs ? > - Full repairs basically take longer than the TTL of the data on my dataset, > so this isn't really effective. > - Even with a 10% chances of doing a repair, we found out that this would be > enough to greatly reduce entropy of the most used data (and if you have > timeseries, you're likely to have a dashboard doing the same important > queries over and over again). > - LOCAL_QUORUM is too expensive (need >3 replicas), QUORUM is too slow. > I'll try to come up with a patch demonstrating how this would work, try it on > our system and report the effects. > cc: [~adejanovski], [~rgerard] as I know you worked on similar issues already. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13418) Allow TWCS to ignore overlaps when dropping fully expired sstables
[ https://issues.apache.org/jira/browse/CASSANDRA-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987267#comment-15987267 ] Jeff Jirsa edited comment on CASSANDRA-13418 at 4/27/17 6:56 PM: - I think Marcus' concern is valid, but having run TWCS in production for a long time, I really wish we just had a dangerous-sounding option that defaulted into a safe state that would let append-only users ignore overlaps when they want to drop sstables. Adding code to flush/split read repaired data to different sstables is a lot more invasive, and would require follow-up changes to TWCS (so as not to try to immediately recompact those sstables with the larger post-window-major large final sstables). We talked about doing this in 9666 (in fact, we committed to it as a condition of merging TWCS), and I think it's probably a perfectly reasonable thing to do, but it's a lot more effort than simply telling cassandra "this table has no deletes, we don't care about overlaps". Maybe the right thing is to get 9779 done so we can block deletes, and then this is a much-less-scary option? was (Author: jjirsa): I think Marcus' concern is valid, but having run TWCS in production for a long time, I really wish we just had a dangerous-sounding option that defaulted into a safe state that would let append-only users ignore overlaps when they want to drop sstables. Adding code to flush read repaired data to different sstables is a lot more invasive, and would require follow-up changes to TWCS (so as not to try to immediately recompact those sstables with the larger post-window-major large final sstables). We talked about doing this in 9666 (in fact, we committed to it as a condition of merging TWCS), and I think it's probably a perfectly reasonable thing to do, but it's a lot more effort than simply telling cassandra "this table has no deletes, we don't care about overlaps". Maybe the right thing is to get 9779 done so we can block deletes, and then this is a much-less-scary option? > Allow TWCS to ignore overlaps when dropping fully expired sstables > -- > > Key: CASSANDRA-13418 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13418 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Corentin Chary > Labels: twcs > > http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html explains it well. If > you really want read-repairs you're going to have sstables blocking the > expiration of other fully expired SSTables because they overlap. > You can set unchecked_tombstone_compaction = true or tombstone_threshold to a > very low value and that will purge the blockers of old data that should > already have expired, thus removing the overlaps and allowing the other > SSTables to expire. > The thing is that this is rather CPU intensive and not optimal. If you have > time series, you might not care if all your data doesn't exactly expire at > the right time, or if data re-appears for some time, as long as it gets > deleted as soon as it can. And in this situation I believe it would be really > beneficial to allow users to simply ignore overlapping SSTables when looking > for fully expired ones. > To the question: why would you need read-repairs ? > - Full repairs basically take longer than the TTL of the data on my dataset, > so this isn't really effective. > - Even with a 10% chances of doing a repair, we found out that this would be > enough to greatly reduce entropy of the most used data (and if you have > timeseries, you're likely to have a dashboard doing the same important > queries over and over again). > - LOCAL_QUORUM is too expensive (need >3 replicas), QUORUM is too slow. > I'll try to come up with a patch demonstrating how this would work, try it on > our system and report the effects. > cc: [~adejanovski], [~rgerard] as I know you worked on similar issues already. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-10263) Insert in Cassandra in a collection, values with single quote
[
https://issues.apache.org/jira/browse/CASSANDRA-10263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987282#comment-15987282
]
Jamie Hankins commented on CASSANDRA-10263:
---
By that logic, why isn't the JSON result also escaped?
{code}
cqlsh:gii> insert into test (name,values) values ('Tu''pac',{'fo''o':'josemy''s
house'});
cqlsh:gii> select * from test;
name| values
-+--
Tu'pac | {'fo''o': 'josemy''s house'}
cqlsh:gii> select json * from test;
[json]
--
{"name": "Tu'pac", "values": {"fo'o": "josemy's house"}}
{code}
The easiest thing is just to say "by design", but returning the escaped
single-quotes is counter to any other DBMS.
> Insert in Cassandra in a collection, values with single quote
> -
>
> Key: CASSANDRA-10263
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10263
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jduarte
>Assignee: Benjamin Lerer
> Attachments: terminal.png
>
>
> I think there is a problem with single quotes in collection fields. This is
> what I try:
> Table: {{CREATE TABLE test (name text, foo map, PRIMARY
> KEY(name));}}
> Insert: {code}INSERT INTO test (name, foo) VALUES (
> 'jos''emy',{'foo':'josemy''s houñse'});{code}
> Query: {{SELECT * FROM test;}}
> {code}
> name| foo
> ---+-
> jos'emy | {'foo': 'josemy''s houñse'}
> {code}
> As you can see, Cassandra replace the double quote in the simple "name"
> field, but does not replace the quotes in the field inside the map. The
> result (I think) should be:
> {code}
> name| foo
> -+-
> jos'emy | {'foo': 'josemy's houñse'}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13418) Allow TWCS to ignore overlaps when dropping fully expired sstables
[ https://issues.apache.org/jira/browse/CASSANDRA-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987304#comment-15987304 ] Jonathan Owens commented on CASSANDRA-13418: We did exactly this about a year ago for our timeseries use case, and it works great. Made compactions a whole lot more efficient. Here's how we did it: https://github.com/newrelic-forks/cassandra/pull/8 > Allow TWCS to ignore overlaps when dropping fully expired sstables > -- > > Key: CASSANDRA-13418 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13418 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Corentin Chary > Labels: twcs > > http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html explains it well. If > you really want read-repairs you're going to have sstables blocking the > expiration of other fully expired SSTables because they overlap. > You can set unchecked_tombstone_compaction = true or tombstone_threshold to a > very low value and that will purge the blockers of old data that should > already have expired, thus removing the overlaps and allowing the other > SSTables to expire. > The thing is that this is rather CPU intensive and not optimal. If you have > time series, you might not care if all your data doesn't exactly expire at > the right time, or if data re-appears for some time, as long as it gets > deleted as soon as it can. And in this situation I believe it would be really > beneficial to allow users to simply ignore overlapping SSTables when looking > for fully expired ones. > To the question: why would you need read-repairs ? > - Full repairs basically take longer than the TTL of the data on my dataset, > so this isn't really effective. > - Even with a 10% chances of doing a repair, we found out that this would be > enough to greatly reduce entropy of the most used data (and if you have > timeseries, you're likely to have a dashboard doing the same important > queries over and over again). > - LOCAL_QUORUM is too expensive (need >3 replicas), QUORUM is too slow. > I'll try to come up with a patch demonstrating how this would work, try it on > our system and report the effects. > cc: [~adejanovski], [~rgerard] as I know you worked on similar issues already. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12576) Setting read_request_timeout_in_ms more than one day causes login fails
[
https://issues.apache.org/jira/browse/CASSANDRA-12576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987348#comment-15987348
]
Varun Gupta commented on CASSANDRA-12576:
-
I am able to cqlsh with read timeout value of 5. I am using version
3.0.11.
Which version of cassandra are you using?
> Setting read_request_timeout_in_ms more than one day causes login fails
> ---
>
> Key: CASSANDRA-12576
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12576
> Project: Cassandra
> Issue Type: Bug
> Components: Configuration
> Environment: Distro: "Debian GNU/Linux 8 (jessie)"
> cqlsh -u kartaca -p kartaca -e "select release_version from system.local;" =>
> 3.5
>
>Reporter: Erol GUZOĞLU
>Priority: Trivial
>
> I try to run "select count( * )" query
> To make query success, I set read_request_timeout_in_ms random big number
> then restart the service.
> When node is UN, login fails started.
> When I try to login cassandra, I came up with this irrelevant error.
> Connection error: ('Unable to connect to any servers', {'127.0.0.1':
> error(111, "Tried connecting to [('127.0.0.1',│
> 9042)]. Last error: Connection refused")})
> It probably occur when setting number bigger than one day. Because when I set
> 5 (~5.7 day) can not login but if i set 5000 (~0.5 day) login
> success.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-10263) Insert in Cassandra in a collection, values with single quote
[
https://issues.apache.org/jira/browse/CASSANDRA-10263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987282#comment-15987282
]
Jamie Hankins edited comment on CASSANDRA-10263 at 4/27/17 7:26 PM:
By that logic, why isn't the JSON result also escaped?
{code}
cqlsh:gii> insert into test (name,values) values ('Tu''pac',{'fo''o':'josemy''s
house'});
cqlsh:gii> select * from test;
name| values
-+--
Tu'pac | {'fo''o': 'josemy''s house'}
cqlsh:gii> select json * from test;
[json]
--
{"name": "Tu'pac", "values": {"fo'o": "josemy's house"}}
{code}
The easiest thing is just to say "by design", but returning the escaped
single-quotes is counter to any other DBMS.
As an aside, PostgreSQL doesn't do maps, but here's what an array looks like:
{code}
derms=# create table test ( name text, items text[] );
CREATE TABLE
derms=# insert into test (name,items) values ('Fred',ARRAY['item''s
delight','Jason''s house']);
INSERT 0 1
derms=# select * from test;
name | items
--+
Fred | {"item's delight","Jason's house"}
{code}
was (Author: jamiehankins):
By that logic, why isn't the JSON result also escaped?
{code}
cqlsh:gii> insert into test (name,values) values ('Tu''pac',{'fo''o':'josemy''s
house'});
cqlsh:gii> select * from test;
name| values
-+--
Tu'pac | {'fo''o': 'josemy''s house'}
cqlsh:gii> select json * from test;
[json]
--
{"name": "Tu'pac", "values": {"fo'o": "josemy's house"}}
{code}
The easiest thing is just to say "by design", but returning the escaped
single-quotes is counter to any other DBMS.
> Insert in Cassandra in a collection, values with single quote
> -
>
> Key: CASSANDRA-10263
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10263
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jduarte
>Assignee: Benjamin Lerer
> Attachments: terminal.png
>
>
> I think there is a problem with single quotes in collection fields. This is
> what I try:
> Table: {{CREATE TABLE test (name text, foo map, PRIMARY
> KEY(name));}}
> Insert: {code}INSERT INTO test (name, foo) VALUES (
> 'jos''emy',{'foo':'josemy''s houñse'});{code}
> Query: {{SELECT * FROM test;}}
> {code}
> name| foo
> ---+-
> jos'emy | {'foo': 'josemy''s houñse'}
> {code}
> As you can see, Cassandra replace the double quote in the simple "name"
> field, but does not replace the quotes in the field inside the map. The
> result (I think) should be:
> {code}
> name| foo
> -+-
> jos'emy | {'foo': 'josemy's houñse'}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13444) Fast and garbage-free Streaming Histogram
[ https://issues.apache.org/jira/browse/CASSANDRA-13444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-13444: --- Fix Version/s: 4.x > Fast and garbage-free Streaming Histogram > - > > Key: CASSANDRA-13444 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13444 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Fuud > Fix For: 4.x > > Attachments: results.csv, results.xlsx > > > StreamingHistogram is cause of high cpu usage and GC pressure. > It was improved at CASSANDRA-13038 by introducing intermediate buffer to try > accumulate different values into the big map before merging them into smaller > one. > But there was not enought for TTL's distributed within large time. Rounding > (also introduced at 13038) can help but it reduce histogram precision > specially in case where TTL's does not distributed uniformly. > There are several improvements that can help to reduce cpu and gc usage. Them > all included in the pool-request as separate revisions thus you can test them > independently. > Improvements list: > # Use Map.computeIfAbsent instead of get->checkIfNull->put chain. In this way > "add-or-accumulate" operation takes one map operation instead of two. But > this method (default-defined in interface Map) is overriden in HashMap but > not in TreeMap. Thus I changed spool type to HashMap. > # As we round incoming values to _roundSeconds_ we can also round value on > merge. It will enlarge hit rate for bin operations. > # Because we inserted only integers into Histogram and rounding values to > integers we can use *int* type everywhere. > # Histogram takes huge amount of time merging values. In merge method largest > amount of time taken by finding nearest points. It can be eliminated by > holding additional TreeSet with differences, sorted from smalest to greatest. > # Because we know max size of _bin_ and _differences_ maps we can replace > them with sorted arrays. Search can be done with _Arrays.binarySearch_ and > insertion/deletions can be done by _System.arraycopy_. Also it helps to merge > some operations into one. > # Because spool map is also limited we can replace it with open address > primitive map. It's finaly reduce allocation rate to zero. > You can see gain given by each step in the attached file. First number is > time for one benchmark invocation and second - is allocation rate in Mb per > operation. > Dependends of payload time is reduced up to 90%. > Overall gain: > |.|.|Payload/SpoolSize|.|.|.|% from original > |.|.|.|original|.|optimized| > |.|.|secondInMonth/0|.|.|.| > |time ms/op|.|.|10747,684|.|5545,063|51,6 > |allocation Mb/op|.|.|2441,38858|.|0,002105713|0 > |.|.|.|.|.|.| > |.|.|secondInMonth/1000|.|.|.| > |time ms/op|.|.|8988,578|.|5791,179|64,4 > |allocation Mb/op|.|.|2440,951141|.|0,017715454|0 > |.|.|.|.|.|.| > |.|.|secondInMonth/1|.|.|.| > |time ms/op|.|.|10711,671|.|5765,243|53,8 > |allocation Mb/op|.|.|2437,022537|.|0,264083862|0 > |.|.|.|.|.|.| > |.|.|secondInMonth/10|.|.|.| > |time ms/op|.|.|13001,841|.|5638,069|43,4 > |allocation Mb/op|.|.|2396,947113|.|2,003662109|0,1 > |.|.|.|.|.|.| > |.|.|secondInDay/0|.|.|.| > |time ms/op|.|.|10381,833|.|5497,804|53 > |allocation Mb/op|.|.|2441,166107|.|0,002105713|0 > |.|.|.|.|.|.| > |.|.|secondInDay/1000|.|.|.| > |time ms/op|.|.|8522,157|.|5929,871|69,6 > |allocation Mb/op|.|.|1973,112381|.|0,017715454|0 > |.|.|.|.|.|.| > |.|.|secondInDay/1|.|.|.| > |time ms/op|.|.|10234,978|.|5480,077|53,5 > |allocation Mb/op|.|.|2306,057404|.|0,262969971|0 > |.|.|.|.|.|.| > |.|.|secondInDay/10|.|.|.| > |time ms/op|.|.|2971,178|.|139,079|4,7 > |allocation Mb/op|.|.|172,1276245|.|2,001721191|1,2 > |.|.|.|.|.|.| > |.|.|secondIn3Hour/0|.|.|.| > |time ms/op|.|.|10663,123|.|5605,672|52,6 > |allocation Mb/op|.|.|2439,456818|.|0,002105713|0 > |.|.|.|.|.|.| > |.|.|secondIn3Hour/1000|.|.|.| > |time ms/op|.|.|9029,788|.|5838,618|64,7 > |allocation Mb/op|.|.|2331,839249|.|0,180664063|0 > |.|.|.|.|.|.| > |.|.|secondIn3Hour/1|.|.|.| > |time ms/op|.|.|4862,409|.|89,001|1,8 > |allocation Mb/op|.|.|965,4871887|.|0,251711652|0 > |.|.|.|.|.|.| > |.|.|secondIn3Hour/10|.|.|.| > |time ms/op|.|.|1484,454|.|95,044|6,4 > |allocation Mb/op|.|.|153,2464722|.|2,001712809|1,3 > |.|.|.|.|.|.| > |.|.|secondInMin/0|.|.|.| > |time ms/op|.|.|875,118|.|424,11|48,5 > |allocation Mb/op|.|.|610,3554993|.|0,001776123|0 > |.|.|.|.|.|.| > |.|.|secondInMin/1000|.|.|.| > |time ms/op|.|.|568,7|.|84,208|14,8 > |allocation Mb/op|.|.|0,007598114|.|0,01810023|238,2 > |.|.|.|.|.|.| > |.|.|secondInMin/1|.|.|.| > |time ms/op|.|.|573,595|.|83,862|14,6 > |allocation Mb/op|.|.|0,007597351|.|0,252473872|3323,2 > |.|.|.|.|.|.| > |.|.|secondInMin/10|.|.|.| > |time ms/op|.|.|584,457|.|86,5
[jira] [Assigned] (CASSANDRA-13346) Failed unregistering mbean during drop keyspace
[
https://issues.apache.org/jira/browse/CASSANDRA-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lerh Chuan Low reassigned CASSANDRA-13346:
--
Assignee: Lerh Chuan Low
> Failed unregistering mbean during drop keyspace
> ---
>
> Key: CASSANDRA-13346
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13346
> Project: Cassandra
> Issue Type: Bug
> Components: Materialized Views
> Environment: Cassandra 3.9
>Reporter: Gábor Auth
>Assignee: Lerh Chuan Low
>Priority: Minor
> Labels: lhf
> Fix For: 3.0.x, 3.11.x
>
> Attachments: 13346-3.0.X.txt, 13346-3.X.txt
>
>
> All node throw exceptions about materialized views during drop keyspace:
> {code}
> WARN [MigrationStage:1] 2017-03-16 16:54:25,016 ColumnFamilyStore.java:535 -
> Failed unregistering mbean:
> org.apache.cassandra.db:type=Tables,keyspace=test20160810,table=unit_by_account
> java.lang.NullPointerException: null
> at
> java.util.concurrent.ConcurrentHashMap.replaceNode(ConcurrentHashMap.java:1106)
> ~[na:1.8.0_121]
> at
> java.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:1097)
> ~[na:1.8.0_121]
> at
> java.util.concurrent.ConcurrentHashMap$KeySetView.remove(ConcurrentHashMap.java:4569)
> ~[na:1.8.0_121]
> at
> org.apache.cassandra.metrics.TableMetrics.release(TableMetrics.java:712)
> ~[apache-cassandra-3.9.0.jar:3.9.0]
> at
> org.apache.cassandra.db.ColumnFamilyStore.unregisterMBean(ColumnFamilyStore.java:570)
> [apache-cassandra-3.9.0.jar:3.9.0]
> at
> org.apache.cassandra.db.ColumnFamilyStore.invalidate(ColumnFamilyStore.java:527)
> [apache-cassandra-3.9.0.jar:3.9.0]
> at
> org.apache.cassandra.db.ColumnFamilyStore.invalidate(ColumnFamilyStore.java:517)
> [apache-cassandra-3.9.0.jar:3.9.0]
> at org.apache.cassandra.db.Keyspace.unloadCf(Keyspace.java:365)
> [apache-cassandra-3.9.0.jar:3.9.0]
> at org.apache.cassandra.db.Keyspace.dropCf(Keyspace.java:358)
> [apache-cassandra-3.9.0.jar:3.9.0]
> at org.apache.cassandra.config.Schema.dropView(Schema.java:744)
> [apache-cassandra-3.9.0.jar:3.9.0]
> at
> org.apache.cassandra.schema.SchemaKeyspace.lambda$mergeSchema$373(SchemaKeyspace.java:1287)
> [apache-cassandra-3.9.0.jar:3.9.0]
> at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_121]
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1287)
> [apache-cassandra-3.9.0.jar:3.9.0]
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1256)
> [apache-cassandra-3.9.0.jar:3.9.0]
> at
> org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:51)
> ~[apache-cassandra-3.9.0.jar:3.9.0]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.9.0.jar:3.9.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_121]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[na:1.8.0_121]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-13480) nodetool repair can hang forever if we lose the notification for the repair completing/failing
Matt Byrd created CASSANDRA-13480: - Summary: nodetool repair can hang forever if we lose the notification for the repair completing/failing Key: CASSANDRA-13480 URL: https://issues.apache.org/jira/browse/CASSANDRA-13480 Project: Cassandra Issue Type: Bug Components: Tools Reporter: Matt Byrd Assignee: Matt Byrd Priority: Minor Fix For: 4.x When a Jmx lost notification occurs, sometimes the lost notification in question is the notification which let's RepairRunner know that the repair is finished (ProgressEventType.COMPLETE or even ERROR for that matter). This results in nodetool process running the repair hanging forever. I have a test which reproduces the issue here: https://github.com/Jollyplum/cassandra-dtest/tree/repair_hang_test To fix this, If on receiving a notification that notifications have been lost (JMXConnectionNotification.NOTIFS_LOST), we instead query a new endpoint via Jmx to receive all the relevant notifications we're interested in, we can replay those we missed and avoid this scenario. It's possible also that the JMXConnectionNotification.NOTIFS_LOST itself might be lost and so for good measure I have made RepairRunner poll periodically to see if there were any notifications that had been sent but we didn't receive (scoped just to the particular tag for the given repair). Users who don't use nodetool but go via jmx directly, can still use this new endpoint and implement similar behaviour in their clients as desired. I'm also expiring the notifications which have been kept on the server side. Please let me know if you've any questions or can think of a different approach, I also tried setting: JVM_OPTS="$JVM_OPTS -Djmx.remote.x.notification.buffer.size=5000" but this didn't fix the test. I suppose it might help under certain scenarios but in this test we don't even send that many notifications so I'm not surprised it doesn't fix it. It seems like getting lost notifications is always a potential problem with jmx as far as I can tell. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-12849) The parameter -XX:HeapDumpPath is not ovewritten by cassandra-env.sh
[
https://issues.apache.org/jira/browse/CASSANDRA-12849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yogesh Nachnani updated CASSANDRA-12849:
Reproduced In: 3.10, 2.1.14 (was: 2.1.14)
Status: Awaiting Feedback (was: Open)
> The parameter -XX:HeapDumpPath is not ovewritten by cassandra-env.sh
> -
>
> Key: CASSANDRA-12849
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12849
> Project: Cassandra
> Issue Type: Bug
>Reporter: jean carlo rivera ura
> Labels: lhf
> Attachments: 12849-trunk.txt
>
>
> The parameter -XX:HeapDumpPath appears twice in the java process
> {panel}
> user@node:~$ sudo ps aux | grep --color HeapDumpPath
> java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar
> -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities
> -XX:ThreadPriorityPolicy=42 -Xms1024M -Xmx1024M -Xmn200M
> -XX:+HeapDumpOnOutOfMemoryError
> -XX:*HeapDumpPath*=/var/lib/cassandra-1477577769-pid1516.hprof -Xss256k
> ...
> -XX:*HeapDumpPath*=/home/cassandra/java_1477577769.hprof
> -XX:ErrorFile=/var/lib/cassandra/hs_err_1477577769.log
> org.apache.cassandra.service.CassandraDaemon
> {panel}
> The problem is when we have an OOM error, the JVM dump goes to
> */home/cassandra/java_1477577769.hprof * when the correct behavior is to go
> to the path defined by cassandra-env.sh
> */var/lib/cassandra-1477577769-pid1516.hprof*
> This is quite annoying because cassandra takes into account only the path
> defined by the script init (usually that disk is not that big to keep 8Gb of
> a heap dump) and not the path defined in cassandra-env.sh
> {noformat}
> user@node:~$ jmx4perl http://localhost:8523/jolokia read
> com.sun.management:type=HotSpotDiagnostic DiagnosticOptions
> {
> name => 'HeapDumpPath',
> origin => 'VM_CREATION',
> value => '/home/cassandra/java_1477043835.hprof',
> writeable => '[true]'
> },
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-12849) The parameter -XX:HeapDumpPath is not ovewritten by cassandra-env.sh
[
https://issues.apache.org/jira/browse/CASSANDRA-12849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yogesh Nachnani updated CASSANDRA-12849:
Attachment: 12849-trunk.txt
It seems the user would have used the -H option while running bin/cassandra The
-H option is processed after processing cassandra-env.sh ; hence-fence the
override behaviour.
[~carlo_4002] - Do let me know exactly how your process is started, so we can
eliminate any other remaining bugs.
Also changed the documentation to nudge users to use cassandra-env.sh rather
than specifying JVM_OPTS on the command line
> The parameter -XX:HeapDumpPath is not ovewritten by cassandra-env.sh
> -
>
> Key: CASSANDRA-12849
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12849
> Project: Cassandra
> Issue Type: Bug
>Reporter: jean carlo rivera ura
> Labels: lhf
> Attachments: 12849-trunk.txt
>
>
> The parameter -XX:HeapDumpPath appears twice in the java process
> {panel}
> user@node:~$ sudo ps aux | grep --color HeapDumpPath
> java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar
> -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities
> -XX:ThreadPriorityPolicy=42 -Xms1024M -Xmx1024M -Xmn200M
> -XX:+HeapDumpOnOutOfMemoryError
> -XX:*HeapDumpPath*=/var/lib/cassandra-1477577769-pid1516.hprof -Xss256k
> ...
> -XX:*HeapDumpPath*=/home/cassandra/java_1477577769.hprof
> -XX:ErrorFile=/var/lib/cassandra/hs_err_1477577769.log
> org.apache.cassandra.service.CassandraDaemon
> {panel}
> The problem is when we have an OOM error, the JVM dump goes to
> */home/cassandra/java_1477577769.hprof * when the correct behavior is to go
> to the path defined by cassandra-env.sh
> */var/lib/cassandra-1477577769-pid1516.hprof*
> This is quite annoying because cassandra takes into account only the path
> defined by the script init (usually that disk is not that big to keep 8Gb of
> a heap dump) and not the path defined in cassandra-env.sh
> {noformat}
> user@node:~$ jmx4perl http://localhost:8523/jolokia read
> com.sun.management:type=HotSpotDiagnostic DiagnosticOptions
> {
> name => 'HeapDumpPath',
> origin => 'VM_CREATION',
> value => '/home/cassandra/java_1477043835.hprof',
> writeable => '[true]'
> },
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-12849) The parameter -XX:HeapDumpPath is not ovewritten by cassandra-env.sh
[
https://issues.apache.org/jira/browse/CASSANDRA-12849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yogesh Nachnani updated CASSANDRA-12849:
Reproduced In: 3.10, 2.1.14 (was: 2.1.14, 3.10)
Status: Patch Available (was: Awaiting Feedback)
> The parameter -XX:HeapDumpPath is not ovewritten by cassandra-env.sh
> -
>
> Key: CASSANDRA-12849
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12849
> Project: Cassandra
> Issue Type: Bug
>Reporter: jean carlo rivera ura
> Labels: lhf
> Attachments: 12849-trunk.txt
>
>
> The parameter -XX:HeapDumpPath appears twice in the java process
> {panel}
> user@node:~$ sudo ps aux | grep --color HeapDumpPath
> java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar
> -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities
> -XX:ThreadPriorityPolicy=42 -Xms1024M -Xmx1024M -Xmn200M
> -XX:+HeapDumpOnOutOfMemoryError
> -XX:*HeapDumpPath*=/var/lib/cassandra-1477577769-pid1516.hprof -Xss256k
> ...
> -XX:*HeapDumpPath*=/home/cassandra/java_1477577769.hprof
> -XX:ErrorFile=/var/lib/cassandra/hs_err_1477577769.log
> org.apache.cassandra.service.CassandraDaemon
> {panel}
> The problem is when we have an OOM error, the JVM dump goes to
> */home/cassandra/java_1477577769.hprof * when the correct behavior is to go
> to the path defined by cassandra-env.sh
> */var/lib/cassandra-1477577769-pid1516.hprof*
> This is quite annoying because cassandra takes into account only the path
> defined by the script init (usually that disk is not that big to keep 8Gb of
> a heap dump) and not the path defined in cassandra-env.sh
> {noformat}
> user@node:~$ jmx4perl http://localhost:8523/jolokia read
> com.sun.management:type=HotSpotDiagnostic DiagnosticOptions
> {
> name => 'HeapDumpPath',
> origin => 'VM_CREATION',
> value => '/home/cassandra/java_1477043835.hprof',
> writeable => '[true]'
> },
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-12849) The parameter -XX:HeapDumpPath is not ovewritten by cassandra-env.sh
[
https://issues.apache.org/jira/browse/CASSANDRA-12849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988033#comment-15988033
]
Yogesh Nachnani edited comment on CASSANDRA-12849 at 4/28/17 1:53 AM:
--
It seems the user would have used the -H option while running bin/cassandra The
-H option is processed after processing cassandra-env.sh ; hence-fence the
override behaviour.
The patch deprecates -H & -E options in bin/cassandra.
AFAIK, these are not documented anywhere in the source or online - so
hopefully, it won't break things for a lot of users
[~carlo_4002] - Do let me know exactly how your process is started, so we can
eliminate any other remaining bugs.
Also changed the documentation to nudge users to use cassandra-env.sh rather
than specifying JVM_OPTS on the command line
was (Author: yogeshnachnani):
It seems the user would have used the -H option while running bin/cassandra The
-H option is processed after processing cassandra-env.sh ; hence-fence the
override behaviour.
[~carlo_4002] - Do let me know exactly how your process is started, so we can
eliminate any other remaining bugs.
Also changed the documentation to nudge users to use cassandra-env.sh rather
than specifying JVM_OPTS on the command line
> The parameter -XX:HeapDumpPath is not ovewritten by cassandra-env.sh
> -
>
> Key: CASSANDRA-12849
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12849
> Project: Cassandra
> Issue Type: Bug
>Reporter: jean carlo rivera ura
> Labels: lhf
> Attachments: 12849-trunk.txt
>
>
> The parameter -XX:HeapDumpPath appears twice in the java process
> {panel}
> user@node:~$ sudo ps aux | grep --color HeapDumpPath
> java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar
> -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities
> -XX:ThreadPriorityPolicy=42 -Xms1024M -Xmx1024M -Xmn200M
> -XX:+HeapDumpOnOutOfMemoryError
> -XX:*HeapDumpPath*=/var/lib/cassandra-1477577769-pid1516.hprof -Xss256k
> ...
> -XX:*HeapDumpPath*=/home/cassandra/java_1477577769.hprof
> -XX:ErrorFile=/var/lib/cassandra/hs_err_1477577769.log
> org.apache.cassandra.service.CassandraDaemon
> {panel}
> The problem is when we have an OOM error, the JVM dump goes to
> */home/cassandra/java_1477577769.hprof * when the correct behavior is to go
> to the path defined by cassandra-env.sh
> */var/lib/cassandra-1477577769-pid1516.hprof*
> This is quite annoying because cassandra takes into account only the path
> defined by the script init (usually that disk is not that big to keep 8Gb of
> a heap dump) and not the path defined in cassandra-env.sh
> {noformat}
> user@node:~$ jmx4perl http://localhost:8523/jolokia read
> com.sun.management:type=HotSpotDiagnostic DiagnosticOptions
> {
> name => 'HeapDumpPath',
> origin => 'VM_CREATION',
> value => '/home/cassandra/java_1477043835.hprof',
> writeable => '[true]'
> },
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13480) nodetool repair can hang forever if we lose the notification for the repair completing/failing
[ https://issues.apache.org/jira/browse/CASSANDRA-13480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988094#comment-15988094 ] Chris Lohfink commented on CASSANDRA-13480: --- how many notifications you see doesnt impact the notification buffer. JMX will create a buffer of notifications and cycle through them indexing new events as they are created. The JMX client will request events with the last index it has seen. Since the server does not store the state of the clients or know what they are listening for, ALL events regardless of listening state are appended into buffer. Even if nothing is listening to them all the storage notifications, the streaming notifications, the jvm hotspot notifications are being pushed onto that buffer. If your client takes too long between polling it will get lost notifications (and it will tell you how many it lost). 5000 still may not be nearly enough, but its gonna cost the heap dearly to make that value too large. Nodetool actually used to just shut down on lost notifications, but in some clusters/workloads its almost impossible for a client to keep up. In CASSANDRA-7909 it was just starting to be logged. Querying a different endpoint wouldnt really help, only the repair coordinator has the events and it doesnt keep it around (and its cycled outta buffer). We could in theory pull expose a JMX operation that checks the repair_history table to determine if the repair has been completed or errored out, and on lost notifications call it to make sure we did not miss a complete event. > nodetool repair can hang forever if we lose the notification for the repair > completing/failing > -- > > Key: CASSANDRA-13480 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13480 > Project: Cassandra > Issue Type: Bug > Components: Tools >Reporter: Matt Byrd >Assignee: Matt Byrd >Priority: Minor > Fix For: 4.x > > > When a Jmx lost notification occurs, sometimes the lost notification in > question is the notification which let's RepairRunner know that the repair is > finished (ProgressEventType.COMPLETE or even ERROR for that matter). > This results in nodetool process running the repair hanging forever. > I have a test which reproduces the issue here: > https://github.com/Jollyplum/cassandra-dtest/tree/repair_hang_test > To fix this, If on receiving a notification that notifications have been lost > (JMXConnectionNotification.NOTIFS_LOST), we instead query a new endpoint via > Jmx to receive all the relevant notifications we're interested in, we can > replay those we missed and avoid this scenario. > It's possible also that the JMXConnectionNotification.NOTIFS_LOST itself > might be lost and so for good measure I have made RepairRunner poll > periodically to see if there were any notifications that had been sent but we > didn't receive (scoped just to the particular tag for the given repair). > Users who don't use nodetool but go via jmx directly, can still use this new > endpoint and implement similar behaviour in their clients as desired. > I'm also expiring the notifications which have been kept on the server side. > Please let me know if you've any questions or can think of a different > approach, I also tried setting: > JVM_OPTS="$JVM_OPTS -Djmx.remote.x.notification.buffer.size=5000" > but this didn't fix the test. I suppose it might help under certain scenarios > but in this test we don't even send that many notifications so I'm not > surprised it doesn't fix it. > It seems like getting lost notifications is always a potential problem with > jmx as far as I can tell. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13480) nodetool repair can hang forever if we lose the notification for the repair completing/failing
[ https://issues.apache.org/jira/browse/CASSANDRA-13480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988094#comment-15988094 ] Chris Lohfink edited comment on CASSANDRA-13480 at 4/28/17 2:51 AM: how many notifications you see doesnt impact the notification buffer. JMX will create a buffer of notifications and cycle through them indexing new events as they are created. The JMX client will request events with the last index it has seen. Since the server does not store the state of the clients or know what they are listening for, ALL events regardless of listening state are appended into buffer. Even if nothing is listening to them all the storage notifications, the streaming notifications, the jvm hotspot notifications are being pushed onto that buffer. If your client takes too long between polling it will get lost notifications (and it will tell you how many it lost). 5000 still may not be nearly enough, but its gonna cost the heap dearly to make that value too large. Nodetool actually used to just shut down on lost notifications, but in some clusters/workloads its almost impossible for a client to keep up. In CASSANDRA-7909 it was just starting to be logged. Querying a different endpoint wouldnt really help, only the repair coordinator has the events and it doesnt keep it around (and its cycled outta buffer). We could in theory pull expose a JMX operation that checks the repair_history table or current repair states to determine if the repair has been completed or errored out, and on lost notifications call it to make sure we did not miss a complete event. was (Author: cnlwsu): how many notifications you see doesnt impact the notification buffer. JMX will create a buffer of notifications and cycle through them indexing new events as they are created. The JMX client will request events with the last index it has seen. Since the server does not store the state of the clients or know what they are listening for, ALL events regardless of listening state are appended into buffer. Even if nothing is listening to them all the storage notifications, the streaming notifications, the jvm hotspot notifications are being pushed onto that buffer. If your client takes too long between polling it will get lost notifications (and it will tell you how many it lost). 5000 still may not be nearly enough, but its gonna cost the heap dearly to make that value too large. Nodetool actually used to just shut down on lost notifications, but in some clusters/workloads its almost impossible for a client to keep up. In CASSANDRA-7909 it was just starting to be logged. Querying a different endpoint wouldnt really help, only the repair coordinator has the events and it doesnt keep it around (and its cycled outta buffer). We could in theory pull expose a JMX operation that checks the repair_history table to determine if the repair has been completed or errored out, and on lost notifications call it to make sure we did not miss a complete event. > nodetool repair can hang forever if we lose the notification for the repair > completing/failing > -- > > Key: CASSANDRA-13480 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13480 > Project: Cassandra > Issue Type: Bug > Components: Tools >Reporter: Matt Byrd >Assignee: Matt Byrd >Priority: Minor > Fix For: 4.x > > > When a Jmx lost notification occurs, sometimes the lost notification in > question is the notification which let's RepairRunner know that the repair is > finished (ProgressEventType.COMPLETE or even ERROR for that matter). > This results in nodetool process running the repair hanging forever. > I have a test which reproduces the issue here: > https://github.com/Jollyplum/cassandra-dtest/tree/repair_hang_test > To fix this, If on receiving a notification that notifications have been lost > (JMXConnectionNotification.NOTIFS_LOST), we instead query a new endpoint via > Jmx to receive all the relevant notifications we're interested in, we can > replay those we missed and avoid this scenario. > It's possible also that the JMXConnectionNotification.NOTIFS_LOST itself > might be lost and so for good measure I have made RepairRunner poll > periodically to see if there were any notifications that had been sent but we > didn't receive (scoped just to the particular tag for the given repair). > Users who don't use nodetool but go via jmx directly, can still use this new > endpoint and implement similar behaviour in their clients as desired. > I'm also expiring the notifications which have been kept on the server side. > Please let me know if you've any questions or can think of a different > approach, I also tried s
[jira] [Updated] (CASSANDRA-12922) Bloom filter miss counts are not measured correctly
[
https://issues.apache.org/jira/browse/CASSANDRA-12922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cameron Zemek updated CASSANDRA-12922:
--
Reproduced In: 3.7
Labels: lhf (was: )
Component/s: Local Write-Read Paths
> Bloom filter miss counts are not measured correctly
> ---
>
> Key: CASSANDRA-12922
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12922
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
>Reporter: Branimir Lambov
>Assignee: Mahdi Mohammadi
> Labels: lhf
>
> Bloom filter hits and misses are evaluated incorrectly in
> {{BigTableReader.getPosition}}: we properly record hits, but not misses. In
> particular, if we don't find a match for a key in the index, which is where
> almost all non-matches will be rejected, [we don't record a bloom filter
> false
> positive|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/sstable/format/big/BigTableReader.java#L228].
> This leads to very misleading output from e.g. {{nodetool tablestats}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-13481) Provide a method for flushing a CQLSSTableWriter
Gabriel Garcia created CASSANDRA-13481: -- Summary: Provide a method for flushing a CQLSSTableWriter Key: CASSANDRA-13481 URL: https://issues.apache.org/jira/browse/CASSANDRA-13481 Project: Cassandra Issue Type: Wish Reporter: Gabriel Garcia Priority: Minor Fix For: 3.11.0 The buffer size estimation in SSTableSimpleUnsortedWriter is not very accurate and causes OOM errors quite often (or lots of GC pressure) for my specific use case (rows that vary greatly in # of columns). I guess if the user knows their data well enough, they can schedule flushes when really needed. It's just an idea, not sure if flushing would actually help. However, playing with bufferSizeInMB is not of much help. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13068) Fully expired sstable not dropped when running out of disk space
[ https://issues.apache.org/jira/browse/CASSANDRA-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cameron Zemek updated CASSANDRA-13068: -- Labels: lhf (was: ) Component/s: Compaction > Fully expired sstable not dropped when running out of disk space > > > Key: CASSANDRA-13068 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13068 > Project: Cassandra > Issue Type: Bug > Components: Compaction >Reporter: Marcus Eriksson > Labels: lhf > > If a fully expired sstable is larger than the remaining disk space we won't > run the compaction that can drop the sstable (ie, in our disk space check > should not include the fully expired sstables) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13481) Provide a method for flushing a CQLSSTableWriter
[ https://issues.apache.org/jira/browse/CASSANDRA-13481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988156#comment-15988156 ] ASF GitHub Bot commented on CASSANDRA-13481: GitHub user Garciat opened a pull request: https://github.com/apache/cassandra/pull/107 [CASSANDRA-13481] Provide a method for flushing a CQLSSTableWriter Jira: https://issues.apache.org/jira/browse/CASSANDRA-13481 You can merge this pull request into a Git repository by running: $ git pull https://github.com/Garciat/cassandra CASSANDRA-13481 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/cassandra/pull/107.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #107 commit 23c913f4642cc2463966f86dba6a24b987bc0f0a Author: Gabriel Garcia Date: 2017-04-28T03:33:28Z Provide a method for flushing a CQLSSTableWriter. > Provide a method for flushing a CQLSSTableWriter > > > Key: CASSANDRA-13481 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13481 > Project: Cassandra > Issue Type: Wish >Reporter: Gabriel Garcia >Priority: Minor > Fix For: 3.11.0 > > > The buffer size estimation in SSTableSimpleUnsortedWriter is not very > accurate and causes OOM errors quite often (or lots of GC pressure) for my > specific use case (rows that vary greatly in # of columns). I guess if the > user knows their data well enough, they can schedule flushes when really > needed. > It's just an idea, not sure if flushing would actually help. However, playing > with bufferSizeInMB is not of much help. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
