[jira] [Comment Edited] (CASSANDRA-8274) Node fails to rejoin cluster on EC2 if private IP is changed
[
https://issues.apache.org/jira/browse/CASSANDRA-8274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177493#comment-16177493
]
Chris mildebrandt edited comment on CASSANDRA-8274 at 9/23/17 7:21 PM:
---
I just hit this issue today with the 3.11.0 docker image running in kubernetes.
I had 4 nodes in the cassandra cluster, two members were restarted (and their
IPs changed) and can't rejoin. There's one seed that is up and reachable from
all the other containers, and one other member that is able to join. The first
exception I see is this:
{noformat}
java.lang.RuntimeException: Cache schema version
38e97a53-563b-3074-b86f-c81efa980524 does not match current schema version
1bfdabae-743e-357e-a661-93984c26bc32
at
org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:206)
~[apache-cassandra-3.11.0.jar:3.11.0]
at
org.apache.cassandra.cache.AutoSavingCache$3.call(AutoSavingCache.java:164)
[apache-cassandra-3.11.0.jar:3.11.0]
at
org.apache.cassandra.cache.AutoSavingCache$3.call(AutoSavingCache.java:160)
[apache-cassandra-3.11.0.jar:3.11.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
{noformat}
Then I see the one related to this issue:
{noformat}
java.lang.RuntimeException: Unable to gossip with any seeds
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1413)
~[apache-cassandra-3.11.0.jar:3.11.0]
at
org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:550)
~[apache-cassandra-3.11.0.jar:3.11.0]
at
org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:801)
~[apache-cassandra-3.11.0.jar:3.11.0]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:666)
~[apache-cassandra-3.11.0.jar:3.11.0]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:612)
~[apache-cassandra-3.11.0.jar:3.11.0]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:393)
[apache-cassandra-3.11.0.jar:3.11.0]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:600)
[apache-cassandra-3.11.0.jar:3.11.0]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:689)
[apache-cassandra-3.11.0.jar:3.11.0]
{noformat}
Restarting the nodes didn't help. nodetool status is now reporting only two
nodes, and nodetool gossipinfo has three "empty" entries (along with the two
good ones not shown):
{noformat}
/100.96.3.164
generation:0
heartbeat:0
TOKENS: not present
/100.96.1.7
generation:0
heartbeat:0
TOKENS: not present
/100.96.2.170
generation:0
heartbeat:0
TOKENS: not present
{noformat}
was (Author: mildebrandt):
I just hit this issue today with the 3.11.0 docker image running in kubernetes.
I had 4 nodes in the cassandra cluster, two members were restarted (and their
IPs changed) and can't rejoin. There's one seed that is up and reachable from
all the other containers, and one other member that is able to join. The first
exception I see is this:
{noformat}
java.lang.RuntimeException: Cache schema version
38e97a53-563b-3074-b86f-c81efa980524 does not match current schema version
1bfdabae-743e-357e-a661-93984c26bc32
at
org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:206)
~[apache-cassandra-3.11.0.jar:3.11.0]
at
org.apache.cassandra.cache.AutoSavingCache$3.call(AutoSavingCache.java:164)
[apache-cassandra-3.11.0.jar:3.11.0]
at
org.apache.cassandra.cache.AutoSavingCache$3.call(AutoSavingCache.java:160)
[apache-cassandra-3.11.0.jar:3.11.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
{noformat}
Then I see the one related to this issue:
{noformat}
java.lang.RuntimeException: Unable to gossip with any seeds
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1413)
~[apache-cassandra-3.11.0.jar:3.11.0]
at
org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:550)
~[apache-cassandra-3.11.0.jar:3.11.0]
at
org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:801)
~[apache-cassandra-3.11.0.jar:3.11.0]
at
[jira] [Commented] (CASSANDRA-13890) Expose current compaction throughput
[
https://issues.apache.org/jira/browse/CASSANDRA-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177876#comment-16177876
]
Jeff Jirsa commented on CASSANDRA-13890:
cc [~cnlwsu] in case he has a better/different idea before you get started.
> Expose current compaction throughput
>
>
> Key: CASSANDRA-13890
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13890
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
>Reporter: Jon Haddad
>Assignee: Jon Haddad
>Priority: Minor
> Labels: lhf
> Fix For: 4.x
>
>
> Getting and setting the current compaction throughput limit is supported, but
> there's no means of knowing if the setting is actually making a difference.
> Let's expose the current throughput being utilized by Cassandra that's in the
> {{compactionRateLimiter}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13890) Expose current compaction throughput
[
https://issues.apache.org/jira/browse/CASSANDRA-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177872#comment-16177872
]
Jon Haddad commented on CASSANDRA-13890:
Great suggestion, thanks Jeff. I'll go forward with that.
> Expose current compaction throughput
>
>
> Key: CASSANDRA-13890
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13890
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
>Reporter: Jon Haddad
>Assignee: Jon Haddad
>Priority: Minor
> Labels: lhf
> Fix For: 4.x
>
>
> Getting and setting the current compaction throughput limit is supported, but
> there's no means of knowing if the setting is actually making a difference.
> Let's expose the current throughput being utilized by Cassandra that's in the
> {{compactionRateLimiter}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13890) Expose current compaction throughput
[
https://issues.apache.org/jira/browse/CASSANDRA-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177868#comment-16177868
]
Jeff Jirsa commented on CASSANDRA-13890:
Could change it from a counter to a meter (or rather, add a meter and deprecate
the counter for a release, and remove the counter later).
> Expose current compaction throughput
>
>
> Key: CASSANDRA-13890
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13890
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
>Reporter: Jon Haddad
>Assignee: Jon Haddad
>Priority: Minor
> Labels: lhf
> Fix For: 4.x
>
>
> Getting and setting the current compaction throughput limit is supported, but
> there's no means of knowing if the setting is actually making a difference.
> Let's expose the current throughput being utilized by Cassandra that's in the
> {{compactionRateLimiter}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13890) Expose current compaction throughput
[
https://issues.apache.org/jira/browse/CASSANDRA-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-13890:
---
Fix Version/s: (was: 4.0)
4.x
> Expose current compaction throughput
>
>
> Key: CASSANDRA-13890
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13890
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
>Reporter: Jon Haddad
>Assignee: Jon Haddad
>Priority: Minor
> Labels: lhf
> Fix For: 4.x
>
>
> Getting and setting the current compaction throughput limit is supported, but
> there's no means of knowing if the setting is actually making a difference.
> Let's expose the current throughput being utilized by Cassandra that's in the
> {{compactionRateLimiter}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13890) Expose current compaction throughput
[
https://issues.apache.org/jira/browse/CASSANDRA-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177861#comment-16177861
]
Jon Haddad commented on CASSANDRA-13890:
I was looking at that, but I don't think it would be very accurate using the
numbers as is. Doesn't that report all the bytes compacted since the server
started? Or are you suggesting updating {{compactionBytesWritten}} to track
the recent rate (maybe a 5 second moving average) in addition to the simple
metric it already tracks? I think this could work well.
Before your comment I was also looking at adding a start time to
{{CompactionInfo}}, then using {{completed}} to determine the current rate, but
it suffers from the same problem as using raw {{compactionBytesWritten}} -
changing the limit via {{nodetool setcompactionthroughput}} would immediately
change the throttle rate but it wouldn't be reflected in the reported rate till
the current compactions ended, so I'm not wild about this approach.
> Expose current compaction throughput
>
>
> Key: CASSANDRA-13890
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13890
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
>Reporter: Jon Haddad
>Assignee: Jon Haddad
>Priority: Minor
> Labels: lhf
> Fix For: 4.0
>
>
> Getting and setting the current compaction throughput limit is supported, but
> there's no means of knowing if the setting is actually making a difference.
> Let's expose the current throughput being utilized by Cassandra that's in the
> {{compactionRateLimiter}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13890) Expose current compaction throughput
[
https://issues.apache.org/jira/browse/CASSANDRA-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177856#comment-16177856
]
Jeff Jirsa commented on CASSANDRA-13890:
You cant get it from the rate limiter, but you may be able to get close from
the bytes-written metric updated from compaction task [e.g.
here|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/compaction/CompactionTask.java#L277]
> Expose current compaction throughput
>
>
> Key: CASSANDRA-13890
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13890
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
>Reporter: Jon Haddad
>Assignee: Jon Haddad
>Priority: Minor
> Labels: lhf
> Fix For: 4.0
>
>
> Getting and setting the current compaction throughput limit is supported, but
> there's no means of knowing if the setting is actually making a difference.
> Let's expose the current throughput being utilized by Cassandra that's in the
> {{compactionRateLimiter}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13890) Expose current compaction throughput
[
https://issues.apache.org/jira/browse/CASSANDRA-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177845#comment-16177845
]
Jon Haddad commented on CASSANDRA-13890:
I figured this would be low hanging fruit, but there's a couple snags. First
off, the api in {{com/google/common/util/concurrent/RateLimiter.java}} only
exposes the ability to acquire "Permits", which is how QPS is regulated.
There's nothing public to determine what current usage is.
We're using an older version of Guava, (18), so I checked the current and it
doesn't expose any additional information.
> Expose current compaction throughput
>
>
> Key: CASSANDRA-13890
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13890
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
>Reporter: Jon Haddad
>Assignee: Jon Haddad
>Priority: Minor
> Labels: lhf
> Fix For: 4.0
>
>
> Getting and setting the current compaction throughput limit is supported, but
> there's no means of knowing if the setting is actually making a difference.
> Let's expose the current throughput being utilized by Cassandra that's in the
> {{compactionRateLimiter}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13897) nodetool compact and flush fail with "error: null"
[
https://issues.apache.org/jira/browse/CASSANDRA-13897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jacob Rhoden updated CASSANDRA-13897:
-
Description:
{{nodetool flush}} and {{nodetool compact}} return a nonsensical error message.
This could probably be improved.
I have one node that always returns this error when invoking {{nodetool
compress}}. And intermittently returns this error when calling {{nodetool
flush}}. I have tried deleting saved_caches, commit logs, doing nodetool
compact/rebuild/scrub, and nothing seems to remove the error.
{noformat}
cass@s5:~/apache-cassandra-3.11.0$ nodetool compact
error: null
-- StackTrace --
java.lang.AssertionError
at
org.apache.cassandra.cache.ChunkCache$CachingRebufferer.(ChunkCache.java:222)
at org.apache.cassandra.cache.ChunkCache.wrap(ChunkCache.java:175)
at
org.apache.cassandra.io.util.FileHandle$Builder.maybeCached(FileHandle.java:412)
at
org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:381)
at
org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:331)
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:333)
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinalEarly(BigTableWriter.java:318)
at
org.apache.cassandra.io.sstable.SSTableRewriter.switchWriter(SSTableRewriter.java:322)
at
org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:370)
at
org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:111)
at
org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
at
org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:121)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:220)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
at
org.apache.cassandra.db.compaction.CompactionManager$10.runMayThrow(CompactionManager.java:733)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
at java.lang.Thread.run(Thread.java:748)
{noformat}
was:
{{nodetool flush}} and {{nodetool compact}} return a nonsensical error message.
This could probably be improved.
I have one node that always returns this error when invoking {{nodetool
compress}}. And intermittently returns this error when calling {{nodetool
flush}}. I have tried deleting saved_caches, commit logs, doing nodetool
compact/rebuild/scrub, and nothing
{noformat}
cass@s5:~/apache-cassandra-3.11.0$ nodetool compact
error: null
-- StackTrace --
java.lang.AssertionError
at
org.apache.cassandra.cache.ChunkCache$CachingRebufferer.(ChunkCache.java:222)
at org.apache.cassandra.cache.ChunkCache.wrap(ChunkCache.java:175)
at
org.apache.cassandra.io.util.FileHandle$Builder.maybeCached(FileHandle.java:412)
at
org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:381)
at
org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:331)
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:333)
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinalEarly(BigTableWriter.java:318)
at
org.apache.cassandra.io.sstable.SSTableRewriter.switchWriter(SSTableRewriter.java:322)
at
org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:370)
at
org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
at
[jira] [Updated] (CASSANDRA-13897) nodetool compact and flush fail with "error: null"
[
https://issues.apache.org/jira/browse/CASSANDRA-13897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jacob Rhoden updated CASSANDRA-13897:
-
Description:
{{nodetool flush}} and {{nodetool compact}} return a nonsensical error message.
This could probably be improved.
I have one node that always returns this error when invoking {{nodetool
compress}}. And intermittently returns this error when calling {{nodetool
flush}}. I have tried deleting saved_caches, commit logs, doing nodetool
compact/rebuild/scrub, and nothing
{noformat}
cass@s5:~/apache-cassandra-3.11.0$ nodetool compact
error: null
-- StackTrace --
java.lang.AssertionError
at
org.apache.cassandra.cache.ChunkCache$CachingRebufferer.(ChunkCache.java:222)
at org.apache.cassandra.cache.ChunkCache.wrap(ChunkCache.java:175)
at
org.apache.cassandra.io.util.FileHandle$Builder.maybeCached(FileHandle.java:412)
at
org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:381)
at
org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:331)
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:333)
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinalEarly(BigTableWriter.java:318)
at
org.apache.cassandra.io.sstable.SSTableRewriter.switchWriter(SSTableRewriter.java:322)
at
org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:370)
at
org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:111)
at
org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
at
org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:121)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:220)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
at
org.apache.cassandra.db.compaction.CompactionManager$10.runMayThrow(CompactionManager.java:733)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
at java.lang.Thread.run(Thread.java:748)
{noformat}
was:
{{nodetool flush}} and {{nodetool compact}} return a nonsensical error message.
This could probably be improved.
I have one node that always returns this error when invoking {{nodetool
compress}}. And intermittently returns this error when calling {{nodetool
flush}}. I have tried deleting saved_caches, commit logs, doing nodetool
compact/rebuild/scrub, and nothing
cass@s5:~/apache-cassandra-3.11.0$ nodetool compact
error: null
-- StackTrace --
java.lang.AssertionError
at
org.apache.cassandra.cache.ChunkCache$CachingRebufferer.(ChunkCache.java:222)
at org.apache.cassandra.cache.ChunkCache.wrap(ChunkCache.java:175)
at
org.apache.cassandra.io.util.FileHandle$Builder.maybeCached(FileHandle.java:412)
at
org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:381)
at
org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:331)
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:333)
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinalEarly(BigTableWriter.java:318)
at
org.apache.cassandra.io.sstable.SSTableRewriter.switchWriter(SSTableRewriter.java:322)
at
org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:370)
at
org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
at
[jira] [Commented] (CASSANDRA-13595) Short read protection doesn't work at the end of a partition
[
https://issues.apache.org/jira/browse/CASSANDRA-13595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177633#comment-16177633
]
ZhaoYang commented on CASSANDRA-13595:
--
Thanks for the feedback. Feel free to take over.
> Short read protection doesn't work at the end of a partition
>
>
> Key: CASSANDRA-13595
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13595
> Project: Cassandra
> Issue Type: Bug
> Components: Coordination
>Reporter: Andrés de la Peña
>Assignee: ZhaoYang
> Labels: Correctness
>
> It seems that short read protection doesn't work when the short read is done
> at the end of a partition in a range query. The final assertion of this dtest
> fails:
> {code}
> def short_read_partitions_delete_test(self):
> cluster = self.cluster
> cluster.set_configuration_options(values={'hinted_handoff_enabled':
> False})
> cluster.set_batch_commitlog(enabled=True)
> cluster.populate(2).start(wait_other_notice=True)
> node1, node2 = self.cluster.nodelist()
> session = self.patient_cql_connection(node1)
> create_ks(session, 'ks', 2)
> session.execute("CREATE TABLE t (k int, c int, PRIMARY KEY(k, c))
> WITH read_repair_chance = 0.0")
> # we write 1 and 2 in a partition: all nodes get it.
> session.execute(SimpleStatement("INSERT INTO t (k, c) VALUES (1, 1)",
> consistency_level=ConsistencyLevel.ALL))
> session.execute(SimpleStatement("INSERT INTO t (k, c) VALUES (2, 1)",
> consistency_level=ConsistencyLevel.ALL))
> # we delete partition 1: only node 1 gets it.
> node2.flush()
> node2.stop(wait_other_notice=True)
> session = self.patient_cql_connection(node1, 'ks',
> consistency_level=ConsistencyLevel.ONE)
> session.execute(SimpleStatement("DELETE FROM t WHERE k = 1"))
> node2.start(wait_other_notice=True)
> # we delete partition 2: only node 2 gets it.
> node1.flush()
> node1.stop(wait_other_notice=True)
> session = self.patient_cql_connection(node2, 'ks',
> consistency_level=ConsistencyLevel.ONE)
> session.execute(SimpleStatement("DELETE FROM t WHERE k = 2"))
> node1.start(wait_other_notice=True)
> # read from both nodes
> session = self.patient_cql_connection(node1, 'ks',
> consistency_level=ConsistencyLevel.ALL)
> assert_none(session, "SELECT * FROM t LIMIT 1")
> {code}
> However, the dtest passes if we remove the {{LIMIT 1}}.
> Short read protection [uses a
> {{SinglePartitionReadCommand}}|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/DataResolver.java#L484],
> maybe it should use a {{PartitionRangeReadCommand}} instead?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-13897) nodetool compact and flush fail with "error: null"
Jacob Rhoden created CASSANDRA-13897:
Summary: nodetool compact and flush fail with "error: null"
Key: CASSANDRA-13897
URL: https://issues.apache.org/jira/browse/CASSANDRA-13897
Project: Cassandra
Issue Type: Bug
Components: Compaction
Environment: * Apache Cassandra 3.11.0
* Linux 4.4.0-92-generic #115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017 x86_64
x86_64 x86_64 GNU/Linux
* jdk1.8.0_144
Reporter: Jacob Rhoden
Priority: Minor
{{nodetool flush}} and {{nodetool compact}} return a nonsensical error message.
This could probably be improved.
I have one node that always returns this error when invoking {{nodetool
compress}}. And intermittently returns this error when calling {{nodetool
flush}}. I have tried deleting saved_caches, commit logs, doing nodetool
compact/rebuild/scrub, and nothing
cass@s5:~/apache-cassandra-3.11.0$ nodetool compact
error: null
-- StackTrace --
java.lang.AssertionError
at
org.apache.cassandra.cache.ChunkCache$CachingRebufferer.(ChunkCache.java:222)
at org.apache.cassandra.cache.ChunkCache.wrap(ChunkCache.java:175)
at
org.apache.cassandra.io.util.FileHandle$Builder.maybeCached(FileHandle.java:412)
at
org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:381)
at
org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:331)
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:333)
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinalEarly(BigTableWriter.java:318)
at
org.apache.cassandra.io.sstable.SSTableRewriter.switchWriter(SSTableRewriter.java:322)
at
org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:370)
at
org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:111)
at
org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
at
org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:121)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:220)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
at
org.apache.cassandra.db.compaction.CompactionManager$10.runMayThrow(CompactionManager.java:733)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
at java.lang.Thread.run(Thread.java:748)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
