[jira] [Created] (CASSANDRA-14932) removenode coordinator, and its hints data will be lost
sunhaihong created CASSANDRA-14932:
--
Summary: removenode coordinator, and its hints data will be lost
Key: CASSANDRA-14932
URL: https://issues.apache.org/jira/browse/CASSANDRA-14932
Project: Cassandra
Issue Type: Bug
Environment: cassandra version 3.11.3
Reporter: sunhaihong
There are four nodes in cluster. assume them are node A, B, C, D. enabled
hinted handoff.
1) create a keyspace with RF=2, and create a table.
2) make node B, C down(nodetool stopdaemon),
3) login in node A with cqlsh，set CONSISTENCY ANY, insert into a row(assume the
row will be stored in node B and C). The row was successfully inserted even
though the node B,C was down, because the consistency level is ANY. the
coordinator(node A) wrote hints.
4) make node A down(nodetool stopdaemon), then remove node A(nodetool
removenode ${nodeA_hostId})
5) make node B, C come back(nodetool start)
6) login in any node of B, C, D. and execute select statement with partition
key of inserted row. But there is no any data that inserted row on step 3.
These steps lead to data(on step 3 was inserted row) lost.
Is there any problem with the steps I performed above?
If yes, How to deal with this situation？
look forward to your reply, thanks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14932) removenode coordinator, and its hints data will be lost
[
https://issues.apache.org/jira/browse/CASSANDRA-14932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sunhaihong updated CASSANDRA-14932:
---
Priority: Major (was: Minor)
> removenode coordinator, and its hints data will be lost
> ---
>
> Key: CASSANDRA-14932
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14932
> Project: Cassandra
> Issue Type: Bug
> Environment: cassandra version 3.11.3
>Reporter: sunhaihong
>Priority: Major
>
> There are four nodes in cluster. assume them are node A, B, C, D. enabled
> hinted handoff.
> 1) create a keyspace with RF=2, and create a table.
> 2) make node B, C down(nodetool stopdaemon),
> 3) login in node A with cqlsh，set CONSISTENCY ANY, insert into a row(assume
> the row will be stored in node B and C). The row was successfully inserted
> even though the node B,C was down, because the consistency level is ANY. the
> coordinator(node A) wrote hints.
> 4) make node A down(nodetool stopdaemon), then remove node A(nodetool
> removenode ${nodeA_hostId})
> 5) make node B, C come back(nodetool start)
> 6) login in any node of B, C, D. and execute select statement with partition
> key of inserted row. But there is no any data that inserted row on step 3.
>
> These steps lead to data(on step 3 was inserted row) lost.
> Is there any problem with the steps I performed above?
> If yes, How to deal with this situation？
> look forward to your reply, thanks.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15669) LeveledCompactionStrategy compact last level throw an ArrayIndexOutOfBoundsException
sunhaihong created CASSANDRA-15669: -- Summary: LeveledCompactionStrategy compact last level throw an ArrayIndexOutOfBoundsException Key: CASSANDRA-15669 URL: https://issues.apache.org/jira/browse/CASSANDRA-15669 Project: Cassandra Issue Type: Bug Reporter: sunhaihong Assignee: sunhaihong Attachments: cfs_compaction_info.png, error_info.png cassandra will throw an ArrayIndexOutOfBoundsException when compact last level. My test is as follows： # Create a table with LeveledCompactionStrategy and its params are 'enabled': 'true', 'fanout_size': '2', 'max_threshold': '32', 'min_threshold': '4', 'sstable_size_in_mb': '2'（fanout_size and sstable_size_in_mb are too small just to make it easier to reproduce the problem）; # Insert data into the table by stress; # Cassandra throw an ArrayIndexOutOfBoundsException when compact level9 sstables(this level score bigger than 1.001) I tested it on cassandra version 3.11.3 & 4.0-alpha3. The exception all happened. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15669) LeveledCompactionStrategy compact last level throw an ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/CASSANDRA-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunhaihong updated CASSANDRA-15669: --- Description: Cassandra will throw an ArrayIndexOutOfBoundsException when compact last level. My test is as follows： # Create a table with LeveledCompactionStrategy and its params are 'enabled': 'true', 'fanout_size': '2', 'max_threshold': '32', 'min_threshold': '4', 'sstable_size_in_mb': '2'（fanout_size and sstable_size_in_mb are too small just to make it easier to reproduce the problem）; # Insert data into the table by stress; # Cassandra throw an ArrayIndexOutOfBoundsException when compact level9 sstables(this level score bigger than 1.001) ERROR [CompactionExecutor:4] 2020-03-28 08:59:00,990 CassandraDaemon.java:442 - Exception in thread Thread[CompactionExecutor:4,1,main] java.lang.ArrayIndexOutOfBoundsException: 9 at org.apache.cassandra.db.compaction.LeveledManifest.getLevel(LeveledManifest.java:814) at org.apache.cassandra.db.compaction.LeveledManifest.getCandidatesFor(LeveledManifest.java:746) at org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:398) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:131) at org.apache.cassandra.db.compaction.CompactionStrategyHolder.lambda$getBackgroundTaskSuppliers$0(CompactionStrategyHolder.java:109) at org.apache.cassandra.db.compaction.AbstractStrategyHolder$TaskSupplier.getTask(AbstractStrategyHolder.java:66) at org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:214) at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:289) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) at java.util.concurrent.FutureTask.run(FutureTask.java) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) I tested it on cassandra version 3.11.3 & 4.0-alpha3. The exception all happened. once it triggers, level1- leveln compaction no longer works, level0 is still valid was: Cassandra will throw an ArrayIndexOutOfBoundsException when compact last level. My test is as follows： # Create a table with LeveledCompactionStrategy and its params are 'enabled': 'true', 'fanout_size': '2', 'max_threshold': '32', 'min_threshold': '4', 'sstable_size_in_mb': '2'（fanout_size and sstable_size_in_mb are too small just to make it easier to reproduce the problem）; # Insert data into the table by stress; # Cassandra throw an ArrayIndexOutOfBoundsException when compact level9 sstables(this level score bigger than 1.001) ERROR [CompactionExecutor:4] 2020-03-28 08:59:00,990 CassandraDaemon.java:442 - Exception in thread Thread[CompactionExecutor:4,1,main] java.lang.ArrayIndexOutOfBoundsException: 9 at org.apache.cassandra.db.compaction.LeveledManifest.getLevel(LeveledManifest.java:814) at org.apache.cassandra.db.compaction.LeveledManifest.getCandidatesFor(LeveledManifest.java:746) at org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:398) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:131) at org.apache.cassandra.db.compaction.CompactionStrategyHolder.lambda$getBackgroundTaskSuppliers$0(CompactionStrategyHolder.java:109) at org.apache.cassandra.db.compaction.AbstractStrategyHolder$TaskSupplier.getTask(AbstractStrategyHolder.java:66) at org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:214) at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:289) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) at java.util.concurrent.FutureTask.run(FutureTask.java) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) I tested it on cassandra version 3.11.3 & 4.0-alpha3. The exception all happened. once it triggers, compaction no longer works > LeveledCompactionStrategy compact last level throw an > ArrayIndexOutOfBoundsException >
[jira] [Updated] (CASSANDRA-15669) LeveledCompactionStrategy compact last level throw an ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/CASSANDRA-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunhaihong updated CASSANDRA-15669: --- Description: Cassandra will throw an ArrayIndexOutOfBoundsException when compact last level. My test is as follows： # Create a table with LeveledCompactionStrategy and its params are 'enabled': 'true', 'fanout_size': '2', 'max_threshold': '32', 'min_threshold': '4', 'sstable_size_in_mb': '2'（fanout_size and sstable_size_in_mb are too small just to make it easier to reproduce the problem）; # Insert data into the table by stress; # Cassandra throw an ArrayIndexOutOfBoundsException when compact level9 sstables(this level score bigger than 1.001) ERROR [CompactionExecutor:4] 2020-03-28 08:59:00,990 CassandraDaemon.java:442 - Exception in thread Thread[CompactionExecutor:4,1,main] java.lang.ArrayIndexOutOfBoundsException: 9 at org.apache.cassandra.db.compaction.LeveledManifest.getLevel(LeveledManifest.java:814) at org.apache.cassandra.db.compaction.LeveledManifest.getCandidatesFor(LeveledManifest.java:746) at org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:398) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:131) at org.apache.cassandra.db.compaction.CompactionStrategyHolder.lambda$getBackgroundTaskSuppliers$0(CompactionStrategyHolder.java:109) at org.apache.cassandra.db.compaction.AbstractStrategyHolder$TaskSupplier.getTask(AbstractStrategyHolder.java:66) at org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:214) at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:289) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) at java.util.concurrent.FutureTask.run(FutureTask.java) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) I tested it on cassandra version 3.11.3 & 4.0-alpha3. The exception all happened. once it triggers, compaction no longer works was: Cassandra will throw an ArrayIndexOutOfBoundsException when compact last level. My test is as follows： # Create a table with LeveledCompactionStrategy and its params are 'enabled': 'true', 'fanout_size': '2', 'max_threshold': '32', 'min_threshold': '4', 'sstable_size_in_mb': '2'（fanout_size and sstable_size_in_mb are too small just to make it easier to reproduce the problem）; # Insert data into the table by stress; # Cassandra throw an ArrayIndexOutOfBoundsException when compact level9 sstables(this level score bigger than 1.001) ERROR [CompactionExecutor:4] 2020-03-28 08:59:00,990 CassandraDaemon.java:442 - Exception in thread Thread[CompactionExecutor:4,1,main] java.lang.ArrayIndexOutOfBoundsException: 9 at org.apache.cassandra.db.compaction.LeveledManifest.getLevel(LeveledManifest.java:814) at org.apache.cassandra.db.compaction.LeveledManifest.getCandidatesFor(LeveledManifest.java:746) at org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:398) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:131) at org.apache.cassandra.db.compaction.CompactionStrategyHolder.lambda$getBackgroundTaskSuppliers$0(CompactionStrategyHolder.java:109) at org.apache.cassandra.db.compaction.AbstractStrategyHolder$TaskSupplier.getTask(AbstractStrategyHolder.java:66) at org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:214) at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:289) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) at java.util.concurrent.FutureTask.run(FutureTask.java) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) I tested it on cassandra version 3.11.3 & 4.0-alpha3. The exception all happened. > LeveledCompactionStrategy compact last level throw an > ArrayIndexOutOfBoundsException > > > Key: CASSANDRA-15669 > URL:
[jira] [Updated] (CASSANDRA-15669) LeveledCompactionStrategy compact last level throw an ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/CASSANDRA-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunhaihong updated CASSANDRA-15669: --- Description: Cassandra will throw an ArrayIndexOutOfBoundsException when compact last level. My test is as follows： # Create a table with LeveledCompactionStrategy and its params are 'enabled': 'true', 'fanout_size': '2', 'max_threshold': '32', 'min_threshold': '4', 'sstable_size_in_mb': '2'（fanout_size and sstable_size_in_mb are too small just to make it easier to reproduce the problem）; # Insert data into the table by stress; # Cassandra throw an ArrayIndexOutOfBoundsException when compact level9 sstables(this level score bigger than 1.001) ERROR [CompactionExecutor:4] 2020-03-28 08:59:00,990 CassandraDaemon.java:442 - Exception in thread Thread[CompactionExecutor:4,1,main] java.lang.ArrayIndexOutOfBoundsException: 9 at org.apache.cassandra.db.compaction.LeveledManifest.getLevel(LeveledManifest.java:814) at org.apache.cassandra.db.compaction.LeveledManifest.getCandidatesFor(LeveledManifest.java:746) at org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:398) at org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:131) at org.apache.cassandra.db.compaction.CompactionStrategyHolder.lambda$getBackgroundTaskSuppliers$0(CompactionStrategyHolder.java:109) at org.apache.cassandra.db.compaction.AbstractStrategyHolder$TaskSupplier.getTask(AbstractStrategyHolder.java:66) at org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:214) at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:289) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) at java.util.concurrent.FutureTask.run(FutureTask.java) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) I tested it on cassandra version 3.11.3 & 4.0-alpha3. The exception all happened. was: cassandra will throw an ArrayIndexOutOfBoundsException when compact last level. My test is as follows： # Create a table with LeveledCompactionStrategy and its params are 'enabled': 'true', 'fanout_size': '2', 'max_threshold': '32', 'min_threshold': '4', 'sstable_size_in_mb': '2'（fanout_size and sstable_size_in_mb are too small just to make it easier to reproduce the problem）; # Insert data into the table by stress; # Cassandra throw an ArrayIndexOutOfBoundsException when compact level9 sstables(this level score bigger than 1.001) I tested it on cassandra version 3.11.3 & 4.0-alpha3. The exception all happened. > LeveledCompactionStrategy compact last level throw an > ArrayIndexOutOfBoundsException > > > Key: CASSANDRA-15669 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15669 > Project: Cassandra > Issue Type: Bug >Reporter: sunhaihong >Assignee: sunhaihong >Priority: Normal > Attachments: cfs_compaction_info.png, error_info.png > > > Cassandra will throw an ArrayIndexOutOfBoundsException when compact last > level. > My test is as follows： > # Create a table with LeveledCompactionStrategy and its params are > 'enabled': 'true', 'fanout_size': '2', 'max_threshold': '32', > 'min_threshold': '4', 'sstable_size_in_mb': '2'（fanout_size and > sstable_size_in_mb are too small just to make it easier to reproduce the > problem）; > # Insert data into the table by stress; > # Cassandra throw an ArrayIndexOutOfBoundsException when compact level9 > sstables(this level score bigger than 1.001) > ERROR [CompactionExecutor:4] 2020-03-28 08:59:00,990 CassandraDaemon.java:442 > - Exception in thread Thread[CompactionExecutor:4,1,main] > java.lang.ArrayIndexOutOfBoundsException: 9 > at > org.apache.cassandra.db.compaction.LeveledManifest.getLevel(LeveledManifest.java:814) > at > org.apache.cassandra.db.compaction.LeveledManifest.getCandidatesFor(LeveledManifest.java:746) > at > org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:398) > at > org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:131) > at > org.apache.cassandra.db.compaction.CompactionStrategyHolder.lambda$getBackgroundTaskSuppliers$0(CompactionStrategyHolder.java:109) > at >
[jira] [Commented] (CASSANDRA-15669) LeveledCompactionStrategy compact last level throw an ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/CASSANDRA-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17325504#comment-17325504 ] sunhaihong commented on CASSANDRA-15669: Hi [~azotcsit]，Thank you for your reply. My problem is the second situation that you described 2. *There is no proper handling of a situation when there is more data than supported.* > LeveledCompactionStrategy compact last level throw an > ArrayIndexOutOfBoundsException > > > Key: CASSANDRA-15669 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15669 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction/LCS >Reporter: sunhaihong >Assignee: Alexey Zotov >Priority: Normal > Fix For: 3.11.x, 4.0.x > > Attachments: cfs_compaction_info.png, error_info.png > > Time Spent: 50m > Remaining Estimate: 0h > > Cassandra will throw an ArrayIndexOutOfBoundsException when compact last > level. > My test is as follows： > # Create a table with LeveledCompactionStrategy and its params are > 'enabled': 'true', 'fanout_size': '2', 'max_threshold': '32', > 'min_threshold': '4', 'sstable_size_in_mb': '2'（fanout_size and > sstable_size_in_mb are too small just to make it easier to reproduce the > problem）; > # Insert data into the table by stress; > # Cassandra throw an ArrayIndexOutOfBoundsException when compact level9 > sstables(this level score bigger than 1.001) > ERROR [CompactionExecutor:4] 2020-03-28 08:59:00,990 CassandraDaemon.java:442 > - Exception in thread Thread[CompactionExecutor:4,1,main] > java.lang.ArrayIndexOutOfBoundsException: 9 > at > org.apache.cassandra.db.compaction.LeveledManifest.getLevel(LeveledManifest.java:814) > at > org.apache.cassandra.db.compaction.LeveledManifest.getCandidatesFor(LeveledManifest.java:746) > at > org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:398) > at > org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:131) > at > org.apache.cassandra.db.compaction.CompactionStrategyHolder.lambda$getBackgroundTaskSuppliers$0(CompactionStrategyHolder.java:109) > at > org.apache.cassandra.db.compaction.AbstractStrategyHolder$TaskSupplier.getTask(AbstractStrategyHolder.java:66) > at > org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:214) > at > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:289) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) > at java.util.concurrent.FutureTask.run(FutureTask.java) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.lang.Thread.run(Thread.java:748) > I tested it on cassandra version 3.11.3 & 4.0-alpha3. The exception all > happened. > once it triggers, level1- leveln compaction no longer works, level0 is still > valid > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
