[ https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Hust updated CASSANDRA-10360: ------------------------------------ Description: When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will get stuck in a compaction loop throwing: {code} java.lang.UnsupportedOperationException at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382) at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147) at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503) at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80) at org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72) at org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100) at org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72) at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223) at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) at org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) {code} It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not surface the issue. It can be reproduced with the following (note -- changing jdks will need to be altered if you aren't on OSX) {code} #!/bin/sh echo "using java7 for cassandra-2.1 instance" export JAVA_HOME=$(/usr/libexec/java_home -v 1.7) ccm stop ccm remove upgrade_nodes ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes ccm start ccm node1 stress write n=500K -rate threads=4 -mode native cql3 sleep 10 for cver in 3.0 do echo "Draining all nodes" ccm node1 nodetool drain ccm stop echo "switching to java 8" export JAVA_HOME=$(/usr/libexec/java_home -v 1.8) echo "Upgrading to git:cassandra-$cver" ccm setdir -v git:cassandra-$cver ccm start echo "Sleeping to all version migrations" sleep 30 echo "Upgrading sstables" ccm node1 nodetool upgradesstables ccm node1 nodetool upgradesstables system ccm node1 nodetool compact system ccm node1 stress write n=500K -rate threads=4 -mode native cql3 sleep 10 done echo "***********" echo "instead of creating churn to cause compaction naturally forcing compaction of system keyspace" echo "***********" ccm node1 nodetool compact system ccm stop {code} The example uses {{nodetool compact system}} but it will also occur with {{nodetool upgradesstables system}}. I'm puzzled by that since the script runs {{upgradesstables}} on each iteration. Is the system keyspace not effected by the command without arguments? Ran against: 2.1: {{e889ee408bec5330c312ff6b72a81a0012fdf2a5}} 2.2: {{e63dacf793fedc8a9eed9c7fc635cde5f9fd68f3}} 3.0: {{9218d7456b36d20ebe78bab23594e67d2f0c4a20}} //CC [~enigmacurry] was: When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will get stuck in a compaction loop throwing: {code} java.lang.UnsupportedOperationException at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382) at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147) at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503) at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80) at org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72) at org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100) at org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72) at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223) at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) at org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) {code} It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not surface the issue. It can be reproduced with the following (note -- changing jdks will need to be altered if you aren't on OSX) {code} #!/bin/sh echo "using java7 for cassandra-2.1 instance" export JAVA_HOME=$(/usr/libexec/java_home -v 1.7) ccm stop ccm remove upgrade_nodes ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes ccm start ccm node1 stress write n=500K -rate threads=4 -mode native cql3 sleep 10 for cver in 2.2 3.0 do echo "Draining all nodes" ccm node1 nodetool drain ccm stop echo "switching to java 8" export JAVA_HOME=$(/usr/libexec/java_home -v 1.8) echo "Upgrading to git:cassandra-$cver" ccm setdir -v git:cassandra-$cver ccm start echo "Sleeping to all version migrations" sleep 30 echo "Upgrading sstables" ccm node1 nodetool upgradesstables ccm node1 nodetool upgradesstables system ccm node1 nodetool compact system ccm node1 stress write n=500K -rate threads=4 -mode native cql3 sleep 10 done echo "***********" echo "instead of creating churn to cause compaction naturally forcing compaction of system keyspace" echo "***********" ccm node1 nodetool compact system ccm stop {code} The example uses {{nodetool compact system}} but it will also occur with {{nodetool upgradesstables system}}. I'm puzzled by that since the script runs {{upgradesstables}} on each iteration. Is the system keyspace not effected by the command without arguments? Ran against: 2.1: {{e889ee408bec5330c312ff6b72a81a0012fdf2a5}} 2.2: {{e63dacf793fedc8a9eed9c7fc635cde5f9fd68f3}} 3.0: {{9218d7456b36d20ebe78bab23594e67d2f0c4a20}} //CC [~enigmacurry] > UnsupportedOperationException when compacting system.size_estimates after 2.1 > -> 2.2 -> 3.0 upgrade > --------------------------------------------------------------------------------------------------- > > Key: CASSANDRA-10360 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10360 > Project: Cassandra > Issue Type: Bug > Reporter: Andrew Hust > Assignee: Sylvain Lebresne > Priority: Blocker > Fix For: 3.0.0 rc2 > > Attachments: size_estimates.tar.bz2, system.log.bz2 > > > When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will > get stuck in a compaction loop throwing: > {code} > java.lang.UnsupportedOperationException > at > org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147) > at > org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80) > at > org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72) > at > org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100) > at > org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72) > at > org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223) > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) > at > org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not > surface the issue. It can be reproduced with the following (note -- changing > jdks will need to be altered if you aren't on OSX) > {code} > #!/bin/sh > echo "using java7 for cassandra-2.1 instance" > export JAVA_HOME=$(/usr/libexec/java_home -v 1.7) > ccm stop > ccm remove upgrade_nodes > ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes > ccm start > ccm node1 stress write n=500K -rate threads=4 -mode native cql3 > sleep 10 > for cver in 3.0 > do > echo "Draining all nodes" > ccm node1 nodetool drain > ccm stop > echo "switching to java 8" > export JAVA_HOME=$(/usr/libexec/java_home -v 1.8) > echo "Upgrading to git:cassandra-$cver" > ccm setdir -v git:cassandra-$cver > ccm start > echo "Sleeping to all version migrations" > sleep 30 > echo "Upgrading sstables" > ccm node1 nodetool upgradesstables > ccm node1 nodetool upgradesstables system > ccm node1 nodetool compact system > ccm node1 stress write n=500K -rate threads=4 -mode native cql3 > sleep 10 > done > echo "***********" > echo "instead of creating churn to cause compaction naturally forcing > compaction of system keyspace" > echo "***********" > ccm node1 nodetool compact system > ccm stop > {code} > The example uses {{nodetool compact system}} but it will also occur with > {{nodetool upgradesstables system}}. I'm puzzled by that since the script > runs {{upgradesstables}} on each iteration. Is the system keyspace not > effected by the command without arguments? > Ran against: > 2.1: {{e889ee408bec5330c312ff6b72a81a0012fdf2a5}} > 2.2: {{e63dacf793fedc8a9eed9c7fc635cde5f9fd68f3}} > 3.0: {{9218d7456b36d20ebe78bab23594e67d2f0c4a20}} > //CC [~enigmacurry] -- This message was sent by Atlassian JIRA (v6.3.4#6332)