[
https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Hust updated CASSANDRA-10360:
------------------------------------
Description:
When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will get
stuck in a compaction loop throwing:
{code}
java.lang.UnsupportedOperationException
at
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382)
at
org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147)
at
org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503)
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80)
at
org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72)
at
org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100)
at
org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72)
at
org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
at
org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not
surface the issue. It can be reproduced with the following (note -- changing
jdks will need to be altered if you aren't on OSX)
{code}
#!/bin/sh
echo "using java7 for cassandra-2.1 instance"
export JAVA_HOME=$(/usr/libexec/java_home -v 1.7)
ccm stop
ccm remove upgrade_nodes
ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes
ccm start
ccm node1 stress write n=500K -rate threads=4 -mode native cql3
sleep 10
for cver in 3.0
do
echo "Draining all nodes"
ccm node1 nodetool drain
ccm stop
echo "switching to java 8"
export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)
echo "Upgrading to git:cassandra-$cver"
ccm setdir -v git:cassandra-$cver
ccm start
echo "Sleeping to all version migrations"
sleep 30
echo "Upgrading sstables"
ccm node1 nodetool upgradesstables
ccm node1 nodetool upgradesstables system
ccm node1 nodetool compact system
ccm node1 stress write n=500K -rate threads=4 -mode native cql3
sleep 10
done
echo "***********"
echo "instead of creating churn to cause compaction naturally forcing
compaction of system keyspace"
echo "***********"
ccm node1 nodetool compact system
ccm stop
{code}
The example uses {{nodetool compact system}} but it will also occur with
{{nodetool upgradesstables system}}. I'm puzzled by that since the script runs
{{upgradesstables}} on each iteration. Is the system keyspace not effected by
the command without arguments?
Ran against:
2.1: {{e889ee408bec5330c312ff6b72a81a0012fdf2a5}}
2.2: {{e63dacf793fedc8a9eed9c7fc635cde5f9fd68f3}}
3.0: {{9218d7456b36d20ebe78bab23594e67d2f0c4a20}}
//CC [~enigmacurry]
was:
When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will get
stuck in a compaction loop throwing:
{code}
java.lang.UnsupportedOperationException
at
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382)
at
org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147)
at
org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503)
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80)
at
org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72)
at
org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100)
at
org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72)
at
org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
at
org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not
surface the issue. It can be reproduced with the following (note -- changing
jdks will need to be altered if you aren't on OSX)
{code}
#!/bin/sh
echo "using java7 for cassandra-2.1 instance"
export JAVA_HOME=$(/usr/libexec/java_home -v 1.7)
ccm stop
ccm remove upgrade_nodes
ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes
ccm start
ccm node1 stress write n=500K -rate threads=4 -mode native cql3
sleep 10
for cver in 2.2 3.0
do
echo "Draining all nodes"
ccm node1 nodetool drain
ccm stop
echo "switching to java 8"
export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)
echo "Upgrading to git:cassandra-$cver"
ccm setdir -v git:cassandra-$cver
ccm start
echo "Sleeping to all version migrations"
sleep 30
echo "Upgrading sstables"
ccm node1 nodetool upgradesstables
ccm node1 nodetool upgradesstables system
ccm node1 nodetool compact system
ccm node1 stress write n=500K -rate threads=4 -mode native cql3
sleep 10
done
echo "***********"
echo "instead of creating churn to cause compaction naturally forcing
compaction of system keyspace"
echo "***********"
ccm node1 nodetool compact system
ccm stop
{code}
The example uses {{nodetool compact system}} but it will also occur with
{{nodetool upgradesstables system}}. I'm puzzled by that since the script runs
{{upgradesstables}} on each iteration. Is the system keyspace not effected by
the command without arguments?
Ran against:
2.1: {{e889ee408bec5330c312ff6b72a81a0012fdf2a5}}
2.2: {{e63dacf793fedc8a9eed9c7fc635cde5f9fd68f3}}
3.0: {{9218d7456b36d20ebe78bab23594e67d2f0c4a20}}
//CC [~enigmacurry]
> UnsupportedOperationException when compacting system.size_estimates after 2.1
> -> 2.2 -> 3.0 upgrade
> ---------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-10360
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10360
> Project: Cassandra
> Issue Type: Bug
> Reporter: Andrew Hust
> Assignee: Sylvain Lebresne
> Priority: Blocker
> Fix For: 3.0.0 rc2
>
> Attachments: size_estimates.tar.bz2, system.log.bz2
>
>
> When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will
> get stuck in a compaction loop throwing:
> {code}
> java.lang.UnsupportedOperationException
> at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382)
> at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147)
> at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80)
> at
> org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100)
> at
> org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72)
> at
> org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not
> surface the issue. It can be reproduced with the following (note -- changing
> jdks will need to be altered if you aren't on OSX)
> {code}
> #!/bin/sh
> echo "using java7 for cassandra-2.1 instance"
> export JAVA_HOME=$(/usr/libexec/java_home -v 1.7)
> ccm stop
> ccm remove upgrade_nodes
> ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes
> ccm start
> ccm node1 stress write n=500K -rate threads=4 -mode native cql3
> sleep 10
> for cver in 3.0
> do
> echo "Draining all nodes"
> ccm node1 nodetool drain
> ccm stop
> echo "switching to java 8"
> export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)
> echo "Upgrading to git:cassandra-$cver"
> ccm setdir -v git:cassandra-$cver
> ccm start
> echo "Sleeping to all version migrations"
> sleep 30
> echo "Upgrading sstables"
> ccm node1 nodetool upgradesstables
> ccm node1 nodetool upgradesstables system
> ccm node1 nodetool compact system
> ccm node1 stress write n=500K -rate threads=4 -mode native cql3
> sleep 10
> done
> echo "***********"
> echo "instead of creating churn to cause compaction naturally forcing
> compaction of system keyspace"
> echo "***********"
> ccm node1 nodetool compact system
> ccm stop
> {code}
> The example uses {{nodetool compact system}} but it will also occur with
> {{nodetool upgradesstables system}}. I'm puzzled by that since the script
> runs {{upgradesstables}} on each iteration. Is the system keyspace not
> effected by the command without arguments?
> Ran against:
> 2.1: {{e889ee408bec5330c312ff6b72a81a0012fdf2a5}}
> 2.2: {{e63dacf793fedc8a9eed9c7fc635cde5f9fd68f3}}
> 3.0: {{9218d7456b36d20ebe78bab23594e67d2f0c4a20}}
> //CC [~enigmacurry]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
