[jira] [Updated] (CASSANDRA-10874) running stress with compaction strategy and replication factor fails on read after write
[
https://issues.apache.org/jira/browse/CASSANDRA-10874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Hust updated CASSANDRA-10874:
Description:
When running a read stress after write stress with a compaction strategy and
replication factor matching the node count will fail with an exception.
{code}
Operation x0 on key(s) [38343433384b34364c30]: Data returned was not validated
{code}
Example run:
{code}
ccm create stress -v git:cassandra-3.0 -n 3 -s
ccm node1 stress write n=10M -rate threads=300 -schema replication\(factor=3\)
compaction\(strategy=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy\)
ccm node1 nodetool flush
ccm node1 nodetool compactionstats # check until quiet
ccm node1 stress read n=10M -rate threads=300
{code}
- This will fail with/out vnodes but will occasionally pass without vnodes.
- Changing the read phase to be CL=QUORUM will make it pass.
- Removing the replication factor on write will make it pass.
- Happens on all compaction strategies
So with that in mind I attempted to add a repair after the write phase. This
leads to 1 of 2 outcomes.
1: a repair that has a greater than 100% completion, usually stalls after a
bit, but have seen it get to >400% progress:
{code}
id compaction typekeyspace
table completed totalunit progress
2d5344c0-9dc8-11e5-9d5f-4fdec8d76c27Validation keyspace1
standard1 94722609949 44035292145 bytes215.11%
{code}
2: a repair that has a greatly inflated completed/total value, it will crunch
for a bit then lockup:
{code}
id compaction typekeyspace
table completed totalunit progress
8c4cf7f0-a34a-11e5-a321-777be88c58aeValidation keyspace1
standard1 0 874811100900 bytes 0.00%
❯ du -sh ~/.ccm/stress/node1/
2.4G ~/.ccm/stress/node1/
❯ du -sh ~/.ccm/stress
7.1G ~/.ccm/stress
{code}
This has been reproduced on cassandra-3.0 and cassandra-2.1 both locally and
using cstar_perf (links below).
A big twist is that cassandra-2.2 will pass the majority of the time. It will
complete successfully without the repair 8 out of 10 runs. This can be seen in
the cstar_perf links below.
cstar_perf runs:
http://cstar.datastax.com/tests/id/c8fa27a4-a205-11e5-8fbc-0256e416528f
http://cstar.datastax.com/tests/id/a254c572-a2ce-11e5-a8b9-0256e416528f
was:
When running a read stress after write stress with a compaction strategy and
replication factor matching the node count will fail with an exception.
{code}
Operation x0 on key(s) [38343433384b34364c30]: Data returned was not validated
{code}
Example run:
{code}
ccm create stress -v git:cassandra-3.0 -n 3 -s
ccm node1 stress write n=10M -rate threads=300 -schema replication\(factor=3\)
compaction\(strategy=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy\)
ccm node1 nodetool flush
ccm node1 nodetool compactionstats # check until quiet
ccm node1 stress read n=10M -rate threads=300
{code}
- This will fail with/out vnodes but will occasionally pass without vnodes.
- Changing the read phase to be CL=QUORUM will make it pass.
- Removing the replication factor on write will make it pass.
- Happens on all compaction strategies
So with that in mind I attempted to add a repair after the write phase. This
leads to 1 of 2 outcomes.
1: a repair that has a greater than 100% completion, usually stalls after a
bit, but have seen it get to >400% progress:
{code}
id compaction typekeyspace
table completed totalunit progress
2d5344c0-9dc8-11e5-9d5f-4fdec8d76c27Validation keyspace1
standard1 94722609949 44035292145 bytes215.11%
{code}
2: a repair that has a greatly inflated completed/total value, it will crunch
for a bit then lockup:
{code}
id compaction typekeyspace
table completed totalunit progress
8c4cf7f0-a34a-11e5-a321-777be88c58aeValidation keyspace1
standard1 0 874811100900 bytes 0.00%
❯ du -sh ~/.ccm/stress/node1/
2.4G ~/.ccm/stress/node1/
❯ du -sh ~/.ccm/stress
7.1G ~/.ccm/stress
{code}
This has been reproduced on cassandra-3.0 and cassandra-2.2 both locally and
using cstar_perf (links below).
A big twist is that cassandra-2.2 will pass the majority of the time. It will
complete successfully without the repair 8 out of 10 runs. This can be seen in
the cstar_perf links below.
cstar_perf runs:
http://cstar.datastax.com/tests/id/c8fa27a4-a205-11e5-8fbc-0256e416528f
http://cstar.datastax.com/tests/id/a254c572-a2ce-11e5-a8b9-0256e416528f
> running stress with compaction strategy and replication factor fails on read
> after write
>
[jira] [Created] (CASSANDRA-10874) running stress with compaction strategy and replication factor fails on read after write
Andrew Hust created CASSANDRA-10874:
---
Summary: running stress with compaction strategy and replication
factor fails on read after write
Key: CASSANDRA-10874
URL: https://issues.apache.org/jira/browse/CASSANDRA-10874
Project: Cassandra
Issue Type: Bug
Components: Tools
Reporter: Andrew Hust
When running a read stress after write stress with a compaction strategy and
replication factor matching the node count will fail with an exception.
{code}
Operation x0 on key(s) [38343433384b34364c30]: Data returned was not validated
{code}
Example run:
{code}
ccm create stress -v git:cassandra-3.0 -n 3 -s
ccm node1 stress write n=10M -rate threads=300 -schema replication\(factor=3\)
compaction\(strategy=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy\)
ccm node1 nodetool flush
ccm node1 nodetool compactionstats # check until quiet
ccm node1 stress read n=10M -rate threads=300
{code}
- This will fail with/out vnodes but will occasionally pass without vnodes.
- Changing the read phase to be CL=QUORUM will make it pass.
- Removing the replication factor on write will make it pass.
- Happens on all compaction strategies
So with that in mind I attempted to add a repair after the write phase. This
leads to 1 of 2 outcomes.
1: a repair that has a greater than 100% completion, usually stalls after a
bit, but have seen it get to >400% progress:
{code}
id compaction typekeyspace
table completed totalunit progress
2d5344c0-9dc8-11e5-9d5f-4fdec8d76c27Validation keyspace1
standard1 94722609949 44035292145 bytes215.11%
{code}
2: a repair that has a greatly inflated completed/total value, it will crunch
for a bit then lockup:
{code}
id compaction typekeyspace
table completed totalunit progress
8c4cf7f0-a34a-11e5-a321-777be88c58aeValidation keyspace1
standard1 0 874811100900 bytes 0.00%
❯ du -sh ~/.ccm/stress/node1/
2.4G ~/.ccm/stress/node1/
❯ du -sh ~/.ccm/stress
7.1G ~/.ccm/stress
{code}
This has been reproduced on cassandra-3.0 and cassandra-2.2 both locally and
using cstar_perf (links below).
A big twist is that cassandra-2.2 will pass the majority of the time. It will
complete successfully without the repair 8 out of 10 runs. This can be seen in
the cstar_perf links below.
cstar_perf runs:
http://cstar.datastax.com/tests/id/a8b6af02-a2ce-11e5-bb72-0256e416528f
http://cstar.datastax.com/tests/id/a254c572-a2ce-11e5-a8b9-0256e416528f
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10874) running stress with compaction strategy and replication factor fails on read after write
[
https://issues.apache.org/jira/browse/CASSANDRA-10874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Hust updated CASSANDRA-10874:
Description:
When running a read stress after write stress with a compaction strategy and
replication factor matching the node count will fail with an exception.
{code}
Operation x0 on key(s) [38343433384b34364c30]: Data returned was not validated
{code}
Example run:
{code}
ccm create stress -v git:cassandra-3.0 -n 3 -s
ccm node1 stress write n=10M -rate threads=300 -schema replication\(factor=3\)
compaction\(strategy=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy\)
ccm node1 nodetool flush
ccm node1 nodetool compactionstats # check until quiet
ccm node1 stress read n=10M -rate threads=300
{code}
- This will fail with/out vnodes but will occasionally pass without vnodes.
- Changing the read phase to be CL=QUORUM will make it pass.
- Removing the replication factor on write will make it pass.
- Happens on all compaction strategies
So with that in mind I attempted to add a repair after the write phase. This
leads to 1 of 2 outcomes.
1: a repair that has a greater than 100% completion, usually stalls after a
bit, but have seen it get to >400% progress:
{code}
id compaction typekeyspace
table completed totalunit progress
2d5344c0-9dc8-11e5-9d5f-4fdec8d76c27Validation keyspace1
standard1 94722609949 44035292145 bytes215.11%
{code}
2: a repair that has a greatly inflated completed/total value, it will crunch
for a bit then lockup:
{code}
id compaction typekeyspace
table completed totalunit progress
8c4cf7f0-a34a-11e5-a321-777be88c58aeValidation keyspace1
standard1 0 874811100900 bytes 0.00%
❯ du -sh ~/.ccm/stress/node1/
2.4G ~/.ccm/stress/node1/
❯ du -sh ~/.ccm/stress
7.1G ~/.ccm/stress
{code}
This has been reproduced on cassandra-3.0 and cassandra-2.2 both locally and
using cstar_perf (links below).
A big twist is that cassandra-2.2 will pass the majority of the time. It will
complete successfully without the repair 8 out of 10 runs. This can be seen in
the cstar_perf links below.
cstar_perf runs:
http://cstar.datastax.com/tests/id/c8fa27a4-a205-11e5-8fbc-0256e416528f
http://cstar.datastax.com/tests/id/a254c572-a2ce-11e5-a8b9-0256e416528f
was:
When running a read stress after write stress with a compaction strategy and
replication factor matching the node count will fail with an exception.
{code}
Operation x0 on key(s) [38343433384b34364c30]: Data returned was not validated
{code}
Example run:
{code}
ccm create stress -v git:cassandra-3.0 -n 3 -s
ccm node1 stress write n=10M -rate threads=300 -schema replication\(factor=3\)
compaction\(strategy=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy\)
ccm node1 nodetool flush
ccm node1 nodetool compactionstats # check until quiet
ccm node1 stress read n=10M -rate threads=300
{code}
- This will fail with/out vnodes but will occasionally pass without vnodes.
- Changing the read phase to be CL=QUORUM will make it pass.
- Removing the replication factor on write will make it pass.
- Happens on all compaction strategies
So with that in mind I attempted to add a repair after the write phase. This
leads to 1 of 2 outcomes.
1: a repair that has a greater than 100% completion, usually stalls after a
bit, but have seen it get to >400% progress:
{code}
id compaction typekeyspace
table completed totalunit progress
2d5344c0-9dc8-11e5-9d5f-4fdec8d76c27Validation keyspace1
standard1 94722609949 44035292145 bytes215.11%
{code}
2: a repair that has a greatly inflated completed/total value, it will crunch
for a bit then lockup:
{code}
id compaction typekeyspace
table completed totalunit progress
8c4cf7f0-a34a-11e5-a321-777be88c58aeValidation keyspace1
standard1 0 874811100900 bytes 0.00%
❯ du -sh ~/.ccm/stress/node1/
2.4G ~/.ccm/stress/node1/
❯ du -sh ~/.ccm/stress
7.1G ~/.ccm/stress
{code}
This has been reproduced on cassandra-3.0 and cassandra-2.2 both locally and
using cstar_perf (links below).
A big twist is that cassandra-2.2 will pass the majority of the time. It will
complete successfully without the repair 8 out of 10 runs. This can be seen in
the cstar_perf links below.
cstar_perf runs:
http://cstar.datastax.com/tests/id/a8b6af02-a2ce-11e5-bb72-0256e416528f
http://cstar.datastax.com/tests/id/a254c572-a2ce-11e5-a8b9-0256e416528f
> running stress with compaction strategy and replication factor fails on read
> after write
>
[jira] [Updated] (CASSANDRA-10697) Leak detected while running offline scrub
[
https://issues.apache.org/jira/browse/CASSANDRA-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Hust updated CASSANDRA-10697:
Component/s: Tools
> Leak detected while running offline scrub
> -
>
> Key: CASSANDRA-10697
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10697
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Environment: C* 2.1.9 on Debian Wheezy
>Reporter: mlowicki
>Priority: Critical
>
> I got couple of those:
> {code}
> ERROR 05:09:15 LEAK DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@3b60e162) to class
> org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier@1433208674:/var/lib/cassandra/data/sync/entity2-e24b5040199b11e5a30f75bb514ae072/sync-entity2-ka-405434
> was not released before the reference was garbage collected
> {code}
> and then:
> {code}
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:99)
> at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:81)
> at
> org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:353)
> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:444)
> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:424)
> at
> org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:378)
> at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:348)
> at
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:327)
> at
> org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:397)
> at
> org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:381)
> at
> org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:75)
> at
> org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52)
> at
> org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:120)
> at
> org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:202)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> com.google.common.collect.Iterators$7.computeNext(Iterators.java:645)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.db.ColumnIndex$Builder.buildForCompaction(ColumnIndex.java:165)
> at
> org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:121)
> at
> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:192)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:127)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.tryAppend(SSTableRewriter.java:158)
> at
> org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:220)
> at
> org.apache.cassandra.tools.StandaloneScrubber.main(StandaloneScrubber.java:116)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10743) Failed upgradesstables (upgrade from 2.2.2 to 3.0.0)
[
https://issues.apache.org/jira/browse/CASSANDRA-10743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15018895#comment-15018895
]
Andrew Hust commented on CASSANDRA-10743:
-
Would it be possible to get the full schema? I can attempt to narrow it down
to find the problem one(s). You can get the full schema via {{cqlsh}} ex:
{{echo -e "desc schema\n" | cqlsh > schema.ddl}}
> Failed upgradesstables (upgrade from 2.2.2 to 3.0.0)
>
>
> Key: CASSANDRA-10743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10743
> Project: Cassandra
> Issue Type: Bug
> Environment: CentOS Linux release 7.1.1503, OpenJDK Runtime
> Environment (build 1.8.0_65-b17), DSC Cassandra 3.0.0 (tar.gz)
>Reporter: Gábor Auth
>
> {code}
> [cassandra@dc01-rack01-cass01 ~]$
> /home/cassandra/dsc-cassandra-3.0.0/bin/nodetool upgradesstables
> error: null
> -- StackTrace --
> java.lang.UnsupportedOperationException
> at
> org.apache.cassandra.db.rows.CellPath$EmptyCellPath.get(CellPath.java:143)
> at
> org.apache.cassandra.db.marshal.CollectionType$CollectionPathSerializer.serializedSize(CollectionType.java:226)
> at
> org.apache.cassandra.db.rows.BufferCell$Serializer.serializedSize(BufferCell.java:325)
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.sizeOfComplexColumn(UnfilteredSerializer.java:297)
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.serializedRowBodySize(UnfilteredSerializer.java:282)
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:163)
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:108)
> at
> org.apache.cassandra.db.ColumnIndex$Builder.add(ColumnIndex.java:144)
> at
> org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:112)
> at
> org.apache.cassandra.db.ColumnIndex.writeAndBuildIndex(ColumnIndex.java:52)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:149)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:121)
> at
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:57)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:110)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:182)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:397)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10715) Filtering on NULL returns ReadFailure exception
[
https://issues.apache.org/jira/browse/CASSANDRA-10715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Hust updated CASSANDRA-10715:
Component/s: CQL
> Filtering on NULL returns ReadFailure exception
> ---
>
> Key: CASSANDRA-10715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10715
> Project: Cassandra
> Issue Type: Bug
> Components: CQL
> Environment: C* 3.0.0 | cqlsh | C# driver 3.0.0beta2 | Windows 2012 R2
>Reporter: Kishan Karunaratne
>
> This is an issue I first noticed through the C# driver, but I was able to
> repro on cqlsh, leading me to believe this is a Cassandra bug.
> Given the following schema:
> {noformat}
> CREATE TABLE "TestKeySpace_4928dc892922"."coolMovies" (
> unique_movie_title text,
> movie_maker text,
> director text,
> list list,
> "mainGuy" text,
> "yearMade" int,
> PRIMARY KEY ((unique_movie_title, movie_maker), director)
> ) WITH CLUSTERING ORDER BY (director ASC)
> {noformat}
> Executing a SELECT with FILTERING on a non-PK column, using a NULL as the
> argument:
> {noformat}
> SELECT "mainGuy", "movie_maker", "unique_movie_title", "list", "director",
> "yearMade" FROM "coolMovies" WHERE "mainGuy" = null ALLOW FILTERING
> {noformat}
> returns a ReadFailure exception:
> {noformat}
> cqlsh:TestKeySpace_4c8f2cf8d5cc> SELECT "mainGuy", "movie_maker",
> "unique_movie_title", "list", "director", "yearMade" FROM "coolMovies" WHERE
> "mainGuy" = null ALLOW FILTERING;
> ←[0;1;31mTraceback (most recent call last):
> File "C:\Users\Kishan\.ccm\repository\3.0.0\bin\\cqlsh.py", line 1216, in
> perform_simple_statement
> result = future.result()
> File
> "C:\Users\Kishan\.ccm\repository\3.0.0\bin\..\lib\cassandra-driver-internal-only-3.0.0a3.post0-3f15725.zip\cassandra-driver-3.0.0a3.post0-3f15725\cassandra\cluster.py",
> line 3118, in result
> raise self._final_exception
> ReadFailure: code=1300 [Replica(s) failed to execute read] message="Operation
> failed - received 0 responses and 1 failures" info={'failures': 1,
> 'received_responses': 0, 'required_responses': 1, 'cons
> istency': 'ONE'}
> ←[0m
> {noformat}
> Cassandra log shows:
> {noformat}
> WARN [SharedPool-Worker-2] 2015-11-16 13:51:00,259
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-2,10,main]: {}
> java.lang.AssertionError: null
> at
> org.apache.cassandra.db.filter.RowFilter$SimpleExpression.isSatisfiedBy(RowFilter.java:581)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.filter.RowFilter$CQLFilter$1IsSatisfiedFilter.applyToRow(RowFilter.java:243)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.transform.BaseRows.applyOne(BaseRows.java:95)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.transform.BaseRows.add(BaseRows.java:86)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.transform.UnfilteredRows.add(UnfilteredRows.java:21)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.transform.Transformation.add(Transformation.java:136)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.transform.Transformation.apply(Transformation.java:102)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.filter.RowFilter$CQLFilter$1IsSatisfiedFilter.applyToPartition(RowFilter.java:233)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.filter.RowFilter$CQLFilter$1IsSatisfiedFilter.applyToPartition(RowFilter.java:227)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:76)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:293)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:136)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:128)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:123)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:288)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1692)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
>
[jira] [Commented] (CASSANDRA-10711) NoSuchElementException when executing empty batch.
[
https://issues.apache.org/jira/browse/CASSANDRA-10711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007570#comment-15007570
]
Andrew Hust commented on CASSANDRA-10711:
-
- confirmed that 2.2 {{73a730f926d25a7d4f693507937b8565b701259c}} does not
throw error
- confirmed both 3.0 {{c0480d8bbddf111e4cd7c67ef7c0daeec3ece2dc}} and trunk
{{0010fce6d2c9a811eb66de077b69a83dce29a6ff}} throw same
{{NoSuchElementException}} in cqlsh
- added [dtest|https://github.com/riptano/cassandra-dtest/pull/662] to verify
fix when made
> NoSuchElementException when executing empty batch.
> --
>
> Key: CASSANDRA-10711
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10711
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra 3.0, OSS 42.1
>Reporter: Jaroslav Kamenik
> Fix For: 3.0.1, 3.1
>
>
> After upgrade to C* 3.0, it fails when executes empty batch:
> java.util.NoSuchElementException: null
> at java.util.ArrayList$Itr.next(ArrayList.java:854) ~[na:1.8.0_60]
> at
> org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:737)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.cql3.statements.BatchStatement.executeWithoutConditions(BatchStatement.java:356)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.cql3.statements.BatchStatement.execute(BatchStatement.java:337)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.cql3.statements.BatchStatement.execute(BatchStatement.java:323)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.cql3.QueryProcessor.processBatch(QueryProcessor.java:490)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.cql3.QueryProcessor.processBatch(QueryProcessor.java:480)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.transport.messages.BatchMessage.execute(BatchMessage.java:217)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507)
> [apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401)
> [apache-cassandra-3.0.0.jar:3.0.0]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
> [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
> [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_60]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> [apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.0.0.jar:3.0.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10711) NoSuchElementException when executing empty batch.
[ https://issues.apache.org/jira/browse/CASSANDRA-10711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Hust updated CASSANDRA-10711: Labels: triaged (was: ) > NoSuchElementException when executing empty batch. > -- > > Key: CASSANDRA-10711 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10711 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 3.0, OSS 42.1 >Reporter: Jaroslav Kamenik > Labels: triaged > Fix For: 3.0.1, 3.1 > > > After upgrade to C* 3.0, it fails when executes empty batch: > java.util.NoSuchElementException: null > at java.util.ArrayList$Itr.next(ArrayList.java:854) ~[na:1.8.0_60] > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:737) > ~[apache-cassandra-3.0.0.jar:3.0.0] > at > org.apache.cassandra.cql3.statements.BatchStatement.executeWithoutConditions(BatchStatement.java:356) > ~[apache-cassandra-3.0.0.jar:3.0.0] > at > org.apache.cassandra.cql3.statements.BatchStatement.execute(BatchStatement.java:337) > ~[apache-cassandra-3.0.0.jar:3.0.0] > at > org.apache.cassandra.cql3.statements.BatchStatement.execute(BatchStatement.java:323) > ~[apache-cassandra-3.0.0.jar:3.0.0] > at > org.apache.cassandra.cql3.QueryProcessor.processBatch(QueryProcessor.java:490) > ~[apache-cassandra-3.0.0.jar:3.0.0] > at > org.apache.cassandra.cql3.QueryProcessor.processBatch(QueryProcessor.java:480) > ~[apache-cassandra-3.0.0.jar:3.0.0] > at > org.apache.cassandra.transport.messages.BatchMessage.execute(BatchMessage.java:217) > ~[apache-cassandra-3.0.0.jar:3.0.0] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507) > [apache-cassandra-3.0.0.jar:3.0.0] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401) > [apache-cassandra-3.0.0.jar:3.0.0] > at > io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_60] > at > org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) > [apache-cassandra-3.0.0.jar:3.0.0] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.0.0.jar:3.0.0] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-10603) Fix CQL syntax errors in upgrade_through_versions_test
[
https://issues.apache.org/jira/browse/CASSANDRA-10603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Hust resolved CASSANDRA-10603.
-
Resolution: Fixed
> Fix CQL syntax errors in upgrade_through_versions_test
> --
>
> Key: CASSANDRA-10603
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10603
> Project: Cassandra
> Issue Type: Sub-task
>Reporter: Jim Witschey
>Assignee: Andrew Hust
>
> In the {{cassandra_upgrade_2.1_to_3.0_proto_v3}} upgrade tests on CassCI,
> some of the tests are failing with the following error:
> {code}
>
> {code}
> The tests that fail this way [(at least as of this
> run)|http://cassci.datastax.com/view/Upgrades/job/cassandra_upgrade_2.1_to_3.0_proto_v3/10/testReport/]
> are the following:
> {code}
> upgrade_through_versions_test.TestUpgrade_from_2_1_latest_tag_to_2_1_HEAD.parallel_upgrade_with_internode_ssl_test
> upgrade_through_versions_test.TestUpgrade_from_2_1_latest_tag_to_2_1_HEAD.parallel_upgrade_test
> upgrade_through_versions_test.TestUpgrade_from_2_1_latest_tag_to_2_1_HEAD.bootstrap_multidc_test
> upgrade_through_versions_test.TestUpgrade_from_2_1_latest_tag_to_2_1_HEAD.bootstrap_test
> upgrade_through_versions_test.TestUpgrade_from_2_1_latest_tag_to_cassandra_3_0_HEAD.bootstrap_multidc_test
> upgrade_through_versions_test.TestUpgrade_from_2_1_latest_tag_to_cassandra_3_0_HEAD.bootstrap_test
> upgrade_through_versions_test.TestUpgrade_from_2_1_latest_tag_to_cassandra_3_0_HEAD.parallel_upgrade_test
> upgrade_through_versions_test.TestUpgrade_from_2_1_latest_tag_to_cassandra_3_0_HEAD.parallel_upgrade_with_internode_ssl_test
> {code}
> There may be other tests in other protocol upgrade jobs that fail this way,
> but I haven't dug through yet to see.
> Assigning to [~rhatch] since, afaik, you're the most likely person to
> understand the problem. Feel free to reassign, of course.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10611) Upgrade test on 2.1->3.0 path fails with configuration problems
[
https://issues.apache.org/jira/browse/CASSANDRA-10611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981090#comment-14981090
]
Andrew Hust commented on CASSANDRA-10611:
-
Fixed with PR: https://github.com/riptano/cassandra-dtest/pull/633
> Upgrade test on 2.1->3.0 path fails with configuration problems
> ---
>
> Key: CASSANDRA-10611
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10611
> Project: Cassandra
> Issue Type: Sub-task
>Reporter: Jim Witschey
>Assignee: Andrew Hust
> Fix For: 3.0.0
>
>
> The following test fails on the uprgrade path from 2.1 to 3.0:
> http://cassci.datastax.com/view/Upgrades/job/cassandra_upgrade_2.1_to_3.0_proto_v3/10/testReport/upgrade_through_versions_test/TestUpgrade_from_3_0_latest_tag_to_3_0_HEAD/bootstrap_multidc_test/
> I believe it's basically a configuration error; the cluster likely just needs
> to be reconfigured in the test:
> {code}
> code=2200 [Invalid query] message="User-defined functions are disabled in
> cassandra.yaml - set enable_user_defined_functions=true to enable"
> {code}
> Assigning to [~rhatch] for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-10611) Upgrade test on 2.1->3.0 path fails with configuration problems
[
https://issues.apache.org/jira/browse/CASSANDRA-10611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Hust resolved CASSANDRA-10611.
-
Resolution: Fixed
> Upgrade test on 2.1->3.0 path fails with configuration problems
> ---
>
> Key: CASSANDRA-10611
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10611
> Project: Cassandra
> Issue Type: Sub-task
>Reporter: Jim Witschey
>Assignee: Andrew Hust
> Fix For: 3.0.0
>
>
> The following test fails on the uprgrade path from 2.1 to 3.0:
> http://cassci.datastax.com/view/Upgrades/job/cassandra_upgrade_2.1_to_3.0_proto_v3/10/testReport/upgrade_through_versions_test/TestUpgrade_from_3_0_latest_tag_to_3_0_HEAD/bootstrap_multidc_test/
> I believe it's basically a configuration error; the cluster likely just needs
> to be reconfigured in the test:
> {code}
> code=2200 [Invalid query] message="User-defined functions are disabled in
> cassandra.yaml - set enable_user_defined_functions=true to enable"
> {code}
> Assigning to [~rhatch] for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-10611) Upgrade test on 2.1->3.0 path fails with configuration problems
[
https://issues.apache.org/jira/browse/CASSANDRA-10611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Hust reassigned CASSANDRA-10611:
---
Assignee: Andrew Hust (was: Russ Hatch)
> Upgrade test on 2.1->3.0 path fails with configuration problems
> ---
>
> Key: CASSANDRA-10611
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10611
> Project: Cassandra
> Issue Type: Sub-task
>Reporter: Jim Witschey
>Assignee: Andrew Hust
> Fix For: 3.0.0
>
>
> The following test fails on the uprgrade path from 2.1 to 3.0:
> http://cassci.datastax.com/view/Upgrades/job/cassandra_upgrade_2.1_to_3.0_proto_v3/10/testReport/upgrade_through_versions_test/TestUpgrade_from_3_0_latest_tag_to_3_0_HEAD/bootstrap_multidc_test/
> I believe it's basically a configuration error; the cluster likely just needs
> to be reconfigured in the test:
> {code}
> code=2200 [Invalid query] message="User-defined functions are disabled in
> cassandra.yaml - set enable_user_defined_functions=true to enable"
> {code}
> Assigning to [~rhatch] for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-10603) Fix CQL syntax errors in upgrade_through_versions_test
[
https://issues.apache.org/jira/browse/CASSANDRA-10603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Hust reassigned CASSANDRA-10603:
---
Assignee: Andrew Hust (was: Russ Hatch)
> Fix CQL syntax errors in upgrade_through_versions_test
> --
>
> Key: CASSANDRA-10603
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10603
> Project: Cassandra
> Issue Type: Sub-task
>Reporter: Jim Witschey
>Assignee: Andrew Hust
>
> In the {{cassandra_upgrade_2.1_to_3.0_proto_v3}} upgrade tests on CassCI,
> some of the tests are failing with the following error:
> {code}
>
> {code}
> The tests that fail this way [(at least as of this
> run)|http://cassci.datastax.com/view/Upgrades/job/cassandra_upgrade_2.1_to_3.0_proto_v3/10/testReport/]
> are the following:
> {code}
> upgrade_through_versions_test.TestUpgrade_from_2_1_latest_tag_to_2_1_HEAD.parallel_upgrade_with_internode_ssl_test
> upgrade_through_versions_test.TestUpgrade_from_2_1_latest_tag_to_2_1_HEAD.parallel_upgrade_test
> upgrade_through_versions_test.TestUpgrade_from_2_1_latest_tag_to_2_1_HEAD.bootstrap_multidc_test
> upgrade_through_versions_test.TestUpgrade_from_2_1_latest_tag_to_2_1_HEAD.bootstrap_test
> upgrade_through_versions_test.TestUpgrade_from_2_1_latest_tag_to_cassandra_3_0_HEAD.bootstrap_multidc_test
> upgrade_through_versions_test.TestUpgrade_from_2_1_latest_tag_to_cassandra_3_0_HEAD.bootstrap_test
> upgrade_through_versions_test.TestUpgrade_from_2_1_latest_tag_to_cassandra_3_0_HEAD.parallel_upgrade_test
> upgrade_through_versions_test.TestUpgrade_from_2_1_latest_tag_to_cassandra_3_0_HEAD.parallel_upgrade_with_internode_ssl_test
> {code}
> There may be other tests in other protocol upgrade jobs that fail this way,
> but I haven't dug through yet to see.
> Assigning to [~rhatch] since, afaik, you're the most likely person to
> understand the problem. Feel free to reassign, of course.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10603) Fix CQL syntax errors in upgrade_through_versions_test
[
https://issues.apache.org/jira/browse/CASSANDRA-10603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981081#comment-14981081
]
Andrew Hust commented on CASSANDRA-10603:
-
Fixed with PR: https://github.com/riptano/cassandra-dtest/pull/633
> Fix CQL syntax errors in upgrade_through_versions_test
> --
>
> Key: CASSANDRA-10603
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10603
> Project: Cassandra
> Issue Type: Sub-task
>Reporter: Jim Witschey
>Assignee: Andrew Hust
>
> In the {{cassandra_upgrade_2.1_to_3.0_proto_v3}} upgrade tests on CassCI,
> some of the tests are failing with the following error:
> {code}
>
> {code}
> The tests that fail this way [(at least as of this
> run)|http://cassci.datastax.com/view/Upgrades/job/cassandra_upgrade_2.1_to_3.0_proto_v3/10/testReport/]
> are the following:
> {code}
> upgrade_through_versions_test.TestUpgrade_from_2_1_latest_tag_to_2_1_HEAD.parallel_upgrade_with_internode_ssl_test
> upgrade_through_versions_test.TestUpgrade_from_2_1_latest_tag_to_2_1_HEAD.parallel_upgrade_test
> upgrade_through_versions_test.TestUpgrade_from_2_1_latest_tag_to_2_1_HEAD.bootstrap_multidc_test
> upgrade_through_versions_test.TestUpgrade_from_2_1_latest_tag_to_2_1_HEAD.bootstrap_test
> upgrade_through_versions_test.TestUpgrade_from_2_1_latest_tag_to_cassandra_3_0_HEAD.bootstrap_multidc_test
> upgrade_through_versions_test.TestUpgrade_from_2_1_latest_tag_to_cassandra_3_0_HEAD.bootstrap_test
> upgrade_through_versions_test.TestUpgrade_from_2_1_latest_tag_to_cassandra_3_0_HEAD.parallel_upgrade_test
> upgrade_through_versions_test.TestUpgrade_from_2_1_latest_tag_to_cassandra_3_0_HEAD.parallel_upgrade_with_internode_ssl_test
> {code}
> There may be other tests in other protocol upgrade jobs that fail this way,
> but I haven't dug through yet to see.
> Assigning to [~rhatch] since, afaik, you're the most likely person to
> understand the problem. Feel free to reassign, of course.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10452) Fix resummable_bootstrap_test and bootstrap_with_reset_bootstrap_state_test
[
https://issues.apache.org/jira/browse/CASSANDRA-10452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976765#comment-14976765
]
Andrew Hust commented on CASSANDRA-10452:
-
Confirmed tests are no longer flapping and pass successfully when ran without
asserts enabled. Closing.
> Fix resummable_bootstrap_test and bootstrap_with_reset_bootstrap_state_test
> ---
>
> Key: CASSANDRA-10452
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10452
> Project: Cassandra
> Issue Type: Sub-task
>Reporter: Jim Witschey
>Assignee: Yuki Morishita
> Fix For: 3.0.0
>
>
> {{bootstrap_test.py:TestBootstrap.resumable_bootstrap_test}} has been
> flapping on CassCI lately:
> http://cassci.datastax.com/view/trunk/job/cassandra-3.0_dtest/lastCompletedBuild/testReport/bootstrap_test/TestBootstrap/resumable_bootstrap_test/history/
> I have not been able to reproduce on OpenStack. I'm assigning [~yukim] for
> now, but feel free to reassign.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10522) counter upgrade dtest fails on 3.0 with JVM assertions disabled
[
https://issues.apache.org/jira/browse/CASSANDRA-10522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959736#comment-14959736
]
Andrew Hust commented on CASSANDRA-10522:
-
Confirmed that these tests (and duplicate jira tests) now pass and no exception
is thrown.
Ran on:
yukim/10522: {{93783039918f8662760195e0f33c4cab20b17c8d}}
> counter upgrade dtest fails on 3.0 with JVM assertions disabled
> ---
>
> Key: CASSANDRA-10522
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10522
> Project: Cassandra
> Issue Type: Sub-task
>Reporter: Andrew Hust
>Assignee: Yuki Morishita
> Fix For: 3.0.0 rc2
>
>
> {{counter_tests.TestCounters.upgrade_test}}
> will fail when run on a cluster with JVM assertions disabled. The tests will
> hang when cassandra throws the following exception:
> {code}
> java.lang.IllegalStateException: No match found
> at java.util.regex.Matcher.group(Matcher.java:536) ~[na:1.8.0_60]
> at org.apache.cassandra.db.lifecycle.LogFile.make(LogFile.java:52)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LogTransaction.removeUnfinishedLeftovers(LogTransaction.java:399)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.removeUnfinishedLeftovers(LifecycleTransaction.java:552)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:571)
> ~[main/:na]
> at
> org.apache.cassandra.service.StartupChecks$7.execute(StartupChecks.java:274)
> ~[main/:na]
> at
> org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:103)
> ~[main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:169)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:548)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:676)
> [main/:na]
> {code}
> These tests both pass with/without JVM assertions on C* 2.2 and pass on 3.0
> when assertions are enabled.
> Ran against:
> apache/cassandra-2.2: {{7cab3272455bdd16b639c510416ae339a8613414}}
> apache/cassandra-3.0: {{f21c888510b0dbbea1a63459476f2dc54093de63}}
> Ran with cmd:
> {{JVM_EXTRA_OPTS=-da PRINT_DEBUG=true nosetests -xsv
> counter_tests.TestCounters.upgrade_test}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10522) counter upgrade dtest fails on 3.0 with JVM assertions disabled
[
https://issues.apache.org/jira/browse/CASSANDRA-10522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957118#comment-14957118
]
Andrew Hust commented on CASSANDRA-10522:
-
Both failures throw the same exception on startup and might be one issue.
> counter upgrade dtest fails on 3.0 with JVM assertions disabled
> ---
>
> Key: CASSANDRA-10522
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10522
> Project: Cassandra
> Issue Type: Sub-task
>Reporter: Andrew Hust
> Fix For: 3.0.0 rc2
>
>
> {{counter_tests.TestCounters.upgrade_test}}
> will fail when run on a cluster with JVM assertions disabled. The tests will
> hang when cassandra throws the following exception:
> {code}
> java.lang.IllegalStateException: No match found
> at java.util.regex.Matcher.group(Matcher.java:536) ~[na:1.8.0_60]
> at org.apache.cassandra.db.lifecycle.LogFile.make(LogFile.java:52)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LogTransaction.removeUnfinishedLeftovers(LogTransaction.java:399)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.removeUnfinishedLeftovers(LifecycleTransaction.java:552)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:571)
> ~[main/:na]
> at
> org.apache.cassandra.service.StartupChecks$7.execute(StartupChecks.java:274)
> ~[main/:na]
> at
> org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:103)
> ~[main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:169)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:548)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:676)
> [main/:na]
> {code}
> These tests both pass with/without JVM assertions on C* 2.2 and pass on 3.0
> when assertions are enabled.
> Ran against:
> apache/cassandra-2.2: {{7cab3272455bdd16b639c510416ae339a8613414}}
> apache/cassandra-3.0: {{f21c888510b0dbbea1a63459476f2dc54093de63}}
> Ran with cmd:
> {{JVM_EXTRA_OPTS=-da PRINT_DEBUG=true nosetests -xsv
> counter_tests.TestCounters.upgrade_test}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-10521) snapshot dtests failing on 3.0 when JVM assertions disabled
Andrew Hust created CASSANDRA-10521:
---
Summary: snapshot dtests failing on 3.0 when JVM assertions
disabled
Key: CASSANDRA-10521
URL: https://issues.apache.org/jira/browse/CASSANDRA-10521
Project: Cassandra
Issue Type: Sub-task
Reporter: Andrew Hust
Fix For: 3.0.0 rc2
{{snapshot_test.TestArchiveCommitlog.test_archive_commitlog_with_active_commitlog}}
and
{{snapshot_test.TestArchiveCommitlog.test_archive_commitlog_point_in_time_with_active_commitlog}}
will fail when run on a cluster with JVM assertions disabled. The tests will
hang when cassandra throws the following exception (in both cases):
{code}
java.lang.IllegalStateException: No match found
at java.util.regex.Matcher.group(Matcher.java:536) ~[na:1.8.0_60]
at org.apache.cassandra.db.lifecycle.LogFile.make(LogFile.java:52)
~[main/:na]
at
org.apache.cassandra.db.lifecycle.LogTransaction.removeUnfinishedLeftovers(LogTransaction.java:399)
~[main/:na]
at
org.apache.cassandra.db.lifecycle.LifecycleTransaction.removeUnfinishedLeftovers(LifecycleTransaction.java:552)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:571)
~[main/:na]
at
org.apache.cassandra.service.StartupChecks$7.execute(StartupChecks.java:274)
~[main/:na]
at
org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:103)
~[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:169)
[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:548)
[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:676)
[main/:na]
{code}
These tests both pass with/without JVM assertions on C* 2.2 and pass on 3.0
when assertions are enabled.
Ran against:
apache/cassandra-2.2: {{7cab3272455bdd16b639c510416ae339a8613414}}
apache/cassandra-3.0: {{f21c888510b0dbbea1a63459476f2dc54093de63}}
Ran with cmd:
{{JVM_EXTRA_OPTS=-da PRINT_DEBUG=true nosetests -xsv
snapshot_test.py:TestArchiveCommitlog.test_archive_commitlog_point_in_time_with_active_commitlog}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-10522) counter upgrade dtest fails on 3.0 with JVM assertions disabled
Andrew Hust created CASSANDRA-10522:
---
Summary: counter upgrade dtest fails on 3.0 with JVM assertions
disabled
Key: CASSANDRA-10522
URL: https://issues.apache.org/jira/browse/CASSANDRA-10522
Project: Cassandra
Issue Type: Sub-task
Reporter: Andrew Hust
Fix For: 3.0.0 rc2
{{counter_tests.TestCounters.upgrade_test}}
will fail when run on a cluster with JVM assertions disabled. The tests will
hang when cassandra throws the following exception:
{code}
java.lang.IllegalStateException: No match found
at java.util.regex.Matcher.group(Matcher.java:536) ~[na:1.8.0_60]
at org.apache.cassandra.db.lifecycle.LogFile.make(LogFile.java:52)
~[main/:na]
at
org.apache.cassandra.db.lifecycle.LogTransaction.removeUnfinishedLeftovers(LogTransaction.java:399)
~[main/:na]
at
org.apache.cassandra.db.lifecycle.LifecycleTransaction.removeUnfinishedLeftovers(LifecycleTransaction.java:552)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:571)
~[main/:na]
at
org.apache.cassandra.service.StartupChecks$7.execute(StartupChecks.java:274)
~[main/:na]
at
org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:103)
~[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:169)
[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:548)
[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:676)
[main/:na]
{code}
These tests both pass with/without JVM assertions on C* 2.2 and pass on 3.0
when assertions are enabled.
Ran against:
apache/cassandra-2.2: {{7cab3272455bdd16b639c510416ae339a8613414}}
apache/cassandra-3.0: {{f21c888510b0dbbea1a63459476f2dc54093de63}}
Ran with cmd:
{{JVM_EXTRA_OPTS=-da PRINT_DEBUG=true nosetests -xsv
counter_tests.TestCounters.upgrade_test}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-10521) snapshot dtests failing on 3.0 when JVM assertions disabled
[
https://issues.apache.org/jira/browse/CASSANDRA-10521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Hust resolved CASSANDRA-10521.
-
Resolution: Duplicate
> snapshot dtests failing on 3.0 when JVM assertions disabled
> ---
>
> Key: CASSANDRA-10521
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10521
> Project: Cassandra
> Issue Type: Sub-task
>Reporter: Andrew Hust
> Fix For: 3.0.0 rc2
>
>
> {{snapshot_test.TestArchiveCommitlog.test_archive_commitlog_with_active_commitlog}}
> and
> {{snapshot_test.TestArchiveCommitlog.test_archive_commitlog_point_in_time_with_active_commitlog}}
> will fail when run on a cluster with JVM assertions disabled. The tests will
> hang when cassandra throws the following exception (in both cases):
> {code}
> java.lang.IllegalStateException: No match found
> at java.util.regex.Matcher.group(Matcher.java:536) ~[na:1.8.0_60]
> at org.apache.cassandra.db.lifecycle.LogFile.make(LogFile.java:52)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LogTransaction.removeUnfinishedLeftovers(LogTransaction.java:399)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.removeUnfinishedLeftovers(LifecycleTransaction.java:552)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:571)
> ~[main/:na]
> at
> org.apache.cassandra.service.StartupChecks$7.execute(StartupChecks.java:274)
> ~[main/:na]
> at
> org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:103)
> ~[main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:169)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:548)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:676)
> [main/:na]
> {code}
> These tests both pass with/without JVM assertions on C* 2.2 and pass on 3.0
> when assertions are enabled.
> Ran against:
> apache/cassandra-2.2: {{7cab3272455bdd16b639c510416ae339a8613414}}
> apache/cassandra-3.0: {{f21c888510b0dbbea1a63459476f2dc54093de63}}
> Ran with cmd:
> {{JVM_EXTRA_OPTS=-da PRINT_DEBUG=true nosetests -xsv
> snapshot_test.py:TestArchiveCommitlog.test_archive_commitlog_point_in_time_with_active_commitlog}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10452) Fix flapping resummable_bootstrap_test
[
https://issues.apache.org/jira/browse/CASSANDRA-10452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14953215#comment-14953215
]
Andrew Hust commented on CASSANDRA-10452:
-
{{bootstrap_test.py:TestBootstrap.resumable_bootstrap_test}} and
{{bootstrap_test.py:TestBootstrap.bootstrap_with_reset_bootstrap_state_test}}
will consistently fail if jvm assertions are disabled.
{{resumable_bootstrap_test}} will fail with this exception:
{code}
ERROR [main] 2015-10-12 09:21:58,088 CassandraDaemon.java:689 - Exception
encountered during startup
java.lang.IllegalStateException: No match found
at java.util.regex.Matcher.group(Matcher.java:536) ~[na:1.8.0_60]
at org.apache.cassandra.db.lifecycle.LogFile.make(LogFile.java:52)
~[main/:na]
at
org.apache.cassandra.db.lifecycle.LogTransaction.removeUnfinishedLeftovers(LogTransaction.java:399)
~[main/:na]
at
org.apache.cassandra.db.lifecycle.LifecycleTransaction.removeUnfinishedLeftovers(LifecycleTransaction.java:552)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:571)
~[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:239)
[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:548)
[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:676)
[main/:na]
{code}
{{bootstrap_with_reset_bootstrap_state_test}} will fail with this exception:
{code}
ERROR [STREAM-IN-/127.0.0.1] 2015-10-12 09:26:04,777 StreamSession.java:520 -
[Stream #8d0ebd40-70f5-11e5-92fe-517de766f243] Streaming error occurred
java.io.EOFException: null
at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
~[na:1.8.0_60]
at
org.apache.cassandra.utils.BytesReadTracker.readUnsignedShort(BytesReadTracker.java:142)
~[main/:na]
at
org.apache.cassandra.utils.ByteBufferUtil.readShortLength(ByteBufferUtil.java:366)
~[main/:na]
at
org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:376)
~[main/:na]
at
org.apache.cassandra.streaming.StreamReader$StreamDeserializer.newPartition(StreamReader.java:198)
~[main/:na]
at
org.apache.cassandra.streaming.StreamReader.writePartition(StreamReader.java:168)
~[main/:na]
at
org.apache.cassandra.streaming.StreamReader.read(StreamReader.java:109)
~[main/:na]
at
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
~[main/:na]
at
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
~[main/:na]
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
~[main/:na]
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
~[main/:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
{code}
Ran against:
apache/cassandra-3.0: {{345be2057383895f8d4c6b07a89357a0e87441be}}
Ran with command:
{{KEEP_LOGS=true JVM_EXTRA_OPTS=-da PRINT_DEBUG=true nosetests -xsv
bootstrap_test.py:TestBootstrap.bootstrap_with_reset_bootstrap_state_test}}
> Fix flapping resummable_bootstrap_test
> --
>
> Key: CASSANDRA-10452
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10452
> Project: Cassandra
> Issue Type: Sub-task
>Reporter: Jim Witschey
>Assignee: Yuki Morishita
> Fix For: 3.0.0 rc2
>
>
> {{bootstrap_test.py:TestBootstrap.resumable_bootstrap_test}} has been
> flapping on CassCI lately:
> http://cassci.datastax.com/view/trunk/job/cassandra-3.0_dtest/lastCompletedBuild/testReport/bootstrap_test/TestBootstrap/resumable_bootstrap_test/history/
> I have not been able to reproduce on OpenStack. I'm assigning [~yukim] for
> now, but feel free to reassign.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10452) Fix resummable_bootstrap_test and bootstrap_with_reset_bootstrap_state_test
[
https://issues.apache.org/jira/browse/CASSANDRA-10452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Hust updated CASSANDRA-10452:
Summary: Fix resummable_bootstrap_test and
bootstrap_with_reset_bootstrap_state_test (was: Fix flapping
resummable_bootstrap_test)
> Fix resummable_bootstrap_test and bootstrap_with_reset_bootstrap_state_test
> ---
>
> Key: CASSANDRA-10452
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10452
> Project: Cassandra
> Issue Type: Sub-task
>Reporter: Jim Witschey
>Assignee: Yuki Morishita
> Fix For: 3.0.0 rc2
>
>
> {{bootstrap_test.py:TestBootstrap.resumable_bootstrap_test}} has been
> flapping on CassCI lately:
> http://cassci.datastax.com/view/trunk/job/cassandra-3.0_dtest/lastCompletedBuild/testReport/bootstrap_test/TestBootstrap/resumable_bootstrap_test/history/
> I have not been able to reproduce on OpenStack. I'm assigning [~yukim] for
> now, but feel free to reassign.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
[
https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14953490#comment-14953490
]
Andrew Hust commented on CASSANDRA-10360:
-
Confirmed that the above upgrade test script now passes without exceptions.
Ran with:
apache/cassandra-2.1: {{4acc3a69d319b0e7e00cbd37b27e988ebfa4df4f}}
apache/cassandra-2.2: {{0c051a46c54fd1a2f151e1a68f4556faca02be8d}}
pcmanus/10360: {{6a56057a4eaab01480fba6292c5a59d4c3d62c49}}
> UnsupportedOperationException when compacting system.size_estimates after 2.1
> -> 2.2 -> 3.0 upgrade
> ---
>
> Key: CASSANDRA-10360
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10360
> Project: Cassandra
> Issue Type: Bug
>Reporter: Andrew Hust
>Assignee: Sylvain Lebresne
>Priority: Blocker
> Fix For: 3.0.0 rc2
>
> Attachments: size_estimates.2.tar.bz2, size_estimates.tar.bz2,
> system.log.2.bz2, system.log.bz2
>
>
> When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will
> get stuck in a compaction loop throwing:
> {code}
> java.lang.UnsupportedOperationException
> at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382)
> at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147)
> at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80)
> at
> org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100)
> at
> org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72)
> at
> org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not
> surface the issue. It can be reproduced with the following (note -- changing
> jdks will need to be altered if you aren't on OSX)
> {code}
> #!/bin/sh
> echo "using java7 for cassandra-2.1 instance"
> export JAVA_HOME=$(/usr/libexec/java_home -v 1.7)
> ccm stop
> ccm remove upgrade_nodes
> ccm create -n 1 -v git:cassandra-2.1
[jira] [Commented] (CASSANDRA-10434) Problem upgrading to 3.0 with UDA
[
https://issues.apache.org/jira/browse/CASSANDRA-10434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14947635#comment-14947635
]
Andrew Hust commented on CASSANDRA-10434:
-
Confirmed patch fixes issue with exception post upgrade and UDA performs as
expected post upgrade. Once this has been merged I'll update the upgrading
dtest to include UDA's in setup and verification.
Ran with:
apache/cassandra-2.2 {{be89dae3ecfd98b2170732c45d7f95807d5c19af}}
snazy/10434-uda-migration-3.0 {{b83a088f252e906faa9924def8e24997e072c109}}
> Problem upgrading to 3.0 with UDA
> -
>
> Key: CASSANDRA-10434
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10434
> Project: Cassandra
> Issue Type: Bug
>Reporter: Sylvain Lebresne
>Assignee: Robert Stupp
> Fix For: 3.0.0 rc2
>
>
> Copy-pasting from [~nutbunnies] comment on CASSANDRA-9756:
> {quote}
> upgrading from 2.2 to 3.0 with a UDA defined will throw the exception below
> and fail to start when upgraded to 3.0.
> Used:
> 2.2: {{ae9b7e05222b2a25eda5618cf9eb17103e4d6d8b}}
> 3.0: {{5c2912d1ce95aacdacb59ccc840b12cd9aa0c8f8}}
> {noformat}
> org.apache.cassandra.exceptions.UnrecognizedEntityException: Undefined name
> function_name in where clause ('function_name = ?')
> at
> org.apache.cassandra.cql3.Relation.toColumnDefinition(Relation.java:259)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.SingleColumnRelation.newEQRestriction(SingleColumnRelation.java:160)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.Relation.toRestriction(Relation.java:137)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.restrictions.StatementRestrictions.(StatementRestrictions.java:151)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepareRestrictions(SelectStatement.java:817)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:764)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:752)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:504)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.parseStatement(QueryProcessor.java:241)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.executeOnceInternal(QueryProcessor.java:336)
> ~[main/:na]
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.query(LegacySchemaMigrator.java:882)
> ~[main/:na]
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.readAggregateMetadata(LegacySchemaMigrator.java:849)
> ~[main/:na]
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.readAggregate(LegacySchemaMigrator.java:830)
> ~[main/:na]
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.lambda$readAggregates$216(LegacySchemaMigrator.java:823)
> ~[main/:na]
> at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_60]
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.readAggregates(LegacySchemaMigrator.java:823)
> ~[main/:na]
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.readKeyspace(LegacySchemaMigrator.java:166)
> ~[main/:na]
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.lambda$readSchema$207(LegacySchemaMigrator.java:154)
> ~[main/:na]
> at java.util.ArrayList.forEach(ArrayList.java:1249) ~[na:1.8.0_60]
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.readSchema(LegacySchemaMigrator.java:154)
> ~[main/:na]
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.migrate(LegacySchemaMigrator.java:77)
> ~[main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:223)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:542)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:668)
> [main/:na]
> {noformat}
> Can be reproduced with:
> {noformat}
> ccm stop
> ccm remove uda_upgrade
> ccm create -n 1 -v git:cassandra-2.2 uda_upgrade
> ccm updateconf 'enable_user_defined_functions: true'
> ccm start
> cat << EOF | ccm node1 cqlsh
> create keyspace ks WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': 1};
> USE ks;
> CREATE FUNCTION func_1(current int, candidate int)
> CALLED ON NULL INPUT
> RETURNS int LANGUAGE java AS
> 'if (current == null) return candidate; else return
> Math.max(current, candidate);';
> CREATE AGGREGATE agg_1(int)
> SFUNC func_1
> STYPE int
> INITCOND null;
> EOF
> sleep 10
> echo "Draining all nodes"
> ccm node1 nodetool drain
> ccm stop
>
[jira] [Updated] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
[
https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Hust updated CASSANDRA-10360:
Attachment: size_estimates.2.tar.bz2
system.log.2.bz2
> UnsupportedOperationException when compacting system.size_estimates after 2.1
> -> 2.2 -> 3.0 upgrade
> ---
>
> Key: CASSANDRA-10360
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10360
> Project: Cassandra
> Issue Type: Bug
>Reporter: Andrew Hust
>Assignee: Sylvain Lebresne
>Priority: Blocker
> Fix For: 3.0.0 rc2
>
> Attachments: size_estimates.2.tar.bz2, size_estimates.tar.bz2,
> system.log.2.bz2, system.log.bz2
>
>
> When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will
> get stuck in a compaction loop throwing:
> {code}
> java.lang.UnsupportedOperationException
> at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382)
> at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147)
> at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80)
> at
> org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100)
> at
> org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72)
> at
> org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not
> surface the issue. It can be reproduced with the following (note -- changing
> jdks will need to be altered if you aren't on OSX)
> {code}
> #!/bin/sh
> echo "using java7 for cassandra-2.1 instance"
> export JAVA_HOME=$(/usr/libexec/java_home -v 1.7)
> ccm stop
> ccm remove upgrade_nodes
> ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes
> ccm start
> ccm node1 stress write n=500K -rate threads=4 -mode native cql3
> sleep 10
> for cver in 3.0
> do
> echo "Draining all nodes"
> ccm node1 nodetool drain
> ccm stop
> echo "switching to java 8"
> export
[jira] [Commented] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
[
https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14947144#comment-14947144
]
Andrew Hust commented on CASSANDRA-10360:
-
I ran your patch thru a 2.1->2.2->3.0 upgrade and it failed with a different
exception. I've attached the system.log and size_estimates sstables from this
run.
Ran with:
apache/cassandra-2.1 {{1b08cbd817dea379ea84175381d3ef151fe65681}}
apache/cassandra-2.2: {{be89dae3ecfd98b2170732c45d7f95807d5c19af}}
pcmanus/10360: {{8f7d524efcbf0157697e67a3b4ff4b883b441a55}}
{code}
java.lang.AssertionError
at
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.checkReady(UnfilteredDeserializer.java:346)
at
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.nextIsRow(UnfilteredDeserializer.java:367)
at
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:384)
at
org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147)
at
org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:369)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:189)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:158)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503)
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80)
at
org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72)
at
org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80)
at
org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72)
at
org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:110)
at
org.apache.cassandra.db.ColumnIndex.writeAndBuildIndex(ColumnIndex.java:52)
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:148)
at
org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
at
org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:57)
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:112)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:182)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
at
org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
> UnsupportedOperationException when compacting system.size_estimates after 2.1
> -> 2.2 -> 3.0 upgrade
>
[jira] [Assigned] (CASSANDRA-10391) sstableloader fails with client SSL enabled with non-standard keystore/truststore location
[
https://issues.apache.org/jira/browse/CASSANDRA-10391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Hust reassigned CASSANDRA-10391:
---
Assignee: Andrew Hust
> sstableloader fails with client SSL enabled with non-standard
> keystore/truststore location
> --
>
> Key: CASSANDRA-10391
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10391
> Project: Cassandra
> Issue Type: Bug
> Environment: [cqlsh 4.1.1 | Cassandra 2.0.14.425 | DSE 4.6.6 | CQL
> spec 3.1.1 | Thrift protocol 19.39.0]
> [cqlsh 5.0.1 | Cassandra 2.1.8.689 | DSE 4.7.3 | CQL spec 3.2.0 | Native
> protocol v3]
>Reporter: Jon Moses
>Assignee: Andrew Hust
>
> If client SSL is enabled, sstableloader is unable to access the keystore and
> truststore if they are not in the expected locations. I reproduce this issue
> providing {{-f /path/to/cassandra.yaml}} as well as manually using the
> {{-ks}} flag with the proper path to the keystore.
> For example:
> {noformat}
> client_encryption_options:
> enabled: true
> keystore: /var/tmp/.keystore
> {noformat}
> {noformat}
> # sstableloader -d 172.31.2.240,172.31.2.241 -f
> /etc/dse/cassandra/cassandra.yaml Keyspace1/Standard1/
> Could not retrieve endpoint ranges:
> java.io.FileNotFoundException: /usr/share/dse/conf/.keystore
> Run with --debug to get full stack trace or --help to get help.
> #
> # sstableloader -d 172.31.2.240,172.31.2.241 -ks /var/tmp/.keystore
> Keyspace1/Standard1/
> Could not retrieve endpoint ranges:
> java.io.FileNotFoundException: /usr/share/dse/conf/.keystore
> Run with --debug to get full stack trace or --help to get help.
> #
> {noformat}
> The full stack is:
> {noformat}
> # sstableloader -d 172.31.2.240,172.31.2.241 -f
> /etc/dse/cassandra/cassandra.yaml --debug Keyspace1/Standard1/
> Could not retrieve endpoint ranges:
> java.io.FileNotFoundException: /usr/share/dse/conf/.keystore
> java.lang.RuntimeException: Could not retrieve endpoint ranges:
> at
> org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:283)
> at
> org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:144)
> at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:95)
> Caused by: java.io.FileNotFoundException: /usr/share/dse/conf/.keystore
> at
> com.datastax.bdp.transport.client.TClientSocketFactory.getSSLSocket(TClientSocketFactory.java:128)
> at
> com.datastax.bdp.transport.client.TClientSocketFactory.openSocket(TClientSocketFactory.java:114)
> at
> com.datastax.bdp.transport.client.TDseClientTransportFactory.openTransport(TDseClientTransportFactory.java:186)
> at
> com.datastax.bdp.transport.client.TDseClientTransportFactory.openTransport(TDseClientTransportFactory.java:120)
> at
> com.datastax.bdp.transport.client.TDseClientTransportFactory.openTransport(TDseClientTransportFactory.java:111)
> at
> org.apache.cassandra.tools.BulkLoader$ExternalClient.createThriftClient(BulkLoader.java:302)
> at
> org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:254)
> ... 2 more
> root@ip-172-31-2-240:/tmp/foo#
> {noformat}.
> If I copy the keystore to the expected location, I get the same error with
> the truststore.
> {noformat}
> # sstableloader -d 172.31.2.240,172.31.2.241 -f
> /etc/dse/cassandra/cassandra.yaml --debug Keyspace1/Standard1/
> Could not retrieve endpoint ranges:
> java.io.FileNotFoundException: /usr/share/dse/conf/.truststore
> java.lang.RuntimeException: Could not retrieve endpoint ranges:
> at
> org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:283)
> at
> org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:144)
> at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:95)
> Caused by: java.io.FileNotFoundException: /usr/share/dse/conf/.truststore
> at
> com.datastax.bdp.transport.client.TClientSocketFactory.getSSLSocket(TClientSocketFactory.java:130)
> at
> com.datastax.bdp.transport.client.TClientSocketFactory.openSocket(TClientSocketFactory.java:114)
> at
> com.datastax.bdp.transport.client.TDseClientTransportFactory.openTransport(TDseClientTransportFactory.java:186)
> at
> com.datastax.bdp.transport.client.TDseClientTransportFactory.openTransport(TDseClientTransportFactory.java:120)
> at
> com.datastax.bdp.transport.client.TDseClientTransportFactory.openTransport(TDseClientTransportFactory.java:111)
> at
> org.apache.cassandra.tools.BulkLoader$ExternalClient.createThriftClient(BulkLoader.java:302)
> at
> org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:254)
> ... 2 more
> #
> {noformat}
> If I copy the truststore, it finds them
[jira] [Updated] (CASSANDRA-10391) sstableloader fails with client SSL enabled with non-standard keystore/truststore location
[
https://issues.apache.org/jira/browse/CASSANDRA-10391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Hust updated CASSANDRA-10391:
Reproduced In: 2.1.8, 2.0.14
> sstableloader fails with client SSL enabled with non-standard
> keystore/truststore location
> --
>
> Key: CASSANDRA-10391
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10391
> Project: Cassandra
> Issue Type: Bug
> Environment: [cqlsh 4.1.1 | Cassandra 2.0.14.425 | DSE 4.6.6 | CQL
> spec 3.1.1 | Thrift protocol 19.39.0]
> [cqlsh 5.0.1 | Cassandra 2.1.8.689 | DSE 4.7.3 | CQL spec 3.2.0 | Native
> protocol v3]
>Reporter: Jon Moses
>Assignee: Andrew Hust
>
> If client SSL is enabled, sstableloader is unable to access the keystore and
> truststore if they are not in the expected locations. I reproduce this issue
> providing {{-f /path/to/cassandra.yaml}} as well as manually using the
> {{-ks}} flag with the proper path to the keystore.
> For example:
> {noformat}
> client_encryption_options:
> enabled: true
> keystore: /var/tmp/.keystore
> {noformat}
> {noformat}
> # sstableloader -d 172.31.2.240,172.31.2.241 -f
> /etc/dse/cassandra/cassandra.yaml Keyspace1/Standard1/
> Could not retrieve endpoint ranges:
> java.io.FileNotFoundException: /usr/share/dse/conf/.keystore
> Run with --debug to get full stack trace or --help to get help.
> #
> # sstableloader -d 172.31.2.240,172.31.2.241 -ks /var/tmp/.keystore
> Keyspace1/Standard1/
> Could not retrieve endpoint ranges:
> java.io.FileNotFoundException: /usr/share/dse/conf/.keystore
> Run with --debug to get full stack trace or --help to get help.
> #
> {noformat}
> The full stack is:
> {noformat}
> # sstableloader -d 172.31.2.240,172.31.2.241 -f
> /etc/dse/cassandra/cassandra.yaml --debug Keyspace1/Standard1/
> Could not retrieve endpoint ranges:
> java.io.FileNotFoundException: /usr/share/dse/conf/.keystore
> java.lang.RuntimeException: Could not retrieve endpoint ranges:
> at
> org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:283)
> at
> org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:144)
> at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:95)
> Caused by: java.io.FileNotFoundException: /usr/share/dse/conf/.keystore
> at
> com.datastax.bdp.transport.client.TClientSocketFactory.getSSLSocket(TClientSocketFactory.java:128)
> at
> com.datastax.bdp.transport.client.TClientSocketFactory.openSocket(TClientSocketFactory.java:114)
> at
> com.datastax.bdp.transport.client.TDseClientTransportFactory.openTransport(TDseClientTransportFactory.java:186)
> at
> com.datastax.bdp.transport.client.TDseClientTransportFactory.openTransport(TDseClientTransportFactory.java:120)
> at
> com.datastax.bdp.transport.client.TDseClientTransportFactory.openTransport(TDseClientTransportFactory.java:111)
> at
> org.apache.cassandra.tools.BulkLoader$ExternalClient.createThriftClient(BulkLoader.java:302)
> at
> org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:254)
> ... 2 more
> root@ip-172-31-2-240:/tmp/foo#
> {noformat}.
> If I copy the keystore to the expected location, I get the same error with
> the truststore.
> {noformat}
> # sstableloader -d 172.31.2.240,172.31.2.241 -f
> /etc/dse/cassandra/cassandra.yaml --debug Keyspace1/Standard1/
> Could not retrieve endpoint ranges:
> java.io.FileNotFoundException: /usr/share/dse/conf/.truststore
> java.lang.RuntimeException: Could not retrieve endpoint ranges:
> at
> org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:283)
> at
> org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:144)
> at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:95)
> Caused by: java.io.FileNotFoundException: /usr/share/dse/conf/.truststore
> at
> com.datastax.bdp.transport.client.TClientSocketFactory.getSSLSocket(TClientSocketFactory.java:130)
> at
> com.datastax.bdp.transport.client.TClientSocketFactory.openSocket(TClientSocketFactory.java:114)
> at
> com.datastax.bdp.transport.client.TDseClientTransportFactory.openTransport(TDseClientTransportFactory.java:186)
> at
> com.datastax.bdp.transport.client.TDseClientTransportFactory.openTransport(TDseClientTransportFactory.java:120)
> at
> com.datastax.bdp.transport.client.TDseClientTransportFactory.openTransport(TDseClientTransportFactory.java:111)
> at
> org.apache.cassandra.tools.BulkLoader$ExternalClient.createThriftClient(BulkLoader.java:302)
> at
> org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:254)
> ... 2 more
> #
> {noformat}
> If I copy the truststore, it finds them
[jira] [Resolved] (CASSANDRA-9456) while starting cassandra using cassandra -f ; encountered an errror ERROR 11:46:42 Exception in thread Thread[MemtableFlushWriter:1,5,main]
[
https://issues.apache.org/jira/browse/CASSANDRA-9456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Hust resolved CASSANDRA-9456.
Resolution: Cannot Reproduce
> while starting cassandra using cassandra -f ; encountered an errror ERROR
> 11:46:42 Exception in thread Thread[MemtableFlushWriter:1,5,main]
> ---
>
> Key: CASSANDRA-9456
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9456
> Project: Cassandra
> Issue Type: Bug
> Components: Config
> Environment: ubuntu14.10
>Reporter: naresh
> Fix For: 3.x
>
>
> I am using openjdk7 and build cassandra successfully using source code from
> github(https://github.com/apache/cassandra.git)
>
> afer successfuly I set the path for cassandra
> and tried to start using cassandra -f
> below is the errror encountered while startup
> {code}
> INFO 11:46:42 Token metadata:
> INFO 11:46:42 Enqueuing flush of local: 653 (0%) on-heap, 0 (0%) off-heap
> INFO 11:46:42 Writing Memtable-local@1257824677(110 serialized bytes, 3 ops,
> 0%/0% of on/off-heap limit)
> ERROR 11:46:42 Exception in thread Thread[MemtableFlushWriter:1,5,main]
> java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
> at
> org.apache.cassandra.utils.memory.MemoryUtil.allocate(MemoryUtil.java:82)
> ~[main/:na]
> at org.apache.cassandra.io.util.Memory.(Memory.java:74)
> ~[main/:na]
> at org.apache.cassandra.io.util.SafeMemory.(SafeMemory.java:32)
> ~[main/:na]
> at
> org.apache.cassandra.io.compress.CompressionMetadata$Writer.(CompressionMetadata.java:274)
> ~[main/:na]
> at
> org.apache.cassandra.io.compress.CompressionMetadata$Writer.open(CompressionMetadata.java:288)
> ~[main/:na]
> at
> org.apache.cassandra.io.compress.CompressedSequentialWriter.(CompressedSequentialWriter.java:75)
> ~[main/:na]
> at
> org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:168)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.(BigTableWriter.java:74)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.big.BigFormat$WriterFactory.open(BigFormat.java:107)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTableWriter.java:84)
> ~[main/:na]
> at
> org.apache.cassandra.db.Memtable$FlushRunnable.createFlushWriter(Memtable.java:396)
> ~[main/:na]
> at
> org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:343)
> ~[main/:na]
> at
> org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:328)
> ~[main/:na]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[main/:na]
> at
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
> ~[guava-16.0.jar:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1085)
> ~[main/:na]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_79]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[na:1.7.0_79]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_79]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9456) while starting cassandra using cassandra -f ; encountered an errror ERROR 11:46:42 Exception in thread Thread[MemtableFlushWriter:1,5,main]
[
https://issues.apache.org/jira/browse/CASSANDRA-9456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14941829#comment-14941829
]
Andrew Hust commented on CASSANDRA-9456:
I'm unable to reproduce this. I was able to successful build and run using
openjdk on ubuntu. If you still see this error please reopen with openjdk
version and git sha your building.
> while starting cassandra using cassandra -f ; encountered an errror ERROR
> 11:46:42 Exception in thread Thread[MemtableFlushWriter:1,5,main]
> ---
>
> Key: CASSANDRA-9456
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9456
> Project: Cassandra
> Issue Type: Bug
> Components: Config
> Environment: ubuntu14.10
>Reporter: naresh
> Fix For: 3.x
>
>
> I am using openjdk7 and build cassandra successfully using source code from
> github(https://github.com/apache/cassandra.git)
>
> afer successfuly I set the path for cassandra
> and tried to start using cassandra -f
> below is the errror encountered while startup
> {code}
> INFO 11:46:42 Token metadata:
> INFO 11:46:42 Enqueuing flush of local: 653 (0%) on-heap, 0 (0%) off-heap
> INFO 11:46:42 Writing Memtable-local@1257824677(110 serialized bytes, 3 ops,
> 0%/0% of on/off-heap limit)
> ERROR 11:46:42 Exception in thread Thread[MemtableFlushWriter:1,5,main]
> java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
> at
> org.apache.cassandra.utils.memory.MemoryUtil.allocate(MemoryUtil.java:82)
> ~[main/:na]
> at org.apache.cassandra.io.util.Memory.(Memory.java:74)
> ~[main/:na]
> at org.apache.cassandra.io.util.SafeMemory.(SafeMemory.java:32)
> ~[main/:na]
> at
> org.apache.cassandra.io.compress.CompressionMetadata$Writer.(CompressionMetadata.java:274)
> ~[main/:na]
> at
> org.apache.cassandra.io.compress.CompressionMetadata$Writer.open(CompressionMetadata.java:288)
> ~[main/:na]
> at
> org.apache.cassandra.io.compress.CompressedSequentialWriter.(CompressedSequentialWriter.java:75)
> ~[main/:na]
> at
> org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:168)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.(BigTableWriter.java:74)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.big.BigFormat$WriterFactory.open(BigFormat.java:107)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTableWriter.java:84)
> ~[main/:na]
> at
> org.apache.cassandra.db.Memtable$FlushRunnable.createFlushWriter(Memtable.java:396)
> ~[main/:na]
> at
> org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:343)
> ~[main/:na]
> at
> org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:328)
> ~[main/:na]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[main/:na]
> at
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
> ~[guava-16.0.jar:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1085)
> ~[main/:na]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_79]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[na:1.7.0_79]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_79]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10233) IndexOutOfBoundsException in HintedHandOffManager
[
https://issues.apache.org/jira/browse/CASSANDRA-10233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14939903#comment-14939903
]
Andrew Hust commented on CASSANDRA-10233:
-
I'll give the patch a look. I'm also curious on how the broken hint got in
there in the first place. I would like to try and reproduce can you give more
details of your upgrade procedure?
- how many nodes?
- assuming rolling upgrade
- jdk change?
- roughly how long was each node unavailable
- gc_grace value of table with broken hint
- values of max_hint_window_in_ms, max_hints_delivery_threads,
hinted_handoff_enabled, hinted_handoff_throttle_in_kb in cassandra.yaml
- what type of mutation was the hint without a target_id?
> IndexOutOfBoundsException in HintedHandOffManager
> -
>
> Key: CASSANDRA-10233
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10233
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 2.2.0
>Reporter: Omri Iluz
>Assignee: Andrew Hust
> Attachments: cassandra-2.1.8-10233-v2.txt, cassandra-2.1.8-10233.txt
>
>
> After upgrading our cluster to 2.2.0, the following error started showing
> exectly every 10 minutes on every server in the cluster:
> {noformat}
> INFO [CompactionExecutor:1381] 2015-08-31 18:31:55,506
> CompactionTask.java:142 - Compacting (8e7e1520-500e-11e5-b1e3-e95897ba4d20)
> [/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/la-540-big-Data.db:level=0,
> ]
> INFO [CompactionExecutor:1381] 2015-08-31 18:31:55,599
> CompactionTask.java:224 - Compacted (8e7e1520-500e-11e5-b1e3-e95897ba4d20) 1
> sstables to
> [/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/la-541-big,]
> to level=0. 1,544,495 bytes to 1,544,495 (~100% of original) in 93ms =
> 15.838121MB/s. 0 total partitions merged to 4. Partition merge counts were
> {1:4, }
> ERROR [HintedHandoff:1] 2015-08-31 18:31:55,600 CassandraDaemon.java:182 -
> Exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.IndexOutOfBoundsException: null
> at java.nio.Buffer.checkIndex(Buffer.java:538) ~[na:1.7.0_79]
> at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:410)
> ~[na:1.7.0_79]
> at org.apache.cassandra.utils.UUIDGen.getUUID(UUIDGen.java:106)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.db.HintedHandOffManager.scheduleAllDeliveries(HintedHandOffManager.java:515)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:88)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.db.HintedHandOffManager$1.run(HintedHandOffManager.java:168)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> [na:1.7.0_79]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> [na:1.7.0_79]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> [na:1.7.0_79]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [na:1.7.0_79]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_79]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_79]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10420) Cassandra server should throw meaningfull exception when thrift_framed_transport_size_in_mb reached
[ https://issues.apache.org/jira/browse/CASSANDRA-10420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14939914#comment-14939914 ] Andrew Hust commented on CASSANDRA-10420: - Can you supply some additional information? What version of cassandra and which language and driver version are you using. Thanks. > Cassandra server should throw meaningfull exception when > thrift_framed_transport_size_in_mb reached > --- > > Key: CASSANDRA-10420 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10420 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Yuan Yao > > In cassandra's configuration, we set "thrift_framed_transport_size_in_mb" as > 15 > When send data large than some threshold max value, java.net.SocketException: > Connection reset will be thrown out from server. This exception doesn't > deliver meaningful message. Client side can't detect what's wrong with the > request > Please throw out meaningful exception in this case to client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10233) IndexOutOfBoundsException in HintedHandOffManager
[
https://issues.apache.org/jira/browse/CASSANDRA-10233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14940540#comment-14940540
]
Andrew Hust commented on CASSANDRA-10233:
-
[~eitikimura] as [~pauloricardomg] mentioned truncating the hints table will
get things rolling again, just make sure to repair in near future to prevent
possible dataloss or zombie records related to the mutations truncated out of
the hints table.
[~fhsgoncalves] thanks, that helps a ton. Couple more questions to help
reproduce it:
- Just want to make sure order of operation is correct, the error showed up
when you added additional nodes then persisted after upgrades?
- Are all the new nodes in the second rack?
- Are you running with vnodes?
- Are your keyspaces set with a replication of NetworkTopologyStrategy,
SimpleStrategy or ?
- In the cassandra.yaml what is the value for: endpoint_snitch
> IndexOutOfBoundsException in HintedHandOffManager
> -
>
> Key: CASSANDRA-10233
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10233
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 2.2.0
>Reporter: Omri Iluz
>Assignee: Andrew Hust
> Attachments: cassandra-2.1.8-10233-v2.txt, cassandra-2.1.8-10233.txt
>
>
> After upgrading our cluster to 2.2.0, the following error started showing
> exectly every 10 minutes on every server in the cluster:
> {noformat}
> INFO [CompactionExecutor:1381] 2015-08-31 18:31:55,506
> CompactionTask.java:142 - Compacting (8e7e1520-500e-11e5-b1e3-e95897ba4d20)
> [/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/la-540-big-Data.db:level=0,
> ]
> INFO [CompactionExecutor:1381] 2015-08-31 18:31:55,599
> CompactionTask.java:224 - Compacted (8e7e1520-500e-11e5-b1e3-e95897ba4d20) 1
> sstables to
> [/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/la-541-big,]
> to level=0. 1,544,495 bytes to 1,544,495 (~100% of original) in 93ms =
> 15.838121MB/s. 0 total partitions merged to 4. Partition merge counts were
> {1:4, }
> ERROR [HintedHandoff:1] 2015-08-31 18:31:55,600 CassandraDaemon.java:182 -
> Exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.IndexOutOfBoundsException: null
> at java.nio.Buffer.checkIndex(Buffer.java:538) ~[na:1.7.0_79]
> at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:410)
> ~[na:1.7.0_79]
> at org.apache.cassandra.utils.UUIDGen.getUUID(UUIDGen.java:106)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.db.HintedHandOffManager.scheduleAllDeliveries(HintedHandOffManager.java:515)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:88)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.db.HintedHandOffManager$1.run(HintedHandOffManager.java:168)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> [na:1.7.0_79]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> [na:1.7.0_79]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> [na:1.7.0_79]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [na:1.7.0_79]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_79]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_79]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-9756) Cleanup UDA code after 6717
[
https://issues.apache.org/jira/browse/CASSANDRA-9756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14940503#comment-14940503
]
Andrew Hust edited comment on CASSANDRA-9756 at 10/1/15 10:09 PM:
--
Not sure if this should be a separate ticket but upgrading from 2.2 to 3.0 with
a UDA defined will throw the exception below and fail to start when upgraded to
3.0.
Used:
2.2: {{ae9b7e05222b2a25eda5618cf9eb17103e4d6d8b}}
3.0: {{5c2912d1ce95aacdacb59ccc840b12cd9aa0c8f8}}
{code}
org.apache.cassandra.exceptions.UnrecognizedEntityException: Undefined name
function_name in where clause ('function_name = ?')
at
org.apache.cassandra.cql3.Relation.toColumnDefinition(Relation.java:259)
~[main/:na]
at
org.apache.cassandra.cql3.SingleColumnRelation.newEQRestriction(SingleColumnRelation.java:160)
~[main/:na]
at org.apache.cassandra.cql3.Relation.toRestriction(Relation.java:137)
~[main/:na]
at
org.apache.cassandra.cql3.restrictions.StatementRestrictions.(StatementRestrictions.java:151)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepareRestrictions(SelectStatement.java:817)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:764)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:752)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:504)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.parseStatement(QueryProcessor.java:241)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.executeOnceInternal(QueryProcessor.java:336)
~[main/:na]
at
org.apache.cassandra.schema.LegacySchemaMigrator.query(LegacySchemaMigrator.java:882)
~[main/:na]
at
org.apache.cassandra.schema.LegacySchemaMigrator.readAggregateMetadata(LegacySchemaMigrator.java:849)
~[main/:na]
at
org.apache.cassandra.schema.LegacySchemaMigrator.readAggregate(LegacySchemaMigrator.java:830)
~[main/:na]
at
org.apache.cassandra.schema.LegacySchemaMigrator.lambda$readAggregates$216(LegacySchemaMigrator.java:823)
~[main/:na]
at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_60]
at
org.apache.cassandra.schema.LegacySchemaMigrator.readAggregates(LegacySchemaMigrator.java:823)
~[main/:na]
at
org.apache.cassandra.schema.LegacySchemaMigrator.readKeyspace(LegacySchemaMigrator.java:166)
~[main/:na]
at
org.apache.cassandra.schema.LegacySchemaMigrator.lambda$readSchema$207(LegacySchemaMigrator.java:154)
~[main/:na]
at java.util.ArrayList.forEach(ArrayList.java:1249) ~[na:1.8.0_60]
at
org.apache.cassandra.schema.LegacySchemaMigrator.readSchema(LegacySchemaMigrator.java:154)
~[main/:na]
at
org.apache.cassandra.schema.LegacySchemaMigrator.migrate(LegacySchemaMigrator.java:77)
~[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:223)
[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:542)
[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:668)
[main/:na]
{code}
Can be reproduced with:
{code}
ccm stop
ccm remove uda_upgrade
ccm create -n 1 -v git:cassandra-2.2 uda_upgrade
ccm updateconf 'enable_user_defined_functions: true'
ccm start
cat << EOF | ccm node1 cqlsh
create keyspace ks WITH replication = {'class': 'SimpleStrategy',
'replication_factor': 1};
USE ks;
CREATE FUNCTION func_1(current int, candidate int)
CALLED ON NULL INPUT
RETURNS int LANGUAGE java AS
'if (current == null) return candidate; else return
Math.max(current, candidate);';
CREATE AGGREGATE agg_1(int)
SFUNC func_1
STYPE int
INITCOND null;
EOF
sleep 10
echo "Draining all nodes"
ccm node1 nodetool drain
ccm stop
echo "Upgrading to git:cassandra-3.0"
ccm setdir -v git:cassandra-3.0
ccm start
echo "Sleeping for version migrations"
sleep 15
ccm checklogerror
ccm stop
{code}
//CC [~enigmacurry]
was (Author: nutbunnies):
Not sure if this should be a separate ticket but upgrading from 2.2 to 3.0 with
a UDA defined will throw the exception below and fail to start when upgraded to
3.0.
Used:
2.2: {{ae9b7e05222b2a25eda5618cf9eb17103e4d6d8b}}
3.0: {{5c2912d1ce95aacdacb59ccc840b12cd9aa0c8f8}}
{code}
org.apache.cassandra.exceptions.UnrecognizedEntityException: Undefined name
function_name in where clause ('function_name = ?')
at
org.apache.cassandra.cql3.Relation.toColumnDefinition(Relation.java:259)
~[main/:na]
at
org.apache.cassandra.cql3.SingleColumnRelation.newEQRestriction(SingleColumnRelation.java:160)
~[main/:na]
at
[jira] [Commented] (CASSANDRA-9756) Cleanup UDA code after 6717
[
https://issues.apache.org/jira/browse/CASSANDRA-9756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14940503#comment-14940503
]
Andrew Hust commented on CASSANDRA-9756:
Not sure if this should be a separate ticket but upgrading from 2.2 to 3.0 with
a UDA defined will throw the exception below and fail to start when upgraded to
3.0.
Used:
2.2: {{ae9b7e05222b2a25eda5618cf9eb17103e4d6d8b}}
3.0: {{5c2912d1ce95aacdacb59ccc840b12cd9aa0c8f8}}
{code}
org.apache.cassandra.exceptions.UnrecognizedEntityException: Undefined name
function_name in where clause ('function_name = ?')
at
org.apache.cassandra.cql3.Relation.toColumnDefinition(Relation.java:259)
~[main/:na]
at
org.apache.cassandra.cql3.SingleColumnRelation.newEQRestriction(SingleColumnRelation.java:160)
~[main/:na]
at org.apache.cassandra.cql3.Relation.toRestriction(Relation.java:137)
~[main/:na]
at
org.apache.cassandra.cql3.restrictions.StatementRestrictions.(StatementRestrictions.java:151)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepareRestrictions(SelectStatement.java:817)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:764)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:752)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:504)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.parseStatement(QueryProcessor.java:241)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.executeOnceInternal(QueryProcessor.java:336)
~[main/:na]
at
org.apache.cassandra.schema.LegacySchemaMigrator.query(LegacySchemaMigrator.java:882)
~[main/:na]
at
org.apache.cassandra.schema.LegacySchemaMigrator.readAggregateMetadata(LegacySchemaMigrator.java:849)
~[main/:na]
at
org.apache.cassandra.schema.LegacySchemaMigrator.readAggregate(LegacySchemaMigrator.java:830)
~[main/:na]
at
org.apache.cassandra.schema.LegacySchemaMigrator.lambda$readAggregates$216(LegacySchemaMigrator.java:823)
~[main/:na]
at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_60]
at
org.apache.cassandra.schema.LegacySchemaMigrator.readAggregates(LegacySchemaMigrator.java:823)
~[main/:na]
at
org.apache.cassandra.schema.LegacySchemaMigrator.readKeyspace(LegacySchemaMigrator.java:166)
~[main/:na]
at
org.apache.cassandra.schema.LegacySchemaMigrator.lambda$readSchema$207(LegacySchemaMigrator.java:154)
~[main/:na]
at java.util.ArrayList.forEach(ArrayList.java:1249) ~[na:1.8.0_60]
at
org.apache.cassandra.schema.LegacySchemaMigrator.readSchema(LegacySchemaMigrator.java:154)
~[main/:na]
at
org.apache.cassandra.schema.LegacySchemaMigrator.migrate(LegacySchemaMigrator.java:77)
~[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:223)
[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:542)
[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:668)
[main/:na]
{code}
Can be reproduced with:
{code}
ccm stop
ccm remove uda_upgrade
ccm create -n 1 -v git:cassandra-2.2 uda_upgrade
ccm updateconf 'enable_user_defined_functions: true'
ccm start
cat << EOF | ccm node1 cqlsh
create keyspace ks WITH replication = {'class': 'SimpleStrategy',
'replication_factor': 1};
USE ks;
CREATE FUNCTION func_1(current int, candidate int)
CALLED ON NULL INPUT
RETURNS int LANGUAGE java AS
'if (current == null) return candidate; else return
Math.max(current, candidate);';
CREATE AGGREGATE agg_1(int)
SFUNC func_1
STYPE int
INITCOND null;
EOF
sleep
echo "Draining all nodes"
ccm node1 nodetool drain
ccm stop
echo "Upgrading to git:cassandra-3.0"
ccm setdir -v git:cassandra-3.0
ccm start
echo "Sleeping for version migrations"
sleep 15
ccm checklogerror
ccm stop
{code}
//CC [~enigmacurry]
> Cleanup UDA code after 6717
> ---
>
> Key: CASSANDRA-9756
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9756
> Project: Cassandra
> Issue Type: Task
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Minor
> Fix For: 3.x
>
>
> After CASSANDRA-6717 has landed, there should be some cleanup of UDF/UDA code
> wrt load from schema tables and handling broken functions.
> /cc [~iamaleksey]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9756) Cleanup UDA code after 6717
[ https://issues.apache.org/jira/browse/CASSANDRA-9756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Hust updated CASSANDRA-9756: --- Fix Version/s: 3.0.0 rc2 > Cleanup UDA code after 6717 > --- > > Key: CASSANDRA-9756 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9756 > Project: Cassandra > Issue Type: Task >Reporter: Robert Stupp >Assignee: Robert Stupp >Priority: Minor > Fix For: 3.x, 3.0.0 rc2 > > > After CASSANDRA-6717 has landed, there should be some cleanup of UDF/UDA code > wrt load from schema tables and handling broken functions. > /cc [~iamaleksey] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
[
https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Hust updated CASSANDRA-10360:
Description:
When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will get
stuck in a compaction loop throwing:
{code}
java.lang.UnsupportedOperationException
at
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382)
at
org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147)
at
org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503)
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80)
at
org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72)
at
org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100)
at
org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72)
at
org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
at
org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not
surface the issue. It can be reproduced with the following (note -- changing
jdks will need to be altered if you aren't on OSX)
{code}
#!/bin/sh
echo "using java7 for cassandra-2.1 instance"
export JAVA_HOME=$(/usr/libexec/java_home -v 1.7)
ccm stop
ccm remove upgrade_nodes
ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes
ccm start
ccm node1 stress write n=500K -rate threads=4 -mode native cql3
sleep 10
for cver in 2.2 3.0
do
echo "Draining all nodes"
ccm node1 nodetool drain
ccm stop
echo "switching to java 8"
export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)
echo "Upgrading to git:cassandra-$cver"
ccm setdir -v git:cassandra-$cver
ccm start
echo "Sleeping to all version migrations"
sleep 30
echo "Upgrading sstables"
ccm node1 nodetool upgradesstables
ccm node1 nodetool upgradesstables system
ccm node1 nodetool compact system
ccm node1 stress write n=500K -rate threads=4 -mode native cql3
sleep 10
done
echo "***"
echo "instead of creating churn to cause compaction naturally forcing
compaction of system keyspace"
echo "***"
ccm node1 nodetool compact system
ccm stop
{code}
The example uses {{nodetool compact system}} but it will also occur with
{{nodetool upgradesstables system}}. I'm puzzled by that since the script runs
{{upgradesstables}} on each iteration. Is the system keyspace not
[jira] [Updated] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
[
https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Hust updated CASSANDRA-10360:
Description:
When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will get
stuck in a compaction loop throwing:
{code}
java.lang.UnsupportedOperationException
at
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382)
at
org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147)
at
org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503)
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80)
at
org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72)
at
org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100)
at
org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72)
at
org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
at
org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not
surface the issue. It can be reproduced with the following (note -- changing
jdks will need to be altered if you aren't on OSX)
{code}
#!/bin/sh
echo "using java7 for cassandra-2.1 instance"
export JAVA_HOME=$(/usr/libexec/java_home -v 1.7)
ccm stop
ccm remove upgrade_nodes
ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes
ccm start
ccm node1 stress write n=500K -rate threads=4 -mode native cql3
sleep 10
for cver in 3.0
do
echo "Draining all nodes"
ccm node1 nodetool drain
ccm stop
echo "switching to java 8"
export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)
echo "Upgrading to git:cassandra-$cver"
ccm setdir -v git:cassandra-$cver
ccm start
echo "Sleeping to all version migrations"
sleep 30
echo "Upgrading sstables"
ccm node1 nodetool upgradesstables
ccm node1 nodetool upgradesstables system
ccm node1 nodetool compact system
ccm node1 stress write n=500K -rate threads=4 -mode native cql3
sleep 10
done
echo "***"
echo "instead of creating churn to cause compaction naturally forcing
compaction of system keyspace"
echo "***"
ccm node1 nodetool compact system
ccm stop
{code}
The example uses {{nodetool compact system}} but it will also occur with
{{nodetool upgradesstables system}}. I'm puzzled by that since the script runs
{{upgradesstables}} on each iteration. Is the system keyspace not effected by
[jira] [Comment Edited] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
[
https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937461#comment-14937461
]
Andrew Hust edited comment on CASSANDRA-10360 at 9/30/15 8:21 PM:
--
I have now reproduced this issue when upgrading from 2.1 to 3.0 when doing a
rolling upgrade with cstar_perf. Attached site_estimates sstables and system
log from one of the nodes. I'll work on getting a isolated script to reproduce.
UPDATE: The above shell script has been updated to show failure from 2.1 ->
3.0. It just needed more activity on the stress operation to trigger the
issue. Change line {{for cver in 3.0}} to {{for cver in 2.2 3.0}} for the
original upgrade path.
was (Author: nutbunnies):
I have now reproduced this issue when upgrading from 2.1 to 3.0 when doing a
rolling upgrade with cstar_perf. Attached site_estimates sstables and system
log from one of the nodes. I'll work on getting a isolated script to reproduce.
> UnsupportedOperationException when compacting system.size_estimates after 2.1
> -> 2.2 -> 3.0 upgrade
> ---
>
> Key: CASSANDRA-10360
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10360
> Project: Cassandra
> Issue Type: Bug
>Reporter: Andrew Hust
>Assignee: Sylvain Lebresne
>Priority: Blocker
> Fix For: 3.0.0 rc2
>
> Attachments: size_estimates.tar.bz2, system.log.bz2
>
>
> When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will
> get stuck in a compaction loop throwing:
> {code}
> java.lang.UnsupportedOperationException
> at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382)
> at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147)
> at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80)
> at
> org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100)
> at
> org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72)
> at
> org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
>
[jira] [Updated] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
[
https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Hust updated CASSANDRA-10360:
Reproduced In: 3.0.0 rc1, 3.0 beta 2 (was: 3.0 beta 2)
Priority: Blocker (was: Major)
> UnsupportedOperationException when compacting system.size_estimates after 2.1
> -> 2.2 -> 3.0 upgrade
> ---
>
> Key: CASSANDRA-10360
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10360
> Project: Cassandra
> Issue Type: Bug
>Reporter: Andrew Hust
>Assignee: Sylvain Lebresne
>Priority: Blocker
> Fix For: 3.0.0 rc2
>
>
> When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will
> get stuck in a compaction loop throwing:
> {code}
> java.lang.UnsupportedOperationException
> at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382)
> at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147)
> at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80)
> at
> org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100)
> at
> org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72)
> at
> org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not
> surface the issue. It can be reproduced with the following (note -- changing
> jdks will need to be altered if you aren't on OSX)
> {code}
> #!/bin/sh
> echo "using java7 for cassandra-2.1 instance"
> export JAVA_HOME=$(/usr/libexec/java_home -v 1.7)
> ccm stop
> ccm remove upgrade_nodes
> ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes
> ccm start
> ccm node1 stress write n=5K -rate threads=4 -mode native cql3
> sleep 10
> for cver in 2.2 3.0
> do
> echo "Draining all nodes"
> ccm node1 nodetool drain
> ccm stop
> echo "switching to java 8"
> export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)
> echo "Upgrading to git:cassandra-$cver"
> ccm setdir
[jira] [Updated] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
[
https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Hust updated CASSANDRA-10360:
Attachment: system.log.bz2
size_estimates.tar.bz2
> UnsupportedOperationException when compacting system.size_estimates after 2.1
> -> 2.2 -> 3.0 upgrade
> ---
>
> Key: CASSANDRA-10360
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10360
> Project: Cassandra
> Issue Type: Bug
>Reporter: Andrew Hust
>Assignee: Sylvain Lebresne
>Priority: Blocker
> Fix For: 3.0.0 rc2
>
> Attachments: size_estimates.tar.bz2, system.log.bz2
>
>
> When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will
> get stuck in a compaction loop throwing:
> {code}
> java.lang.UnsupportedOperationException
> at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382)
> at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147)
> at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80)
> at
> org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100)
> at
> org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72)
> at
> org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not
> surface the issue. It can be reproduced with the following (note -- changing
> jdks will need to be altered if you aren't on OSX)
> {code}
> #!/bin/sh
> echo "using java7 for cassandra-2.1 instance"
> export JAVA_HOME=$(/usr/libexec/java_home -v 1.7)
> ccm stop
> ccm remove upgrade_nodes
> ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes
> ccm start
> ccm node1 stress write n=5K -rate threads=4 -mode native cql3
> sleep 10
> for cver in 2.2 3.0
> do
> echo "Draining all nodes"
> ccm node1 nodetool drain
> ccm stop
> echo "switching to java 8"
> export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)
> echo "Upgrading to
[jira] [Commented] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
[
https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937461#comment-14937461
]
Andrew Hust commented on CASSANDRA-10360:
-
I have now reproduced this issue when upgrading from 2.1 to 3.0 when doing a
rolling upgrade with cstar_perf. Attached site_estimates sstables and system
log from one of the nodes. I'll work on getting a isolated script to reproduce.
> UnsupportedOperationException when compacting system.size_estimates after 2.1
> -> 2.2 -> 3.0 upgrade
> ---
>
> Key: CASSANDRA-10360
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10360
> Project: Cassandra
> Issue Type: Bug
>Reporter: Andrew Hust
>Assignee: Sylvain Lebresne
>Priority: Blocker
> Fix For: 3.0.0 rc2
>
> Attachments: size_estimates.tar.bz2, system.log.bz2
>
>
> When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will
> get stuck in a compaction loop throwing:
> {code}
> java.lang.UnsupportedOperationException
> at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382)
> at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147)
> at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80)
> at
> org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100)
> at
> org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72)
> at
> org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not
> surface the issue. It can be reproduced with the following (note -- changing
> jdks will need to be altered if you aren't on OSX)
> {code}
> #!/bin/sh
> echo "using java7 for cassandra-2.1 instance"
> export JAVA_HOME=$(/usr/libexec/java_home -v 1.7)
> ccm stop
> ccm remove upgrade_nodes
> ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes
> ccm start
> ccm node1 stress write n=5K -rate threads=4 -mode native cql3
> sleep 10
[jira] [Updated] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
[
https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Hust updated CASSANDRA-10360:
Description:
When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will get
stuck in a compaction loop throwing:
{code}
java.lang.UnsupportedOperationException
at
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382)
at
org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147)
at
org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503)
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80)
at
org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72)
at
org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100)
at
org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72)
at
org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
at
org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not
surface the issue. It can be reproduced with the following (note -- changing
jdks will need to be altered if you aren't on OSX)
{code}
#!/bin/sh
echo "using java7 for cassandra-2.1 instance"
export JAVA_HOME=$(/usr/libexec/java_home -v 1.7)
ccm stop
ccm remove upgrade_nodes
ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes
ccm start
ccm node1 stress write n=5K -rate threads=4 -mode native cql3
sleep 10
for cver in 2.2 3.0
do
echo "Draining all nodes"
ccm node1 nodetool drain
ccm stop
echo "switching to java 8"
export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)
echo "Upgrading to git:cassandra-$cver"
ccm setdir -v git:cassandra-$cver
ccm start
echo "Sleeping to all version migrations"
sleep 30
echo "Upgrading sstables"
ccm node1 nodetool upgradesstables
ccm node1 stress write n=5K -rate threads=4 -mode native cql3
sleep 10
done
echo "***"
echo "instead of creating churn to cause compaction naturally forcing
compaction of system keyspace"
echo "***"
ccm node1 nodetool compact system
ccm stop
{code}
The example uses {{nodetool compact system}} but it will also occur with
{{nodetool upgradesstables system}}. I'm puzzled by that since the script runs
{{upgradesstables}} on each iteration. Is the system keyspace not effected by
the command without arguments?
Ran against:
2.1: {{e889ee408bec5330c312ff6b72a81a0012fdf2a5}}
2.2:
[jira] [Created] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
Andrew Hust created CASSANDRA-10360:
---
Summary: UnsupportedOperationException when compacting
system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade
Key: CASSANDRA-10360
URL: https://issues.apache.org/jira/browse/CASSANDRA-10360
Project: Cassandra
Issue Type: Bug
Reporter: Andrew Hust
When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will get
stuck in a compaction loop throwing:
{code}
java.lang.UnsupportedOperationException
at
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382)
at
org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147)
at
org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503)
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80)
at
org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72)
at
org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100)
at
org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72)
at
org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
at
org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
It will only occur when upgrading thru 2.2. Going from 2.1 -> 3.0 will not
surface the issue. It can be reproduced with the following (note -- changing
jdks will need to be altered if you aren't on OSX)
{code}
#!/bin/sh
echo "using java7 for cassandra-2.1 instance"
export JAVA_HOME=$(/usr/libexec/java_home -v 1.7)
ccm stop
ccm remove upgrade_nodes
ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes
ccm start
ccm node1 stress write n=5K -rate threads=4 -mode native cql3
sleep 10
for cver in 2.2 3.0
do
echo "Draining all nodes"
ccm node1 nodetool drain
ccm stop
echo "switching to java 8"
export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)
echo "Upgrading to git:cassandra-$cver"
ccm setdir -v git:cassandra-$cver
ccm start
echo "Sleeping to all version migrations"
sleep 30
echo "Upgrading sstables"
ccm node1 nodetool upgradesstables
ccm node1 stress write n=5K -rate threads=4 -mode native cql3
sleep 10
done
echo "***"
echo "instead of creating churn to cause compaction naturally forcing
compaction of system keyspace"
echo "***"
ccm node1 nodetool compact system
ccm stop
{code}
The example uses {{nodetool compact system}} but it will also occur with
{{nodetool upgradesstables system}}. I'm puzzled by that since the script runs
[jira] [Created] (CASSANDRA-10350) cqlsh describe keyspace output no longers keeps indexes in sorted order
Andrew Hust created CASSANDRA-10350:
---
Summary: cqlsh describe keyspace output no longers keeps indexes
in sorted order
Key: CASSANDRA-10350
URL: https://issues.apache.org/jira/browse/CASSANDRA-10350
Project: Cassandra
Issue Type: Bug
Reporter: Andrew Hust
cqlsh command {{describe keyspace }} no longer keeps indexes in alpha
sorted order. This was caught with a dtest on
[cassci|http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/lastCompletedBuild/testReport/cqlsh_tests.cqlsh_tests/TestCqlsh/test_describe/].
Tested on: C* {{b4544846def2bdd00ff841c7e3d9f2559620827b}}
Can be reproduced with the following:
{code}
ccm stop
ccm remove describe_order
ccm create -n 1 -v git:cassandra-2.2 describe_order
ccm start
cat << EOF | ccm node1 cqlsh
CREATE KEYSPACE ks1 WITH replication = {'class': 'SimpleStrategy',
'replication_factor': 1};
USE ks1;
CREATE TABLE ks1.test (id int, col int, val text, val2 text, val3 text, PRIMARY
KEY(id, col));
CREATE INDEX ix0 ON ks1.test (col);
CREATE INDEX ix3 ON ks1.test (val3);
CREATE INDEX ix2 ON ks1.test (val2);
CREATE INDEX ix1 ON ks1.test (val);
DESCRIBE KEYSPACE ks1;
EOF
ccm stop
ccm setdir -v git:cassandra-3.0
ccm start
sleep 15
cat << EOF | ccm node1 cqlsh
DESCRIBE KEYSPACE ks1;
EOF
ccm stop
{code}
Ouput on <= cassandra-2.2:
{code}
CREATE INDEX ix0 ON ks1.test (col);
CREATE INDEX ix1 ON ks1.test (val);
CREATE INDEX ix2 ON ks1.test (val2);
CREATE INDEX ix3 ON ks1.test (val3);
{code}
Output on cassandra-3.0:
{code}
CREATE INDEX ix2 ON ks1.test (val2);
CREATE INDEX ix3 ON ks1.test (val3);
CREATE INDEX ix0 ON ks1.test (col);
CREATE INDEX ix1 ON ks1.test (val);
{code}
//CC [~enigmacurry]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7771) Allow multiple 2ndary index on the same column
[
https://issues.apache.org/jira/browse/CASSANDRA-7771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14729220#comment-14729220
]
Andrew Hust commented on CASSANDRA-7771:
When creating separate indexes on both the key and value of a map column the
ddl for the table in cqlsh only contains the index on the value. Both indexes
are functional and queries return expected results. When querying metadata
from the python driver (3.0.0a2) both indexes are present and using the
function as_cql_query produces the correct ddl. This might just be an out of
date python lib in cqlsh.
Tested on C*: {{66b0e1d7889d0858753c6e364e77d86fe278eee4}}
Can be reproduced with the following shell commands and ccm:
{code}
ccm remove 2i_test
ccm create -n 1 -v git:cassandra-3.0 -s 2i_test
ccm start
cat << EOF | ccm node1 cqlsh
CREATE KEYSPACE index_test_ks WITH replication = {'class': 'SimpleStrategy',
'replication_factor': 1};
USE index_test_ks;
CREATE TABLE tbl1 (id uuid primary key, ds map, c1 int);
INSERT INTO tbl1 (id, ds, c1) values (uuid(), {'foo': 1, 'bar': 2}, 1);
INSERT INTO tbl1 (id, ds, c1) values (uuid(), {'faz': 1, 'baz': 2}, 2);
CREATE INDEX ix_tbl1_map_values ON tbl1(ds);
CREATE INDEX ix_tbl1_map_keys ON tbl1(keys(ds));
SELECT * FROM tbl1 where ds contains 1;
SELECT * FROM tbl1 where ds contains key 'foo';
// ***
// DDL only has ix_tbl1_map_values present
// ***
DESC TABLE tbl1;
// ***
// system_schema.indexes is correct
// ***
SELECT * FROM system_schema.indexes;
EOF
ccm stop
{code}
Example output:
{code}
CREATE TABLE index_test_ks.tbl1 (
id uuid PRIMARY KEY,
c1 int,
ds map
) WITH bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';
CREATE INDEX ix_tbl1_map_values ON index_test_ks.tbl1 (ds);
keyspace_name | table_name | index_name | index_type | options
| target_columns | target_type
---++++--++-
index_test_ks | tbl1 | ix_tbl1_map_keys | COMPOSITES |
{'index_keys': ''} | {'ds'} | COLUMN
index_test_ks | tbl1 | ix_tbl1_map_values | COMPOSITES |
{'index_values': ''} | {'ds'} | COLUMN
{code}
> Allow multiple 2ndary index on the same column
> --
>
> Key: CASSANDRA-7771
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7771
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sylvain Lebresne
>Assignee: Sam Tunnicliffe
> Labels: client-impacting
> Fix For: 3.0 beta 1
>
>
> Currently, the code assumes that we'll only have one 2ndary index per column.
> This has been reasonable so far but stop being it with CASSANDRA-6382 (you
> might want to index multiple fields of the same UDT column) and
> CASSANDRA-7458 (you may want to have one "normal" index an multiple
> functional index for the same column). So we should consider removing that
> assumption in the code, which is mainly 2 places:
> # in the schema: each ColumnDefinition only has infos for one index. This
> part should probably be tackled in CASSANDRA-6717 so I'm marking this issue
> as a follow-up of CASSANDRA-6717.
> # in the 2ndary index API: this is the part I'm suggesting we fix in this
> issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (CASSANDRA-9459) SecondaryIndex API redesign
[
https://issues.apache.org/jira/browse/CASSANDRA-9459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Hust updated CASSANDRA-9459:
---
Comment: was deleted
(was: When creating separate indexes on both the key and value of a map column
the ddl for the table in cqlsh only contains the index on the value. Both
indexes are functional and queries return expected results. When querying
metadata from the python driver (3.0.0a2) both indexes are present and using
the function as_cql_query produces the correct ddl. This might just be an out
of date python lib in cqlsh.
Tested on C*: {{66b0e1d7889d0858753c6e364e77d86fe278eee4}}
Can be reproduced with the following shell commands and ccm:
{code}
ccm remove 2i_test
ccm create -n 1 -v git:cassandra-3.0 -s 2i_test
ccm start
cat << EOF | ccm node1 cqlsh
CREATE KEYSPACE index_test_ks WITH replication = {'class': 'SimpleStrategy',
'replication_factor': 1};
USE index_test_ks;
CREATE TABLE tbl1 (id uuid primary key, ds map, c1 int);
INSERT INTO tbl1 (id, ds, c1) values (uuid(), {'foo': 1, 'bar': 2}, 1);
INSERT INTO tbl1 (id, ds, c1) values (uuid(), {'faz': 1, 'baz': 2}, 2);
CREATE INDEX ix_tbl1_map_values ON tbl1(ds);
CREATE INDEX ix_tbl1_map_keys ON tbl1(keys(ds));
SELECT * FROM tbl1 where ds contains 1;
SELECT * FROM tbl1 where ds contains key 'foo';
// ***
// DDL only has ix_tbl1_map_values present
// ***
DESC TABLE tbl1;
// ***
// system_schema.indexes is correct
// ***
SELECT * FROM system_schema.indexes;
EOF
ccm stop
{code}
Example output:
{code}
CREATE TABLE index_test_ks.tbl1 (
id uuid PRIMARY KEY,
c1 int,
ds map
) WITH bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';
CREATE INDEX ix_tbl1_map_values ON index_test_ks.tbl1 (ds);
keyspace_name | table_name | index_name | index_type | options
| target_columns | target_type
---++++--++-
index_test_ks | tbl1 | ix_tbl1_map_keys | COMPOSITES |
{'index_keys': ''} | {'ds'} | COLUMN
index_test_ks | tbl1 | ix_tbl1_map_values | COMPOSITES |
{'index_values': ''} | {'ds'} | COLUMN
{code})
> SecondaryIndex API redesign
> ---
>
> Key: CASSANDRA-9459
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9459
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
> Fix For: 3.0 beta 1
>
>
> For some time now the index subsystem has been a pain point and in large part
> this is due to the way that the APIs and principal classes have grown
> organically over the years. It would be a good idea to conduct a wholesale
> review of the area and see if we can come up with something a bit more
> coherent.
> A few starting points:
> * There's a lot in AbstractPerColumnSecondaryIndex & its subclasses which
> could be pulled up into SecondaryIndexSearcher (note that to an extent, this
> is done in CASSANDRA-8099).
> * SecondayIndexManager is overly complex and several of its functions should
> be simplified/re-examined. The handling of which columns are indexed and
> index selection on both the read and write paths are somewhat dense and
> unintuitive.
> * The SecondaryIndex class hierarchy is rather convoluted and could use some
> serious rework.
> There are a number of outstanding tickets which we should be able to roll
> into this higher level one as subtasks (but I'll defer doing that until
> getting into the details of the redesign):
> * CASSANDRA-7771
> * CASSANDRA-8103
> * CASSANDRA-9041
> * CASSANDRA-4458
> * CASSANDRA-8505
> Whilst they're not hard dependencies, I propose that this be done on top of
> both CASSANDRA-8099 and CASSANDRA-6717. The former largely because the
> storage engine changes may facilitate a friendlier index API, but also
> because of the changes to SIS mentioned above. As for 6717, the changes to
> schema tables there will help facilitate CASSANDRA-7771.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9459) SecondaryIndex API redesign
[
https://issues.apache.org/jira/browse/CASSANDRA-9459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14727859#comment-14727859
]
Andrew Hust commented on CASSANDRA-9459:
When creating separate indexes on both the key and value of a map column the
ddl for the table in cqlsh only contains the index on the value. Both indexes
are functional and queries return expected results. When querying metadata
from the python driver (3.0.0a2) both indexes are present and using the
function as_cql_query produces the correct ddl. This might just be an out of
date python lib in cqlsh.
Tested on C*: {{66b0e1d7889d0858753c6e364e77d86fe278eee4}}
Can be reproduced with the following shell commands and ccm:
{code}
ccm remove 2i_test
ccm create -n 1 -v git:cassandra-3.0 -s 2i_test
ccm start
cat << EOF | ccm node1 cqlsh
CREATE KEYSPACE index_test_ks WITH replication = {'class': 'SimpleStrategy',
'replication_factor': 1};
USE index_test_ks;
CREATE TABLE tbl1 (id uuid primary key, ds map, c1 int);
INSERT INTO tbl1 (id, ds, c1) values (uuid(), {'foo': 1, 'bar': 2}, 1);
INSERT INTO tbl1 (id, ds, c1) values (uuid(), {'faz': 1, 'baz': 2}, 2);
CREATE INDEX ix_tbl1_map_values ON tbl1(ds);
CREATE INDEX ix_tbl1_map_keys ON tbl1(keys(ds));
SELECT * FROM tbl1 where ds contains 1;
SELECT * FROM tbl1 where ds contains key 'foo';
// ***
// DDL only has ix_tbl1_map_values present
// ***
DESC TABLE tbl1;
// ***
// system_schema.indexes is correct
// ***
SELECT * FROM system_schema.indexes;
EOF
ccm stop
{code}
Example output:
{code}
CREATE TABLE index_test_ks.tbl1 (
id uuid PRIMARY KEY,
c1 int,
ds map
) WITH bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';
CREATE INDEX ix_tbl1_map_values ON index_test_ks.tbl1 (ds);
keyspace_name | table_name | index_name | index_type | options
| target_columns | target_type
---++++--++-
index_test_ks | tbl1 | ix_tbl1_map_keys | COMPOSITES |
{'index_keys': ''} | {'ds'} | COLUMN
index_test_ks | tbl1 | ix_tbl1_map_values | COMPOSITES |
{'index_values': ''} | {'ds'} | COLUMN
{code}
> SecondaryIndex API redesign
> ---
>
> Key: CASSANDRA-9459
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9459
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
> Fix For: 3.0 beta 1
>
>
> For some time now the index subsystem has been a pain point and in large part
> this is due to the way that the APIs and principal classes have grown
> organically over the years. It would be a good idea to conduct a wholesale
> review of the area and see if we can come up with something a bit more
> coherent.
> A few starting points:
> * There's a lot in AbstractPerColumnSecondaryIndex & its subclasses which
> could be pulled up into SecondaryIndexSearcher (note that to an extent, this
> is done in CASSANDRA-8099).
> * SecondayIndexManager is overly complex and several of its functions should
> be simplified/re-examined. The handling of which columns are indexed and
> index selection on both the read and write paths are somewhat dense and
> unintuitive.
> * The SecondaryIndex class hierarchy is rather convoluted and could use some
> serious rework.
> There are a number of outstanding tickets which we should be able to roll
> into this higher level one as subtasks (but I'll defer doing that until
> getting into the details of the redesign):
> * CASSANDRA-7771
> * CASSANDRA-8103
> * CASSANDRA-9041
> * CASSANDRA-4458
> * CASSANDRA-8505
> Whilst they're not hard dependencies, I propose that this be done on top of
> both CASSANDRA-8099 and CASSANDRA-6717. The former largely because the
> storage engine changes may facilitate a friendlier index API, but also
> because of the changes to SIS mentioned above. As for 6717, the changes to
> schema tables there will help facilitate CASSANDRA-7771.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10250) Executing lots of schema alters concurrently can lead to dropped alters
[
https://issues.apache.org/jira/browse/CASSANDRA-10250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Hust updated CASSANDRA-10250:
Description:
A recently added
[dtest|http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/132/testReport/junit/concurrent_schema_changes_test/TestConcurrentSchemaChanges/create_lots_of_schema_churn_test/]
has been flapping on cassci and has exposed an issue with running lots of
schema alterations concurrently. The failures occur on healthy clusters but
seem to occur at higher rates when 1 node is down during the alters.
The test executes the following – 440 total commands:
- Create 20 new tables
- Drop 7 columns one at time across 20 tables
- Add 7 columns on at time across 20 tables
- Add one column index on each of the 7 columns on 20 tables
Outcome is random. Majority of the failures are dropped columns still being
present, but new columns and indexes have been observed to be incorrect. The
logs are don’t have exceptions and the columns/indexes that are incorrect don’t
seem to follow a pattern. Running a {{nodetool describecluster}} on each node
shows the same schema id on all nodes.
Attached is a python script extracted from the dtest. Running against a local
3 node cluster will reproduce the issue (with enough runs – fails ~20% on my
machine).
Also attached is the node logs from a run with when a dropped column
(alter_me_7 table, column s1) is still present. Checking the system_schema
tables for this case shows the s1 column in both the columns and drop_columns
tables.
This has been flapping on cassci on versions 2+ and doesn’t seem to be related
to changes in 3.0. More testing needs to be done though.
//cc [~enigmacurry]
was:
A recently added
[dtest|http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/132/testReport/junit/concurrent_schema_changes_test/TestConcurrentSchemaChanges/create_lots_of_schema_churn_test/]
has been flapping on cassci and has exposed an issue with running lots of
schema alterations concurrently. The failures occur on healthy clusters but
seem to occur at higher rates when 1 node is down during the alters.
The test executes the following – 440 total commands:
- Create 20 new tables
- Drop 7 columns one at time across 20 tables
- Add 7 columns on at time across 20 tables
- Add one column index on each of the 7 columns on 20 tables
Outcome is random. Majority of the failures are dropped columns still being
present, but new columns and indexes have been observed to be incorrect. The
logs are don’t have exceptions and the columns/indexes that are incorrect don’t
seem to follow a pattern. Running a {{nodetool describecluster}} on each node
shows the same schema id on all nodes.
Attached is a python script extracted from the dtest. Running against a local
3 node cluster will reproduce the issue (with enough runs – fails ~20% on my
machine).
Also attached is the node logs from a run with when a dropped column
(alter_me_7 table, column s1) is still present. Checking the system_schema
tables for this case shows the s1 column in both the columns and drop_columns
tables.
This has been flapping on cassci on versions 2+ and doesn’t seem to be related
to changes in 3.0. More testing needs to be done though.
> Executing lots of schema alters concurrently can lead to dropped alters
> ---
>
> Key: CASSANDRA-10250
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10250
> Project: Cassandra
> Issue Type: Bug
>Reporter: Andrew Hust
> Attachments: concurrent_schema_changes.py, node1.log, node2.log,
> node3.log
>
>
> A recently added
> [dtest|http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/132/testReport/junit/concurrent_schema_changes_test/TestConcurrentSchemaChanges/create_lots_of_schema_churn_test/]
> has been flapping on cassci and has exposed an issue with running lots of
> schema alterations concurrently. The failures occur on healthy clusters but
> seem to occur at higher rates when 1 node is down during the alters.
> The test executes the following – 440 total commands:
> - Create 20 new tables
> - Drop 7 columns one at time across 20 tables
> - Add 7 columns on at time across 20 tables
> - Add one column index on each of the 7 columns on 20 tables
> Outcome is random. Majority of the failures are dropped columns still being
> present, but new columns and indexes have been observed to be incorrect. The
> logs are don’t have exceptions and the columns/indexes that are incorrect
> don’t seem to follow a pattern. Running a {{nodetool describecluster}} on
> each node shows the same schema id on all nodes.
> Attached is a python script extracted from the
[jira] [Updated] (CASSANDRA-10250) Executing lots of schema alters concurrently can lead to dropped alters
[
https://issues.apache.org/jira/browse/CASSANDRA-10250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Hust updated CASSANDRA-10250:
Description:
A recently added
[dtest|http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/132/testReport/junit/concurrent_schema_changes_test/TestConcurrentSchemaChanges/create_lots_of_schema_churn_test/]
has been flapping on cassci and has exposed an issue with running lots of
schema alterations concurrently. The failures occur on healthy clusters but
seem to occur at higher rates when 1 node is down during the alters.
The test executes the following – 440 total commands:
- Create 20 new tables
- Drop 7 columns one at time across 20 tables
- Add 7 columns one at time across 20 tables
- Add one column index on each of the 7 columns on 20 tables
Outcome is random. Majority of the failures are dropped columns still being
present, but new columns and indexes have been observed to be incorrect. The
logs are don’t have exceptions and the columns/indexes that are incorrect don’t
seem to follow a pattern. Running a {{nodetool describecluster}} on each node
shows the same schema id on all nodes.
Attached is a python script extracted from the dtest. Running against a local
3 node cluster will reproduce the issue (with enough runs – fails ~20% on my
machine).
Also attached is the node logs from a run with when a dropped column
(alter_me_7 table, column s1) is still present. Checking the system_schema
tables for this case shows the s1 column in both the columns and drop_columns
tables.
This has been flapping on cassci on versions 2+ and doesn’t seem to be related
to changes in 3.0. More testing needs to be done though.
//cc [~enigmacurry]
was:
A recently added
[dtest|http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/132/testReport/junit/concurrent_schema_changes_test/TestConcurrentSchemaChanges/create_lots_of_schema_churn_test/]
has been flapping on cassci and has exposed an issue with running lots of
schema alterations concurrently. The failures occur on healthy clusters but
seem to occur at higher rates when 1 node is down during the alters.
The test executes the following – 440 total commands:
- Create 20 new tables
- Drop 7 columns one at time across 20 tables
- Add 7 columns on at time across 20 tables
- Add one column index on each of the 7 columns on 20 tables
Outcome is random. Majority of the failures are dropped columns still being
present, but new columns and indexes have been observed to be incorrect. The
logs are don’t have exceptions and the columns/indexes that are incorrect don’t
seem to follow a pattern. Running a {{nodetool describecluster}} on each node
shows the same schema id on all nodes.
Attached is a python script extracted from the dtest. Running against a local
3 node cluster will reproduce the issue (with enough runs – fails ~20% on my
machine).
Also attached is the node logs from a run with when a dropped column
(alter_me_7 table, column s1) is still present. Checking the system_schema
tables for this case shows the s1 column in both the columns and drop_columns
tables.
This has been flapping on cassci on versions 2+ and doesn’t seem to be related
to changes in 3.0. More testing needs to be done though.
//cc [~enigmacurry]
> Executing lots of schema alters concurrently can lead to dropped alters
> ---
>
> Key: CASSANDRA-10250
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10250
> Project: Cassandra
> Issue Type: Bug
>Reporter: Andrew Hust
> Attachments: concurrent_schema_changes.py, node1.log, node2.log,
> node3.log
>
>
> A recently added
> [dtest|http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/132/testReport/junit/concurrent_schema_changes_test/TestConcurrentSchemaChanges/create_lots_of_schema_churn_test/]
> has been flapping on cassci and has exposed an issue with running lots of
> schema alterations concurrently. The failures occur on healthy clusters but
> seem to occur at higher rates when 1 node is down during the alters.
> The test executes the following – 440 total commands:
> - Create 20 new tables
> - Drop 7 columns one at time across 20 tables
> - Add 7 columns one at time across 20 tables
> - Add one column index on each of the 7 columns on 20 tables
> Outcome is random. Majority of the failures are dropped columns still being
> present, but new columns and indexes have been observed to be incorrect. The
> logs are don’t have exceptions and the columns/indexes that are incorrect
> don’t seem to follow a pattern. Running a {{nodetool describecluster}} on
> each node shows the same schema id on all nodes.
> Attached is a python script
[jira] [Commented] (CASSANDRA-10250) Executing lots of schema alters concurrently can lead to dropped alters
[
https://issues.apache.org/jira/browse/CASSANDRA-10250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14726721#comment-14726721
]
Andrew Hust commented on CASSANDRA-10250:
-
example script run -- missing c4 column
{code}
❯ python concurrent_schema_changes.py
creating base tables to be added/altered
executing creation of tables, add/drop column and index creation
sleeping 20 to make sure things are settled
verifing schema status
Errors found:
alter_me_8 expected c1 -> c7, id, got: [u'c1', u'c2', u'c3', u'c5', u'c6',
u'c7', u'id']
{code}
> Executing lots of schema alters concurrently can lead to dropped alters
> ---
>
> Key: CASSANDRA-10250
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10250
> Project: Cassandra
> Issue Type: Bug
>Reporter: Andrew Hust
> Attachments: concurrent_schema_changes.py, node1.log, node2.log,
> node3.log
>
>
> A recently added
> [dtest|http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/132/testReport/junit/concurrent_schema_changes_test/TestConcurrentSchemaChanges/create_lots_of_schema_churn_test/]
> has been flapping on cassci and has exposed an issue with running lots of
> schema alterations concurrently. The failures occur on healthy clusters but
> seem to occur at higher rates when 1 node is down during the alters.
> The test executes the following – 440 total commands:
> - Create 20 new tables
> - Drop 7 columns one at time across 20 tables
> - Add 7 columns on at time across 20 tables
> - Add one column index on each of the 7 columns on 20 tables
> Outcome is random. Majority of the failures are dropped columns still being
> present, but new columns and indexes have been observed to be incorrect. The
> logs are don’t have exceptions and the columns/indexes that are incorrect
> don’t seem to follow a pattern. Running a {{nodetool describecluster}} on
> each node shows the same schema id on all nodes.
> Attached is a python script extracted from the dtest. Running against a
> local 3 node cluster will reproduce the issue (with enough runs – fails ~20%
> on my machine).
> Also attached is the node logs from a run with when a dropped column
> (alter_me_7 table, column s1) is still present. Checking the system_schema
> tables for this case shows the s1 column in both the columns and drop_columns
> tables.
> This has been flapping on cassci on versions 2+ and doesn’t seem to be
> related to changes in 3.0. More testing needs to be done though.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-10250) Executing lots of schema alters concurrently can lead to dropped alters
Andrew Hust created CASSANDRA-10250:
---
Summary: Executing lots of schema alters concurrently can lead to
dropped alters
Key: CASSANDRA-10250
URL: https://issues.apache.org/jira/browse/CASSANDRA-10250
Project: Cassandra
Issue Type: Bug
Reporter: Andrew Hust
A recently added
[dtest|http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/132/testReport/junit/concurrent_schema_changes_test/TestConcurrentSchemaChanges/create_lots_of_schema_churn_test/]
has been flapping on cassci and has exposed an issue with running lots of
schema alterations concurrently. The failures occur on healthy clusters but
seem to occur at higher rates when 1 node is down during the alters.
The test executes the following – 440 total commands:
- Create 20 new tables
- Drop 7 columns one at time across 20 tables
- Add 7 columns on at time across 20 tables
- Add one column index on each of the 7 columns on 20 tables
Outcome is random. Majority of the failures are dropped columns still being
present, but new columns and indexes have been observed to be incorrect. The
logs are don’t have exceptions and the columns/indexes that are incorrect don’t
seem to follow a pattern. Running a {{nodetool describecluster}} on each node
shows the same schema id on all nodes.
Attached is a python script extracted from the dtest. Running against a local
3 node cluster will reproduce the issue (with enough runs – fails ~20% on my
machine).
Also attached is the node logs from a run with when a dropped column
(alter_me_7 table, column s1) is still present. Checking the system_schema
tables for this case shows the s1 column in both the columns and drop_columns
tables.
This has been flapping on cassci on versions 2+ and doesn’t seem to be related
to changes in 3.0. More testing needs to be done though.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10250) Executing lots of schema alters concurrently can lead to dropped alters
[
https://issues.apache.org/jira/browse/CASSANDRA-10250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Hust updated CASSANDRA-10250:
Attachment: node3.log
node2.log
node1.log
concurrent_schema_changes.py
> Executing lots of schema alters concurrently can lead to dropped alters
> ---
>
> Key: CASSANDRA-10250
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10250
> Project: Cassandra
> Issue Type: Bug
>Reporter: Andrew Hust
> Attachments: concurrent_schema_changes.py, node1.log, node2.log,
> node3.log
>
>
> A recently added
> [dtest|http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/132/testReport/junit/concurrent_schema_changes_test/TestConcurrentSchemaChanges/create_lots_of_schema_churn_test/]
> has been flapping on cassci and has exposed an issue with running lots of
> schema alterations concurrently. The failures occur on healthy clusters but
> seem to occur at higher rates when 1 node is down during the alters.
> The test executes the following – 440 total commands:
> - Create 20 new tables
> - Drop 7 columns one at time across 20 tables
> - Add 7 columns on at time across 20 tables
> - Add one column index on each of the 7 columns on 20 tables
> Outcome is random. Majority of the failures are dropped columns still being
> present, but new columns and indexes have been observed to be incorrect. The
> logs are don’t have exceptions and the columns/indexes that are incorrect
> don’t seem to follow a pattern. Running a {{nodetool describecluster}} on
> each node shows the same schema id on all nodes.
> Attached is a python script extracted from the dtest. Running against a
> local 3 node cluster will reproduce the issue (with enough runs – fails ~20%
> on my machine).
> Also attached is the node logs from a run with when a dropped column
> (alter_me_7 table, column s1) is still present. Checking the system_schema
> tables for this case shows the s1 column in both the columns and drop_columns
> tables.
> This has been flapping on cassci on versions 2+ and doesn’t seem to be
> related to changes in 3.0. More testing needs to be done though.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10156) Creating Materialized views concurrently leads to missing data
[
https://issues.apache.org/jira/browse/CASSANDRA-10156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14717657#comment-14717657
]
Andrew Hust commented on CASSANDRA-10156:
-
Confirmed patch corrects problem:
Failed as expected on:
origin/trunk : {{f744b6c05537b84b60f53917fafb32bc1d8b1b03}}
Passes on:
tjake/10156 : {{0edab864030817d1fe3739832a72453f084179b4}}
Once this is merged I'll un-skip the
[dtest|https://github.com/riptano/cassandra-dtest/blob/master/concurrent_schema_changes_test.py#L243]
Creating Materialized views concurrently leads to missing data
--
Key: CASSANDRA-10156
URL: https://issues.apache.org/jira/browse/CASSANDRA-10156
Project: Cassandra
Issue Type: Bug
Reporter: Alan Boudreault
Assignee: T Jake Luciani
Fix For: 3.0 beta 2
Attachments: mv_test_bad.sh, mv_test_good.sh
[~nutbunnies] was writing dtests that create multiple tables concurrently. He
also wrote a test that creates multiple MV but has not been able to get it
works properly. After some debugging outside of dtest, it seems that there is
an issue if we create more than 1 MV at the same time. There is no errors in
the log but the MV are never entirely populated and are missing data.
I've attached 2 scripts:
[^mv_test_bad.sh]: is the one that reproduce the issue. It creates 4 MVs at
the same time. At the end, some data are missing in the MVs and there is
nothing in system.hints or system.batchlog.
[^mv_test_good.sh]: is the same script but that waits 10 seconds between each
MV creation, which results in 4 MVs with all the data.
Some more notes from Andrew:
{code}
- lowering the number of rows inserted below ~1000 won't exhibit the
inconsistent behavior
- adding more columns/MV make it worse -- more of the MVs counts are
consistently wrong
- multiple runs will range in disagreement -- usually one of the MVs is
correct though
- the describe cluster and system.mv* queries always look good
{code}
Thanks Andrew for finding this bug and providing the test scripts!
//cc [~carlyeks] [~tjake] [~enigmacurry]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6717) Modernize schema tables
[
https://issues.apache.org/jira/browse/CASSANDRA-6717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701618#comment-14701618
]
Andrew Hust commented on CASSANDRA-6717:
There appears to be a couple issues with UDT and UDF migration from 2.2-3.0.
Both seem to be related to the number of nodes in the cluster during the
upgrade. More nodes cause more duplication of field data but it's inconsistent
and can range from the correct value to the number of nodes. I haven’t been
able reproduce on single or 2 node clusters. I’ve verified that the metadata
returned from the driver does match the reality of the system_schema.functions
and system_schema.user_types tables.
I've extracted the failures from the upgrade test to simplify and remove other
test influences.
UDT:field_names duplication can be reproduced with this
[gist|https://gist.github.com/nutbunnies/4d1dcfe96556d0218b15]
UDF:argument_names duplication can be reproduced with this
[gist|https://gist.github.com/nutbunnies/51b61d245f00106b1641]
Both can also be reproduced by enabling the establish/verify methods in the
upgrade dtest in this [PR|https://github.com/riptano/cassandra-dtest/pull/447].
This was done with C* at {{06c130e3cb85577041b475084400c08c505d8f9e}} and the
python-driver at {{caf58cc5e72afe664759b50dbcc92993902beb3e}}
Modernize schema tables
---
Key: CASSANDRA-6717
URL: https://issues.apache.org/jira/browse/CASSANDRA-6717
Project: Cassandra
Issue Type: Sub-task
Reporter: Sylvain Lebresne
Assignee: Aleksey Yeschenko
Labels: client-impacting, doc-impacting
Fix For: 3.0 beta 2
There is a few problems/improvements that can be done with the way we store
schema:
# CASSANDRA-4988: as explained on the ticket, storing the comparator is now
redundant (or almost, we'd need to store whether the table is COMPACT or not
too, which we don't currently is easy and probably a good idea anyway), it
can be entirely reconstructed from the infos in schema_columns (the same is
true of key_validator and subcomparator, and replacing default_validator by a
COMPACT_VALUE column in all case is relatively simple). And storing the
comparator as an opaque string broke concurrent updates of sub-part of said
comparator (concurrent collection addition or altering 2 separate clustering
columns typically) so it's really worth removing it.
# CASSANDRA-4603: it's time to get rid of those ugly json maps. I'll note
that schema_keyspaces is a problem due to its use of COMPACT STORAGE, but I
think we should fix it once and for-all nonetheless (see below).
# For CASSANDRA-6382 and to allow indexing both map keys and values at the
same time, we'd need to be able to have more than one index definition for a
given column.
# There is a few mismatches in table options between the one stored in the
schema and the one used when declaring/altering a table which would be nice
to fix. The compaction, compression and replication maps are one already
mentioned from CASSANDRA-4603, but also for some reason
'dclocal_read_repair_chance' in CQL is called just 'local_read_repair_chance'
in the schema table, and 'min/max_compaction_threshold' are column families
option in the schema but just compaction options for CQL (which makes more
sense).
None of those issues are major, and we could probably deal with them
independently but it might be simpler to just fix them all in one shot so I
wanted to sum them all up here. In particular, the fact that
'schema_keyspaces' uses COMPACT STORAGE is annoying (for the replication map,
but it may limit future stuff too) which suggest we should migrate it to a
new, non COMPACT table. And while that's arguably a detail, it wouldn't hurt
to rename schema_columnfamilies to schema_tables for the years to come since
that's the prefered vernacular for CQL.
Overall, what I would suggest is to move all schema tables to a new keyspace,
named 'schema' for instance (or 'system_schema' but I prefer the shorter
version), and fix all the issues above at once. Since we currently don't
exchange schema between nodes of different versions, all we'd need to do that
is a one shot startup migration, and overall, I think it could be simpler for
clients to deal with one clear migration than to have to handle minor
individual changes all over the place. I also think it's somewhat cleaner
conceptually to have schema tables in their own keyspace since they are
replicated through a different mechanism than other system tables.
If we do that, we could, for instance, migrate to the following schema tables
(details up for discussion of course):
{noformat}
CREATE TYPE user_type (
name text,
column_names listtext,
column_types listtext
)
CREATE TABLE keyspaces (
[jira] [Commented] (CASSANDRA-6717) Modernize schema tables
[
https://issues.apache.org/jira/browse/CASSANDRA-6717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14662105#comment-14662105
]
Andrew Hust commented on CASSANDRA-6717:
In adding additional table
[dtests|https://github.com/riptano/cassandra-dtest/pull/444/files] I came
across a o.a.c.e.UnrecognizedEntityException when migrating aggregates from 2.2
to 3.0. This failure can be exercised outside of the upgrade dtest with this
[gist|https://gist.github.com/nutbunnies/3a544826cc44c92a3131].
Modernize schema tables
---
Key: CASSANDRA-6717
URL: https://issues.apache.org/jira/browse/CASSANDRA-6717
Project: Cassandra
Issue Type: Sub-task
Reporter: Sylvain Lebresne
Assignee: Sam Tunnicliffe
Labels: client-impacting, doc-impacting
Fix For: 3.0 beta 1
There is a few problems/improvements that can be done with the way we store
schema:
# CASSANDRA-4988: as explained on the ticket, storing the comparator is now
redundant (or almost, we'd need to store whether the table is COMPACT or not
too, which we don't currently is easy and probably a good idea anyway), it
can be entirely reconstructed from the infos in schema_columns (the same is
true of key_validator and subcomparator, and replacing default_validator by a
COMPACT_VALUE column in all case is relatively simple). And storing the
comparator as an opaque string broke concurrent updates of sub-part of said
comparator (concurrent collection addition or altering 2 separate clustering
columns typically) so it's really worth removing it.
# CASSANDRA-4603: it's time to get rid of those ugly json maps. I'll note
that schema_keyspaces is a problem due to its use of COMPACT STORAGE, but I
think we should fix it once and for-all nonetheless (see below).
# For CASSANDRA-6382 and to allow indexing both map keys and values at the
same time, we'd need to be able to have more than one index definition for a
given column.
# There is a few mismatches in table options between the one stored in the
schema and the one used when declaring/altering a table which would be nice
to fix. The compaction, compression and replication maps are one already
mentioned from CASSANDRA-4603, but also for some reason
'dclocal_read_repair_chance' in CQL is called just 'local_read_repair_chance'
in the schema table, and 'min/max_compaction_threshold' are column families
option in the schema but just compaction options for CQL (which makes more
sense).
None of those issues are major, and we could probably deal with them
independently but it might be simpler to just fix them all in one shot so I
wanted to sum them all up here. In particular, the fact that
'schema_keyspaces' uses COMPACT STORAGE is annoying (for the replication map,
but it may limit future stuff too) which suggest we should migrate it to a
new, non COMPACT table. And while that's arguably a detail, it wouldn't hurt
to rename schema_columnfamilies to schema_tables for the years to come since
that's the prefered vernacular for CQL.
Overall, what I would suggest is to move all schema tables to a new keyspace,
named 'schema' for instance (or 'system_schema' but I prefer the shorter
version), and fix all the issues above at once. Since we currently don't
exchange schema between nodes of different versions, all we'd need to do that
is a one shot startup migration, and overall, I think it could be simpler for
clients to deal with one clear migration than to have to handle minor
individual changes all over the place. I also think it's somewhat cleaner
conceptually to have schema tables in their own keyspace since they are
replicated through a different mechanism than other system tables.
If we do that, we could, for instance, migrate to the following schema tables
(details up for discussion of course):
{noformat}
CREATE TYPE user_type (
name text,
column_names listtext,
column_types listtext
)
CREATE TABLE keyspaces (
name text PRIMARY KEY,
durable_writes boolean,
replication mapstring, string,
user_types mapstring, user_type
)
CREATE TYPE trigger_definition (
name text,
options maptex, text
)
CREATE TABLE tables (
keyspace text,
name text,
id uuid,
table_type text, // COMPACT, CQL or SUPER
dropped_columns maptext, bigint,
triggers maptext, trigger_definition,
// options
comment text,
compaction maptext, text,
compression maptext, text,
read_repair_chance double,
dclocal_read_repair_chance double,
gc_grace_seconds int,
caching text,
rows_per_partition_to_cache text,
default_time_to_live int,
min_index_interval int,
max_index_interval int,
speculative_retry text,
populate_io_cache_on_flush boolean,
bloom_filter_fp_chance double
[jira] [Commented] (CASSANDRA-6717) Modernize schema tables
[
https://issues.apache.org/jira/browse/CASSANDRA-6717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14658951#comment-14658951
]
Andrew Hust commented on CASSANDRA-6717:
I added some
[dtests|https://github.com/riptano/cassandra-dtest/commit/c346c6b87ec081956330e5b3cab2178e4c5b8a23]
to check schema metadata -- anything additional that should be added?
Modernize schema tables
---
Key: CASSANDRA-6717
URL: https://issues.apache.org/jira/browse/CASSANDRA-6717
Project: Cassandra
Issue Type: Sub-task
Reporter: Sylvain Lebresne
Assignee: Aleksey Yeschenko
Labels: client-impacting, doc-impacting
Fix For: 3.0 beta 1
There is a few problems/improvements that can be done with the way we store
schema:
# CASSANDRA-4988: as explained on the ticket, storing the comparator is now
redundant (or almost, we'd need to store whether the table is COMPACT or not
too, which we don't currently is easy and probably a good idea anyway), it
can be entirely reconstructed from the infos in schema_columns (the same is
true of key_validator and subcomparator, and replacing default_validator by a
COMPACT_VALUE column in all case is relatively simple). And storing the
comparator as an opaque string broke concurrent updates of sub-part of said
comparator (concurrent collection addition or altering 2 separate clustering
columns typically) so it's really worth removing it.
# CASSANDRA-4603: it's time to get rid of those ugly json maps. I'll note
that schema_keyspaces is a problem due to its use of COMPACT STORAGE, but I
think we should fix it once and for-all nonetheless (see below).
# For CASSANDRA-6382 and to allow indexing both map keys and values at the
same time, we'd need to be able to have more than one index definition for a
given column.
# There is a few mismatches in table options between the one stored in the
schema and the one used when declaring/altering a table which would be nice
to fix. The compaction, compression and replication maps are one already
mentioned from CASSANDRA-4603, but also for some reason
'dclocal_read_repair_chance' in CQL is called just 'local_read_repair_chance'
in the schema table, and 'min/max_compaction_threshold' are column families
option in the schema but just compaction options for CQL (which makes more
sense).
None of those issues are major, and we could probably deal with them
independently but it might be simpler to just fix them all in one shot so I
wanted to sum them all up here. In particular, the fact that
'schema_keyspaces' uses COMPACT STORAGE is annoying (for the replication map,
but it may limit future stuff too) which suggest we should migrate it to a
new, non COMPACT table. And while that's arguably a detail, it wouldn't hurt
to rename schema_columnfamilies to schema_tables for the years to come since
that's the prefered vernacular for CQL.
Overall, what I would suggest is to move all schema tables to a new keyspace,
named 'schema' for instance (or 'system_schema' but I prefer the shorter
version), and fix all the issues above at once. Since we currently don't
exchange schema between nodes of different versions, all we'd need to do that
is a one shot startup migration, and overall, I think it could be simpler for
clients to deal with one clear migration than to have to handle minor
individual changes all over the place. I also think it's somewhat cleaner
conceptually to have schema tables in their own keyspace since they are
replicated through a different mechanism than other system tables.
If we do that, we could, for instance, migrate to the following schema tables
(details up for discussion of course):
{noformat}
CREATE TYPE user_type (
name text,
column_names listtext,
column_types listtext
)
CREATE TABLE keyspaces (
name text PRIMARY KEY,
durable_writes boolean,
replication mapstring, string,
user_types mapstring, user_type
)
CREATE TYPE trigger_definition (
name text,
options maptex, text
)
CREATE TABLE tables (
keyspace text,
name text,
id uuid,
table_type text, // COMPACT, CQL or SUPER
dropped_columns maptext, bigint,
triggers maptext, trigger_definition,
// options
comment text,
compaction maptext, text,
compression maptext, text,
read_repair_chance double,
dclocal_read_repair_chance double,
gc_grace_seconds int,
caching text,
rows_per_partition_to_cache text,
default_time_to_live int,
min_index_interval int,
max_index_interval int,
speculative_retry text,
populate_io_cache_on_flush boolean,
bloom_filter_fp_chance double
memtable_flush_period_in_ms int,
PRIMARY KEY (keyspace, name)
)
CREATE TYPE index_definition (
name text,
index_type text,
options maptext, text
[jira] [Commented] (CASSANDRA-6717) Modernize schema tables
[
https://issues.apache.org/jira/browse/CASSANDRA-6717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649507#comment-14649507
]
Andrew Hust commented on CASSANDRA-6717:
I’ve started adding dtests for this and ran into a small issue. With the
schema_* tables being renamed some of them conflict with the cql describe
command. Example when scoped to the system_schema keyspace running `desc
tables` gives a list of tables in the keyspace instead of the cql for the
tables table. Using `desc system_schema.tables` works as expected.
Does any documentation or ? need to be updated to explain that behavior?
Modernize schema tables
---
Key: CASSANDRA-6717
URL: https://issues.apache.org/jira/browse/CASSANDRA-6717
Project: Cassandra
Issue Type: Sub-task
Reporter: Sylvain Lebresne
Assignee: Aleksey Yeschenko
Labels: client-impacting, doc-impacting
Fix For: 3.0 beta 1
There is a few problems/improvements that can be done with the way we store
schema:
# CASSANDRA-4988: as explained on the ticket, storing the comparator is now
redundant (or almost, we'd need to store whether the table is COMPACT or not
too, which we don't currently is easy and probably a good idea anyway), it
can be entirely reconstructed from the infos in schema_columns (the same is
true of key_validator and subcomparator, and replacing default_validator by a
COMPACT_VALUE column in all case is relatively simple). And storing the
comparator as an opaque string broke concurrent updates of sub-part of said
comparator (concurrent collection addition or altering 2 separate clustering
columns typically) so it's really worth removing it.
# CASSANDRA-4603: it's time to get rid of those ugly json maps. I'll note
that schema_keyspaces is a problem due to its use of COMPACT STORAGE, but I
think we should fix it once and for-all nonetheless (see below).
# For CASSANDRA-6382 and to allow indexing both map keys and values at the
same time, we'd need to be able to have more than one index definition for a
given column.
# There is a few mismatches in table options between the one stored in the
schema and the one used when declaring/altering a table which would be nice
to fix. The compaction, compression and replication maps are one already
mentioned from CASSANDRA-4603, but also for some reason
'dclocal_read_repair_chance' in CQL is called just 'local_read_repair_chance'
in the schema table, and 'min/max_compaction_threshold' are column families
option in the schema but just compaction options for CQL (which makes more
sense).
None of those issues are major, and we could probably deal with them
independently but it might be simpler to just fix them all in one shot so I
wanted to sum them all up here. In particular, the fact that
'schema_keyspaces' uses COMPACT STORAGE is annoying (for the replication map,
but it may limit future stuff too) which suggest we should migrate it to a
new, non COMPACT table. And while that's arguably a detail, it wouldn't hurt
to rename schema_columnfamilies to schema_tables for the years to come since
that's the prefered vernacular for CQL.
Overall, what I would suggest is to move all schema tables to a new keyspace,
named 'schema' for instance (or 'system_schema' but I prefer the shorter
version), and fix all the issues above at once. Since we currently don't
exchange schema between nodes of different versions, all we'd need to do that
is a one shot startup migration, and overall, I think it could be simpler for
clients to deal with one clear migration than to have to handle minor
individual changes all over the place. I also think it's somewhat cleaner
conceptually to have schema tables in their own keyspace since they are
replicated through a different mechanism than other system tables.
If we do that, we could, for instance, migrate to the following schema tables
(details up for discussion of course):
{noformat}
CREATE TYPE user_type (
name text,
column_names listtext,
column_types listtext
)
CREATE TABLE keyspaces (
name text PRIMARY KEY,
durable_writes boolean,
replication mapstring, string,
user_types mapstring, user_type
)
CREATE TYPE trigger_definition (
name text,
options maptex, text
)
CREATE TABLE tables (
keyspace text,
name text,
id uuid,
table_type text, // COMPACT, CQL or SUPER
dropped_columns maptext, bigint,
triggers maptext, trigger_definition,
// options
comment text,
compaction maptext, text,
compression maptext, text,
read_repair_chance double,
dclocal_read_repair_chance double,
gc_grace_seconds int,
caching text,
rows_per_partition_to_cache text,
default_time_to_live int,
min_index_interval int,
max_index_interval int,
[jira] [Commented] (CASSANDRA-9884) Error on encrypted node communication upgrading from 2.1.6 to 2.2.0
[ https://issues.apache.org/jira/browse/CASSANDRA-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14646629#comment-14646629 ] Andrew Hust commented on CASSANDRA-9884: This appears to be just a 2.2 issue and not related to the process of upgrading from 2.1.x to 2.2. As far as I can tell no new configuration is need for internode SSL communication on 2.2 and a fresh 2.2 cluster shows this failure. Applying the above patch does resolve the issue in my testing. I have added a simple internode SSL verification test and updated the upgrading test to run with/without internode SSL to casasandra-dtest. https://github.com/riptano/cassandra-dtest/pull/427 Error on encrypted node communication upgrading from 2.1.6 to 2.2.0 --- Key: CASSANDRA-9884 URL: https://issues.apache.org/jira/browse/CASSANDRA-9884 Project: Cassandra Issue Type: Bug Components: Config, Core Environment: Ubuntu 14.04.2 LTS 64 bits. Java version 1.8.0_45 Java(TM) SE Runtime Environment (build 1.8.0_45-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode) Reporter: Carlos Scheidecker Priority: Critical Labels: security Fix For: 2.2.x After updating to Cassandra 2.2.0 from 2.1.6 I am having SSL issues. The configuration had not changed from one version to the other, the JVM is still the same however on 2.2.0 it is erroring. I am yet to investigate the source code for it. But for now, this is the information I have to share on it: My JVM is java version 1.8.0_45 Java(TM) SE Runtime Environment (build 1.8.0_45-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode) Ubuntu 14.04.2 LTS is on all nodes, they are the same. Below is the encryption settings from cassandra.yaml of all nodes. I am using the same keystore and trustore as I had used before on 2.1.6 # Enable or disable inter-node encryption # Default settings are TLS v1, RSA 1024-bit keys (it is imperative that # users generate their own keys) TLS_RSA_WITH_AES_128_CBC_SHA as the cipher # suite for authentication, key exchange and encryption of the actual data transfers. # Use the DHE/ECDHE ciphers if running in FIPS 140 compliant mode. # NOTE: No custom encryption options are enabled at the moment # The available internode options are : all, none, dc, rack # # If set to dc cassandra will encrypt the traffic between the DCs # If set to rack cassandra will encrypt the traffic between the racks # # The passwords used in these options must match the passwords used when generating # the keystore and truststore. For instructions on generating these files, see: # http://download.oracle.com/javase/6/docs/technotes/guides/security/jsse/JSSERefGuide.html#CreateKeystore # server_encryption_options: internode_encryption: all keystore: /etc/cassandra/certs/node.keystore keystore_password: mypasswd truststore: /etc/cassandra/certs/global.truststore truststore_password: mypasswd # More advanced defaults below: # protocol: TLS # algorithm: SunX509 # store_type: JKS cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA] require_client_auth: false # enable or disable client/server encryption. Nodes cannot talk to each other as per SSL errors bellow. WARN [MessagingService-Outgoing-/192.168.1.31] 2015-07-22 17:29:48,764 SSLFactory.java:163 - Filtering out TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA as it isnt supported by the socket ERROR [MessagingService-Outgoing-/192.168.1.31] 2015-07-22 17:29:48,764 OutboundTcpConnection.java:229 - error processing a message intended for /192.168.1.31 java.lang.NullPointerException: null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213) ~[guava-16.0.jar:na] at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.init(BufferedDataOutputStreamPlus.java:74) ~[apache-cassandra-2.2.0.jar:2.2.0] at org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:404) ~[apache-cassandra-2.2.0.jar:2.2.0] at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218) ~[apache-cassandra-2.2.0.jar:2.2.0] ERROR [MessagingService-Outgoing-/192.168.1.31] 2015-07-22 17:29:48,764 OutboundTcpConnection.java:316 - error writing to /192.168.1.31 java.lang.NullPointerException: null at org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323) [apache-cassandra-2.2.0.jar:2.2.0] at
