[jira] [Commented] (CASSANDRA-7281) SELECT on tuple relations are broken for mixed ASC/DESC clustering order
[
https://issues.apache.org/jira/browse/CASSANDRA-7281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568783#comment-14568783
]
Benjamin Lerer commented on CASSANDRA-7281:
---
Sorry, my comment was really not that clear.
In 2.1: I think that your code should tackle the problem within the
{{SelectStatement.buildBound}} method. The tests for that method are within
{{SelectStatementTest}}.
In 2.2 you are right the code should be within
{{MultiColumnRestriction.Slice.appendBoundTo}} method and the tests are within
{{PrimaryKeyRestrictionSetTest }}.
I expect that if you manage to fix the problem in 2.1 as part of the
{{SelectStatement.buildBound}} method you should be able to make it work within
2.2 without too much difficulties by extracting your fix and putting it within
{{MultiColumnRestriction.Slice.appendBoundTo}}.
Hope it helps.
Do not hesitate to contact me on IRC or by email if you have more questions
SELECT on tuple relations are broken for mixed ASC/DESC clustering order
Key: CASSANDRA-7281
URL: https://issues.apache.org/jira/browse/CASSANDRA-7281
Project: Cassandra
Issue Type: Bug
Reporter: Sylvain Lebresne
Assignee: Marcin Szymaniuk
Fix For: 2.1.x
Attachments:
0001-CASSANDRA-7281-SELECT-on-tuple-relations-are-broken-.patch,
0001-CASSANDRA-7281-SELECT-on-tuple-relations-are-broken-v2.patch,
0001-CASSANDRA-7281-SELECT-on-tuple-relations-are-broken-v3.patch,
0001-CASSANDRA-7281-SELECT-on-tuple-relations-are-broken-v4.patch
As noted on
[CASSANDRA-6875|https://issues.apache.org/jira/browse/CASSANDRA-6875?focusedCommentId=13992153page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13992153],
the tuple notation is broken when the clustering order mixes ASC and DESC
directives because the range of data they describe don't correspond to a
single continuous slice internally. To copy the example from CASSANDRA-6875:
{noformat}
cqlsh:ks create table foo (a int, b int, c int, PRIMARY KEY (a, b, c)) WITH
CLUSTERING ORDER BY (b DESC, c ASC);
cqlsh:ks INSERT INTO foo (a, b, c) VALUES (0, 2, 0);
cqlsh:ks INSERT INTO foo (a, b, c) VALUES (0, 1, 0);
cqlsh:ks INSERT INTO foo (a, b, c) VALUES (0, 1, 1);
cqlsh:ks INSERT INTO foo (a, b, c) VALUES (0, 0, 0);
cqlsh:ks SELECT * FROM foo WHERE a=0;
a | b | c
---+---+---
0 | 2 | 0
0 | 1 | 0
0 | 1 | 1
0 | 0 | 0
(4 rows)
cqlsh:ks SELECT * FROM foo WHERE a=0 AND (b, c) (1, 0);
a | b | c
---+---+---
0 | 2 | 0
(1 rows)
{noformat}
The last query should really return {{(0, 2, 0)}} and {{(0, 1, 1)}}.
For that specific example we should generate 2 internal slices, but I believe
that with more clustering columns we may have more slices.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8656) long-test LongLeveledCompactionStrategyTest flaps in 2.0
[
https://issues.apache.org/jira/browse/CASSANDRA-8656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568835#comment-14568835
]
Stefania commented on CASSANDRA-8656:
-
Thanks [~mshuler], with a loop it is indeed easy to reproduce and if fails on
average 2 or 3 times every 10 iterations.
I cherry picked the commit for CASSANDRA-8739 and that seems to have fixed it.
I've run the loop twice and I got no failures. See also the attached branch, a
2.0 with CASSANDRA-8739, hopefully it will start running on Jenkins soon.
[~krummas] do you agree with the analysis and if so can you merge CASSANRA-8739
to 2.0?
Here is the bash code if someone wants to test this further:
{code}
#!/bin/bash
for i in `seq 1 10`;
do
echo
echo Run iteration $i
ant long-test -Dtest.name=LongLeveledCompactionStrategyTest
done
{code}
long-test LongLeveledCompactionStrategyTest flaps in 2.0
Key: CASSANDRA-8656
URL: https://issues.apache.org/jira/browse/CASSANDRA-8656
Project: Cassandra
Issue Type: Test
Components: Tests
Reporter: Michael Shuler
Assignee: Stefania
Priority: Minor
Labels: test-failure
Fix For: 2.0.x
Attachments: system.log
LongLeveledCompactionStrategyTest periodically fails with:
{noformat}
[junit] Testsuite:
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest
[junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
54.412 sec
[junit]
[junit] Testcase:
testParallelLeveledCompaction(org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest):
Caused an ERROR
[junit] java.util.concurrent.ExecutionException:
java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) =
current key DecoratedKey(313236, 313236) writing into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] java.lang.RuntimeException:
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last
written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236,
313236) writing into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:413)
[junit] at
org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:402)
[junit] at
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest.testParallelLeveledCompaction(LongLeveledCompactionStrategyTest.java:97)
[junit] Caused by: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) =
current key DecoratedKey(313236, 313236) writing into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] at java.util.concurrent.FutureTask.report(FutureTask.java:122)
[junit] at java.util.concurrent.FutureTask.get(FutureTask.java:188)
[junit] at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:409)
[junit] Caused by: java.lang.RuntimeException: Last written key
DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing
into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
[junit] at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
[junit] at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167)
[junit] at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
[junit] at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
[junit] at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
[junit] at
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest$1.run(LongLeveledCompactionStrategyTest.java:87)
[junit] at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[junit] at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit] at java.lang.Thread.run(Thread.java:745)
[junit]
[junit]
[junit] Test
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest FAILED
{noformat}
I would guess the failure is
[jira] [Commented] (CASSANDRA-7688) Add data sizing to a system table
[
https://issues.apache.org/jira/browse/CASSANDRA-7688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568708#comment-14568708
]
Piotr Kołaczkowski commented on CASSANDRA-7688:
---
So what is this JMX call to refresh the estimates?
I tried setting {{-Dcassandra.size_recorder_interval=1}} (I know, ridiculously
low value) and the size estimates table is still empty when running the tests.
Add data sizing to a system table
-
Key: CASSANDRA-7688
URL: https://issues.apache.org/jira/browse/CASSANDRA-7688
Project: Cassandra
Issue Type: New Feature
Reporter: Jeremiah Jordan
Assignee: Aleksey Yeschenko
Fix For: 2.1.5
Attachments: 7688.txt
Currently you can't implement something similar to describe_splits_ex purely
from the a native protocol driver.
https://datastax-oss.atlassian.net/browse/JAVA-312 is open to expose easily
getting ownership information to a client in the java-driver. But you still
need the data sizing part to get splits of a given size. We should add the
sizing information to a system table so that native clients can get to it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9486) LazilyCompactedRow accumulates all expired RangeTombstones
[ https://issues.apache.org/jira/browse/CASSANDRA-9486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568806#comment-14568806 ] Sylvain Lebresne commented on CASSANDRA-9486: - bq. we switched maxOrderingSet to a LinkedList Make sense, I've pushed the update. I'll wait for the CI to run on it and commit, unless you have remarks on that last versions. LazilyCompactedRow accumulates all expired RangeTombstones -- Key: CASSANDRA-9486 URL: https://issues.apache.org/jira/browse/CASSANDRA-9486 Project: Cassandra Issue Type: Bug Components: Core Reporter: Benedict Assignee: Sylvain Lebresne Priority: Critical Fix For: 3.x, 2.1.x, 2.0.x, 2.2.x, 1.2.x Attachments: 0001-9486.patch LazilyCompactedRow initializes a ColumnIndex.Builder to use its RangeTombstone.Tracker, but it only calls update() with a RT argument, never an atom. The Tracker only ever _adds_ if it receives a RT, never removes. So all the RT ever seen for the partition (that have expired) remain in memory until the compaction completes. To make matters worse, this then forces a linear scan of all of these RT for each live cell we add, so this extra load hangs around for a long time, and compactions stall. This issue is biting one of our users badly (at least, it seems likely to be this issue), and there may be others. This user is not even making use of RT extensively themselves, only collections (presumably with a complete overwrite of the contents of the collection, resulting in a RT being generated). Probably the best solution is to make the RT addition itself remove any already present that are no longer helpful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9486) LazilyCompactedRow accumulates all expired RangeTombstones
[
https://issues.apache.org/jira/browse/CASSANDRA-9486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568827#comment-14568827
]
Benedict commented on CASSANDRA-9486:
-
LGTM. There's a double in (i.e. in in) in one of your comment
modifications. Also, {{insertBefore}} doesn't need to call {{next()}} for the
behaviour here, but not at all fussed since the semantics of an arbitrary
method labelled {{insertBefore}} might well want to do so.
LazilyCompactedRow accumulates all expired RangeTombstones
--
Key: CASSANDRA-9486
URL: https://issues.apache.org/jira/browse/CASSANDRA-9486
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Benedict
Assignee: Sylvain Lebresne
Priority: Critical
Fix For: 3.x, 2.1.x, 2.0.x, 2.2.x, 1.2.x
Attachments: 0001-9486.patch
LazilyCompactedRow initializes a ColumnIndex.Builder to use its
RangeTombstone.Tracker, but it only calls update() with a RT argument, never
an atom. The Tracker only ever _adds_ if it receives a RT, never removes. So
all the RT ever seen for the partition (that have expired) remain in memory
until the compaction completes. To make matters worse, this then forces a
linear scan of all of these RT for each live cell we add, so this extra load
hangs around for a long time, and compactions stall.
This issue is biting one of our users badly (at least, it seems likely to be
this issue), and there may be others. This user is not even making use of RT
extensively themselves, only collections (presumably with a complete
overwrite of the contents of the collection, resulting in a RT being
generated).
Probably the best solution is to make the RT addition itself remove any
already present that are no longer helpful.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8609) Remove depency of hadoop to internals (Cell/CellName)
[
https://issues.apache.org/jira/browse/CASSANDRA-8609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Tunnicliffe updated CASSANDRA-8609:
---
Summary: Remove depency of hadoop to internals (Cell/CellName) (was: t)
Remove depency of hadoop to internals (Cell/CellName)
-
Key: CASSANDRA-8609
URL: https://issues.apache.org/jira/browse/CASSANDRA-8609
Project: Cassandra
Issue Type: Bug
Reporter: Sylvain Lebresne
Assignee: Sam Tunnicliffe
Fix For: 2.2.0 rc1
Attachments: 8609-2.2-2.txt, 8609-2.2.txt,
CASSANDRA-8609-3.0-branch.txt
For some reason most of the Hadoop code (ColumnFamilyRecordReader,
CqlStorage, ...) uses the {{Cell}} and {{CellName}} classes. That dependency
is entirely artificial: all this code is really client code that communicate
with Cassandra over thrift/native protocol and there is thus no reason for it
to use internal classes. And in fact, thoses classes are used in a very crude
way, as a {{PairByteBuffer, ByteBuffer}} really.
But this dependency is really painful when we make changes to the internals.
Further, every time we do so, I believe we break some of those the APIs due
to the change. This has been painful for CASSANDRA-5417 and this is now
painful for CASSANDRA-8099. But while I somewhat hack over it in
CASSANDRA-5417, this was a mistake and we should have removed the depency
back then. So let do that now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7281) SELECT on tuple relations are broken for mixed ASC/DESC clustering order
[
https://issues.apache.org/jira/browse/CASSANDRA-7281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568757#comment-14568757
]
Marcin Szymaniuk commented on CASSANDRA-7281:
-
In 2.2 I cannot see classes/methods mentioned in your comment.
From quick look it seems to me that:
- MultiColumnRestriction.Slice#appendBoundTo is the place to implement the logic
- PrimaryKeyRestrictionSetTest is good place to create more of unit-style tests.
Can you please have a look and confirm?
SELECT on tuple relations are broken for mixed ASC/DESC clustering order
Key: CASSANDRA-7281
URL: https://issues.apache.org/jira/browse/CASSANDRA-7281
Project: Cassandra
Issue Type: Bug
Reporter: Sylvain Lebresne
Assignee: Marcin Szymaniuk
Fix For: 2.1.x
Attachments:
0001-CASSANDRA-7281-SELECT-on-tuple-relations-are-broken-.patch,
0001-CASSANDRA-7281-SELECT-on-tuple-relations-are-broken-v2.patch,
0001-CASSANDRA-7281-SELECT-on-tuple-relations-are-broken-v3.patch,
0001-CASSANDRA-7281-SELECT-on-tuple-relations-are-broken-v4.patch
As noted on
[CASSANDRA-6875|https://issues.apache.org/jira/browse/CASSANDRA-6875?focusedCommentId=13992153page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13992153],
the tuple notation is broken when the clustering order mixes ASC and DESC
directives because the range of data they describe don't correspond to a
single continuous slice internally. To copy the example from CASSANDRA-6875:
{noformat}
cqlsh:ks create table foo (a int, b int, c int, PRIMARY KEY (a, b, c)) WITH
CLUSTERING ORDER BY (b DESC, c ASC);
cqlsh:ks INSERT INTO foo (a, b, c) VALUES (0, 2, 0);
cqlsh:ks INSERT INTO foo (a, b, c) VALUES (0, 1, 0);
cqlsh:ks INSERT INTO foo (a, b, c) VALUES (0, 1, 1);
cqlsh:ks INSERT INTO foo (a, b, c) VALUES (0, 0, 0);
cqlsh:ks SELECT * FROM foo WHERE a=0;
a | b | c
---+---+---
0 | 2 | 0
0 | 1 | 0
0 | 1 | 1
0 | 0 | 0
(4 rows)
cqlsh:ks SELECT * FROM foo WHERE a=0 AND (b, c) (1, 0);
a | b | c
---+---+---
0 | 2 | 0
(1 rows)
{noformat}
The last query should really return {{(0, 2, 0)}} and {{(0, 1, 1)}}.
For that specific example we should generate 2 internal slices, but I believe
that with more clustering columns we may have more slices.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8609) t
[
https://issues.apache.org/jira/browse/CASSANDRA-8609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Tunnicliffe updated CASSANDRA-8609:
---
Summary: t (was: Remove depency of hadoop to internals (Cell/CellName))
t
-
Key: CASSANDRA-8609
URL: https://issues.apache.org/jira/browse/CASSANDRA-8609
Project: Cassandra
Issue Type: Bug
Reporter: Sylvain Lebresne
Assignee: Sam Tunnicliffe
Fix For: 2.2.0 rc1
Attachments: 8609-2.2-2.txt, 8609-2.2.txt,
CASSANDRA-8609-3.0-branch.txt
For some reason most of the Hadoop code (ColumnFamilyRecordReader,
CqlStorage, ...) uses the {{Cell}} and {{CellName}} classes. That dependency
is entirely artificial: all this code is really client code that communicate
with Cassandra over thrift/native protocol and there is thus no reason for it
to use internal classes. And in fact, thoses classes are used in a very crude
way, as a {{PairByteBuffer, ByteBuffer}} really.
But this dependency is really painful when we make changes to the internals.
Further, every time we do so, I believe we break some of those the APIs due
to the change. This has been painful for CASSANDRA-5417 and this is now
painful for CASSANDRA-8099. But while I somewhat hack over it in
CASSANDRA-5417, this was a mistake and we should have removed the depency
back then. So let do that now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-8656) long-test LongLeveledCompactionStrategyTest flaps in 2.0
[
https://issues.apache.org/jira/browse/CASSANDRA-8656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania reassigned CASSANDRA-8656:
---
Assignee: Stefania
long-test LongLeveledCompactionStrategyTest flaps in 2.0
Key: CASSANDRA-8656
URL: https://issues.apache.org/jira/browse/CASSANDRA-8656
Project: Cassandra
Issue Type: Test
Components: Tests
Reporter: Michael Shuler
Assignee: Stefania
Priority: Minor
Labels: test-failure
Fix For: 2.0.x
Attachments: system.log
LongLeveledCompactionStrategyTest periodically fails with:
{noformat}
[junit] Testsuite:
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest
[junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
54.412 sec
[junit]
[junit] Testcase:
testParallelLeveledCompaction(org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest):
Caused an ERROR
[junit] java.util.concurrent.ExecutionException:
java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) =
current key DecoratedKey(313236, 313236) writing into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] java.lang.RuntimeException:
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last
written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236,
313236) writing into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:413)
[junit] at
org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:402)
[junit] at
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest.testParallelLeveledCompaction(LongLeveledCompactionStrategyTest.java:97)
[junit] Caused by: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) =
current key DecoratedKey(313236, 313236) writing into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] at java.util.concurrent.FutureTask.report(FutureTask.java:122)
[junit] at java.util.concurrent.FutureTask.get(FutureTask.java:188)
[junit] at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:409)
[junit] Caused by: java.lang.RuntimeException: Last written key
DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing
into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
[junit] at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
[junit] at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167)
[junit] at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
[junit] at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
[junit] at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
[junit] at
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest$1.run(LongLeveledCompactionStrategyTest.java:87)
[junit] at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[junit] at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit] at java.lang.Thread.run(Thread.java:745)
[junit]
[junit]
[junit] Test
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest FAILED
{noformat}
I would guess the failure is 10-20% of the time, looping over the test
repeatedly.
On the 2.1 branch, the failure is different, so perhaps this could also be
updated.
{noformat}
[junit] Testsuite:
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest
[junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
7.04 sec
[junit]
[junit] Testcase:
testParallelLeveledCompaction(org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest):
Caused an ERROR
[junit] org.apache.cassandra.db.compaction.WrappingCompactionStrategy
cannot be cast to org.apache.cassandra.db.compaction.LeveledCompactionStrategy
[junit] java.lang.ClassCastException:
[jira] [Updated] (CASSANDRA-9529) Standardize quoting in Unix scripts
[
https://issues.apache.org/jira/browse/CASSANDRA-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania updated CASSANDRA-9529:
Description: {{$CLASSPATH}} and {{$JAVA}} are quoted in some scripts and
not quoted in other scripts. Since it's best practice to use quotes, we should
update the scripts without quotes around these variables. (was: {{$CLASSPATH}}
and {{$JAVA}} are quoted in some scripts and not quoted in other scripts. Since
it's best practice to use quotes, we should update the scripts were the quotes
around these variables are missing.)
Standardize quoting in Unix scripts
---
Key: CASSANDRA-9529
URL: https://issues.apache.org/jira/browse/CASSANDRA-9529
Project: Cassandra
Issue Type: Improvement
Environment: All Unix based systems.
Reporter: Stefania
Assignee: Stefania
Priority: Minor
Fix For: 3.x, 2.1.x, 2.2.x, 2.0.16
{{$CLASSPATH}} and {{$JAVA}} are quoted in some scripts and not quoted in
other scripts. Since it's best practice to use quotes, we should update the
scripts without quotes around these variables.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9533) Make batch commitlog mode easier to tune
[
https://issues.apache.org/jira/browse/CASSANDRA-9533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569489#comment-14569489
]
Jonathan Ellis commented on CASSANDRA-9533:
---
It's pretty clear that the new behavior is worse for most users. I thought we
did it that way because we don't have a queue of operations to peek into
anymore, so it's difficult to provide the old behavior of stop sleeping when
the queue is empty.
Strictly speaking the new behavior matches the comment *less* well since the
comment said {{it will wait *up to*}} the window, which is in fact what it did.
The new behavior waits *at least* the window.
Make batch commitlog mode easier to tune
Key: CASSANDRA-9533
URL: https://issues.apache.org/jira/browse/CASSANDRA-9533
Project: Cassandra
Issue Type: Improvement
Reporter: Jonathan Ellis
Assignee: Benedict
Fix For: 3.x
As discussed in CASSANDRA-9504, 2.1 changed commitlog_sync_batch_window_in_ms
from a maximum time to wait between fsync to the minimum time, so one must be
very careful to keep it small enough that most writers aren't kept waiting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5543) Ant issues when building gen-cql2-grammar
[
https://issues.apache.org/jira/browse/CASSANDRA-5543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569605#comment-14569605
]
Antoine Levy-Lambert commented on CASSANDRA-5543:
-
do you have a stack trace of when the java command invoked by ant is failing ?
maybe running ant -verbose could give more output.
can you give a pointer to the ant build file you are running ?
Ant issues when building gen-cql2-grammar
-
Key: CASSANDRA-5543
URL: https://issues.apache.org/jira/browse/CASSANDRA-5543
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.2.3
Reporter: Joaquin Casares
Assignee: Dave Brosius
Priority: Trivial
Below are the commands and outputs that were returned.
The first `ant` command fails on gen-cql2-grammar, but if I don't run `ant
realclean` then it works fine after a second pass.
{CODE}
ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant realclean
Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml
clean:
[delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build/test
[delete] Deleting directory
/home/ubuntu/.ccm/repository/1.2.3/build/classes
[delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/src/gen-java
[delete] Deleting: /home/ubuntu/.ccm/repository/1.2.3/build/internode.avpr
realclean:
[delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build
BUILD SUCCESSFUL
Total time: 0 seconds
{CODE}
{CODE}
ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant
Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml
maven-ant-tasks-localrepo:
maven-ant-tasks-download:
[echo] Downloading Maven ANT Tasks...
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build
[get] Getting:
http://repo2.maven.org/maven2/org/apache/maven/maven-ant-tasks/2.1.3/maven-ant-tasks-2.1.3.jar
[get] To:
/home/ubuntu/.ccm/repository/1.2.3/build/maven-ant-tasks-2.1.3.jar
maven-ant-tasks-init:
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/lib
maven-declare-dependencies:
maven-ant-tasks-retrieve-build:
[artifact:dependencies] Downloading: asm/asm/3.2/asm-3.2-sources.jar from
repository central at http://repo1.maven.org/maven2
[artifact:dependencies] [INFO] Unable to find resource
'hsqldb:hsqldb:java-source:sources:1.8.0.10' in repository java.net2
(http://download.java.net/maven/2)
[artifact:dependencies] Building ant file:
/home/ubuntu/.ccm/repository/1.2.3/build/build-dependencies.xml
[copy] Copying 45 files to
/home/ubuntu/.ccm/repository/1.2.3/build/lib/jars
[copy] Copying 35 files to
/home/ubuntu/.ccm/repository/1.2.3/build/lib/sources
init:
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/classes/main
[mkdir] Created dir:
/home/ubuntu/.ccm/repository/1.2.3/build/classes/thrift
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/lib
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/classes
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/src/gen-java
check-avro-generate:
avro-interface-generate-internode:
[echo] Generating Avro internode code...
avro-generate:
build-subprojects:
check-gen-cli-grammar:
gen-cli-grammar:
[echo] Building Grammar
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cli/Cli.g
check-gen-cql2-grammar:
gen-cql2-grammar:
[echo] Building Grammar
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g
...
[java] warning(200):
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20:
Decision can match input such as IDENT using multiple alternatives: 1, 2
[java] As a result, alternative(s) 2 were disabled for that input
[java] warning(200):
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20:
Decision can match input such as K_KEY using multiple alternatives: 1, 2
[java] As a result, alternative(s) 2 were disabled for that input
[java] warning(200):
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20:
Decision can match input such as QMARK using multiple alternatives: 1, 2
[java] As a result, alternative(s) 2 were disabled for that input
[java] warning(200):
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20:
Decision can match input such as FLOAT using multiple alternatives: 1, 2
[java] As a result, alternative(s) 2 were disabled for that input
[java] warning(200):
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20:
Decision can match input such as STRING_LITERAL using multiple
alternatives: 1, 2
[java] As a result, alternative(s)
[jira] [Commented] (CASSANDRA-9536) The failure detector becomes more sensitive when the network is flakey
[
https://issues.apache.org/jira/browse/CASSANDRA-9536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569629#comment-14569629
]
Ron Kuris commented on CASSANDRA-9536:
--
There are a few possible fixes to this, each has their pros and cons:
# Don't record values less than 1 second; these are probably retransmits. This
will prevent the phi average from being dropped below 1, which means that all
phi can do is make the system less sensitive. The disadvantage to this is that
flakey networks will still not receive any phi bonuses.
# Whenever a packet is received longer than MAX_INTERVAL_IN_NANOS, just record
an entry with MAX_INTERVAL_IN_NANOS instead of just dropping it.
# Do both of the above
I am leaning toward both, since this means that flakier networks will raise the
phi average closer to 2, giving them twice the resilience before being marked
down.
The failure detector becomes more sensitive when the network is flakey
--
Key: CASSANDRA-9536
URL: https://issues.apache.org/jira/browse/CASSANDRA-9536
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Ron Kuris
Priority: Minor
I added considerable instrumentation into the failure detector, and then
blocked port 7000 for a random 5-6 second interval, then resumed traffic for
the same amount of time, with a script like:
{code}while :
do
iptables -A INPUT -p tcp --destination-port 7000 -j DROP
v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)]
s=5.${v:1:2}${v:4:3}
echo offline for $s
sleep $s
iptables -F
v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)]
s=5.${v:1:2}${v:4:3}
echo online for $s
sleep $s
done{code}
When I do this, I watch the values being reported to the FailureDetector. The
median actually goes down, as low as 850ms. The reason is that the very slow
packets are not recorded (they exceed MAX_INTERVAL_IN_NANO which is 2
seconds) and the retransmitted packets arrive very quickly in succession,
lowering the overall average.
Once the average is lowered, the node becomes much more sensitive to shorter
outages. If you run this code for a while, the average drops down to 800ms or
less, which means that the node will go down 20% quicker than expected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8609) Remove depency of hadoop to internals (Cell/CellName)
[
https://issues.apache.org/jira/browse/CASSANDRA-8609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569536#comment-14569536
]
Sam Tunnicliffe commented on CASSANDRA-8609:
Not just yet I'm afraid, there are still some dependencies on internals like
{{IPartitioner}}, {{AbstractType}} in determining splits.
Ideally, the Hadoop integration code shouldn't have any dependencies beyond the
cql driver and thrift (and client-utils). For that to happen though, we'll need
to duplicate a few internal things to enable the split sizing done entirely
client side along the lines of
[SPARKC-94|https://datastax-oss.atlassian.net/browse/SPARKC-94]). Maybe that
should be done as part of CASSANDRA-9353.
There are also some dependencies in {{CqlBulkRecordWriter}} ({{CFMetaData}},
{{Config}}, {{DatabaseDescriptor}}) which can't easily be removed yet.
Remove depency of hadoop to internals (Cell/CellName)
-
Key: CASSANDRA-8609
URL: https://issues.apache.org/jira/browse/CASSANDRA-8609
Project: Cassandra
Issue Type: Bug
Reporter: Sylvain Lebresne
Assignee: Sam Tunnicliffe
Fix For: 2.2.0 rc1
Attachments: 8609-2.2-2.txt, 8609-2.2.txt,
CASSANDRA-8609-3.0-branch.txt
For some reason most of the Hadoop code (ColumnFamilyRecordReader,
CqlStorage, ...) uses the {{Cell}} and {{CellName}} classes. That dependency
is entirely artificial: all this code is really client code that communicate
with Cassandra over thrift/native protocol and there is thus no reason for it
to use internal classes. And in fact, thoses classes are used in a very crude
way, as a {{PairByteBuffer, ByteBuffer}} really.
But this dependency is really painful when we make changes to the internals.
Further, every time we do so, I believe we break some of those the APIs due
to the change. This has been painful for CASSANDRA-5417 and this is now
painful for CASSANDRA-8099. But while I somewhat hack over it in
CASSANDRA-5417, this was a mistake and we should have removed the depency
back then. So let do that now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-9535) KeyCache code should handle assertion errors from corrupt keycache better
Philip Thompson created CASSANDRA-9535:
--
Summary: KeyCache code should handle assertion errors from corrupt
keycache better
Key: CASSANDRA-9535
URL: https://issues.apache.org/jira/browse/CASSANDRA-9535
Project: Cassandra
Issue Type: Bug
Reporter: Philip Thompson
Fix For: 3.x, 2.2.x
While running 2.2.0-beta1, a user ran into the following exception from a
corrupt keycache file:
{code}
ERROR [main] 2015-05-30 10:31:18,603 CassandraDaemon.java:571 - Exception
encountered during startup
java.lang.AssertionError: null
at
org.apache.cassandra.db.RowIndexEntry$IndexedEntry.init(RowIndexEntry.java:202)
~[main/:na]
at
org.apache.cassandra.db.RowIndexEntry$IndexedEntry.init(RowIndexEntry.java:190)
~[main/:na]
at
org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:144)
~[main/:na]
at
org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:446)
~[main/:na]
at
org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:142)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.init(ColumnFamilyStore.java:328)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:455)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
~[main/:na]
at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:324)
~[main/:na]
at org.apache.cassandra.db.Keyspace.init(Keyspace.java:275)
~[main/:na]
at org.apache.cassandra.db.Keyspace.open(Keyspace.java:117) ~[main/:na]
at org.apache.cassandra.db.Keyspace.open(Keyspace.java:94) ~[main/:na]
at
org.apache.cassandra.service.StartupChecks$8.execute(StartupChecks.java:281)
~[main/:na]
at
org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:103)
~[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:128)
[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:451)
[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:558)
[main/:na]
{code}
This is from the failed assert of {{columnsIndex.size() 1}} at
{code}
private IndexedEntry(long position, DeletionTime deletionTime,
ListIndexHelper.IndexInfo columnsIndex)
{
super(position);
assert deletionTime != null;
assert columnsIndex != null columnsIndex.size() 1;
this.deletionTime = deletionTime;
this.columnsIndex = columnsIndex;
}
{code}
Discussing with [~thobbs], he felt that C* could do a better job of handling
this, even though the root issue was a corrupt file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9071) CQLSSTableWriter gives java.lang.AssertionError: Empty partition
[ https://issues.apache.org/jira/browse/CASSANDRA-9071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569561#comment-14569561 ] Benedict commented on CASSANDRA-9071: - Heh. Thanks for pointing that out! I've pushed a fix and unit test [here|https://github.com/belliottsmith/cassandra/tree/9051] CQLSSTableWriter gives java.lang.AssertionError: Empty partition Key: CASSANDRA-9071 URL: https://issues.apache.org/jira/browse/CASSANDRA-9071 Project: Cassandra Issue Type: Bug Components: Tools Environment: java 7 / 8 cassandra 2.1.3 snapshot build locally with last commit https://github.com/apache/cassandra/commit/6ee4b0989d9a3ae3e704918622024fa57fdf63e7 macos Yosemite 10.10.2 Reporter: Ajit Joglekar Assignee: Fabien Rousseau Fix For: 2.1.x Attachments: EmailWriter.java, Screen Shot 2015-04-15 at 11.14.40 PM.png, cassandra-2.1-9071.txt, data.csv.gz I am always getting the following error: Exception in thread main java.lang.AssertionError: Empty partition at org.apache.cassandra.io.sstable.SSTableSimpleUnsortedWriter$DiskWriter.run(SSTableSimpleUnsortedWriter.java:228) It happens at a certain point that seems to be repeatable. Only issue is I am converting 400 million records into multiple SSTables and creating small test is a challenge Last comment from Benedict looks relevant here https://issues.apache.org/jira/browse/CASSANDRA-8619 Is there a work around, quick fix, fix that I can try out locally? Thanks, -Ajit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9533) Make batch commitlog mode easier to tune
[
https://issues.apache.org/jira/browse/CASSANDRA-9533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569535#comment-14569535
]
Benedict commented on CASSANDRA-9533:
-
bq. matches the comment *less* well
You're being a bit selective in which parts you bold :)
it will wait up to implies _it will wait_ - which it would not, at all. The
reference to Postgres' behaviour also indicates it will actually wait that
period (although with separate sibling requirements, and on a microsecond time
horizon which is much more sensible). Further, our docs say To avoid syncing
after every write, Cassandra groups the mutations into batches and *syncs
every* {{commitlog_batch_window_in_ms.}} Not at least that often, but -
implcitly - exactly that often, as we do now.
bq. I thought we did it that way because we don't have a queue of operations
to peek into anymore, so it's difficult to provide the old behavior of stop
sleeping when the queue is empty.
No, unfortunately the best I can find of the etymology of this change is some
offline discussion between Brandon, Jeremiah and myself, which occurred 2-3
months after commit:
{quote}
Benedict Elliott Smith so, just figured out why that CL unit test @yukim
found went from hero to zero in 2.1
Benedict Elliott Smith SHANP on IRC has found the old Batch CL code doesn't
behave in the same way
Benedict Elliott Smith the window only serves as a maximum for buffering
Benedict Elliott Smith so if you get one record arrive, it will immediately
sync
Benedict Elliott Smith in 2.1 this changed. i'm not sure if I got the wrong
end of the stick (and it seems everyone else who's been discussing it since,
maybe), or if this is a mistake that's been present all along
Benedict Elliott Smith but we should probably decide which behaviour we want
to go with in 2.1
dr driftxI don't think it was a mistake, exactly, I've explained it
lots of times in training that way (pre-2.1 behavior)
Benedict Elliott Smith right
Benedict Elliott Smith so question is: did I just misunderstand, or did
somebody tell me to implement it this way? and ignoring the answer, do we want
to restore the old behaviour?
Benedict Elliott Smith To avoid syncing after every write, Cassandra groups
the mutations into batches and syncs every {{commitlog_batch_window_in_ms. }}
Benedict Elliott Smith anyway, i'm easy, and heading to bed. if nobody says
anything i'll leave it how it is :-)
ZJeremiah D Jordan As long as the window time is the max time between
syncs and writes block until they sync. I think your code is fine. As that is
how we have documented it working all along
Benedict Elliott Smith @jd my version has it as the the exact time between
syncs
{quote}
So it is not at all clear why it happened. My recollection was that we
discussed it and decided to normalise on the doc behaviour, but I cannot find a
reference to that, so it is possible I simply implemented it how the docs
described it, and you reviewed it with the same lens. Either way, it was
decided to let the change stand due to it matching the doc descriptions and the
lack of further feedback.
We can certainly wait for the queue to empty, by issuing Barriers on the
OpOrder; we already issue one and wait for any active at that moment to
complete, which is probably good enough. If we want to wait until _none_ can be
active, we can just (on completion of the barrier), check if there are any now
running and issue another if there are.
Make batch commitlog mode easier to tune
Key: CASSANDRA-9533
URL: https://issues.apache.org/jira/browse/CASSANDRA-9533
Project: Cassandra
Issue Type: Improvement
Reporter: Jonathan Ellis
Assignee: Benedict
Fix For: 3.x
As discussed in CASSANDRA-9504, 2.1 changed commitlog_sync_batch_window_in_ms
from a maximum time to wait between fsync to the minimum time, so one must be
very careful to keep it small enough that most writers aren't kept waiting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9533) Make batch commitlog mode easier to tune
[ https://issues.apache.org/jira/browse/CASSANDRA-9533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569552#comment-14569552 ] Jonathan Ellis commented on CASSANDRA-9533: --- bq. We can certainly wait for the queue to empty, by issuing Barriers on the OpOrder; we already issue one and wait for any active at that moment to complete, which is probably good enough. WFM. Make batch commitlog mode easier to tune Key: CASSANDRA-9533 URL: https://issues.apache.org/jira/browse/CASSANDRA-9533 Project: Cassandra Issue Type: Improvement Reporter: Jonathan Ellis Assignee: Benedict Fix For: 3.x As discussed in CASSANDRA-9504, 2.1 changed commitlog_sync_batch_window_in_ms from a maximum time to wait between fsync to the minimum time, so one must be very careful to keep it small enough that most writers aren't kept waiting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9535) KeyCache code should handle assertion errors from corrupt keycache better
[
https://issues.apache.org/jira/browse/CASSANDRA-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philip Thompson updated CASSANDRA-9535:
---
Issue Type: Improvement (was: New Feature)
KeyCache code should handle assertion errors from corrupt keycache better
-
Key: CASSANDRA-9535
URL: https://issues.apache.org/jira/browse/CASSANDRA-9535
Project: Cassandra
Issue Type: Improvement
Reporter: Philip Thompson
Fix For: 3.x, 2.2.x
While running 2.2.0-beta1, a user ran into the following exception from a
corrupt keycache file:
{code}
ERROR [main] 2015-05-30 10:31:18,603 CassandraDaemon.java:571 - Exception
encountered during startup
java.lang.AssertionError: null
at
org.apache.cassandra.db.RowIndexEntry$IndexedEntry.init(RowIndexEntry.java:202)
~[main/:na]
at
org.apache.cassandra.db.RowIndexEntry$IndexedEntry.init(RowIndexEntry.java:190)
~[main/:na]
at
org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:144)
~[main/:na]
at
org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:446)
~[main/:na]
at
org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:142)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.init(ColumnFamilyStore.java:328)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:455)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
~[main/:na]
at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:324)
~[main/:na]
at org.apache.cassandra.db.Keyspace.init(Keyspace.java:275)
~[main/:na]
at org.apache.cassandra.db.Keyspace.open(Keyspace.java:117)
~[main/:na]
at org.apache.cassandra.db.Keyspace.open(Keyspace.java:94) ~[main/:na]
at
org.apache.cassandra.service.StartupChecks$8.execute(StartupChecks.java:281)
~[main/:na]
at
org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:103)
~[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:128)
[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:451)
[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:558)
[main/:na]
{code}
This is from the failed assert of {{columnsIndex.size() 1}} at
{code}
private IndexedEntry(long position, DeletionTime deletionTime,
ListIndexHelper.IndexInfo columnsIndex)
{
super(position);
assert deletionTime != null;
assert columnsIndex != null columnsIndex.size() 1;
this.deletionTime = deletionTime;
this.columnsIndex = columnsIndex;
}
{code}
Discussing with [~thobbs], he felt that C* could do a better job of handling
this, even though the root issue was a corrupt file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9535) KeyCache code should handle assertion errors from corrupt keycache better
[
https://issues.apache.org/jira/browse/CASSANDRA-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philip Thompson updated CASSANDRA-9535:
---
Issue Type: New Feature (was: Bug)
KeyCache code should handle assertion errors from corrupt keycache better
-
Key: CASSANDRA-9535
URL: https://issues.apache.org/jira/browse/CASSANDRA-9535
Project: Cassandra
Issue Type: New Feature
Reporter: Philip Thompson
Fix For: 3.x, 2.2.x
While running 2.2.0-beta1, a user ran into the following exception from a
corrupt keycache file:
{code}
ERROR [main] 2015-05-30 10:31:18,603 CassandraDaemon.java:571 - Exception
encountered during startup
java.lang.AssertionError: null
at
org.apache.cassandra.db.RowIndexEntry$IndexedEntry.init(RowIndexEntry.java:202)
~[main/:na]
at
org.apache.cassandra.db.RowIndexEntry$IndexedEntry.init(RowIndexEntry.java:190)
~[main/:na]
at
org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:144)
~[main/:na]
at
org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:446)
~[main/:na]
at
org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:142)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.init(ColumnFamilyStore.java:328)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:455)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
~[main/:na]
at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:324)
~[main/:na]
at org.apache.cassandra.db.Keyspace.init(Keyspace.java:275)
~[main/:na]
at org.apache.cassandra.db.Keyspace.open(Keyspace.java:117)
~[main/:na]
at org.apache.cassandra.db.Keyspace.open(Keyspace.java:94) ~[main/:na]
at
org.apache.cassandra.service.StartupChecks$8.execute(StartupChecks.java:281)
~[main/:na]
at
org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:103)
~[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:128)
[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:451)
[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:558)
[main/:na]
{code}
This is from the failed assert of {{columnsIndex.size() 1}} at
{code}
private IndexedEntry(long position, DeletionTime deletionTime,
ListIndexHelper.IndexInfo columnsIndex)
{
super(position);
assert deletionTime != null;
assert columnsIndex != null columnsIndex.size() 1;
this.deletionTime = deletionTime;
this.columnsIndex = columnsIndex;
}
{code}
Discussing with [~thobbs], he felt that C* could do a better job of handling
this, even though the root issue was a corrupt file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9536) The failure detector becomes more sensitive when the network is flakey
[
https://issues.apache.org/jira/browse/CASSANDRA-9536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis updated CASSANDRA-9536:
--
Priority: Minor (was: Major)
The failure detector becomes more sensitive when the network is flakey
--
Key: CASSANDRA-9536
URL: https://issues.apache.org/jira/browse/CASSANDRA-9536
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Ron Kuris
Priority: Minor
I added considerable instrumentation into the failure detector, and then
blocked port 7000 for a random 5-6 second interval, then resumed traffic for
the same amount of time, with a script like:
{code}while :
do
iptables -A INPUT -p tcp --destination-port 7000 -j DROP
v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)]
s=5.${v:1:2}${v:4:3}
echo offline for $s
sleep $s
iptables -F
v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)]
s=5.${v:1:2}${v:4:3}
echo online for $s
sleep $s
done{code}
When I do this, I watch the values being reported to the FailureDetector. The
median actually goes down, as low as 850ms. The reason is that the very slow
packets are not recorded (they exceed MAX_INTERVAL_IN_NANO which is 2
seconds) and the retransmitted packets arrive very quickly in succession,
lowering the overall average.
Once the average is lowered, the node becomes much more sensitive to shorter
outages. If you run this code for a while, the average drops down to 800ms or
less, which means that the node will go down 20% quicker than expected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9536) The failure detector becomes more sensitive when the network is flakey
[
https://issues.apache.org/jira/browse/CASSANDRA-9536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis updated CASSANDRA-9536:
--
Issue Type: Improvement (was: Bug)
The failure detector becomes more sensitive when the network is flakey
--
Key: CASSANDRA-9536
URL: https://issues.apache.org/jira/browse/CASSANDRA-9536
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Ron Kuris
Priority: Minor
I added considerable instrumentation into the failure detector, and then
blocked port 7000 for a random 5-6 second interval, then resumed traffic for
the same amount of time, with a script like:
{code}while :
do
iptables -A INPUT -p tcp --destination-port 7000 -j DROP
v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)]
s=5.${v:1:2}${v:4:3}
echo offline for $s
sleep $s
iptables -F
v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)]
s=5.${v:1:2}${v:4:3}
echo online for $s
sleep $s
done{code}
When I do this, I watch the values being reported to the FailureDetector. The
median actually goes down, as low as 850ms. The reason is that the very slow
packets are not recorded (they exceed MAX_INTERVAL_IN_NANO which is 2
seconds) and the retransmitted packets arrive very quickly in succession,
lowering the overall average.
Once the average is lowered, the node becomes much more sensitive to shorter
outages. If you run this code for a while, the average drops down to 800ms or
less, which means that the node will go down 20% quicker than expected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-9536) The failure detector becomes more sensitive when the network is flakey
Ron Kuris created CASSANDRA-9536:
Summary: The failure detector becomes more sensitive when the
network is flakey
Key: CASSANDRA-9536
URL: https://issues.apache.org/jira/browse/CASSANDRA-9536
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Ron Kuris
I added considerable instrumentation into the failure detector, and then
blocked port 7000 for a random 5-6 second interval, then resumed traffic for
the same amount of time, with a script like:
{code}while :
do
iptables -A INPUT -p tcp --destination-port 7000 -j DROP
v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)]
s=5.${v:1:2}${v:4:3}
echo offline for $s
sleep $s
iptables -F
v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)]
s=5.${v:1:2}${v:4:3}
echo online for $s
sleep $s
done{code}
When I do this, I watch the values being reported to the FailureDetector. The
median actually goes down, as low as 850ms. The reason is that the very slow
packets are not recorded (they exceed MAX_INTERVAL_IN_NANO which is 2 seconds)
and the retransmitted packets arrive very quickly in succession, lowering the
overall average.
Once the average is lowered, the node becomes much more sensitive to shorter
outages. If you run this code for a while, the average drops down to 800ms or
less, which means that the node will go down 20% quicker than expected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9536) The failure detector becomes more sensitive when the network is flakey
[
https://issues.apache.org/jira/browse/CASSANDRA-9536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569642#comment-14569642
]
Ron Kuris commented on CASSANDRA-9536:
--
Some additional notes from my research on how this code works:
FailureDetector's report() is called each time the gossiper receives a message
from that node. The message should arrive every second (this is not tunable).
The failure detector records the time since the last message, up to 1000
samples, which will take about 16 minutes, 40 seconds. It does not record the
time if the message was sent more than 2 seconds ago.
In the gossiper's status check, it calls FailureDetector's interpret() method.
This calculates phi, which is the time
since the last message, divided by the average of the collected samples. It
then divides by a constant, PHI_FACTOR, which is about 0.434, then compares to
phi_convict_threshold. If the adjusted phi is greater than the threshold, the
node is marked down.
If the average collected samples drops below 1, this increases phi.
I ran several additional tests, including changing the packet dropping time to
0-1 seconds, which severely lowers the average and causes the node to flip flop
UP/DOWN.
I'm very open to helping to fix this, but I'm afraid it might require a
different implementation of the FailureDetector. Perhaps I should make it
pluggable, where we have a new implementation of IFailureDetector? I'm open to
suggestions.
The failure detector becomes more sensitive when the network is flakey
--
Key: CASSANDRA-9536
URL: https://issues.apache.org/jira/browse/CASSANDRA-9536
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Ron Kuris
Priority: Minor
I added considerable instrumentation into the failure detector, and then
blocked port 7000 for a random 5-6 second interval, then resumed traffic for
the same amount of time, with a script like:
{code}while :
do
iptables -A INPUT -p tcp --destination-port 7000 -j DROP
v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)]
s=5.${v:1:2}${v:4:3}
echo offline for $s
sleep $s
iptables -F
v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)]
s=5.${v:1:2}${v:4:3}
echo online for $s
sleep $s
done{code}
When I do this, I watch the values being reported to the FailureDetector. The
median actually goes down, as low as 850ms. The reason is that the very slow
packets are not recorded (they exceed MAX_INTERVAL_IN_NANO which is 2
seconds) and the retransmitted packets arrive very quickly in succession,
lowering the overall average.
Once the average is lowered, the node becomes much more sensitive to shorter
outages. If you run this code for a while, the average drops down to 800ms or
less, which means that the node will go down 20% quicker than expected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9527) Cannot create secondary index on a table WITH COMPACT STORAGE
[ https://issues.apache.org/jira/browse/CASSANDRA-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569645#comment-14569645 ] Benjamin Lerer commented on CASSANDRA-9527: --- [~thobbs] can you review? Cannot create secondary index on a table WITH COMPACT STORAGE - Key: CASSANDRA-9527 URL: https://issues.apache.org/jira/browse/CASSANDRA-9527 Project: Cassandra Issue Type: Bug Components: Core Environment: cassandra v2.0.13 Reporter: fuggy_yama Assignee: Benjamin Lerer Priority: Minor Fix For: 2.0.x Attachments: 9527.txt In CASSANDRA-8156 it is said that secondary indexes are not allowed on clustering columns in COMPACT tables. However, I found that it is not possible to create a secondary index on the value column in a COMPACT table: CREATE TABLE t ( a INT, b INT, c INT, PRIMARY KEY (a, b) ) WITH COMPACT STORAGE; CREATE INDEX ON t (c); *Bad Request: Secondary indexes are not supported on PRIMARY KEY columns in COMPACT STORAGE tables* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9536) The failure detector becomes more sensitive when the network is flakey
[
https://issues.apache.org/jira/browse/CASSANDRA-9536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Kuris updated CASSANDRA-9536:
-
Description:
I added considerable instrumentation into the failure detector, and then
blocked port 7000 for a random 5-6 second interval, then resumed traffic for
the same amount of time, with a script like:
{code}while :
do
iptables -A INPUT -p tcp --destination-port 7000 -j DROP
v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)]
s=5.${v:1:2}${v:4:3}
echo offline for $s
sleep $s
iptables -F
v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)]
s=5.${v:1:2}${v:4:3}
echo online for $s
sleep $s
done{code}
When I do this, I watch the values being reported to the FailureDetector. The
mean actually goes down, as low as 850ms. The reason is that the very slow
packets are not recorded (they exceed MAX_INTERVAL_IN_NANO which is 2 seconds)
and the retransmitted packets arrive very quickly in succession, lowering the
overall average.
Once the average is lowered, the node becomes much more sensitive to shorter
outages. If you run this code for a while, the average drops down to 800ms or
less, which means that the node will go down 20% quicker than expected.
was:
I added considerable instrumentation into the failure detector, and then
blocked port 7000 for a random 5-6 second interval, then resumed traffic for
the same amount of time, with a script like:
{code}while :
do
iptables -A INPUT -p tcp --destination-port 7000 -j DROP
v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)]
s=5.${v:1:2}${v:4:3}
echo offline for $s
sleep $s
iptables -F
v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)]
s=5.${v:1:2}${v:4:3}
echo online for $s
sleep $s
done{code}
When I do this, I watch the values being reported to the FailureDetector. The
median actually goes down, as low as 850ms. The reason is that the very slow
packets are not recorded (they exceed MAX_INTERVAL_IN_NANO which is 2 seconds)
and the retransmitted packets arrive very quickly in succession, lowering the
overall average.
Once the average is lowered, the node becomes much more sensitive to shorter
outages. If you run this code for a while, the average drops down to 800ms or
less, which means that the node will go down 20% quicker than expected.
The failure detector becomes more sensitive when the network is flakey
--
Key: CASSANDRA-9536
URL: https://issues.apache.org/jira/browse/CASSANDRA-9536
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Ron Kuris
Priority: Minor
I added considerable instrumentation into the failure detector, and then
blocked port 7000 for a random 5-6 second interval, then resumed traffic for
the same amount of time, with a script like:
{code}while :
do
iptables -A INPUT -p tcp --destination-port 7000 -j DROP
v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)]
s=5.${v:1:2}${v:4:3}
echo offline for $s
sleep $s
iptables -F
v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)]
s=5.${v:1:2}${v:4:3}
echo online for $s
sleep $s
done{code}
When I do this, I watch the values being reported to the FailureDetector. The
mean actually goes down, as low as 850ms. The reason is that the very slow
packets are not recorded (they exceed MAX_INTERVAL_IN_NANO which is 2
seconds) and the retransmitted packets arrive very quickly in succession,
lowering the overall average.
Once the average is lowered, the node becomes much more sensitive to shorter
outages. If you run this code for a while, the average drops down to 800ms or
less, which means that the node will go down 20% quicker than expected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-5543) Ant issues when building gen-cql2-grammar
[
https://issues.apache.org/jira/browse/CASSANDRA-5543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Lococo updated CASSANDRA-5543:
---
Attachment: 5543-ant.log
The ant build-file I'm using is the top-level build.xml, if I understand your
questions correctly.
The results of ant -verbose 5543-ant.log are attached.
The stderr also contains a stacktrace:
$ ant -verbose ant.log
BUILD FAILED
/home/ubuntu/cassandra/build.xml:222: Java returned: 1
at org.apache.tools.ant.taskdefs.Java.execute(Java.java:111)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:292)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:435)
at org.apache.tools.ant.Target.performTasks(Target.java:456)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1393)
at org.apache.tools.ant.Project.executeTarget(Project.java:1364)
at
org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
at org.apache.tools.ant.Project.executeTargets(Project.java:1248)
at org.apache.tools.ant.Main.runBuild(Main.java:851)
at org.apache.tools.ant.Main.startAnt(Main.java:235)
at org.apache.tools.ant.launch.Launcher.run(Launcher.java:280)
at org.apache.tools.ant.launch.Launcher.main(Launcher.java:109)
Total time: 20 seconds
Ant issues when building gen-cql2-grammar
-
Key: CASSANDRA-5543
URL: https://issues.apache.org/jira/browse/CASSANDRA-5543
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.2.3
Reporter: Joaquin Casares
Assignee: Dave Brosius
Priority: Trivial
Attachments: 5543-ant.log
Below are the commands and outputs that were returned.
The first `ant` command fails on gen-cql2-grammar, but if I don't run `ant
realclean` then it works fine after a second pass.
{CODE}
ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant realclean
Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml
clean:
[delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build/test
[delete] Deleting directory
/home/ubuntu/.ccm/repository/1.2.3/build/classes
[delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/src/gen-java
[delete] Deleting: /home/ubuntu/.ccm/repository/1.2.3/build/internode.avpr
realclean:
[delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build
BUILD SUCCESSFUL
Total time: 0 seconds
{CODE}
{CODE}
ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant
Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml
maven-ant-tasks-localrepo:
maven-ant-tasks-download:
[echo] Downloading Maven ANT Tasks...
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build
[get] Getting:
http://repo2.maven.org/maven2/org/apache/maven/maven-ant-tasks/2.1.3/maven-ant-tasks-2.1.3.jar
[get] To:
/home/ubuntu/.ccm/repository/1.2.3/build/maven-ant-tasks-2.1.3.jar
maven-ant-tasks-init:
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/lib
maven-declare-dependencies:
maven-ant-tasks-retrieve-build:
[artifact:dependencies] Downloading: asm/asm/3.2/asm-3.2-sources.jar from
repository central at http://repo1.maven.org/maven2
[artifact:dependencies] [INFO] Unable to find resource
'hsqldb:hsqldb:java-source:sources:1.8.0.10' in repository java.net2
(http://download.java.net/maven/2)
[artifact:dependencies] Building ant file:
/home/ubuntu/.ccm/repository/1.2.3/build/build-dependencies.xml
[copy] Copying 45 files to
/home/ubuntu/.ccm/repository/1.2.3/build/lib/jars
[copy] Copying 35 files to
/home/ubuntu/.ccm/repository/1.2.3/build/lib/sources
init:
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/classes/main
[mkdir] Created dir:
/home/ubuntu/.ccm/repository/1.2.3/build/classes/thrift
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/lib
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/classes
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/src/gen-java
check-avro-generate:
avro-interface-generate-internode:
[echo] Generating Avro internode code...
avro-generate:
build-subprojects:
check-gen-cli-grammar:
gen-cli-grammar:
[echo] Building Grammar
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cli/Cli.g
check-gen-cql2-grammar:
gen-cql2-grammar:
[echo] Building Grammar
[jira] [Commented] (CASSANDRA-5543) Ant issues when building gen-cql2-grammar
[
https://issues.apache.org/jira/browse/CASSANDRA-5543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569749#comment-14569749
]
Mike Lococo commented on CASSANDRA-5543:
I'm also happy to provide direct access to a test-box (a temporary aws box
created solely to test this issue) where this is happening if that's useful.
Reach out to me via email with your ssh public-key and I'll respond with the ip
address if that sounds helpful.
Ant issues when building gen-cql2-grammar
-
Key: CASSANDRA-5543
URL: https://issues.apache.org/jira/browse/CASSANDRA-5543
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.2.3
Reporter: Joaquin Casares
Assignee: Dave Brosius
Priority: Trivial
Attachments: 5543-ant.log
Below are the commands and outputs that were returned.
The first `ant` command fails on gen-cql2-grammar, but if I don't run `ant
realclean` then it works fine after a second pass.
{CODE}
ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant realclean
Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml
clean:
[delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build/test
[delete] Deleting directory
/home/ubuntu/.ccm/repository/1.2.3/build/classes
[delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/src/gen-java
[delete] Deleting: /home/ubuntu/.ccm/repository/1.2.3/build/internode.avpr
realclean:
[delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build
BUILD SUCCESSFUL
Total time: 0 seconds
{CODE}
{CODE}
ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant
Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml
maven-ant-tasks-localrepo:
maven-ant-tasks-download:
[echo] Downloading Maven ANT Tasks...
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build
[get] Getting:
http://repo2.maven.org/maven2/org/apache/maven/maven-ant-tasks/2.1.3/maven-ant-tasks-2.1.3.jar
[get] To:
/home/ubuntu/.ccm/repository/1.2.3/build/maven-ant-tasks-2.1.3.jar
maven-ant-tasks-init:
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/lib
maven-declare-dependencies:
maven-ant-tasks-retrieve-build:
[artifact:dependencies] Downloading: asm/asm/3.2/asm-3.2-sources.jar from
repository central at http://repo1.maven.org/maven2
[artifact:dependencies] [INFO] Unable to find resource
'hsqldb:hsqldb:java-source:sources:1.8.0.10' in repository java.net2
(http://download.java.net/maven/2)
[artifact:dependencies] Building ant file:
/home/ubuntu/.ccm/repository/1.2.3/build/build-dependencies.xml
[copy] Copying 45 files to
/home/ubuntu/.ccm/repository/1.2.3/build/lib/jars
[copy] Copying 35 files to
/home/ubuntu/.ccm/repository/1.2.3/build/lib/sources
init:
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/classes/main
[mkdir] Created dir:
/home/ubuntu/.ccm/repository/1.2.3/build/classes/thrift
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/lib
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/classes
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/src/gen-java
check-avro-generate:
avro-interface-generate-internode:
[echo] Generating Avro internode code...
avro-generate:
build-subprojects:
check-gen-cli-grammar:
gen-cli-grammar:
[echo] Building Grammar
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cli/Cli.g
check-gen-cql2-grammar:
gen-cql2-grammar:
[echo] Building Grammar
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g
...
[java] warning(200):
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20:
Decision can match input such as IDENT using multiple alternatives: 1, 2
[java] As a result, alternative(s) 2 were disabled for that input
[java] warning(200):
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20:
Decision can match input such as K_KEY using multiple alternatives: 1, 2
[java] As a result, alternative(s) 2 were disabled for that input
[java] warning(200):
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20:
Decision can match input such as QMARK using multiple alternatives: 1, 2
[java] As a result, alternative(s) 2 were disabled for that input
[java] warning(200):
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20:
Decision can match input such as FLOAT using multiple alternatives: 1, 2
[java] As a result, alternative(s) 2 were disabled for that input
[java] warning(200):
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20:
Decision can match input such as
[jira] [Commented] (CASSANDRA-9534) Parent repair data not always saved to system_distributed
[
https://issues.apache.org/jira/browse/CASSANDRA-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569762#comment-14569762
]
Jim Witschey commented on CASSANDRA-9534:
-
If [this dtest
failure|http://cassci.datastax.com/view/trunk/job/trunk_dtest/200/testReport/repair_test/TestRepairDataSystemTable/repair_table_test/]
is any indication, the regular repair history table is also not populated
sometimes.
Parent repair data not always saved to system_distributed
-
Key: CASSANDRA-9534
URL: https://issues.apache.org/jira/browse/CASSANDRA-9534
Project: Cassandra
Issue Type: Bug
Reporter: Jim Witschey
Assignee: Marcus Eriksson
Fix For: 2.2.0 rc1
About 1 out of 5 times, in a ccm-managed cluster, a repair doesn't write
anything to the {{system_distributed.parent_repair_history}} table. [This
dtest|https://github.com/riptano/cassandra-dtest/blob/master/repair_test.py#L296]
reproduces the error on a 5-node cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9531) NullPointerException logged when running MessagePayloadTest
[
https://issues.apache.org/jira/browse/CASSANDRA-9531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569684#comment-14569684
]
Benjamin Lerer commented on CASSANDRA-9531:
---
I tested the patch on my machine. It fixes the issue.
+1
Thanks
NullPointerException logged when running MessagePayloadTest
---
Key: CASSANDRA-9531
URL: https://issues.apache.org/jira/browse/CASSANDRA-9531
Project: Cassandra
Issue Type: Bug
Reporter: Benjamin Lerer
Assignee: T Jake Luciani
Priority: Minor
Fix For: 2.2.0 rc1
When I run the {{MessagePayloadTest}} I have the following error in my log
output:
{code}
juin 02, 2015 3:27:31 PM com.google.common.cache.LocalCache
processPendingNotifications
AVERTISSEMENT: Exception thrown by removal listener
java.lang.NullPointerException
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.getTotalBufferSize(CompressedRandomAccessReader.java:260)
at
org.apache.cassandra.service.FileCacheService$2.onRemoval(FileCacheService.java:105)
at
com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1954)
at
com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3457)
at
com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3433)
at
com.google.common.cache.LocalCache$Segment.remove(LocalCache.java:3104)
at com.google.common.cache.LocalCache.remove(LocalCache.java:4170)
at
com.google.common.cache.LocalCache$LocalManualCache.invalidate(LocalCache.java:4762)
at
org.apache.cassandra.service.FileCacheService.invalidate(FileCacheService.java:182)
at
org.apache.cassandra.io.util.PoolingSegmentedFile$Cleanup.tidy(PoolingSegmentedFile.java:53)
at
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Cleanup.tidy(CompressedPoolingSegmentedFile.java:77)
at
org.apache.cassandra.utils.concurrent.Ref$GlobalState.release(Ref.java:285)
at
org.apache.cassandra.utils.concurrent.Ref$State.ensureReleased(Ref.java:164)
at org.apache.cassandra.utils.concurrent.Ref.ensureReleased(Ref.java:89)
at
org.apache.cassandra.utils.concurrent.SharedCloseableImpl.close(SharedCloseableImpl.java:45)
at
org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier$1.run(SSTableReader.java:1937)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
The problem is due to the fact that when {{getTotalBufferSize}} is called the
segment has already been desallocated. The reason being that the segment has
been put twice in the queue of the {{CacheBucket}}.
I tracked down the problem to {{CompactionTask.runMayThrow}} which trigger 2
calls to the {{close}} method of {{BigTableScanner}}.
One here:
{code}
at
org.apache.cassandra.io.sstable.format.big.BigTableScanner.close(BigTableScanner.java:197)
at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy$ScannerList.close(AbstractCompactionStrategy.java:333)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:207)
{code}
and the other here:
{code}
at
org.apache.cassandra.io.sstable.format.big.BigTableScanner.close(BigTableScanner.java:197)
at org.apache.cassandra.utils.MergeIterator.close(MergeIterator.java:62)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:206)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5543) Ant issues when building gen-cql2-grammar
[
https://issues.apache.org/jira/browse/CASSANDRA-5543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569755#comment-14569755
]
Antoine Levy-Lambert commented on CASSANDRA-5543:
-
the interesting part in the attachment is that :
{code}
[java] error(208):
/home/ubuntu/cassandra/src/java/org/apache/cassandra/cql/Cql.g:654:1: The
following token definitions can never be matched because prior tokens match the
same input:
T__93,T__94,T__97,T__98,T__101,T__105,T__107,K_WITH,K_USING,K_USE,K_FIRST,K_COUNT,K_SET,K_APPLY,K_BATCH,K_TRUNCATE,K_IN,K_CREATE,K_KEYSPACE,K_COLUMNFAMILY,K_INDEX,K_ON,K_DROP,K_INTO,K_TIMESTAMP,K_TTL,K_ALTER,K_ADD,K_TYPE,RANGEOP,FLOAT,COMPIDENT,UUID,MULTILINE_COMMENT
{code}
I do not know antlr (I think this is some antlr generation). Someone should
study this Cql.g to see whether it indeed does not comply with antlr's
expectations and what to do about that.
The stack trace above is just saying something like ant was executing an
external java program with the java task, and the external java program failed
You could change the build file to make the java/ invocation not fail on
error but this might be hiding the problem.
Ant issues when building gen-cql2-grammar
-
Key: CASSANDRA-5543
URL: https://issues.apache.org/jira/browse/CASSANDRA-5543
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.2.3
Reporter: Joaquin Casares
Assignee: Dave Brosius
Priority: Trivial
Attachments: 5543-ant.log
Below are the commands and outputs that were returned.
The first `ant` command fails on gen-cql2-grammar, but if I don't run `ant
realclean` then it works fine after a second pass.
{CODE}
ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant realclean
Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml
clean:
[delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build/test
[delete] Deleting directory
/home/ubuntu/.ccm/repository/1.2.3/build/classes
[delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/src/gen-java
[delete] Deleting: /home/ubuntu/.ccm/repository/1.2.3/build/internode.avpr
realclean:
[delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build
BUILD SUCCESSFUL
Total time: 0 seconds
{CODE}
{CODE}
ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant
Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml
maven-ant-tasks-localrepo:
maven-ant-tasks-download:
[echo] Downloading Maven ANT Tasks...
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build
[get] Getting:
http://repo2.maven.org/maven2/org/apache/maven/maven-ant-tasks/2.1.3/maven-ant-tasks-2.1.3.jar
[get] To:
/home/ubuntu/.ccm/repository/1.2.3/build/maven-ant-tasks-2.1.3.jar
maven-ant-tasks-init:
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/lib
maven-declare-dependencies:
maven-ant-tasks-retrieve-build:
[artifact:dependencies] Downloading: asm/asm/3.2/asm-3.2-sources.jar from
repository central at http://repo1.maven.org/maven2
[artifact:dependencies] [INFO] Unable to find resource
'hsqldb:hsqldb:java-source:sources:1.8.0.10' in repository java.net2
(http://download.java.net/maven/2)
[artifact:dependencies] Building ant file:
/home/ubuntu/.ccm/repository/1.2.3/build/build-dependencies.xml
[copy] Copying 45 files to
/home/ubuntu/.ccm/repository/1.2.3/build/lib/jars
[copy] Copying 35 files to
/home/ubuntu/.ccm/repository/1.2.3/build/lib/sources
init:
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/classes/main
[mkdir] Created dir:
/home/ubuntu/.ccm/repository/1.2.3/build/classes/thrift
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/lib
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/classes
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/src/gen-java
check-avro-generate:
avro-interface-generate-internode:
[echo] Generating Avro internode code...
avro-generate:
build-subprojects:
check-gen-cli-grammar:
gen-cli-grammar:
[echo] Building Grammar
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cli/Cli.g
check-gen-cql2-grammar:
gen-cql2-grammar:
[echo] Building Grammar
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g
...
[java] warning(200):
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20:
Decision can match input such as IDENT using multiple alternatives: 1, 2
[java] As a result, alternative(s) 2 were disabled for that input
[java] warning(200):
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20:
Decision can match input such as K_KEY using multiple alternatives: 1, 2
[jira] [Commented] (CASSANDRA-5543) Ant issues when building gen-cql2-grammar
[
https://issues.apache.org/jira/browse/CASSANDRA-5543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569774#comment-14569774
]
Antoine Levy-Lambert commented on CASSANDRA-5543:
-
I would focus on the error happening the first time that one builds.
the second time that one builds antlr might be in no-op mode because it sees
output files.
in that case setting failonerror = false on the java task in the ant build file
might be an acceptable workaround.
but best to see what is the offending part in the grammar file.
Ant issues when building gen-cql2-grammar
-
Key: CASSANDRA-5543
URL: https://issues.apache.org/jira/browse/CASSANDRA-5543
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.2.3
Reporter: Joaquin Casares
Assignee: Dave Brosius
Priority: Trivial
Attachments: 5543-ant.log
Below are the commands and outputs that were returned.
The first `ant` command fails on gen-cql2-grammar, but if I don't run `ant
realclean` then it works fine after a second pass.
{CODE}
ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant realclean
Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml
clean:
[delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build/test
[delete] Deleting directory
/home/ubuntu/.ccm/repository/1.2.3/build/classes
[delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/src/gen-java
[delete] Deleting: /home/ubuntu/.ccm/repository/1.2.3/build/internode.avpr
realclean:
[delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build
BUILD SUCCESSFUL
Total time: 0 seconds
{CODE}
{CODE}
ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant
Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml
maven-ant-tasks-localrepo:
maven-ant-tasks-download:
[echo] Downloading Maven ANT Tasks...
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build
[get] Getting:
http://repo2.maven.org/maven2/org/apache/maven/maven-ant-tasks/2.1.3/maven-ant-tasks-2.1.3.jar
[get] To:
/home/ubuntu/.ccm/repository/1.2.3/build/maven-ant-tasks-2.1.3.jar
maven-ant-tasks-init:
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/lib
maven-declare-dependencies:
maven-ant-tasks-retrieve-build:
[artifact:dependencies] Downloading: asm/asm/3.2/asm-3.2-sources.jar from
repository central at http://repo1.maven.org/maven2
[artifact:dependencies] [INFO] Unable to find resource
'hsqldb:hsqldb:java-source:sources:1.8.0.10' in repository java.net2
(http://download.java.net/maven/2)
[artifact:dependencies] Building ant file:
/home/ubuntu/.ccm/repository/1.2.3/build/build-dependencies.xml
[copy] Copying 45 files to
/home/ubuntu/.ccm/repository/1.2.3/build/lib/jars
[copy] Copying 35 files to
/home/ubuntu/.ccm/repository/1.2.3/build/lib/sources
init:
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/classes/main
[mkdir] Created dir:
/home/ubuntu/.ccm/repository/1.2.3/build/classes/thrift
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/lib
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/classes
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/src/gen-java
check-avro-generate:
avro-interface-generate-internode:
[echo] Generating Avro internode code...
avro-generate:
build-subprojects:
check-gen-cli-grammar:
gen-cli-grammar:
[echo] Building Grammar
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cli/Cli.g
check-gen-cql2-grammar:
gen-cql2-grammar:
[echo] Building Grammar
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g
...
[java] warning(200):
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20:
Decision can match input such as IDENT using multiple alternatives: 1, 2
[java] As a result, alternative(s) 2 were disabled for that input
[java] warning(200):
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20:
Decision can match input such as K_KEY using multiple alternatives: 1, 2
[java] As a result, alternative(s) 2 were disabled for that input
[java] warning(200):
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20:
Decision can match input such as QMARK using multiple alternatives: 1, 2
[java] As a result, alternative(s) 2 were disabled for that input
[java] warning(200):
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20:
Decision can match input such as FLOAT using multiple alternatives: 1, 2
[java] As a result, alternative(s) 2 were disabled for that input
[java] warning(200):
[jira] [Commented] (CASSANDRA-9486) LazilyCompactedRow accumulates all expired RangeTombstones
[ https://issues.apache.org/jira/browse/CASSANDRA-9486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569721#comment-14569721 ] Brent Haines commented on CASSANDRA-9486: - FWIW -- I created a patch from [~slebresne]'s proposed fix, tested it locally and tried it on one of our impacted nodes in our data cluster. We are running 2.1.5, but the variance was easy to sift through. Compactions that used to take a day finished in 30 seconds. We haven't seen any spikes in heap space either, but that will take more time to evaluate. I do not know if there are any residual or unexpected behaviors resulting from the patch yet. We'll leave it in place for a few days to see. In the mean time, I thought it might help to confirm that this does indeed seem to fix our issue. Now, if we can stop creating so many RT's in the first place... :) LazilyCompactedRow accumulates all expired RangeTombstones -- Key: CASSANDRA-9486 URL: https://issues.apache.org/jira/browse/CASSANDRA-9486 Project: Cassandra Issue Type: Bug Components: Core Reporter: Benedict Assignee: Sylvain Lebresne Priority: Critical Fix For: 3.x, 2.1.x, 2.0.x, 2.2.x, 1.2.x Attachments: 0001-9486.patch LazilyCompactedRow initializes a ColumnIndex.Builder to use its RangeTombstone.Tracker, but it only calls update() with a RT argument, never an atom. The Tracker only ever _adds_ if it receives a RT, never removes. So all the RT ever seen for the partition (that have expired) remain in memory until the compaction completes. To make matters worse, this then forces a linear scan of all of these RT for each live cell we add, so this extra load hangs around for a long time, and compactions stall. This issue is biting one of our users badly (at least, it seems likely to be this issue), and there may be others. This user is not even making use of RT extensively themselves, only collections (presumably with a complete overwrite of the contents of the collection, resulting in a RT being generated). Probably the best solution is to make the RT addition itself remove any already present that are no longer helpful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5543) Ant issues when building gen-cql2-grammar
[
https://issues.apache.org/jira/browse/CASSANDRA-5543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569779#comment-14569779
]
Mike Lococo commented on CASSANDRA-5543:
Interesting speculation as to why subsequent runs may be working, that didn't
occur to me.
If it's not obvious, I have no antlr background either. I'll be watching this
ticket and am happy to provide additional data or direct access to a box that
reproduces this bug, but won't be digging into that potentially faulty grammar
to find and fix the bug.
Ant issues when building gen-cql2-grammar
-
Key: CASSANDRA-5543
URL: https://issues.apache.org/jira/browse/CASSANDRA-5543
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.2.3
Reporter: Joaquin Casares
Assignee: Dave Brosius
Priority: Trivial
Attachments: 5543-ant.log
Below are the commands and outputs that were returned.
The first `ant` command fails on gen-cql2-grammar, but if I don't run `ant
realclean` then it works fine after a second pass.
{CODE}
ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant realclean
Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml
clean:
[delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build/test
[delete] Deleting directory
/home/ubuntu/.ccm/repository/1.2.3/build/classes
[delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/src/gen-java
[delete] Deleting: /home/ubuntu/.ccm/repository/1.2.3/build/internode.avpr
realclean:
[delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build
BUILD SUCCESSFUL
Total time: 0 seconds
{CODE}
{CODE}
ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant
Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml
maven-ant-tasks-localrepo:
maven-ant-tasks-download:
[echo] Downloading Maven ANT Tasks...
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build
[get] Getting:
http://repo2.maven.org/maven2/org/apache/maven/maven-ant-tasks/2.1.3/maven-ant-tasks-2.1.3.jar
[get] To:
/home/ubuntu/.ccm/repository/1.2.3/build/maven-ant-tasks-2.1.3.jar
maven-ant-tasks-init:
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/lib
maven-declare-dependencies:
maven-ant-tasks-retrieve-build:
[artifact:dependencies] Downloading: asm/asm/3.2/asm-3.2-sources.jar from
repository central at http://repo1.maven.org/maven2
[artifact:dependencies] [INFO] Unable to find resource
'hsqldb:hsqldb:java-source:sources:1.8.0.10' in repository java.net2
(http://download.java.net/maven/2)
[artifact:dependencies] Building ant file:
/home/ubuntu/.ccm/repository/1.2.3/build/build-dependencies.xml
[copy] Copying 45 files to
/home/ubuntu/.ccm/repository/1.2.3/build/lib/jars
[copy] Copying 35 files to
/home/ubuntu/.ccm/repository/1.2.3/build/lib/sources
init:
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/classes/main
[mkdir] Created dir:
/home/ubuntu/.ccm/repository/1.2.3/build/classes/thrift
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/lib
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/classes
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/src/gen-java
check-avro-generate:
avro-interface-generate-internode:
[echo] Generating Avro internode code...
avro-generate:
build-subprojects:
check-gen-cli-grammar:
gen-cli-grammar:
[echo] Building Grammar
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cli/Cli.g
check-gen-cql2-grammar:
gen-cql2-grammar:
[echo] Building Grammar
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g
...
[java] warning(200):
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20:
Decision can match input such as IDENT using multiple alternatives: 1, 2
[java] As a result, alternative(s) 2 were disabled for that input
[java] warning(200):
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20:
Decision can match input such as K_KEY using multiple alternatives: 1, 2
[java] As a result, alternative(s) 2 were disabled for that input
[java] warning(200):
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20:
Decision can match input such as QMARK using multiple alternatives: 1, 2
[java] As a result, alternative(s) 2 were disabled for that input
[java] warning(200):
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20:
Decision can match input such as FLOAT using multiple alternatives: 1, 2
[java] As a result, alternative(s) 2 were disabled for that input
[java] warning(200):
[jira] [Updated] (CASSANDRA-9527) Cannot create secondary index on a table WITH COMPACT STORAGE
[ https://issues.apache.org/jira/browse/CASSANDRA-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Lerer updated CASSANDRA-9527: -- Attachment: 9527.txt The patch make sure that we log a different error message for this specific case. Cannot create secondary index on a table WITH COMPACT STORAGE - Key: CASSANDRA-9527 URL: https://issues.apache.org/jira/browse/CASSANDRA-9527 Project: Cassandra Issue Type: Bug Components: Core Environment: cassandra v2.0.13 Reporter: fuggy_yama Assignee: Benjamin Lerer Priority: Minor Fix For: 2.0.x Attachments: 9527.txt In CASSANDRA-8156 it is said that secondary indexes are not allowed on clustering columns in COMPACT tables. However, I found that it is not possible to create a secondary index on the value column in a COMPACT table: CREATE TABLE t ( a INT, b INT, c INT, PRIMARY KEY (a, b) ) WITH COMPACT STORAGE; CREATE INDEX ON t (c); *Bad Request: Secondary indexes are not supported on PRIMARY KEY columns in COMPACT STORAGE tables* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8576) Primary Key Pushdown For Hadoop
[ https://issues.apache.org/jira/browse/CASSANDRA-8576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569713#comment-14569713 ] Philip Thompson commented on CASSANDRA-8576: This does not break any of the existing pig tests. I ran some additional tests, and found no major issues. As far as a mixed version cluster, I spun up a 3 node cluster of C*, with two nodes running this patch, the third without. I connected Pig to the cluster, using the unmodified node as the initial address. I then performed some map reduce jobs to select data from the cluster. The jobs succeeded, and I did see any errors in the log. +1 from me. Primary Key Pushdown For Hadoop --- Key: CASSANDRA-8576 URL: https://issues.apache.org/jira/browse/CASSANDRA-8576 Project: Cassandra Issue Type: Improvement Components: Hadoop Reporter: Russell Alexander Spitzer Assignee: Alex Liu Fix For: 2.1.x Attachments: 8576-2.1-branch.txt, 8576-trunk.txt, CASSANDRA-8576-v2-2.1-branch.txt, CASSANDRA-8576-v3-2.1-branch.txt I've heard reports from several users that they would like to have predicate pushdown functionality for hadoop (Hive in particular) based services. Example usecase Table with wide partitions, one per customer Application team has HQL they would like to run on a single customer Currently time to complete scales with number of customers since Input Format can't pushdown primary key predicate Current implementation requires a full table scan (since it can't recognize that a single partition was specified) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9534) Repair data not always saved to system_distributed
[
https://issues.apache.org/jira/browse/CASSANDRA-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Witschey updated CASSANDRA-9534:
Summary: Repair data not always saved to system_distributed (was: Parent
repair data not always saved to system_distributed)
Repair data not always saved to system_distributed
--
Key: CASSANDRA-9534
URL: https://issues.apache.org/jira/browse/CASSANDRA-9534
Project: Cassandra
Issue Type: Bug
Reporter: Jim Witschey
Assignee: Marcus Eriksson
Fix For: 2.2.0 rc1
About 1 out of 5 times, in a ccm-managed cluster, a repair doesn't write
anything to the {{system_distributed.parent_repair_history}} table. [This
dtest|https://github.com/riptano/cassandra-dtest/blob/master/repair_test.py#L296]
reproduces the error on a 5-node cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5543) Ant issues when building gen-cql2-grammar
[
https://issues.apache.org/jira/browse/CASSANDRA-5543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569767#comment-14569767
]
Mike Lococo commented on CASSANDRA-5543:
To be clear, it also works if you just run ant a second time with no additional
changes. So...
$ ant # error as noted above
$ ant # works
$ ant clean # works
$ ant # same error
$ ant # works
The main thing this hurts is automated workflows on top of the ant build. I'm
not sure how much of that is going on in the wild. I'm working around this for
now by omitting the source build from the automated workflow I'm building. In
my case it's not essential.
I did want to drop the extra details I've found into this bug report, though,
since it seems like it's been an irritation for some time.
Ant issues when building gen-cql2-grammar
-
Key: CASSANDRA-5543
URL: https://issues.apache.org/jira/browse/CASSANDRA-5543
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.2.3
Reporter: Joaquin Casares
Assignee: Dave Brosius
Priority: Trivial
Attachments: 5543-ant.log
Below are the commands and outputs that were returned.
The first `ant` command fails on gen-cql2-grammar, but if I don't run `ant
realclean` then it works fine after a second pass.
{CODE}
ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant realclean
Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml
clean:
[delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build/test
[delete] Deleting directory
/home/ubuntu/.ccm/repository/1.2.3/build/classes
[delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/src/gen-java
[delete] Deleting: /home/ubuntu/.ccm/repository/1.2.3/build/internode.avpr
realclean:
[delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build
BUILD SUCCESSFUL
Total time: 0 seconds
{CODE}
{CODE}
ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant
Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml
maven-ant-tasks-localrepo:
maven-ant-tasks-download:
[echo] Downloading Maven ANT Tasks...
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build
[get] Getting:
http://repo2.maven.org/maven2/org/apache/maven/maven-ant-tasks/2.1.3/maven-ant-tasks-2.1.3.jar
[get] To:
/home/ubuntu/.ccm/repository/1.2.3/build/maven-ant-tasks-2.1.3.jar
maven-ant-tasks-init:
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/lib
maven-declare-dependencies:
maven-ant-tasks-retrieve-build:
[artifact:dependencies] Downloading: asm/asm/3.2/asm-3.2-sources.jar from
repository central at http://repo1.maven.org/maven2
[artifact:dependencies] [INFO] Unable to find resource
'hsqldb:hsqldb:java-source:sources:1.8.0.10' in repository java.net2
(http://download.java.net/maven/2)
[artifact:dependencies] Building ant file:
/home/ubuntu/.ccm/repository/1.2.3/build/build-dependencies.xml
[copy] Copying 45 files to
/home/ubuntu/.ccm/repository/1.2.3/build/lib/jars
[copy] Copying 35 files to
/home/ubuntu/.ccm/repository/1.2.3/build/lib/sources
init:
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/classes/main
[mkdir] Created dir:
/home/ubuntu/.ccm/repository/1.2.3/build/classes/thrift
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/lib
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/classes
[mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/src/gen-java
check-avro-generate:
avro-interface-generate-internode:
[echo] Generating Avro internode code...
avro-generate:
build-subprojects:
check-gen-cli-grammar:
gen-cli-grammar:
[echo] Building Grammar
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cli/Cli.g
check-gen-cql2-grammar:
gen-cql2-grammar:
[echo] Building Grammar
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g
...
[java] warning(200):
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20:
Decision can match input such as IDENT using multiple alternatives: 1, 2
[java] As a result, alternative(s) 2 were disabled for that input
[java] warning(200):
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20:
Decision can match input such as K_KEY using multiple alternatives: 1, 2
[java] As a result, alternative(s) 2 were disabled for that input
[java] warning(200):
/home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20:
Decision can match input such as QMARK using multiple alternatives: 1, 2
[java] As a result, alternative(s) 2 were disabled for that input
[java] warning(200):
[jira] [Commented] (CASSANDRA-9409) Ensure that UDF and UDAs are keyspace-isolated
[
https://issues.apache.org/jira/browse/CASSANDRA-9409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570009#comment-14570009
]
Zachary Kurey commented on CASSANDRA-9409:
--
Is it expected that built in functions cannot be resolved by a UDA unless we
specify the system keyspace? For example:
{code}
CREATE AGGREGATE if not exists ks.my_agg(int)
SFUNC ks.adder
STYPE int
FINALFUNC intasblob
;
{code}
Gives me me the error:
{quote}Final function ks.intasblob(int) does not exist or is not a scalar
function{quote}
Prefixing with 'system.' fixes the issue. But I was expecting it to resolve as
if I were using a built in function elsewhere.
Ensure that UDF and UDAs are keyspace-isolated
--
Key: CASSANDRA-9409
URL: https://issues.apache.org/jira/browse/CASSANDRA-9409
Project: Cassandra
Issue Type: Bug
Reporter: Aleksey Yeschenko
Assignee: Robert Stupp
Labels: qa-resolved
Fix For: 2.2.0 rc1
Attachments: 9409-with-system.txt, 9409.txt
In table columns we don't allow to use UDTs from other keyspaces.
We should also make sure that the following is *not* allowed:
- UDFs taking UDTs from other keyspaces as arguments or return types
- UDAs using UDFs from other keyspaces as its subfunctions
The only exception should be made for {{system}} keyspace. UDAs and UDFs from
any keyspace should be able to reference and reuse those.
Having no dependencies between keyspaces makes this consistent with the way
we treat UDTs, which is important, but also simplifies auth in multi-tenant
environments, and is also crucial to upcoming 3.X strongly consistent schema
work (the ability to treat keyspace as a unit of change and not having to
worry about cross-keyspace dependencies).
P.S. Should *probably* still allow using other keyspaces UDFs and UDAs in
{{SELECT}} statements. Strict isolation in schema dependencies is what
matters to me here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9536) The failure detector becomes more sensitive when the network is flakey
[
https://issues.apache.org/jira/browse/CASSANDRA-9536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569894#comment-14569894
]
Ron Kuris commented on CASSANDRA-9536:
--
One additional thought here is that it's probably fairly safe to record some
extra 2 values since that errs on the side of not bringing the node down as
fast. In fact, the first sample received is automatically recorded at value 2
so it's kind of a known quantity. The average quickly drops as values close to
1 are added to the list.
The failure detector becomes more sensitive when the network is flakey
--
Key: CASSANDRA-9536
URL: https://issues.apache.org/jira/browse/CASSANDRA-9536
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Ron Kuris
Priority: Minor
Fix For: 2.2.x
I added considerable instrumentation into the failure detector, and then
blocked port 7000 for a random 5-6 second interval, then resumed traffic for
the same amount of time, with a script like:
{code}while :
do
iptables -A INPUT -p tcp --destination-port 7000 -j DROP
v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)]
s=5.${v:1:2}${v:4:3}
echo offline for $s
sleep $s
iptables -F
v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)]
s=5.${v:1:2}${v:4:3}
echo online for $s
sleep $s
done{code}
When I do this, I watch the values being reported to the FailureDetector. The
mean actually goes down, as low as 850ms. The reason is that the very slow
packets are not recorded (they exceed MAX_INTERVAL_IN_NANO which is 2
seconds) and the retransmitted packets arrive very quickly in succession,
lowering the overall average.
Once the average is lowered, the node becomes much more sensitive to shorter
outages. If you run this code for a while, the average drops down to 800ms or
less, which means that the node will go down 20% quicker than expected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9536) The failure detector becomes more sensitive when the network is flakey
[
https://issues.apache.org/jira/browse/CASSANDRA-9536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis updated CASSANDRA-9536:
--
Fix Version/s: 2.2.x
The failure detector becomes more sensitive when the network is flakey
--
Key: CASSANDRA-9536
URL: https://issues.apache.org/jira/browse/CASSANDRA-9536
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Ron Kuris
Priority: Minor
Fix For: 2.2.x
I added considerable instrumentation into the failure detector, and then
blocked port 7000 for a random 5-6 second interval, then resumed traffic for
the same amount of time, with a script like:
{code}while :
do
iptables -A INPUT -p tcp --destination-port 7000 -j DROP
v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)]
s=5.${v:1:2}${v:4:3}
echo offline for $s
sleep $s
iptables -F
v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)]
s=5.${v:1:2}${v:4:3}
echo online for $s
sleep $s
done{code}
When I do this, I watch the values being reported to the FailureDetector. The
mean actually goes down, as low as 850ms. The reason is that the very slow
packets are not recorded (they exceed MAX_INTERVAL_IN_NANO which is 2
seconds) and the retransmitted packets arrive very quickly in succession,
lowering the overall average.
Once the average is lowered, the node becomes much more sensitive to shorter
outages. If you run this code for a while, the average drops down to 800ms or
less, which means that the node will go down 20% quicker than expected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8735) Batch log replication is not randomized when there are only 2 racks
[ https://issues.apache.org/jira/browse/CASSANDRA-8735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mihai Suteu updated CASSANDRA-8735: --- Attachment: CASSANDRA-8735.patch Created a patch for cassandra-2.2 The file location changed since the original issue was for cassandra 2.0.10, but the bug still persisted. Batch log replication is not randomized when there are only 2 racks --- Key: CASSANDRA-8735 URL: https://issues.apache.org/jira/browse/CASSANDRA-8735 Project: Cassandra Issue Type: Bug Reporter: Yuki Morishita Priority: Minor Attachments: CASSANDRA-8735.patch Batch log replication is not randomized and the same 2 nodes can be picked up when there are only 2 racks in the cluster. https://github.com/apache/cassandra/blob/cassandra-2.0.11/src/java/org/apache/cassandra/service/BatchlogEndpointSelector.java#L72-73 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-8735) Batch log replication is not randomized when there are only 2 racks
[ https://issues.apache.org/jira/browse/CASSANDRA-8735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mihai Suteu reassigned CASSANDRA-8735: -- Assignee: Mihai Suteu Batch log replication is not randomized when there are only 2 racks --- Key: CASSANDRA-8735 URL: https://issues.apache.org/jira/browse/CASSANDRA-8735 Project: Cassandra Issue Type: Bug Reporter: Yuki Morishita Assignee: Mihai Suteu Priority: Minor Attachments: CASSANDRA-8735.patch Batch log replication is not randomized and the same 2 nodes can be picked up when there are only 2 racks in the cluster. https://github.com/apache/cassandra/blob/cassandra-2.0.11/src/java/org/apache/cassandra/service/BatchlogEndpointSelector.java#L72-73 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8735) Batch log replication is not randomized when there are only 2 racks
[ https://issues.apache.org/jira/browse/CASSANDRA-8735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570189#comment-14570189 ] Mikhail Stepura commented on CASSANDRA-8735: [~yukim]. I don't remember. It looks like I made that test case based on the assumption that the behavior from Jonathan's patch CASSANDRA-6551 #2 *is* the expected one. Batch log replication is not randomized when there are only 2 racks --- Key: CASSANDRA-8735 URL: https://issues.apache.org/jira/browse/CASSANDRA-8735 Project: Cassandra Issue Type: Bug Reporter: Yuki Morishita Assignee: Mihai Suteu Priority: Minor Fix For: 2.1.x, 2.2.x Attachments: CASSANDRA-8735.patch Batch log replication is not randomized and the same 2 nodes can be picked up when there are only 2 racks in the cluster. https://github.com/apache/cassandra/blob/cassandra-2.0.11/src/java/org/apache/cassandra/service/BatchlogEndpointSelector.java#L72-73 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8735) Batch log replication is not randomized when there are only 2 racks
[
https://issues.apache.org/jira/browse/CASSANDRA-8735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570130#comment-14570130
]
Yuki Morishita commented on CASSANDRA-8735:
---
So the patch will make
{{BatchlogEndpointFilterTest#shouldSelectTwoFirstHostsFromSingleOtherRack}}
fail occasionally since the test is expected to be always the first two.
[~mishail] is this intended behavior? I saw the case where the same 2 nodes
always getting a lot of batch log traffic.
Batch log replication is not randomized when there are only 2 racks
---
Key: CASSANDRA-8735
URL: https://issues.apache.org/jira/browse/CASSANDRA-8735
Project: Cassandra
Issue Type: Bug
Reporter: Yuki Morishita
Assignee: Mihai Suteu
Priority: Minor
Fix For: 2.1.x, 2.2.x
Attachments: CASSANDRA-8735.patch
Batch log replication is not randomized and the same 2 nodes can be picked up
when there are only 2 racks in the cluster.
https://github.com/apache/cassandra/blob/cassandra-2.0.11/src/java/org/apache/cassandra/service/BatchlogEndpointSelector.java#L72-73
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9515) Bytes map encoding does not match protocol spec
[
https://issues.apache.org/jira/browse/CASSANDRA-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Lerer updated CASSANDRA-9515:
--
Attachment: 9515.txt
The patch replace the {{byte[]}} type used for the values in the {{Map}} by the
{{ByteBuffer}} type and use the {{readValue}} and {{writeValue}} methods from
{{CBUtil}} to read and write the values.
Bytes map encoding does not match protocol spec
---
Key: CASSANDRA-9515
URL: https://issues.apache.org/jira/browse/CASSANDRA-9515
Project: Cassandra
Issue Type: Bug
Reporter: Olivier Michallat
Assignee: Benjamin Lerer
Fix For: 2.2.0 rc1
Attachments: 9515.txt
In native_protocol_v4.spec, [bytes map] is defined as having [bytes] values,
where [bytes] is an int n followed by n bytes.
In practice, {{CBUtil#writeBytesMap}} encodes each value with
{{CBUtil#writeBytes}}, which writes the length as a short (so a [short bytes]
instead of a [bytes]).
This is used for the custom payload introduced by CASSANDRA-8553.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8743) Repair on NFS in version 2.1.2
[ https://issues.apache.org/jira/browse/CASSANDRA-8743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568892#comment-14568892 ] Nicolas Guyomar commented on CASSANDRA-8743: HI, I just reproduced this 'exact' same issue on 2.1.5 while doing a repair (nodetool repair) after changing my cluster snitch My cluster is also on an NFS mounted drive I ran again the repair and got the same exception (keyspace name and table are voluntary changed in the stack) INFO [AntiEntropyStage:1] 2015-06-02 10:56:29,133 Validator.java:257 - [repair #412b6e80-0905-11e5-aa92-f327be1d597d] Sending completed merkle tree to /10.106.199.9 for mykeyspace/mytable2 ERROR [ValidationExecutor:3] 2015-06-02 10:56:29,142 Validator.java:245 - Failed creating a merkle tree for [repair #412b6e80-0905-11e5-aa92-f327be1d597d on mykeyspace/mytable, (1997653793039594215,2135846523050696776]], /10.106.199.9 (see log for details) ERROR [ValidationExecutor:3] 2015-06-02 10:56:29,158 CassandraDaemon.java:223 - Exception in thread Thread[ValidationExecutor:3,1,main] org.apache.cassandra.io.FSWriteError: java.nio.file.DirectoryNotEmptyException: /var/opt/data/flat/cassandra/mykeyspace/mytable-648d9b70abc011e49f8747338c540143/snapshots/412b6e80-0905-11e5-aa92-f327be1d597d at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:135) ~[apache-cassandra-2.1.5.jar:2.1.5] at org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:381) ~[apache-cassandra-2.1.5.jar:2.1.5] at org.apache.cassandra.db.Directories.clearSnapshot(Directories.java:616) ~[apache-cassandra-2.1.5.jar:2.1.5] at org.apache.cassandra.db.ColumnFamilyStore.clearSnapshot(ColumnFamilyStore.java:2341) ~[apache-cassandra-2.1.5.jar:2.1.5] at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1013) ~[apache-cassandra-2.1.5.jar:2.1.5] at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:95) ~[apache-cassandra-2.1.5.jar:2.1.5] at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:617) ~[apache-cassandra-2.1.5.jar:2.1.5] at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_67] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.7.0_67] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.7.0_67] at java.lang.Thread.run(Unknown Source) [na:1.7.0_67] Caused by: java.nio.file.DirectoryNotEmptyException: /var/opt/data/flat/cassandra/mykeyspace/mytable-648d9b70abc011e49f8747338c540143/snapshots/412b6e80-0905-11e5-aa92-f327be1d597d at sun.nio.fs.UnixFileSystemProvider.implDelete(Unknown Source) ~[na:1.7.0_67] at sun.nio.fs.AbstractFileSystemProvider.delete(Unknown Source) ~[na:1.7.0_67] at java.nio.file.Files.delete(Unknown Source) ~[na:1.7.0_67] at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:131) ~[apache-cassandra-2.1.5.jar:2.1.5] ... 10 common frames omitted ERROR [ValidationExecutor:3] 2015-06-02 10:56:29,166 StorageService.java:445 - Stopping gossiper WARN [ValidationExecutor:3] 2015-06-02 10:56:29,167 StorageService.java:351 - Stopping gossip by operator request INFO [ValidationExecutor:3] 2015-06-02 10:56:29,167 Gossiper.java:1410 - Announcing shutdown Repair on NFS in version 2.1.2 -- Key: CASSANDRA-8743 URL: https://issues.apache.org/jira/browse/CASSANDRA-8743 Project: Cassandra Issue Type: Bug Components: Core Reporter: Tamar Nirenberg Assignee: Joshua McKenzie Priority: Minor Running repair over NFS in Cassandra 2.1.2 encounters this error and crashes the ring: ERROR [ValidationExecutor:2] 2015-01-22 11:48:14,811 Validator.java:232 - Failed creating a merkle tree for [repair #c84c7c70-a21b-11e4-aeca-19e6d7fa2595 on ATTRIBUTES/LINKS, (11621838520493020277529637175352775759,11853478749048239324667887059881170862]], /10.1.234.63 (see log for details) ERROR [ValidationExecutor:2] 2015-01-22 11:48:14,827 CassandraDaemon.java:153 - Exception in thread Thread[ValidationExecutor:2,1,main] org.apache.cassandra.io.FSWriteError: java.nio.file.DirectoryNotEmptyException: /exlibris/cassandra/local/data/data/ATTRIBUTES/LINKS/snapshots/c84c7c70-a21b-11e4-aeca-19e6d7fa2595 at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:135) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:381) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.Directories.clearSnapshot(Directories.java:547) ~[apache-cassandra-2.1.2.jar:2.1.2] at
[jira] [Updated] (CASSANDRA-9515) Bytes map encoding does not match protocol spec
[
https://issues.apache.org/jira/browse/CASSANDRA-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sylvain Lebresne updated CASSANDRA-9515:
Reviewer: Sylvain Lebresne
Bytes map encoding does not match protocol spec
---
Key: CASSANDRA-9515
URL: https://issues.apache.org/jira/browse/CASSANDRA-9515
Project: Cassandra
Issue Type: Bug
Reporter: Olivier Michallat
Assignee: Benjamin Lerer
Fix For: 2.2.0 rc1
Attachments: 9515.txt
In native_protocol_v4.spec, [bytes map] is defined as having [bytes] values,
where [bytes] is an int n followed by n bytes.
In practice, {{CBUtil#writeBytesMap}} encodes each value with
{{CBUtil#writeBytes}}, which writes the length as a short (so a [short bytes]
instead of a [bytes]).
This is used for the custom payload introduced by CASSANDRA-8553.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-7688) Add data sizing to a system table
[ https://issues.apache.org/jira/browse/CASSANDRA-7688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568971#comment-14568971 ] Piotr Kołaczkowski edited comment on CASSANDRA-7688 at 6/2/15 11:29 AM: I double checked the cassandra.size_recorder_interval option does indeed work, but because many tests are creating fresh keyspaces.tables, the interval of 1 second is still too large and the test manages to run before the estimates are created. We'd need to put Thread.sleep before all the tests, which we're not going to do, because it would significantly increase runtime of the whole test suite. So far we just removed the warning about missing estimates. But it would be nice if C* filled those estimate entries on table creation (even with zeroes). There is a difference between we don't know the estimates and we know there are no data. was (Author: pkolaczk): I double checked the cassandra.size_recorder_interval option does indeed work, but because many tests are creating fresh keyspaces.tables, the interval of 1 second is still too large and the test manages to run before the estimates are created. We'd need to put Thread.sleep before all the tests, which we're not going to do, because it would So far we just removed the warning about missing estimates. But it would be nice if C* filled those estimate entries on table creation (even with zeroes). There is a difference between we don't know the estimates and we know there are no data. Add data sizing to a system table - Key: CASSANDRA-7688 URL: https://issues.apache.org/jira/browse/CASSANDRA-7688 Project: Cassandra Issue Type: New Feature Reporter: Jeremiah Jordan Assignee: Aleksey Yeschenko Fix For: 2.1.5 Attachments: 7688.txt Currently you can't implement something similar to describe_splits_ex purely from the a native protocol driver. https://datastax-oss.atlassian.net/browse/JAVA-312 is open to expose easily getting ownership information to a client in the java-driver. But you still need the data sizing part to get splits of a given size. We should add the sizing information to a system table so that native clients can get to it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9527) Cannot create secondary index on a table WITH COMPACT STORAGE
[
https://issues.apache.org/jira/browse/CASSANDRA-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569144#comment-14569144
]
Benjamin Lerer commented on CASSANDRA-9527:
---
The issue is valid.
Our test is:
{code}
if ((cfm.getCfDef().isCompact || !cfm.getCfDef().isComposite) cd.type !=
ColumnDefinition.Type.REGULAR)
throw new InvalidRequestException(Secondary indexes are not
supported on PRIMARY KEY columns in COMPACT STORAGE tables);
{code}
The problem is that the type of the c column is
{{ColumnDefinition.Type.COMPACT_VALUE}} and not
{{ColumnDefinition.Type.REGULAR}}.
Cannot create secondary index on a table WITH COMPACT STORAGE
-
Key: CASSANDRA-9527
URL: https://issues.apache.org/jira/browse/CASSANDRA-9527
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: cassandra v2.0.13
Reporter: fuggy_yama
Assignee: Benjamin Lerer
Priority: Minor
Fix For: 2.0.x
In CASSANDRA-8156 it is said that secondary indexes are not allowed on
clustering columns in COMPACT tables.
However, I found that it is not possible to create a secondary index on the
value column in a COMPACT table:
CREATE TABLE t (
a INT,
b INT,
c INT,
PRIMARY KEY (a, b)
) WITH COMPACT STORAGE;
CREATE INDEX ON t (c);
*Bad Request: Secondary indexes are not supported on PRIMARY KEY columns in
COMPACT STORAGE tables*
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-9531) NullPointerException logged when running MessagePayloadTest
[
https://issues.apache.org/jira/browse/CASSANDRA-9531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
T Jake Luciani reassigned CASSANDRA-9531:
-
Assignee: T Jake Luciani
NullPointerException logged when running MessagePayloadTest
---
Key: CASSANDRA-9531
URL: https://issues.apache.org/jira/browse/CASSANDRA-9531
Project: Cassandra
Issue Type: Bug
Reporter: Benjamin Lerer
Assignee: T Jake Luciani
Priority: Minor
When I run the {{MessagePayloadTest}} I have the following error in my log
output:
{code}
juin 02, 2015 3:27:31 PM com.google.common.cache.LocalCache
processPendingNotifications
AVERTISSEMENT: Exception thrown by removal listener
java.lang.NullPointerException
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.getTotalBufferSize(CompressedRandomAccessReader.java:260)
at
org.apache.cassandra.service.FileCacheService$2.onRemoval(FileCacheService.java:105)
at
com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1954)
at
com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3457)
at
com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3433)
at
com.google.common.cache.LocalCache$Segment.remove(LocalCache.java:3104)
at com.google.common.cache.LocalCache.remove(LocalCache.java:4170)
at
com.google.common.cache.LocalCache$LocalManualCache.invalidate(LocalCache.java:4762)
at
org.apache.cassandra.service.FileCacheService.invalidate(FileCacheService.java:182)
at
org.apache.cassandra.io.util.PoolingSegmentedFile$Cleanup.tidy(PoolingSegmentedFile.java:53)
at
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Cleanup.tidy(CompressedPoolingSegmentedFile.java:77)
at
org.apache.cassandra.utils.concurrent.Ref$GlobalState.release(Ref.java:285)
at
org.apache.cassandra.utils.concurrent.Ref$State.ensureReleased(Ref.java:164)
at org.apache.cassandra.utils.concurrent.Ref.ensureReleased(Ref.java:89)
at
org.apache.cassandra.utils.concurrent.SharedCloseableImpl.close(SharedCloseableImpl.java:45)
at
org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier$1.run(SSTableReader.java:1937)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
The problem is due to the fact that when {{getTotalBufferSize}} is called the
segment has already been desallocated. The reason being that the segment has
been put twice in the queue of the {{CacheBucket}}.
I tracked down the problem to {{CompactionTask.runMayThrow}} which trigger 2
calls to the {{close}} method of {{BigTableScanner}}.
One here:
{code}
at
org.apache.cassandra.io.sstable.format.big.BigTableScanner.close(BigTableScanner.java:197)
at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy$ScannerList.close(AbstractCompactionStrategy.java:333)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:207)
{code}
and the other here:
{code}
at
org.apache.cassandra.io.sstable.format.big.BigTableScanner.close(BigTableScanner.java:197)
at org.apache.cassandra.utils.MergeIterator.close(MergeIterator.java:62)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:206)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9486) LazilyCompactedRow accumulates all expired RangeTombstones
[
https://issues.apache.org/jira/browse/CASSANDRA-9486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569175#comment-14569175
]
Sylvain Lebresne commented on CASSANDRA-9486:
-
I had actually forgot one add in {{update}} which was breaking tests. I've
pushed the fix and I'll wait once the tests have ran.
bq. the semantics of an arbitrary method labelled insertBefore might well want
to do so
Right, felt more logical so makes hypothetical future misuse of the method less
likely.
LazilyCompactedRow accumulates all expired RangeTombstones
--
Key: CASSANDRA-9486
URL: https://issues.apache.org/jira/browse/CASSANDRA-9486
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Benedict
Assignee: Sylvain Lebresne
Priority: Critical
Fix For: 3.x, 2.1.x, 2.0.x, 2.2.x, 1.2.x
Attachments: 0001-9486.patch
LazilyCompactedRow initializes a ColumnIndex.Builder to use its
RangeTombstone.Tracker, but it only calls update() with a RT argument, never
an atom. The Tracker only ever _adds_ if it receives a RT, never removes. So
all the RT ever seen for the partition (that have expired) remain in memory
until the compaction completes. To make matters worse, this then forces a
linear scan of all of these RT for each live cell we add, so this extra load
hangs around for a long time, and compactions stall.
This issue is biting one of our users badly (at least, it seems likely to be
this issue), and there may be others. This user is not even making use of RT
extensively themselves, only collections (presumably with a complete
overwrite of the contents of the collection, resulting in a RT being
generated).
Probably the best solution is to make the RT addition itself remove any
already present that are no longer helpful.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9526) Provide a JMX hook to monitor phi values in the FailureDetector
[ https://issues.apache.org/jira/browse/CASSANDRA-9526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569127#comment-14569127 ] Brandon Williams commented on CASSANDRA-9526: - Thanks for the patch, Ron, this is good work. A couple of nits here: You disabled the logger.isTraceEnabled() in one call (which is fine) but then repeated the pattern you undid thereafter. I can be fine with either since they don't matter, but let's be consistent. Your IDE incorrectly reordered the imports. Provide a JMX hook to monitor phi values in the FailureDetector --- Key: CASSANDRA-9526 URL: https://issues.apache.org/jira/browse/CASSANDRA-9526 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Ron Kuris Fix For: 2.0.x Attachments: Monitor-Phi-JMX.patch.txt, PHI-Log-Debug-When-Close.patch.txt, PHI-Race-Condition.patch.txt phi_convict_threshold can be tuned, but there's currently no way to monitor the phi values to see if you're getting close. The attached patch adds the ability to get these values via JMX. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9531) NullPointerException logged when running MessagePayloadTest
[
https://issues.apache.org/jira/browse/CASSANDRA-9531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
T Jake Luciani updated CASSANDRA-9531:
--
Fix Version/s: 2.2.0 rc1
NullPointerException logged when running MessagePayloadTest
---
Key: CASSANDRA-9531
URL: https://issues.apache.org/jira/browse/CASSANDRA-9531
Project: Cassandra
Issue Type: Bug
Reporter: Benjamin Lerer
Assignee: T Jake Luciani
Priority: Minor
Fix For: 2.2.0 rc1
When I run the {{MessagePayloadTest}} I have the following error in my log
output:
{code}
juin 02, 2015 3:27:31 PM com.google.common.cache.LocalCache
processPendingNotifications
AVERTISSEMENT: Exception thrown by removal listener
java.lang.NullPointerException
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.getTotalBufferSize(CompressedRandomAccessReader.java:260)
at
org.apache.cassandra.service.FileCacheService$2.onRemoval(FileCacheService.java:105)
at
com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1954)
at
com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3457)
at
com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3433)
at
com.google.common.cache.LocalCache$Segment.remove(LocalCache.java:3104)
at com.google.common.cache.LocalCache.remove(LocalCache.java:4170)
at
com.google.common.cache.LocalCache$LocalManualCache.invalidate(LocalCache.java:4762)
at
org.apache.cassandra.service.FileCacheService.invalidate(FileCacheService.java:182)
at
org.apache.cassandra.io.util.PoolingSegmentedFile$Cleanup.tidy(PoolingSegmentedFile.java:53)
at
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Cleanup.tidy(CompressedPoolingSegmentedFile.java:77)
at
org.apache.cassandra.utils.concurrent.Ref$GlobalState.release(Ref.java:285)
at
org.apache.cassandra.utils.concurrent.Ref$State.ensureReleased(Ref.java:164)
at org.apache.cassandra.utils.concurrent.Ref.ensureReleased(Ref.java:89)
at
org.apache.cassandra.utils.concurrent.SharedCloseableImpl.close(SharedCloseableImpl.java:45)
at
org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier$1.run(SSTableReader.java:1937)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
The problem is due to the fact that when {{getTotalBufferSize}} is called the
segment has already been desallocated. The reason being that the segment has
been put twice in the queue of the {{CacheBucket}}.
I tracked down the problem to {{CompactionTask.runMayThrow}} which trigger 2
calls to the {{close}} method of {{BigTableScanner}}.
One here:
{code}
at
org.apache.cassandra.io.sstable.format.big.BigTableScanner.close(BigTableScanner.java:197)
at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy$ScannerList.close(AbstractCompactionStrategy.java:333)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:207)
{code}
and the other here:
{code}
at
org.apache.cassandra.io.sstable.format.big.BigTableScanner.close(BigTableScanner.java:197)
at org.apache.cassandra.utils.MergeIterator.close(MergeIterator.java:62)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:206)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8656) long-test LongLeveledCompactionStrategyTest flaps in 2.0
[
https://issues.apache.org/jira/browse/CASSANDRA-8656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569201#comment-14569201
]
Stefania commented on CASSANDRA-8656:
-
I've attached a patch proposed by Marcus, that too has passed 2 rounds of 10
iterations on my box. Would you mind testing some more Michael?
long-test LongLeveledCompactionStrategyTest flaps in 2.0
Key: CASSANDRA-8656
URL: https://issues.apache.org/jira/browse/CASSANDRA-8656
Project: Cassandra
Issue Type: Test
Components: Tests
Reporter: Michael Shuler
Assignee: Stefania
Priority: Minor
Labels: test-failure
Fix For: 2.0.x
Attachments: system.log
LongLeveledCompactionStrategyTest periodically fails with:
{noformat}
[junit] Testsuite:
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest
[junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
54.412 sec
[junit]
[junit] Testcase:
testParallelLeveledCompaction(org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest):
Caused an ERROR
[junit] java.util.concurrent.ExecutionException:
java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) =
current key DecoratedKey(313236, 313236) writing into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] java.lang.RuntimeException:
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last
written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236,
313236) writing into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:413)
[junit] at
org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:402)
[junit] at
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest.testParallelLeveledCompaction(LongLeveledCompactionStrategyTest.java:97)
[junit] Caused by: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) =
current key DecoratedKey(313236, 313236) writing into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] at java.util.concurrent.FutureTask.report(FutureTask.java:122)
[junit] at java.util.concurrent.FutureTask.get(FutureTask.java:188)
[junit] at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:409)
[junit] Caused by: java.lang.RuntimeException: Last written key
DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing
into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
[junit] at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
[junit] at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167)
[junit] at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
[junit] at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
[junit] at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
[junit] at
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest$1.run(LongLeveledCompactionStrategyTest.java:87)
[junit] at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[junit] at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit] at java.lang.Thread.run(Thread.java:745)
[junit]
[junit]
[junit] Test
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest FAILED
{noformat}
I would guess the failure is 10-20% of the time, looping over the test
repeatedly.
On the 2.1 branch, the failure is different, so perhaps this could also be
updated.
{noformat}
[junit] Testsuite:
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest
[junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
7.04 sec
[junit]
[junit] Testcase:
testParallelLeveledCompaction(org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest):
Caused an ERROR
[junit]
[jira] [Commented] (CASSANDRA-7688) Add data sizing to a system table
[ https://issues.apache.org/jira/browse/CASSANDRA-7688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568971#comment-14568971 ] Piotr Kołaczkowski commented on CASSANDRA-7688: --- I double checked the cassandra.size_recorder_interval option does indeed work, but because many tests are creating fresh keyspaces.tables, the interval of 1 second is still too large and the test manages to run before the estimates are created. We'd need to put Thread.sleep before all the tests, which we're not going to do, because it would So far we just removed the warning about missing estimates. But it would be nice if C* filled those estimate entries on table creation (even with zeroes). There is a difference between we don't know the estimates and we know there are no data. Add data sizing to a system table - Key: CASSANDRA-7688 URL: https://issues.apache.org/jira/browse/CASSANDRA-7688 Project: Cassandra Issue Type: New Feature Reporter: Jeremiah Jordan Assignee: Aleksey Yeschenko Fix For: 2.1.5 Attachments: 7688.txt Currently you can't implement something similar to describe_splits_ex purely from the a native protocol driver. https://datastax-oss.atlassian.net/browse/JAVA-312 is open to expose easily getting ownership information to a client in the java-driver. But you still need the data sizing part to get splits of a given size. We should add the sizing information to a system table so that native clients can get to it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9527) Cannot create secondary index on a table WITH COMPACT STORAGE
[
https://issues.apache.org/jira/browse/CASSANDRA-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569146#comment-14569146
]
Benjamin Lerer commented on CASSANDRA-9527:
---
The issue is valid.
Our test is:
{code}
if ((cfm.getCfDef().isCompact || !cfm.getCfDef().isComposite) cd.type !=
ColumnDefinition.Type.REGULAR)
throw new InvalidRequestException(Secondary indexes are not
supported on PRIMARY KEY columns in COMPACT STORAGE tables);
{code}
The problem is that the type of the c column is
{{ColumnDefinition.Type.COMPACT_VALUE}} and not
{{ColumnDefinition.Type.REGULAR}}.
Cannot create secondary index on a table WITH COMPACT STORAGE
-
Key: CASSANDRA-9527
URL: https://issues.apache.org/jira/browse/CASSANDRA-9527
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: cassandra v2.0.13
Reporter: fuggy_yama
Assignee: Benjamin Lerer
Priority: Minor
Fix For: 2.0.x
In CASSANDRA-8156 it is said that secondary indexes are not allowed on
clustering columns in COMPACT tables.
However, I found that it is not possible to create a secondary index on the
value column in a COMPACT table:
CREATE TABLE t (
a INT,
b INT,
c INT,
PRIMARY KEY (a, b)
) WITH COMPACT STORAGE;
CREATE INDEX ON t (c);
*Bad Request: Secondary indexes are not supported on PRIMARY KEY columns in
COMPACT STORAGE tables*
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8656) long-test LongLeveledCompactionStrategyTest flaps in 2.0
[
https://issues.apache.org/jira/browse/CASSANDRA-8656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569151#comment-14569151
]
Michael Shuler commented on CASSANDRA-8656:
---
I see there's a little irc discussion on this patch, but I've run 40 times over
2.0 HEAD +
https://github.com/stef1927/cassandra/commit/6f7950e9db2ac7ad904e4962a9e5bd1cddfbb8e1.patch
with no failures.
long-test LongLeveledCompactionStrategyTest flaps in 2.0
Key: CASSANDRA-8656
URL: https://issues.apache.org/jira/browse/CASSANDRA-8656
Project: Cassandra
Issue Type: Test
Components: Tests
Reporter: Michael Shuler
Assignee: Stefania
Priority: Minor
Labels: test-failure
Fix For: 2.0.x
Attachments: system.log
LongLeveledCompactionStrategyTest periodically fails with:
{noformat}
[junit] Testsuite:
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest
[junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
54.412 sec
[junit]
[junit] Testcase:
testParallelLeveledCompaction(org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest):
Caused an ERROR
[junit] java.util.concurrent.ExecutionException:
java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) =
current key DecoratedKey(313236, 313236) writing into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] java.lang.RuntimeException:
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last
written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236,
313236) writing into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:413)
[junit] at
org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:402)
[junit] at
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest.testParallelLeveledCompaction(LongLeveledCompactionStrategyTest.java:97)
[junit] Caused by: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) =
current key DecoratedKey(313236, 313236) writing into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] at java.util.concurrent.FutureTask.report(FutureTask.java:122)
[junit] at java.util.concurrent.FutureTask.get(FutureTask.java:188)
[junit] at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:409)
[junit] Caused by: java.lang.RuntimeException: Last written key
DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing
into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
[junit] at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
[junit] at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167)
[junit] at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
[junit] at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
[junit] at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
[junit] at
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest$1.run(LongLeveledCompactionStrategyTest.java:87)
[junit] at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[junit] at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit] at java.lang.Thread.run(Thread.java:745)
[junit]
[junit]
[junit] Test
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest FAILED
{noformat}
I would guess the failure is 10-20% of the time, looping over the test
repeatedly.
On the 2.1 branch, the failure is different, so perhaps this could also be
updated.
{noformat}
[junit] Testsuite:
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest
[junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
7.04 sec
[junit]
[junit] Testcase:
[2/2] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1 Conflicts: CHANGES.txt src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java test/unit/org/apache/cassandra/db/compaction/BlacklistingCompactionsTest.java Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c0f96e1d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c0f96e1d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c0f96e1d Branch: refs/heads/cassandra-2.1 Commit: c0f96e1d46d664a9d4ca7f982d21cdfcaed1a24a Parents: 14a3324 9b10928 Author: Sylvain Lebresne sylv...@datastax.com Authored: Tue Jun 2 14:51:20 2015 +0200 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue Jun 2 14:51:20 2015 +0200 -- CHANGES.txt | 1 + .../io/sstable/SSTableIdentityIterator.java | 10 +++ .../cassandra/io/sstable/SSTableReader.java | 68 ++-- .../compaction/BlacklistingCompactionsTest.java | 17 +++-- 4 files changed, 69 insertions(+), 27 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0f96e1d/CHANGES.txt -- diff --cc CHANGES.txt index 71ce442,1aad965..4a7e174 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,37 -1,5 +1,38 @@@ -2.0.16: +2.1.6 + * Consistent error message when a table mixes counter and non-counter + columns (CASSANDRA-9492) + * Avoid getting unreadable keys during anticompaction (CASSANDRA-9508) + * (cqlsh) Better float precision by default (CASSANDRA-9224) + * Improve estimated row count (CASSANDRA-9107) + * Optimize range tombstone memory footprint (CASSANDRA-8603) + * Use configured gcgs in anticompaction (CASSANDRA-9397) + * Warn on misuse of unlogged batches (CASSANDRA-9282) + * Failure detector detects and ignores local pauses (CASSANDRA-9183) + * Add utility class to support for rate limiting a given log statement (CASSANDRA-9029) + * Add missing consistency levels to cassandra-stess (CASSANDRA-9361) + * Fix commitlog getCompletedTasks to not increment (CASSANDRA-9339) + * Fix for harmless exceptions logged as ERROR (CASSANDRA-8564) + * Delete processed sstables in sstablesplit/sstableupgrade (CASSANDRA-8606) + * Improve sstable exclusion from partition tombstones (CASSANDRA-9298) + * Validate the indexed column rather than the cell's contents for 2i (CASSANDRA-9057) + * Add support for top-k custom 2i queries (CASSANDRA-8717) + * Fix error when dropping table during compaction (CASSANDRA-9251) + * cassandra-stress supports validation operations over user profiles (CASSANDRA-8773) + * Add support for rate limiting log messages (CASSANDRA-9029) + * Log the partition key with tombstone warnings (CASSANDRA-8561) + * Reduce runWithCompactionsDisabled poll interval to 1ms (CASSANDRA-9271) + * Fix PITR commitlog replay (CASSANDRA-9195) + * GCInspector logs very different times (CASSANDRA-9124) + * Fix deleting from an empty list (CASSANDRA-9198) + * Update tuple and collection types that use a user-defined type when that UDT + is modified (CASSANDRA-9148, CASSANDRA-9192) + * Use higher timeout for prepair and snapshot in repair (CASSANDRA-9261) + * Fix anticompaction blocking ANTI_ENTROPY stage (CASSANDRA-9151) + * Repair waits for anticompaction to finish (CASSANDRA-9097) + * Fix streaming not holding ref when stream error (CASSANDRA-9295) + * Fix canonical view returning early opened SSTables (CASSANDRA-9396) +Merged from 2.0: + * Always mark sstable suspect when corrupted (CASSANDRA-9478) * Add database users and permissions to CQL3 documentation (CASSANDRA-7558) * Allow JVM_OPTS to be passed to standalone tools (CASSANDRA-5969) * Fix bad condition in RangeTombstoneList (CASSANDRA-9485) http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0f96e1d/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java -- diff --cc src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java index b784a7e,8b45005..498ad26 --- a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java +++ b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java @@@ -79,17 -95,18 +82,18 @@@ public class SSTableIdentityIterator im this.filename = filename; this.key = key; this.dataSize = dataSize; -this.expireBefore = (int)(System.currentTimeMillis() / 1000); this.flag = flag; this.validateColumns = checkData; -this.dataVersion = sstable == null ? Descriptor.Version.CURRENT : sstable.descriptor.version; + this.sstable = sstable; +Descriptor.Version dataVersion = sstable == null ?
[1/2] cassandra git commit: Always mark sstable suspected on corruption
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.1 14a332470 - c0f96e1d4
Always mark sstable suspected on corruption
patch by slebresne; reviewed by benedict for CASSANDRA-9478
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9b10928c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9b10928c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9b10928c
Branch: refs/heads/cassandra-2.1
Commit: 9b10928c159317160fb3049727679a48232b6041
Parents: 63819cb
Author: Sylvain Lebresne sylv...@datastax.com
Authored: Mon May 25 18:26:56 2015 +0200
Committer: Sylvain Lebresne sylv...@datastax.com
Committed: Tue Jun 2 14:46:09 2015 +0200
--
CHANGES.txt | 1 +
.../io/sstable/SSTableIdentityIterator.java | 45
.../compaction/BlacklistingCompactionsTest.java | 16 ---
3 files changed, 48 insertions(+), 14 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9b10928c/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index d23661d..1aad965 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.16:
+ * Always mark sstable suspect when corrupted (CASSANDRA-9478)
* Add database users and permissions to CQL3 documentation (CASSANDRA-7558)
* Allow JVM_OPTS to be passed to standalone tools (CASSANDRA-5969)
* Fix bad condition in RangeTombstoneList (CASSANDRA-9485)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9b10928c/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
--
diff --git
a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
index 52da9bb..8b45005 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
@@ -50,6 +50,9 @@ public class SSTableIdentityIterator implements
ComparableSSTableIdentityIterat
private final boolean validateColumns;
private final String filename;
+// Not every SSTableIdentifyIterator is attached to a sstable, so this can
be null.
+private final SSTableReader sstable;
+
/**
* Used to iterate through the columns of a row.
* @param sstable SSTable we are reading ffrom.
@@ -96,6 +99,7 @@ public class SSTableIdentityIterator implements
ComparableSSTableIdentityIterat
this.flag = flag;
this.validateColumns = checkData;
this.dataVersion = sstable == null ? Descriptor.Version.CURRENT :
sstable.descriptor.version;
+this.sstable = sstable;
try
{
@@ -132,9 +136,15 @@ public class SSTableIdentityIterator implements
ComparableSSTableIdentityIterat
{
// catch here b/c atomIterator is an AbstractIterator; hasNext
reads the value
if (e.getCause() instanceof IOException)
+{
+if (sstable != null)
+sstable.markSuspect();
throw new CorruptSSTableException((IOException)e.getCause(),
filename);
+}
else
+{
throw e;
+}
}
}
@@ -181,22 +191,39 @@ public class SSTableIdentityIterator implements
ComparableSSTableIdentityIterat
{
ColumnFamily cf = columnFamily.cloneMeShallow(containerFactory, false);
// since we already read column count, just pass that value and
continue deserialization
-IteratorOnDiskAtom iter = cf.metadata().getOnDiskIterator(in,
columnCount, flag, expireBefore, dataVersion);
-while (iter.hasNext())
-cf.addAtom(iter.next());
+try
+{
+IteratorOnDiskAtom iter = cf.metadata().getOnDiskIterator(in,
columnCount, flag, expireBefore, dataVersion);
+while (iter.hasNext())
+cf.addAtom(iter.next());
-if (validateColumns)
+if (validateColumns)
+{
+try
+{
+cf.metadata().validateColumns(cf);
+}
+catch (MarshalException e)
+{
+throw new RuntimeException(Error validating row + key,
e);
+}
+}
+return cf;
+}
+catch (IOError e)
{
-try
+// catch here b/c atomIterator is an AbstractIterator; hasNext
reads the value
+if (e.getCause() instanceof IOException)
{
-cf.metadata().validateColumns(cf);
+if (sstable != null)
+
[5/6] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
Merge branch 'cassandra-2.1' into cassandra-2.2 Conflicts: src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a30d8bd2 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a30d8bd2 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a30d8bd2 Branch: refs/heads/trunk Commit: a30d8bd212a9a2fb4499130d2493eca34c3715a2 Parents: 94a68a1 c0f96e1 Author: Sylvain Lebresne sylv...@datastax.com Authored: Tue Jun 2 14:57:44 2015 +0200 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue Jun 2 14:57:44 2015 +0200 -- CHANGES.txt | 1 + .../io/sstable/SSTableIdentityIterator.java | 10 .../io/sstable/format/SSTableReader.java| 57 +++- .../compaction/BlacklistingCompactionsTest.java | 17 +++--- 4 files changed, 67 insertions(+), 18 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/a30d8bd2/CHANGES.txt -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/a30d8bd2/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java -- diff --cc src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java index 6300749,498ad26..17f9a8d --- a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java +++ b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java @@@ -78,10 -81,12 +81,11 @@@ import org.apache.cassandra.serializers this.in = in; this.filename = filename; this.key = key; -this.dataSize = dataSize; this.flag = flag; this.validateColumns = checkData; + this.sstable = sstable; -Descriptor.Version dataVersion = sstable == null ? Descriptor.Version.CURRENT : sstable.descriptor.version; +Version dataVersion = sstable == null ? DatabaseDescriptor.getSSTableFormat().info.getLatestVersion() : sstable.descriptor.version; int expireBefore = (int) (System.currentTimeMillis() / 1000); columnFamily = ArrayBackedSortedColumns.factory.create(metadata);
[2/6] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1 Conflicts: CHANGES.txt src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java test/unit/org/apache/cassandra/db/compaction/BlacklistingCompactionsTest.java Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c0f96e1d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c0f96e1d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c0f96e1d Branch: refs/heads/trunk Commit: c0f96e1d46d664a9d4ca7f982d21cdfcaed1a24a Parents: 14a3324 9b10928 Author: Sylvain Lebresne sylv...@datastax.com Authored: Tue Jun 2 14:51:20 2015 +0200 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue Jun 2 14:51:20 2015 +0200 -- CHANGES.txt | 1 + .../io/sstable/SSTableIdentityIterator.java | 10 +++ .../cassandra/io/sstable/SSTableReader.java | 68 ++-- .../compaction/BlacklistingCompactionsTest.java | 17 +++-- 4 files changed, 69 insertions(+), 27 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0f96e1d/CHANGES.txt -- diff --cc CHANGES.txt index 71ce442,1aad965..4a7e174 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,37 -1,5 +1,38 @@@ -2.0.16: +2.1.6 + * Consistent error message when a table mixes counter and non-counter + columns (CASSANDRA-9492) + * Avoid getting unreadable keys during anticompaction (CASSANDRA-9508) + * (cqlsh) Better float precision by default (CASSANDRA-9224) + * Improve estimated row count (CASSANDRA-9107) + * Optimize range tombstone memory footprint (CASSANDRA-8603) + * Use configured gcgs in anticompaction (CASSANDRA-9397) + * Warn on misuse of unlogged batches (CASSANDRA-9282) + * Failure detector detects and ignores local pauses (CASSANDRA-9183) + * Add utility class to support for rate limiting a given log statement (CASSANDRA-9029) + * Add missing consistency levels to cassandra-stess (CASSANDRA-9361) + * Fix commitlog getCompletedTasks to not increment (CASSANDRA-9339) + * Fix for harmless exceptions logged as ERROR (CASSANDRA-8564) + * Delete processed sstables in sstablesplit/sstableupgrade (CASSANDRA-8606) + * Improve sstable exclusion from partition tombstones (CASSANDRA-9298) + * Validate the indexed column rather than the cell's contents for 2i (CASSANDRA-9057) + * Add support for top-k custom 2i queries (CASSANDRA-8717) + * Fix error when dropping table during compaction (CASSANDRA-9251) + * cassandra-stress supports validation operations over user profiles (CASSANDRA-8773) + * Add support for rate limiting log messages (CASSANDRA-9029) + * Log the partition key with tombstone warnings (CASSANDRA-8561) + * Reduce runWithCompactionsDisabled poll interval to 1ms (CASSANDRA-9271) + * Fix PITR commitlog replay (CASSANDRA-9195) + * GCInspector logs very different times (CASSANDRA-9124) + * Fix deleting from an empty list (CASSANDRA-9198) + * Update tuple and collection types that use a user-defined type when that UDT + is modified (CASSANDRA-9148, CASSANDRA-9192) + * Use higher timeout for prepair and snapshot in repair (CASSANDRA-9261) + * Fix anticompaction blocking ANTI_ENTROPY stage (CASSANDRA-9151) + * Repair waits for anticompaction to finish (CASSANDRA-9097) + * Fix streaming not holding ref when stream error (CASSANDRA-9295) + * Fix canonical view returning early opened SSTables (CASSANDRA-9396) +Merged from 2.0: + * Always mark sstable suspect when corrupted (CASSANDRA-9478) * Add database users and permissions to CQL3 documentation (CASSANDRA-7558) * Allow JVM_OPTS to be passed to standalone tools (CASSANDRA-5969) * Fix bad condition in RangeTombstoneList (CASSANDRA-9485) http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0f96e1d/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java -- diff --cc src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java index b784a7e,8b45005..498ad26 --- a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java +++ b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java @@@ -79,17 -95,18 +82,18 @@@ public class SSTableIdentityIterator im this.filename = filename; this.key = key; this.dataSize = dataSize; -this.expireBefore = (int)(System.currentTimeMillis() / 1000); this.flag = flag; this.validateColumns = checkData; -this.dataVersion = sstable == null ? Descriptor.Version.CURRENT : sstable.descriptor.version; + this.sstable = sstable; +Descriptor.Version dataVersion = sstable == null ? Descriptor.Version.CURRENT :
[1/6] cassandra git commit: Always mark sstable suspected on corruption
Repository: cassandra
Updated Branches:
refs/heads/trunk 525579d0d - f90207c06
Always mark sstable suspected on corruption
patch by slebresne; reviewed by benedict for CASSANDRA-9478
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9b10928c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9b10928c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9b10928c
Branch: refs/heads/trunk
Commit: 9b10928c159317160fb3049727679a48232b6041
Parents: 63819cb
Author: Sylvain Lebresne sylv...@datastax.com
Authored: Mon May 25 18:26:56 2015 +0200
Committer: Sylvain Lebresne sylv...@datastax.com
Committed: Tue Jun 2 14:46:09 2015 +0200
--
CHANGES.txt | 1 +
.../io/sstable/SSTableIdentityIterator.java | 45
.../compaction/BlacklistingCompactionsTest.java | 16 ---
3 files changed, 48 insertions(+), 14 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9b10928c/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index d23661d..1aad965 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.16:
+ * Always mark sstable suspect when corrupted (CASSANDRA-9478)
* Add database users and permissions to CQL3 documentation (CASSANDRA-7558)
* Allow JVM_OPTS to be passed to standalone tools (CASSANDRA-5969)
* Fix bad condition in RangeTombstoneList (CASSANDRA-9485)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9b10928c/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
--
diff --git
a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
index 52da9bb..8b45005 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
@@ -50,6 +50,9 @@ public class SSTableIdentityIterator implements
ComparableSSTableIdentityIterat
private final boolean validateColumns;
private final String filename;
+// Not every SSTableIdentifyIterator is attached to a sstable, so this can
be null.
+private final SSTableReader sstable;
+
/**
* Used to iterate through the columns of a row.
* @param sstable SSTable we are reading ffrom.
@@ -96,6 +99,7 @@ public class SSTableIdentityIterator implements
ComparableSSTableIdentityIterat
this.flag = flag;
this.validateColumns = checkData;
this.dataVersion = sstable == null ? Descriptor.Version.CURRENT :
sstable.descriptor.version;
+this.sstable = sstable;
try
{
@@ -132,9 +136,15 @@ public class SSTableIdentityIterator implements
ComparableSSTableIdentityIterat
{
// catch here b/c atomIterator is an AbstractIterator; hasNext
reads the value
if (e.getCause() instanceof IOException)
+{
+if (sstable != null)
+sstable.markSuspect();
throw new CorruptSSTableException((IOException)e.getCause(),
filename);
+}
else
+{
throw e;
+}
}
}
@@ -181,22 +191,39 @@ public class SSTableIdentityIterator implements
ComparableSSTableIdentityIterat
{
ColumnFamily cf = columnFamily.cloneMeShallow(containerFactory, false);
// since we already read column count, just pass that value and
continue deserialization
-IteratorOnDiskAtom iter = cf.metadata().getOnDiskIterator(in,
columnCount, flag, expireBefore, dataVersion);
-while (iter.hasNext())
-cf.addAtom(iter.next());
+try
+{
+IteratorOnDiskAtom iter = cf.metadata().getOnDiskIterator(in,
columnCount, flag, expireBefore, dataVersion);
+while (iter.hasNext())
+cf.addAtom(iter.next());
-if (validateColumns)
+if (validateColumns)
+{
+try
+{
+cf.metadata().validateColumns(cf);
+}
+catch (MarshalException e)
+{
+throw new RuntimeException(Error validating row + key,
e);
+}
+}
+return cf;
+}
+catch (IOError e)
{
-try
+// catch here b/c atomIterator is an AbstractIterator; hasNext
reads the value
+if (e.getCause() instanceof IOException)
{
-cf.metadata().validateColumns(cf);
+if (sstable != null)
+sstable.markSuspect();
[3/6] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
http://git-wip-us.apache.org/repos/asf/cassandra/blob/a30d8bd2/test/unit/org/apache/cassandra/db/compaction/BlacklistingCompactionsTest.java
--
diff --cc
test/unit/org/apache/cassandra/db/compaction/BlacklistingCompactionsTest.java
index 5420b1b,572ad36..2b6a62a
---
a/test/unit/org/apache/cassandra/db/compaction/BlacklistingCompactionsTest.java
+++
b/test/unit/org/apache/cassandra/db/compaction/BlacklistingCompactionsTest.java
@@@ -22,11 -22,8 +22,9 @@@ package org.apache.cassandra.db.compact
import java.io.RandomAccessFile;
- import java.util.Collection;
- import java.util.HashSet;
- import java.util.Set;
+ import java.util.*;
+import org.apache.cassandra.io.sstable.format.SSTableReader;
import org.junit.After;
import org.junit.AfterClass;
import org.junit.BeforeClass;
@@@ -43,12 -38,12 +41,13 @@@ import org.apache.cassandra.utils.ByteB
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
+ import static org.junit.Assert.assertTrue;
import static org.apache.cassandra.Util.cellname;
-public class BlacklistingCompactionsTest extends SchemaLoader
+public class BlacklistingCompactionsTest
{
-public static final String KEYSPACE = Keyspace1;
+private static final String KEYSPACE1 = BlacklistingCompactionsTest;
+private static final String CF_STANDARD1 = Standard1;
@After
public void leakDetect() throws InterruptedException
[6/6] cassandra git commit: Merge branch 'cassandra-2.2' into trunk
Merge branch 'cassandra-2.2' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f90207c0 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f90207c0 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f90207c0 Branch: refs/heads/trunk Commit: f90207c0661812ad1dc6bc6d7672898406cf9ec0 Parents: 525579d a30d8bd Author: Sylvain Lebresne sylv...@datastax.com Authored: Tue Jun 2 14:58:18 2015 +0200 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue Jun 2 14:58:18 2015 +0200 -- CHANGES.txt | 1 + .../io/sstable/SSTableIdentityIterator.java | 10 .../io/sstable/format/SSTableReader.java| 57 +++- .../compaction/BlacklistingCompactionsTest.java | 17 +++--- 4 files changed, 67 insertions(+), 18 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/f90207c0/CHANGES.txt --
[4/6] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
http://git-wip-us.apache.org/repos/asf/cassandra/blob/a30d8bd2/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
--
diff --cc src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
index ba634b6,000..f4a59a2
mode 100644,00..100644
--- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
@@@ -1,2142 -1,0 +1,2175 @@@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.io.sstable.format;
+
+import java.io.*;
+import java.nio.ByteBuffer;
+import java.util.*;
+import java.util.concurrent.*;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Predicate;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Ordering;
+import com.google.common.primitives.Longs;
+import com.google.common.util.concurrent.RateLimiter;
+
+import com.clearspring.analytics.stream.cardinality.CardinalityMergeException;
+import com.clearspring.analytics.stream.cardinality.HyperLogLogPlus;
+import com.clearspring.analytics.stream.cardinality.ICardinality;
+import org.apache.cassandra.cache.CachingOptions;
+import org.apache.cassandra.cache.InstrumentingCache;
+import org.apache.cassandra.cache.KeyCacheKey;
+import org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor;
+import org.apache.cassandra.concurrent.ScheduledExecutors;
+import org.apache.cassandra.config.*;
+import org.apache.cassandra.db.*;
+import org.apache.cassandra.db.columniterator.OnDiskAtomIterator;
+import org.apache.cassandra.db.commitlog.ReplayPosition;
+import org.apache.cassandra.db.composites.CellName;
+import org.apache.cassandra.db.filter.ColumnSlice;
+import org.apache.cassandra.db.index.SecondaryIndex;
+import org.apache.cassandra.db.lifecycle.Tracker;
+import org.apache.cassandra.dht.*;
+import org.apache.cassandra.io.compress.CompressionMetadata;
+import org.apache.cassandra.io.sstable.*;
+import org.apache.cassandra.io.sstable.metadata.*;
+import org.apache.cassandra.io.util.*;
+import org.apache.cassandra.metrics.RestorableMeter;
+import org.apache.cassandra.metrics.StorageMetrics;
+import org.apache.cassandra.service.ActiveRepairService;
+import org.apache.cassandra.service.CacheService;
+import org.apache.cassandra.service.StorageService;
+import org.apache.cassandra.utils.*;
+import org.apache.cassandra.utils.concurrent.OpOrder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.cassandra.utils.concurrent.Ref;
+import org.apache.cassandra.utils.concurrent.SelfRefCounted;
+
+import static
org.apache.cassandra.db.Directories.SECONDARY_INDEX_NAME_SEPARATOR;
+
+/**
+ * An SSTableReader can be constructed in a number of places, but typically
is either
+ * read from disk at startup, or constructed from a flushed memtable, or
after compaction
+ * to replace some existing sstables. However once created, an sstablereader
may also be modified.
+ *
+ * A reader's OpenReason describes its current stage in its lifecycle, as
follows:
+ *
+ *
+ * pre {@code
+ * NORMAL
+ * From: None= Reader has been read from disk, either at
startup or from a flushed memtable
+ * EARLY = Reader is the final result of a compaction
+ * MOVED_START = Reader WAS being compacted, but this failed and
it has been restored to NORMAL status
+ *
+ * EARLY
+ * From: None= Reader is a compaction replacement that is
either incomplete and has been opened
+ *to represent its partial result status, or has
been finished but the compaction
+ *it is a part of has not yet completed fully
+ * EARLY = Same as from None, only it is not the first
time it has been
+ *
+ * MOVED_START
+ * From: NORMAL = Reader is being compacted. This compaction has
not finished, but the compaction result
+ *
[jira] [Updated] (CASSANDRA-9527) Cannot create secondary index on a table WITH COMPACT STORAGE
[ https://issues.apache.org/jira/browse/CASSANDRA-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-9527: --- Assignee: Benjamin Lerer Cannot create secondary index on a table WITH COMPACT STORAGE - Key: CASSANDRA-9527 URL: https://issues.apache.org/jira/browse/CASSANDRA-9527 Project: Cassandra Issue Type: Bug Components: Core Environment: cassandra v2.0.13 Reporter: fuggy_yama Assignee: Benjamin Lerer Priority: Minor Fix For: 2.0.x In CASSANDRA-8156 it is said that secondary indexes are not allowed on clustering columns in COMPACT tables. However, I found that it is not possible to create a secondary index on the value column in a COMPACT table: CREATE TABLE t ( a INT, b INT, c INT, PRIMARY KEY (a, b) ) WITH COMPACT STORAGE; CREATE INDEX ON t (c); *Bad Request: Secondary indexes are not supported on PRIMARY KEY columns in COMPACT STORAGE tables* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[1/2] cassandra git commit: Fix custom payload encoding decoding to match protocol spec
Repository: cassandra
Updated Branches:
refs/heads/trunk f90207c06 - ae2daea52
Fix custom payload encoding decoding to match protocol spec
patch by blerer; reviewed by slebresne for CASSANDRA-9515
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6f93bd1f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6f93bd1f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6f93bd1f
Branch: refs/heads/trunk
Commit: 6f93bd1f65888104e33da2f9f01056b6115952e5
Parents: a30d8bd
Author: Benjamin Lerer benjamin.le...@datastax.com
Authored: Tue Jun 2 15:34:31 2015 +0200
Committer: Sylvain Lebresne sylv...@datastax.com
Committed: Tue Jun 2 15:34:31 2015 +0200
--
CHANGES.txt | 1 +
.../CustomPayloadMirroringQueryHandler.java | 18 +++--
.../org/apache/cassandra/cql3/QueryHandler.java | 20 --
.../apache/cassandra/cql3/QueryProcessor.java | 22 --
.../org/apache/cassandra/transport/CBUtil.java | 18 ++---
.../org/apache/cassandra/transport/Message.java | 13 ++--
.../cassandra/transport/MessagePayloadTest.java | 73
7 files changed, 110 insertions(+), 55 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/6f93bd1f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 7f0ef51..db94c76 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.2
+ * Fix custom payload coding/decoding to match the spec (CASSANDRA-9515)
* ant test-all results incomplete when parsed (CASSANDRA-9463)
* Disallow frozen types in function arguments and return types for
clarity (CASSANDRA-9411)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/6f93bd1f/src/java/org/apache/cassandra/cql3/CustomPayloadMirroringQueryHandler.java
--
diff --git
a/src/java/org/apache/cassandra/cql3/CustomPayloadMirroringQueryHandler.java
b/src/java/org/apache/cassandra/cql3/CustomPayloadMirroringQueryHandler.java
index 3930e9c..02a6df9 100644
--- a/src/java/org/apache/cassandra/cql3/CustomPayloadMirroringQueryHandler.java
+++ b/src/java/org/apache/cassandra/cql3/CustomPayloadMirroringQueryHandler.java
@@ -17,6 +17,7 @@
*/
package org.apache.cassandra.cql3;
+import java.nio.ByteBuffer;
import java.util.Map;
import org.apache.cassandra.cql3.statements.BatchStatement;
@@ -34,14 +35,17 @@ public class CustomPayloadMirroringQueryHandler implements
QueryHandler
{
static QueryProcessor queryProcessor = QueryProcessor.instance;
-public ResultMessage process(String query, QueryState state, QueryOptions
options, MapString, byte[] customPayload)
+public ResultMessage process(String query,
+ QueryState state,
+ QueryOptions options,
+ MapString, ByteBuffer customPayload)
{
ResultMessage result = queryProcessor.process(query, state, options,
customPayload);
result.setCustomPayload(customPayload);
return result;
}
-public ResultMessage.Prepared prepare(String query, QueryState state,
MapString, byte[] customPayload)
+public ResultMessage.Prepared prepare(String query, QueryState state,
MapString, ByteBuffer customPayload)
{
ResultMessage.Prepared prepared = queryProcessor.prepare(query, state,
customPayload);
prepared.setCustomPayload(customPayload);
@@ -58,14 +62,20 @@ public class CustomPayloadMirroringQueryHandler implements
QueryHandler
return queryProcessor.getPreparedForThrift(id);
}
-public ResultMessage processPrepared(CQLStatement statement, QueryState
state, QueryOptions options, MapString, byte[] customPayload)
+public ResultMessage processPrepared(CQLStatement statement,
+ QueryState state,
+ QueryOptions options,
+ MapString, ByteBuffer customPayload)
{
ResultMessage result = queryProcessor.processPrepared(statement,
state, options, customPayload);
result.setCustomPayload(customPayload);
return result;
}
-public ResultMessage processBatch(BatchStatement statement, QueryState
state, BatchQueryOptions options, MapString, byte[] customPayload)
+public ResultMessage processBatch(BatchStatement statement,
+ QueryState state,
+ BatchQueryOptions options,
+ MapString, ByteBuffer customPayload)
{
ResultMessage result = queryProcessor.processBatch(statement, state,
options, customPayload);
[5/5] cassandra git commit: Merge branch 'cassandra-2.2' into trunk
Merge branch 'cassandra-2.2' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e27e9741 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e27e9741 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e27e9741 Branch: refs/heads/trunk Commit: e27e974139d371c00c49ad70504f0283caaba356 Parents: ae2daea 4cfb17f Author: Brandon Williams brandonwilli...@apache.org Authored: Tue Jun 2 08:43:33 2015 -0500 Committer: Brandon Williams brandonwilli...@apache.org Committed: Tue Jun 2 08:43:33 2015 -0500 -- CHANGES.txt | 1 + src/java/org/apache/cassandra/gms/Gossiper.java | 16 -- .../cassandra/service/StorageService.java | 54 +--- 3 files changed, 26 insertions(+), 45 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/e27e9741/CHANGES.txt -- diff --cc CHANGES.txt index 474476f,42846d7..4597175 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,10 -1,5 +1,11 @@@ +3.0: + * Decommissioned nodes will not rejoin the cluster (CASSANDRA-8801) + * Change gossip stabilization to use endpoit size (CASSANDRA-9401) + * Change default garbage collector to G1 (CASSANDRA-7486) + + 2.2 + * Clean up gossiper logic for old versions (CASSANDRA-9370) * Fix custom payload coding/decoding to match the spec (CASSANDRA-9515) * ant test-all results incomplete when parsed (CASSANDRA-9463) * Disallow frozen types in function arguments and return types for http://git-wip-us.apache.org/repos/asf/cassandra/blob/e27e9741/src/java/org/apache/cassandra/service/StorageService.java --
[2/5] cassandra git commit: Clean up gossiper logic for old versions
Clean up gossiper logic for old versions
Patch by brandonwilliams, reviewed by aleksey for CASSANDRA-9370
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/23d7a558
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/23d7a558
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/23d7a558
Branch: refs/heads/trunk
Commit: 23d7a558ee824a22d382e1b645814638a2cd2985
Parents: 94a68a1
Author: Brandon Williams brandonwilli...@apache.org
Authored: Tue Jun 2 08:42:28 2015 -0500
Committer: Brandon Williams brandonwilli...@apache.org
Committed: Tue Jun 2 08:42:28 2015 -0500
--
CHANGES.txt | 1 +
src/java/org/apache/cassandra/gms/Gossiper.java | 16 --
.../cassandra/service/StorageService.java | 54 +---
3 files changed, 26 insertions(+), 45 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/23d7a558/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 165f76b..41f5cfb 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.2
+ * Clean up gossiper logic for old versions (CASSANDRA-9370)
* ant test-all results incomplete when parsed (CASSANDRA-9463)
* Disallow frozen types in function arguments and return types for
clarity (CASSANDRA-9411)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/23d7a558/src/java/org/apache/cassandra/gms/Gossiper.java
--
diff --git a/src/java/org/apache/cassandra/gms/Gossiper.java
b/src/java/org/apache/cassandra/gms/Gossiper.java
index 0c449ec..e131da6 100644
--- a/src/java/org/apache/cassandra/gms/Gossiper.java
+++ b/src/java/org/apache/cassandra/gms/Gossiper.java
@@ -798,24 +798,8 @@ public class Gossiper implements
IFailureDetectionEventListener, GossiperMBean
return endpointStateMap.entrySet();
}
-public boolean usesHostId(InetAddress endpoint)
-{
-if (MessagingService.instance().knowsVersion(endpoint))
-return true;
-else if
(getEndpointStateForEndpoint(endpoint).getApplicationState(ApplicationState.NET_VERSION)
!= null)
-return true;
-return false;
-}
-
-public boolean usesVnodes(InetAddress endpoint)
-{
-return usesHostId(endpoint)
getEndpointStateForEndpoint(endpoint).getApplicationState(ApplicationState.TOKENS)
!= null;
-}
-
public UUID getHostId(InetAddress endpoint)
{
-if (!usesHostId(endpoint))
-throw new RuntimeException(Host + endpoint + does not use
new-style tokens!);
return
UUID.fromString(getEndpointStateForEndpoint(endpoint).getApplicationState(ApplicationState.HOST_ID).value);
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/23d7a558/src/java/org/apache/cassandra/service/StorageService.java
--
diff --git a/src/java/org/apache/cassandra/service/StorageService.java
b/src/java/org/apache/cassandra/service/StorageService.java
index bfbf1a8..8afb93f 100644
--- a/src/java/org/apache/cassandra/service/StorageService.java
+++ b/src/java/org/apache/cassandra/service/StorageService.java
@@ -1825,8 +1825,7 @@ public class StorageService extends
NotificationBroadcasterSupport implements IE
tokenMetadata.addBootstrapTokens(tokens, endpoint);
PendingRangeCalculatorService.instance.update();
-if (Gossiper.instance.usesHostId(endpoint))
-tokenMetadata.updateHostId(Gossiper.instance.getHostId(endpoint),
endpoint);
+tokenMetadata.updateHostId(Gossiper.instance.getHostId(endpoint),
endpoint);
}
/**
@@ -1855,39 +1854,36 @@ public class StorageService extends
NotificationBroadcasterSupport implements IE
updatePeerInfo(endpoint);
// Order Matters, TM.updateHostID() should be called before
TM.updateNormalToken(), (see CASSANDRA-4300).
-if (Gossiper.instance.usesHostId(endpoint))
+UUID hostId = Gossiper.instance.getHostId(endpoint);
+InetAddress existing = tokenMetadata.getEndpointForHostId(hostId);
+if (replacing
Gossiper.instance.getEndpointStateForEndpoint(DatabaseDescriptor.getReplaceAddress())
!= null
(hostId.equals(Gossiper.instance.getHostId(DatabaseDescriptor.getReplaceAddress()
+logger.warn(Not updating token metadata for {} because I am
replacing it, endpoint);
+else
{
-UUID hostId = Gossiper.instance.getHostId(endpoint);
-InetAddress existing = tokenMetadata.getEndpointForHostId(hostId);
-if (replacing
Gossiper.instance.getEndpointStateForEndpoint(DatabaseDescriptor.getReplaceAddress())
!= null
[jira] [Updated] (CASSANDRA-9527) Cannot create secondary index on a table WITH COMPACT STORAGE
[ https://issues.apache.org/jira/browse/CASSANDRA-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-9527: --- Reproduced In: 2.0.13 Fix Version/s: 2.0.x Cannot create secondary index on a table WITH COMPACT STORAGE - Key: CASSANDRA-9527 URL: https://issues.apache.org/jira/browse/CASSANDRA-9527 Project: Cassandra Issue Type: Bug Components: Core Environment: cassandra v2.0.13 Reporter: fuggy_yama Priority: Minor Fix For: 2.0.x In CASSANDRA-8156 it is said that secondary indexes are not allowed on clustering columns in COMPACT tables. However, I found that it is not possible to create a secondary index on the value column in a COMPACT table: CREATE TABLE t ( a INT, b INT, c INT, PRIMARY KEY (a, b) ) WITH COMPACT STORAGE; CREATE INDEX ON t (c); *Bad Request: Secondary indexes are not supported on PRIMARY KEY columns in COMPACT STORAGE tables* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[2/2] cassandra git commit: Merge branch 'cassandra-2.2' into trunk
Merge branch 'cassandra-2.2' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ae2daea5 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ae2daea5 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ae2daea5 Branch: refs/heads/trunk Commit: ae2daea52456bd460663d7aecff9180810063834 Parents: f90207c 6f93bd1 Author: Sylvain Lebresne sylv...@datastax.com Authored: Tue Jun 2 15:35:37 2015 +0200 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue Jun 2 15:35:37 2015 +0200 -- CHANGES.txt | 1 + .../CustomPayloadMirroringQueryHandler.java | 18 +++-- .../org/apache/cassandra/cql3/QueryHandler.java | 20 -- .../apache/cassandra/cql3/QueryProcessor.java | 22 -- .../org/apache/cassandra/transport/CBUtil.java | 18 ++--- .../org/apache/cassandra/transport/Message.java | 13 ++-- .../cassandra/transport/MessagePayloadTest.java | 73 7 files changed, 110 insertions(+), 55 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/ae2daea5/CHANGES.txt -- diff --cc CHANGES.txt index 59bdadc,db94c76..474476f --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,10 -1,5 +1,11 @@@ +3.0: + * Decommissioned nodes will not rejoin the cluster (CASSANDRA-8801) + * Change gossip stabilization to use endpoit size (CASSANDRA-9401) + * Change default garbage collector to G1 (CASSANDRA-7486) + + 2.2 + * Fix custom payload coding/decoding to match the spec (CASSANDRA-9515) * ant test-all results incomplete when parsed (CASSANDRA-9463) * Disallow frozen types in function arguments and return types for clarity (CASSANDRA-9411)
cassandra git commit: Fix custom payload encoding decoding to match protocol spec
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.2 a30d8bd21 - 6f93bd1f6
Fix custom payload encoding decoding to match protocol spec
patch by blerer; reviewed by slebresne for CASSANDRA-9515
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6f93bd1f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6f93bd1f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6f93bd1f
Branch: refs/heads/cassandra-2.2
Commit: 6f93bd1f65888104e33da2f9f01056b6115952e5
Parents: a30d8bd
Author: Benjamin Lerer benjamin.le...@datastax.com
Authored: Tue Jun 2 15:34:31 2015 +0200
Committer: Sylvain Lebresne sylv...@datastax.com
Committed: Tue Jun 2 15:34:31 2015 +0200
--
CHANGES.txt | 1 +
.../CustomPayloadMirroringQueryHandler.java | 18 +++--
.../org/apache/cassandra/cql3/QueryHandler.java | 20 --
.../apache/cassandra/cql3/QueryProcessor.java | 22 --
.../org/apache/cassandra/transport/CBUtil.java | 18 ++---
.../org/apache/cassandra/transport/Message.java | 13 ++--
.../cassandra/transport/MessagePayloadTest.java | 73
7 files changed, 110 insertions(+), 55 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/6f93bd1f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 7f0ef51..db94c76 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.2
+ * Fix custom payload coding/decoding to match the spec (CASSANDRA-9515)
* ant test-all results incomplete when parsed (CASSANDRA-9463)
* Disallow frozen types in function arguments and return types for
clarity (CASSANDRA-9411)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/6f93bd1f/src/java/org/apache/cassandra/cql3/CustomPayloadMirroringQueryHandler.java
--
diff --git
a/src/java/org/apache/cassandra/cql3/CustomPayloadMirroringQueryHandler.java
b/src/java/org/apache/cassandra/cql3/CustomPayloadMirroringQueryHandler.java
index 3930e9c..02a6df9 100644
--- a/src/java/org/apache/cassandra/cql3/CustomPayloadMirroringQueryHandler.java
+++ b/src/java/org/apache/cassandra/cql3/CustomPayloadMirroringQueryHandler.java
@@ -17,6 +17,7 @@
*/
package org.apache.cassandra.cql3;
+import java.nio.ByteBuffer;
import java.util.Map;
import org.apache.cassandra.cql3.statements.BatchStatement;
@@ -34,14 +35,17 @@ public class CustomPayloadMirroringQueryHandler implements
QueryHandler
{
static QueryProcessor queryProcessor = QueryProcessor.instance;
-public ResultMessage process(String query, QueryState state, QueryOptions
options, MapString, byte[] customPayload)
+public ResultMessage process(String query,
+ QueryState state,
+ QueryOptions options,
+ MapString, ByteBuffer customPayload)
{
ResultMessage result = queryProcessor.process(query, state, options,
customPayload);
result.setCustomPayload(customPayload);
return result;
}
-public ResultMessage.Prepared prepare(String query, QueryState state,
MapString, byte[] customPayload)
+public ResultMessage.Prepared prepare(String query, QueryState state,
MapString, ByteBuffer customPayload)
{
ResultMessage.Prepared prepared = queryProcessor.prepare(query, state,
customPayload);
prepared.setCustomPayload(customPayload);
@@ -58,14 +62,20 @@ public class CustomPayloadMirroringQueryHandler implements
QueryHandler
return queryProcessor.getPreparedForThrift(id);
}
-public ResultMessage processPrepared(CQLStatement statement, QueryState
state, QueryOptions options, MapString, byte[] customPayload)
+public ResultMessage processPrepared(CQLStatement statement,
+ QueryState state,
+ QueryOptions options,
+ MapString, ByteBuffer customPayload)
{
ResultMessage result = queryProcessor.processPrepared(statement,
state, options, customPayload);
result.setCustomPayload(customPayload);
return result;
}
-public ResultMessage processBatch(BatchStatement statement, QueryState
state, BatchQueryOptions options, MapString, byte[] customPayload)
+public ResultMessage processBatch(BatchStatement statement,
+ QueryState state,
+ BatchQueryOptions options,
+ MapString, ByteBuffer customPayload)
{
ResultMessage result = queryProcessor.processBatch(statement, state,
options,
[2/5] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1 Conflicts: CHANGES.txt src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java test/unit/org/apache/cassandra/db/compaction/BlacklistingCompactionsTest.java Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c0f96e1d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c0f96e1d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c0f96e1d Branch: refs/heads/cassandra-2.2 Commit: c0f96e1d46d664a9d4ca7f982d21cdfcaed1a24a Parents: 14a3324 9b10928 Author: Sylvain Lebresne sylv...@datastax.com Authored: Tue Jun 2 14:51:20 2015 +0200 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue Jun 2 14:51:20 2015 +0200 -- CHANGES.txt | 1 + .../io/sstable/SSTableIdentityIterator.java | 10 +++ .../cassandra/io/sstable/SSTableReader.java | 68 ++-- .../compaction/BlacklistingCompactionsTest.java | 17 +++-- 4 files changed, 69 insertions(+), 27 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0f96e1d/CHANGES.txt -- diff --cc CHANGES.txt index 71ce442,1aad965..4a7e174 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,37 -1,5 +1,38 @@@ -2.0.16: +2.1.6 + * Consistent error message when a table mixes counter and non-counter + columns (CASSANDRA-9492) + * Avoid getting unreadable keys during anticompaction (CASSANDRA-9508) + * (cqlsh) Better float precision by default (CASSANDRA-9224) + * Improve estimated row count (CASSANDRA-9107) + * Optimize range tombstone memory footprint (CASSANDRA-8603) + * Use configured gcgs in anticompaction (CASSANDRA-9397) + * Warn on misuse of unlogged batches (CASSANDRA-9282) + * Failure detector detects and ignores local pauses (CASSANDRA-9183) + * Add utility class to support for rate limiting a given log statement (CASSANDRA-9029) + * Add missing consistency levels to cassandra-stess (CASSANDRA-9361) + * Fix commitlog getCompletedTasks to not increment (CASSANDRA-9339) + * Fix for harmless exceptions logged as ERROR (CASSANDRA-8564) + * Delete processed sstables in sstablesplit/sstableupgrade (CASSANDRA-8606) + * Improve sstable exclusion from partition tombstones (CASSANDRA-9298) + * Validate the indexed column rather than the cell's contents for 2i (CASSANDRA-9057) + * Add support for top-k custom 2i queries (CASSANDRA-8717) + * Fix error when dropping table during compaction (CASSANDRA-9251) + * cassandra-stress supports validation operations over user profiles (CASSANDRA-8773) + * Add support for rate limiting log messages (CASSANDRA-9029) + * Log the partition key with tombstone warnings (CASSANDRA-8561) + * Reduce runWithCompactionsDisabled poll interval to 1ms (CASSANDRA-9271) + * Fix PITR commitlog replay (CASSANDRA-9195) + * GCInspector logs very different times (CASSANDRA-9124) + * Fix deleting from an empty list (CASSANDRA-9198) + * Update tuple and collection types that use a user-defined type when that UDT + is modified (CASSANDRA-9148, CASSANDRA-9192) + * Use higher timeout for prepair and snapshot in repair (CASSANDRA-9261) + * Fix anticompaction blocking ANTI_ENTROPY stage (CASSANDRA-9151) + * Repair waits for anticompaction to finish (CASSANDRA-9097) + * Fix streaming not holding ref when stream error (CASSANDRA-9295) + * Fix canonical view returning early opened SSTables (CASSANDRA-9396) +Merged from 2.0: + * Always mark sstable suspect when corrupted (CASSANDRA-9478) * Add database users and permissions to CQL3 documentation (CASSANDRA-7558) * Allow JVM_OPTS to be passed to standalone tools (CASSANDRA-5969) * Fix bad condition in RangeTombstoneList (CASSANDRA-9485) http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0f96e1d/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java -- diff --cc src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java index b784a7e,8b45005..498ad26 --- a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java +++ b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java @@@ -79,17 -95,18 +82,18 @@@ public class SSTableIdentityIterator im this.filename = filename; this.key = key; this.dataSize = dataSize; -this.expireBefore = (int)(System.currentTimeMillis() / 1000); this.flag = flag; this.validateColumns = checkData; -this.dataVersion = sstable == null ? Descriptor.Version.CURRENT : sstable.descriptor.version; + this.sstable = sstable; +Descriptor.Version dataVersion = sstable == null ?
[5/5] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
Merge branch 'cassandra-2.1' into cassandra-2.2 Conflicts: src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a30d8bd2 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a30d8bd2 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a30d8bd2 Branch: refs/heads/cassandra-2.2 Commit: a30d8bd212a9a2fb4499130d2493eca34c3715a2 Parents: 94a68a1 c0f96e1 Author: Sylvain Lebresne sylv...@datastax.com Authored: Tue Jun 2 14:57:44 2015 +0200 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue Jun 2 14:57:44 2015 +0200 -- CHANGES.txt | 1 + .../io/sstable/SSTableIdentityIterator.java | 10 .../io/sstable/format/SSTableReader.java| 57 +++- .../compaction/BlacklistingCompactionsTest.java | 17 +++--- 4 files changed, 67 insertions(+), 18 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/a30d8bd2/CHANGES.txt -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/a30d8bd2/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java -- diff --cc src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java index 6300749,498ad26..17f9a8d --- a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java +++ b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java @@@ -78,10 -81,12 +81,11 @@@ import org.apache.cassandra.serializers this.in = in; this.filename = filename; this.key = key; -this.dataSize = dataSize; this.flag = flag; this.validateColumns = checkData; + this.sstable = sstable; -Descriptor.Version dataVersion = sstable == null ? Descriptor.Version.CURRENT : sstable.descriptor.version; +Version dataVersion = sstable == null ? DatabaseDescriptor.getSSTableFormat().info.getLatestVersion() : sstable.descriptor.version; int expireBefore = (int) (System.currentTimeMillis() / 1000); columnFamily = ArrayBackedSortedColumns.factory.create(metadata);
[4/5] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
http://git-wip-us.apache.org/repos/asf/cassandra/blob/a30d8bd2/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
--
diff --cc src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
index ba634b6,000..f4a59a2
mode 100644,00..100644
--- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
@@@ -1,2142 -1,0 +1,2175 @@@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.io.sstable.format;
+
+import java.io.*;
+import java.nio.ByteBuffer;
+import java.util.*;
+import java.util.concurrent.*;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Predicate;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Ordering;
+import com.google.common.primitives.Longs;
+import com.google.common.util.concurrent.RateLimiter;
+
+import com.clearspring.analytics.stream.cardinality.CardinalityMergeException;
+import com.clearspring.analytics.stream.cardinality.HyperLogLogPlus;
+import com.clearspring.analytics.stream.cardinality.ICardinality;
+import org.apache.cassandra.cache.CachingOptions;
+import org.apache.cassandra.cache.InstrumentingCache;
+import org.apache.cassandra.cache.KeyCacheKey;
+import org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor;
+import org.apache.cassandra.concurrent.ScheduledExecutors;
+import org.apache.cassandra.config.*;
+import org.apache.cassandra.db.*;
+import org.apache.cassandra.db.columniterator.OnDiskAtomIterator;
+import org.apache.cassandra.db.commitlog.ReplayPosition;
+import org.apache.cassandra.db.composites.CellName;
+import org.apache.cassandra.db.filter.ColumnSlice;
+import org.apache.cassandra.db.index.SecondaryIndex;
+import org.apache.cassandra.db.lifecycle.Tracker;
+import org.apache.cassandra.dht.*;
+import org.apache.cassandra.io.compress.CompressionMetadata;
+import org.apache.cassandra.io.sstable.*;
+import org.apache.cassandra.io.sstable.metadata.*;
+import org.apache.cassandra.io.util.*;
+import org.apache.cassandra.metrics.RestorableMeter;
+import org.apache.cassandra.metrics.StorageMetrics;
+import org.apache.cassandra.service.ActiveRepairService;
+import org.apache.cassandra.service.CacheService;
+import org.apache.cassandra.service.StorageService;
+import org.apache.cassandra.utils.*;
+import org.apache.cassandra.utils.concurrent.OpOrder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.cassandra.utils.concurrent.Ref;
+import org.apache.cassandra.utils.concurrent.SelfRefCounted;
+
+import static
org.apache.cassandra.db.Directories.SECONDARY_INDEX_NAME_SEPARATOR;
+
+/**
+ * An SSTableReader can be constructed in a number of places, but typically
is either
+ * read from disk at startup, or constructed from a flushed memtable, or
after compaction
+ * to replace some existing sstables. However once created, an sstablereader
may also be modified.
+ *
+ * A reader's OpenReason describes its current stage in its lifecycle, as
follows:
+ *
+ *
+ * pre {@code
+ * NORMAL
+ * From: None= Reader has been read from disk, either at
startup or from a flushed memtable
+ * EARLY = Reader is the final result of a compaction
+ * MOVED_START = Reader WAS being compacted, but this failed and
it has been restored to NORMAL status
+ *
+ * EARLY
+ * From: None= Reader is a compaction replacement that is
either incomplete and has been opened
+ *to represent its partial result status, or has
been finished but the compaction
+ *it is a part of has not yet completed fully
+ * EARLY = Same as from None, only it is not the first
time it has been
+ *
+ * MOVED_START
+ * From: NORMAL = Reader is being compacted. This compaction has
not finished, but the compaction result
+ *
[3/5] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
http://git-wip-us.apache.org/repos/asf/cassandra/blob/a30d8bd2/test/unit/org/apache/cassandra/db/compaction/BlacklistingCompactionsTest.java
--
diff --cc
test/unit/org/apache/cassandra/db/compaction/BlacklistingCompactionsTest.java
index 5420b1b,572ad36..2b6a62a
---
a/test/unit/org/apache/cassandra/db/compaction/BlacklistingCompactionsTest.java
+++
b/test/unit/org/apache/cassandra/db/compaction/BlacklistingCompactionsTest.java
@@@ -22,11 -22,8 +22,9 @@@ package org.apache.cassandra.db.compact
import java.io.RandomAccessFile;
- import java.util.Collection;
- import java.util.HashSet;
- import java.util.Set;
+ import java.util.*;
+import org.apache.cassandra.io.sstable.format.SSTableReader;
import org.junit.After;
import org.junit.AfterClass;
import org.junit.BeforeClass;
@@@ -43,12 -38,12 +41,13 @@@ import org.apache.cassandra.utils.ByteB
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
+ import static org.junit.Assert.assertTrue;
import static org.apache.cassandra.Util.cellname;
-public class BlacklistingCompactionsTest extends SchemaLoader
+public class BlacklistingCompactionsTest
{
-public static final String KEYSPACE = Keyspace1;
+private static final String KEYSPACE1 = BlacklistingCompactionsTest;
+private static final String CF_STANDARD1 = Standard1;
@After
public void leakDetect() throws InterruptedException
[1/5] cassandra git commit: Always mark sstable suspected on corruption
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.2 94a68a17c - a30d8bd21
Always mark sstable suspected on corruption
patch by slebresne; reviewed by benedict for CASSANDRA-9478
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9b10928c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9b10928c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9b10928c
Branch: refs/heads/cassandra-2.2
Commit: 9b10928c159317160fb3049727679a48232b6041
Parents: 63819cb
Author: Sylvain Lebresne sylv...@datastax.com
Authored: Mon May 25 18:26:56 2015 +0200
Committer: Sylvain Lebresne sylv...@datastax.com
Committed: Tue Jun 2 14:46:09 2015 +0200
--
CHANGES.txt | 1 +
.../io/sstable/SSTableIdentityIterator.java | 45
.../compaction/BlacklistingCompactionsTest.java | 16 ---
3 files changed, 48 insertions(+), 14 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9b10928c/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index d23661d..1aad965 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.16:
+ * Always mark sstable suspect when corrupted (CASSANDRA-9478)
* Add database users and permissions to CQL3 documentation (CASSANDRA-7558)
* Allow JVM_OPTS to be passed to standalone tools (CASSANDRA-5969)
* Fix bad condition in RangeTombstoneList (CASSANDRA-9485)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9b10928c/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
--
diff --git
a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
index 52da9bb..8b45005 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
@@ -50,6 +50,9 @@ public class SSTableIdentityIterator implements
ComparableSSTableIdentityIterat
private final boolean validateColumns;
private final String filename;
+// Not every SSTableIdentifyIterator is attached to a sstable, so this can
be null.
+private final SSTableReader sstable;
+
/**
* Used to iterate through the columns of a row.
* @param sstable SSTable we are reading ffrom.
@@ -96,6 +99,7 @@ public class SSTableIdentityIterator implements
ComparableSSTableIdentityIterat
this.flag = flag;
this.validateColumns = checkData;
this.dataVersion = sstable == null ? Descriptor.Version.CURRENT :
sstable.descriptor.version;
+this.sstable = sstable;
try
{
@@ -132,9 +136,15 @@ public class SSTableIdentityIterator implements
ComparableSSTableIdentityIterat
{
// catch here b/c atomIterator is an AbstractIterator; hasNext
reads the value
if (e.getCause() instanceof IOException)
+{
+if (sstable != null)
+sstable.markSuspect();
throw new CorruptSSTableException((IOException)e.getCause(),
filename);
+}
else
+{
throw e;
+}
}
}
@@ -181,22 +191,39 @@ public class SSTableIdentityIterator implements
ComparableSSTableIdentityIterat
{
ColumnFamily cf = columnFamily.cloneMeShallow(containerFactory, false);
// since we already read column count, just pass that value and
continue deserialization
-IteratorOnDiskAtom iter = cf.metadata().getOnDiskIterator(in,
columnCount, flag, expireBefore, dataVersion);
-while (iter.hasNext())
-cf.addAtom(iter.next());
+try
+{
+IteratorOnDiskAtom iter = cf.metadata().getOnDiskIterator(in,
columnCount, flag, expireBefore, dataVersion);
+while (iter.hasNext())
+cf.addAtom(iter.next());
-if (validateColumns)
+if (validateColumns)
+{
+try
+{
+cf.metadata().validateColumns(cf);
+}
+catch (MarshalException e)
+{
+throw new RuntimeException(Error validating row + key,
e);
+}
+}
+return cf;
+}
+catch (IOError e)
{
-try
+// catch here b/c atomIterator is an AbstractIterator; hasNext
reads the value
+if (e.getCause() instanceof IOException)
{
-cf.metadata().validateColumns(cf);
+if (sstable != null)
+
cassandra git commit: Always mark sstable suspected on corruption
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.0 63819cbee - 9b10928c1
Always mark sstable suspected on corruption
patch by slebresne; reviewed by benedict for CASSANDRA-9478
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9b10928c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9b10928c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9b10928c
Branch: refs/heads/cassandra-2.0
Commit: 9b10928c159317160fb3049727679a48232b6041
Parents: 63819cb
Author: Sylvain Lebresne sylv...@datastax.com
Authored: Mon May 25 18:26:56 2015 +0200
Committer: Sylvain Lebresne sylv...@datastax.com
Committed: Tue Jun 2 14:46:09 2015 +0200
--
CHANGES.txt | 1 +
.../io/sstable/SSTableIdentityIterator.java | 45
.../compaction/BlacklistingCompactionsTest.java | 16 ---
3 files changed, 48 insertions(+), 14 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9b10928c/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index d23661d..1aad965 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.16:
+ * Always mark sstable suspect when corrupted (CASSANDRA-9478)
* Add database users and permissions to CQL3 documentation (CASSANDRA-7558)
* Allow JVM_OPTS to be passed to standalone tools (CASSANDRA-5969)
* Fix bad condition in RangeTombstoneList (CASSANDRA-9485)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9b10928c/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
--
diff --git
a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
index 52da9bb..8b45005 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
@@ -50,6 +50,9 @@ public class SSTableIdentityIterator implements
ComparableSSTableIdentityIterat
private final boolean validateColumns;
private final String filename;
+// Not every SSTableIdentifyIterator is attached to a sstable, so this can
be null.
+private final SSTableReader sstable;
+
/**
* Used to iterate through the columns of a row.
* @param sstable SSTable we are reading ffrom.
@@ -96,6 +99,7 @@ public class SSTableIdentityIterator implements
ComparableSSTableIdentityIterat
this.flag = flag;
this.validateColumns = checkData;
this.dataVersion = sstable == null ? Descriptor.Version.CURRENT :
sstable.descriptor.version;
+this.sstable = sstable;
try
{
@@ -132,9 +136,15 @@ public class SSTableIdentityIterator implements
ComparableSSTableIdentityIterat
{
// catch here b/c atomIterator is an AbstractIterator; hasNext
reads the value
if (e.getCause() instanceof IOException)
+{
+if (sstable != null)
+sstable.markSuspect();
throw new CorruptSSTableException((IOException)e.getCause(),
filename);
+}
else
+{
throw e;
+}
}
}
@@ -181,22 +191,39 @@ public class SSTableIdentityIterator implements
ComparableSSTableIdentityIterat
{
ColumnFamily cf = columnFamily.cloneMeShallow(containerFactory, false);
// since we already read column count, just pass that value and
continue deserialization
-IteratorOnDiskAtom iter = cf.metadata().getOnDiskIterator(in,
columnCount, flag, expireBefore, dataVersion);
-while (iter.hasNext())
-cf.addAtom(iter.next());
+try
+{
+IteratorOnDiskAtom iter = cf.metadata().getOnDiskIterator(in,
columnCount, flag, expireBefore, dataVersion);
+while (iter.hasNext())
+cf.addAtom(iter.next());
-if (validateColumns)
+if (validateColumns)
+{
+try
+{
+cf.metadata().validateColumns(cf);
+}
+catch (MarshalException e)
+{
+throw new RuntimeException(Error validating row + key,
e);
+}
+}
+return cf;
+}
+catch (IOError e)
{
-try
+// catch here b/c atomIterator is an AbstractIterator; hasNext
reads the value
+if (e.getCause() instanceof IOException)
{
-cf.metadata().validateColumns(cf);
+if (sstable != null)
+
[jira] [Commented] (CASSANDRA-9527) Cannot create secondary index on a table WITH COMPACT STORAGE
[ https://issues.apache.org/jira/browse/CASSANDRA-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569098#comment-14569098 ] Philip Thompson commented on CASSANDRA-9527: I can confirm this is happening on 2.0.15. [~blerer], should this be possible? If not, then it's just the error message that's wrong. Cannot create secondary index on a table WITH COMPACT STORAGE - Key: CASSANDRA-9527 URL: https://issues.apache.org/jira/browse/CASSANDRA-9527 Project: Cassandra Issue Type: Bug Components: Core Environment: cassandra v2.0.13 Reporter: fuggy_yama Assignee: Benjamin Lerer Priority: Minor Fix For: 2.0.x In CASSANDRA-8156 it is said that secondary indexes are not allowed on clustering columns in COMPACT tables. However, I found that it is not possible to create a secondary index on the value column in a COMPACT table: CREATE TABLE t ( a INT, b INT, c INT, PRIMARY KEY (a, b) ) WITH COMPACT STORAGE; CREATE INDEX ON t (c); *Bad Request: Secondary indexes are not supported on PRIMARY KEY columns in COMPACT STORAGE tables* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9530) SSTable corruption can trigger OOM
Sylvain Lebresne created CASSANDRA-9530:
---
Summary: SSTable corruption can trigger OOM
Key: CASSANDRA-9530
URL: https://issues.apache.org/jira/browse/CASSANDRA-9530
Project: Cassandra
Issue Type: Bug
Reporter: Sylvain Lebresne
Priority: Minor
If a sstable is corrupted so that the length of a given is bogus, we'll still
happily try to allocate a buffer of that bogus size to read the value, which
can easily lead to an OOM.
We should probably protect against this. In practice, a given value can be so
big since it's limited by the protocol frame size in the first place. Maybe we
could add a max_value_size_in_mb setting and we'd considered a sstable
corrupted if it was containing a value bigger than that.
I'll note that this ticket would be a good occasion to improve
{{BlacklistingCompactionsTest}}. Typically, it currently generate empty values
which makes it pretty much impossible to get the problem described here. And as
described in CASSANDRA-9478, it also doesn't test properly for thing like early
opening of compaction results. We could try to randomize as much of the
parameters of this test as possible to make it more likely to catch any type of
corruption that could happen.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[1/5] cassandra git commit: Clean up gossiper logic for old versions
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.2 6f93bd1f6 - 4cfb17f48
refs/heads/trunk ae2daea52 - e27e97413
Clean up gossiper logic for old versions
Patch by brandonwilliams, reviewed by aleksey for CASSANDRA-9370
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/23d7a558
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/23d7a558
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/23d7a558
Branch: refs/heads/cassandra-2.2
Commit: 23d7a558ee824a22d382e1b645814638a2cd2985
Parents: 94a68a1
Author: Brandon Williams brandonwilli...@apache.org
Authored: Tue Jun 2 08:42:28 2015 -0500
Committer: Brandon Williams brandonwilli...@apache.org
Committed: Tue Jun 2 08:42:28 2015 -0500
--
CHANGES.txt | 1 +
src/java/org/apache/cassandra/gms/Gossiper.java | 16 --
.../cassandra/service/StorageService.java | 54 +---
3 files changed, 26 insertions(+), 45 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/23d7a558/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 165f76b..41f5cfb 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.2
+ * Clean up gossiper logic for old versions (CASSANDRA-9370)
* ant test-all results incomplete when parsed (CASSANDRA-9463)
* Disallow frozen types in function arguments and return types for
clarity (CASSANDRA-9411)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/23d7a558/src/java/org/apache/cassandra/gms/Gossiper.java
--
diff --git a/src/java/org/apache/cassandra/gms/Gossiper.java
b/src/java/org/apache/cassandra/gms/Gossiper.java
index 0c449ec..e131da6 100644
--- a/src/java/org/apache/cassandra/gms/Gossiper.java
+++ b/src/java/org/apache/cassandra/gms/Gossiper.java
@@ -798,24 +798,8 @@ public class Gossiper implements
IFailureDetectionEventListener, GossiperMBean
return endpointStateMap.entrySet();
}
-public boolean usesHostId(InetAddress endpoint)
-{
-if (MessagingService.instance().knowsVersion(endpoint))
-return true;
-else if
(getEndpointStateForEndpoint(endpoint).getApplicationState(ApplicationState.NET_VERSION)
!= null)
-return true;
-return false;
-}
-
-public boolean usesVnodes(InetAddress endpoint)
-{
-return usesHostId(endpoint)
getEndpointStateForEndpoint(endpoint).getApplicationState(ApplicationState.TOKENS)
!= null;
-}
-
public UUID getHostId(InetAddress endpoint)
{
-if (!usesHostId(endpoint))
-throw new RuntimeException(Host + endpoint + does not use
new-style tokens!);
return
UUID.fromString(getEndpointStateForEndpoint(endpoint).getApplicationState(ApplicationState.HOST_ID).value);
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/23d7a558/src/java/org/apache/cassandra/service/StorageService.java
--
diff --git a/src/java/org/apache/cassandra/service/StorageService.java
b/src/java/org/apache/cassandra/service/StorageService.java
index bfbf1a8..8afb93f 100644
--- a/src/java/org/apache/cassandra/service/StorageService.java
+++ b/src/java/org/apache/cassandra/service/StorageService.java
@@ -1825,8 +1825,7 @@ public class StorageService extends
NotificationBroadcasterSupport implements IE
tokenMetadata.addBootstrapTokens(tokens, endpoint);
PendingRangeCalculatorService.instance.update();
-if (Gossiper.instance.usesHostId(endpoint))
-tokenMetadata.updateHostId(Gossiper.instance.getHostId(endpoint),
endpoint);
+tokenMetadata.updateHostId(Gossiper.instance.getHostId(endpoint),
endpoint);
}
/**
@@ -1855,39 +1854,36 @@ public class StorageService extends
NotificationBroadcasterSupport implements IE
updatePeerInfo(endpoint);
// Order Matters, TM.updateHostID() should be called before
TM.updateNormalToken(), (see CASSANDRA-4300).
-if (Gossiper.instance.usesHostId(endpoint))
+UUID hostId = Gossiper.instance.getHostId(endpoint);
+InetAddress existing = tokenMetadata.getEndpointForHostId(hostId);
+if (replacing
Gossiper.instance.getEndpointStateForEndpoint(DatabaseDescriptor.getReplaceAddress())
!= null
(hostId.equals(Gossiper.instance.getHostId(DatabaseDescriptor.getReplaceAddress()
+logger.warn(Not updating token metadata for {} because I am
replacing it, endpoint);
+else
{
-UUID hostId = Gossiper.instance.getHostId(endpoint);
-InetAddress existing =
[4/5] cassandra git commit: Merge branch 'cassandra-2.2' of https://git-wip-us.apache.org/repos/asf/cassandra into cassandra-2.2
Merge branch 'cassandra-2.2' of https://git-wip-us.apache.org/repos/asf/cassandra into cassandra-2.2 Conflicts: CHANGES.txt Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4cfb17f4 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4cfb17f4 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4cfb17f4 Branch: refs/heads/cassandra-2.2 Commit: 4cfb17f48ea88465b39e7eb762985cda172e1fb3 Parents: 23d7a55 6f93bd1 Author: Brandon Williams brandonwilli...@apache.org Authored: Tue Jun 2 08:43:15 2015 -0500 Committer: Brandon Williams brandonwilli...@apache.org Committed: Tue Jun 2 08:43:15 2015 -0500 -- CHANGES.txt | 2 + .../CustomPayloadMirroringQueryHandler.java | 18 +++-- .../org/apache/cassandra/cql3/QueryHandler.java | 20 -- .../apache/cassandra/cql3/QueryProcessor.java | 22 -- .../io/sstable/SSTableIdentityIterator.java | 10 +++ .../io/sstable/format/SSTableReader.java| 57 +++ .../org/apache/cassandra/transport/CBUtil.java | 18 ++--- .../org/apache/cassandra/transport/Message.java | 13 ++-- .../compaction/BlacklistingCompactionsTest.java | 17 +++-- .../cassandra/transport/MessagePayloadTest.java | 73 10 files changed, 177 insertions(+), 73 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/4cfb17f4/CHANGES.txt -- diff --cc CHANGES.txt index 41f5cfb,db94c76..42846d7 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,5 -1,5 +1,6 @@@ 2.2 + * Clean up gossiper logic for old versions (CASSANDRA-9370) + * Fix custom payload coding/decoding to match the spec (CASSANDRA-9515) * ant test-all results incomplete when parsed (CASSANDRA-9463) * Disallow frozen types in function arguments and return types for clarity (CASSANDRA-9411)
[3/5] cassandra git commit: Merge branch 'cassandra-2.2' of https://git-wip-us.apache.org/repos/asf/cassandra into cassandra-2.2
Merge branch 'cassandra-2.2' of https://git-wip-us.apache.org/repos/asf/cassandra into cassandra-2.2 Conflicts: CHANGES.txt Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4cfb17f4 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4cfb17f4 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4cfb17f4 Branch: refs/heads/trunk Commit: 4cfb17f48ea88465b39e7eb762985cda172e1fb3 Parents: 23d7a55 6f93bd1 Author: Brandon Williams brandonwilli...@apache.org Authored: Tue Jun 2 08:43:15 2015 -0500 Committer: Brandon Williams brandonwilli...@apache.org Committed: Tue Jun 2 08:43:15 2015 -0500 -- CHANGES.txt | 2 + .../CustomPayloadMirroringQueryHandler.java | 18 +++-- .../org/apache/cassandra/cql3/QueryHandler.java | 20 -- .../apache/cassandra/cql3/QueryProcessor.java | 22 -- .../io/sstable/SSTableIdentityIterator.java | 10 +++ .../io/sstable/format/SSTableReader.java| 57 +++ .../org/apache/cassandra/transport/CBUtil.java | 18 ++--- .../org/apache/cassandra/transport/Message.java | 13 ++-- .../compaction/BlacklistingCompactionsTest.java | 17 +++-- .../cassandra/transport/MessagePayloadTest.java | 73 10 files changed, 177 insertions(+), 73 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/4cfb17f4/CHANGES.txt -- diff --cc CHANGES.txt index 41f5cfb,db94c76..42846d7 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,5 -1,5 +1,6 @@@ 2.2 + * Clean up gossiper logic for old versions (CASSANDRA-9370) + * Fix custom payload coding/decoding to match the spec (CASSANDRA-9515) * ant test-all results incomplete when parsed (CASSANDRA-9463) * Disallow frozen types in function arguments and return types for clarity (CASSANDRA-9411)
[jira] [Created] (CASSANDRA-9531) NullPointerException logged when running MessagePayloadTest
Benjamin Lerer created CASSANDRA-9531:
-
Summary: NullPointerException logged when running
MessagePayloadTest
Key: CASSANDRA-9531
URL: https://issues.apache.org/jira/browse/CASSANDRA-9531
Project: Cassandra
Issue Type: Bug
Reporter: Benjamin Lerer
Priority: Minor
When I run the {{MessagePayloadTest}} I have the following error in my log
output:
{code}
juin 02, 2015 3:27:31 PM com.google.common.cache.LocalCache
processPendingNotifications
AVERTISSEMENT: Exception thrown by removal listener
java.lang.NullPointerException
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.getTotalBufferSize(CompressedRandomAccessReader.java:260)
at
org.apache.cassandra.service.FileCacheService$2.onRemoval(FileCacheService.java:105)
at
com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1954)
at
com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3457)
at
com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3433)
at
com.google.common.cache.LocalCache$Segment.remove(LocalCache.java:3104)
at com.google.common.cache.LocalCache.remove(LocalCache.java:4170)
at
com.google.common.cache.LocalCache$LocalManualCache.invalidate(LocalCache.java:4762)
at
org.apache.cassandra.service.FileCacheService.invalidate(FileCacheService.java:182)
at
org.apache.cassandra.io.util.PoolingSegmentedFile$Cleanup.tidy(PoolingSegmentedFile.java:53)
at
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Cleanup.tidy(CompressedPoolingSegmentedFile.java:77)
at
org.apache.cassandra.utils.concurrent.Ref$GlobalState.release(Ref.java:285)
at
org.apache.cassandra.utils.concurrent.Ref$State.ensureReleased(Ref.java:164)
at org.apache.cassandra.utils.concurrent.Ref.ensureReleased(Ref.java:89)
at
org.apache.cassandra.utils.concurrent.SharedCloseableImpl.close(SharedCloseableImpl.java:45)
at
org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier$1.run(SSTableReader.java:1937)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
The problem is due to the fact that when {{getTotalBufferSize}} is called the
segment has already been desallocated. The reason being that the segment has
been put twice in the queue of the {{CacheBucket}}.
I tracked down the problem to {{CompactionTask.runMayThrow}} which trigger 2
calls to the {{close}} method of {{BigTableScanner}}.
One here:
{code}
at
org.apache.cassandra.io.sstable.format.big.BigTableScanner.close(BigTableScanner.java:197)
at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy$ScannerList.close(AbstractCompactionStrategy.java:333)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:207)
{code}
and the other here:
{code}
at
org.apache.cassandra.io.sstable.format.big.BigTableScanner.close(BigTableScanner.java:197)
at org.apache.cassandra.utils.MergeIterator.close(MergeIterator.java:62)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:206)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9514) TrackerTest.testAddSSTables fails in 2.2 and trunk
[
https://issues.apache.org/jira/browse/CASSANDRA-9514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569287#comment-14569287
]
Michael Shuler commented on CASSANDRA-9514:
---
http://cassci.datastax.com/job/trunk_utest/229/testReport/org.apache.cassandra.db.lifecycle/TrackerTest/testAddInitialSSTables/
{noformat}
junit.framework.AssertionFailedError
at
org.apache.cassandra.utils.concurrent.Ref$State.assertNotReleased(Ref.java:157)
at org.apache.cassandra.utils.concurrent.Ref.ref(Ref.java:113)
at
org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy.get(SSTableReader.java:2111)
at
org.apache.cassandra.io.sstable.format.SSTableReader$DescriptorTypeTidy.init(SSTableReader.java:1980)
at
org.apache.cassandra.io.sstable.format.SSTableReader$DescriptorTypeTidy.get(SSTableReader.java:2017)
at
org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.setup(SSTableReader.java:1897)
at
org.apache.cassandra.io.sstable.format.SSTableReader.setup(SSTableReader.java:1842)
at
org.apache.cassandra.io.sstable.format.SSTableReader.internalOpen(SSTableReader.java:530)
at org.apache.cassandra.MockSchema.sstable(MockSchema.java:128)
at org.apache.cassandra.MockSchema.sstable(MockSchema.java:92)
at org.apache.cassandra.MockSchema.sstable(MockSchema.java:87)
at
org.apache.cassandra.db.lifecycle.TrackerTest.testAddInitialSSTables(TrackerTest.java:146)
{noformat}
This was on COMMIT = 525579d which looks to be a commit or 2 after this fix was
committed - may re-open this..
TrackerTest.testAddSSTables fails in 2.2 and trunk
--
Key: CASSANDRA-9514
URL: https://issues.apache.org/jira/browse/CASSANDRA-9514
Project: Cassandra
Issue Type: Test
Components: Tests
Reporter: Michael Shuler
Assignee: Stefania
Labels: test-failure
Fix For: 3.x, 2.2.x
Attachments: system.log.gz, system.log.gz
Error:
{{expected:1 but was:2}}
http://cassci.datastax.com/job/cassandra-2.2_testall/33/testReport/org.apache.cassandra.db.lifecycle/TrackerTest/testTryModify/
http://cassci.datastax.com/job/trunk_testall/125/testReport/org.apache.cassandra.db.lifecycle/TrackerTest/testDropSSTables/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[2/2] cassandra git commit: Merge branch 'cassandra-2.2' into trunk
Merge branch 'cassandra-2.2' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bde7615c Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bde7615c Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bde7615c Branch: refs/heads/trunk Commit: bde7615c2f8f8d8f6a8a9be4ca01c02fc9e03e54 Parents: e27e974 b2e4d10 Author: Marcus Eriksson marc...@apache.org Authored: Tue Jun 2 17:37:23 2015 +0200 Committer: Marcus Eriksson marc...@apache.org Committed: Tue Jun 2 17:37:23 2015 +0200 -- src/java/org/apache/cassandra/service/StorageService.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/bde7615c/src/java/org/apache/cassandra/service/StorageService.java --
cassandra git commit: Make it possible to run dc local repairs
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.2 4cfb17f48 - b2e4d1015
Make it possible to run dc local repairs
Patch by marcuse; reviewed by yukim for CASSANDRA-9521
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b2e4d101
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b2e4d101
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b2e4d101
Branch: refs/heads/cassandra-2.2
Commit: b2e4d10155afa3618ba1a9066c6ffc13489c73d8
Parents: 4cfb17f
Author: Marcus Eriksson marc...@apache.org
Authored: Tue Jun 2 17:36:42 2015 +0200
Committer: Marcus Eriksson marc...@apache.org
Committed: Tue Jun 2 17:36:42 2015 +0200
--
src/java/org/apache/cassandra/service/StorageService.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/b2e4d101/src/java/org/apache/cassandra/service/StorageService.java
--
diff --git a/src/java/org/apache/cassandra/service/StorageService.java
b/src/java/org/apache/cassandra/service/StorageService.java
index 8afb93f..4fdb7b7 100644
--- a/src/java/org/apache/cassandra/service/StorageService.java
+++ b/src/java/org/apache/cassandra/service/StorageService.java
@@ -2966,7 +2966,7 @@ public class StorageService extends
NotificationBroadcasterSupport implements IE
private FutureTaskObject createRepairTask(final int cmd, final String
keyspace, final RepairOption options)
{
-if (!options.getDataCenters().isEmpty()
options.getDataCenters().contains(DatabaseDescriptor.getLocalDataCenter()))
+if (!options.getDataCenters().isEmpty()
!options.getDataCenters().contains(DatabaseDescriptor.getLocalDataCenter()))
{
throw new IllegalArgumentException(the local data center must be
part of the repair);
}
[1/2] cassandra git commit: Make it possible to run dc local repairs
Repository: cassandra
Updated Branches:
refs/heads/trunk e27e97413 - bde7615c2
Make it possible to run dc local repairs
Patch by marcuse; reviewed by yukim for CASSANDRA-9521
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b2e4d101
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b2e4d101
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b2e4d101
Branch: refs/heads/trunk
Commit: b2e4d10155afa3618ba1a9066c6ffc13489c73d8
Parents: 4cfb17f
Author: Marcus Eriksson marc...@apache.org
Authored: Tue Jun 2 17:36:42 2015 +0200
Committer: Marcus Eriksson marc...@apache.org
Committed: Tue Jun 2 17:36:42 2015 +0200
--
src/java/org/apache/cassandra/service/StorageService.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/b2e4d101/src/java/org/apache/cassandra/service/StorageService.java
--
diff --git a/src/java/org/apache/cassandra/service/StorageService.java
b/src/java/org/apache/cassandra/service/StorageService.java
index 8afb93f..4fdb7b7 100644
--- a/src/java/org/apache/cassandra/service/StorageService.java
+++ b/src/java/org/apache/cassandra/service/StorageService.java
@@ -2966,7 +2966,7 @@ public class StorageService extends
NotificationBroadcasterSupport implements IE
private FutureTaskObject createRepairTask(final int cmd, final String
keyspace, final RepairOption options)
{
-if (!options.getDataCenters().isEmpty()
options.getDataCenters().contains(DatabaseDescriptor.getLocalDataCenter()))
+if (!options.getDataCenters().isEmpty()
!options.getDataCenters().contains(DatabaseDescriptor.getLocalDataCenter()))
{
throw new IllegalArgumentException(the local data center must be
part of the repair);
}
[jira] [Commented] (CASSANDRA-8897) Remove FileCacheService, instead pooling the buffers
[ https://issues.apache.org/jira/browse/CASSANDRA-8897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569296#comment-14569296 ] Benedict commented on CASSANDRA-8897: - Thanks [~stef1927]! I've pushed one last round of suggestions. If you can yay/nay them, and make any comments you feel necessary, I'll then see about committing. They're more involved than I'd first intended, so please don't shy away from any criticism you consider warranted. To summarise, as far as basic functionality is concerned, I've: # modified the chunk queue logic to always be a prefix of non-null items and to always put the new chunk at the end, so it is always tried last # implemented the immediate freeing of a chunk that is owned by the thread that is freeing (to keep retained memory to a minimum) # merged the CAS to perform the recycle with the update that sets it to fully free # some minor renaming/refactoring, such as separating the adoption of a new Chunk from the allocation of a buffer from it Then, I've fixed a bug. This bug crept in due to the mix of suggestions I made to you, and how they combined in this instance. The bug is a pretty subtle and rare concurrency failure, wherein a thread could be late recycling a buffer that has already been recycled and is concurrently being re-adopted by another thread, immediately making it unavailable for allocation by that other thread. In most situations this would end up simply occupying some space in the owner's queue until it got evicted, but it would be possible for it to be reallocated to another thread while still avaiable and owned by the first, and trigger assertion failures. In order to fix this bug, I've rolled back the reuse of the same Chunk instance; there are other solutions, but they are likely not worth the trouble. In order to _exercise_ this bug, I've revamped the long test. The new version attempts to increase the level of direct interleaving of concurrent operations. Sleeps in particular mitigate this kind of behaviour, so they are avoided. Occasionally we yield, if no safe progress can be made by a test thread. There are fewer threads, and these each reduce the concurrent operations they participate in that are not directly incurred in the BufferPool. To summarise these changes: # Utilises DynamicList from stress, that is moved into the main package (and split into LockedDynamicList and DynamicList), and permits us to maintain a per-thread collection of buffers to validate # Constrains the buffer pool's memory so that it can see high churn and reuse of the same chunks # Ensures the total amount of memory in use is at or around the size of the buffer pool # Ensures that all chunks are recycled periodically (so that none are leaked/lost) #* This involved introducing some debug logic into the BufferPool path, to track the recycling of chunks; this isn't super pretty, but is very helpful to have confidence of the behaviour # Randomly frees buffers from its local collection, either itself or via a neighbour # Each thread only frees buffers from one of its neighbours, so that the message passing between them can be kept extremely cheap (reducing cost for none BufferPool operations) # Amortizes the cost of System.nanoTime(), reducing competition within it (again, non-BufferPool concurrent op) # Introduces two new threads, whose sole job is to churn chunks at a high rate. This is to most directly increase the incidence of the bug. # Randomly burst allocates the entire pool contents I've also slightly modified your version, and uploaded it [here|https://github.com/belliottsmith/cassandra/tree/8897-testimprovements] with the new long test. This change was just to further increase the likelihood of exercising this bug, by hitting a failure-to-allocate in the thread that has its freeSlots ripped away from it. Remove FileCacheService, instead pooling the buffers Key: CASSANDRA-8897 URL: https://issues.apache.org/jira/browse/CASSANDRA-8897 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Benedict Assignee: Stefania Fix For: 3.x Attachments: 8897.yaml, 9240_test_results.txt, snapshot-1431582436640-cpu-backtraces.png, snapshot-1431582436640-cpu-calltree-compression-8897.nps, snapshot-1431582436640-cpu-calltree-compression-trunk.nps After CASSANDRA-8893, a RAR will be a very lightweight object and will not need caching, so we can eliminate this cache entirely. Instead we should have a pool of buffers that are page-aligned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9531) NullPointerException logged when running MessagePayloadTest
[
https://issues.apache.org/jira/browse/CASSANDRA-9531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569298#comment-14569298
]
T Jake Luciani commented on CASSANDRA-9531:
---
Branch here https://github.com/tjake/cassandra/tree/9531
I wasn't able to reproduce the issue. But the patch makes the scanner.close
call idempotent. Can you verify [~blerer]?
NullPointerException logged when running MessagePayloadTest
---
Key: CASSANDRA-9531
URL: https://issues.apache.org/jira/browse/CASSANDRA-9531
Project: Cassandra
Issue Type: Bug
Reporter: Benjamin Lerer
Assignee: T Jake Luciani
Priority: Minor
Fix For: 2.2.0 rc1
When I run the {{MessagePayloadTest}} I have the following error in my log
output:
{code}
juin 02, 2015 3:27:31 PM com.google.common.cache.LocalCache
processPendingNotifications
AVERTISSEMENT: Exception thrown by removal listener
java.lang.NullPointerException
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.getTotalBufferSize(CompressedRandomAccessReader.java:260)
at
org.apache.cassandra.service.FileCacheService$2.onRemoval(FileCacheService.java:105)
at
com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1954)
at
com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3457)
at
com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3433)
at
com.google.common.cache.LocalCache$Segment.remove(LocalCache.java:3104)
at com.google.common.cache.LocalCache.remove(LocalCache.java:4170)
at
com.google.common.cache.LocalCache$LocalManualCache.invalidate(LocalCache.java:4762)
at
org.apache.cassandra.service.FileCacheService.invalidate(FileCacheService.java:182)
at
org.apache.cassandra.io.util.PoolingSegmentedFile$Cleanup.tidy(PoolingSegmentedFile.java:53)
at
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Cleanup.tidy(CompressedPoolingSegmentedFile.java:77)
at
org.apache.cassandra.utils.concurrent.Ref$GlobalState.release(Ref.java:285)
at
org.apache.cassandra.utils.concurrent.Ref$State.ensureReleased(Ref.java:164)
at org.apache.cassandra.utils.concurrent.Ref.ensureReleased(Ref.java:89)
at
org.apache.cassandra.utils.concurrent.SharedCloseableImpl.close(SharedCloseableImpl.java:45)
at
org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier$1.run(SSTableReader.java:1937)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
The problem is due to the fact that when {{getTotalBufferSize}} is called the
segment has already been desallocated. The reason being that the segment has
been put twice in the queue of the {{CacheBucket}}.
I tracked down the problem to {{CompactionTask.runMayThrow}} which trigger 2
calls to the {{close}} method of {{BigTableScanner}}.
One here:
{code}
at
org.apache.cassandra.io.sstable.format.big.BigTableScanner.close(BigTableScanner.java:197)
at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy$ScannerList.close(AbstractCompactionStrategy.java:333)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:207)
{code}
and the other here:
{code}
at
org.apache.cassandra.io.sstable.format.big.BigTableScanner.close(BigTableScanner.java:197)
at org.apache.cassandra.utils.MergeIterator.close(MergeIterator.java:62)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:206)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8656) long-test LongLeveledCompactionStrategyTest flaps in 2.0
[
https://issues.apache.org/jira/browse/CASSANDRA-8656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569313#comment-14569313
]
Michael Shuler commented on CASSANDRA-8656:
---
30x over https://github.com/stef1927/cassandra/commit/25ecbff with no failures.
:)
long-test LongLeveledCompactionStrategyTest flaps in 2.0
Key: CASSANDRA-8656
URL: https://issues.apache.org/jira/browse/CASSANDRA-8656
Project: Cassandra
Issue Type: Test
Components: Tests
Reporter: Michael Shuler
Assignee: Stefania
Priority: Minor
Labels: test-failure
Fix For: 2.0.x
Attachments: system.log
LongLeveledCompactionStrategyTest periodically fails with:
{noformat}
[junit] Testsuite:
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest
[junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
54.412 sec
[junit]
[junit] Testcase:
testParallelLeveledCompaction(org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest):
Caused an ERROR
[junit] java.util.concurrent.ExecutionException:
java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) =
current key DecoratedKey(313236, 313236) writing into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] java.lang.RuntimeException:
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last
written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236,
313236) writing into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:413)
[junit] at
org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:402)
[junit] at
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest.testParallelLeveledCompaction(LongLeveledCompactionStrategyTest.java:97)
[junit] Caused by: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) =
current key DecoratedKey(313236, 313236) writing into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] at java.util.concurrent.FutureTask.report(FutureTask.java:122)
[junit] at java.util.concurrent.FutureTask.get(FutureTask.java:188)
[junit] at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:409)
[junit] Caused by: java.lang.RuntimeException: Last written key
DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing
into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
[junit] at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
[junit] at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167)
[junit] at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
[junit] at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
[junit] at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
[junit] at
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest$1.run(LongLeveledCompactionStrategyTest.java:87)
[junit] at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[junit] at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit] at java.lang.Thread.run(Thread.java:745)
[junit]
[junit]
[junit] Test
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest FAILED
{noformat}
I would guess the failure is 10-20% of the time, looping over the test
repeatedly.
On the 2.1 branch, the failure is different, so perhaps this could also be
updated.
{noformat}
[junit] Testsuite:
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest
[junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
7.04 sec
[junit]
[junit] Testcase:
testParallelLeveledCompaction(org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest):
Caused an ERROR
[junit] org.apache.cassandra.db.compaction.WrappingCompactionStrategy
[jira] [Commented] (CASSANDRA-9527) Cannot create secondary index on a table WITH COMPACT STORAGE
[ https://issues.apache.org/jira/browse/CASSANDRA-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569314#comment-14569314 ] Benjamin Lerer commented on CASSANDRA-9527: --- Sorry, I was wrong. We support secondary indices only on the compact tables that does not have clustering columns. Which means that we should change the error message for this specific case. Cannot create secondary index on a table WITH COMPACT STORAGE - Key: CASSANDRA-9527 URL: https://issues.apache.org/jira/browse/CASSANDRA-9527 Project: Cassandra Issue Type: Bug Components: Core Environment: cassandra v2.0.13 Reporter: fuggy_yama Assignee: Benjamin Lerer Priority: Minor Fix For: 2.0.x In CASSANDRA-8156 it is said that secondary indexes are not allowed on clustering columns in COMPACT tables. However, I found that it is not possible to create a secondary index on the value column in a COMPACT table: CREATE TABLE t ( a INT, b INT, c INT, PRIMARY KEY (a, b) ) WITH COMPACT STORAGE; CREATE INDEX ON t (c); *Bad Request: Secondary indexes are not supported on PRIMARY KEY columns in COMPACT STORAGE tables* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8897) Remove FileCacheService, instead pooling the buffers
[ https://issues.apache.org/jira/browse/CASSANDRA-8897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569299#comment-14569299 ] Benedict commented on CASSANDRA-8897: - Thanks [~stef1927]! I've pushed one last round of suggestions. If you can yay/nay them, and make any comments you feel necessary, I'll then see about committing. They're more involved than I'd first intended, so please don't shy away from any criticism you consider warranted. To summarise, as far as basic functionality is concerned, I've: # modified the chunk queue logic to always be a prefix of non-null items and to always put the new chunk at the end, so it is always tried last # implemented the immediate freeing of a chunk that is owned by the thread that is freeing (to keep retained memory to a minimum) # merged the CAS to perform the recycle with the update that sets it to fully free # some minor renaming/refactoring, such as separating the adoption of a new Chunk from the allocation of a buffer from it Then, I've fixed a bug. This bug crept in due to the mix of suggestions I made to you, and how they combined in this instance. The bug is a pretty subtle and rare concurrency failure, wherein a thread could be late recycling a buffer that has already been recycled and is concurrently being re-adopted by another thread, immediately making it unavailable for allocation by that other thread. In most situations this would end up simply occupying some space in the owner's queue until it got evicted, but it would be possible for it to be reallocated to another thread while still avaiable and owned by the first, and trigger assertion failures. In order to fix this bug, I've rolled back the reuse of the same Chunk instance; there are other solutions, but they are likely not worth the trouble. In order to _exercise_ this bug, I've revamped the long test. The new version attempts to increase the level of direct interleaving of concurrent operations. Sleeps in particular mitigate this kind of behaviour, so they are avoided. Occasionally we yield, if no safe progress can be made by a test thread. There are fewer threads, and these each reduce the concurrent operations they participate in that are not directly incurred in the BufferPool. To summarise these changes: # Utilises DynamicList from stress, that is moved into the main package (and split into LockedDynamicList and DynamicList), and permits us to maintain a per-thread collection of buffers to validate # Constrains the buffer pool's memory so that it can see high churn and reuse of the same chunks # Ensures the total amount of memory in use is at or around the size of the buffer pool # Ensures that all chunks are recycled periodically (so that none are leaked/lost) #* This involved introducing some debug logic into the BufferPool path, to track the recycling of chunks; this isn't super pretty, but is very helpful to have confidence of the behaviour # Randomly frees buffers from its local collection, either itself or via a neighbour # Each thread only frees buffers from one of its neighbours, so that the message passing between them can be kept extremely cheap (reducing cost for none BufferPool operations) # Amortizes the cost of System.nanoTime(), reducing competition within it (again, non-BufferPool concurrent op) # Introduces two new threads, whose sole job is to churn chunks at a high rate. This is to most directly increase the incidence of the bug. # Randomly burst allocates the entire pool contents I've also slightly modified your version, and uploaded it [here|https://github.com/belliottsmith/cassandra/tree/8897-testimprovements] with the new long test. This change was just to further increase the likelihood of exercising this bug, by hitting a failure-to-allocate in the thread that has its freeSlots ripped away from it. Remove FileCacheService, instead pooling the buffers Key: CASSANDRA-8897 URL: https://issues.apache.org/jira/browse/CASSANDRA-8897 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Benedict Assignee: Stefania Fix For: 3.x Attachments: 8897.yaml, 9240_test_results.txt, snapshot-1431582436640-cpu-backtraces.png, snapshot-1431582436640-cpu-calltree-compression-8897.nps, snapshot-1431582436640-cpu-calltree-compression-trunk.nps After CASSANDRA-8893, a RAR will be a very lightweight object and will not need caching, so we can eliminate this cache entirely. Instead we should have a pool of buffers that are page-aligned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8805) runWithCompactionsDisabled only cancels compactions, which is not the only source of markCompacted
[
https://issues.apache.org/jira/browse/CASSANDRA-8805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569384#comment-14569384
]
Benedict commented on CASSANDRA-8805:
-
The rollback logic in the event of CIE is, unfortunately, likely to break
things. Once a reader replaces another, it needs to be published or itself
replaced. So we should either make visible the newly sampled readers (excluding
the ones we haven't completed), or we should construct a new instance with the
old summaries that are publised.
Since the operation can also be very expensive, we should probably extract the
resampling into a method that can itself check for the {{isStopRequested}},
otherwise it could take a long time to acquiesce.
runWithCompactionsDisabled only cancels compactions, which is not the only
source of markCompacted
--
Key: CASSANDRA-8805
URL: https://issues.apache.org/jira/browse/CASSANDRA-8805
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Benedict
Assignee: Carl Yeksigian
Fix For: 2.1.x
Attachments: 8805-2.1.txt
Operations like repair that may operate over all sstables cancel compactions
before beginning, and fail if there are any files marked compacting after
doing so. Redistribution of index summaries is not a compaction, so is not
cancelled by this action, but does mark sstables as compacting, so such an
action will fail to initiate if there is an index summary redistribution in
progress. It seems that IndexSummaryManager needs to register itself as
interruptible along with compactions (AFAICT no other actions that may
markCompacting are not themselves compactions).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (CASSANDRA-8897) Remove FileCacheService, instead pooling the buffers
[ https://issues.apache.org/jira/browse/CASSANDRA-8897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-8897: Comment: was deleted (was: Thanks [~stef1927]! I've pushed one last round of suggestions. If you can yay/nay them, and make any comments you feel necessary, I'll then see about committing. They're more involved than I'd first intended, so please don't shy away from any criticism you consider warranted. To summarise, as far as basic functionality is concerned, I've: # modified the chunk queue logic to always be a prefix of non-null items and to always put the new chunk at the end, so it is always tried last # implemented the immediate freeing of a chunk that is owned by the thread that is freeing (to keep retained memory to a minimum) # merged the CAS to perform the recycle with the update that sets it to fully free # some minor renaming/refactoring, such as separating the adoption of a new Chunk from the allocation of a buffer from it Then, I've fixed a bug. This bug crept in due to the mix of suggestions I made to you, and how they combined in this instance. The bug is a pretty subtle and rare concurrency failure, wherein a thread could be late recycling a buffer that has already been recycled and is concurrently being re-adopted by another thread, immediately making it unavailable for allocation by that other thread. In most situations this would end up simply occupying some space in the owner's queue until it got evicted, but it would be possible for it to be reallocated to another thread while still avaiable and owned by the first, and trigger assertion failures. In order to fix this bug, I've rolled back the reuse of the same Chunk instance; there are other solutions, but they are likely not worth the trouble. In order to _exercise_ this bug, I've revamped the long test. The new version attempts to increase the level of direct interleaving of concurrent operations. Sleeps in particular mitigate this kind of behaviour, so they are avoided. Occasionally we yield, if no safe progress can be made by a test thread. There are fewer threads, and these each reduce the concurrent operations they participate in that are not directly incurred in the BufferPool. To summarise these changes: # Utilises DynamicList from stress, that is moved into the main package (and split into LockedDynamicList and DynamicList), and permits us to maintain a per-thread collection of buffers to validate # Constrains the buffer pool's memory so that it can see high churn and reuse of the same chunks # Ensures the total amount of memory in use is at or around the size of the buffer pool # Ensures that all chunks are recycled periodically (so that none are leaked/lost) #* This involved introducing some debug logic into the BufferPool path, to track the recycling of chunks; this isn't super pretty, but is very helpful to have confidence of the behaviour # Randomly frees buffers from its local collection, either itself or via a neighbour # Each thread only frees buffers from one of its neighbours, so that the message passing between them can be kept extremely cheap (reducing cost for none BufferPool operations) # Amortizes the cost of System.nanoTime(), reducing competition within it (again, non-BufferPool concurrent op) # Introduces two new threads, whose sole job is to churn chunks at a high rate. This is to most directly increase the incidence of the bug. # Randomly burst allocates the entire pool contents I've also slightly modified your version, and uploaded it [here|https://github.com/belliottsmith/cassandra/tree/8897-testimprovements] with the new long test. This change was just to further increase the likelihood of exercising this bug, by hitting a failure-to-allocate in the thread that has its freeSlots ripped away from it.) Remove FileCacheService, instead pooling the buffers Key: CASSANDRA-8897 URL: https://issues.apache.org/jira/browse/CASSANDRA-8897 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Benedict Assignee: Stefania Fix For: 3.x Attachments: 8897.yaml, 9240_test_results.txt, snapshot-1431582436640-cpu-backtraces.png, snapshot-1431582436640-cpu-calltree-compression-8897.nps, snapshot-1431582436640-cpu-calltree-compression-trunk.nps After CASSANDRA-8893, a RAR will be a very lightweight object and will not need caching, so we can eliminate this cache entirely. Instead we should have a pool of buffers that are page-aligned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (CASSANDRA-8656) long-test LongLeveledCompactionStrategyTest flaps in 2.0
[
https://issues.apache.org/jira/browse/CASSANDRA-8656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Shuler updated CASSANDRA-8656:
--
Comment: was deleted
(was: 30x over https://github.com/stef1927/cassandra/commit/25ecbff with no
failures. :))
long-test LongLeveledCompactionStrategyTest flaps in 2.0
Key: CASSANDRA-8656
URL: https://issues.apache.org/jira/browse/CASSANDRA-8656
Project: Cassandra
Issue Type: Test
Components: Tests
Reporter: Michael Shuler
Assignee: Stefania
Priority: Minor
Labels: test-failure
Fix For: 2.0.x
Attachments: system.log
LongLeveledCompactionStrategyTest periodically fails with:
{noformat}
[junit] Testsuite:
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest
[junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
54.412 sec
[junit]
[junit] Testcase:
testParallelLeveledCompaction(org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest):
Caused an ERROR
[junit] java.util.concurrent.ExecutionException:
java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) =
current key DecoratedKey(313236, 313236) writing into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] java.lang.RuntimeException:
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last
written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236,
313236) writing into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:413)
[junit] at
org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:402)
[junit] at
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest.testParallelLeveledCompaction(LongLeveledCompactionStrategyTest.java:97)
[junit] Caused by: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) =
current key DecoratedKey(313236, 313236) writing into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] at java.util.concurrent.FutureTask.report(FutureTask.java:122)
[junit] at java.util.concurrent.FutureTask.get(FutureTask.java:188)
[junit] at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:409)
[junit] Caused by: java.lang.RuntimeException: Last written key
DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing
into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
[junit] at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
[junit] at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167)
[junit] at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
[junit] at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
[junit] at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
[junit] at
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest$1.run(LongLeveledCompactionStrategyTest.java:87)
[junit] at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[junit] at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit] at java.lang.Thread.run(Thread.java:745)
[junit]
[junit]
[junit] Test
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest FAILED
{noformat}
I would guess the failure is 10-20% of the time, looping over the test
repeatedly.
On the 2.1 branch, the failure is different, so perhaps this could also be
updated.
{noformat}
[junit] Testsuite:
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest
[junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
7.04 sec
[junit]
[junit] Testcase:
testParallelLeveledCompaction(org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest):
Caused an ERROR
[junit] org.apache.cassandra.db.compaction.WrappingCompactionStrategy
cannot be cast to
[jira] [Commented] (CASSANDRA-8656) long-test LongLeveledCompactionStrategyTest flaps in 2.0
[
https://issues.apache.org/jira/browse/CASSANDRA-8656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569301#comment-14569301
]
Michael Shuler commented on CASSANDRA-8656:
---
30x over https://github.com/stef1927/cassandra/commit/25ecbff with no failures.
:)
long-test LongLeveledCompactionStrategyTest flaps in 2.0
Key: CASSANDRA-8656
URL: https://issues.apache.org/jira/browse/CASSANDRA-8656
Project: Cassandra
Issue Type: Test
Components: Tests
Reporter: Michael Shuler
Assignee: Stefania
Priority: Minor
Labels: test-failure
Fix For: 2.0.x
Attachments: system.log
LongLeveledCompactionStrategyTest periodically fails with:
{noformat}
[junit] Testsuite:
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest
[junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
54.412 sec
[junit]
[junit] Testcase:
testParallelLeveledCompaction(org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest):
Caused an ERROR
[junit] java.util.concurrent.ExecutionException:
java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) =
current key DecoratedKey(313236, 313236) writing into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] java.lang.RuntimeException:
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last
written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236,
313236) writing into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:413)
[junit] at
org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:402)
[junit] at
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest.testParallelLeveledCompaction(LongLeveledCompactionStrategyTest.java:97)
[junit] Caused by: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) =
current key DecoratedKey(313236, 313236) writing into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] at java.util.concurrent.FutureTask.report(FutureTask.java:122)
[junit] at java.util.concurrent.FutureTask.get(FutureTask.java:188)
[junit] at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:409)
[junit] Caused by: java.lang.RuntimeException: Last written key
DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing
into
build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db
[junit] at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
[junit] at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
[junit] at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167)
[junit] at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
[junit] at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
[junit] at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
[junit] at
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest$1.run(LongLeveledCompactionStrategyTest.java:87)
[junit] at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[junit] at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit] at java.lang.Thread.run(Thread.java:745)
[junit]
[junit]
[junit] Test
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest FAILED
{noformat}
I would guess the failure is 10-20% of the time, looping over the test
repeatedly.
On the 2.1 branch, the failure is different, so perhaps this could also be
updated.
{noformat}
[junit] Testsuite:
org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest
[junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
7.04 sec
[junit]
[junit] Testcase:
testParallelLeveledCompaction(org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest):
Caused an ERROR
[junit] org.apache.cassandra.db.compaction.WrappingCompactionStrategy
[jira] [Created] (CASSANDRA-9532) Provide access to select statement's real column definitions
mck created CASSANDRA-9532:
--
Summary: Provide access to select statement's real column
definitions
Key: CASSANDRA-9532
URL: https://issues.apache.org/jira/browse/CASSANDRA-9532
Project: Cassandra
Issue Type: Improvement
Reporter: mck
Assignee: mck
Currently there is no way to get access to the real ColumnDefinitions being
used in a SelectStatement.
This information is there in
{{selectStatement.selection.columns}} but is private.
Giving public access would make it possible for third-party implementations of
a {{QueryHandler}} to work accurately with the real columns being queried and
not have to work-around column aliases (or when the rawSelectors don't map
directly to ColumnDefinitions, eg in Selection.fromSelectors(..), like
functions), which is what one has to do today with going through
ResultSet.metadata.names.
This issue provides a very minimal patch to provide access to the already final
and immutable fields.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8915) Improve MergeIterator performance
[
https://issues.apache.org/jira/browse/CASSANDRA-8915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569392#comment-14569392
]
Branimir Lambov commented on CASSANDRA-8915:
I wasn't completely sure of the correctness of the flagging approach and I
wrote the following proof to make sure I'm not missing something:
{panel}
In the discussion below ρ(N) stands for parent of N and σ(N) stands for the
'equalParent' predicate, i.e. σ(N) ↔ ρ(N) = N.
We define ep-heap leading to M to be a binary tree of nodes where each node N
satisfies ρ(N) ≤ N σ(N) ↔ ρ(N) = N and the root has M as its parent. Full
ep-heaps are ep-heaps leading to top, where 'top' is assumed to be a special
value strictly smaller than any other (thus σ(R) is always false for the root
of a full ep-heap). For simplicity we will take all heaps to end at a special
node called 'bottom' whose value is assumed to be greater than any other value.
The critical step here is adding a new value C M to combine two existing
well-formed ep-heaps P and R leading to the same M into a bigger single
well-formed ep-heap leading to M. Without loss of generality we can take P ≤ R
(otherwise we can swap the places of P and R in the discussion below).
We distinguish three cases:
# C P (necessarily ¬σ(P)): We place C at the top by setting ρ(P), ρ(R) := C,
ρ(C) := M and leaving σ(P), σ(R) := false (Note that σ(R) cannot be true as
that contradicts with C M). σ(C) := false. The ep-heap rooted at C is
well-formed leading to M.
# C = P (necessarily ¬σ(P)): We place C at the top, setting ρ(P), ρ(R) := C,
ρ(C) := M, σ(P) := true and σ(R) := (P = R). σ(C) := false. The heap rooted at
C is well-formed as C = P σ(P) as well as C = P ≤ R C = R if and only if
σ(R).
# C P (σ(P) can be true or false): Set ρ(R) := P. If ¬σ(P), set σ(R) := (P =
R), otherwise leave σ(R) unchanged. Use the algorithm recursively to turn the
two ep-subheaps leading to P at its children and the new value C P into an
ep-heap leading to P. Let its root be Q. The necessary conditions for an
element of an ep-heap hold for Q. They are also true for R as P ≤ R and either
σ(P) in which case P = M and σ(R) if and only if R = M = P, or ¬σ(P) and σ(R)
is explicitly set to be true if and only if P = R. We also have ρ(P) = M as
well as P = M if and only if σ(P), thus the constructed ep-heap rooted at P
leads to M.
The recursion always ends for finite heaps as we must reach the first case due
to the ordering of bottom.
For performance the order of comparing elements and identifying the case to use
can be changed to sequentially checking:
- σ(P): apply pt. 3 for P and return (P must be ≤ R). No update of σ(P)/σ(R)
necessary.
- σ(R): apply pt. 3 for R and return (R must now be P). No update of
σ(P)/σ(R) necessary.
- compare P and R.
- If R P, perform the next steps with P and R swapped.
- compare C and P.
- if C P apply pt. 1.
- if C = P apply pt. 2.
- if C P apply pt. 3.
Once an ep-heap with root R is formed, it trivially follows that for each
element E in the heap E = R if and only if σ is true for E and the chain of
parents of E leading to but not including the root. To consume the equal
elements we can thus walk the sub-tree formed by the children which have σ set.
To advance the elements equal to the root, we can:
- Process the elements in the deepest level of the σ sub-tree in any order. For
each of them:
-- Advance the element. Let the new value be C.
-- The children P, R of the element must have ¬σ(P), ¬σ(R). Overwrite ρ(P)
and ρ(R) with top*. P and R now form ep-heaps leading to top.
-- Use the algorithm to combine P, R and C into a new ep-heap leading to top.
-- Let its root be Q. Because the heap leads to top we have ¬σ(Q).
- Continue with the next deepest level of the hierarchy until the top is
reached.
\* The algorithm never actually uses the values of the parent function ρ, only
the fact that σ is set correctly with respect to that parent. This means this
step does not need to be actually performed.
{panel}
Updated [the
branch|https://github.com/apache/cassandra/compare/trunk...blambov:8915-mergeiterator]
to use this solution and cleaned the code a bit.
Improve MergeIterator performance
-
Key: CASSANDRA-8915
URL: https://issues.apache.org/jira/browse/CASSANDRA-8915
Project: Cassandra
Issue Type: Improvement
Reporter: Branimir Lambov
Assignee: Branimir Lambov
Priority: Minor
Labels: compaction, performance
Fix For: 3.x
The implementation of {{MergeIterator}} uses a priority queue and applies a
pair of {{poll}}+{{add}} operations for every item in the resulting sequence.
This is quite inefficient as {{poll}} necessarily applies at least {{log N}}
comparisons (up to {{2log N}}), and {{add}}
