[jira] [Updated] (CASSANDRA-10571) ClusteringIndexNamesFilter::shouldInclude is not implemented, SinglePartitionNamesCommand not discarding the sstables it could

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-10571:
--
Description: 
Now that we use {{SinglePartitionNamesCommand}} in more places - where we'd 
previously use what is now {{SinglePartitionSliceCommand}} - not being able to 
skip sstables with non-overlapping clusterings is actually a performance 
regression.

{{SinglePartitionNamesCommand::queryMemtableAndDiskInternal}} should prune 
sstables based on {{ClusteringIndexNamesFilter::shouldInclude}} output, and the 
latter must be replaced with an actual implementation instead of a {{TODO}}.

This is also a potentially a big regression in performance for counter writes 
(say, with DTCS), since before 3.0, the read-before-write code would use 
{{collectAllData}}, and that *was* pruning sstable with non-overlapping 
clusterings.

  was:
Now that we use {{SinglePartitionNamesCommand}} in more places - where we'd 
previously use what is now {{SinglePartitionSliceCommand}} - not being able to 
skip sstables with non-overlapping clusterings is actually a performance 
regression.

{{SinglePartitionNamesCommand::queryMemtableAndDiskInternal}} should prune 
sstables based on {{ClusteringIndexNamesFilter::shouldInclude}} output, and the 
latter must be replaced with an actual implementation instead of a {{TODO}}.

Not a regression, but not doing so affects counts write path negatively.


> ClusteringIndexNamesFilter::shouldInclude is not implemented, 
> SinglePartitionNamesCommand not discarding the sstables it could
> --
>
> Key: CASSANDRA-10571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10571
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Aleksey Yeschenko
>Assignee: Sylvain Lebresne
> Fix For: 3.0.0
>
>
> Now that we use {{SinglePartitionNamesCommand}} in more places - where we'd 
> previously use what is now {{SinglePartitionSliceCommand}} - not being able 
> to skip sstables with non-overlapping clusterings is actually a performance 
> regression.
> {{SinglePartitionNamesCommand::queryMemtableAndDiskInternal}} should prune 
> sstables based on {{ClusteringIndexNamesFilter::shouldInclude}} output, and 
> the latter must be replaced with an actual implementation instead of a 
> {{TODO}}.
> This is also a potentially a big regression in performance for counter writes 
> (say, with DTCS), since before 3.0, the read-before-write code would use 
> {{collectAllData}}, and that *was* pruning sstable with non-overlapping 
> clusterings.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-10572) SinglePartitionNamesCommand::canRemoveRow omits counter cells it shouldn't

[Aleksey Yeschenko (JIRA)]

Aleksey Yeschenko created CASSANDRA-10572:
-

 Summary: SinglePartitionNamesCommand::canRemoveRow omits counter 
cells it shouldn't
 Key: CASSANDRA-10572
 URL: https://issues.apache.org/jira/browse/CASSANDRA-10572
 Project: Cassandra
  Issue Type: Bug
Reporter: Aleksey Yeschenko
Assignee: Sylvain Lebresne
 Fix For: 3.0.0


Like with collections, we can never be sure we've exhausted all the sstables 
when reducing the filter. The sstables can contain legacy local shards that 
need to be summed up. Unless we look into the content of the cells (?), or 
until we get rid of the pre-2.1 values, we must include all the sstables in the 
read (so long as clusterings overlap).

This was not an issue with 2.1/2.2 as we've forced {{collectAllData}} always on 
that path, but now we don't, and {{SinglePartitionNamesCommand}} should handle 
the case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10258) Counter table written with CQLSSTableWriter generates exceptions and become corrupted at first use

[Paulo Motta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-10258:

Fix Version/s: 3.0.x
   2.2.x

> Counter table written with CQLSSTableWriter generates exceptions and become 
> corrupted at first use
> --
>
> Key: CASSANDRA-10258
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10258
> Project: Cassandra
>  Issue Type: Bug
>  Components: API
> Environment: Linux Debian Wheezie 7.8 / Oracle Java 1.7.0_67
> Ubuntu 14.04.3 LTS / Oracle Java 1.7.0_75
> Cassandra 2.0.12 2.1.5 2.1.8
>Reporter: Guillaume VIEL
>Assignee: Paulo Motta
> Fix For: 2.1.x, 2.2.x, 3.0.x
>
>
> We use CQLSStableWriter to produce testing datasets.
> Here are the steps to reproduce this issue :
> 1) definition of a table with counter
> {code}
> CREATE TABLE my_counter (
>   my_id text,
>   my_counter counter,
>   PRIMARY KEY (my_id)
> )
> {code}
> 2) with CQLSSTableWriter initialize this table (about 2millions entries) with 
> this insert order (one insert / key only)
> {{UPDATE myks.my_counter SET my_counter = my_counter + ? WHERE my_id = ?}}
> 3) load the files written by CQLSSTableWriter with sstableloader in your 
> cassandra cluster (tested on a single node and a 3 nodes cluster)
> 4) start a process that updates the counters (we used 3millions entries 
> distributed on the key my_id)
> 5) after a while try to query a key in the my_counter table
> {{cqlsh:myks> select * from my_counter where my_id='001';}}
> Request did not complete within rpc_timeout.
> In the logs of cassandra (2.0.12) :
> {code}
> ERROR [CompactionExecutor:3] 2015-05-28 15:53:39,491 CassandraDaemon.java 
> (line 258) Exception in thread Thread[CompactionExecutor:3,1,main]
> java.lang.AssertionError: Wrong class type.
> at 
> org.apache.cassandra.db.CounterUpdateColumn.reconcile(CounterUpdateColumn.java:70)
> at 
> org.apache.cassandra.db.ArrayBackedSortedColumns.resolveAgainst(ArrayBackedSortedColumns.java:147)
> at 
> org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:126)
> at 
> org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:121)
> at 
> org.apache.cassandra.db.compaction.PrecompactedRow$1.reduce(PrecompactedRow.java:120)
> at 
> org.apache.cassandra.db.compaction.PrecompactedRow$1.reduce(PrecompactedRow.java:115)
> at 
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:112)
> at 
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98)
> at 
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at 
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at 
> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:191)
> at 
> org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:144)
> at 
> org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:103)
> at 
> org.apache.cassandra.db.compaction.PrecompactedRow.(PrecompactedRow.java:85)
> at 
> org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:196)
> at 
> org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:74)
> at 
> org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:55)
> at 
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:115)
> at 
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98)
> at 
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at 
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:164)
> at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at 
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at 
> 

[jira] [Commented] (CASSANDRA-10558) Fix out-of-order scrub test

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968682#comment-14968682
 ] 

Sylvain Lebresne commented on CASSANDRA-10558:
--

+1

> Fix out-of-order scrub test
> ---
>
> Key: CASSANDRA-10558
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10558
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: Sylvain Lebresne
>Assignee: Yuki Morishita
> Fix For: 3.0.0
>
>
> This is kind of a followup of CASSANDRA-9880.
> As the name imply, the out-of-order scrub test 
> ({{ScrubTest.testScrubOutOfOrder}}) test scrubs detects a sstable with 
> out-of-order partitions, which imply we have a generated sstable with out of 
> order partitions. That latter part being not trivial since we have a sanity 
> check in {{BigTableWriter}} that we never write out-of-order partitions. And 
> hence, generating such a sstable used to be manual: you had to comment the 
> check in {{BigTableWriter}} and uncomment some code in {{ScrubTest}} to 
> generate the sstable. That did imply you had to manually regenerate the 
> sstable each time the sstable format was changed.
> I had do to such manipulation in CASSANDRA-9705 (as there was some change to 
> the sstable format) and unfortunately I screwed up and included the 
> uncommenting of the sanity check in {{BigTableWriter}} in the commit.
> In the meantime, came CASSANDRA-9880, which aimed to make the manual process 
> of generating out-of-order sstable automatic. Unfortunately, that ticket was 
> done with the {{BigTableWriter}} sanity check commented, which did make 
> things a lot easier for the test, but was kind of wrong.
> So anyway, commenting the check in {{BigTableWriter}} was an oversight and I 
> uncommented it on the cassandra-3.0/trunk branches. The problem is that 
> {{ScrubTest.testScrubOutOfOrder}} now fails due to that check. We need to 
> find a way to make that test work without disabling our sanity check on 
> writes. Not sure what's the best solution though to be honest. Maybe we'll 
> need a flag in {{BigTableWriter}} to disable that check just for the test?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10544) Make cqlsh tests work when authentication is configured

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968683#comment-14968683
 ] 

Stefania commented on CASSANDRA-10544:
--

Patch for 2.1 [here|https://github.com/stef1927/cassandra/tree/10544-2.1]

CI: 
http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-10544-2.1-dtest

[~blerer] would you mind reviewing?

> Make cqlsh tests work when authentication is configured
> ---
>
> Key: CASSANDRA-10544
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10544
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tests, Tools
>Reporter: Adam Holmberg
>Assignee: Stefania
>Priority: Trivial
>  Labels: cqlsh
> Fix For: 2.1.x, 2.2.x, 3.0.x
>
>
> cqlsh tests break if the runner has an authentication section in their 
> ~/.cassandra/cqlshrc, because cqlsh changes the prompt and the tests scan 
> output for a prompt. It manifests as read timeouts while waiting for a prompt 
> in test/run_cqlsh.py.
> [This 
> pattern|https://github.com/mambocab/cassandra/blob/1c27f9be1ba8ea10dbe843d513e23de6238dede8/pylib/cqlshlib/test/run_cqlsh.py#L30]
>  could be generalized to match the "@cqlsh..." prompt that arises 
> with this config.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10544) Make cqlsh tests work when authentication is configured

[Stefania (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefania updated CASSANDRA-10544:
-
Fix Version/s: 2.1.x

> Make cqlsh tests work when authentication is configured
> ---
>
> Key: CASSANDRA-10544
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10544
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tests, Tools
>Reporter: Adam Holmberg
>Assignee: Stefania
>Priority: Trivial
>  Labels: cqlsh
> Fix For: 2.1.x, 2.2.x, 3.0.x
>
>
> cqlsh tests break if the runner has an authentication section in their 
> ~/.cassandra/cqlshrc, because cqlsh changes the prompt and the tests scan 
> output for a prompt. It manifests as read timeouts while waiting for a prompt 
> in test/run_cqlsh.py.
> [This 
> pattern|https://github.com/mambocab/cassandra/blob/1c27f9be1ba8ea10dbe843d513e23de6238dede8/pylib/cqlshlib/test/run_cqlsh.py#L30]
>  could be generalized to match the "@cqlsh..." prompt that arises 
> with this config.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10091) Align JMX authentication with internal authentication

[Jan Karlsson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968712#comment-14968712
 ] 

Jan Karlsson commented on CASSANDRA-10091:
--

{quote}
For instance, do we actually need to enact fine grained control over nodetool 
at the keyspace or table level, such that a user with permissions on keyspace 
ks_a would be able to run nodetool status ks_a, but not to run nodetool status 
ks_b? I think that's overkill and not really needed by most admins. 
{quote}
The current patch does not restrict access from nodetool in terms of which 
keyspace/table the command is run on. This is due to nodetool calling methods 
in the {{StorageProxy}}. However if someone were to call these methods from a 
specific columnfamily, it would prevent that. I believe preventing users from 
initiating operations like major compaction on a some tables but not on others 
is a fairly common use case. Especially when we provide so many potentially, in 
some cases detrimental operations like {{compact}}. Unfortunately this patch 
does not give this distinction on nodetool level, because you either have the 
permission for StorageProxy or you do not. However it does give you the choice 
to make that distinction on non-nodetool users.

{quote}
So for example, this would enable us to grant read access to all the 
ColumnFamily mbeans with GRANT SELECT ON ALL MBEANS IN 
'org.apache.cassandra.db:type=ColumnFamily', e.g. for running nodetool cfstats. 
What it doesn't permit is restricting access to a particular subset of 
ColumnFamily beans.
{quote}
Another disadvantage is that if the client application (for example I observed 
jconsole doing this) sends a jmx request with a wildcard mbean. For instance it 
might send something like {{java.lang:*}} or a wildcard would be send in when a 
program is trying to receive the names of all mbeans. Now the latter instance 
might not be so difficult to handle with your proposal, since {{queryNames}} 
and {{isInstanceOf}} are granted to everyone, but there might be other cases 
where wildcard mbean are being passed in. We would have to handle this somehow. 
Otherwise applications who pass wildcard mbean will have to have root 
permission.

{quote}
Also, I noticed one other thing regarding the MBeanServerForwarder 
implementation. We should create a new ClientState and log the 
AuthenticatedUser derived from the subject into it, which would have a couple 
of benefits. Firstly, the check that the user has the LOGIN privilege would be 
performed which isn't the case in the current patch. Second, the permissions 
check could include the full resource hierarchy using ensureHasPermission, 
rather than directly by calling the IAuthorizer::authorize.
{quote}
+1

Another aspect we need to remember is that currently there is no way to 
ascertain which mbeans are needed for a particular nodetool commands or for the 
different tools that exist (like jconsole). We probably need to document this 
somewhere.

> Align JMX authentication with internal authentication
> -
>
> Key: CASSANDRA-10091
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10091
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Core
>Reporter: Jan Karlsson
>Assignee: Jan Karlsson
>Priority: Minor
> Fix For: 3.x
>
>
> It would be useful to authenticate with JMX through Cassandra's internal 
> authentication. This would reduce the overhead of keeping passwords in files 
> on the machine and would consolidate passwords to one location. It would also 
> allow the possibility to handle JMX permissions in Cassandra.
> It could be done by creating our own JMX server and setting custom classes 
> for the authenticator and authorizer. We could then add some parameters where 
> the user could specify what authenticator and authorizer to use in case they 
> want to make their own.
> This could also be done by creating a premain method which creates a jmx 
> server. This would give us the feature without changing the Cassandra code 
> itself. However I believe this would be a good feature to have in Cassandra.
> I am currently working on a solution which creates a JMX server and uses a 
> custom authenticator and authorizer. It is currently build as a premain, 
> however it would be great if we could put this in Cassandra instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9304) COPY TO improvements

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968782#comment-14968782
 ] 

Stefania commented on CASSANDRA-9304:
-

A really interesting fact, it takes about 55 seconds to export 500K rows on my 
box, with a single node running. If instead of formatting rows we just dump 
binary rows, then it takes 5 seconds total. 

> COPY TO improvements
> 
>
> Key: CASSANDRA-9304
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9304
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Jonathan Ellis
>Assignee: Stefania
>Priority: Minor
>  Labels: cqlsh
> Fix For: 3.x, 2.1.x, 2.2.x
>
>
> COPY FROM has gotten a lot of love.  COPY TO not so much.  One obvious 
> improvement could be to parallelize reading and writing (write one page of 
> data while fetching the next).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-10568) LegacySSTableTest failed on cassandra 2.1.10

[Meerabo Shah (JIRA)]

Meerabo Shah created CASSANDRA-10568:


 Summary: LegacySSTableTest failed on cassandra 2.1.10
 Key: CASSANDRA-10568
 URL: https://issues.apache.org/jira/browse/CASSANDRA-10568
 Project: Cassandra
  Issue Type: Bug
  Components: Tests
 Environment: s390x RHEL7 IBM Systemz
Reporter: Meerabo Shah
 Fix For: 2.1.10


[junit] -  ---
[junit] - Standard Error -
[junit] Failed to read jb
[junit] -  ---
[junit] Testcase: 
testStreaming(org.apache.cassandra.io.sstable.LegacySSTableTest): FAILED
[junit] null
[junit] junit.framework.AssertionFailedError
[junit] at 
org.apache.cassandra.io.sstable.IndexSummary.(IndexSummary.java:80)
[junit] at 
org.apache.cassandra.io.sstable.IndexSummary$IndexSummarySerializer.deserialize(IndexSummary.java:317)
[junit] at 
org.apache.cassandra.io.sstable.SSTableReader.loadSummary(SSTableReader.java:877)
[junit] at 
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:730)
[junit] at 
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:692)
[junit] at 
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:480)
[junit] at 
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:376)
[junit] at 
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:371)
[junit] at 
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:363)
[junit] at 
org.apache.cassandra.io.sstable.LegacySSTableTest.testStreaming(LegacySSTableTest.java:115)
[junit] at 
org.apache.cassandra.io.sstable.LegacySSTableTest.testStreaming(LegacySSTableTest.java:110)
[junit]
[junit]
[junit] Testcase: 
testVersions(org.apache.cassandra.io.sstable.LegacySSTableTest):  FAILED
[junit] null
[junit] junit.framework.AssertionFailedError
[junit] at 
org.apache.cassandra.io.sstable.IndexSummary.(IndexSummary.java:80)
[junit] at 
org.apache.cassandra.io.sstable.IndexSummary$IndexSummarySerializer.deserialize(IndexSummary.java:317)
[junit] at 
org.apache.cassandra.io.sstable.SSTableReader.loadSummary(SSTableReader.java:877)
[junit] at 
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:730)
[junit] at 
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:692)
[junit] at 
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:480)
[junit] at 
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:376)
[junit] at 
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:371)
[junit] at 
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:363)
[junit] at 
org.apache.cassandra.io.sstable.LegacySSTableTest.testVersion(LegacySSTableTest.java:165)
[junit] at 
org.apache.cassandra.io.sstable.LegacySSTableTest.testVersions(LegacySSTableTest.java:154)
[junit]
[junit]
[junit] Test org.apache.cassandra.io.sstable.LegacySSTableTest FAILED
[junitreport] Processing 
/apache-cassandra-2.1.10-src/build/test/TESTS-TestSuites.xml to 
/tmp/null1914220216
[junitreport] Loading stylesheet 
jar:file:/apache-ant-1.9.2/lib/ant-junit.jar!/org/apache/tools/ant/taskdefs/optional/junit/xsl/junit-frames.xsl
[junitreport] Transform time: 2188ms
[junitreport] Deleting: /tmp/null1914220216

BUILD FAILED



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10544) Make cqlsh tests work when authentication is configured

[Stefania (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefania updated CASSANDRA-10544:
-
Labels: cqlsh test  (was: cqlsh)

> Make cqlsh tests work when authentication is configured
> ---
>
> Key: CASSANDRA-10544
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10544
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tests, Tools
>Reporter: Adam Holmberg
>Assignee: Stefania
>Priority: Trivial
>  Labels: cqlsh, test
> Fix For: 2.1.x, 2.2.x, 3.0.x
>
>
> cqlsh tests break if the runner has an authentication section in their 
> ~/.cassandra/cqlshrc, because cqlsh changes the prompt and the tests scan 
> output for a prompt. It manifests as read timeouts while waiting for a prompt 
> in test/run_cqlsh.py.
> [This 
> pattern|https://github.com/mambocab/cassandra/blob/1c27f9be1ba8ea10dbe843d513e23de6238dede8/pylib/cqlshlib/test/run_cqlsh.py#L30]
>  could be generalized to match the "@cqlsh..." prompt that arises 
> with this config.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10544) Make cqlsh tests work when authentication is configured

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968684#comment-14968684
 ] 

Stefania commented on CASSANDRA-10544:
--

The patch applies to 2.2+ as well so I did not create multiple versions.

> Make cqlsh tests work when authentication is configured
> ---
>
> Key: CASSANDRA-10544
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10544
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tests, Tools
>Reporter: Adam Holmberg
>Assignee: Stefania
>Priority: Trivial
>  Labels: cqlsh
> Fix For: 2.1.x, 2.2.x, 3.0.x
>
>
> cqlsh tests break if the runner has an authentication section in their 
> ~/.cassandra/cqlshrc, because cqlsh changes the prompt and the tests scan 
> output for a prompt. It manifests as read timeouts while waiting for a prompt 
> in test/run_cqlsh.py.
> [This 
> pattern|https://github.com/mambocab/cassandra/blob/1c27f9be1ba8ea10dbe843d513e23de6238dede8/pylib/cqlshlib/test/run_cqlsh.py#L30]
>  could be generalized to match the "@cqlsh..." prompt that arises 
> with this config.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10544) Make cqlsh tests work when authentication is configured

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer updated CASSANDRA-10544:
---
Assignee: Stefania  (was: Benjamin Lerer)
Reviewer: Benjamin Lerer

> Make cqlsh tests work when authentication is configured
> ---
>
> Key: CASSANDRA-10544
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10544
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tests, Tools
>Reporter: Adam Holmberg
>Assignee: Stefania
>Priority: Trivial
>  Labels: cqlsh, test
> Fix For: 2.1.x, 2.2.x, 3.0.x
>
>
> cqlsh tests break if the runner has an authentication section in their 
> ~/.cassandra/cqlshrc, because cqlsh changes the prompt and the tests scan 
> output for a prompt. It manifests as read timeouts while waiting for a prompt 
> in test/run_cqlsh.py.
> [This 
> pattern|https://github.com/mambocab/cassandra/blob/1c27f9be1ba8ea10dbe843d513e23de6238dede8/pylib/cqlshlib/test/run_cqlsh.py#L30]
>  could be generalized to match the "@cqlsh..." prompt that arises 
> with this config.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (CASSANDRA-10544) Make cqlsh tests work when authentication is configured

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer reassigned CASSANDRA-10544:
--

Assignee: Benjamin Lerer  (was: Stefania)

> Make cqlsh tests work when authentication is configured
> ---
>
> Key: CASSANDRA-10544
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10544
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tests, Tools
>Reporter: Adam Holmberg
>Assignee: Benjamin Lerer
>Priority: Trivial
>  Labels: cqlsh, test
> Fix For: 2.1.x, 2.2.x, 3.0.x
>
>
> cqlsh tests break if the runner has an authentication section in their 
> ~/.cassandra/cqlshrc, because cqlsh changes the prompt and the tests scan 
> output for a prompt. It manifests as read timeouts while waiting for a prompt 
> in test/run_cqlsh.py.
> [This 
> pattern|https://github.com/mambocab/cassandra/blob/1c27f9be1ba8ea10dbe843d513e23de6238dede8/pylib/cqlshlib/test/run_cqlsh.py#L30]
>  could be generalized to match the "@cqlsh..." prompt that arises 
> with this config.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-10564) AntiCompactionTest failed on cassandra 2.1.10 due to antiCompactionSizeTest method failed on s390x

[Meerabo Shah (JIRA)]

Meerabo Shah created CASSANDRA-10564:


 Summary: AntiCompactionTest failed on cassandra 2.1.10 due to 
antiCompactionSizeTest method failed on s390x
 Key: CASSANDRA-10564
 URL: https://issues.apache.org/jira/browse/CASSANDRA-10564
 Project: Cassandra
  Issue Type: Bug
  Components: Tests
 Environment: s390x - RHEL7 IBM systemz
Reporter: Meerabo Shah
Priority: Blocker
 Fix For: 2.1.10






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10564) AntiCompactionTest failed on cassandra 2.1.10 due to antiCompactionSizeTest method failed on s390x

[Meerabo Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Meerabo Shah updated CASSANDRA-10564:
-
Description: 
testlist:
 [echo] running test bucket 0 tests
[junit] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[junit] Testsuite: org.apache.cassandra.db.compaction.AntiCompactionTest
[junit] Testsuite: org.apache.cassandra.db.compaction.AntiCompactionTest
[junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 
sec
[junit]
[junit] Testcase: 
org.apache.cassandra.db.compaction.AntiCompactionTest:antiCompactionSizeTest:   
  Caused an ERROR
[junit] Timeout occurred. Please note the time in the report does not 
reflect the time until the timeout.
[junit] junit.framework.AssertionFailedError: Timeout occurred. Please note 
the time in the report does not reflect the time until the timeout.
[junit] at java.lang.Thread.run(Thread.java:745)
[junit]
[junit]
[junit] Test org.apache.cassandra.db.compaction.AntiCompactionTest FAILED 
(timeout)
[junitreport] Processing 
/apache-cassandra-2.1.10-src/build/test/TESTS-TestSuites.xml to 
/tmp/null155310269
[junitreport] Loading stylesheet 
jar:file:/apache-ant-1.9.2/lib/ant-junit.jar!/org/apache/tools/ant/taskdefs/optional/junit/xsl/junit-frames.xsl
[junitreport] Transform time: 1421ms
[junitreport] Deleting: /tmp/null155310269

BUILD FAILED
/apache-cassandra-2.1.10-src/build.xml:1146: Some test bucket 0 test(s) failed.

Total time: 1 minute 11 seconds


> AntiCompactionTest failed on cassandra 2.1.10 due to antiCompactionSizeTest 
> method failed on s390x
> --
>
> Key: CASSANDRA-10564
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10564
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tests
> Environment: s390x - RHEL7 IBM systemz
>Reporter: Meerabo Shah
>Priority: Blocker
> Fix For: 2.1.10
>
>
> testlist:
>  [echo] running test bucket 0 tests
> [junit] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
> [junit] Testsuite: org.apache.cassandra.db.compaction.AntiCompactionTest
> [junit] Testsuite: org.apache.cassandra.db.compaction.AntiCompactionTest
> [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 
> sec
> [junit]
> [junit] Testcase: 
> org.apache.cassandra.db.compaction.AntiCompactionTest:antiCompactionSizeTest: 
> Caused an ERROR
> [junit] Timeout occurred. Please note the time in the report does not 
> reflect the time until the timeout.
> [junit] junit.framework.AssertionFailedError: Timeout occurred. Please 
> note the time in the report does not reflect the time until the timeout.
> [junit] at java.lang.Thread.run(Thread.java:745)
> [junit]
> [junit]
> [junit] Test org.apache.cassandra.db.compaction.AntiCompactionTest FAILED 
> (timeout)
> [junitreport] Processing 
> /apache-cassandra-2.1.10-src/build/test/TESTS-TestSuites.xml to 
> /tmp/null155310269
> [junitreport] Loading stylesheet 
> jar:file:/apache-ant-1.9.2/lib/ant-junit.jar!/org/apache/tools/ant/taskdefs/optional/junit/xsl/junit-frames.xsl
> [junitreport] Transform time: 1421ms
> [junitreport] Deleting: /tmp/null155310269
> BUILD FAILED
> /apache-cassandra-2.1.10-src/build.xml:1146: Some test bucket 0 test(s) 
> failed.
> Total time: 1 minute 11 seconds



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

cassandra git commit: Make cqlsh support python-2.6 (FOLLOW-UP CASSANDRA-10415)

[blerer]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.1 29ff1f2ac -> 5a1d37648


Make cqlsh support python-2.6 (FOLLOW-UP CASSANDRA-10415)

patch by Stefania Alborghetti; reviewed by Benjamin Lerer for CASSANDRA-10415


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5a1d3764
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5a1d3764
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5a1d3764

Branch: refs/heads/cassandra-2.1
Commit: 5a1d376489e5884b381e3bee3afd22130945782d
Parents: 29ff1f2
Author: Stefania Alborghetti 
Authored: Thu Oct 22 10:12:04 2015 +0200
Committer: blerer 
Committed: Thu Oct 22 10:17:17 2015 +0200

--
 pylib/cqlshlib/cqlhandling.py| 2 +-
 pylib/cqlshlib/test/test_cqlsh_completion.py | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5a1d3764/pylib/cqlshlib/cqlhandling.py
--
diff --git a/pylib/cqlshlib/cqlhandling.py b/pylib/cqlshlib/cqlhandling.py
index 9ea30cd..6ee3cf5 100644
--- a/pylib/cqlshlib/cqlhandling.py
+++ b/pylib/cqlshlib/cqlhandling.py
@@ -63,7 +63,7 @@ class CqlParsingRuleSet(pylexotron.ParsingRuleSet):
 We cannot let resreved cql keywords be simple 'identifier' since this 
caused
 problems with completion, see CASSANDRA-10415
 """
-syntax = ' ::= /(' + '|'.join(r'\b{}\b'.format(k) 
for k in keywords) + ')/ ;'
+syntax = ' ::= /(' + '|'.join(r'\b%s\b' % (k,) 
for k in keywords) + ')/ ;'
 self.append_rules(syntax)
 
 def completer_for(self, rulename, symname):

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5a1d3764/pylib/cqlshlib/test/test_cqlsh_completion.py
--
diff --git a/pylib/cqlshlib/test/test_cqlsh_completion.py 
b/pylib/cqlshlib/test/test_cqlsh_completion.py
index 5f7b6e4..04cee5d 100644
--- a/pylib/cqlshlib/test/test_cqlsh_completion.py
+++ b/pylib/cqlshlib/test/test_cqlsh_completion.py
@@ -76,7 +76,7 @@ class CqlshCompletionCase(BaseTestCase):
 # ensure the last line of the completion is the prompt
 prompt_regex = self.cqlsh.prompt.lstrip() + re.escape(inputstring)
 msg = ('Double-tab completion '
-   'does not print prompt for input "{}"'.format(inputstring))
+   'does not print prompt for input "%s"' % (inputstring, ))
 self.assertRegexpMatches(choice_lines[-1], prompt_regex, msg=msg)
 
 choice_lines = [line.strip() for line in choice_lines[:-1]]

[jira] [Commented] (CASSANDRA-10415) Fix cqlsh bugs

[Benjamin Lerer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968786#comment-14968786
 ] 

Benjamin Lerer commented on CASSANDRA-10415:


Committed in 2.1 at 5a1d376489e5884b381e3bee3afd22130945782d. In 2.2+ we will 
only support python-2.7 (see CASSANDRA-10561)

> Fix cqlsh bugs
> --
>
> Key: CASSANDRA-10415
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10415
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: Jim Witschey
>Assignee: Stefania
>  Labels: cqlsh
> Fix For: 3.0.0 rc2, 2.2.4, 2.1.12
>
>
> This is followup to CASSANDRA-10289
> The tests currently failing should be:
> * 
> {{cqlshlib.test.test_cqlsh_completion.TestCqlshCompletion.test_complete_in_create_columnfamily}}
> ** uses {{create_columnfamily_table_template}}. Stefania says "the {{(}} 
> after {{CREATE ... IF}} does not look valid to me."
> * 
> {{cqlshlib.test.test_cqlsh_completion.TestCqlshCompletion.test_complete_in_create_table}}
> ** uses {{create_columnfamily_table_template}}, see above.
> * 
> {{cqlshlib.test.test_cqlsh_completion.TestCqlshCompletion.test_complete_in_delete}}
> ** Stefania says: "I don't think keyspaces are a valid completion after 
> {{DELETE a [}} and after {{DELETE FROM twenty_rows_composite_table USING 
> TIMESTAMP 0 WHERE TOKEN(a) >=}}. From a quick analysis of {{cqlhandling.py}} 
> I think it comes from {{}}, which picks up {{}}, which 
> was changed to include {{ks.}} by CASSANDRA-7556.
> * 
> {{cqlshlib.test.test_cqlsh_completion.TestCqlshCompletion.test_complete_in_drop_keyspace}}
> ** Stefania says: "the {{;}} after {{DROP KEYSPACE IF}} is not valid.
> * {{cqlshlib.test.test_cqlsh_output.TestCqlshOutput.test_timestamp_output}}
> ** already documented with CASSANDRA-10313 and CASSANDRA-10397
> I'm happy to break these out into separate tickets if necessary. 
> To run the tests locally, I cd to {{cassandra/pylib/cqlshlib}} and run the 
> following:
> {code}
> ccm create -n 1 --install-dir=../.. test
> ccm start --wait-for-binary-proto
> nosetests test 2>&1
> ccm remove
> {code}
> This requires nose and ccm. Until CASSANDRA-10289 is resolved, you'll have to 
> use my branch here: https://github.com/mambocab/cassandra/tree/fix-cqlsh-tests
> Tests for this branch are run (non-continuously) here:
> http://cassci.datastax.com/job/scratch_mambocab-fix_cqlsh/
> Assigning [~Stefania] for now, since she's already looked at 10289, but feel 
> free to reassign.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10421) Potential issue with LogTransaction as it only checks in a single directory for files

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968676#comment-14968676
 ] 

Stefania commented on CASSANDRA-10421:
--

CI testall seems fine now. After 2 runs, failing tests are similar to 3.0 
({{testRecoverPITUnordered}} also timed out on 3.0, build #202).

> Potential issue with LogTransaction as it only checks in a single directory 
> for files
> -
>
> Key: CASSANDRA-10421
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10421
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Stefania
>Priority: Blocker
> Fix For: 3.0.0
>
>
> When creating a new LogTransaction we try to create the new logfile in the 
> same directory as the one we are writing to, but as we use 
> {{[directories.getDirectoryForNewSSTables()|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/lifecycle/LogTransaction.java#L125]}}
>  this might end up in "any" of the configured data directories. If it does, 
> we will not be able to clean up leftovers as we check for files in the same 
> directory as the logfile was created: 
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/lifecycle/LogRecord.java#L163
> cc [~Stefania]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (CASSANDRA-10561) Add a check to cqlsh to require Python-2.7 for version >= 2.2

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer reassigned CASSANDRA-10561:
--

Assignee: Benjamin Lerer

> Add a check to cqlsh to require Python-2.7 for version >= 2.2
> -
>
> Key: CASSANDRA-10561
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10561
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Benjamin Lerer
>Assignee: Benjamin Lerer
>Priority: Minor
>
> Not everybody is reading the {{README}} file. Up to now cqlsh was still 
> working with python 2.6 by consequence user complains when we use some 
> python-2.7 features.
> We should still allow python-2.6 with {{2.1}} but enforce the use of python 
> 2.7 in {{2.2+}}. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-10565) NanoTimeToCurrentTimeMillisTest Failed on s390x

[Meerabo Shah (JIRA)]

Meerabo Shah created CASSANDRA-10565:


 Summary: NanoTimeToCurrentTimeMillisTest Failed on s390x
 Key: CASSANDRA-10565
 URL: https://issues.apache.org/jira/browse/CASSANDRA-10565
 Project: Cassandra
  Issue Type: Bug
  Components: Tests
 Environment: s390x RHEL7 IBM systemz
Reporter: Meerabo Shah
 Fix For: 2.1.10


testlist:
 [echo] running test bucket 0 tests
[junit] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[junit] Testsuite: 
org.apache.cassandra.utils.NanoTimeToCurrentTimeMillisTest
[junit] Testsuite: 
org.apache.cassandra.utils.NanoTimeToCurrentTimeMillisTest
[junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 
sec
[junit]
[junit] Testcase: 
org.apache.cassandra.utils.NanoTimeToCurrentTimeMillisTest:testTimestampOrdering:
 Caused an ERROR
[junit] Timeout occurred. Please note the time in the report does not 
reflect the time until the timeout.
[junit] junit.framework.AssertionFailedError: Timeout occurred. Please note 
the time in the report does not reflect the time until the timeout.
[junit] at java.lang.Thread.run(Thread.java:745)
[junit]
[junit]
[junit] Test org.apache.cassandra.utils.NanoTimeToCurrentTimeMillisTest 
FAILED (timeout)
[junitreport] Processing 
/apache-cassandra-2.1.10-src/build/test/TESTS-TestSuites.xml to 
/tmp/null714671406
[junitreport] Loading stylesheet 
jar:file:/apache-ant-1.9.2/lib/ant-junit.jar!/org/apache/tools/ant/taskdefs/optional/junit/xsl/junit-frames.xsl
[junitreport] Transform time: 1430ms
[junitreport] Deleting: /tmp/null714671406

BUILD FAILED
/apache-cassandra-2.1.10-src/build.xml:1146: Some test bucket 0 test(s) failed.

Total time: 1 minute 8 seconds



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-10570) HSHA test_closing_connections test flapping on 3.0

[Jim Witschey (JIRA)]

Jim Witschey created CASSANDRA-10570:


 Summary: HSHA test_closing_connections test flapping on 3.0
 Key: CASSANDRA-10570
 URL: https://issues.apache.org/jira/browse/CASSANDRA-10570
 Project: Cassandra
  Issue Type: Sub-task
Reporter: Jim Witschey


See history here:

http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/280/testReport/thrift_hsha_test/ThriftHSHATest/test_closing_connections/history/

The 3 most recent failures were:

http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/272/testReport/junit/thrift_hsha_test/ThriftHSHATest/test_closing_connections/
http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/272/testReport/junit/thrift_hsha_test/ThriftHSHATest/test_closing_connections/
http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/280/testReport/junit/thrift_hsha_test/ThriftHSHATest/test_closing_connections/

I haven't seen this failure on CassCI on the 2.2 branch, so it may be a 
regression; I'm not sure.

This is related to CASSANDRA-9369; it presents itself the same way. Looks like 
that ticket was closed after a fix was merged to the dtests, but it didn't fix 
this particular error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (CASSANDRA-10570) HSHA test_closing_connections test flapping on 3.0

[Carl Yeksigian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Yeksigian reassigned CASSANDRA-10570:
--

Assignee: Carl Yeksigian

> HSHA test_closing_connections test flapping on 3.0
> --
>
> Key: CASSANDRA-10570
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10570
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: Jim Witschey
>Assignee: Carl Yeksigian
> Fix For: 3.0.0
>
>
> See history here:
> http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/280/testReport/thrift_hsha_test/ThriftHSHATest/test_closing_connections/history/
> The 3 most recent failures were:
> http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/272/testReport/junit/thrift_hsha_test/ThriftHSHATest/test_closing_connections/
> http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/272/testReport/junit/thrift_hsha_test/ThriftHSHATest/test_closing_connections/
> http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/280/testReport/junit/thrift_hsha_test/ThriftHSHATest/test_closing_connections/
> I haven't seen this failure on CassCI on the 2.2 branch, so it may be a 
> regression; I'm not sure.
> This is related to CASSANDRA-9369; it presents itself the same way. Looks 
> like that ticket was closed after a fix was merged to the dtests, but it 
> didn't fix this particular error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10554) Auto Bootstraping a new node fails

[Alan Boudreault (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969752#comment-14969752
 ] 

Alan Boudreault commented on CASSANDRA-10554:
-

Thanks [~yukim]. Let me know if I can help in any way.

> Auto Bootstraping a new node fails
> --
>
> Key: CASSANDRA-10554
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10554
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Alan Boudreault
>Assignee: Yuki Morishita
> Fix For: 3.0.0
>
> Attachments: 0001-Add-debug.txt, debug.log, system.log, test.sh
>
>
> I've been trying to add a new node in my 3.0 cluster and it seems to fail. 
> All my nodes are using apache/cassandra-3.0.0 branch. At the beginning, I can 
> see the following error:
> {code}
> INFO  18:45:55 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a ID#0] Prepare 
> completed. Receiving 42 files(1910066622 bytes), sending 0 files(0 bytes)
> WARN  18:45:55 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Retrying for 
> following error
> java.lang.RuntimeException: Unknown column added_time during deserialization
> at 
> org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.StreamReader.createWriter(StreamReader.java:136)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:77)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
>  [main/:na]
> at 
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
>  [main/:na]
> at 
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
>  [main/:na]
> at 
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
>  [main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> ERROR 18:45:55 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Streaming error 
> occurred
> java.lang.IllegalArgumentException: Unknown type 0
> at 
> org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:97)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
>  ~[main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> INFO  18:45:55 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Session with 
> /54.210.187.114 is complete
> INFO  18:45:56 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a ID#0] Prepare 
> completed. Receiving 38 files(2323537628 bytes), sending 0 files(0 bytes)
> WARN  18:45:56 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Retrying for 
> following error
> java.lang.RuntimeException: Unknown column added_time during deserialization
> at 
> org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.StreamReader.createWriter(StreamReader.java:136)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:77)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
>  [main/:na]
> at 
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
>  [main/:na]
> at 
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
>  [main/:na]
> at 
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
>  [main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> ERROR 18:45:56 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Streaming error 
> occurred
> java.lang.IllegalArgumentException: Unknown type 0
> at 
> org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:97)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
>  ~[main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> INFO  18:45:56 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Session with 
> /54.210.184.198 is complete
> INFO  18:45:56 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a ID#0] Prepare 
> completed. Receiving 35 

[jira] [Commented] (CASSANDRA-9975) Flatten Iterator call hierarchy with a shared Transformer

[Benedict (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969544#comment-14969544
 ] 

Benedict commented on CASSANDRA-9975:
-

bq. Wondering if BasePartitions.hasNext/applyOne shouldn't take responsibility 
of closing next if applyToPartitions returned null since other aspects of the 
transformer do close automatically?

I have been back and forth on this one myself. I just started to do it again, 
and remembered why I chose not to: half of the places we close the iterator in 
a transformation still return a non-null result. So the help offered is 
inconsistent. It also (slightly) makes the main transformation loop uglier. I'm 
happy to do this, as I don't have a strong feeling on it, but both times I've 
looked at it I've ultimately reversed course.

bq. toQuery in checkForShortRead no longer has division-by-zero protection. Is 
it no longer necessary?

Line 345 has added a check for if we've received any live data in the most 
recent request, which captures this AFAICT. i.e., if we get zero rows first 
time, there is no point asking for more, since there aren't any.

I've rebased and split out into a separate package. I've also run 
eclipse-warnings, but all I see is "[java] incorrect classpath: 
/home/benedict/git/cassandra/build/cobertura/classes". [~tjake], is this 
expected / indicates we're otherwise good? I assume I'm meant to see a warning 
or two (else Branimir probably wouldn't have mentioned it)

CI looked goodish last run. AFAICT it was equivalent to mainline. I'll see if I 
can get some certainty before commit.

> Flatten Iterator call hierarchy with a shared Transformer
> -
>
> Key: CASSANDRA-9975
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9975
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Core
>Reporter: Benedict
>Assignee: Benedict
> Fix For: 3.0.0
>
>
> Stepping through a read response is made exceedingly difficult by the sheer 
> depth of the call hierarchy, and how rapidly your context jumps around. This 
> ticket intend to partially address that, by flattening one of the main causes 
> of this: iterator transformations.
> I have a patch that attempts to mitigate (but not entirely eliminate) this, 
> through the introduction of a {{RowTransformer}} class that all 
> transformations are applied through. If a transformation has already been 
> applied, the {{RowTransformer}} class does not wrap a new iterator, but 
> instead returns a new {{RowTransformer}} that wraps the original underlying 
> (untransformed) iterator and both transformations. This can accumulate an 
> arbitrary number of transformations and, quite importantly, can apply the 
> filtration step {{Unfiltered -> Row}}  in the same instance as well. The 
> intention being that a majority of control flow happens inside this 
> {{RowTransformer}}, so there is far less context jumping to cope with.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10554) Auto Bootstraping a new node fails

[Yuki Morishita (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969557#comment-14969557
 ] 

Yuki Morishita commented on CASSANDRA-10554:


So I added some debug and found interesting trace:

{code}
ERROR [SSTableBatchOpen:2] 2015-10-22 12:14:00,827 SSTableReader.java:541 - 
Cannot read sstable 
/home/yuki/.ccm/10544/node2/data/manual/song_to_user-2350231078e011e5877ca9fac1d00bce/ma-2-big=[Summary.db,
 Statistics.db, Filter.db, CompressionInfo.db, Digest.crc32, Index.db, TOC.txt, 
Data.db]; other IO error, skipping table
java.lang.RuntimeException: Unknown column artist_name during deserialization
at 
org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
 ~[main/:na]
at 
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:477)
 ~[main/:na]
at 
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:366)
 ~[main/:na]
at 
org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:525)
 ~[main/:na]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[na:1.8.0_51]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
[na:1.8.0_51]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_51]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_51]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
{code}

Since {{manual.song_to_user}} doesn't have {{artist_name}} in its schema 
(schema bellow), how does SSTable happen to have such column?

{code}
CREATE TABLE manual.song_to_user (
song_id text,
user_name text,
playlist_name text,
added_time bigint,
PRIMARY KEY (song_id, user_name, playlist_name)
)
{code}

Will investigate more.

> Auto Bootstraping a new node fails
> --
>
> Key: CASSANDRA-10554
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10554
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Alan Boudreault
>Assignee: Yuki Morishita
> Fix For: 3.0.0
>
> Attachments: 0001-Add-debug.txt, debug.log, system.log, test.sh
>
>
> I've been trying to add a new node in my 3.0 cluster and it seems to fail. 
> All my nodes are using apache/cassandra-3.0.0 branch. At the beginning, I can 
> see the following error:
> {code}
> INFO  18:45:55 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a ID#0] Prepare 
> completed. Receiving 42 files(1910066622 bytes), sending 0 files(0 bytes)
> WARN  18:45:55 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Retrying for 
> following error
> java.lang.RuntimeException: Unknown column added_time during deserialization
> at 
> org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.StreamReader.createWriter(StreamReader.java:136)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:77)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
>  [main/:na]
> at 
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
>  [main/:na]
> at 
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
>  [main/:na]
> at 
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
>  [main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> ERROR 18:45:55 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Streaming error 
> occurred
> java.lang.IllegalArgumentException: Unknown type 0
> at 
> org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:97)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
>  ~[main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> INFO  18:45:55 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Session with 
> /54.210.187.114 is complete
> INFO  18:45:56 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a ID#0] Prepare 
> completed. Receiving 38 files(2323537628 bytes), sending 0 files(0 bytes)
> WARN  18:45:56 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Retrying for 
> following error
> java.lang.RuntimeException: Unknown column added_time during deserialization
> at 
> 

[jira] [Commented] (CASSANDRA-10140) Enable GC logging by default

[Ariel Weisberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969771#comment-14969771
 ] 

Ariel Weisberg commented on CASSANDRA-10140:


After losing almost an entire day to testing this on Windows I can confirm that 
the GC logs appear in the right spot with the 2.2 and 3.0 patch.

> Enable GC logging by default
> 
>
> Key: CASSANDRA-10140
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10140
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Config
>Reporter: Chris Lohfink
>Assignee: Chris Lohfink
>Priority: Minor
> Fix For: 2.2.x, 3.0.x
>
> Attachments: CASSANDRA-10140-2-2.txt, CASSANDRA-10140-v2.txt, 
> CASSANDRA-10140.txt, cassandra-2.2-10140-v2.txt, cassandra-2.2-10140-v3.txt
>
>
> Overhead for the gc logging is very small (with cycling logs in 7+) and it 
> provides a ton of useful information. This will open up more for C* 
> diagnostic tools to provide feedback as well without requiring restarts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10421) Potential issue with LogTransaction as it only checks in a single directory for files

[Ariel Weisberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969495#comment-14969495
 ] 

Ariel Weisberg commented on CASSANDRA-10421:


Can you update the Windows branch so we can see how it does on Windows before 
committing?

I reviewed the additional commit and it looks good. You are fixing multiples 
failures right? Let's  see if I can list them so I am sure I understand.

The first was the bad assumption in LogRecord.getExistingFiles, which caused it 
to throw, which exposed the missing exception handling in Memtable (leaking the 
transaction) which exposed the fact that leaked transactions when cleaned up by 
the garbage collector fail to cleanup due to an incorrect assertion.

So when you fixed the assertion you call abort. What happens if a transaction 
commits, but isn't closed and that leads to the commit followed by an abort? Or 
is closing the only way to get a commit?

> Potential issue with LogTransaction as it only checks in a single directory 
> for files
> -
>
> Key: CASSANDRA-10421
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10421
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Stefania
>Priority: Blocker
> Fix For: 3.0.0
>
>
> When creating a new LogTransaction we try to create the new logfile in the 
> same directory as the one we are writing to, but as we use 
> {{[directories.getDirectoryForNewSSTables()|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/lifecycle/LogTransaction.java#L125]}}
>  this might end up in "any" of the configured data directories. If it does, 
> we will not be able to clean up leftovers as we check for files in the same 
> directory as the logfile was created: 
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/lifecycle/LogRecord.java#L163
> cc [~Stefania]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9975) Flatten Iterator call hierarchy with a shared Transformer

[T Jake Luciani (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969574#comment-14969574
 ] 

T Jake Luciani commented on CASSANDRA-9975:
---

That's fine/unrelated.

[~benedict] Since I'm on this ticket now some questions. Do you have any 
results showing what the stack used to look like vs what it looks like now? You 
mention it improves performance do we have any results to support that?




> Flatten Iterator call hierarchy with a shared Transformer
> -
>
> Key: CASSANDRA-9975
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9975
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Core
>Reporter: Benedict
>Assignee: Benedict
> Fix For: 3.0.0
>
>
> Stepping through a read response is made exceedingly difficult by the sheer 
> depth of the call hierarchy, and how rapidly your context jumps around. This 
> ticket intend to partially address that, by flattening one of the main causes 
> of this: iterator transformations.
> I have a patch that attempts to mitigate (but not entirely eliminate) this, 
> through the introduction of a {{RowTransformer}} class that all 
> transformations are applied through. If a transformation has already been 
> applied, the {{RowTransformer}} class does not wrap a new iterator, but 
> instead returns a new {{RowTransformer}} that wraps the original underlying 
> (untransformed) iterator and both transformations. This can accumulate an 
> arbitrary number of transformations and, quite importantly, can apply the 
> filtration step {{Unfiltered -> Row}}  in the same instance as well. The 
> intention being that a majority of control flow happens inside this 
> {{RowTransformer}}, so there is far less context jumping to cope with.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-10571) ClusteringIndexNamesFilter::shouldInclude is not implemented, SinglePartitionNamesCommand not discarding the sstables it could

[Aleksey Yeschenko (JIRA)]

Aleksey Yeschenko created CASSANDRA-10571:
-

 Summary: ClusteringIndexNamesFilter::shouldInclude is not 
implemented, SinglePartitionNamesCommand not discarding the sstables it could
 Key: CASSANDRA-10571
 URL: https://issues.apache.org/jira/browse/CASSANDRA-10571
 Project: Cassandra
  Issue Type: Bug
Reporter: Aleksey Yeschenko
Assignee: Sylvain Lebresne
 Fix For: 3.0.0


Now that we use {{SinglePartitionNamesCommand}} in more places - where we'd 
previously use what is now {{SinglePartitionSliceCommand}} - not being able to 
skip sstables with non-overlapping clusterings is actually a performance 
regression.

{{SinglePartitionNamesCommand::queryMemtableAndDiskInternal}} should prune 
sstables based on {{ClusteringIndexNamesFilter::shouldInclude}} output, and the 
latter must be replaced with an actual implementation instead of a {{TODO}}.

Not a regression, but not doing so affects counts write path negatively.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10561) Add a check to cqlsh to require Python-2.7 for version >= 2.2

[Benjamin Lerer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969692#comment-14969692
 ] 

Benjamin Lerer commented on CASSANDRA-10561:


The dtest results for 2.2 are 
[here|http://cassci.datastax.com/view/Dev/view/blerer/job/blerer-10561-2.2-dtest/1/]

> Add a check to cqlsh to require Python-2.7 for version >= 2.2
> -
>
> Key: CASSANDRA-10561
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10561
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Benjamin Lerer
>Assignee: Benjamin Lerer
>Priority: Minor
> Attachments: 10561-2.2.txt
>
>
> Not everybody is reading the {{README}} file. Up to now cqlsh was still 
> working with python 2.6 by consequence user complains when we use some 
> python-2.7 features.
> We should still allow python-2.6 with {{2.1}} but enforce the use of python 
> 2.7 in {{2.2+}}. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10360) UnsupportedOperationException when compacting system.size_estimates after 2.1 -> 2.2 -> 3.0 upgrade

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968696#comment-14968696
 ] 

Sylvain Lebresne commented on CASSANDRA-10360:
--

Ok, I've closed this issue then. Thanks!

> UnsupportedOperationException when compacting system.size_estimates after 2.1 
> -> 2.2 -> 3.0 upgrade
> ---
>
> Key: CASSANDRA-10360
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10360
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Andrew Hust
>Assignee: Sylvain Lebresne
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: size_estimates.2.tar.bz2, size_estimates.tar.bz2, 
> system.log.2.bz2, system.log.bz2
>
>
> When upgrading from 2.1 -> 2.2 -> 3.0 the system.size_estimates table will 
> get stuck in a compaction loop throwing:
> {code}
> java.lang.UnsupportedOperationException
> at 
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.readNext(UnfilteredDeserializer.java:382)
> at 
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:147)
> at 
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.computeNext(SSTableSimpleIterator.java:96)
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at 
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
> at 
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at 
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
> at 
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at 
> org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460)
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:503)
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:363)
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at 
> org.apache.cassandra.db.rows.WrappingUnfilteredRowIterator.hasNext(WrappingUnfilteredRowIterator.java:80)
> at 
> org.apache.cassandra.db.rows.AlteringUnfilteredRowIterator.hasNext(AlteringUnfilteredRowIterator.java:72)
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:100)
> at 
> org.apache.cassandra.db.partitions.PurgingPartitionIterator.hasNext(PurgingPartitionIterator.java:72)
> at 
> org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:223)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
> at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
> at 
> org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:539)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> It will only occur when upgrading thru 2.2.  Going from 2.1 -> 3.0 will not 
> surface the issue.  It can be reproduced with the following (note -- changing 
> jdks will need to be altered if you aren't on OSX)
> {code}
> #!/bin/sh
> echo "using java7 for cassandra-2.1 instance"
> export JAVA_HOME=$(/usr/libexec/java_home -v 1.7)
> ccm stop
> ccm remove upgrade_nodes
> ccm create -n 1 -v git:cassandra-2.1 upgrade_nodes
> ccm start
> ccm node1 stress write n=500K -rate threads=4 -mode native cql3
> sleep 10
> for cver in 3.0
> do
> echo "Draining all nodes"
> ccm node1 nodetool drain
> ccm stop
> echo "switching to java 8"
> 

[jira] [Created] (CASSANDRA-10566) IndexSummaryTest Failed on s390x , testDownsample method caused error in cassandra 2.1.10

[Meerabo Shah (JIRA)]

Meerabo Shah created CASSANDRA-10566:


 Summary: IndexSummaryTest Failed on s390x , testDownsample method 
caused error in cassandra 2.1.10
 Key: CASSANDRA-10566
 URL: https://issues.apache.org/jira/browse/CASSANDRA-10566
 Project: Cassandra
  Issue Type: Bug
  Components: Tests
 Environment: s390x RHEL7 IBM systemz
Reporter: Meerabo Shah
 Fix For: 2.1.10


testlist:
 [echo] running test bucket 0 tests
[junit] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[junit] Testsuite: org.apache.cassandra.io.sstable.IndexSummaryTest
[junit] Testsuite: org.apache.cassandra.io.sstable.IndexSummaryTest
[junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 
sec
[junit]
[junit] Testcase: 
org.apache.cassandra.io.sstable.IndexSummaryTest:testDownsample:  Caused an 
ERROR
[junit] Timeout occurred. Please note the time in the report does not 
reflect the time until the timeout.
[junit] junit.framework.AssertionFailedError: Timeout occurred. Please note 
the time in the report does not reflect the time until the timeout.
[junit] at java.lang.Thread.run(Thread.java:745)
[junit]
[junit]
[junit] Test org.apache.cassandra.io.sstable.IndexSummaryTest FAILED 
(timeout)
[junitreport] Processing 
/apache-cassandra-2.1.10-src/build/test/TESTS-TestSuites.xml to 
/tmp/null1454176845
[junitreport] Loading stylesheet 
jar:file:/apache-ant-1.9.2/lib/ant-junit.jar!/org/apache/tools/ant/taskdefs/optional/junit/xsl/junit-frames.xsl
[junitreport] Transform time: 1684ms
[junitreport] Deleting: /tmp/null1454176845

BUILD FAILED



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] cassandra git commit: Make cqlsh support python-2.6 (FOLLOW-UP CASSANDRA-10415)

[blerer]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.2 12e38f7f8 -> bbaa9630c


Make cqlsh support python-2.6 (FOLLOW-UP CASSANDRA-10415)

patch by Stefania Alborghetti; reviewed by Benjamin Lerer for CASSANDRA-10415


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5a1d3764
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5a1d3764
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5a1d3764

Branch: refs/heads/cassandra-2.2
Commit: 5a1d376489e5884b381e3bee3afd22130945782d
Parents: 29ff1f2
Author: Stefania Alborghetti 
Authored: Thu Oct 22 10:12:04 2015 +0200
Committer: blerer 
Committed: Thu Oct 22 10:17:17 2015 +0200

--
 pylib/cqlshlib/cqlhandling.py| 2 +-
 pylib/cqlshlib/test/test_cqlsh_completion.py | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5a1d3764/pylib/cqlshlib/cqlhandling.py
--
diff --git a/pylib/cqlshlib/cqlhandling.py b/pylib/cqlshlib/cqlhandling.py
index 9ea30cd..6ee3cf5 100644
--- a/pylib/cqlshlib/cqlhandling.py
+++ b/pylib/cqlshlib/cqlhandling.py
@@ -63,7 +63,7 @@ class CqlParsingRuleSet(pylexotron.ParsingRuleSet):
 We cannot let resreved cql keywords be simple 'identifier' since this 
caused
 problems with completion, see CASSANDRA-10415
 """
-syntax = ' ::= /(' + '|'.join(r'\b{}\b'.format(k) 
for k in keywords) + ')/ ;'
+syntax = ' ::= /(' + '|'.join(r'\b%s\b' % (k,) 
for k in keywords) + ')/ ;'
 self.append_rules(syntax)
 
 def completer_for(self, rulename, symname):

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5a1d3764/pylib/cqlshlib/test/test_cqlsh_completion.py
--
diff --git a/pylib/cqlshlib/test/test_cqlsh_completion.py 
b/pylib/cqlshlib/test/test_cqlsh_completion.py
index 5f7b6e4..04cee5d 100644
--- a/pylib/cqlshlib/test/test_cqlsh_completion.py
+++ b/pylib/cqlshlib/test/test_cqlsh_completion.py
@@ -76,7 +76,7 @@ class CqlshCompletionCase(BaseTestCase):
 # ensure the last line of the completion is the prompt
 prompt_regex = self.cqlsh.prompt.lstrip() + re.escape(inputstring)
 msg = ('Double-tab completion '
-   'does not print prompt for input "{}"'.format(inputstring))
+   'does not print prompt for input "%s"' % (inputstring, ))
 self.assertRegexpMatches(choice_lines[-1], prompt_regex, msg=msg)
 
 choice_lines = [line.strip() for line in choice_lines[:-1]]

[2/2] cassandra git commit: Merge branch cassandra-2.1 into cassandra-2.2

[blerer]

Merge branch cassandra-2.1 into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bbaa9630
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bbaa9630
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bbaa9630

Branch: refs/heads/cassandra-2.2
Commit: bbaa9630c4be9a9451f69ddd25e045dc2e92c057
Parents: 12e38f7 5a1d376
Author: blerer 
Authored: Thu Oct 22 10:22:01 2015 +0200
Committer: blerer 
Committed: Thu Oct 22 10:22:12 2015 +0200

--

--

[jira] [Created] (CASSANDRA-10567) FastByteOperationsTest failed on s390x for cassandra 2.1.10, estFastByteComparisons: Caused an ERROR

[Meerabo Shah (JIRA)]

Meerabo Shah created CASSANDRA-10567:


 Summary: FastByteOperationsTest failed on s390x for cassandra 
2.1.10, estFastByteComparisons: Caused an ERROR
 Key: CASSANDRA-10567
 URL: https://issues.apache.org/jira/browse/CASSANDRA-10567
 Project: Cassandra
  Issue Type: Bug
  Components: Tests
 Environment: s390x RHEL7 IBM systemz
Reporter: Meerabo Shah
 Fix For: 2.1.10


testlist:
 [echo] running test bucket 0 tests
[junit] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[junit] Testsuite: org.apache.cassandra.utils.FastByteOperationsTest
[junit] Testsuite: org.apache.cassandra.utils.FastByteOperationsTest
[junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 
sec
[junit]
[junit] Testcase: 
org.apache.cassandra.utils.FastByteOperationsTest:testFastByteComparisons:  
  Caused an ERROR
[junit] Timeout occurred. Please note the time in the report does not 
reflect the time until the timeout.
[junit] junit.framework.AssertionFailedError: Timeout occurred. Please note 
the time in the report does not reflect the time until the timeout.
[junit] at java.lang.Thread.run(Thread.java:745)
[junit]
[junit]
[junit] Test org.apache.cassandra.utils.FastByteOperationsTest FAILED 
(timeout)
[junitreport] Processing 
/apache-cassandra-2.1.10-src/build/test/TESTS-TestSuites.xml to 
/tmp/null1905802535
[junitreport] Loading stylesheet 
jar:file:/apache-ant-1.9.2/lib/ant-junit.jar!/org/apache/tools/ant/taskdefs/optional/junit/xsl/junit-frames.xsl
[junitreport] Transform time: 1906ms
[junitreport] Deleting: /tmp/null1905802535

BUILD FAILED



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10561) Add a check to cqlsh to require Python-2.7 for version >= 2.2

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer updated CASSANDRA-10561:
---
Attachment: 10561-2.2.txt

> Add a check to cqlsh to require Python-2.7 for version >= 2.2
> -
>
> Key: CASSANDRA-10561
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10561
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Benjamin Lerer
>Assignee: Benjamin Lerer
>Priority: Minor
> Attachments: 10561-2.2.txt
>
>
> Not everybody is reading the {{README}} file. Up to now cqlsh was still 
> working with python 2.6 by consequence user complains when we use some 
> python-2.7 features.
> We should still allow python-2.6 with {{2.1}} but enforce the use of python 
> 2.7 in {{2.2+}}. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968973#comment-14968973
 ] 

Marcus Eriksson commented on CASSANDRA-10540:
-

Pushed a new branch 
[here|https://github.com/krummas/cassandra/commits/marcuse/rangeawarecompaction]
 for early feedback, still needs cleanup and tests

Enable like this:
{code}ALTER TABLE x.y WITH compaction={'class':'LeveledCompactionStrategy', 
'range_aware_compaction':'true', 'min_range_sstable_size_in_mb':'15'}{code}

* Run a compaction strategy instance per owned range (with num_tokens=256 and 
rf=3, we will have 768 * 2 instances (repaired/unrepaired data)). 
* To avoid getting very many tiny sstables in the per-range strategies, we keep 
them outside the strategy until the estimated size of a range-sstable is larger 
than {{'min_range_sstable_size_in_mb'}}. 
([estimation|https://github.com/krummas/cassandra/blob/09c58eb4689230d471ef4319733fb0e85399bd3a/src/java/org/apache/cassandra/db/compaction/writers/RangeAwareCompactionWriter.java#L115]
 usually gets within a few % of the actual value).
* We do STCS among the many-range-sstables (called "L0" which might not be 
optimal due to LCS)
* We currently prioritize compaction in L0 to get sstables out of there as 
quickly as possible
* If an sstable fits within a range, it is added to that corresponding 
range-compaction strategy - this should avoid getting a lot of L0 sstables 
after streaming for example
* Adds a {{describecompactionstrategy}} nodetool command which displays 
information about the configured compaction strategy (like sstables per range 
etc). Example with only unrepaired data and 2 data directories - we first split 
the owned ranges over those 2 directories, and then we split on a per range 
basis, so the first RangeAwareCompactionStrategy is responsible for half the 
data and the second one is responsible for the rest: {code} $ bin/nodetool 
describecompactionstrategy keyspace1 standard1

-- keyspace1.standard1 
--
Strategy=class org.apache.cassandra.db.compaction.RangeAwareCompactionStrategy, 
for 167 unrepaired sstables, boundary tokens=min(-9223372036854775808) -> 
max(-4095785201827646), location=/home/marcuse/c/d1
Inner strategy: class 
org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy (257 instances, 
162 total sstables)
  sstable counts: 
0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 
23 24 25 26 27 28 29
  
--
  0.. 29 |  1  3  0  0  2  3  0  3  3  0  3  0  2  1  0  1  0  1  0  3  3  4  1 
 0  3  1  0  0  0  0
 30.. 59 |  0  0  0  3  0  2  2  0  3  0  3  3  0  1  3  3  3  0  2  0  1  2  0 
 0  0  1  0  3  0  0
 60.. 89 |  1  0  0  1  1  1  1  0  1  0  2  3  1  0  3  1  2  3  2  0  0  3  2 
 1  1  0  0  2  3  1
 90..119 |  0  1  2  0  0  3  0  3  3  1  0  0  3  0  2  0  2  0  2  1  3  0  2 
 1  1  3  1  0  3  0
120..149 |  2  0  3  1  3  0  0  3  3  1  0  0  0  0  0  0  0  0  0  0  0  0  0 
 0  0  0  0  0  0  0
150..179 |  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 
 0  0  0  0  0  0  0
180..209 |  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 
 0  0  0  0  0  0  0
210..239 |  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 
 0  0  0  0  0  0  0
240..257 |  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
Strategy=class org.apache.cassandra.db.compaction.RangeAwareCompactionStrategy, 
for 221 unrepaired sstables, boundary tokens=max(-4095785201827646) -> 
max(9223372036854775807), location=/var/lib/c1
Inner strategy: class 
org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy (257 instances, 
215 total sstables)
  sstable counts: 
0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 
23 24 25 26 27 28 29
  
--
  0.. 29 |  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 
 0  0  0  0  0  0  0
 30.. 59 |  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 
 0  0  0  0  0  0  0
 60.. 89 |  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 
 0  0  0  0  0  0  0
 90..119 |  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 
 0  0  0  0  0  0  0
120..149 |  0  0  0  0  0  0  0  0  0  0  1  6  0  0  3  0  3  0  3  3  3  3  1 
 0  1  0  2  0  3  2
150..179 |  3  3  3  0  0  3  3  0  3  2  3  1  3  3  3  3  0  0  0  3  0  1  1 
 0  6  3  3  0  3  3
180..209 |  0  1  1  3  1  3  1  3  3  2  3  3  0  3  0  3  1  0  0  1  2  3  0 
 0  1  1  0  0  3  3
210..239 |  3  3  3  2  0  6  1  3  0  0  3  3  3  1  3  4  3  3  3  0  3  0  3 
 1  2  2  0  2  0  0
240..257 |  1  0  3  1  0  3  

[jira] [Commented] (CASSANDRA-10476) Fix upgrade paging dtest failures on 2.2->3.0 path

[Jim Witschey (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969877#comment-14969877
 ] 

Jim Witschey commented on CASSANDRA-10476:
--

Looks like these 2 tests flap with timeouts as well:

http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/280/testReport/upgrade_tests.paging_test/TestPagingWithDeletions/test_single_partition_deletions/history/
http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/280/testReport/upgrade_tests.paging_test/TestPagingWithDeletions/test_multiple_partition_deletions/history/

[~thobbs] Who do you recommend look at these tests?

> Fix upgrade paging dtest failures on 2.2->3.0 path
> --
>
> Key: CASSANDRA-10476
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10476
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: Jim Witschey
> Fix For: 3.0.0
>
>
> The following upgrade tests for paging features fail or flap on the upgrade 
> path from 2.2 to 3.0:
> - {{upgrade_tests/paging_test.py:TestPagingData.static_columns_paging_test}}
> - 
> {{upgrade_tests/paging_test.py:TestPagingSize.test_undefined_page_size_default}}
> - 
> {{upgrade_tests/paging_test.py:TestPagingSize.test_with_more_results_than_page_size}}
> - 
> {{upgrade_tests/paging_test.py:TestPagingWithDeletions.test_failure_threshold_deletions}}
> - 
> {{upgrade_tests/paging_test.py:TestPagingWithDeletions.test_multiple_cell_deletions}}
> - 
> {{upgrade_tests/paging_test.py:TestPagingWithDeletions.test_single_cell_deletions}}
> - 
> {{upgrade_tests/paging_test.py:TestPagingWithDeletions.test_single_row_deletions}}
> - 
> {{upgrade_tests/paging_test.py:TestPagingDatasetChanges.test_cell_TTL_expiry_during_paging/}}
> I've grouped them all together because I don't know how to tell if they're 
> related; once someone triages them, it may be appropriate to break this out 
> into multiple tickets.
> The failures can be found here:
> http://cassci.datastax.com/view/Upgrades/job/storage_engine_upgrade_dtest-22_tarball-30_HEAD/44/testReport/upgrade_tests.paging_test/TestPagingData/static_columns_paging_test/history/
> http://cassci.datastax.com/view/Upgrades/job/storage_engine_upgrade_dtest-22_tarball-30_HEAD/44/testReport/upgrade_tests.paging_test/TestPagingSize/test_undefined_page_size_default/history/
> http://cassci.datastax.com/view/Upgrades/job/storage_engine_upgrade_dtest-22_tarball-30_HEAD/42/testReport/upgrade_tests.paging_test/TestPagingSize/test_with_more_results_than_page_size/history/
> http://cassci.datastax.com/view/Upgrades/job/storage_engine_upgrade_dtest-22_tarball-30_HEAD/44/testReport/upgrade_tests.paging_test/TestPagingWithDeletions/test_failure_threshold_deletions/history/
> http://cassci.datastax.com/view/Upgrades/job/storage_engine_upgrade_dtest-22_tarball-30_HEAD/44/testReport/upgrade_tests.paging_test/TestPagingWithDeletions/test_multiple_cell_deletions/history/
> http://cassci.datastax.com/view/Upgrades/job/storage_engine_upgrade_dtest-22_tarball-30_HEAD/44/testReport/upgrade_tests.paging_test/TestPagingWithDeletions/test_single_cell_deletions/history/
> http://cassci.datastax.com/view/Upgrades/job/storage_engine_upgrade_dtest-22_tarball-30_HEAD/44/testReport/upgrade_tests.paging_test/TestPagingWithDeletions/test_single_row_deletions/history/
> http://cassci.datastax.com/view/Upgrades/job/storage_engine_upgrade_dtest-22_tarball-30_HEAD/44/testReport/upgrade_tests.paging_test/TestPagingDatasetChanges/test_cell_TTL_expiry_during_paging/
> Once [this dtest PR|https://github.com/riptano/cassandra-dtest/pull/586] is 
> merged, these tests should also run with this upgrade path on normal 3.0 
> jobs. Until then, you can run them with the following command:
> {code}
> SKIP=false CASSANDRA_VERSION=binary:2.2.0 UPGRADE_TO=git:cassandra-3.0 
> nosetests 
> upgrade_tests/paging_test.py:TestPagingData.static_columns_paging_test 
> upgrade_tests/paging_test.py:TestPagingSize.test_undefined_page_size_default 
> upgrade_tests/paging_test.py:TestPagingSize.test_with_more_results_than_page_size
>  
> upgrade_tests/paging_test.py:TestPagingWithDeletions.test_failure_threshold_deletions
>  
> upgrade_tests/paging_test.py:TestPagingWithDeletions.test_multiple_cell_deletions
>  
> upgrade_tests/paging_test.py:TestPagingWithDeletions.test_single_cell_deletions
>  
> upgrade_tests/paging_test.py:TestPagingWithDeletions.test_single_row_deletions
> upgrade_tests/paging_test.py:TestPagingDatasetChanges.test_cell_TTL_expiry_during_paging
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10554) Auto Bootstraping a new node fails

[Yuki Morishita (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuki Morishita updated CASSANDRA-10554:
---
Attachment: 10554.cql

I think I found the problem.
Batch statement seems to mix partitioned columns from those in its statement.
Memtable serializes those mixed {{PartitionColumns}} to Stats.db

CQL file to reproduce is attached.

To reproduce:

{code}
$ ccm node1 start # start node
$ cqlsh -f 10554.cql
$ nodetool flush manual # flush manual keyspace
// then restart or anything that reads SSTable offline will throw error
{code}



> Auto Bootstraping a new node fails
> --
>
> Key: CASSANDRA-10554
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10554
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Alan Boudreault
>Assignee: Yuki Morishita
> Fix For: 3.0.0
>
> Attachments: 0001-Add-debug.txt, 10554.cql, debug.log, system.log, 
> test.sh
>
>
> I've been trying to add a new node in my 3.0 cluster and it seems to fail. 
> All my nodes are using apache/cassandra-3.0.0 branch. At the beginning, I can 
> see the following error:
> {code}
> INFO  18:45:55 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a ID#0] Prepare 
> completed. Receiving 42 files(1910066622 bytes), sending 0 files(0 bytes)
> WARN  18:45:55 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Retrying for 
> following error
> java.lang.RuntimeException: Unknown column added_time during deserialization
> at 
> org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.StreamReader.createWriter(StreamReader.java:136)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:77)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
>  [main/:na]
> at 
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
>  [main/:na]
> at 
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
>  [main/:na]
> at 
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
>  [main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> ERROR 18:45:55 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Streaming error 
> occurred
> java.lang.IllegalArgumentException: Unknown type 0
> at 
> org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:97)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
>  ~[main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> INFO  18:45:55 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Session with 
> /54.210.187.114 is complete
> INFO  18:45:56 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a ID#0] Prepare 
> completed. Receiving 38 files(2323537628 bytes), sending 0 files(0 bytes)
> WARN  18:45:56 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Retrying for 
> following error
> java.lang.RuntimeException: Unknown column added_time during deserialization
> at 
> org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.StreamReader.createWriter(StreamReader.java:136)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:77)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
>  [main/:na]
> at 
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
>  [main/:na]
> at 
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
>  [main/:na]
> at 
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
>  [main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> ERROR 18:45:56 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Streaming error 
> occurred
> java.lang.IllegalArgumentException: Unknown type 0
> at 
> org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:97)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58)
>  ~[main/:na]
> at 
> 

[jira] [Commented] (CASSANDRA-9091) Cassandra-stress should support map column type

[Sebastian Estevez (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969829#comment-14969829
 ] 

Sebastian Estevez commented on CASSANDRA-9091:
--

Do we need to wait for a major overhaul to add map type support? It doesn't 
seem like it would be that challenging to add it give our current structure.

> Cassandra-stress should support map column type
> ---
>
> Key: CASSANDRA-9091
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9091
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Asya Lisak
>Priority: Minor
> Fix For: 2.1.5
>
>
> Currently cassandra-stress does not support map data type, even though 
> cassandra itself supports it.  UnsupportedOperationException is thrown if 
> table under stress test has column with map type.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-9091) Cassandra-stress should support map column type

[Sebastian Estevez (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969829#comment-14969829
 ] 

Sebastian Estevez edited comment on CASSANDRA-9091 at 10/22/15 8:26 PM:


Do we need to wait for a major overhaul to add map type support? It doesn't 
seem like it would be that challenging to add it given the state of the current 
stress tool and the fact that lists and sets are already supported.


was (Author: sebastian.este...@datastax.com):
Do we need to wait for a major overhaul to add map type support? It doesn't 
seem like it would be that challenging to add it give our current structure.

> Cassandra-stress should support map column type
> ---
>
> Key: CASSANDRA-9091
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9091
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Asya Lisak
>Priority: Minor
> Fix For: 2.1.5
>
>
> Currently cassandra-stress does not support map data type, even though 
> cassandra itself supports it.  UnsupportedOperationException is thrown if 
> table under stress test has column with map type.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10513) Update cqlsh for new driver execution API

[Adam Holmberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Holmberg updated CASSANDRA-10513:
--
Attachment: 10513.txt

> Update cqlsh for new driver execution API
> -
>
> Key: CASSANDRA-10513
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10513
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Tools
>Reporter: Adam Holmberg
>Assignee: Paulo Motta
>Priority: Minor
>  Labels: cqlsh
> Fix For: 2.2.x, 3.0.x
>
> Attachments: 10513.txt
>
>
> The 3.0 Python driver will have a few tweaks to the execution API. We'll need 
> to update cqlsh in a couple of minor ways.
> [Results are always returned as an iterable 
> ResultSet|https://datastax-oss.atlassian.net/browse/PYTHON-368]
> [Trace data is always attached to the 
> ResultSet|https://datastax-oss.atlassian.net/browse/PYTHON-318] (instead of 
> being attached to a statement)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10513) Update cqlsh for new driver execution API

[Adam Holmberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Holmberg updated CASSANDRA-10513:
--
Attachment: (was: 10513.txt)

> Update cqlsh for new driver execution API
> -
>
> Key: CASSANDRA-10513
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10513
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Tools
>Reporter: Adam Holmberg
>Assignee: Paulo Motta
>Priority: Minor
>  Labels: cqlsh
> Fix For: 2.2.x, 3.0.x
>
> Attachments: 10513.txt
>
>
> The 3.0 Python driver will have a few tweaks to the execution API. We'll need 
> to update cqlsh in a couple of minor ways.
> [Results are always returned as an iterable 
> ResultSet|https://datastax-oss.atlassian.net/browse/PYTHON-368]
> [Trace data is always attached to the 
> ResultSet|https://datastax-oss.atlassian.net/browse/PYTHON-318] (instead of 
> being attached to a statement)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-10573) select_distinct_test flapping on 2.2 -> 3.0 upgrade path

[Jim Witschey (JIRA)]

Jim Witschey created CASSANDRA-10573:


 Summary: select_distinct_test flapping on 2.2 -> 3.0 upgrade path
 Key: CASSANDRA-10573
 URL: https://issues.apache.org/jira/browse/CASSANDRA-10573
 Project: Cassandra
  Issue Type: Sub-task
Reporter: Jim Witschey
 Fix For: 3.0.0


The {{upgrade_tests/cql_tests.py:TestCQLNodes2RF1.select_distinct_test flaps}} 
dtest flaps on CassCI. It flaps locally for me as well:

{code}
CASSANDRA_VERSION=git:cassandra-3.0 nosetests 
upgrade_tests/cql_tests.py:TestCQLNodes2RF1.select_distinct_test
{code}

Once [this dtest pr|https://github.com/riptano/cassandra-dtest/pull/624] is 
merged, you can run against a local directory with 

{code}
CASSANDRA_VERSION=git:cassandra-3.0 OLD_CASSANDRA_VERSION=binary:2.2.3 
UPGRADE_TO_DIR=~/path/to/cassandra nosetests 
upgrade_tests/cql_tests.py:TestCQLNodes2RF1.select_distinct_test --no-skip
{code}

Apologies for the ugly interface for that. To run on Windows, use {{set}} to 
set environment variables like {{CASSANDRA_VERSION}} instead of including them 
in the command.

The history for the test is here:

http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/280/testReport/junit/upgrade_tests.cql_tests/TestCQL/select_distinct_test/

Note that the name of the class containing the test has changed from 
{{TestCQL}} to ; its behavior hasn't.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10554) Batch that updates two or more table can produce unreadable SSTable (was: Auto Bootstraping a new node fails)

[Yuki Morishita (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuki Morishita updated CASSANDRA-10554:
---
Assignee: Sylvain Lebresne  (was: Yuki Morishita)
Priority: Critical  (was: Major)
 Summary: Batch that updates two or more table can produce unreadable 
SSTable (was: Auto Bootstraping a new node fails)  (was: Auto Bootstraping a 
new node fails)

> Batch that updates two or more table can produce unreadable SSTable (was: 
> Auto Bootstraping a new node fails)
> -
>
> Key: CASSANDRA-10554
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10554
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Alan Boudreault
>Assignee: Sylvain Lebresne
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: 0001-Add-debug.txt, 10554.cql, debug.log, system.log, 
> test.sh
>
>
> I've been trying to add a new node in my 3.0 cluster and it seems to fail. 
> All my nodes are using apache/cassandra-3.0.0 branch. At the beginning, I can 
> see the following error:
> {code}
> INFO  18:45:55 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a ID#0] Prepare 
> completed. Receiving 42 files(1910066622 bytes), sending 0 files(0 bytes)
> WARN  18:45:55 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Retrying for 
> following error
> java.lang.RuntimeException: Unknown column added_time during deserialization
> at 
> org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.StreamReader.createWriter(StreamReader.java:136)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:77)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
>  [main/:na]
> at 
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
>  [main/:na]
> at 
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
>  [main/:na]
> at 
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
>  [main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> ERROR 18:45:55 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Streaming error 
> occurred
> java.lang.IllegalArgumentException: Unknown type 0
> at 
> org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:97)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
>  ~[main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> INFO  18:45:55 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Session with 
> /54.210.187.114 is complete
> INFO  18:45:56 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a ID#0] Prepare 
> completed. Receiving 38 files(2323537628 bytes), sending 0 files(0 bytes)
> WARN  18:45:56 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Retrying for 
> following error
> java.lang.RuntimeException: Unknown column added_time during deserialization
> at 
> org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.StreamReader.createWriter(StreamReader.java:136)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:77)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
>  [main/:na]
> at 
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
>  [main/:na]
> at 
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
>  [main/:na]
> at 
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
>  [main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> ERROR 18:45:56 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Streaming error 
> occurred
> java.lang.IllegalArgumentException: Unknown type 0
> at 
> org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:97)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58)
>  ~[main/:na]
> at 
> 

[jira] [Commented] (CASSANDRA-10544) Make cqlsh tests work when authentication is configured

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970209#comment-14970209
 ] 

Stefania commented on CASSANDRA-10544:
--

[~aholmber] would you mind cross testing this patch since [~blerer] cannot run 
these tests on Windows yet?

> Make cqlsh tests work when authentication is configured
> ---
>
> Key: CASSANDRA-10544
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10544
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tests, Tools
>Reporter: Adam Holmberg
>Assignee: Stefania
>Priority: Trivial
>  Labels: cqlsh, test
> Fix For: 2.1.x, 2.2.x, 3.0.x
>
>
> cqlsh tests break if the runner has an authentication section in their 
> ~/.cassandra/cqlshrc, because cqlsh changes the prompt and the tests scan 
> output for a prompt. It manifests as read timeouts while waiting for a prompt 
> in test/run_cqlsh.py.
> [This 
> pattern|https://github.com/mambocab/cassandra/blob/1c27f9be1ba8ea10dbe843d513e23de6238dede8/pylib/cqlshlib/test/run_cqlsh.py#L30]
>  could be generalized to match the "@cqlsh..." prompt that arises 
> with this config.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10513) Update cqlsh for new driver execution API

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970103#comment-14970103
 ] 

Paulo Motta commented on CASSANDRA-10513:
-

Thanks for the patch. Code looks good. Applied patch locally and tested a few 
commands (use, select, paging, tracing, cas insert), and everything seems to 
work fine (apart from some minor nit already addressed offline by [~aholmber]).

Submitted a [3.0 
branch|https://github.com/pauloricardomg/cassandra/tree/3.0-10513] to cassci: 
[dtest 
link|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-3.0-10513-dtest/]

Waiting for submission of 2.1 and 2.2 patches ([~aholmber] is merging it, 
thanks!).

> Update cqlsh for new driver execution API
> -
>
> Key: CASSANDRA-10513
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10513
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Tools
>Reporter: Adam Holmberg
>Assignee: Paulo Motta
>Priority: Minor
>  Labels: cqlsh
> Fix For: 2.2.x, 3.0.x
>
> Attachments: 10513.txt
>
>
> The 3.0 Python driver will have a few tweaks to the execution API. We'll need 
> to update cqlsh in a couple of minor ways.
> [Results are always returned as an iterable 
> ResultSet|https://datastax-oss.atlassian.net/browse/PYTHON-368]
> [Trace data is always attached to the 
> ResultSet|https://datastax-oss.atlassian.net/browse/PYTHON-318] (instead of 
> being attached to a statement)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8844) Change Data Capture (CDC)

[Yong Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970135#comment-14970135
 ] 

Yong Zhang commented on CASSANDRA-8844:
---

Currently we parse the backup SSTable files to integration the data from C* to 
other storage system. And it is hard to capture change data this way. So I will 
definitely vote for this feature.

I always wonder what is it in the commit log of Cassandra? Won't it be exactly 
the change data? It is a black box right now for us, like bin log of MySQL 
database?

> Change Data Capture (CDC)
> -
>
> Key: CASSANDRA-8844
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8844
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Core
>Reporter: Tupshin Harper
>Assignee: Joshua McKenzie
>Priority: Critical
> Fix For: 3.x
>
>
> "In databases, change data capture (CDC) is a set of software design patterns 
> used to determine (and track) the data that has changed so that action can be 
> taken using the changed data. Also, Change data capture (CDC) is an approach 
> to data integration that is based on the identification, capture and delivery 
> of the changes made to enterprise data sources."
> -Wikipedia
> As Cassandra is increasingly being used as the Source of Record (SoR) for 
> mission critical data in large enterprises, it is increasingly being called 
> upon to act as the central hub of traffic and data flow to other systems. In 
> order to try to address the general need, we (cc [~brianmhess]), propose 
> implementing a simple data logging mechanism to enable per-table CDC patterns.
> h2. The goals:
> # Use CQL as the primary ingestion mechanism, in order to leverage its 
> Consistency Level semantics, and in order to treat it as the single 
> reliable/durable SoR for the data.
> # To provide a mechanism for implementing good and reliable 
> (deliver-at-least-once with possible mechanisms for deliver-exactly-once ) 
> continuous semi-realtime feeds of mutations going into a Cassandra cluster.
> # To eliminate the developmental and operational burden of users so that they 
> don't have to do dual writes to other systems.
> # For users that are currently doing batch export from a Cassandra system, 
> give them the opportunity to make that realtime with a minimum of coding.
> h2. The mechanism:
> We propose a durable logging mechanism that functions similar to a commitlog, 
> with the following nuances:
> - Takes place on every node, not just the coordinator, so RF number of copies 
> are logged.
> - Separate log per table.
> - Per-table configuration. Only tables that are specified as CDC_LOG would do 
> any logging.
> - Per DC. We are trying to keep the complexity to a minimum to make this an 
> easy enhancement, but most likely use cases would prefer to only implement 
> CDC logging in one (or a subset) of the DCs that are being replicated to
> - In the critical path of ConsistencyLevel acknowledgment. Just as with the 
> commitlog, failure to write to the CDC log should fail that node's write. If 
> that means the requested consistency level was not met, then clients *should* 
> experience UnavailableExceptions.
> - Be written in a Row-centric manner such that it is easy for consumers to 
> reconstitute rows atomically.
> - Written in a simple format designed to be consumed *directly* by daemons 
> written in non JVM languages
> h2. Nice-to-haves
> I strongly suspect that the following features will be asked for, but I also 
> believe that they can be deferred for a subsequent release, and to guage 
> actual interest.
> - Multiple logs per table. This would make it easy to have multiple 
> "subscribers" to a single table's changes. A workaround would be to create a 
> forking daemon listener, but that's not a great answer.
> - Log filtering. Being able to apply filters, including UDF-based filters 
> would make Casandra a much more versatile feeder into other systems, and 
> again, reduce complexity that would otherwise need to be built into the 
> daemons.
> h2. Format and Consumption
> - Cassandra would only write to the CDC log, and never delete from it. 
> - Cleaning up consumed logfiles would be the client daemon's responibility
> - Logfile size should probably be configurable.
> - Logfiles should be named with a predictable naming schema, making it 
> triivial to process them in order.
> - Daemons should be able to checkpoint their work, and resume from where they 
> left off. This means they would have to leave some file artifact in the CDC 
> log's directory.
> - A sophisticated daemon should be able to be written that could 
> -- Catch up, in written-order, even when it is multiple logfiles behind in 
> processing
> -- Be able to continuously "tail" the most recent logfile and get 
> low-latency(ms?) access to the 

[jira] [Commented] (CASSANDRA-10421) Potential issue with LogTransaction as it only checks in a single directory for files

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970203#comment-14970203
 ] 

Stefania commented on CASSANDRA-10421:
--

bq. Can you update the Windows branch so we can see how it does on Windows 
before committing?

Windows branch updated and jobs are scheduled:

http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-10421-windows-3.0-utest_win32/
http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-10421-windows-3.0-dtest_win32/

bq. The first was the bad assumption in LogRecord.getExistingFiles, which 
caused it to throw, which exposed the missing exception handling in Memtable 
(leaking the transaction) which exposed the fact that leaked transactions when 
cleaned up by the garbage collector fail to cleanup due to an incorrect 
assertion.

Your 100% correct. This once again highlights how beneficial testing this 
component with failure injection would be. Another example is CASSANDRA-10538.

bq. So when you fixed the assertion you call abort. What happens if a 
transaction commits, but isn't closed and that leads to the commit followed by 
an abort? Or is closing the only way to get a commit?

If a transaction commits then it is completed and we will not call abort. I 
think the only risk is if we throw an exception in the commit operation _after 
writing to file and before updating the records in memory_, in which case we 
would end up with a corrupt file as you said. But after CASSANDRA-10538 is 
applied, we will update the records just after writing to file, so this should 
be safe.

> Potential issue with LogTransaction as it only checks in a single directory 
> for files
> -
>
> Key: CASSANDRA-10421
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10421
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Stefania
>Priority: Blocker
> Fix For: 3.0.0
>
>
> When creating a new LogTransaction we try to create the new logfile in the 
> same directory as the one we are writing to, but as we use 
> {{[directories.getDirectoryForNewSSTables()|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/lifecycle/LogTransaction.java#L125]}}
>  this might end up in "any" of the configured data directories. If it does, 
> we will not be able to clean up leftovers as we check for files in the same 
> directory as the logfile was created: 
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/lifecycle/LogRecord.java#L163
> cc [~Stefania]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10561) Add a check to cqlsh to require Python-2.7 for version >= 2.2

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970227#comment-14970227
 ] 

Stefania commented on CASSANDRA-10561:
--

I would add an s to {{support}}, from {{CQL Shell support only Python 2.7}} to 
{{CQL Shell supports only Python 2.7}}.

Otherwise +1, verified with python 2.6 and 2.7.

> Add a check to cqlsh to require Python-2.7 for version >= 2.2
> -
>
> Key: CASSANDRA-10561
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10561
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Benjamin Lerer
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 2.2.4, 3.0.0
>
> Attachments: 10561-2.2.txt
>
>
> Not everybody is reading the {{README}} file. Up to now cqlsh was still 
> working with python 2.6 by consequence user complains when we use some 
> python-2.7 features.
> We should still allow python-2.6 with {{2.1}} but enforce the use of python 
> 2.7 in {{2.2+}}. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9387) Add snitch supporting Windows Azure

[Matt Kennedy (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970263#comment-14970263
 ] 

Matt Kennedy commented on CASSANDRA-9387:
-

This item is waiting on the fault-domain and update-domain data to be exposed 
via the Instance Metadata Service similar to how the instance event metadata is 
exposed in this article: 
https://azure.microsoft.com/en-us/blog/what-just-happened-to-my-vm-in-vm-metadata-service/

> Add snitch supporting Windows Azure
> ---
>
> Key: CASSANDRA-9387
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9387
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Config
>Reporter: Jonathan Ellis
>Assignee: Matt Kennedy
> Fix For: 2.1.x
>
>
> Looks like regions / fault domains are a pretty close analogue to C* 
> DCs/racks.
> http://blogs.technet.com/b/yungchou/archive/2011/05/16/window-azure-fault-domain-and-update-domain-explained-for-it-pros.aspx



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10540) RangeAwareCompaction

[Marcus Eriksson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-10540:

Description: 
Broken out from CASSANDRA-6696, we should split sstables based on ranges during 
compaction.

Requirements;
* dont create tiny sstables - keep them bunched together until a single vnode 
is big enough (configurable how big that is)
* make it possible to run existing compaction strategies on the per-range 
sstables

We should probably add a global compaction strategy parameter that states 
whether this should be enabled or not.

  was:
Broken out from CASSANDRA-6696, we should split sstables based on ranges during 
compaction.

Requirements;
* dont create tiny sstables - keep them bunched together until a single vnode 
is big enough (configurable how big that is)
* make it possible to run existing compaction strategies on the per-range 
sstables

We should probably add a global compaction strategy parameter that states 
whether this should be enabled or not.

My wip-branch is here (broken out from 6696, probably does not build); 
https://github.com/krummas/cassandra/commits/marcuse/vnodeawarecompaction - 
naming is wrong - we should split based on local ranges even without vnodes


> RangeAwareCompaction
> 
>
> Key: CASSANDRA-10540
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10540
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
> Fix For: 3.2
>
>
> Broken out from CASSANDRA-6696, we should split sstables based on ranges 
> during compaction.
> Requirements;
> * dont create tiny sstables - keep them bunched together until a single vnode 
> is big enough (configurable how big that is)
> * make it possible to run existing compaction strategies on the per-range 
> sstables
> We should probably add a global compaction strategy parameter that states 
> whether this should be enabled or not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9549) Memory leak in Ref.GlobalState due to pathological ConcurrentLinkedQueue.remove behaviour

[Maxim Podkolzine (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968882#comment-14968882
 ] 

Maxim Podkolzine commented on CASSANDRA-9549:
-

Is this bug fixed in Cassandra 2.2.0?

> Memory leak in Ref.GlobalState due to pathological 
> ConcurrentLinkedQueue.remove behaviour
> -
>
> Key: CASSANDRA-9549
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9549
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Cassandra 2.1.5. 9 node cluster in EC2 (m1.large nodes, 
> 2 cores 7.5G memory, 800G platter for cassandra data, root partition and 
> commit log are on SSD EBS with sufficient IOPS), 3 nodes/availablity zone, 1 
> replica/zone
> JVM: /usr/java/jdk1.8.0_40/jre/bin/java 
> JVM Flags besides CP: -ea -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar 
> -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities 
> -XX:ThreadPriorityPolicy=42 -Xms2G -Xmx2G -Xmn200M 
> -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103 
> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled 
> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 
> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly 
> -XX:+UseTLAB -XX:CompileCommandFile=/etc/cassandra/conf/hotspot_compiler 
> -XX:CMSWaitDuration=1 -XX:+CMSParallelInitialMarkEnabled 
> -XX:+CMSEdenChunksRecordAlways -XX:CMSWaitDuration=1 -XX:+UseCondCardMark 
> -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 
> -Dcom.sun.management.jmxremote.rmi.port=7199 
> -Dcom.sun.management.jmxremote.ssl=false 
> -Dcom.sun.management.jmxremote.authenticate=false 
> -Dlogback.configurationFile=logback.xml -Dcassandra.logdir=/var/log/cassandra 
> -Dcassandra.storagedir= -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid 
> Kernel: Linux 2.6.32-504.16.2.el6.x86_64 #1 SMP x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Ivar Thorson
>Assignee: Benedict
>Priority: Critical
> Fix For: 2.1.7
>
> Attachments: c4_system.log, c7fromboot.zip, cassandra.yaml, 
> cpu-load.png, memoryuse.png, ref-java-errors.jpeg, suspect.png, two-loads.png
>
>
> We have been experiencing a severe memory leak with Cassandra 2.1.5 that, 
> over the period of a couple of days, eventually consumes all of the available 
> JVM heap space, putting the JVM into GC hell where it keeps trying CMS 
> collection but can't free up any heap space. This pattern happens for every 
> node in our cluster and is requiring rolling cassandra restarts just to keep 
> the cluster running. We have upgraded the cluster per Datastax docs from the 
> 2.0 branch a couple of months ago and have been using the data from this 
> cluster for more than a year without problem.
> As the heap fills up with non-GC-able objects, the CPU/OS load average grows 
> along with it. Heap dumps reveal an increasing number of 
> java.util.concurrent.ConcurrentLinkedQueue$Node objects. We took heap dumps 
> over a 2 day period, and watched the number of Node objects go from 4M, to 
> 19M, to 36M, and eventually about 65M objects before the node stops 
> responding. The screen capture of our heap dump is from the 19M measurement.
> Load on the cluster is minimal. We can see this effect even with only a 
> handful of writes per second. (See attachments for Opscenter snapshots during 
> very light loads and heavier loads). Even with only 5 reads a sec we see this 
> behavior.
> Log files show repeated errors in Ref.java:181 and Ref.java:279 and "LEAK 
> detected" messages:
> {code}
> ERROR [CompactionExecutor:557] 2015-06-01 18:27:36,978 Ref.java:279 - Error 
> when closing class 
> org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier@1302301946:/data1/data/ourtablegoeshere-ka-1150
> java.util.concurrent.RejectedExecutionException: Task 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@32680b31 
> rejected from 
> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@573464d6[Terminated,
>  pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1644]
> {code}
> {code}
> ERROR [Reference-Reaper:1] 2015-06-01 18:27:37,083 Ref.java:181 - LEAK 
> DETECTED: a reference 
> (org.apache.cassandra.utils.concurrent.Ref$State@74b5df92) to class 
> org.apache.cassandra.io.sstable.SSTableReader$DescriptorTypeTidy@2054303604:/data2/data/ourtablegoeshere-ka-1151
>  was not released before the reference was garbage collected
> {code}
> This might be related to [CASSANDRA-8723]?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-7385) sstableloader OutOfMemoryError: Java heap space

[Stuart Bishop (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968863#comment-14968863
 ] 

Stuart Bishop commented on CASSANDRA-7385:
--

This did not get resolved by CASSANDRA-5969. /usr/bin/sstablescrub was fixed, 
but /usr/bin/sstableloader still has a hard coded -Mx256M that can't be easily 
overridden.

> sstableloader OutOfMemoryError: Java heap space
> ---
>
> Key: CASSANDRA-7385
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7385
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Mike Heffner
>
> We hit the following exception with sstableloader while attempting to bulk 
> load about 100GB of SSTs. We are now employing this workaround before 
> starting an sstableloader run:
> sed -i -e 's/-Xmx256M/-Xmx8G/g' /usr/bin/sstableloader
> {code}
> ERROR 19:25:45,060 Error in ThreadPoolExecutor
> java.lang.OutOfMemoryError: Java heap space
>   at 
> org.apache.cassandra.io.util.FastByteArrayOutputStream.expand(FastByteArrayOutputStream.java:104)
>   at 
> org.apache.cassandra.io.util.FastByteArrayOutputStream.write(FastByteArrayOutputStream.java:235)
>   at java.io.DataOutputStream.writeInt(DataOutputStream.java:199)
>   at 
> org.apache.cassandra.io.compress.CompressionMetadata$ChunkSerializer.serialize(CompressionMetadata.java:412)
>   at 
> org.apache.cassandra.io.compress.CompressionMetadata$ChunkSerializer.serialize(CompressionMetadata.java:407)
>   at 
> org.apache.cassandra.streaming.compress.CompressionInfo$CompressionInfoSerializer.serialize(CompressionInfo.java:59)
>   at 
> org.apache.cassandra.streaming.compress.CompressionInfo$CompressionInfoSerializer.serialize(CompressionInfo.java:46)
>   at 
> org.apache.cassandra.streaming.PendingFile$PendingFileSerializer.serialize(PendingFile.java:142)
>   at 
> org.apache.cassandra.streaming.StreamHeader$StreamHeaderSerializer.serialize(StreamHeader.java:67)
>   at 
> org.apache.cassandra.streaming.StreamHeader$StreamHeaderSerializer.serialize(StreamHeader.java:58)
>   at 
> org.apache.cassandra.net.MessagingService.constructStreamHeader(MessagingService.java:782)
>   at 
> org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:65)
>   at 
> org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Exception in thread "Streaming to /10.167.a.b:1" java.lang.OutOfMemoryError: 
> Java heap space
>   at 
> org.apache.cassandra.io.util.FastByteArrayOutputStream.expand(FastByteArrayOutputStream.java:104)
>   at 
> org.apache.cassandra.io.util.FastByteArrayOutputStream.write(FastByteArrayOutputStream.java:235)
>   at java.io.DataOutputStream.writeInt(DataOutputStream.java:199)
>   at 
> org.apache.cassandra.io.compress.CompressionMetadata$ChunkSerializer.serialize(CompressionMetadata.java:412)
>   at 
> org.apache.cassandra.io.compress.CompressionMetadata$ChunkSerializer.serialize(CompressionMetadata.java:407)
>   at 
> org.apache.cassandra.streaming.compress.CompressionInfo$CompressionInfoSerializer.serialize(CompressionInfo.java:59)
>   at 
> org.apache.cassandra.streaming.compress.CompressionInfo$CompressionInfoSerializer.serialize(CompressionInfo.java:46)
>   at 
> org.apache.cassandra.streaming.PendingFile$PendingFileSerializer.serialize(PendingFile.java:142)
>   at 
> org.apache.cassandra.streaming.StreamHeader$StreamHeaderSerializer.serialize(StreamHeader.java:67)
>   at 
> org.apache.cassandra.streaming.StreamHeader$StreamHeaderSerializer.serialize(StreamHeader.java:58)
>   at 
> org.apache.cassandra.net.MessagingService.constructStreamHeader(MessagingService.java:782)
>   at 
> org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:65)
>   at 
> org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit 
> exceeded
>   at 
> org.apache.cassandra.io.compress.CompressionMetadata.getChunksForSections(CompressionMetadata.java:210)
>   at 
> 

[jira] [Assigned] (CASSANDRA-9778) CQL support for time series aggregation

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer reassigned CASSANDRA-9778:
-

Assignee: Benjamin Lerer

> CQL support for time series aggregation
> ---
>
> Key: CASSANDRA-9778
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9778
> Project: Cassandra
>  Issue Type: New Feature
>  Components: API, Core
>Reporter: Jonathan Ellis
>Assignee: Benjamin Lerer
> Fix For: 3.x
>
>
> Along with MV (CASSANDRA-6477), time series aggregation or "rollups" are a 
> common design pattern in cassandra applications.  I'd like to add CQL support 
> for this along these lines:
> {code}
> CREATE MATERIALIZED VIEW stocks_by_hour AS
> SELECT exchange, day, day_time(1h) AS hour, symbol, avg(price), sum(volume)
> FROM stocks
> GROUP BY exchange, day, symbol, hour
> PRIMARY KEY  ((exchange, day), hour, symbol);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10449) OOM on bootstrap after long GC pause

[Robbie Strickland (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969215#comment-14969215
 ] 

Robbie Strickland commented on CASSANDRA-10449:
---

Also, for reference, tpstats shows nothing in the queues:

{noformat}
ubuntu@eventcass4x087:~$ nodetool tpstats
Pool NameActive   Pending  Completed   Blocked  All 
time blocked
MutationStage 0 0   85431226 0  
   0
ReadStage 0 0  0 0  
   0
RequestResponseStage  0 0 48 0  
   0
ReadRepairStage   0 0  0 0  
   0
CounterMutationStage  0 0  0 0  
   0
MiscStage 0 0  0 0  
   0
HintedHandoff 0 0 29 0  
   0
GossipStage   0 0 565556 0  
   0
CacheCleanupExecutor  0 0  0 0  
   0
InternalResponseStage 0 0  0 0  
   0
CommitLogArchiver 0 0  0 0  
   0
CompactionExecutor0 0  12774 0  
   0
ValidationExecutor0 0  0 0  
   0
MigrationStage0 0  0 0  
   0
AntiEntropyStage  0 0  0 0  
   0
PendingRangeCalculator0 0  3 0  
   0
Sampler   0 0  0 0  
   0
MemtableFlushWriter   0 0   7157 0  
   0
MemtablePostFlush 0 0  10083 0  
   0
MemtableReclaimMemory 0 0   9340 0  
   0

Message type   Dropped
READ 0
RANGE_SLICE  0
_TRACE   0
MUTATION 0
COUNTER_MUTATION 0
BINARY   0
REQUEST_RESPONSE 0
PAGED_RANGE  0
READ_REPAIR  0
{noformat}

> OOM on bootstrap after long GC pause
> 
>
> Key: CASSANDRA-10449
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10449
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Ubuntu 14.04, AWS
>Reporter: Robbie Strickland
>  Labels: gc
> Fix For: 2.1.x
>
> Attachments: GCpath.txt, heap_dump.png, system.log.10-05, 
> thread_dump.log, threads.txt
>
>
> I have a 20-node cluster (i2.4xlarge) with vnodes (default of 256) and 
> 500-700GB per node.  SSTable counts are <10 per table.  I am attempting to 
> provision additional nodes, but bootstrapping OOMs every time after about 10 
> hours with a sudden long GC pause:
> {noformat}
> INFO  [Service Thread] 2015-10-05 23:33:33,373 GCInspector.java:252 - G1 Old 
> Generation GC in 1586126ms.  G1 Old Gen: 49213756976 -> 49072277176;
> ...
> ERROR [MemtableFlushWriter:454] 2015-10-05 23:33:33,380 
> CassandraDaemon.java:223 - Exception in thread 
> Thread[MemtableFlushWriter:454,5,main]
> java.lang.OutOfMemoryError: Java heap space
> {noformat}
> I have tried increasing max heap to 48G just to get through the bootstrap, to 
> no avail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10501) Failure to start up Cassandra when temporary compaction files are not all renamed after kill/crash (FSReadError)

[Fred A (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969226#comment-14969226
 ] 

Fred A commented on CASSANDRA-10501:


Is it safe to remove the files from the generation even in an single-node 
cluster scenario without data-loss?

> Failure to start up Cassandra when temporary compaction files are not all 
> renamed after kill/crash (FSReadError)
> 
>
> Key: CASSANDRA-10501
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10501
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Cassandra 2.1.6
> Redhat Linux
>Reporter: Mathieu Roy
>Assignee: Marcus Eriksson
>  Labels: compaction, triage
>
> We have seen an issue intermittently but repeatedly over the last few months 
> where, after exiting the Cassandra process, it fails to start with an 
> FSReadError (stack trace below). The FSReadError refers to a 'statistics' 
> file for a  that doesn't exist, though a corresponding temporary file does 
> exist (eg. there is no 
> /media/data/cassandraDB/data/clusteradmin/singleton_token-01a92ed069b511e59b2c53679a538c14/clusteradmin-singleton_token-ka-9-Statistics.db
>  file, but there is a 
> /media/data/cassandraDB/data/clusteradmin/singleton_token-01a92ed069b511e59b2c53679a538c14/clusteradmin-singleton_token-tmp-ka-9-Statistics.db
>  file.)
> We tracked down the issue to the fact that the process exited with leftover 
> compactions and some of the 'tmp' files for the SSTable had been renamed to 
> final files, but not all of them - the issue happens if the 'Statistics' file 
> is not renamed but others are. The scenario we've seen on the last two 
> occurrences involves the 'CompressionInfo' file being a final file while all 
> other files for the SSTable generation were left with 'tmp' names.
> When this occurs, Cassandra cannot start until the file issue is resolved; 
> we've worked around it by deleting the SSTable files from the same 
> generation, both final and tmp, which at least allows Cassandra to start. 
> Renaming all files to either tmp or final names would also work.
> We've done some debugging in Cassandra and have been unable to cause the 
> issue without renaming the files manually. The rename code at 
> SSTableWriter.rename() looks like it could result in this if the process 
> exits in the middle of the rename, but in every occurrence we've debugged 
> through, the Set of components is ordered and Statistics is the first file 
> renamed.
> However the comments in SSTableWriter.rename() suggest that the 'Data' file 
> is meant to be used as meaning the files were completely renamed. The method 
> ColumnFamilyStore. removeUnfinishedCompactionLeftovers(), however, will 
> proceed assuming the compaction is complete if any of the component files has 
> a final name, and will skip temporary files when reading the list. If the 
> 'Statistics' file is temporary then it won't be read, and the defaults does 
> not include a list of ancestors, leading to the NullPointerException.
> It appears that ColumnFamilyStore. removeUnfinishedCompactionLeftovers() 
> should perhaps either ensure that all 'tmp' files are properly renamed before 
> it uses them, or skip SSTable files that don't have either the 'Data' or 
> 'Statistics' file in final form.
> Stack trace: 
> {code}
> FSReadError in Failed to remove unfinished compaction leftovers (file: 
> /media/data/cassandraDB/data/clusteradmin/singleton_token-01a92ed069b511e59b2c53679a538c14/clusteradmin-singleton_token-ka-9-Statistics.db).
>   See log for details.
> at 
> org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:617)
> at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:302)
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:536)
> at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:625)
> Caused by: java.lang.NullPointerException
> at 
> org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:609)
> ... 3 more
> Exception encountered during startup: java.lang.NullPointerException
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/3] cassandra git commit: Make ScrubTest pass

[yukim]

Make ScrubTest pass

patch by yukim; reviewed by slebresne for CASSANDRA-10558


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/07989057
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/07989057
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/07989057

Branch: refs/heads/trunk
Commit: 079890579ba2997b270d8a42bdf7a708f4907aa2
Parents: bc5f167
Author: Yuki Morishita 
Authored: Wed Oct 21 16:05:51 2015 -0500
Committer: Yuki Morishita 
Committed: Thu Oct 22 09:44:01 2015 -0500

--
 .../io/sstable/SimpleSSTableMultiWriter.java|  2 +-
 .../io/sstable/format/big/BigTableWriter.java   |  4 +-
 .../unit/org/apache/cassandra/db/ScrubTest.java | 52 
 3 files changed, 45 insertions(+), 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/07989057/src/java/org/apache/cassandra/io/sstable/SimpleSSTableMultiWriter.java
--
diff --git 
a/src/java/org/apache/cassandra/io/sstable/SimpleSSTableMultiWriter.java 
b/src/java/org/apache/cassandra/io/sstable/SimpleSSTableMultiWriter.java
index e034d90..fd1b9a7 100644
--- a/src/java/org/apache/cassandra/io/sstable/SimpleSSTableMultiWriter.java
+++ b/src/java/org/apache/cassandra/io/sstable/SimpleSSTableMultiWriter.java
@@ -35,7 +35,7 @@ public class SimpleSSTableMultiWriter implements 
SSTableMultiWriter
 {
 private final SSTableWriter writer;
 
-private SimpleSSTableMultiWriter(SSTableWriter writer)
+protected SimpleSSTableMultiWriter(SSTableWriter writer)
 {
 this.writer = writer;
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/07989057/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java
--
diff --git 
a/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java 
b/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java
index fdc1c64..e02c919 100644
--- a/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java
@@ -52,7 +52,7 @@ public class BigTableWriter extends SSTableWriter
 
 private final IndexWriter iwriter;
 private final SegmentedFile.Builder dbuilder;
-private final SequentialWriter dataFile;
+protected final SequentialWriter dataFile;
 private DecoratedKey lastWrittenKey;
 private FileMark dataMark;
 
@@ -98,7 +98,7 @@ public class BigTableWriter extends SSTableWriter
 /**
  * Perform sanity checks on @param decoratedKey and @return the position 
in the data file before any data is written
  */
-private long beforeAppend(DecoratedKey decoratedKey)
+protected long beforeAppend(DecoratedKey decoratedKey)
 {
 assert decoratedKey != null : "Keys must not be null"; // empty keys 
ARE allowed b/c of indexed column values
 if (lastWrittenKey != null && lastWrittenKey.compareTo(decoratedKey) 
>= 0)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/07989057/test/unit/org/apache/cassandra/db/ScrubTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/ScrubTest.java 
b/test/unit/org/apache/cassandra/db/ScrubTest.java
index ab99750..a553fe8 100644
--- a/test/unit/org/apache/cassandra/db/ScrubTest.java
+++ b/test/unit/org/apache/cassandra/db/ScrubTest.java
@@ -29,6 +29,7 @@ import org.junit.Test;
 import org.junit.runner.RunWith;
 
 import org.apache.cassandra.*;
+import org.apache.cassandra.config.CFMetaData;
 import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.cql3.Operator;
 import org.apache.cassandra.cql3.QueryProcessor;
@@ -49,7 +50,11 @@ import 
org.apache.cassandra.exceptions.RequestExecutionException;
 import org.apache.cassandra.exceptions.WriteTimeoutException;
 import org.apache.cassandra.io.compress.CompressionMetadata;
 import org.apache.cassandra.io.sstable.*;
+import org.apache.cassandra.io.sstable.format.SSTableFormat;
 import org.apache.cassandra.io.sstable.format.SSTableReader;
+import org.apache.cassandra.io.sstable.format.SSTableWriter;
+import org.apache.cassandra.io.sstable.format.big.BigTableWriter;
+import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
 import org.apache.cassandra.io.util.FileUtils;
 import org.apache.cassandra.schema.KeyspaceParams;
 import org.apache.cassandra.utils.ByteBufferUtil;
@@ -321,14 +326,8 @@ public class ScrubTest
 String filename = cfs.getSSTablePath(tempDataDir);
 Descriptor desc = Descriptor.fromFilename(filename);
 
-try (SSTableTxnWriter writer = SSTableTxnWriter.create(cfs, 

[3/3] cassandra git commit: Merge branch 'cassandra-3.0' into trunk

[yukim]

Merge branch 'cassandra-3.0' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/04fa6457
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/04fa6457
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/04fa6457

Branch: refs/heads/trunk
Commit: 04fa645743e2d9ce864a39fa59d261f21d1c62d4
Parents: a252e3f 0798905
Author: Yuki Morishita 
Authored: Thu Oct 22 09:47:29 2015 -0500
Committer: Yuki Morishita 
Committed: Thu Oct 22 09:47:29 2015 -0500

--
 .../io/sstable/SimpleSSTableMultiWriter.java|  2 +-
 .../io/sstable/format/big/BigTableWriter.java   |  4 +-
 .../unit/org/apache/cassandra/db/ScrubTest.java | 52 
 3 files changed, 45 insertions(+), 13 deletions(-)
--

[jira] [Updated] (CASSANDRA-10092) Generalize PerRowSecondaryIndex validation

[JIRA]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrés de la Peña updated CASSANDRA-10092:
--
Attachment: CASSANDRA-10092_v2.patch

> Generalize PerRowSecondaryIndex validation
> --
>
> Key: CASSANDRA-10092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10092
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Andrés de la Peña
>Assignee: Andrés de la Peña
>Priority: Minor
>  Labels: 2i, secondary_index, validation
> Fix For: 2.1.x, 2.2.x
>
> Attachments: CASSANDRA-10092_v2.patch, improve_2i_validation.patch
>
>
> Index validation is currently done in a per-cell basis. However, per-row 
> secondary index developers can be interested in validating all the written 
> columns at once, because some implementations need to check the validity of a 
> row write by comparing some column values against others. For example, a per 
> row 2i implementation indexing time ranges (composed by a start date column 
> and an end date column) should check that the start date is before the stop 
> date.
> I'm attaching a patch adding a new method to {{PerRowSecondaryIndex}}:
> {code:java}
> public void validate(ByteBuffer key, ColumnFamily cf) throws 
> InvalidRequestException {}
> {code}
> and a new method to {{SecondaryIndexManager}}:
> {code:java}
> public void validateRowLevelIndexes(ByteBuffer key, ColumnFamily cf) throws 
> InvalidRequestException
>   {
>   for (SecondaryIndex index : rowLevelIndexMap.values())
>   {
>   ((PerRowSecondaryIndex) index).validate(key, cf);
>   }
>   }
> {code}
> This method is invoked in CQL {{UpdateStatement#validateIndexedColumns}}. 
> This way, {{PerRowSecondaryIndex}} could perform complex write validation.
> I have tried to do the patch in the least invasive way possible. Maybe the 
> current method {{SecondaryIndex#validate(ByteBuffer, Cell)}} should be moved 
> to {{PerColumnSecondaryIndex}}, and the {{InvalidRequestException}} that 
> informs about the particular 64k limitation should be thrown by 
> {{AbstractSimplePerColumnSecondaryIndex}}. However, given the incoming  
> [CASSANDRA-9459|https://issues.apache.org/jira/browse/CASSANDRA-9459], I 
> think that the proposed patch is more than enough to provide rich validation 
> features to 2i implementations based on 2.1.x and 2.2.x.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10557) Streaming can throw exception when trying to retry

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969257#comment-14969257
 ] 

Paulo Motta commented on CASSANDRA-10557:
-

Is this the same issue as CASSANDRA-10448? Were you able to reproduce?

Patch looks good and should help, but I don't understand how it relates to the 
{{java.lang.IllegalArgumentException: Unknown type 0}} issue, because as per 
CASSANDRA-10557 logs there was no retry and that exception was not thrown 
during {{SSTableWriter.abort()}} call. Could you please clarify?

> Streaming can throw exception when trying to retry
> --
>
> Key: CASSANDRA-10557
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10557
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Yuki Morishita
>Assignee: Yuki Morishita
>Priority: Minor
> Fix For: 2.1.x, 2.2.x, 3.0.0
>
>
> Streaming can throw below exception when trying to retry.
> This seems to be happening when underlining cause is not catched properly.
> {code}
> ERROR 18:45:56 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Streaming error 
> occurred
> java.lang.IllegalArgumentException: Unknown type 0
> at 
> org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:97)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
>  ~[main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10341) Streaming does not guarantee cache invalidation

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969144#comment-14969144
 ] 

Paulo Motta commented on CASSANDRA-10341:
-

Thanks for the review. Updated patches with suggested changes - tests are still 
running.

Basically, created and tested utility method 
{{Bounds.getNonOverlappingBounds()}} which is now used by 
{{ColumnFamilyStore.invalidateRowCache}} before iterating on bounds. Also 
modified naming from inclusive ranges to bounds.

> Streaming does not guarantee cache invalidation
> ---
>
> Key: CASSANDRA-10341
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10341
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Benedict
>Assignee: Paulo Motta
> Fix For: 2.1.x, 2.2.x, 3.0.x
>
>
> Looking at the code, we attempt to invalidate the row cache for any rows we 
> receive via streaming, however we invalidate them immediately, before the new 
> data is available. So, if it is requested (which is likely if it is "hot") in 
> the interval, it will be re-cached and not invalidated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10501) Failure to start up Cassandra when temporary compaction files are not all renamed after kill/crash (FSReadError)

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969168#comment-14969168
 ] 

Marcus Eriksson commented on CASSANDRA-10501:
-

Looks like the reason is that if we have renamed a single sstable component, we 
will enter 
[this|https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L647]
 loop and get the exception.

It should be safe to remove the files from the generation mentioned in the 
FSReadError until we have a fix out.

> Failure to start up Cassandra when temporary compaction files are not all 
> renamed after kill/crash (FSReadError)
> 
>
> Key: CASSANDRA-10501
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10501
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Cassandra 2.1.6
> Redhat Linux
>Reporter: Mathieu Roy
>Assignee: Marcus Eriksson
>  Labels: compaction, triage
>
> We have seen an issue intermittently but repeatedly over the last few months 
> where, after exiting the Cassandra process, it fails to start with an 
> FSReadError (stack trace below). The FSReadError refers to a 'statistics' 
> file for a  that doesn't exist, though a corresponding temporary file does 
> exist (eg. there is no 
> /media/data/cassandraDB/data/clusteradmin/singleton_token-01a92ed069b511e59b2c53679a538c14/clusteradmin-singleton_token-ka-9-Statistics.db
>  file, but there is a 
> /media/data/cassandraDB/data/clusteradmin/singleton_token-01a92ed069b511e59b2c53679a538c14/clusteradmin-singleton_token-tmp-ka-9-Statistics.db
>  file.)
> We tracked down the issue to the fact that the process exited with leftover 
> compactions and some of the 'tmp' files for the SSTable had been renamed to 
> final files, but not all of them - the issue happens if the 'Statistics' file 
> is not renamed but others are. The scenario we've seen on the last two 
> occurrences involves the 'CompressionInfo' file being a final file while all 
> other files for the SSTable generation were left with 'tmp' names.
> When this occurs, Cassandra cannot start until the file issue is resolved; 
> we've worked around it by deleting the SSTable files from the same 
> generation, both final and tmp, which at least allows Cassandra to start. 
> Renaming all files to either tmp or final names would also work.
> We've done some debugging in Cassandra and have been unable to cause the 
> issue without renaming the files manually. The rename code at 
> SSTableWriter.rename() looks like it could result in this if the process 
> exits in the middle of the rename, but in every occurrence we've debugged 
> through, the Set of components is ordered and Statistics is the first file 
> renamed.
> However the comments in SSTableWriter.rename() suggest that the 'Data' file 
> is meant to be used as meaning the files were completely renamed. The method 
> ColumnFamilyStore. removeUnfinishedCompactionLeftovers(), however, will 
> proceed assuming the compaction is complete if any of the component files has 
> a final name, and will skip temporary files when reading the list. If the 
> 'Statistics' file is temporary then it won't be read, and the defaults does 
> not include a list of ancestors, leading to the NullPointerException.
> It appears that ColumnFamilyStore. removeUnfinishedCompactionLeftovers() 
> should perhaps either ensure that all 'tmp' files are properly renamed before 
> it uses them, or skip SSTable files that don't have either the 'Data' or 
> 'Statistics' file in final form.
> Stack trace: 
> {code}
> FSReadError in Failed to remove unfinished compaction leftovers (file: 
> /media/data/cassandraDB/data/clusteradmin/singleton_token-01a92ed069b511e59b2c53679a538c14/clusteradmin-singleton_token-ka-9-Statistics.db).
>   See log for details.
> at 
> org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:617)
> at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:302)
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:536)
> at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:625)
> Caused by: java.lang.NullPointerException
> at 
> org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:609)
> ... 3 more
> Exception encountered during startup: java.lang.NullPointerException
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/3] cassandra git commit: Make ScrubTest pass

[yukim]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 bc5f16736 -> 079890579
  refs/heads/trunk a252e3f77 -> 04fa64574


Make ScrubTest pass

patch by yukim; reviewed by slebresne for CASSANDRA-10558


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/07989057
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/07989057
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/07989057

Branch: refs/heads/cassandra-3.0
Commit: 079890579ba2997b270d8a42bdf7a708f4907aa2
Parents: bc5f167
Author: Yuki Morishita 
Authored: Wed Oct 21 16:05:51 2015 -0500
Committer: Yuki Morishita 
Committed: Thu Oct 22 09:44:01 2015 -0500

--
 .../io/sstable/SimpleSSTableMultiWriter.java|  2 +-
 .../io/sstable/format/big/BigTableWriter.java   |  4 +-
 .../unit/org/apache/cassandra/db/ScrubTest.java | 52 
 3 files changed, 45 insertions(+), 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/07989057/src/java/org/apache/cassandra/io/sstable/SimpleSSTableMultiWriter.java
--
diff --git 
a/src/java/org/apache/cassandra/io/sstable/SimpleSSTableMultiWriter.java 
b/src/java/org/apache/cassandra/io/sstable/SimpleSSTableMultiWriter.java
index e034d90..fd1b9a7 100644
--- a/src/java/org/apache/cassandra/io/sstable/SimpleSSTableMultiWriter.java
+++ b/src/java/org/apache/cassandra/io/sstable/SimpleSSTableMultiWriter.java
@@ -35,7 +35,7 @@ public class SimpleSSTableMultiWriter implements 
SSTableMultiWriter
 {
 private final SSTableWriter writer;
 
-private SimpleSSTableMultiWriter(SSTableWriter writer)
+protected SimpleSSTableMultiWriter(SSTableWriter writer)
 {
 this.writer = writer;
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/07989057/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java
--
diff --git 
a/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java 
b/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java
index fdc1c64..e02c919 100644
--- a/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java
@@ -52,7 +52,7 @@ public class BigTableWriter extends SSTableWriter
 
 private final IndexWriter iwriter;
 private final SegmentedFile.Builder dbuilder;
-private final SequentialWriter dataFile;
+protected final SequentialWriter dataFile;
 private DecoratedKey lastWrittenKey;
 private FileMark dataMark;
 
@@ -98,7 +98,7 @@ public class BigTableWriter extends SSTableWriter
 /**
  * Perform sanity checks on @param decoratedKey and @return the position 
in the data file before any data is written
  */
-private long beforeAppend(DecoratedKey decoratedKey)
+protected long beforeAppend(DecoratedKey decoratedKey)
 {
 assert decoratedKey != null : "Keys must not be null"; // empty keys 
ARE allowed b/c of indexed column values
 if (lastWrittenKey != null && lastWrittenKey.compareTo(decoratedKey) 
>= 0)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/07989057/test/unit/org/apache/cassandra/db/ScrubTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/ScrubTest.java 
b/test/unit/org/apache/cassandra/db/ScrubTest.java
index ab99750..a553fe8 100644
--- a/test/unit/org/apache/cassandra/db/ScrubTest.java
+++ b/test/unit/org/apache/cassandra/db/ScrubTest.java
@@ -29,6 +29,7 @@ import org.junit.Test;
 import org.junit.runner.RunWith;
 
 import org.apache.cassandra.*;
+import org.apache.cassandra.config.CFMetaData;
 import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.cql3.Operator;
 import org.apache.cassandra.cql3.QueryProcessor;
@@ -49,7 +50,11 @@ import 
org.apache.cassandra.exceptions.RequestExecutionException;
 import org.apache.cassandra.exceptions.WriteTimeoutException;
 import org.apache.cassandra.io.compress.CompressionMetadata;
 import org.apache.cassandra.io.sstable.*;
+import org.apache.cassandra.io.sstable.format.SSTableFormat;
 import org.apache.cassandra.io.sstable.format.SSTableReader;
+import org.apache.cassandra.io.sstable.format.SSTableWriter;
+import org.apache.cassandra.io.sstable.format.big.BigTableWriter;
+import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
 import org.apache.cassandra.io.util.FileUtils;
 import org.apache.cassandra.schema.KeyspaceParams;
 import org.apache.cassandra.utils.ByteBufferUtil;
@@ -321,14 +326,8 @@ public class ScrubTest
 String filename = cfs.getSSTablePath(tempDataDir);
   

[jira] [Commented] (CASSANDRA-10449) OOM on bootstrap after long GC pause

[Robbie Strickland (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969125#comment-14969125
 ] 

Robbie Strickland commented on CASSANDRA-10449:
---

I decided to try upgrading to 2.1.11 to see if the issue was resolved by 
CASSANDRA-9681.  The node has been joining for over 24 hours, even though it 
appears to have finished streaming after about 6 hours:

{{noformat}}
ubuntu@eventcass4x087:~$ nodetool netstats | grep -v 100%
Mode: JOINING
Bootstrap 7047c510-7732-11e5-a7e7-63f53bbd2778
Receiving 171 files, 95313491312 bytes total. Already received 171 
files, 95313491312 bytes total
Receiving 165 files, 78860134041 bytes total. Already received 165 
files, 78860134041 bytes total
Receiving 158 files, 77709354374 bytes total. Already received 158 
files, 77709354374 bytes total
Receiving 184 files, 106710570690 bytes total. Already received 184 
files, 106710570690 bytes total
Receiving 136 files, 35699286217 bytes total. Already received 136 
files, 35699286217 bytes total
Receiving 169 files, 53498180215 bytes total. Already received 169 
files, 53498180215 bytes total
Receiving 197 files, 129020987979 bytes total. Already received 197 
files, 129020987979 bytes total
Receiving 196 files, 113904035360 bytes total. Already received 196 
files, 113904035360 bytes total
Receiving 172 files, 47685647028 bytes total. Already received 172 
files, 47685647028 bytes total
Read Repair Statistics:
Attempted: 0
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool NameActive   Pending  Completed
Commandsn/a 1  0
Responses   n/a 0   83743675
{{noformat}}

It doesn't appear to still be building indexes either:

{{noformat}}
ubuntu@eventcass4x087:~$ nodetool compactionstats
pending tasks: 2
   compaction typekeyspace  table   completed   total   
 unit   progress
Compaction   prod_analytics_events   wuevents   163704673   201033961   
bytes 81.43%
Active compaction remaining time :n/a
{{noformat}}

So I'm not sure why it's still joining.  Any thoughts?

> OOM on bootstrap after long GC pause
> 
>
> Key: CASSANDRA-10449
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10449
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Ubuntu 14.04, AWS
>Reporter: Robbie Strickland
>  Labels: gc
> Fix For: 2.1.x
>
> Attachments: GCpath.txt, heap_dump.png, system.log.10-05, 
> thread_dump.log, threads.txt
>
>
> I have a 20-node cluster (i2.4xlarge) with vnodes (default of 256) and 
> 500-700GB per node.  SSTable counts are <10 per table.  I am attempting to 
> provision additional nodes, but bootstrapping OOMs every time after about 10 
> hours with a sudden long GC pause:
> {noformat}
> INFO  [Service Thread] 2015-10-05 23:33:33,373 GCInspector.java:252 - G1 Old 
> Generation GC in 1586126ms.  G1 Old Gen: 49213756976 -> 49072277176;
> ...
> ERROR [MemtableFlushWriter:454] 2015-10-05 23:33:33,380 
> CassandraDaemon.java:223 - Exception in thread 
> Thread[MemtableFlushWriter:454,5,main]
> java.lang.OutOfMemoryError: Java heap space
> {noformat}
> I have tried increasing max heap to 48G just to get through the bootstrap, to 
> no avail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-10449) OOM on bootstrap after long GC pause

[Robbie Strickland (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969125#comment-14969125
 ] 

Robbie Strickland edited comment on CASSANDRA-10449 at 10/22/15 1:08 PM:
-

I decided to try upgrading to 2.1.11 to see if the issue was resolved by 
CASSANDRA-9681.  The node has been joining for over 24 hours, even though it 
appears to have finished streaming after about 6 hours:

{noformat}
ubuntu@eventcass4x087:~$ nodetool netstats | grep -v 100%
Mode: JOINING
Bootstrap 7047c510-7732-11e5-a7e7-63f53bbd2778
Receiving 171 files, 95313491312 bytes total. Already received 171 
files, 95313491312 bytes total
Receiving 165 files, 78860134041 bytes total. Already received 165 
files, 78860134041 bytes total
Receiving 158 files, 77709354374 bytes total. Already received 158 
files, 77709354374 bytes total
Receiving 184 files, 106710570690 bytes total. Already received 184 
files, 106710570690 bytes total
Receiving 136 files, 35699286217 bytes total. Already received 136 
files, 35699286217 bytes total
Receiving 169 files, 53498180215 bytes total. Already received 169 
files, 53498180215 bytes total
Receiving 197 files, 129020987979 bytes total. Already received 197 
files, 129020987979 bytes total
Receiving 196 files, 113904035360 bytes total. Already received 196 
files, 113904035360 bytes total
Receiving 172 files, 47685647028 bytes total. Already received 172 
files, 47685647028 bytes total
Read Repair Statistics:
Attempted: 0
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool NameActive   Pending  Completed
Commandsn/a 1  0
Responses   n/a 0   83743675
{noformat}

It doesn't appear to still be building indexes either:

{noformat}
ubuntu@eventcass4x087:~$ nodetool compactionstats
pending tasks: 2
   compaction typekeyspace  table   completed   total   
 unit   progress
Compaction   prod_analytics_events   wuevents   163704673   201033961   
bytes 81.43%
Active compaction remaining time :n/a
{noformat}

So I'm not sure why it's still joining.  Any thoughts?


was (Author: rstrickland):
I decided to try upgrading to 2.1.11 to see if the issue was resolved by 
CASSANDRA-9681.  The node has been joining for over 24 hours, even though it 
appears to have finished streaming after about 6 hours:

{{noformat}}
ubuntu@eventcass4x087:~$ nodetool netstats | grep -v 100%
Mode: JOINING
Bootstrap 7047c510-7732-11e5-a7e7-63f53bbd2778
Receiving 171 files, 95313491312 bytes total. Already received 171 
files, 95313491312 bytes total
Receiving 165 files, 78860134041 bytes total. Already received 165 
files, 78860134041 bytes total
Receiving 158 files, 77709354374 bytes total. Already received 158 
files, 77709354374 bytes total
Receiving 184 files, 106710570690 bytes total. Already received 184 
files, 106710570690 bytes total
Receiving 136 files, 35699286217 bytes total. Already received 136 
files, 35699286217 bytes total
Receiving 169 files, 53498180215 bytes total. Already received 169 
files, 53498180215 bytes total
Receiving 197 files, 129020987979 bytes total. Already received 197 
files, 129020987979 bytes total
Receiving 196 files, 113904035360 bytes total. Already received 196 
files, 113904035360 bytes total
Receiving 172 files, 47685647028 bytes total. Already received 172 
files, 47685647028 bytes total
Read Repair Statistics:
Attempted: 0
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool NameActive   Pending  Completed
Commandsn/a 1  0
Responses   n/a 0   83743675
{{noformat}}

It doesn't appear to still be building indexes either:

{{noformat}}
ubuntu@eventcass4x087:~$ nodetool compactionstats
pending tasks: 2
   compaction typekeyspace  table   completed   total   
 unit   progress
Compaction   prod_analytics_events   wuevents   163704673   201033961   
bytes 81.43%
Active compaction remaining time :n/a
{{noformat}}

So I'm not sure why it's still joining.  Any thoughts?

> OOM on bootstrap after long GC pause
> 
>
> Key: CASSANDRA-10449
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10449
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Ubuntu 14.04, AWS
>Reporter: Robbie Strickland
>  Labels: gc
> Fix For: 2.1.x
>
> Attachments: GCpath.txt, heap_dump.png, system.log.10-05, 
> thread_dump.log, threads.txt
>
>
> I have a 20-node cluster (i2.4xlarge) with vnodes (default 

[jira] [Comment Edited] (CASSANDRA-10449) OOM on bootstrap after long GC pause

[Robbie Strickland (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946624#comment-14946624
 ] 

Robbie Strickland edited comment on CASSANDRA-10449 at 10/22/15 1:10 PM:
-

I increased max heap to 96GB and tried again.  Now doing netstats shows 
progress ground to a halt:

9pm:

{noformat}
ubuntu@eventcass4x024:~$ nodetool netstats | grep -v 100%
Mode: JOINING
Bootstrap 45d8dec0-6c12-11e5-90ef-f7a8e02e59c0
Receiving 139 files, 36548040412 bytes total. Already received 139 
files, 36548040412 bytes total
Receiving 171 files, 6431853 bytes total. Already received 171 
files, 6431853 bytes total
Receiving 147 files, 78458709168 bytes total. Already received 79 
files, 55003961646 bytes total

/var/lib/cassandra/xvdd/data/prod_analytics_events/wuevents-ffa99ad05af911e596f05987bbaaffad/prod_analytics_events-wuevents-tmp-ka-295-Data.db
 955162267/4105438496 bytes(23%) received from idx:0/x.x.x.x
Receiving 141 files, 36700837768 bytes total. Already received 141 
files, 36700837768 bytes total
Receiving 176 files, 79676288976 bytes total. Already received 98 
files, 55932809644 bytes total

/var/lib/cassandra/xvdb/data/prod_analytics_events/wuevents-ffa99ad05af911e596f05987bbaaffad/prod_analytics_events-wuevents-tmp-ka-329-Data.db
 174070078/7326235809 bytes(2%) received from idx:0/x.x.x.x
Receiving 170 files, 85920995638 bytes total. Already received 94 
files, 54985226700 bytes total

/var/lib/cassandra/xvdd/data/prod_analytics_events/wuevents-ffa99ad05af911e596f05987bbaaffad/prod_analytics_events-wuevents-tmp-ka-265-Data.db
 4875660361/22821083384 bytes(21%) received from idx:0/x.x.x.x
Receiving 174 files, 87064163973 bytes total. Already received 91 
files, 53930233899 bytes total

/var/lib/cassandra/xvdb/data/prod_analytics_events/wuevents-ffa99ad05af911e596f05987bbaaffad/prod_analytics_events-wuevents-tmp-ka-157-Data.db
 17064156850/25823860172 bytes(66%) received from idx:0/x.x.x.x
Receiving 164 files, 46351636573 bytes total. Already received 164 
files, 46351636573 bytes total
Receiving 158 files, 62899520151 bytes total. Already received 158 
files, 62899520151 bytes total
Receiving 164 files, 48771232182 bytes total. Already received 164 
files, 48771232182 bytes total
Read Repair Statistics:
Attempted: 0
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool NameActive   Pending  Completed
Commandsn/a19 56
Responses   n/a 0   35515795
{noformat}

6am:

{noformat}
ubuntu@eventcass4x024:~$ nodetool netstats | grep -v 100%
Mode: JOINING
Bootstrap 45d8dec0-6c12-11e5-90ef-f7a8e02e59c0
Receiving 139 files, 36548040412 bytes total. Already received 139 
files, 36548040412 bytes total
Receiving 171 files, 6431853 bytes total. Already received 171 
files, 6431853 bytes total
Receiving 147 files, 78458709168 bytes total. Already received 79 
files, 55003961646 bytes total

/var/lib/cassandra/xvdd/data/prod_analytics_events/wuevents-ffa99ad05af911e596f05987bbaaffad/prod_analytics_events-wuevents-tmp-ka-295-Data.db
 955162267/4105438496 bytes(23%) received from idx:0/x.x.x.x
Receiving 141 files, 36700837768 bytes total. Already received 141 
files, 36700837768 bytes total
Receiving 176 files, 79676288976 bytes total. Already received 98 
files, 55932809644 bytes total

/var/lib/cassandra/xvdb/data/prod_analytics_events/wuevents-ffa99ad05af911e596f05987bbaaffad/prod_analytics_events-wuevents-tmp-ka-329-Data.db
 174070078/7326235809 bytes(2%) received from idx:0/x.x.x.x
Receiving 170 files, 85920995638 bytes total. Already received 94 
files, 54985226700 bytes total

/var/lib/cassandra/xvdd/data/prod_analytics_events/wuevents-ffa99ad05af911e596f05987bbaaffad/prod_analytics_events-wuevents-tmp-ka-265-Data.db
 4875660361/22821083384 bytes(21%) received from idx:0/x.x.x.x
Receiving 174 files, 87064163973 bytes total. Already received 91 
files, 53930233899 bytes total

/var/lib/cassandra/xvdb/data/prod_analytics_events/wuevents-ffa99ad05af911e596f05987bbaaffad/prod_analytics_events-wuevents-tmp-ka-157-Data.db
 17064156850/25823860172 bytes(66%) received from idx:0/x.x.x.x
Receiving 164 files, 46351636573 bytes total. Already received 164 
files, 46351636573 bytes total
Receiving 158 files, 62899520151 bytes total. Already received 158 
files, 62899520151 bytes total
Receiving 164 files, 48771232182 bytes total. Already received 164 
files, 48771232182 bytes total
Read Repair Statistics:
Attempted: 0
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool NameActive   Pending  Completed
Commands 

[jira] [Commented] (CASSANDRA-10554) Auto Bootstraping a new node fails

[Alan Boudreault (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969187#comment-14969187
 ] 

Alan Boudreault commented on CASSANDRA-10554:
-

While trying to produce a local test case, I've noticed that I can reproduce 
the error easily just by restarting a node. The error might not be related to 
the bootstraping anymore at this point since I can see it with a simple node 
restart. Here the error produce by my test case:

{code}
ERROR [SSTableBatchOpen:2] 2015-10-22 09:48:04,721 CassandraDaemon.java:195 - 
Exception in thread Thread[SSTableBatchOpen:2,5,main]
java.lang.RuntimeException: Unknown column added_time during deserialization
at 
org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
 ~[main/:na]
at 
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:477)
 ~[main/:na]
at 
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:366)
 ~[main/:na]
at 
org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:525)
 ~[main/:na]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
~[na:1.8.0_45]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[na:1.8.0_45]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
ERROR [SSTableBatchOpen:2] 2015-10-22 09:48:04,721 CassandraDaemon.java:195 - 
Exception in thread Thread[SSTableBatchOpen:2,5,main]
java.lang.RuntimeException: Unknown column added_time during deserialization
at 
org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
 ~[main/:na]
at 
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:477)
 ~[main/:na]
at 
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:366)
 ~[main/:na]
at 
org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:525)
 ~[main/:na]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
~[na:1.8.0_45]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[na:1.8.0_45]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
ERROR [SSTableBatchOpen:1] 2015-10-22 09:48:04,721 CassandraDaemon.java:195 - 
Exception in thread Thread[SSTableBatchOpen:1,5,main]
java.lang.RuntimeException: Unknown column added_time during deserialization
at 
org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
 ~[main/:na]
at 
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:477)
 ~[main/:na]
at 
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:366)
 ~[main/:na]
at 
org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:525)
 ~[main/:na]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
~[na:1.8.0_45]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[na:1.8.0_45]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
ERROR [SSTableBatchOpen:1] 2015-10-22 09:48:04,721 CassandraDaemon.java:195 - 
Exception in thread Thread[SSTableBatchOpen:1,5,main]
java.lang.RuntimeException: Unknown column added_time during deserialization
at 
org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
 ~[main/:na]
at 
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:477)
 ~[main/:na]
at 
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:366)
 ~[main/:na]
at 
org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:525)
 ~[main/:na]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
~[na:1.8.0_45]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[na:1.8.0_45]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]

[jira] [Updated] (CASSANDRA-10554) Auto Bootstraping a new node fails

[Alan Boudreault (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Boudreault updated CASSANDRA-10554:

Attachment: test.sh

> Auto Bootstraping a new node fails
> --
>
> Key: CASSANDRA-10554
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10554
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Alan Boudreault
>Assignee: Yuki Morishita
> Fix For: 3.0.0
>
> Attachments: 0001-Add-debug.txt, debug.log, system.log, test.sh
>
>
> I've been trying to add a new node in my 3.0 cluster and it seems to fail. 
> All my nodes are using apache/cassandra-3.0.0 branch. At the beginning, I can 
> see the following error:
> {code}
> INFO  18:45:55 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a ID#0] Prepare 
> completed. Receiving 42 files(1910066622 bytes), sending 0 files(0 bytes)
> WARN  18:45:55 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Retrying for 
> following error
> java.lang.RuntimeException: Unknown column added_time during deserialization
> at 
> org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.StreamReader.createWriter(StreamReader.java:136)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:77)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
>  [main/:na]
> at 
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
>  [main/:na]
> at 
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
>  [main/:na]
> at 
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
>  [main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> ERROR 18:45:55 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Streaming error 
> occurred
> java.lang.IllegalArgumentException: Unknown type 0
> at 
> org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:97)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
>  ~[main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> INFO  18:45:55 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Session with 
> /54.210.187.114 is complete
> INFO  18:45:56 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a ID#0] Prepare 
> completed. Receiving 38 files(2323537628 bytes), sending 0 files(0 bytes)
> WARN  18:45:56 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Retrying for 
> following error
> java.lang.RuntimeException: Unknown column added_time during deserialization
> at 
> org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.StreamReader.createWriter(StreamReader.java:136)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:77)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
>  [main/:na]
> at 
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
>  [main/:na]
> at 
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
>  [main/:na]
> at 
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
>  [main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> ERROR 18:45:56 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Streaming error 
> occurred
> java.lang.IllegalArgumentException: Unknown type 0
> at 
> org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:97)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58)
>  ~[main/:na]
> at 
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
>  ~[main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> INFO  18:45:56 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Session with 
> /54.210.184.198 is complete
> INFO  18:45:56 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a ID#0] Prepare 
> completed. Receiving 35 files(2069893759 bytes), sending 0 files(0 bytes)
> WARN  18:45:57 [Stream 

[jira] [Commented] (CASSANDRA-9975) Flatten Iterator call hierarchy with a shared Transformer

[Branimir Lambov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969268#comment-14969268
 ] 

Branimir Lambov commented on CASSANDRA-9975:


The code is complicated, but looks correct and is well documented. I think it's 
practically ready to commit once CI is clean.

Wondering if {{BasePartitions.hasNext/applyOne}} shouldn't take responsibility 
of closing {{next}} if {{applyToPartitions}} returned {{null}} since other 
aspects of the transformer do close automatically?

[toQuery in 
checkForShortRead|https://github.com/apache/cassandra/compare/trunk...belliottsmith:9975#diff-8781f9483cca1cfc87145c767295cc79R358]
 no longer has division-by-zero protection. Is it no longer necessary?

You should run the {{eclipse-warnings}} ant target and suppress any new 
warnings.


> Flatten Iterator call hierarchy with a shared Transformer
> -
>
> Key: CASSANDRA-9975
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9975
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Core
>Reporter: Benedict
>Assignee: Benedict
> Fix For: 3.0.0
>
>
> Stepping through a read response is made exceedingly difficult by the sheer 
> depth of the call hierarchy, and how rapidly your context jumps around. This 
> ticket intend to partially address that, by flattening one of the main causes 
> of this: iterator transformations.
> I have a patch that attempts to mitigate (but not entirely eliminate) this, 
> through the introduction of a {{RowTransformer}} class that all 
> transformations are applied through. If a transformation has already been 
> applied, the {{RowTransformer}} class does not wrap a new iterator, but 
> instead returns a new {{RowTransformer}} that wraps the original underlying 
> (untransformed) iterator and both transformations. This can accumulate an 
> arbitrary number of transformations and, quite importantly, can apply the 
> filtration step {{Unfiltered -> Row}}  in the same instance as well. The 
> intention being that a majority of control flow happens inside this 
> {{RowTransformer}}, so there is far less context jumping to cope with.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9975) Flatten Iterator call hierarchy with a shared Transformer

[Branimir Lambov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969306#comment-14969306
 ] 

Branimir Lambov commented on CASSANDRA-9975:


It does serve a lot of purposes, that's where the complexity comes from.

It may be better if the {{Transformer}} class grew into a package, so that each 
of its subclasses can be in its own source file, to turn them into easier 
chunks for digesting.

> Flatten Iterator call hierarchy with a shared Transformer
> -
>
> Key: CASSANDRA-9975
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9975
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Core
>Reporter: Benedict
>Assignee: Benedict
> Fix For: 3.0.0
>
>
> Stepping through a read response is made exceedingly difficult by the sheer 
> depth of the call hierarchy, and how rapidly your context jumps around. This 
> ticket intend to partially address that, by flattening one of the main causes 
> of this: iterator transformations.
> I have a patch that attempts to mitigate (but not entirely eliminate) this, 
> through the introduction of a {{RowTransformer}} class that all 
> transformations are applied through. If a transformation has already been 
> applied, the {{RowTransformer}} class does not wrap a new iterator, but 
> instead returns a new {{RowTransformer}} that wraps the original underlying 
> (untransformed) iterator and both transformations. This can accumulate an 
> arbitrary number of transformations and, quite importantly, can apply the 
> filtration step {{Unfiltered -> Row}}  in the same instance as well. The 
> intention being that a majority of control flow happens inside this 
> {{RowTransformer}}, so there is far less context jumping to cope with.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9975) Flatten Iterator call hierarchy with a shared Transformer

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969292#comment-14969292
 ] 

Sylvain Lebresne commented on CASSANDRA-9975:
-

bq. The code is complicated, but looks correct and is well documented.

Do you have ideas/suggestions to simplify it out of curiosity? Or does it 
appears justified/hard to avoid.

> Flatten Iterator call hierarchy with a shared Transformer
> -
>
> Key: CASSANDRA-9975
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9975
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Core
>Reporter: Benedict
>Assignee: Benedict
> Fix For: 3.0.0
>
>
> Stepping through a read response is made exceedingly difficult by the sheer 
> depth of the call hierarchy, and how rapidly your context jumps around. This 
> ticket intend to partially address that, by flattening one of the main causes 
> of this: iterator transformations.
> I have a patch that attempts to mitigate (but not entirely eliminate) this, 
> through the introduction of a {{RowTransformer}} class that all 
> transformations are applied through. If a transformation has already been 
> applied, the {{RowTransformer}} class does not wrap a new iterator, but 
> instead returns a new {{RowTransformer}} that wraps the original underlying 
> (untransformed) iterator and both transformations. This can accumulate an 
> arbitrary number of transformations and, quite importantly, can apply the 
> filtration step {{Unfiltered -> Row}}  in the same instance as well. The 
> intention being that a majority of control flow happens inside this 
> {{RowTransformer}}, so there is far less context jumping to cope with.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10092) Generalize PerRowSecondaryIndex validation

[JIRA]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969270#comment-14969270
 ] 

Andrés de la Peña commented on CASSANDRA-10092:
---

I have uploaded a new version of the patch with the requested changes. I have 
added equivalent validation on Thrift path, and a couple of tests in 
{{PerRowSecondaryIndexTest}}, one for CQL and another one for Thrift. These 
tests are based on the existing {{PerRowSecondaryIndexTest#testInvalidSearch}}.

> Generalize PerRowSecondaryIndex validation
> --
>
> Key: CASSANDRA-10092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10092
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Andrés de la Peña
>Assignee: Andrés de la Peña
>Priority: Minor
>  Labels: 2i, secondary_index, validation
> Fix For: 2.1.x, 2.2.x
>
> Attachments: CASSANDRA-10092_v2.patch, improve_2i_validation.patch
>
>
> Index validation is currently done in a per-cell basis. However, per-row 
> secondary index developers can be interested in validating all the written 
> columns at once, because some implementations need to check the validity of a 
> row write by comparing some column values against others. For example, a per 
> row 2i implementation indexing time ranges (composed by a start date column 
> and an end date column) should check that the start date is before the stop 
> date.
> I'm attaching a patch adding a new method to {{PerRowSecondaryIndex}}:
> {code:java}
> public void validate(ByteBuffer key, ColumnFamily cf) throws 
> InvalidRequestException {}
> {code}
> and a new method to {{SecondaryIndexManager}}:
> {code:java}
> public void validateRowLevelIndexes(ByteBuffer key, ColumnFamily cf) throws 
> InvalidRequestException
>   {
>   for (SecondaryIndex index : rowLevelIndexMap.values())
>   {
>   ((PerRowSecondaryIndex) index).validate(key, cf);
>   }
>   }
> {code}
> This method is invoked in CQL {{UpdateStatement#validateIndexedColumns}}. 
> This way, {{PerRowSecondaryIndex}} could perform complex write validation.
> I have tried to do the patch in the least invasive way possible. Maybe the 
> current method {{SecondaryIndex#validate(ByteBuffer, Cell)}} should be moved 
> to {{PerColumnSecondaryIndex}}, and the {{InvalidRequestException}} that 
> informs about the particular 64k limitation should be thrown by 
> {{AbstractSimplePerColumnSecondaryIndex}}. However, given the incoming  
> [CASSANDRA-9459|https://issues.apache.org/jira/browse/CASSANDRA-9459], I 
> think that the proposed patch is more than enough to provide rich validation 
> features to 2i implementations based on 2.1.x and 2.2.x.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-7645) cqlsh: show partial trace if incomplete after max_trace_wait

[Adam Holmberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-7645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969281#comment-14969281
 ] 

Adam Holmberg commented on CASSANDRA-7645:
--

Sorry I'm late to the party. I don't think this will work as expected because 
[ResponseFuture.get_query_trace raises TraceUnavailable if it's 
incomplete|http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.ResponseFuture.get_query_trace].
 I can address in CASSANDRA-10513 if you prefer not to update.

> cqlsh: show partial trace if incomplete after max_trace_wait
> 
>
> Key: CASSANDRA-7645
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7645
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Tyler Hobbs
>Assignee: Carl Yeksigian
>Priority: Trivial
> Fix For: 3.0.0 rc2, 2.2.4, 2.1.12
>
>
> If a trace hasn't completed within {{max_trace_wait}}, cqlsh will say the 
> trace is unavailable and not show anything.  It (and the underlying python 
> driver) determines when the trace is complete by checking if the {{duration}} 
> column in {{system_traces.sessions}} is non-null.  If {{duration}} is null 
> but we still have some trace events when the timeout is hit, cqlsh should 
> print whatever trace events we have along with a warning about it being 
> incomplete.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-10569) Keyspace validation errors are getting lost in system_add_keyspace

[Mike Adamson (JIRA)]

Mike Adamson created CASSANDRA-10569:


 Summary: Keyspace validation errors are getting lost in 
system_add_keyspace
 Key: CASSANDRA-10569
 URL: https://issues.apache.org/jira/browse/CASSANDRA-10569
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: Mike Adamson
 Fix For: 3.0.0


The following:
{noformat}
cassandraserver.system_add_keyspace(
new KsDef("xxx", SimpleStrategy.class.getSimpleName(), 
  Lists.newArrayList()));
{noformat}
used to throw an {{InvalidRequestException}} in 2.1. 

In 3.0 the strategy validation has been removed from 
{{KeyspaceMetadata.validate}} so the strategy errors don't get picked up until 
the schema change has been announced. As a result the {{ConfigurationError}} is 
swallowed in {{FBUtilities.waitOnFuture}} and thrown on as a 
{{RuntimeException}}.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10415) Fix cqlsh bugs

[Benjamin Lerer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968761#comment-14968761
 ] 

Benjamin Lerer commented on CASSANDRA-10415:


Thanks for the patch. It looks good to me.

> Fix cqlsh bugs
> --
>
> Key: CASSANDRA-10415
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10415
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: Jim Witschey
>Assignee: Stefania
>  Labels: cqlsh
> Fix For: 3.0.0 rc2, 2.2.4, 2.1.12
>
>
> This is followup to CASSANDRA-10289
> The tests currently failing should be:
> * 
> {{cqlshlib.test.test_cqlsh_completion.TestCqlshCompletion.test_complete_in_create_columnfamily}}
> ** uses {{create_columnfamily_table_template}}. Stefania says "the {{(}} 
> after {{CREATE ... IF}} does not look valid to me."
> * 
> {{cqlshlib.test.test_cqlsh_completion.TestCqlshCompletion.test_complete_in_create_table}}
> ** uses {{create_columnfamily_table_template}}, see above.
> * 
> {{cqlshlib.test.test_cqlsh_completion.TestCqlshCompletion.test_complete_in_delete}}
> ** Stefania says: "I don't think keyspaces are a valid completion after 
> {{DELETE a [}} and after {{DELETE FROM twenty_rows_composite_table USING 
> TIMESTAMP 0 WHERE TOKEN(a) >=}}. From a quick analysis of {{cqlhandling.py}} 
> I think it comes from {{}}, which picks up {{}}, which 
> was changed to include {{ks.}} by CASSANDRA-7556.
> * 
> {{cqlshlib.test.test_cqlsh_completion.TestCqlshCompletion.test_complete_in_drop_keyspace}}
> ** Stefania says: "the {{;}} after {{DROP KEYSPACE IF}} is not valid.
> * {{cqlshlib.test.test_cqlsh_output.TestCqlshOutput.test_timestamp_output}}
> ** already documented with CASSANDRA-10313 and CASSANDRA-10397
> I'm happy to break these out into separate tickets if necessary. 
> To run the tests locally, I cd to {{cassandra/pylib/cqlshlib}} and run the 
> following:
> {code}
> ccm create -n 1 --install-dir=../.. test
> ccm start --wait-for-binary-proto
> nosetests test 2>&1
> ccm remove
> {code}
> This requires nose and ccm. Until CASSANDRA-10289 is resolved, you'll have to 
> use my branch here: https://github.com/mambocab/cassandra/tree/fix-cqlsh-tests
> Tests for this branch are run (non-continuously) here:
> http://cassci.datastax.com/job/scratch_mambocab-fix_cqlsh/
> Assigning [~Stefania] for now, since she's already looked at 10289, but feel 
> free to reassign.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10513) Update cqlsh for new driver execution API

[Adam Holmberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Holmberg updated CASSANDRA-10513:
--
Attachment: (was: 10513.txt)

> Update cqlsh for new driver execution API
> -
>
> Key: CASSANDRA-10513
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10513
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Tools
>Reporter: Adam Holmberg
>Assignee: Adam Holmberg
>Priority: Minor
>  Labels: cqlsh
> Fix For: 2.2.x, 3.0.x
>
>
> The 3.0 Python driver will have a few tweaks to the execution API. We'll need 
> to update cqlsh in a couple of minor ways.
> [Results are always returned as an iterable 
> ResultSet|https://datastax-oss.atlassian.net/browse/PYTHON-368]
> [Trace data is always attached to the 
> ResultSet|https://datastax-oss.atlassian.net/browse/PYTHON-318] (instead of 
> being attached to a statement)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10513) Update cqlsh for new driver execution API

[Adam Holmberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Holmberg updated CASSANDRA-10513:
--
Attachment: 10513.txt

> Update cqlsh for new driver execution API
> -
>
> Key: CASSANDRA-10513
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10513
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Tools
>Reporter: Adam Holmberg
>Assignee: Adam Holmberg
>Priority: Minor
>  Labels: cqlsh
> Fix For: 2.2.x, 3.0.x
>
> Attachments: 10513.txt
>
>
> The 3.0 Python driver will have a few tweaks to the execution API. We'll need 
> to update cqlsh in a couple of minor ways.
> [Results are always returned as an iterable 
> ResultSet|https://datastax-oss.atlassian.net/browse/PYTHON-368]
> [Trace data is always attached to the 
> ResultSet|https://datastax-oss.atlassian.net/browse/PYTHON-318] (instead of 
> being attached to a statement)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (CASSANDRA-10513) Update cqlsh for new driver execution API

[Adam Holmberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Holmberg reassigned CASSANDRA-10513:
-

Assignee: Paulo Motta  (was: Adam Holmberg)

Updated patch attached.

> Update cqlsh for new driver execution API
> -
>
> Key: CASSANDRA-10513
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10513
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Tools
>Reporter: Adam Holmberg
>Assignee: Paulo Motta
>Priority: Minor
>  Labels: cqlsh
> Fix For: 2.2.x, 3.0.x
>
> Attachments: 10513.txt
>
>
> The 3.0 Python driver will have a few tweaks to the execution API. We'll need 
> to update cqlsh in a couple of minor ways.
> [Results are always returned as an iterable 
> ResultSet|https://datastax-oss.atlassian.net/browse/PYTHON-368]
> [Trace data is always attached to the 
> ResultSet|https://datastax-oss.atlassian.net/browse/PYTHON-318] (instead of 
> being attached to a statement)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-7645) cqlsh: show partial trace if incomplete after max_trace_wait

[Adam Holmberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-7645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969398#comment-14969398
 ] 

Adam Holmberg edited comment on CASSANDRA-7645 at 10/22/15 4:33 PM:


bq. since the driver ignores the TraceUnavailable during ResultFuture 
construction.
That's why he originally changed it to attach the trace_id to the statement. 
That's also changing in CASSANDRA-10513 and driver 3.0.

bq. maybe we should create a driver ticket to support partial query traces 
first?
Yes. I think QueryTrace should have the option to populate events without 
waiting for the session. 
https://datastax-oss.atlassian.net/browse/PYTHON-438


was (Author: aholmber):
bq. since the driver ignores the TraceUnavailable during ResultFuture 
construction.
That's why he originally changed it to attach the trace_id to the statement. 
That's also changing in CASSANDRA-10513 and driver 3.0.

bq. maybe we should create a driver ticket to support partial query traces 
first?
Yes. I think QueryTrace should have the option to populate events without 
waiting for the session. 
https://datastax-oss.atlassian.net/browse/PYTHON-438

I defer on whether to reopen this ticket or create a new one.

> cqlsh: show partial trace if incomplete after max_trace_wait
> 
>
> Key: CASSANDRA-7645
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7645
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Tyler Hobbs
>Assignee: Carl Yeksigian
>Priority: Trivial
> Fix For: 3.0.0 rc2, 2.2.4, 2.1.12
>
>
> If a trace hasn't completed within {{max_trace_wait}}, cqlsh will say the 
> trace is unavailable and not show anything.  It (and the underlying python 
> driver) determines when the trace is complete by checking if the {{duration}} 
> column in {{system_traces.sessions}} is non-null.  If {{duration}} is null 
> but we still have some trace events when the timeout is hit, cqlsh should 
> print whatever trace events we have along with a warning about it being 
> incomplete.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10569) Keyspace validation errors are getting lost in system_add_keyspace

[Mike Adamson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Adamson updated CASSANDRA-10569:
-
Description: 
The following:
{noformat}
cassandraserver.system_add_keyspace(
new KsDef("xxx", SimpleStrategy.class.getSimpleName(), 
  Lists.newArrayList()));
{noformat}
used to throw an {{InvalidRequestException}} in 2.1. 

In 3.0 the strategy validation has been removed from 
{{KeyspaceMetadata.validate}} so the strategy errors don't get picked up until 
the schema change has been announced. As a result the {{ConfigurationError}} is 
swallowed in {{FBUtilities.waitOnFuture}} and thrown on as a 
{{RuntimeException}}.

This possibly affects {{system_update_keyspace}} as well.


  was:
The following:
{noformat}
cassandraserver.system_add_keyspace(
new KsDef("xxx", SimpleStrategy.class.getSimpleName(), 
  Lists.newArrayList()));
{noformat}
used to throw an {{InvalidRequestException}} in 2.1. 

In 3.0 the strategy validation has been removed from 
{{KeyspaceMetadata.validate}} so the strategy errors don't get picked up until 
the schema change has been announced. As a result the {{ConfigurationError}} is 
swallowed in {{FBUtilities.waitOnFuture}} and thrown on as a 
{{RuntimeException}}.



> Keyspace validation errors are getting lost in system_add_keyspace
> --
>
> Key: CASSANDRA-10569
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10569
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Mike Adamson
>Assignee: Sam Tunnicliffe
> Fix For: 3.0.0
>
>
> The following:
> {noformat}
> cassandraserver.system_add_keyspace(
> new KsDef("xxx", SimpleStrategy.class.getSimpleName(), 
>   Lists.newArrayList()));
> {noformat}
> used to throw an {{InvalidRequestException}} in 2.1. 
> In 3.0 the strategy validation has been removed from 
> {{KeyspaceMetadata.validate}} so the strategy errors don't get picked up 
> until the schema change has been announced. As a result the 
> {{ConfigurationError}} is swallowed in {{FBUtilities.waitOnFuture}} and 
> thrown on as a {{RuntimeException}}.
> This possibly affects {{system_update_keyspace}} as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-7645) cqlsh: show partial trace if incomplete after max_trace_wait

[Adam Holmberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-7645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969365#comment-14969365
 ] 

Adam Holmberg commented on CASSANDRA-7645:
--

Also, the fallback mechanism `show_session` uses the same QueryTrace.populate 
mechanism to query the trace, but with a max_wait of 2.0 seconds. We would need 
to bypass that if we really want to see a partial trace.

> cqlsh: show partial trace if incomplete after max_trace_wait
> 
>
> Key: CASSANDRA-7645
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7645
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Tyler Hobbs
>Assignee: Carl Yeksigian
>Priority: Trivial
> Fix For: 3.0.0 rc2, 2.2.4, 2.1.12
>
>
> If a trace hasn't completed within {{max_trace_wait}}, cqlsh will say the 
> trace is unavailable and not show anything.  It (and the underlying python 
> driver) determines when the trace is complete by checking if the {{duration}} 
> column in {{system_traces.sessions}} is non-null.  If {{duration}} is null 
> but we still have some trace events when the timeout is hit, cqlsh should 
> print whatever trace events we have along with a warning about it being 
> incomplete.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (CASSANDRA-10569) Keyspace validation errors are getting lost in system_add_keyspace

[Sam Tunnicliffe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Tunnicliffe reassigned CASSANDRA-10569:
---

Assignee: Sam Tunnicliffe

> Keyspace validation errors are getting lost in system_add_keyspace
> --
>
> Key: CASSANDRA-10569
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10569
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Mike Adamson
>Assignee: Sam Tunnicliffe
> Fix For: 3.0.0
>
>
> The following:
> {noformat}
> cassandraserver.system_add_keyspace(
> new KsDef("xxx", SimpleStrategy.class.getSimpleName(), 
>   Lists.newArrayList()));
> {noformat}
> used to throw an {{InvalidRequestException}} in 2.1. 
> In 3.0 the strategy validation has been removed from 
> {{KeyspaceMetadata.validate}} so the strategy errors don't get picked up 
> until the schema change has been announced. As a result the 
> {{ConfigurationError}} is swallowed in {{FBUtilities.waitOnFuture}} and 
> thrown on as a {{RuntimeException}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (CASSANDRA-7645) cqlsh: show partial trace if incomplete after max_trace_wait

[Paulo Motta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-7645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta reopened CASSANDRA-7645:


> cqlsh: show partial trace if incomplete after max_trace_wait
> 
>
> Key: CASSANDRA-7645
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7645
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Tyler Hobbs
>Assignee: Carl Yeksigian
>Priority: Trivial
> Fix For: 3.0.0 rc2, 2.2.4, 2.1.12
>
>
> If a trace hasn't completed within {{max_trace_wait}}, cqlsh will say the 
> trace is unavailable and not show anything.  It (and the underlying python 
> driver) determines when the trace is complete by checking if the {{duration}} 
> column in {{system_traces.sessions}} is non-null.  If {{duration}} is null 
> but we still have some trace events when the timeout is hit, cqlsh should 
> print whatever trace events we have along with a warning about it being 
> incomplete.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-7645) cqlsh: show partial trace if incomplete after max_trace_wait

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-7645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969376#comment-14969376
 ] 

Paulo Motta commented on CASSANDRA-7645:


bq. Also, the fallback mechanism (show_session) uses the same 
QueryTrace.populate mechanism to query the trace, but with a max_wait of 2.0 
seconds.

actually only the fallback mechanism is not working, since the driver 
[ignores|https://github.com/carlyeks/python-driver/blob/58536f57467db9998417eff57304ef40562f5975/cassandra/cluster.py#L1420]
 the {{TraceUnavailable}} during {{ResultFuture}} construction.

maybe we should create a driver ticket to support partial query traces first?

> cqlsh: show partial trace if incomplete after max_trace_wait
> 
>
> Key: CASSANDRA-7645
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7645
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Tyler Hobbs
>Assignee: Carl Yeksigian
>Priority: Trivial
> Fix For: 3.0.0 rc2, 2.2.4, 2.1.12
>
>
> If a trace hasn't completed within {{max_trace_wait}}, cqlsh will say the 
> trace is unavailable and not show anything.  It (and the underlying python 
> driver) determines when the trace is complete by checking if the {{duration}} 
> column in {{system_traces.sessions}} is non-null.  If {{duration}} is null 
> but we still have some trace events when the timeout is hit, cqlsh should 
> print whatever trace events we have along with a warning about it being 
> incomplete.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-7645) cqlsh: show partial trace if incomplete after max_trace_wait

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-7645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969376#comment-14969376
 ] 

Paulo Motta edited comment on CASSANDRA-7645 at 10/22/15 4:18 PM:
--

bq. Also, the fallback mechanism (show_session) uses the same 
QueryTrace.populate mechanism to query the trace, but with a max_wait of 2.0 
seconds.

actually only the fallback mechanism may not work due to {{TraceUnavailable}}, 
since the driver 
[ignores|https://github.com/carlyeks/python-driver/blob/58536f57467db9998417eff57304ef40562f5975/cassandra/cluster.py#L1420]
 the {{TraceUnavailable}} during {{ResultFuture}} construction.

maybe we should create a driver ticket to support partial query traces first?


was (Author: pauloricardomg):
bq. Also, the fallback mechanism (show_session) uses the same 
QueryTrace.populate mechanism to query the trace, but with a max_wait of 2.0 
seconds.

actually only the fallback mechanism is not working, since the driver 
[ignores|https://github.com/carlyeks/python-driver/blob/58536f57467db9998417eff57304ef40562f5975/cassandra/cluster.py#L1420]
 the {{TraceUnavailable}} during {{ResultFuture}} construction.

maybe we should create a driver ticket to support partial query traces first?

> cqlsh: show partial trace if incomplete after max_trace_wait
> 
>
> Key: CASSANDRA-7645
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7645
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Tyler Hobbs
>Assignee: Carl Yeksigian
>Priority: Trivial
> Fix For: 3.0.0 rc2, 2.2.4, 2.1.12
>
>
> If a trace hasn't completed within {{max_trace_wait}}, cqlsh will say the 
> trace is unavailable and not show anything.  It (and the underlying python 
> driver) determines when the trace is complete by checking if the {{duration}} 
> column in {{system_traces.sessions}} is non-null.  If {{duration}} is null 
> but we still have some trace events when the timeout is hit, cqlsh should 
> print whatever trace events we have along with a warning about it being 
> incomplete.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9975) Flatten Iterator call hierarchy with a shared Transformer

[Benedict (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969381#comment-14969381
 ] 

Benedict commented on CASSANDRA-9975:
-

bq. It may be better if the Transformer class grew into a package

More than happy to make that change. I agree there's a lot in there, especially 
boilerplate (static methods etc), which might be easier to digest in smaller 
chunks.

As for general complexity: this should just be taken as a first step, in my 
book. There's a lot that can potentially be done in follow ups, but I didn't 
want to eat the whole world. An imperfect first step is still a step in the 
right direction. The interaction with counters, for instance, could be more 
tightly coupled to perhaps reduce complexity wrt "stopping early". Possibly the 
control flow can be simplified for the {{MoreContents}} work. There are many 
avenues to follow, in fact.

My main goal here wasn't to make _this_ code simple (although I tried to make 
it as good as I could personally manage in a reasonable time frame, given the 
headline goal), but to make layers of the transformations themselves simple to 
follow.


> Flatten Iterator call hierarchy with a shared Transformer
> -
>
> Key: CASSANDRA-9975
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9975
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Core
>Reporter: Benedict
>Assignee: Benedict
> Fix For: 3.0.0
>
>
> Stepping through a read response is made exceedingly difficult by the sheer 
> depth of the call hierarchy, and how rapidly your context jumps around. This 
> ticket intend to partially address that, by flattening one of the main causes 
> of this: iterator transformations.
> I have a patch that attempts to mitigate (but not entirely eliminate) this, 
> through the introduction of a {{RowTransformer}} class that all 
> transformations are applied through. If a transformation has already been 
> applied, the {{RowTransformer}} class does not wrap a new iterator, but 
> instead returns a new {{RowTransformer}} that wraps the original underlying 
> (untransformed) iterator and both transformations. This can accumulate an 
> arbitrary number of transformations and, quite importantly, can apply the 
> filtration step {{Unfiltered -> Row}}  in the same instance as well. The 
> intention being that a majority of control flow happens inside this 
> {{RowTransformer}}, so there is far less context jumping to cope with.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-7645) cqlsh: show partial trace if incomplete after max_trace_wait

[Adam Holmberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-7645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969365#comment-14969365
 ] 

Adam Holmberg edited comment on CASSANDRA-7645 at 10/22/15 4:09 PM:


Also, the fallback mechanism (show_session) uses the same QueryTrace.populate 
mechanism to query the trace, but with a max_wait of 2.0 seconds. We would need 
to bypass that if we really want to see a partial trace.


was (Author: aholmber):
Also, the fallback mechanism `show_session` uses the same QueryTrace.populate 
mechanism to query the trace, but with a max_wait of 2.0 seconds. We would need 
to bypass that if we really want to see a partial trace.

> cqlsh: show partial trace if incomplete after max_trace_wait
> 
>
> Key: CASSANDRA-7645
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7645
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Tyler Hobbs
>Assignee: Carl Yeksigian
>Priority: Trivial
> Fix For: 3.0.0 rc2, 2.2.4, 2.1.12
>
>
> If a trace hasn't completed within {{max_trace_wait}}, cqlsh will say the 
> trace is unavailable and not show anything.  It (and the underlying python 
> driver) determines when the trace is complete by checking if the {{duration}} 
> column in {{system_traces.sessions}} is non-null.  If {{duration}} is null 
> but we still have some trace events when the timeout is hit, cqlsh should 
> print whatever trace events we have along with a warning about it being 
> incomplete.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-7645) cqlsh: show partial trace if incomplete after max_trace_wait

[Adam Holmberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-7645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969398#comment-14969398
 ] 

Adam Holmberg commented on CASSANDRA-7645:
--

bq. since the driver ignores the TraceUnavailable during ResultFuture 
construction.
That's why he originally changed it to attach the trace_id to the statement. 
That's also changing in CASSANDRA-10513 and driver 3.0.

bq. maybe we should create a driver ticket to support partial query traces 
first?
Yes. I think QueryTrace should have the option to populate events without 
waiting for the session. 
https://datastax-oss.atlassian.net/browse/PYTHON-438

I defer on whether to reopen this ticket or create a new one.

> cqlsh: show partial trace if incomplete after max_trace_wait
> 
>
> Key: CASSANDRA-7645
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7645
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Tyler Hobbs
>Assignee: Carl Yeksigian
>Priority: Trivial
> Fix For: 3.0.0 rc2, 2.2.4, 2.1.12
>
>
> If a trace hasn't completed within {{max_trace_wait}}, cqlsh will say the 
> trace is unavailable and not show anything.  It (and the underlying python 
> driver) determines when the trace is complete by checking if the {{duration}} 
> column in {{system_traces.sessions}} is non-null.  If {{duration}} is null 
> but we still have some trace events when the timeout is hit, cqlsh should 
> print whatever trace events we have along with a warning about it being 
> incomplete.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-7645) cqlsh: show partial trace if incomplete after max_trace_wait

[Adam Holmberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-7645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969463#comment-14969463
 ] 

Adam Holmberg commented on CASSANDRA-7645:
--

Proposed API update in [driver 
branch|https://github.com/datastax/python-driver/tree/438].

> cqlsh: show partial trace if incomplete after max_trace_wait
> 
>
> Key: CASSANDRA-7645
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7645
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Tyler Hobbs
>Assignee: Carl Yeksigian
>Priority: Trivial
> Fix For: 3.0.0 rc2, 2.2.4, 2.1.12
>
>
> If a trace hasn't completed within {{max_trace_wait}}, cqlsh will say the 
> trace is unavailable and not show anything.  It (and the underlying python 
> driver) determines when the trace is complete by checking if the {{duration}} 
> column in {{system_traces.sessions}} is non-null.  If {{duration}} is null 
> but we still have some trace events when the timeout is hit, cqlsh should 
> print whatever trace events we have along with a warning about it being 
> incomplete.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)