[jira] [Commented] (CASSANDRA-11678) cassandra crush when enable hints_compression

[Weijian Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265109#comment-15265109
 ] 

Weijian Lin commented on CASSANDRA-11678:
-

Every time I re-initialized all the cluster node and inserted data, some of the 
nodes crushed. I retried many times with V3.0.5 and V3.5.0. It came out the 
same result.  But after I comment out the hints_compression settings, it's ok. 




all the hints settings:

hinted_handoff_enabled: true
max_hint_window_in_ms: 1080 # 3 hours
hinted_handoff_throttle_in_kb: 1024
max_hints_delivery_threads: 2
hints_directory: /data1/cassandra/hints
hints_flush_period_in_ms: 1
max_hints_file_size_in_mb: 128
hints_compression:
   - class_name: LZ4Compressor
 parameters:
 -

> cassandra crush when enable hints_compression
> -
>
> Key: CASSANDRA-11678
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11678
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core, Local Write-Read Paths
> Environment: Centos 7
>Reporter: Weijian Lin
>Assignee: Blake Eggleston
>Priority: Critical
>
> When I enable hints_compression and set the compression class to
> LZ4Compressor，the
> cassandra (v3.05, V3.5.0) will crush。That is a bug, or any conf is wrong?
> *Exception   in V 3.5.0  *
> ERROR [HintsDispatcher:2] 2016-04-26 15:02:56,970
> HintsDispatchExecutor.java:225 - Failed to dispatch hints file
> abc4dda2-b551-427e-bb0b-e383d4a392e1-1461654138963-1.hints: file is
> corrupted ({})
> org.apache.cassandra.io.FSReadError: java.io.EOFException
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:284)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:254)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:91)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:259)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:242)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:220)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:199)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_65]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> Caused by: java.io.EOFException: null
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:146)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readPrimitiveSlowly(RebufferingInputStream.java:108)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readInt(RebufferingInputStream.java:188)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNextInternal(HintsReader.java:297)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:280)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 15 common frames omitted
> *Exception   in V 3.0.5  *
> ERROR [HintsDispatcher:2] 2016-04-26 15:54:46,294
> HintsDispatchExecutor.java:225 - Failed to dispatch hints file
> 8603be13-6878-4de3-8bc3-a7a7146b0376-1461657251205-1.hints: file is
> corrupted ({})
> org.apache.cassandra.io.FSReadError: java.io.EOFException
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:282)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:252)
> 

[jira] [Commented] (CASSANDRA-11678) cassandra crush when enable hints_compression

[Weijian Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265110#comment-15265110
 ] 

Weijian Lin commented on CASSANDRA-11678:
-

Every time I re-initialized all the cluster node and inserted data, some of the 
nodes crushed. I retried many times with V3.0.5 and V3.5.0. It came out the 
same result.  But after I comment out the hints_compression settings, it's ok. 




all the hints settings:

hinted_handoff_enabled: true
max_hint_window_in_ms: 1080 # 3 hours
hinted_handoff_throttle_in_kb: 1024
max_hints_delivery_threads: 2
hints_directory: /data1/cassandra/hints
hints_flush_period_in_ms: 1
max_hints_file_size_in_mb: 128
hints_compression:
   - class_name: LZ4Compressor
 parameters:
 -

> cassandra crush when enable hints_compression
> -
>
> Key: CASSANDRA-11678
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11678
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core, Local Write-Read Paths
> Environment: Centos 7
>Reporter: Weijian Lin
>Assignee: Blake Eggleston
>Priority: Critical
>
> When I enable hints_compression and set the compression class to
> LZ4Compressor，the
> cassandra (v3.05, V3.5.0) will crush。That is a bug, or any conf is wrong?
> *Exception   in V 3.5.0  *
> ERROR [HintsDispatcher:2] 2016-04-26 15:02:56,970
> HintsDispatchExecutor.java:225 - Failed to dispatch hints file
> abc4dda2-b551-427e-bb0b-e383d4a392e1-1461654138963-1.hints: file is
> corrupted ({})
> org.apache.cassandra.io.FSReadError: java.io.EOFException
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:284)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:254)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:91)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:259)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:242)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:220)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:199)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_65]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> Caused by: java.io.EOFException: null
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:146)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readPrimitiveSlowly(RebufferingInputStream.java:108)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readInt(RebufferingInputStream.java:188)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNextInternal(HintsReader.java:297)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:280)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 15 common frames omitted
> *Exception   in V 3.0.5  *
> ERROR [HintsDispatcher:2] 2016-04-26 15:54:46,294
> HintsDispatchExecutor.java:225 - Failed to dispatch hints file
> 8603be13-6878-4de3-8bc3-a7a7146b0376-1461657251205-1.hints: file is
> corrupted ({})
> org.apache.cassandra.io.FSReadError: java.io.EOFException
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:282)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:252)
> 

[jira] [Commented] (CASSANDRA-11683) Code refactor for StorageServiceMBean.forceRepair* methods

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265039#comment-15265039
 ] 

ASF GitHub Bot commented on CASSANDRA-11683:


Github user peoplehlj commented on the pull request:

https://github.com/apache/cassandra/pull/68#issuecomment-215919791
  
2.1 has reached end of development and only accepting critical bug fixes at 
this stage.


> Code refactor for StorageServiceMBean.forceRepair* methods
> --
>
> Key: CASSANDRA-11683
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11683
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Lijun Huang
>Priority: Trivial
> Fix For: 2.1.x
>
>
> For the class StorageServiceMBean, it has below methods, 
> {code:title=org/apache/cassandra/service/StorageServiceMBean.java|borderStyle=solid}
> public int forceRepairAsync(String keyspace, boolean isSequential, 
> Collection dataCenters, Collection hosts,  boolean 
> primaryRange, boolean repairedAt, String... columnFamilies) throws 
> IOException;
> public int forceRepairRangeAsync(String beginToken, String endToken, String 
> keyspaceName, boolean isSequential, Collection dataCenters, 
> Collection hosts, boolean repairedAt, String... columnFamilies) 
> throws IOException;
> public int forceRepairRangeAsync(String beginToken, String endToken, String 
> keyspaceName, boolean isSequential, boolean isLocal, boolean repairedAt, 
> String... columnFamilies);
> {code}
> But in the implementation, the arguments are different from this MBean, 
> please notice the last second argument, from *repairedAt* to *fullRepair*, 
> and actually *fullRepair* is more clear for this API.
> {code:title=org/apache/cassandra/service/StorageService.java|borderStyle=solid}
> public int forceRepairAsync(String keyspace, boolean isSequential, 
> Collection dataCenters, Collection hosts, boolean 
> primaryRange, boolean fullRepair, String... columnFamilies) throws 
> IOException{}
> public int forceRepairRangeAsync(String beginToken, String endToken, String 
> keyspaceName, boolean isSequential, Collection dataCenters, 
> Collection hosts, boolean fullRepair, String... columnFamilies) 
> throws IOException{}
> public int forceRepairRangeAsync(String beginToken, String endToken, String 
> keyspaceName, boolean isSequential, Collection dataCenters, 
> Collection hosts, boolean fullRepair, String... columnFamilies) 
> throws IOException{}
> {code}
> This will make developers confused, for example I met an issue that we didn't 
> want to do a "full" repair but from the MBean I didn't know how to pass the 
> argument.
> I will send out a patch soon, and please help to merge it if we want to fix 
> this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11683) Code refactor for StorageServiceMBean.forceRepair* methods

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265040#comment-15265040
 ] 

ASF GitHub Bot commented on CASSANDRA-11683:


Github user peoplehlj closed the pull request at:

https://github.com/apache/cassandra/pull/68


> Code refactor for StorageServiceMBean.forceRepair* methods
> --
>
> Key: CASSANDRA-11683
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11683
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Lijun Huang
>Priority: Trivial
> Fix For: 2.1.x
>
>
> For the class StorageServiceMBean, it has below methods, 
> {code:title=org/apache/cassandra/service/StorageServiceMBean.java|borderStyle=solid}
> public int forceRepairAsync(String keyspace, boolean isSequential, 
> Collection dataCenters, Collection hosts,  boolean 
> primaryRange, boolean repairedAt, String... columnFamilies) throws 
> IOException;
> public int forceRepairRangeAsync(String beginToken, String endToken, String 
> keyspaceName, boolean isSequential, Collection dataCenters, 
> Collection hosts, boolean repairedAt, String... columnFamilies) 
> throws IOException;
> public int forceRepairRangeAsync(String beginToken, String endToken, String 
> keyspaceName, boolean isSequential, boolean isLocal, boolean repairedAt, 
> String... columnFamilies);
> {code}
> But in the implementation, the arguments are different from this MBean, 
> please notice the last second argument, from *repairedAt* to *fullRepair*, 
> and actually *fullRepair* is more clear for this API.
> {code:title=org/apache/cassandra/service/StorageService.java|borderStyle=solid}
> public int forceRepairAsync(String keyspace, boolean isSequential, 
> Collection dataCenters, Collection hosts, boolean 
> primaryRange, boolean fullRepair, String... columnFamilies) throws 
> IOException{}
> public int forceRepairRangeAsync(String beginToken, String endToken, String 
> keyspaceName, boolean isSequential, Collection dataCenters, 
> Collection hosts, boolean fullRepair, String... columnFamilies) 
> throws IOException{}
> public int forceRepairRangeAsync(String beginToken, String endToken, String 
> keyspaceName, boolean isSequential, Collection dataCenters, 
> Collection hosts, boolean fullRepair, String... columnFamilies) 
> throws IOException{}
> {code}
> This will make developers confused, for example I met an issue that we didn't 
> want to do a "full" repair but from the MBean I didn't know how to pass the 
> argument.
> I will send out a patch soon, and please help to merge it if we want to fix 
> this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11683) Code refactor for StorageServiceMBean.forceRepair* methods

[Lijun Huang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265037#comment-15265037
 ] 

Lijun Huang commented on CASSANDRA-11683:
-

[~pauloricardomg], thanks, yes, it got fix on the more recent versions.

> Code refactor for StorageServiceMBean.forceRepair* methods
> --
>
> Key: CASSANDRA-11683
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11683
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Lijun Huang
>Priority: Trivial
> Fix For: 2.1.x
>
>
> For the class StorageServiceMBean, it has below methods, 
> {code:title=org/apache/cassandra/service/StorageServiceMBean.java|borderStyle=solid}
> public int forceRepairAsync(String keyspace, boolean isSequential, 
> Collection dataCenters, Collection hosts,  boolean 
> primaryRange, boolean repairedAt, String... columnFamilies) throws 
> IOException;
> public int forceRepairRangeAsync(String beginToken, String endToken, String 
> keyspaceName, boolean isSequential, Collection dataCenters, 
> Collection hosts, boolean repairedAt, String... columnFamilies) 
> throws IOException;
> public int forceRepairRangeAsync(String beginToken, String endToken, String 
> keyspaceName, boolean isSequential, boolean isLocal, boolean repairedAt, 
> String... columnFamilies);
> {code}
> But in the implementation, the arguments are different from this MBean, 
> please notice the last second argument, from *repairedAt* to *fullRepair*, 
> and actually *fullRepair* is more clear for this API.
> {code:title=org/apache/cassandra/service/StorageService.java|borderStyle=solid}
> public int forceRepairAsync(String keyspace, boolean isSequential, 
> Collection dataCenters, Collection hosts, boolean 
> primaryRange, boolean fullRepair, String... columnFamilies) throws 
> IOException{}
> public int forceRepairRangeAsync(String beginToken, String endToken, String 
> keyspaceName, boolean isSequential, Collection dataCenters, 
> Collection hosts, boolean fullRepair, String... columnFamilies) 
> throws IOException{}
> public int forceRepairRangeAsync(String beginToken, String endToken, String 
> keyspaceName, boolean isSequential, Collection dataCenters, 
> Collection hosts, boolean fullRepair, String... columnFamilies) 
> throws IOException{}
> {code}
> This will make developers confused, for example I met an issue that we didn't 
> want to do a "full" repair but from the MBean I didn't know how to pass the 
> argument.
> I will send out a patch soon, and please help to merge it if we want to fix 
> this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11483) Enhance sstablemetadata

[Yuki Morishita (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265034#comment-15265034
 ] 

Yuki Morishita commented on CASSANDRA-11483:


On windows, this runs without problems.
I fixed some styling and ran tests.

||branch||testall||dtest||
|[11483|https://github.com/yukim/cassandra/tree/11483]|[testall|http://cassci.datastax.com/view/Dev/view/yukim/job/yukim-11483-testall/lastCompletedBuild/testReport/]|[dtest|http://cassci.datastax.com/view/Dev/view/yukim/job/yukim-11483-dtest/lastCompletedBuild/testReport/]|

As you can see in dtest, some of them rely on the output of sstablemetadata, so 
we either fix those dtests as well, or support old output format.
Between those two, I prefer supporting old output format, since it can be least 
suprise for user who rely on output of the tool as well.
WDYT?

> Enhance sstablemetadata
> ---
>
> Key: CASSANDRA-11483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11483
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability
>Reporter: Chris Lohfink
>Assignee: Chris Lohfink
>Priority: Minor
> Fix For: 3.x
>
> Attachments: CASSANDRA-11483.txt, CASSANDRA-11483v2.txt, 
> CASSANDRA-11483v3.txt, CASSANDRA-11483v4.txt, Screen Shot 2016-04-03 at 
> 11.40.32 PM.png
>
>
> sstablemetadata provides quite a bit of useful information but theres a few 
> hiccups I would like to see addressed:
> * Does not use client mode
> * Units are not provided (or anything for that matter). There is data in 
> micros, millis, seconds as durations and timestamps from epoch. But there is 
> no way to tell what one is without a non-trival code dive
> * in general pretty frustrating to parse



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-11573) cqlsh fails with undefined symbol: PyUnicodeUCS2_DecodeUTF8

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler resolved CASSANDRA-11573.

   Resolution: Fixed
Fix Version/s: 3.6

Closing ticket, since root cause ticket has been committed for the upcoming 3.6 
release. Thanks for letting us know!

> cqlsh fails with undefined symbol: PyUnicodeUCS2_DecodeUTF8
> ---
>
> Key: CASSANDRA-11573
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11573
> Project: Cassandra
>  Issue Type: Bug
> Environment: centos 7, datastax ddc 3.5
> installed according to 
> http://docs.datastax.com/en/cassandra/3.x/cassandra/install/installRHEL.html
> JVM vendor/version: OpenJDK 64-Bit Server VM/1.8.0_77
> Cassandra version: 3.5.0
>Reporter: Oli Schacher
>Assignee: Michael Shuler
> Fix For: 3.6
>
>
> trying to run cqlsh produces:
> {quote}
> cqlsh
> Traceback (most recent call last):
>   File "/usr/bin/cqlsh.py", line 170, in 
> from cqlshlib.copyutil import ExportTask, ImportTask
> ImportError: /usr/lib/python2.7/site-packages/cqlshlib/copyutil.so: undefined 
> symbol: PyUnicodeUCS2_DecodeUTF8
> {quote}
> with 3.4 the error does not happen.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11694) sstabledump doesn't represent static columns correctly

[Joel Knighton (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264930#comment-15264930
 ] 

Joel Knighton commented on CASSANDRA-11694:
---

I don't think the negative timestamp output in "-d" is incorrect - this 
indicates the empty LivenessInfo on the static row, which I would prefer to see 
in debug output.

> sstabledump doesn't represent static columns correctly
> --
>
> Key: CASSANDRA-11694
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11694
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Wei Deng
>Assignee: Chris Lohfink
>
> It appears that the latest trunk code (after fixing CASSANDRA-11654, 
> CASSANDRA-11655 and CASSANDRA-11656) of sstabledump still doesn't handle 
> static columns correctly.
> Take a look at the following example:
> {noformat}
> root@node0:/mnt/ephemeral/cassandra/data/testks/test_static_column-ab5ce7c20b8411e695aeebb3bfdd5790#
>  ~/cassandra-trunk/tools/bin/sstabledump ma-1-big-Data.db -t
> [
>   {
> "partition" : {
>   "key" : [ "1" ],
>   "position" : 0
> },
> "rows" : [
>   {
> "type" : "static_block",
> "position" : 40,
> "cells" : [
>   { "name" : "static0_int", "value" : "3000", "tstamp" : 
> "1461657675565767" },
>   { "name" : "static1_int", "value" : "4000", "tstamp" : 
> "1461657675565767" }
> ]
>   },
>   {
> "type" : "row",
> "position" : 40,
> "clustering" : [ "c1" ],
> "liveness_info" : { "tstamp" : "1461657663393419" },
> "cells" : [
>   { "name" : "val0_int", "value" : "100" },
>   { "name" : "val1_set_of_int", "deletion_info" : { "marked_deleted" 
> : "1461657663393418", "local_delete_time" : "1461657663" } },
>   { "name" : "val1_set_of_int", "path" : [ "1" ], "value" : "" },
>   { "name" : "val1_set_of_int", "path" : [ "2" ], "value" : "" },
>   { "name" : "val1_set_of_int", "path" : [ "3" ], "value" : "" },
>   { "name" : "val1_set_of_int", "path" : [ "4" ], "value" : "" },
>   { "name" : "val1_set_of_int", "path" : [ "5" ], "value" : "" }
> ]
>   },
>   {
> "type" : "row",
> "position" : 92,
> "clustering" : [ "c2" ],
> "liveness_info" : { "tstamp" : "1461657675565767" },
> "cells" : [
>   { "name" : "val0_int", "value" : "200" },
>   { "name" : "val1_set_of_int", "deletion_info" : { "marked_deleted" 
> : "1461657675565766", "local_delete_time" : "1461657675" } },
>   { "name" : "val1_set_of_int", "path" : [ "1" ], "value" : "" },
>   { "name" : "val1_set_of_int", "path" : [ "2" ], "value" : "" },
>   { "name" : "val1_set_of_int", "path" : [ "3" ], "value" : "" },
>   { "name" : "val1_set_of_int", "path" : [ "4" ], "value" : "" },
>   { "name" : "val1_set_of_int", "path" : [ "5" ], "value" : "" }
> ]
>   },
>   {
> "type" : "row",
> "position" : 144,
> "clustering" : [ "c3" ],
> "liveness_info" : { "tstamp" : "1461657634639043" },
> "cells" : [
>   { "name" : "val0_int", "value" : "300" },
>   { "name" : "val1_set_of_int", "deletion_info" : { "marked_deleted" 
> : "1461657634639042", "local_delete_time" : "1461657634" } },
>   { "name" : "val1_set_of_int", "path" : [ "1" ], "value" : "" },
>   { "name" : "val1_set_of_int", "path" : [ "2" ], "value" : "" },
>   { "name" : "val1_set_of_int", "path" : [ "3" ], "value" : "" },
>   { "name" : "val1_set_of_int", "path" : [ "4" ], "value" : "" },
>   { "name" : "val1_set_of_int", "path" : [ "5" ], "value" : "" }
> ]
>   }
> ]
>   }
> ]
> {noformat}
> Note the "position" for the "static_block" and the first "row" are the same 
> (40), which should be incorrect.
> If you print out in debug mode, you will see the following:
> {noformat}
> root@node0:/mnt/ephemeral/cassandra/data/testks/test_static_column-ab5ce7c20b8411e695aeebb3bfdd5790#
>  ~/cassandra-trunk/tools/bin/sstabledump ma-1-big-Data.db -t -d
> [1]@0 Row[info=[ts=-9223372036854775808] ]: STATIC | [static0_int=3000 
> ts=1461657675565767], [static1_int=4000 ts=1461657675565767]
> [1]@0 Row[info=[ts=1461657663393419] ]: c1 | [val0_int=100 
> ts=1461657663393419], del(val1_set_of_int)=deletedAt=1461657663393418, 
> localDeletion=1461657663, [val1_set_of_int[1]= ts=1461657663393419], 
> [val1_set_of_int[2]= ts=1461657663393419], [val1_set_of_int[3]= 
> ts=1461657663393419], [val1_set_of_int[4]= ts=1461657663393419], 
> [val1_set_of_int[5]= ts=1461657663393419]
> [1]@92 Row[info=[ts=1461657675565767] ]: c2 | [val0_int=200 
> ts=1461657675565767], 

[jira] [Updated] (CASSANDRA-11694) sstabledump doesn't represent static columns correctly

[Wei Deng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei Deng updated CASSANDRA-11694:
-
Assignee: Chris Lohfink

> sstabledump doesn't represent static columns correctly
> --
>
> Key: CASSANDRA-11694
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11694
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Wei Deng
>Assignee: Chris Lohfink
>
> It appears that the latest trunk code (after fixing CASSANDRA-11654, 
> CASSANDRA-11655 and CASSANDRA-11656) of sstabledump still doesn't handle 
> static columns correctly.
> Take a look at the following example:
> {noformat}
> root@node0:/mnt/ephemeral/cassandra/data/testks/test_static_column-ab5ce7c20b8411e695aeebb3bfdd5790#
>  ~/cassandra-trunk/tools/bin/sstabledump ma-1-big-Data.db -t
> [
>   {
> "partition" : {
>   "key" : [ "1" ],
>   "position" : 0
> },
> "rows" : [
>   {
> "type" : "static_block",
> "position" : 40,
> "cells" : [
>   { "name" : "static0_int", "value" : "3000", "tstamp" : 
> "1461657675565767" },
>   { "name" : "static1_int", "value" : "4000", "tstamp" : 
> "1461657675565767" }
> ]
>   },
>   {
> "type" : "row",
> "position" : 40,
> "clustering" : [ "c1" ],
> "liveness_info" : { "tstamp" : "1461657663393419" },
> "cells" : [
>   { "name" : "val0_int", "value" : "100" },
>   { "name" : "val1_set_of_int", "deletion_info" : { "marked_deleted" 
> : "1461657663393418", "local_delete_time" : "1461657663" } },
>   { "name" : "val1_set_of_int", "path" : [ "1" ], "value" : "" },
>   { "name" : "val1_set_of_int", "path" : [ "2" ], "value" : "" },
>   { "name" : "val1_set_of_int", "path" : [ "3" ], "value" : "" },
>   { "name" : "val1_set_of_int", "path" : [ "4" ], "value" : "" },
>   { "name" : "val1_set_of_int", "path" : [ "5" ], "value" : "" }
> ]
>   },
>   {
> "type" : "row",
> "position" : 92,
> "clustering" : [ "c2" ],
> "liveness_info" : { "tstamp" : "1461657675565767" },
> "cells" : [
>   { "name" : "val0_int", "value" : "200" },
>   { "name" : "val1_set_of_int", "deletion_info" : { "marked_deleted" 
> : "1461657675565766", "local_delete_time" : "1461657675" } },
>   { "name" : "val1_set_of_int", "path" : [ "1" ], "value" : "" },
>   { "name" : "val1_set_of_int", "path" : [ "2" ], "value" : "" },
>   { "name" : "val1_set_of_int", "path" : [ "3" ], "value" : "" },
>   { "name" : "val1_set_of_int", "path" : [ "4" ], "value" : "" },
>   { "name" : "val1_set_of_int", "path" : [ "5" ], "value" : "" }
> ]
>   },
>   {
> "type" : "row",
> "position" : 144,
> "clustering" : [ "c3" ],
> "liveness_info" : { "tstamp" : "1461657634639043" },
> "cells" : [
>   { "name" : "val0_int", "value" : "300" },
>   { "name" : "val1_set_of_int", "deletion_info" : { "marked_deleted" 
> : "1461657634639042", "local_delete_time" : "1461657634" } },
>   { "name" : "val1_set_of_int", "path" : [ "1" ], "value" : "" },
>   { "name" : "val1_set_of_int", "path" : [ "2" ], "value" : "" },
>   { "name" : "val1_set_of_int", "path" : [ "3" ], "value" : "" },
>   { "name" : "val1_set_of_int", "path" : [ "4" ], "value" : "" },
>   { "name" : "val1_set_of_int", "path" : [ "5" ], "value" : "" }
> ]
>   }
> ]
>   }
> ]
> {noformat}
> Note the "position" for the "static_block" and the first "row" are the same 
> (40), which should be incorrect.
> If you print out in debug mode, you will see the following:
> {noformat}
> root@node0:/mnt/ephemeral/cassandra/data/testks/test_static_column-ab5ce7c20b8411e695aeebb3bfdd5790#
>  ~/cassandra-trunk/tools/bin/sstabledump ma-1-big-Data.db -t -d
> [1]@0 Row[info=[ts=-9223372036854775808] ]: STATIC | [static0_int=3000 
> ts=1461657675565767], [static1_int=4000 ts=1461657675565767]
> [1]@0 Row[info=[ts=1461657663393419] ]: c1 | [val0_int=100 
> ts=1461657663393419], del(val1_set_of_int)=deletedAt=1461657663393418, 
> localDeletion=1461657663, [val1_set_of_int[1]= ts=1461657663393419], 
> [val1_set_of_int[2]= ts=1461657663393419], [val1_set_of_int[3]= 
> ts=1461657663393419], [val1_set_of_int[4]= ts=1461657663393419], 
> [val1_set_of_int[5]= ts=1461657663393419]
> [1]@92 Row[info=[ts=1461657675565767] ]: c2 | [val0_int=200 
> ts=1461657675565767], del(val1_set_of_int)=deletedAt=1461657675565766, 
> localDeletion=1461657675, [val1_set_of_int[1]= ts=1461657675565767], 
> [val1_set_of_int[2]= ts=1461657675565767], [val1_set_of_int[3]= 
> ts=1461657675565767], 

[jira] [Created] (CASSANDRA-11694) sstabledump doesn't represent static columns correctly

[Wei Deng (JIRA)]

Wei Deng created CASSANDRA-11694:


 Summary: sstabledump doesn't represent static columns correctly
 Key: CASSANDRA-11694
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11694
 Project: Cassandra
  Issue Type: Bug
  Components: Tools
Reporter: Wei Deng


It appears that the latest trunk code (after fixing CASSANDRA-11654, 
CASSANDRA-11655 and CASSANDRA-11656) of sstabledump still doesn't handle static 
columns correctly.

Take a look at the following example:

{noformat}
root@node0:/mnt/ephemeral/cassandra/data/testks/test_static_column-ab5ce7c20b8411e695aeebb3bfdd5790#
 ~/cassandra-trunk/tools/bin/sstabledump ma-1-big-Data.db -t
[
  {
"partition" : {
  "key" : [ "1" ],
  "position" : 0
},
"rows" : [
  {
"type" : "static_block",
"position" : 40,
"cells" : [
  { "name" : "static0_int", "value" : "3000", "tstamp" : 
"1461657675565767" },
  { "name" : "static1_int", "value" : "4000", "tstamp" : 
"1461657675565767" }
]
  },
  {
"type" : "row",
"position" : 40,
"clustering" : [ "c1" ],
"liveness_info" : { "tstamp" : "1461657663393419" },
"cells" : [
  { "name" : "val0_int", "value" : "100" },
  { "name" : "val1_set_of_int", "deletion_info" : { "marked_deleted" : 
"1461657663393418", "local_delete_time" : "1461657663" } },
  { "name" : "val1_set_of_int", "path" : [ "1" ], "value" : "" },
  { "name" : "val1_set_of_int", "path" : [ "2" ], "value" : "" },
  { "name" : "val1_set_of_int", "path" : [ "3" ], "value" : "" },
  { "name" : "val1_set_of_int", "path" : [ "4" ], "value" : "" },
  { "name" : "val1_set_of_int", "path" : [ "5" ], "value" : "" }
]
  },
  {
"type" : "row",
"position" : 92,
"clustering" : [ "c2" ],
"liveness_info" : { "tstamp" : "1461657675565767" },
"cells" : [
  { "name" : "val0_int", "value" : "200" },
  { "name" : "val1_set_of_int", "deletion_info" : { "marked_deleted" : 
"1461657675565766", "local_delete_time" : "1461657675" } },
  { "name" : "val1_set_of_int", "path" : [ "1" ], "value" : "" },
  { "name" : "val1_set_of_int", "path" : [ "2" ], "value" : "" },
  { "name" : "val1_set_of_int", "path" : [ "3" ], "value" : "" },
  { "name" : "val1_set_of_int", "path" : [ "4" ], "value" : "" },
  { "name" : "val1_set_of_int", "path" : [ "5" ], "value" : "" }
]
  },
  {
"type" : "row",
"position" : 144,
"clustering" : [ "c3" ],
"liveness_info" : { "tstamp" : "1461657634639043" },
"cells" : [
  { "name" : "val0_int", "value" : "300" },
  { "name" : "val1_set_of_int", "deletion_info" : { "marked_deleted" : 
"1461657634639042", "local_delete_time" : "1461657634" } },
  { "name" : "val1_set_of_int", "path" : [ "1" ], "value" : "" },
  { "name" : "val1_set_of_int", "path" : [ "2" ], "value" : "" },
  { "name" : "val1_set_of_int", "path" : [ "3" ], "value" : "" },
  { "name" : "val1_set_of_int", "path" : [ "4" ], "value" : "" },
  { "name" : "val1_set_of_int", "path" : [ "5" ], "value" : "" }
]
  }
]
  }
]
{noformat}

Note the "position" for the "static_block" and the first "row" are the same 
(40), which should be incorrect.

If you print out in debug mode, you will see the following:

{noformat}
root@node0:/mnt/ephemeral/cassandra/data/testks/test_static_column-ab5ce7c20b8411e695aeebb3bfdd5790#
 ~/cassandra-trunk/tools/bin/sstabledump ma-1-big-Data.db -t -d
[1]@0 Row[info=[ts=-9223372036854775808] ]: STATIC | [static0_int=3000 
ts=1461657675565767], [static1_int=4000 ts=1461657675565767]
[1]@0 Row[info=[ts=1461657663393419] ]: c1 | [val0_int=100 
ts=1461657663393419], del(val1_set_of_int)=deletedAt=1461657663393418, 
localDeletion=1461657663, [val1_set_of_int[1]= ts=1461657663393419], 
[val1_set_of_int[2]= ts=1461657663393419], [val1_set_of_int[3]= 
ts=1461657663393419], [val1_set_of_int[4]= ts=1461657663393419], 
[val1_set_of_int[5]= ts=1461657663393419]
[1]@92 Row[info=[ts=1461657675565767] ]: c2 | [val0_int=200 
ts=1461657675565767], del(val1_set_of_int)=deletedAt=1461657675565766, 
localDeletion=1461657675, [val1_set_of_int[1]= ts=1461657675565767], 
[val1_set_of_int[2]= ts=1461657675565767], [val1_set_of_int[3]= 
ts=1461657675565767], [val1_set_of_int[4]= ts=1461657675565767], 
[val1_set_of_int[5]= ts=1461657675565767]
[1]@144 Row[info=[ts=1461657634639043] ]: c3 | [val0_int=300 
ts=1461657634639043], del(val1_set_of_int)=deletedAt=1461657634639042, 
localDeletion=1461657634, [val1_set_of_int[1]= ts=1461657634639043], 
[val1_set_of_int[2]= ts=1461657634639043], [val1_set_of_int[3]= 
ts=1461657634639043], [val1_set_of_int[4]= ts=1461657634639043], 

[jira] [Created] (CASSANDRA-11693) dtest failure in scrub_test.TestScrubIndexes.test_scrub_static_table

[Russ Hatch (JIRA)]

Russ Hatch created CASSANDRA-11693:
--

 Summary: dtest failure in 
scrub_test.TestScrubIndexes.test_scrub_static_table
 Key: CASSANDRA-11693
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11693
 Project: Cassandra
  Issue Type: Test
Reporter: Russ Hatch
Assignee: DS Test Eng


intermittent failure: 
http://cassci.datastax.com/job/trunk_offheap_dtest/166/testReport/scrub_test/TestScrubIndexes/test_scrub_static_table/

looks distinct from another reported failure on this test for windows 
(CASSANDRA-11284)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11692) dtest failure in sstableutil_test.SSTableUtilTest.abortedcompaction_test

[Russ Hatch (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264875#comment-15264875
 ] 

Russ Hatch commented on CASSANDRA-11692:


/cc [~Stefania]

> dtest failure in sstableutil_test.SSTableUtilTest.abortedcompaction_test
> 
>
> Key: CASSANDRA-11692
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11692
> Project: Cassandra
>  Issue Type: Test
>Reporter: Russ Hatch
>Assignee: DS Test Eng
>  Labels: dtest
>
> example failure:
> {noformat}
> Lists differ: ['/mnt/tmp/dtest-hXZ_VA/test/n... != 
> ['/mnt/tmp/dtest-hXZ_VA/test/n...
> First differing element 16:
> /mnt/tmp/dtest-hXZ_VA/test/node1/data2/keyspace1/standard1-483ee2700d5911e6b19a879d803a6aae/ma-3-big-CRC.db
> /mnt/tmp/dtest-hXZ_VA/test/node1/data2/keyspace1/standard1-483ee2700d5911e6b19a879d803a6aae/ma-5-big-CRC.db
> Diff is 5376 characters long. Set self.maxDiff to None to see it.
> {noformat}
> http://cassci.datastax.com/job/trunk_novnode_dtest/360/testReport/sstableutil_test/SSTableUtilTest/abortedcompaction_test
> Failed on CassCI build trunk_novnode_dtest #360



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11692) dtest failure in sstableutil_test.SSTableUtilTest.abortedcompaction_test

[Russ Hatch (JIRA)]

Russ Hatch created CASSANDRA-11692:
--

 Summary: dtest failure in 
sstableutil_test.SSTableUtilTest.abortedcompaction_test
 Key: CASSANDRA-11692
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11692
 Project: Cassandra
  Issue Type: Test
Reporter: Russ Hatch
Assignee: DS Test Eng


example failure:

{noformat}
Lists differ: ['/mnt/tmp/dtest-hXZ_VA/test/n... != 
['/mnt/tmp/dtest-hXZ_VA/test/n...

First differing element 16:
/mnt/tmp/dtest-hXZ_VA/test/node1/data2/keyspace1/standard1-483ee2700d5911e6b19a879d803a6aae/ma-3-big-CRC.db
/mnt/tmp/dtest-hXZ_VA/test/node1/data2/keyspace1/standard1-483ee2700d5911e6b19a879d803a6aae/ma-5-big-CRC.db

Diff is 5376 characters long. Set self.maxDiff to None to see it.
{noformat}

http://cassci.datastax.com/job/trunk_novnode_dtest/360/testReport/sstableutil_test/SSTableUtilTest/abortedcompaction_test

Failed on CassCI build trunk_novnode_dtest #360



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11691) (windows) dtest failure in replace_address_test.TestReplaceAddress.fail_without_replace_test

[Russ Hatch (JIRA)]

Russ Hatch created CASSANDRA-11691:
--

 Summary: (windows) dtest failure in 
replace_address_test.TestReplaceAddress.fail_without_replace_test
 Key: CASSANDRA-11691
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11691
 Project: Cassandra
  Issue Type: Test
Reporter: Russ Hatch
Assignee: DS Test Eng


looks to be a new test, first run failing with:

{noformat}
[Error 5] Access is denied: 
'd:\\temp\\dtest-kof1fj\\test\\node3\\commitlogs\\CommitLog-6-1461863368272.log'
{noformat}

maybe a windows python compat issue?

http://cassci.datastax.com/job/trunk_dtest_win32/401/testReport/replace_address_test/TestReplaceAddress/fail_without_replace_test

Failed on CassCI build trunk_dtest_win32 #401



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11686) dtest failure in replication_test.SnitchConfigurationUpdateTest.test_rf_expand_gossiping_property_file_snitch_multi_dc

[Russ Hatch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russ Hatch updated CASSANDRA-11686:
---
Assignee: DS Test Eng

> dtest failure in 
> replication_test.SnitchConfigurationUpdateTest.test_rf_expand_gossiping_property_file_snitch_multi_dc
> --
>
> Key: CASSANDRA-11686
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11686
> Project: Cassandra
>  Issue Type: Test
>Reporter: Russ Hatch
>Assignee: DS Test Eng
>  Labels: dtest
>
> intermittent failure. this test also fails on windows but looks to be for 
> another reason (CASSANDRA-11439)
> http://cassci.datastax.com/job/cassandra-3.0_dtest/682/testReport/replication_test/SnitchConfigurationUpdateTest/test_rf_expand_gossiping_property_file_snitch_multi_dc/
> {noformat}
> Nodetool command '/home/automaton/cassandra/bin/nodetool -h localhost -p 7400 
> getendpoints testing rf_test dummy' failed; exit status: 1; stderr: nodetool: 
> Failed to connect to 'localhost:7400' - ConnectException: 'Connection 
> refused'.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11542) Create a benchmark to compare HDFS and Cassandra bulk read times

[vincent.poncet (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264867#comment-15264867
 ] 

vincent.poncet commented on CASSANDRA-11542:


Cassandra tests have a way higher normalized std deviation (std dev / mean) 
than hdfs tests. and looking in you previous tests, first run of cassandra is 
way way higher than next runs. So, it really looks like there is a cache 
somewhere which is biaising your cassandra numbers. 

> Create a benchmark to compare HDFS and Cassandra bulk read times
> 
>
> Key: CASSANDRA-11542
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11542
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Testing
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 3.x
>
> Attachments: spark-load-perf-results-001.zip, 
> spark-load-perf-results-002.zip
>
>
> I propose creating a benchmark for comparing Cassandra and HDFS bulk reading 
> performance. Simple Spark queries will be performed on data stored in HDFS or 
> Cassandra, and the entire duration will be measured. An example query would 
> be the max or min of a column or a count\(*\).
> This benchmark should allow determining the impact of:
> * partition size
> * number of clustering columns
> * number of value columns (cells)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11690) dtest faile in test_rf_collapse_gossiping_property_file_snitch_multi_dc

[Russ Hatch (JIRA)]

Russ Hatch created CASSANDRA-11690:
--

 Summary: dtest faile in 
test_rf_collapse_gossiping_property_file_snitch_multi_dc
 Key: CASSANDRA-11690
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11690
 Project: Cassandra
  Issue Type: Test
Reporter: Russ Hatch
Assignee: DS Test Eng


looks like a possible resource constraint issue:
{noformat}
[Errno 12] Cannot allocate memory
{noformat}

more than one failure in recent history.

http://cassci.datastax.com/job/trunk_dtest/1173/testReport/replication_test/SnitchConfigurationUpdateTest/test_rf_collapse_gossiping_property_file_snitch_multi_dc/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11689) dtest failures in internode_ssl_test tests

[Russ Hatch (JIRA)]

Russ Hatch created CASSANDRA-11689:
--

 Summary: dtest failures in internode_ssl_test tests
 Key: CASSANDRA-11689
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11689
 Project: Cassandra
  Issue Type: Test
Reporter: Russ Hatch
Assignee: DS Test Eng


has happened a few times on trunk, two different tests:

http://cassci.datastax.com/job/trunk_dtest/1179/testReport/internode_ssl_test/TestInternodeSSL/putget_with_internode_ssl_without_compression_test

http://cassci.datastax.com/job/trunk_dtest/1169/testReport/internode_ssl_test/TestInternodeSSL/putget_with_internode_ssl_test/

Failed on CassCI build trunk_dtest #1179



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11626) cqlsh fails and exits on non-ascii chars

[Tyler Hobbs (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tyler Hobbs updated CASSANDRA-11626:

Reproduced In: 3.5, 3.0.4  (was: 3.0.4, 3.5)
   Status: Open  (was: Patch Available)

Hmm, looks like there are some dtest failures, let me fix those up.

> cqlsh fails and exits on non-ascii chars
> 
>
> Key: CASSANDRA-11626
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11626
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Robert Stupp
>Assignee: Tyler Hobbs
>Priority: Minor
>  Labels: cqlsh
> Fix For: 2.2.x, 3.0.x, 3.x
>
>
> Just seen on cqlsh on current trunk:
> To repro, copy {{ä}} (german umlaut) to cqlsh and press return.
> cqlsh errors out and immediately exits.
> {code}
> $ bin/cqlsh
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.13-SNAPSHOT | CQL spec 3.2.1 | Native protocol 
> v3]
> Use HELP for help.
> cqlsh> ä
> Invalid syntax at line 1, char 1
> Traceback (most recent call last):
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2636, in 
> 
> main(*read_options(sys.argv[1:], os.environ))
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2625, in main
> shell.cmdloop()
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 1114, in 
> cmdloop
> if self.onecmd(self.statement.getvalue()):
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 1139, in onecmd
> self.printerr('  %s' % statementline)
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2314, in 
> printerr
> self.writeresult(text, color, newline=newline, out=sys.stderr)
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2303, in 
> writeresult
> out.write(self.applycolor(str(text), color) + ('\n' if newline else ''))
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 
> 2: ordinal not in range(128)
> $ 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8844) Change Data Capture (CDC)

[DOAN DuyHai (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264753#comment-15264753
 ] 

DOAN DuyHai commented on CASSANDRA-8844:


> I don't see reject all being viable since it'll shut down your cluster, and 
> reject none on a slow consumer scenario would just lead to disk space 
> exhaustion and lost CDC data.

 Didn't we agreed in the past on the idea of having a parameter in Yaml that 
works similar to disk_failure_policy and let users decide which behavior they 
want on CDC overflow (reject writes/stop creating CDC/ ) ?

> Change Data Capture (CDC)
> -
>
> Key: CASSANDRA-8844
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8844
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Coordination, Local Write-Read Paths
>Reporter: Tupshin Harper
>Assignee: Joshua McKenzie
>Priority: Critical
> Fix For: 3.x
>
>
> "In databases, change data capture (CDC) is a set of software design patterns 
> used to determine (and track) the data that has changed so that action can be 
> taken using the changed data. Also, Change data capture (CDC) is an approach 
> to data integration that is based on the identification, capture and delivery 
> of the changes made to enterprise data sources."
> -Wikipedia
> As Cassandra is increasingly being used as the Source of Record (SoR) for 
> mission critical data in large enterprises, it is increasingly being called 
> upon to act as the central hub of traffic and data flow to other systems. In 
> order to try to address the general need, we (cc [~brianmhess]), propose 
> implementing a simple data logging mechanism to enable per-table CDC patterns.
> h2. The goals:
> # Use CQL as the primary ingestion mechanism, in order to leverage its 
> Consistency Level semantics, and in order to treat it as the single 
> reliable/durable SoR for the data.
> # To provide a mechanism for implementing good and reliable 
> (deliver-at-least-once with possible mechanisms for deliver-exactly-once ) 
> continuous semi-realtime feeds of mutations going into a Cassandra cluster.
> # To eliminate the developmental and operational burden of users so that they 
> don't have to do dual writes to other systems.
> # For users that are currently doing batch export from a Cassandra system, 
> give them the opportunity to make that realtime with a minimum of coding.
> h2. The mechanism:
> We propose a durable logging mechanism that functions similar to a commitlog, 
> with the following nuances:
> - Takes place on every node, not just the coordinator, so RF number of copies 
> are logged.
> - Separate log per table.
> - Per-table configuration. Only tables that are specified as CDC_LOG would do 
> any logging.
> - Per DC. We are trying to keep the complexity to a minimum to make this an 
> easy enhancement, but most likely use cases would prefer to only implement 
> CDC logging in one (or a subset) of the DCs that are being replicated to
> - In the critical path of ConsistencyLevel acknowledgment. Just as with the 
> commitlog, failure to write to the CDC log should fail that node's write. If 
> that means the requested consistency level was not met, then clients *should* 
> experience UnavailableExceptions.
> - Be written in a Row-centric manner such that it is easy for consumers to 
> reconstitute rows atomically.
> - Written in a simple format designed to be consumed *directly* by daemons 
> written in non JVM languages
> h2. Nice-to-haves
> I strongly suspect that the following features will be asked for, but I also 
> believe that they can be deferred for a subsequent release, and to guage 
> actual interest.
> - Multiple logs per table. This would make it easy to have multiple 
> "subscribers" to a single table's changes. A workaround would be to create a 
> forking daemon listener, but that's not a great answer.
> - Log filtering. Being able to apply filters, including UDF-based filters 
> would make Casandra a much more versatile feeder into other systems, and 
> again, reduce complexity that would otherwise need to be built into the 
> daemons.
> h2. Format and Consumption
> - Cassandra would only write to the CDC log, and never delete from it. 
> - Cleaning up consumed logfiles would be the client daemon's responibility
> - Logfile size should probably be configurable.
> - Logfiles should be named with a predictable naming schema, making it 
> triivial to process them in order.
> - Daemons should be able to checkpoint their work, and resume from where they 
> left off. This means they would have to leave some file artifact in the CDC 
> log's directory.
> - A sophisticated daemon should be able to be written that could 
> -- Catch up, in written-order, even when it is multiple logfiles behind in 
> processing
> -- Be able to 

[jira] [Comment Edited] (CASSANDRA-8844) Change Data Capture (CDC)

[DOAN DuyHai (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264753#comment-15264753
 ] 

DOAN DuyHai edited comment on CASSANDRA-8844 at 4/29/16 9:11 PM:
-

bq. I don't see reject all being viable since it'll shut down your cluster, and 
reject none on a slow consumer scenario would just lead to disk space 
exhaustion and lost CDC data.

 Didn't we agree in the past on the idea of having a parameter in Yaml that 
works similar to disk_failure_policy and let users decide which behavior they 
want on CDC overflow (reject writes/stop creating CDC/ ) ?


was (Author: doanduyhai):
> I don't see reject all being viable since it'll shut down your cluster, and 
> reject none on a slow consumer scenario would just lead to disk space 
> exhaustion and lost CDC data.

 Didn't we agreed in the past on the idea of having a parameter in Yaml that 
works similar to disk_failure_policy and let users decide which behavior they 
want on CDC overflow (reject writes/stop creating CDC/ ) ?

> Change Data Capture (CDC)
> -
>
> Key: CASSANDRA-8844
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8844
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Coordination, Local Write-Read Paths
>Reporter: Tupshin Harper
>Assignee: Joshua McKenzie
>Priority: Critical
> Fix For: 3.x
>
>
> "In databases, change data capture (CDC) is a set of software design patterns 
> used to determine (and track) the data that has changed so that action can be 
> taken using the changed data. Also, Change data capture (CDC) is an approach 
> to data integration that is based on the identification, capture and delivery 
> of the changes made to enterprise data sources."
> -Wikipedia
> As Cassandra is increasingly being used as the Source of Record (SoR) for 
> mission critical data in large enterprises, it is increasingly being called 
> upon to act as the central hub of traffic and data flow to other systems. In 
> order to try to address the general need, we (cc [~brianmhess]), propose 
> implementing a simple data logging mechanism to enable per-table CDC patterns.
> h2. The goals:
> # Use CQL as the primary ingestion mechanism, in order to leverage its 
> Consistency Level semantics, and in order to treat it as the single 
> reliable/durable SoR for the data.
> # To provide a mechanism for implementing good and reliable 
> (deliver-at-least-once with possible mechanisms for deliver-exactly-once ) 
> continuous semi-realtime feeds of mutations going into a Cassandra cluster.
> # To eliminate the developmental and operational burden of users so that they 
> don't have to do dual writes to other systems.
> # For users that are currently doing batch export from a Cassandra system, 
> give them the opportunity to make that realtime with a minimum of coding.
> h2. The mechanism:
> We propose a durable logging mechanism that functions similar to a commitlog, 
> with the following nuances:
> - Takes place on every node, not just the coordinator, so RF number of copies 
> are logged.
> - Separate log per table.
> - Per-table configuration. Only tables that are specified as CDC_LOG would do 
> any logging.
> - Per DC. We are trying to keep the complexity to a minimum to make this an 
> easy enhancement, but most likely use cases would prefer to only implement 
> CDC logging in one (or a subset) of the DCs that are being replicated to
> - In the critical path of ConsistencyLevel acknowledgment. Just as with the 
> commitlog, failure to write to the CDC log should fail that node's write. If 
> that means the requested consistency level was not met, then clients *should* 
> experience UnavailableExceptions.
> - Be written in a Row-centric manner such that it is easy for consumers to 
> reconstitute rows atomically.
> - Written in a simple format designed to be consumed *directly* by daemons 
> written in non JVM languages
> h2. Nice-to-haves
> I strongly suspect that the following features will be asked for, but I also 
> believe that they can be deferred for a subsequent release, and to guage 
> actual interest.
> - Multiple logs per table. This would make it easy to have multiple 
> "subscribers" to a single table's changes. A workaround would be to create a 
> forking daemon listener, but that's not a great answer.
> - Log filtering. Being able to apply filters, including UDF-based filters 
> would make Casandra a much more versatile feeder into other systems, and 
> again, reduce complexity that would otherwise need to be built into the 
> daemons.
> h2. Format and Consumption
> - Cassandra would only write to the CDC log, and never delete from it. 
> - Cleaning up consumed logfiles would be the client daemon's responibility
> - Logfile size should probably be 

[jira] [Comment Edited] (CASSANDRA-11542) Create a benchmark to compare HDFS and Cassandra bulk read times

[Russell Alexander Spitzer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264698#comment-15264698
 ] 

Russell Alexander Spitzer edited comment on CASSANDRA-11542 at 4/29/16 9:04 PM:


-Hmm i'm a little confused that case classes don't help but Dataframes do...-  
https://datastax-oss.atlassian.net/browse/SPARKC-373 Saw that we are always 
doing the conversion to CassandraRow with RDDs, dataframes go directly to the 
internal SQL Type. 

The code you presented looks good to me, there is the potential issue of 
blocking on resultsets that take a long time to complete while other 
result-sets are already on the driver but i'm not sure if this is a big deal. 
Do you have any idea of the parallelization in these test? How many partitions 
are the different runs generating?




was (Author: rspitzer):
Hmm i'm a little confused that case classes don't help but Dataframes do... The 
code you presented looks good to me, there is the potential issue of blocking 
on resultsets that take a long time to complete while other result-sets are 
already on the driver but i'm not sure if this is a big deal. Do you have any 
idea of the parallelization in these test? How many partitions are the 
different runs generating?

> Create a benchmark to compare HDFS and Cassandra bulk read times
> 
>
> Key: CASSANDRA-11542
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11542
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Testing
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 3.x
>
> Attachments: spark-load-perf-results-001.zip, 
> spark-load-perf-results-002.zip
>
>
> I propose creating a benchmark for comparing Cassandra and HDFS bulk reading 
> performance. Simple Spark queries will be performed on data stored in HDFS or 
> Cassandra, and the entire duration will be measured. An example query would 
> be the max or min of a column or a count\(*\).
> This benchmark should allow determining the impact of:
> * partition size
> * number of clustering columns
> * number of value columns (cells)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11688) Replace_address should sanity check prior node state before migrating tokens

[Jonathan Shook (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Shook updated CASSANDRA-11688:
---
Description: 
During a node replacement, a replace_address was used which was associated with 
a different node than the intended one. The result was that both nodes remained 
active after the node came up. This caused several other issues which were 
difficult to diagnose, including invalid gossip state, etc.

Replace_address should be more robust in this scenario. It would be much more 
user friendly if the replace_address logic would first do some basic sanity 
checks, possibly to include:

- Pinging the other node to see if it is indeed “down”, if the address is 
different than all local interface addresses
- Checking gossip state of the node to verify that it is not known to peers.

It may even be safest to require that both address reachability and gossip 
state are required to show the replace_address as down by default before 
allowing any token migration or other replace_address actions to occur.

In the case that the replace_address is not ready to be replaced, the log 
should indicate that you are trying to replace an active node, and cassandra 
should refuse to start.

  was:
During a node replacement, a customer used an ip address associated with a 
different node than the intended one. The result was that both nodes remained 
active after the node came up. This caused several other issues which were 
difficult to diagnose, including invalid gossip state, etc.

Replace_address should be more robust in this scenario. It would be much more 
user friendly if the replace_address logic would first do some basic sanity 
checks, possibly to include:

- Pinging the other node to see if it is indeed “down”, if the address is 
different than all local interface addresses
- Checking gossip state of the node to verify that it is not known to peers.

It may even be safest to require that both address reachability and gossip 
state are required to show the replace_address as down by default before 
allowing any token migration or other replace_address actions to occur.

In the case that the replace_address is not ready to be replaced, the log 
should indicate that you are trying to replace an active node, and cassandra 
should refuse to start.


> Replace_address should sanity check prior node state before migrating tokens
> 
>
> Key: CASSANDRA-11688
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11688
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Jonathan Shook
>
> During a node replacement, a replace_address was used which was associated 
> with a different node than the intended one. The result was that both nodes 
> remained active after the node came up. This caused several other issues 
> which were difficult to diagnose, including invalid gossip state, etc.
> Replace_address should be more robust in this scenario. It would be much more 
> user friendly if the replace_address logic would first do some basic sanity 
> checks, possibly to include:
> - Pinging the other node to see if it is indeed “down”, if the address is 
> different than all local interface addresses
> - Checking gossip state of the node to verify that it is not known to peers.
> It may even be safest to require that both address reachability and gossip 
> state are required to show the replace_address as down by default before 
> allowing any token migration or other replace_address actions to occur.
> In the case that the replace_address is not ready to be replaced, the log 
> should indicate that you are trying to replace an active node, and cassandra 
> should refuse to start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11688) Replace_address should sanity check prior node state before migrating tokens

[Jonathan Shook (JIRA)]

Jonathan Shook created CASSANDRA-11688:
--

 Summary: Replace_address should sanity check prior node state 
before migrating tokens
 Key: CASSANDRA-11688
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11688
 Project: Cassandra
  Issue Type: Improvement
Reporter: Jonathan Shook


During a node replacement, a customer used an ip address associated with a 
different node than the intended one. The result was that both nodes remained 
active after the node came up. This caused several other issues which were 
difficult to diagnose, including invalid gossip state, etc.

Replace_address should be more robust in this scenario. It would be much more 
user friendly if the replace_address logic would first do some basic sanity 
checks, possibly to include:

- Pinging the other node to see if it is indeed “down”, if the address is 
different than all local interface addresses
- Checking gossip state of the node to verify that it is not known to peers.

It may even be safest to require that both address reachability and gossip 
state are required to show the replace_address as down by default before 
allowing any token migration or other replace_address actions to occur.

In the case that the replace_address is not ready to be replaced, the log 
should indicate that you are trying to replace an active node, and cassandra 
should refuse to start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11570) Concurrent execution of prepared statement returns invalid JSON as result

[Tyler Hobbs (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264724#comment-15264724
 ] 

Tyler Hobbs commented on CASSANDRA-11570:
-

[~neptunao] any news?

> Concurrent execution of prepared statement returns invalid JSON as result
> -
>
> Key: CASSANDRA-11570
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11570
> Project: Cassandra
>  Issue Type: Bug
> Environment: Cassandra 3.2, C++ or C# driver
>Reporter: Alexander Ryabets
>Assignee: Tyler Hobbs
> Attachments: CassandraPreparedStatementsTest.zip, broken_output.txt, 
> test_neptunao.cql, valid_output.txt
>
>
> When I use prepared statement for async execution of multiple statements I 
> get JSON with broken data. Keys got totally corrupted when values seems to be 
> normal though.
> First I encoutered this issue when I were performing stress testing of our 
> project using custom script. We are using DataStax C++ driver and execute 
> statements from different fibers.
> Then I was trying to isolate problem and wrote simple C# program which starts 
> multiple Tasks in a loop. Each task uses the once created prepared statement 
> to read data from the base. As you can see results are totally mess.
> I 've attached archive with console C# project (1 cs file) which just print 
> resulting JSON to user. 
> Here is the main part of C# code.
> {noformat}
> static void Main(string[] args)
> {
>   const int task_count = 300;
>   using(var cluster = Cluster.Builder().AddContactPoints(/*contact points 
> here*/).Build())
>   {
> using(var session = cluster.Connect())
> {
>   var prepared = session.Prepare("select json * from test_neptunao.ubuntu 
> where id=?");
>   var tasks = new Task[task_count];
>   for(int i = 0; i < task_count; i++)
>   {
> tasks[i] = Query(prepared, session);
>   }
>   Task.WaitAll(tasks);
> }
>   }
>   Console.ReadKey();
> }
> private static Task Query(PreparedStatement prepared, ISession session)
> {
>   string id = GetIdOfRandomRow();
>   var stmt = prepared.Bind(id);
>   stmt.SetConsistencyLevel(ConsistencyLevel.One);
>   return session.ExecuteAsync(stmt).ContinueWith(tr =>
>   {
> foreach(var row in tr.Result)
> {
>   var value = row.GetValue(0);
>   //some kind of output
> }
>   });
> }
> {noformat}
> I also attached cql script with test DB schema.
> {noformat}
> CREATE KEYSPACE IF NOT EXISTS test_neptunao
> WITH replication = {
>   'class' : 'SimpleStrategy',
>   'replication_factor' : 3
> };
> use test_neptunao;
> create table if not exists ubuntu (
>   id timeuuid PRIMARY KEY,
>   precise_pangolin text,
>   trusty_tahr text,
>   wily_werewolf text, 
>   vivid_vervet text,
>   saucy_salamander text,
>   lucid_lynx text
> );
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11609) Nested UDTs cause error when migrating 2.x schema to trunk

[Tyler Hobbs (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tyler Hobbs updated CASSANDRA-11609:

   Resolution: Fixed
Fix Version/s: (was: 3.x)
   3.6
   Status: Resolved  (was: Patch Available)

Thanks, committed as 
[{{d62b2cf7c77b01723c4d312bcc249c91a73b4e45}}|https://github.com/apache/cassandra/commit/d62b2cf7c77b01723c4d312bcc249c91a73b4e45]
 to trunk.

> Nested UDTs cause error when migrating 2.x schema to trunk
> --
>
> Key: CASSANDRA-11609
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11609
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Russ Hatch
>Assignee: Tyler Hobbs
> Fix For: 3.6
>
>
> This was found in the upgrades user_types_test.
> Can also be repro'd with ccm.
> To repro using ccm:
> Create a 1 node cluster on 2.2.x
> Create this schema:
> {noformat}
> create keyspace test2 with replication = {'class':'SimpleStrategy', 
> 'replication_factor':1};
> use test2;
> CREATE TYPE address (
>  street text,
>  city text,
>  zip_code int,
>  phones set
>  );
> CREATE TYPE fullname (
>  irstname text,
>  astname text
>  );
> CREATE TABLE users (
>  d uuid PRIMARY KEY,
>  ame frozen,
>  ddresses map
>  );
> {noformat}
> Upgrade the single node to trunk, attempt to start the node up. Start will 
> fail with this exception:
> {noformat}
> ERROR [main] 2016-04-19 11:33:19,218 CassandraDaemon.java:704 - Exception 
> encountered during startup
> org.apache.cassandra.exceptions.InvalidRequestException: Non-frozen UDTs are 
> not allowed inside collections: map
> at 
> org.apache.cassandra.cql3.CQL3Type$Raw$RawCollection.throwNestedNonFrozenError(CQL3Type.java:686)
>  ~[main/:na]
> at 
> org.apache.cassandra.cql3.CQL3Type$Raw$RawCollection.prepare(CQL3Type.java:652)
>  ~[main/:na]
> at 
> org.apache.cassandra.cql3.CQL3Type$Raw$RawCollection.prepareInternal(CQL3Type.java:644)
>  ~[main/:na]
> at 
> org.apache.cassandra.schema.CQLTypeParser.parse(CQLTypeParser.java:53) 
> ~[main/:na]
> at 
> org.apache.cassandra.schema.SchemaKeyspace.createColumnFromRow(SchemaKeyspace.java:1022)
>  ~[main/:na]
> at 
> org.apache.cassandra.schema.SchemaKeyspace.lambda$fetchColumns$12(SchemaKeyspace.java:1006)
>  ~[main/:na]
> at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_77]
> at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchColumns(SchemaKeyspace.java:1006)
>  ~[main/:na]
> at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:960)
>  ~[main/:na]
> at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:939)
>  ~[main/:na]
> at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:902)
>  ~[main/:na]
> at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:879)
>  ~[main/:na]
> at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:867)
>  ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:134) 
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:124) 
> ~[main/:na]
> at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:229) 
> [main/:na]
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:558)
>  [main/:na]
> at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:687) 
> [main/:na]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

cassandra git commit: Freeze implicitly frozen types in 2.x -> 3.x schema migration

[tylerhobbs]

Repository: cassandra
Updated Branches:
  refs/heads/trunk c6778c5af -> d62b2cf7c


Freeze implicitly frozen types in 2.x -> 3.x schema migration

Patch by Tyler Hobbs; reviewed by Benjamin Lerer for CASSANDRA-11609


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d62b2cf7
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d62b2cf7
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d62b2cf7

Branch: refs/heads/trunk
Commit: d62b2cf7c77b01723c4d312bcc249c91a73b4e45
Parents: c6778c5
Author: Tyler Hobbs 
Authored: Fri Apr 29 15:51:34 2016 -0500
Committer: Tyler Hobbs 
Committed: Fri Apr 29 15:51:34 2016 -0500

--
 .../org/apache/cassandra/db/marshal/AbstractType.java | 13 +
 .../org/apache/cassandra/db/marshal/ListType.java |  9 +
 src/java/org/apache/cassandra/db/marshal/MapType.java | 14 ++
 src/java/org/apache/cassandra/db/marshal/SetType.java |  9 +
 .../org/apache/cassandra/db/marshal/UserType.java | 12 
 .../apache/cassandra/schema/LegacySchemaMigrator.java |  5 +
 6 files changed, 62 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d62b2cf7/src/java/org/apache/cassandra/db/marshal/AbstractType.java
--
diff --git a/src/java/org/apache/cassandra/db/marshal/AbstractType.java 
b/src/java/org/apache/cassandra/db/marshal/AbstractType.java
index 1b94a74..7a67433 100644
--- a/src/java/org/apache/cassandra/db/marshal/AbstractType.java
+++ b/src/java/org/apache/cassandra/db/marshal/AbstractType.java
@@ -333,6 +333,19 @@ public abstract class AbstractType implements 
Comparator
 }
 
 /**
+ * Returns an AbstractType instance that is equivalent to this one, but 
with all nested UDTs explicitly frozen and
+ * all collections in UDTs explicitly frozen.
+ *
+ * This is only necessary for 2.x -> 3.x schema migrations, and can be 
removed in Cassandra 4.0.
+ *
+ * See CASSANDRA-11609
+ */
+public AbstractType freezeNestedUDTs()
+{
+return this;
+}
+
+/**
  * Returns {@code true} for types where empty should be handled like 
{@code null} like {@link Int32Type}.
  */
 public boolean isEmptyValueMeaningless()

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d62b2cf7/src/java/org/apache/cassandra/db/marshal/ListType.java
--
diff --git a/src/java/org/apache/cassandra/db/marshal/ListType.java 
b/src/java/org/apache/cassandra/db/marshal/ListType.java
index 4480dcb..b3fc4f5 100644
--- a/src/java/org/apache/cassandra/db/marshal/ListType.java
+++ b/src/java/org/apache/cassandra/db/marshal/ListType.java
@@ -110,6 +110,15 @@ public class ListType extends CollectionType
 }
 
 @Override
+public AbstractType freezeNestedUDTs()
+{
+if (elements.isUDT() && elements.isMultiCell())
+return getInstance(elements.freeze(), isMultiCell);
+else
+return getInstance(elements.freezeNestedUDTs(), isMultiCell);
+}
+
+@Override
 public boolean isMultiCell()
 {
 return isMultiCell;

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d62b2cf7/src/java/org/apache/cassandra/db/marshal/MapType.java
--
diff --git a/src/java/org/apache/cassandra/db/marshal/MapType.java 
b/src/java/org/apache/cassandra/db/marshal/MapType.java
index 425ffc2..3c5af99 100644
--- a/src/java/org/apache/cassandra/db/marshal/MapType.java
+++ b/src/java/org/apache/cassandra/db/marshal/MapType.java
@@ -117,6 +117,20 @@ public class MapType extends CollectionType>
 }
 
 @Override
+public AbstractType freezeNestedUDTs()
+{
+AbstractType keyType = (keys.isUDT() && keys.isMultiCell())
+? keys.freeze()
+: keys.freezeNestedUDTs();
+
+AbstractType valueType = (values.isUDT() && values.isMultiCell())
+  ? values.freeze()
+  : values.freezeNestedUDTs();
+
+return getInstance(keyType, valueType, isMultiCell);
+}
+
+@Override
 public boolean isCompatibleWithFrozen(CollectionType previous)
 {
 assert !isMultiCell;

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d62b2cf7/src/java/org/apache/cassandra/db/marshal/SetType.java
--
diff --git a/src/java/org/apache/cassandra/db/marshal/SetType.java 
b/src/java/org/apache/cassandra/db/marshal/SetType.java
index 22577b3..46c0741 100644
--- 

[jira] [Created] (CASSANDRA-11687) dtest failure in rebuild_test.TestRebuild.simple_rebuild_test

[Russ Hatch (JIRA)]

Russ Hatch created CASSANDRA-11687:
--

 Summary: dtest failure in 
rebuild_test.TestRebuild.simple_rebuild_test
 Key: CASSANDRA-11687
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11687
 Project: Cassandra
  Issue Type: Test
Reporter: Russ Hatch
Assignee: DS Test Eng


single failure on most recent run (3.0 no-vnode)

{noformat}
concurrent rebuild should not be allowed, but one rebuild command should have 
succeeded.
{noformat}

http://cassci.datastax.com/job/cassandra-3.0_novnode_dtest/217/testReport/rebuild_test/TestRebuild/simple_rebuild_test

Failed on CassCI build cassandra-3.0_novnode_dtest #217



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11542) Create a benchmark to compare HDFS and Cassandra bulk read times

[Russell Alexander Spitzer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264698#comment-15264698
 ] 

Russell Alexander Spitzer commented on CASSANDRA-11542:
---

Hmm i'm a little confused that case classes don't help but Dataframes do... The 
code you presented looks good to me, there is the potential issue of blocking 
on resultsets that take a long time to complete while other result-sets are 
already on the driver but i'm not sure if this is a big deal. Do you have any 
idea of the parallelization in these test? How many partitions are the 
different runs generating?

> Create a benchmark to compare HDFS and Cassandra bulk read times
> 
>
> Key: CASSANDRA-11542
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11542
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Testing
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 3.x
>
> Attachments: spark-load-perf-results-001.zip, 
> spark-load-perf-results-002.zip
>
>
> I propose creating a benchmark for comparing Cassandra and HDFS bulk reading 
> performance. Simple Spark queries will be performed on data stored in HDFS or 
> Cassandra, and the entire duration will be measured. An example query would 
> be the max or min of a column or a count\(*\).
> This benchmark should allow determining the impact of:
> * partition size
> * number of clustering columns
> * number of value columns (cells)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11391) "class declared as inner class" error when using UDF

[Tyler Hobbs (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tyler Hobbs updated CASSANDRA-11391:

Status: Ready to Commit  (was: Patch Available)

> "class declared as inner class" error when using UDF
> 
>
> Key: CASSANDRA-11391
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11391
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: C* 3.4
>Reporter: DOAN DuyHai
>Assignee: Robert Stupp
>Priority: Critical
> Fix For: 3.x
>
>
> {noformat}
> cqlsh:music> CREATE FUNCTION testMapEntry(my_map map)
>  ... CALLED ON NULL INPUT
>  ... RETURNS text
>  ... LANGUAGE java
>  ... AS $$
>  ... String buffer = "";
>  ... for(java.util.Map.Entry entry: 
> my_map.entrySet()) {
>  ... buffer = buffer + entry.getKey() + ": " + 
> entry.getValue() + ", ";
>  ... }
>  ... return buffer;
>  ... $$;
> InvalidRequest: code=2200 [Invalid query] 
> message="Could not compile function 'music.testmapentry' from Java source: 
> org.apache.cassandra.exceptions.InvalidRequestException: 
> Java UDF validation failed: [class declared as inner class]"
> {noformat}
> When I try to decompile the source code into byte code, below is the result:
> {noformat}
>   public java.lang.String test(java.util.Map java.lang.String>);
> Code:
>0: ldc   #2  // String
>2: astore_2
>3: aload_1
>4: invokeinterface #3,  1// InterfaceMethod 
> java/util/Map.entrySet:()Ljava/util/Set;
>9: astore_3
>   10: aload_3
>   11: invokeinterface #4,  1// InterfaceMethod 
> java/util/Set.iterator:()Ljava/util/Iterator;
>   16: astore4
>   18: aload 4
>   20: invokeinterface #5,  1// InterfaceMethod 
> java/util/Iterator.hasNext:()Z
>   25: ifeq  94
>   28: aload 4
>   30: invokeinterface #6,  1// InterfaceMethod 
> java/util/Iterator.next:()Ljava/lang/Object;
>   35: checkcast #7  // class java/util/Map$Entry
>   38: astore5
>   40: new   #8  // class java/lang/StringBuilder
>   43: dup
>   44: invokespecial #9  // Method 
> java/lang/StringBuilder."":()V
>   47: aload_2
>   48: invokevirtual #10 // Method 
> java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
>   51: aload 5
>   53: invokeinterface #11,  1   // InterfaceMethod 
> java/util/Map$Entry.getKey:()Ljava/lang/Object;
>   58: checkcast #12 // class java/lang/String
>   61: invokevirtual #10 // Method 
> java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
>   64: ldc   #13 // String :
>   66: invokevirtual #10 // Method 
> java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
>   69: aload 5
>   71: invokeinterface #14,  1   // InterfaceMethod 
> java/util/Map$Entry.getValue:()Ljava/lang/Object;
>   76: checkcast #12 // class java/lang/String
>   79: invokevirtual #10 // Method 
> java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
>   82: ldc   #15 // String ,
>   84: invokevirtual #10 // Method 
> java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
>   87: invokevirtual #16 // Method 
> java/lang/StringBuilder.toString:()Ljava/lang/String;
>   90: astore_2
>   91: goto  18
>   94: aload_2
>   95: areturn
> {noformat}
>  There is nothing that could trigger inner class creation ...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11391) "class declared as inner class" error when using UDF

[Tyler Hobbs (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264672#comment-15264672
 ] 

Tyler Hobbs commented on CASSANDRA-11391:
-

The new test results look much better, so +1 from me.

> "class declared as inner class" error when using UDF
> 
>
> Key: CASSANDRA-11391
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11391
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: C* 3.4
>Reporter: DOAN DuyHai
>Assignee: Robert Stupp
>Priority: Critical
> Fix For: 3.x
>
>
> {noformat}
> cqlsh:music> CREATE FUNCTION testMapEntry(my_map map)
>  ... CALLED ON NULL INPUT
>  ... RETURNS text
>  ... LANGUAGE java
>  ... AS $$
>  ... String buffer = "";
>  ... for(java.util.Map.Entry entry: 
> my_map.entrySet()) {
>  ... buffer = buffer + entry.getKey() + ": " + 
> entry.getValue() + ", ";
>  ... }
>  ... return buffer;
>  ... $$;
> InvalidRequest: code=2200 [Invalid query] 
> message="Could not compile function 'music.testmapentry' from Java source: 
> org.apache.cassandra.exceptions.InvalidRequestException: 
> Java UDF validation failed: [class declared as inner class]"
> {noformat}
> When I try to decompile the source code into byte code, below is the result:
> {noformat}
>   public java.lang.String test(java.util.Map java.lang.String>);
> Code:
>0: ldc   #2  // String
>2: astore_2
>3: aload_1
>4: invokeinterface #3,  1// InterfaceMethod 
> java/util/Map.entrySet:()Ljava/util/Set;
>9: astore_3
>   10: aload_3
>   11: invokeinterface #4,  1// InterfaceMethod 
> java/util/Set.iterator:()Ljava/util/Iterator;
>   16: astore4
>   18: aload 4
>   20: invokeinterface #5,  1// InterfaceMethod 
> java/util/Iterator.hasNext:()Z
>   25: ifeq  94
>   28: aload 4
>   30: invokeinterface #6,  1// InterfaceMethod 
> java/util/Iterator.next:()Ljava/lang/Object;
>   35: checkcast #7  // class java/util/Map$Entry
>   38: astore5
>   40: new   #8  // class java/lang/StringBuilder
>   43: dup
>   44: invokespecial #9  // Method 
> java/lang/StringBuilder."":()V
>   47: aload_2
>   48: invokevirtual #10 // Method 
> java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
>   51: aload 5
>   53: invokeinterface #11,  1   // InterfaceMethod 
> java/util/Map$Entry.getKey:()Ljava/lang/Object;
>   58: checkcast #12 // class java/lang/String
>   61: invokevirtual #10 // Method 
> java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
>   64: ldc   #13 // String :
>   66: invokevirtual #10 // Method 
> java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
>   69: aload 5
>   71: invokeinterface #14,  1   // InterfaceMethod 
> java/util/Map$Entry.getValue:()Ljava/lang/Object;
>   76: checkcast #12 // class java/lang/String
>   79: invokevirtual #10 // Method 
> java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
>   82: ldc   #15 // String ,
>   84: invokevirtual #10 // Method 
> java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
>   87: invokevirtual #16 // Method 
> java/lang/StringBuilder.toString:()Ljava/lang/String;
>   90: astore_2
>   91: goto  18
>   94: aload_2
>   95: areturn
> {noformat}
>  There is nothing that could trigger inner class creation ...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11686) dtest failure in replication_test.SnitchConfigurationUpdateTest.test_rf_expand_gossiping_property_file_snitch_multi_dc

[Russ Hatch (JIRA)]

Russ Hatch created CASSANDRA-11686:
--

 Summary: dtest failure in 
replication_test.SnitchConfigurationUpdateTest.test_rf_expand_gossiping_property_file_snitch_multi_dc
 Key: CASSANDRA-11686
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11686
 Project: Cassandra
  Issue Type: Test
Reporter: Russ Hatch


intermittent failure. this test also fails on windows but looks to be for 
another reason (CASSANDRA-11439)

http://cassci.datastax.com/job/cassandra-3.0_dtest/682/testReport/replication_test/SnitchConfigurationUpdateTest/test_rf_expand_gossiping_property_file_snitch_multi_dc/

{noformat}
Nodetool command '/home/automaton/cassandra/bin/nodetool -h localhost -p 7400 
getendpoints testing rf_test dummy' failed; exit status: 1; stderr: nodetool: 
Failed to connect to 'localhost:7400' - ConnectException: 'Connection refused'.
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8844) Change Data Capture (CDC)

[Joshua McKenzie (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264667#comment-15264667
 ] 

Joshua McKenzie commented on CASSANDRA-8844:


bq. This means we'll be bringing the checking of whether or not we have a CDC 
write back into the CommitLog. One of the things that I really liked about the 
direction this proposal is going in was that we were removing a lot of the 
runtime dependencies on state, as well as providing a much lower burden to the 
C* process, at the cost of an additional outside process that would need to be 
managed if CDC was enabled on a node.

I agree, but in order for us to selectively reject mutations w/CDC-enabled CF's 
in them, this is something we're going to have to do. Only alternatives I can 
see are:
# full reject of all mutations if cdc is enabled and we're at limit
# don't allow setting a cdc limit and strongly advise users to put it on its 
own disk, so we don't backpressure from it, so we don't have to check 
mutations. And you can lose CDC data if the drive fills.

I don't see reject all being viable since it'll shut down your cluster, and 
reject none on a slow consumer scenario would just lead to disk space 
exhaustion and lost CDC data.

> Change Data Capture (CDC)
> -
>
> Key: CASSANDRA-8844
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8844
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Coordination, Local Write-Read Paths
>Reporter: Tupshin Harper
>Assignee: Joshua McKenzie
>Priority: Critical
> Fix For: 3.x
>
>
> "In databases, change data capture (CDC) is a set of software design patterns 
> used to determine (and track) the data that has changed so that action can be 
> taken using the changed data. Also, Change data capture (CDC) is an approach 
> to data integration that is based on the identification, capture and delivery 
> of the changes made to enterprise data sources."
> -Wikipedia
> As Cassandra is increasingly being used as the Source of Record (SoR) for 
> mission critical data in large enterprises, it is increasingly being called 
> upon to act as the central hub of traffic and data flow to other systems. In 
> order to try to address the general need, we (cc [~brianmhess]), propose 
> implementing a simple data logging mechanism to enable per-table CDC patterns.
> h2. The goals:
> # Use CQL as the primary ingestion mechanism, in order to leverage its 
> Consistency Level semantics, and in order to treat it as the single 
> reliable/durable SoR for the data.
> # To provide a mechanism for implementing good and reliable 
> (deliver-at-least-once with possible mechanisms for deliver-exactly-once ) 
> continuous semi-realtime feeds of mutations going into a Cassandra cluster.
> # To eliminate the developmental and operational burden of users so that they 
> don't have to do dual writes to other systems.
> # For users that are currently doing batch export from a Cassandra system, 
> give them the opportunity to make that realtime with a minimum of coding.
> h2. The mechanism:
> We propose a durable logging mechanism that functions similar to a commitlog, 
> with the following nuances:
> - Takes place on every node, not just the coordinator, so RF number of copies 
> are logged.
> - Separate log per table.
> - Per-table configuration. Only tables that are specified as CDC_LOG would do 
> any logging.
> - Per DC. We are trying to keep the complexity to a minimum to make this an 
> easy enhancement, but most likely use cases would prefer to only implement 
> CDC logging in one (or a subset) of the DCs that are being replicated to
> - In the critical path of ConsistencyLevel acknowledgment. Just as with the 
> commitlog, failure to write to the CDC log should fail that node's write. If 
> that means the requested consistency level was not met, then clients *should* 
> experience UnavailableExceptions.
> - Be written in a Row-centric manner such that it is easy for consumers to 
> reconstitute rows atomically.
> - Written in a simple format designed to be consumed *directly* by daemons 
> written in non JVM languages
> h2. Nice-to-haves
> I strongly suspect that the following features will be asked for, but I also 
> believe that they can be deferred for a subsequent release, and to guage 
> actual interest.
> - Multiple logs per table. This would make it easy to have multiple 
> "subscribers" to a single table's changes. A workaround would be to create a 
> forking daemon listener, but that's not a great answer.
> - Log filtering. Being able to apply filters, including UDF-based filters 
> would make Casandra a much more versatile feeder into other systems, and 
> again, reduce complexity that would otherwise need to be built into the 
> daemons.
> h2. Format and Consumption
> - Cassandra 

[jira] [Commented] (CASSANDRA-8844) Change Data Capture (CDC)

[Carl Yeksigian (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264624#comment-15264624
 ] 

Carl Yeksigian commented on CASSANDRA-8844:
---

bq. Initial thoughts on ways around this: On segment creation, if there is not 
room to accommodate another CLS in cdc_overflow we can flag the segment as not 
accepting CDC writes. .allocate calls on Mutation containing CDC-enabled CF's 
reject if acceptingCDCWrites is false, use current logic to re-check on-disk 
size w/RateLimiter, and if cdc_overflow + segment size <= allowable on return, 
we toggle accepting on the active segment back on. On a successful CDC 
mutations, we idempotently enable a different bool to indicate that a segment 
has accepted a CDC write. On segment discard, check if segment accepted any CDC 
writes and, if so, move to cdc_overflow. If not, delete.

This means we'll be bringing the checking of whether or not we have a CDC write 
back into the CommitLog. One of the things that I really liked about the 
direction this proposal is going in was that we were removing a lot of the 
runtime dependencies on state, as well as providing a much lower burden to the 
C* process, at the cost of an additional outside process that would need to be 
managed if CDC was enabled on a node.

bq. Maybe flush all cdc-enabled CF on daemon shutdown to prevent this? Or 
document it and recommend flushing all CDC-enabled CF before toggling the 
setting.
Yeah, documenting is probably the right way of helping this. It seems like the 
caveat is that the data written between the last flush and changing the CDC 
setting is not guaranteed to follow the CDC setting.

bq. At-least-once semantics should allow an edge-case like this where some data 
gets re-compacted and delivered multiple times. 

Perfect; I was thinking that we might have issues with more-than-once delivery 
at the node-level, but there is always the possibility we'd replay hints and 
cause the identical mutation to be in the commit log twice, so would always 
have had to have been handled.

> Change Data Capture (CDC)
> -
>
> Key: CASSANDRA-8844
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8844
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Coordination, Local Write-Read Paths
>Reporter: Tupshin Harper
>Assignee: Joshua McKenzie
>Priority: Critical
> Fix For: 3.x
>
>
> "In databases, change data capture (CDC) is a set of software design patterns 
> used to determine (and track) the data that has changed so that action can be 
> taken using the changed data. Also, Change data capture (CDC) is an approach 
> to data integration that is based on the identification, capture and delivery 
> of the changes made to enterprise data sources."
> -Wikipedia
> As Cassandra is increasingly being used as the Source of Record (SoR) for 
> mission critical data in large enterprises, it is increasingly being called 
> upon to act as the central hub of traffic and data flow to other systems. In 
> order to try to address the general need, we (cc [~brianmhess]), propose 
> implementing a simple data logging mechanism to enable per-table CDC patterns.
> h2. The goals:
> # Use CQL as the primary ingestion mechanism, in order to leverage its 
> Consistency Level semantics, and in order to treat it as the single 
> reliable/durable SoR for the data.
> # To provide a mechanism for implementing good and reliable 
> (deliver-at-least-once with possible mechanisms for deliver-exactly-once ) 
> continuous semi-realtime feeds of mutations going into a Cassandra cluster.
> # To eliminate the developmental and operational burden of users so that they 
> don't have to do dual writes to other systems.
> # For users that are currently doing batch export from a Cassandra system, 
> give them the opportunity to make that realtime with a minimum of coding.
> h2. The mechanism:
> We propose a durable logging mechanism that functions similar to a commitlog, 
> with the following nuances:
> - Takes place on every node, not just the coordinator, so RF number of copies 
> are logged.
> - Separate log per table.
> - Per-table configuration. Only tables that are specified as CDC_LOG would do 
> any logging.
> - Per DC. We are trying to keep the complexity to a minimum to make this an 
> easy enhancement, but most likely use cases would prefer to only implement 
> CDC logging in one (or a subset) of the DCs that are being replicated to
> - In the critical path of ConsistencyLevel acknowledgment. Just as with the 
> commitlog, failure to write to the CDC log should fail that node's write. If 
> that means the requested consistency level was not met, then clients *should* 
> experience UnavailableExceptions.
> - Be written in a Row-centric manner such that it is easy for consumers to 
> 

[jira] [Commented] (CASSANDRA-11626) cqlsh fails and exits on non-ascii chars

[Tyler Hobbs (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264620#comment-15264620
 ] 

Tyler Hobbs commented on CASSANDRA-11626:
-

In a few places in cqlsh we were only handling ascii chars in error messages.  
I've changed this to use the specified encoding (defaults to UTF-8).

Branches and pending CI runs (merges from 2.2 up are trivial and can be 
ignored):
||branch||dtest||
|[CASSANDRA-11626-2.2|https://github.com/thobbs/cassandra/tree/CASSANDRA-11626-2.2]|[dtest|http://cassci.datastax.com/view/Dev/view/thobbs/job/thobbs-CASSANDRA-11626-2.2-dtest]|
|[CASSANDRA-11626-3.0|https://github.com/thobbs/cassandra/tree/CASSANDRA-11626-3.0]|[dtest|http://cassci.datastax.com/view/Dev/view/thobbs/job/thobbs-CASSANDRA-11626-3.0-dtest]|
|[CASSANDRA-11626-trunk|https://github.com/thobbs/cassandra/tree/CASSANDRA-11626-trunk]|[dtest|http://cassci.datastax.com/view/Dev/view/thobbs/job/thobbs-CASSANDRA-11626-trunk-dtest]|

I've also opened a [dtest pull 
request|https://github.com/riptano/cassandra-dtest/pull/965] that covers a 
couple of the unicode error message scenarios.

> cqlsh fails and exits on non-ascii chars
> 
>
> Key: CASSANDRA-11626
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11626
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Robert Stupp
>Assignee: Tyler Hobbs
>Priority: Minor
>  Labels: cqlsh
> Fix For: 2.2.x, 3.0.x, 3.x
>
>
> Just seen on cqlsh on current trunk:
> To repro, copy {{ä}} (german umlaut) to cqlsh and press return.
> cqlsh errors out and immediately exits.
> {code}
> $ bin/cqlsh
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.13-SNAPSHOT | CQL spec 3.2.1 | Native protocol 
> v3]
> Use HELP for help.
> cqlsh> ä
> Invalid syntax at line 1, char 1
> Traceback (most recent call last):
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2636, in 
> 
> main(*read_options(sys.argv[1:], os.environ))
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2625, in main
> shell.cmdloop()
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 1114, in 
> cmdloop
> if self.onecmd(self.statement.getvalue()):
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 1139, in onecmd
> self.printerr('  %s' % statementline)
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2314, in 
> printerr
> self.writeresult(text, color, newline=newline, out=sys.stderr)
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2303, in 
> writeresult
> out.write(self.applycolor(str(text), color) + ('\n' if newline else ''))
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 
> 2: ordinal not in range(128)
> $ 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11626) cqlsh fails and exits on non-ascii chars

[Tyler Hobbs (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tyler Hobbs updated CASSANDRA-11626:

Reproduced In: 3.5, 3.0.4  (was: 3.0.4, 3.5)
  Component/s: Tools

> cqlsh fails and exits on non-ascii chars
> 
>
> Key: CASSANDRA-11626
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11626
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Robert Stupp
>Assignee: Tyler Hobbs
>Priority: Minor
>  Labels: cqlsh
> Fix For: 2.2.x, 3.0.x, 3.x
>
>
> Just seen on cqlsh on current trunk:
> To repro, copy {{ä}} (german umlaut) to cqlsh and press return.
> cqlsh errors out and immediately exits.
> {code}
> $ bin/cqlsh
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.13-SNAPSHOT | CQL spec 3.2.1 | Native protocol 
> v3]
> Use HELP for help.
> cqlsh> ä
> Invalid syntax at line 1, char 1
> Traceback (most recent call last):
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2636, in 
> 
> main(*read_options(sys.argv[1:], os.environ))
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2625, in main
> shell.cmdloop()
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 1114, in 
> cmdloop
> if self.onecmd(self.statement.getvalue()):
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 1139, in onecmd
> self.printerr('  %s' % statementline)
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2314, in 
> printerr
> self.writeresult(text, color, newline=newline, out=sys.stderr)
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2303, in 
> writeresult
> out.write(self.applycolor(str(text), color) + ('\n' if newline else ''))
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 
> 2: ordinal not in range(128)
> $ 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11626) cqlsh fails and exits on non-ascii chars

[Tyler Hobbs (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tyler Hobbs updated CASSANDRA-11626:

   Labels: cqlsh  (was: )
 Reviewer: Paulo Motta
Fix Version/s: 3.x
   3.0.x
   2.2.x
Reproduced In: 3.5, 3.0.4  (was: 3.0.4, 3.5)
   Status: Patch Available  (was: In Progress)

> cqlsh fails and exits on non-ascii chars
> 
>
> Key: CASSANDRA-11626
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11626
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Robert Stupp
>Assignee: Tyler Hobbs
>Priority: Minor
>  Labels: cqlsh
> Fix For: 2.2.x, 3.0.x, 3.x
>
>
> Just seen on cqlsh on current trunk:
> To repro, copy {{ä}} (german umlaut) to cqlsh and press return.
> cqlsh errors out and immediately exits.
> {code}
> $ bin/cqlsh
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.13-SNAPSHOT | CQL spec 3.2.1 | Native protocol 
> v3]
> Use HELP for help.
> cqlsh> ä
> Invalid syntax at line 1, char 1
> Traceback (most recent call last):
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2636, in 
> 
> main(*read_options(sys.argv[1:], os.environ))
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2625, in main
> shell.cmdloop()
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 1114, in 
> cmdloop
> if self.onecmd(self.statement.getvalue()):
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 1139, in onecmd
> self.printerr('  %s' % statementline)
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2314, in 
> printerr
> self.writeresult(text, color, newline=newline, out=sys.stderr)
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2303, in 
> writeresult
> out.write(self.applycolor(str(text), color) + ('\n' if newline else ''))
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 
> 2: ordinal not in range(128)
> $ 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-11656) sstabledump has inconsistency in deletion_time printout

[Yuki Morishita (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuki Morishita resolved CASSANDRA-11656.

Resolution: Fixed
  Assignee: Chris Lohfink

Fixed in CASSANDRA-11655.

> sstabledump has inconsistency in deletion_time printout
> ---
>
> Key: CASSANDRA-11656
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11656
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Wei Deng
>Assignee: Chris Lohfink
>  Labels: Tools
>
> See the following output (note the deletion info under the second row):
> {noformat}
> [
>   {
> "partition" : {
>   "key" : [ "1" ],
>   "position" : 0
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 18,
> "clustering" : [ "c1" ],
> "liveness_info" : { "tstamp" : 1461646542601774 },
> "cells" : [
>   { "name" : "val0_int", "deletion_time" : 1461647421, "tstamp" : 
> 1461647421344759 },
>   { "name" : "val1_set_of_int", "path" : [ "1" ], "deletion_time" : 
> 1461647320, "tstamp" : 1461647320160261 },
>   { "name" : "val1_set_of_int", "path" : [ "10" ], "value" : "", 
> "tstamp" : 1461647295880444 },
>   { "name" : "val1_set_of_int", "path" : [ "11" ], "value" : "", 
> "tstamp" : 1461647295880444 },
>   { "name" : "val1_set_of_int", "path" : [ "12" ], "value" : "", 
> "tstamp" : 1461647295880444 }
> ]
>   },
>   {
> "type" : "row",
> "position" : 85,
> "clustering" : [ "c2" ],
> "deletion_info" : { "deletion_time" : 1461647588089843, "tstamp" : 
> 1461647588 },
> "cells" : [ ]
>   }
> ]
>   }
> ]
> {noformat}
> To avoid confusion, we need to have consistency in printing out the 
> DeletionTime object. By definition, markedForDeleteAt is in microseconds 
> since epoch and marks the time when the "delete" mutation happens; 
> localDeletionTime is in seconds since epoch and allows GC to collect the 
> tombstone if the current epoch second is greater than localDeletionTime + 
> gc_grace_seconds. I'm ok to use "tstamp" to represent markedForDeleteAt 
> because markedForDeleteAt does represent this delete mutation's timestamp, 
> but we need to be consistent everywhere.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11655) sstabledump doesn't print out tombstone information for deleted collection column

[Yuki Morishita (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuki Morishita updated CASSANDRA-11655:
---
   Resolution: Fixed
Fix Version/s: 3.0.6
   3.6
   Status: Resolved  (was: Patch Available)

+1 as well.
Committed as {{620efdc8c4968e45994496b23cd7dcdfbccdad6d}}.

> sstabledump doesn't print out tombstone information for deleted collection 
> column
> -
>
> Key: CASSANDRA-11655
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11655
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Wei Deng
>Assignee: Chris Lohfink
>  Labels: Tools
> Fix For: 3.6, 3.0.6
>
> Attachments: CASSANDRA-11655.patch, trunk-11655v2.patch, 
> trunk-11655v3.patch
>
>
> Pretty trivial to reproduce.
> {noformat}
> echo "CREATE KEYSPACE IF NOT EXISTS testks WITH replication = {'class': 
> 'SimpleStrategy', 'replication_factor': '1'};" | cqlsh
> echo "CREATE TABLE IF NOT EXISTS testks.testcf ( k int, c text, val0_int int, 
> val1_set_of_int set, PRIMARY KEY (k, c) );" | cqlsh
> echo "INSERT INTO testks.testcf (k, c, val0_int, val1_set_of_int) VALUES (1, 
> 'c1', 100, {1, 2, 3, 4, 5});" | cqlsh
> echo "delete val1_set_of_int from testks.testcf where k=1 and c='c1';" | cqlsh
> echo "select * from testks.testcf;" | cqlsh
> nodetool flush testks testcf
> {noformat}
> Now if you run sstabledump (even after taking the 
> [patch|https://github.com/yukim/cassandra/tree/11654-3.0] for 
> CASSANDRA-11654) against the newly generated SSTable like the following:
> {noformat}
> ~/cassandra-trunk/tools/bin/sstabledump ma-1-big-Data.db
> [
>   {
> "partition" : {
>   "key" : [ "1" ],
>   "position" : 0
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 18,
> "clustering" : [ "c1" ],
> "liveness_info" : { "tstamp" : 1461645231352208 },
> "cells" : [
>   { "name" : "val0_int", "value" : "100" }
> ]
>   }
> ]
>   }
> ]
> {noformat}
> You will see that the collection-level Deletion Info is nowhere to be found, 
> so you will not be able to know "markedForDeleteAt" or "localDeletionTime" 
> for this collection tombstone.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/3] cassandra git commit: Improve tombstone printing in sstabledump

[yukim]

Improve tombstone printing in sstabledump

patch by clohfink; reviewed by Wei Deng for CASSANDRA-11655


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/620efdc8
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/620efdc8
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/620efdc8

Branch: refs/heads/trunk
Commit: 620efdc8c4968e45994496b23cd7dcdfbccdad6d
Parents: b15983e
Author: Chris Lohfink 
Authored: Thu Apr 28 20:34:29 2016 -0500
Committer: Yuki Morishita 
Committed: Fri Apr 29 14:01:02 2016 -0500

--
 CHANGES.txt |   1 +
 .../apache/cassandra/tools/JsonTransformer.java | 112 +--
 .../apache/cassandra/tools/SSTableExport.java   |  11 +-
 3 files changed, 85 insertions(+), 39 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/620efdc8/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 3184cce..64bcbd8 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.6
+ * Improve tombstone printing in sstabledump (CASSANDRA-11655)
  * Fix paging for range queries where all clustering columns are specified 
(CASSANDRA-11669)
  * Don't require HEAP_NEW_SIZE to be set when using G1 (CASSANDRA-11600)
  * Fix sstabledump not showing cells after tombstone marker (CASSANDRA-11654)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/620efdc8/src/java/org/apache/cassandra/tools/JsonTransformer.java
--
diff --git a/src/java/org/apache/cassandra/tools/JsonTransformer.java 
b/src/java/org/apache/cassandra/tools/JsonTransformer.java
index 93bb686..364070e 100644
--- a/src/java/org/apache/cassandra/tools/JsonTransformer.java
+++ b/src/java/org/apache/cassandra/tools/JsonTransformer.java
@@ -4,7 +4,9 @@ import java.io.IOException;
 import java.io.OutputStream;
 import java.io.OutputStreamWriter;
 import java.nio.ByteBuffer;
+import java.time.Instant;
 import java.util.List;
+import java.util.concurrent.TimeUnit;
 import java.util.stream.Stream;
 
 import org.apache.cassandra.config.CFMetaData;
@@ -18,6 +20,8 @@ import org.apache.cassandra.db.marshal.AbstractType;
 import org.apache.cassandra.db.marshal.CollectionType;
 import org.apache.cassandra.db.marshal.CompositeType;
 import org.apache.cassandra.db.rows.Cell;
+import org.apache.cassandra.db.rows.ColumnData;
+import org.apache.cassandra.db.rows.ComplexColumnData;
 import org.apache.cassandra.db.rows.RangeTombstoneBoundMarker;
 import org.apache.cassandra.db.rows.RangeTombstoneBoundaryMarker;
 import org.apache.cassandra.db.rows.RangeTombstoneMarker;
@@ -51,13 +55,16 @@ public final class JsonTransformer
 
 private final ISSTableScanner currentScanner;
 
+private boolean rawTime = false;
+
 private long currentPosition = 0;
 
-private JsonTransformer(JsonGenerator json, ISSTableScanner 
currentScanner, CFMetaData metadata)
+private JsonTransformer(JsonGenerator json, ISSTableScanner 
currentScanner, boolean rawTime, CFMetaData metadata)
 {
 this.json = json;
 this.metadata = metadata;
 this.currentScanner = currentScanner;
+this.rawTime = rawTime;
 
 DefaultPrettyPrinter prettyPrinter = new DefaultPrettyPrinter();
 prettyPrinter.indentObjectsWith(objectIndenter);
@@ -65,23 +72,23 @@ public final class JsonTransformer
 json.setPrettyPrinter(prettyPrinter);
 }
 
-public static void toJson(ISSTableScanner currentScanner, 
Stream partitions, CFMetaData metadata, OutputStream out)
+public static void toJson(ISSTableScanner currentScanner, 
Stream partitions, boolean rawTime, CFMetaData metadata, 
OutputStream out)
 throws IOException
 {
 try (JsonGenerator json = jsonFactory.createJsonGenerator(new 
OutputStreamWriter(out, "UTF-8")))
 {
-JsonTransformer transformer = new JsonTransformer(json, 
currentScanner, metadata);
+JsonTransformer transformer = new JsonTransformer(json, 
currentScanner, rawTime, metadata);
 json.writeStartArray();
 partitions.forEach(transformer::serializePartition);
 json.writeEndArray();
 }
 }
 
-public static void keysToJson(ISSTableScanner currentScanner, 
Stream keys, CFMetaData metadata, OutputStream out) throws 
IOException
+public static void keysToJson(ISSTableScanner currentScanner, 
Stream keys, boolean rawTime, CFMetaData metadata, OutputStream 
out) throws IOException
 {
 try (JsonGenerator json = jsonFactory.createJsonGenerator(new 
OutputStreamWriter(out, "UTF-8")))
 {
-JsonTransformer transformer = new 

[1/3] cassandra git commit: Improve tombstone printing in sstabledump

[yukim]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 b15983e83 -> 620efdc8c
  refs/heads/trunk ff4d0f9ab -> c6778c5af


Improve tombstone printing in sstabledump

patch by clohfink; reviewed by Wei Deng for CASSANDRA-11655


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/620efdc8
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/620efdc8
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/620efdc8

Branch: refs/heads/cassandra-3.0
Commit: 620efdc8c4968e45994496b23cd7dcdfbccdad6d
Parents: b15983e
Author: Chris Lohfink 
Authored: Thu Apr 28 20:34:29 2016 -0500
Committer: Yuki Morishita 
Committed: Fri Apr 29 14:01:02 2016 -0500

--
 CHANGES.txt |   1 +
 .../apache/cassandra/tools/JsonTransformer.java | 112 +--
 .../apache/cassandra/tools/SSTableExport.java   |  11 +-
 3 files changed, 85 insertions(+), 39 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/620efdc8/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 3184cce..64bcbd8 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.6
+ * Improve tombstone printing in sstabledump (CASSANDRA-11655)
  * Fix paging for range queries where all clustering columns are specified 
(CASSANDRA-11669)
  * Don't require HEAP_NEW_SIZE to be set when using G1 (CASSANDRA-11600)
  * Fix sstabledump not showing cells after tombstone marker (CASSANDRA-11654)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/620efdc8/src/java/org/apache/cassandra/tools/JsonTransformer.java
--
diff --git a/src/java/org/apache/cassandra/tools/JsonTransformer.java 
b/src/java/org/apache/cassandra/tools/JsonTransformer.java
index 93bb686..364070e 100644
--- a/src/java/org/apache/cassandra/tools/JsonTransformer.java
+++ b/src/java/org/apache/cassandra/tools/JsonTransformer.java
@@ -4,7 +4,9 @@ import java.io.IOException;
 import java.io.OutputStream;
 import java.io.OutputStreamWriter;
 import java.nio.ByteBuffer;
+import java.time.Instant;
 import java.util.List;
+import java.util.concurrent.TimeUnit;
 import java.util.stream.Stream;
 
 import org.apache.cassandra.config.CFMetaData;
@@ -18,6 +20,8 @@ import org.apache.cassandra.db.marshal.AbstractType;
 import org.apache.cassandra.db.marshal.CollectionType;
 import org.apache.cassandra.db.marshal.CompositeType;
 import org.apache.cassandra.db.rows.Cell;
+import org.apache.cassandra.db.rows.ColumnData;
+import org.apache.cassandra.db.rows.ComplexColumnData;
 import org.apache.cassandra.db.rows.RangeTombstoneBoundMarker;
 import org.apache.cassandra.db.rows.RangeTombstoneBoundaryMarker;
 import org.apache.cassandra.db.rows.RangeTombstoneMarker;
@@ -51,13 +55,16 @@ public final class JsonTransformer
 
 private final ISSTableScanner currentScanner;
 
+private boolean rawTime = false;
+
 private long currentPosition = 0;
 
-private JsonTransformer(JsonGenerator json, ISSTableScanner 
currentScanner, CFMetaData metadata)
+private JsonTransformer(JsonGenerator json, ISSTableScanner 
currentScanner, boolean rawTime, CFMetaData metadata)
 {
 this.json = json;
 this.metadata = metadata;
 this.currentScanner = currentScanner;
+this.rawTime = rawTime;
 
 DefaultPrettyPrinter prettyPrinter = new DefaultPrettyPrinter();
 prettyPrinter.indentObjectsWith(objectIndenter);
@@ -65,23 +72,23 @@ public final class JsonTransformer
 json.setPrettyPrinter(prettyPrinter);
 }
 
-public static void toJson(ISSTableScanner currentScanner, 
Stream partitions, CFMetaData metadata, OutputStream out)
+public static void toJson(ISSTableScanner currentScanner, 
Stream partitions, boolean rawTime, CFMetaData metadata, 
OutputStream out)
 throws IOException
 {
 try (JsonGenerator json = jsonFactory.createJsonGenerator(new 
OutputStreamWriter(out, "UTF-8")))
 {
-JsonTransformer transformer = new JsonTransformer(json, 
currentScanner, metadata);
+JsonTransformer transformer = new JsonTransformer(json, 
currentScanner, rawTime, metadata);
 json.writeStartArray();
 partitions.forEach(transformer::serializePartition);
 json.writeEndArray();
 }
 }
 
-public static void keysToJson(ISSTableScanner currentScanner, 
Stream keys, CFMetaData metadata, OutputStream out) throws 
IOException
+public static void keysToJson(ISSTableScanner currentScanner, 
Stream keys, boolean rawTime, CFMetaData metadata, OutputStream 
out) throws IOException
 {
 try (JsonGenerator 

[3/3] cassandra git commit: Merge branch 'cassandra-3.0' into trunk

[yukim]

Merge branch 'cassandra-3.0' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c6778c5a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c6778c5a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c6778c5a

Branch: refs/heads/trunk
Commit: c6778c5afb2456b0f625bc05de9a8caffc2898d8
Parents: ff4d0f9 620efdc
Author: Yuki Morishita 
Authored: Fri Apr 29 14:33:41 2016 -0500
Committer: Yuki Morishita 
Committed: Fri Apr 29 14:33:41 2016 -0500

--
 CHANGES.txt |   1 +
 .../apache/cassandra/tools/JsonTransformer.java | 112 +--
 .../apache/cassandra/tools/SSTableExport.java   |  11 +-
 3 files changed, 85 insertions(+), 39 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/c6778c5a/CHANGES.txt
--
diff --cc CHANGES.txt
index 8ee39dc,64bcbd8..b1ca6dd
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,67 -1,5 +1,68 @@@
 -3.0.6
 +3.6
 + * Add units to stress ouput (CASSANDRA-11352)
 + * Fix PER PARTITION LIMIT for single and multi partitions queries 
(CASSANDRA-11603)
 + * Add uncompressed chunk cache for RandomAccessReader (CASSANDRA-5863)
 + * Clarify ClusteringPrefix hierarchy (CASSANDRA-11213)
 + * Always perform collision check before joining ring (CASSANDRA-10134)
 + * SSTableWriter output discrepancy (CASSANDRA-11646)
 + * Fix potential timeout in NativeTransportService.testConcurrentDestroys 
(CASSANDRA-10756)
 + * Support large partitions on the 3.0 sstable format (CASSANDRA-11206)
 + * Add support to rebuild from specific range (CASSANDRA-10406)
 + * Optimize the overlapping lookup by calculating all the
 +   bounds in advance (CASSANDRA-11571)
 + * Support json/yaml output in noetool tablestats (CASSANDRA-5977)
 + * (stress) Add datacenter option to -node options (CASSANDRA-11591)
 + * Fix handling of empty slices (CASSANDRA-11513)
 + * Make number of cores used by cqlsh COPY visible to testing code 
(CASSANDRA-11437)
 + * Allow filtering on clustering columns for queries without secondary 
indexes (CASSANDRA-11310)
 + * Refactor Restriction hierarchy (CASSANDRA-11354)
 + * Eliminate allocations in R/W path (CASSANDRA-11421)
 + * Update Netty to 4.0.36 (CASSANDRA-11567)
 + * Fix PER PARTITION LIMIT for queries requiring post-query ordering 
(CASSANDRA-11556)
 + * Allow instantiation of UDTs and tuples in UDFs (CASSANDRA-10818)
 + * Support UDT in CQLSSTableWriter (CASSANDRA-10624)
 + * Support for non-frozen user-defined types, updating
 +   individual fields of user-defined types (CASSANDRA-7423)
 + * Make LZ4 compression level configurable (CASSANDRA-11051)
 + * Allow per-partition LIMIT clause in CQL (CASSANDRA-7017)
 + * Make custom filtering more extensible with UserExpression (CASSANDRA-11295)
 + * Improve field-checking and error reporting in cassandra.yaml 
(CASSANDRA-10649)
 + * Print CAS stats in nodetool proxyhistograms (CASSANDRA-11507)
 + * More user friendly error when providing an invalid token to nodetool 
(CASSANDRA-9348)
 + * Add static column support to SASI index (CASSANDRA-11183)
 + * Support EQ/PREFIX queries in SASI CONTAINS mode without tokenization 
(CASSANDRA-11434)
 + * Support LIKE operator in prepared statements (CASSANDRA-11456)
 + * Add a command to see if a Materialized View has finished building 
(CASSANDRA-9967)
 + * Log endpoint and port associated with streaming operation (CASSANDRA-8777)
 + * Print sensible units for all log messages (CASSANDRA-9692)
 + * Upgrade Netty to version 4.0.34 (CASSANDRA-11096)
 + * Break the CQL grammar into separate Parser and Lexer (CASSANDRA-11372)
 + * Compress only inter-dc traffic by default (CASSANDRA-)
 + * Add metrics to track write amplification (CASSANDRA-11420)
 + * cassandra-stress: cannot handle "value-less" tables (CASSANDRA-7739)
 + * Add/drop multiple columns in one ALTER TABLE statement (CASSANDRA-10411)
 + * Add require_endpoint_verification opt for internode encryption 
(CASSANDRA-9220)
 + * Add auto import java.util for UDF code block (CASSANDRA-11392)
 + * Add --hex-format option to nodetool getsstables (CASSANDRA-11337)
 + * sstablemetadata should print sstable min/max token (CASSANDRA-7159)
 + * Do not wrap CassandraException in TriggerExecutor (CASSANDRA-9421)
 + * COPY TO should have higher double precision (CASSANDRA-11255)
 + * Stress should exit with non-zero status after failure (CASSANDRA-10340)
 + * Add client to cqlsh SHOW_SESSION (CASSANDRA-8958)
 + * Fix nodetool tablestats keyspace level metrics (CASSANDRA-11226)
 + * Store repair options in parent_repair_history (CASSANDRA-11244)
 + * Print current leveling in sstableofflinerelevel (CASSANDRA-9588)
 + * Change repair message for keyspaces with RF 

[jira] [Created] (CASSANDRA-11685) (windows) dtest failure in replace_address_test.TestReplaceAddress.replace_shutdown_node_test

[Russ Hatch (JIRA)]

Russ Hatch created CASSANDRA-11685:
--

 Summary: (windows) dtest failure in 
replace_address_test.TestReplaceAddress.replace_shutdown_node_test
 Key: CASSANDRA-11685
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11685
 Project: Cassandra
  Issue Type: Test
Reporter: Russ Hatch
Assignee: DS Test Eng


single recent failure, may just be a routine timeout type issue:
{noformat}
('Unable to complete the operation against any hosts', {: ConnectionException('Host has been marked down or removed',)})
 >> begin captured logging << 
dtest: DEBUG: cluster ccm directory: d:\temp\dtest-ceprzc
dtest: DEBUG: Custom init_config not found. Setting defaults.
dtest: DEBUG: Done setting configuration options:
{   'initial_token': None,
'num_tokens': '32',
'phi_convict_threshold': 5,
'start_rpc': 'true'}
dtest: DEBUG: Starting cluster with 3 nodes.
dtest: DEBUG: 32
dtest: DEBUG: Inserting Data...
{noformat}
http://cassci.datastax.com/job/cassandra-2.2_dtest_win32/233/testReport/replace_address_test/TestReplaceAddress/replace_shutdown_node_test

Failed on CassCI build cassandra-2.2_dtest_win32 #233



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8488) Filter by UDF

[Brian Hess (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264523#comment-15264523
 ] 

 Brian Hess commented on CASSANDRA-8488:


There is a way around CASSANDRA-8273 for UDFs, but it might not be the best 
(though, then again, there doesn't appear to be a solution for 8273).  We could 
not have replica-based filtering if the filter is a UDF.  That is, you will 
bring all rows that meet the *other* filters back to the coordinator and do the 
filtering of the UDF(s) on the replica after ConsistencyLevel is applied.

> Filter by UDF
> -
>
> Key: CASSANDRA-8488
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8488
> Project: Cassandra
>  Issue Type: New Feature
>  Components: CQL, Local Write-Read Paths
>Reporter: Jonathan Ellis
>  Labels: client-impacting, cql, udf
> Fix For: 3.x
>
>
> Allow user-defined functions in WHERE clause with ALLOW FILTERING.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8844) Change Data Capture (CDC)

[Joshua McKenzie (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264516#comment-15264516
 ] 

Joshua McKenzie commented on CASSANDRA-8844:


bq. How will cdc space be measured?
2 contexts:
# from C*, same as the current branch. Check on discard, check on failed alloc 
if CDC on, only care about cdc_overflow.
# from compactor, check size of target folder / config in options, if at limit, 
just stop compacting. Backpressure flows into C*, writes for cdc-enabled get 
rejected.

bq. We will have to reject all updates...
Initial thoughts on ways around this: On segment creation, if there is not room 
to accommodate another CLS in cdc_overflow we can flag the segment as not 
accepting CDC writes. .allocate calls on Mutation containing CDC-enabled CF's 
reject if acceptingCDCWrites is false, use current logic to re-check on-disk 
size w/RateLimiter, and if cdc_overflow + segment size <= allowable on return, 
we toggle accepting on the active segment back on. On a successful CDC 
mutations, we idempotently enable a different bool to indicate that a segment 
has accepted a CDC write. On segment discard, check if segment accepted any CDC 
writes and, if so, move to cdc_overflow. If not, delete.

Few immediate concerns I see
# idempotent hammering of volatile bool on each CDC mutation write probably has 
some perf implications
# possibility of overallocation since there's a temporal split between when we 
allocate a segment and when we discard it
# Determining whether or not a Mutation is CDC-enabled w/out sequential 
scanning contained PartitionUpdate CFMetaData might matter. Might not. Worth 
microbenching, and it'd only affect nodes w/CDC-enabled anyway.
# Doesn't really allow for toggling CDC acceptance back off during run, but I'm 
personally fine w/us tolerating an extra segment or two slipping into 
cdc_overflow past the allowable limit rather than constantly checking and 
toggling CDC acceptance on and off in a segment.

bq. Not sure how we can handle someone turning off CDC and then restarting 
their daemon
Maybe flush all cdc-enabled CF on daemon shutdown to prevent this? Or document 
it and recommend flushing all CDC-enabled CF before toggling the setting.

bq. How will we guarantee atomicity between the overflow and the compacted 
files?
Don't think we have to. At-least-once semantics should allow an edge-case like 
this where some data gets re-compacted and delivered multiple times. It's a 
little cpu overhead but not worth tackling on the complexity front IMO.

> Change Data Capture (CDC)
> -
>
> Key: CASSANDRA-8844
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8844
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Coordination, Local Write-Read Paths
>Reporter: Tupshin Harper
>Assignee: Joshua McKenzie
>Priority: Critical
> Fix For: 3.x
>
>
> "In databases, change data capture (CDC) is a set of software design patterns 
> used to determine (and track) the data that has changed so that action can be 
> taken using the changed data. Also, Change data capture (CDC) is an approach 
> to data integration that is based on the identification, capture and delivery 
> of the changes made to enterprise data sources."
> -Wikipedia
> As Cassandra is increasingly being used as the Source of Record (SoR) for 
> mission critical data in large enterprises, it is increasingly being called 
> upon to act as the central hub of traffic and data flow to other systems. In 
> order to try to address the general need, we (cc [~brianmhess]), propose 
> implementing a simple data logging mechanism to enable per-table CDC patterns.
> h2. The goals:
> # Use CQL as the primary ingestion mechanism, in order to leverage its 
> Consistency Level semantics, and in order to treat it as the single 
> reliable/durable SoR for the data.
> # To provide a mechanism for implementing good and reliable 
> (deliver-at-least-once with possible mechanisms for deliver-exactly-once ) 
> continuous semi-realtime feeds of mutations going into a Cassandra cluster.
> # To eliminate the developmental and operational burden of users so that they 
> don't have to do dual writes to other systems.
> # For users that are currently doing batch export from a Cassandra system, 
> give them the opportunity to make that realtime with a minimum of coding.
> h2. The mechanism:
> We propose a durable logging mechanism that functions similar to a commitlog, 
> with the following nuances:
> - Takes place on every node, not just the coordinator, so RF number of copies 
> are logged.
> - Separate log per table.
> - Per-table configuration. Only tables that are specified as CDC_LOG would do 
> any logging.
> - Per DC. We are trying to keep the complexity to a minimum to make this an 
> easy enhancement, but 

[jira] [Updated] (CASSANDRA-8844) Change Data Capture (CDC)

[Joshua McKenzie (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua McKenzie updated CASSANDRA-8844:
---
Status: Open  (was: Patch Available)

> Change Data Capture (CDC)
> -
>
> Key: CASSANDRA-8844
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8844
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Coordination, Local Write-Read Paths
>Reporter: Tupshin Harper
>Assignee: Joshua McKenzie
>Priority: Critical
> Fix For: 3.x
>
>
> "In databases, change data capture (CDC) is a set of software design patterns 
> used to determine (and track) the data that has changed so that action can be 
> taken using the changed data. Also, Change data capture (CDC) is an approach 
> to data integration that is based on the identification, capture and delivery 
> of the changes made to enterprise data sources."
> -Wikipedia
> As Cassandra is increasingly being used as the Source of Record (SoR) for 
> mission critical data in large enterprises, it is increasingly being called 
> upon to act as the central hub of traffic and data flow to other systems. In 
> order to try to address the general need, we (cc [~brianmhess]), propose 
> implementing a simple data logging mechanism to enable per-table CDC patterns.
> h2. The goals:
> # Use CQL as the primary ingestion mechanism, in order to leverage its 
> Consistency Level semantics, and in order to treat it as the single 
> reliable/durable SoR for the data.
> # To provide a mechanism for implementing good and reliable 
> (deliver-at-least-once with possible mechanisms for deliver-exactly-once ) 
> continuous semi-realtime feeds of mutations going into a Cassandra cluster.
> # To eliminate the developmental and operational burden of users so that they 
> don't have to do dual writes to other systems.
> # For users that are currently doing batch export from a Cassandra system, 
> give them the opportunity to make that realtime with a minimum of coding.
> h2. The mechanism:
> We propose a durable logging mechanism that functions similar to a commitlog, 
> with the following nuances:
> - Takes place on every node, not just the coordinator, so RF number of copies 
> are logged.
> - Separate log per table.
> - Per-table configuration. Only tables that are specified as CDC_LOG would do 
> any logging.
> - Per DC. We are trying to keep the complexity to a minimum to make this an 
> easy enhancement, but most likely use cases would prefer to only implement 
> CDC logging in one (or a subset) of the DCs that are being replicated to
> - In the critical path of ConsistencyLevel acknowledgment. Just as with the 
> commitlog, failure to write to the CDC log should fail that node's write. If 
> that means the requested consistency level was not met, then clients *should* 
> experience UnavailableExceptions.
> - Be written in a Row-centric manner such that it is easy for consumers to 
> reconstitute rows atomically.
> - Written in a simple format designed to be consumed *directly* by daemons 
> written in non JVM languages
> h2. Nice-to-haves
> I strongly suspect that the following features will be asked for, but I also 
> believe that they can be deferred for a subsequent release, and to guage 
> actual interest.
> - Multiple logs per table. This would make it easy to have multiple 
> "subscribers" to a single table's changes. A workaround would be to create a 
> forking daemon listener, but that's not a great answer.
> - Log filtering. Being able to apply filters, including UDF-based filters 
> would make Casandra a much more versatile feeder into other systems, and 
> again, reduce complexity that would otherwise need to be built into the 
> daemons.
> h2. Format and Consumption
> - Cassandra would only write to the CDC log, and never delete from it. 
> - Cleaning up consumed logfiles would be the client daemon's responibility
> - Logfile size should probably be configurable.
> - Logfiles should be named with a predictable naming schema, making it 
> triivial to process them in order.
> - Daemons should be able to checkpoint their work, and resume from where they 
> left off. This means they would have to leave some file artifact in the CDC 
> log's directory.
> - A sophisticated daemon should be able to be written that could 
> -- Catch up, in written-order, even when it is multiple logfiles behind in 
> processing
> -- Be able to continuously "tail" the most recent logfile and get 
> low-latency(ms?) access to the data as it is written.
> h2. Alternate approach
> In order to make consuming a change log easy and efficient to do with low 
> latency, the following could supplement the approach outlined above
> - Instead of writing to a logfile, by default, Cassandra could expose a 
> socket for a daemon to connect to, and from 

[jira] [Commented] (CASSANDRA-11597) dtest failure in upgrade_supercolumns_test.TestSCUpgrade.upgrade_with_counters_test

[Russ Hatch (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264498#comment-15264498
 ] 

Russ Hatch commented on CASSANDRA-11597:


pr merged. this test will now use flaky so it can retry in narrow circumstances 
when dealing with routine timeouts.

> dtest failure in 
> upgrade_supercolumns_test.TestSCUpgrade.upgrade_with_counters_test
> ---
>
> Key: CASSANDRA-11597
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11597
> Project: Cassandra
>  Issue Type: Test
>Reporter: Jim Witschey
>Assignee: Russ Hatch
>  Labels: dtest
>
> Looks like a new flap. Example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/447/testReport/upgrade_supercolumns_test/TestSCUpgrade/upgrade_with_counters_test
> Failed on CassCI build cassandra-2.1_dtest #447 - 2.1.14-tentative
> {code}
> Error Message
> TimedOutException(acknowledged_by=0, paxos_in_progress=None, 
> acknowledged_by_batchlog=None)
>  >> begin captured logging << 
> dtest: DEBUG: cluster ccm directory: /mnt/tmp/dtest-1Fi9qz
> dtest: DEBUG: Custom init_config not found. Setting defaults.
> dtest: DEBUG: Done setting configuration options:
> {   'initial_token': None,
> 'num_tokens': '32',
> 'phi_convict_threshold': 5,
> 'range_request_timeout_in_ms': 1,
> 'read_request_timeout_in_ms': 1,
> 'request_timeout_in_ms': 1,
> 'truncate_request_timeout_in_ms': 1,
> 'write_request_timeout_in_ms': 1}
> dtest: DEBUG: Upgrading to binary:2.0.17
> dtest: DEBUG: Shutting down node: node1
> dtest: DEBUG: Set new cassandra dir for node1: 
> /home/automaton/.ccm/repository/2.0.17
> dtest: DEBUG: Starting node1 on new version (binary:2.0.17)
> - >> end captured logging << -
> Stacktrace
>   File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
>   File "/home/automaton/cassandra-dtest/upgrade_supercolumns_test.py", line 
> 215, in upgrade_with_counters_test
> client.add('Counter1', column_parent, column, ThriftConsistencyLevel.ONE)
>   File "/home/automaton/cassandra-dtest/thrift_bindings/v22/Cassandra.py", 
> line 985, in add
> self.recv_add()
>   File "/home/automaton/cassandra-dtest/thrift_bindings/v22/Cassandra.py", 
> line 1013, in recv_add
> raise result.te
> "TimedOutException(acknowledged_by=0, paxos_in_progress=None, 
> acknowledged_by_batchlog=None)\n >> begin captured 
> logging << \ndtest: DEBUG: cluster ccm directory: 
> /mnt/tmp/dtest-1Fi9qz\ndtest: DEBUG: Custom init_config not found. Setting 
> defaults.\ndtest: DEBUG: Done setting configuration options:\n{   
> 'initial_token': None,\n'num_tokens': '32',\n'phi_convict_threshold': 
> 5,\n'range_request_timeout_in_ms': 1,\n
> 'read_request_timeout_in_ms': 1,\n'request_timeout_in_ms': 1,\n   
>  'truncate_request_timeout_in_ms': 1,\n'write_request_timeout_in_ms': 
> 1}\ndtest: DEBUG: Upgrading to binary:2.0.17\ndtest: DEBUG: Shutting down 
> node: node1\ndtest: DEBUG: Set new cassandra dir for node1: 
> /home/automaton/.ccm/repository/2.0.17\ndtest: DEBUG: Starting node1 on new 
> version (binary:2.0.17)\n- >> end captured logging << 
> -"
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-11597) dtest failure in upgrade_supercolumns_test.TestSCUpgrade.upgrade_with_counters_test

[Russ Hatch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russ Hatch resolved CASSANDRA-11597.

Resolution: Fixed

> dtest failure in 
> upgrade_supercolumns_test.TestSCUpgrade.upgrade_with_counters_test
> ---
>
> Key: CASSANDRA-11597
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11597
> Project: Cassandra
>  Issue Type: Test
>Reporter: Jim Witschey
>Assignee: Russ Hatch
>  Labels: dtest
>
> Looks like a new flap. Example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/447/testReport/upgrade_supercolumns_test/TestSCUpgrade/upgrade_with_counters_test
> Failed on CassCI build cassandra-2.1_dtest #447 - 2.1.14-tentative
> {code}
> Error Message
> TimedOutException(acknowledged_by=0, paxos_in_progress=None, 
> acknowledged_by_batchlog=None)
>  >> begin captured logging << 
> dtest: DEBUG: cluster ccm directory: /mnt/tmp/dtest-1Fi9qz
> dtest: DEBUG: Custom init_config not found. Setting defaults.
> dtest: DEBUG: Done setting configuration options:
> {   'initial_token': None,
> 'num_tokens': '32',
> 'phi_convict_threshold': 5,
> 'range_request_timeout_in_ms': 1,
> 'read_request_timeout_in_ms': 1,
> 'request_timeout_in_ms': 1,
> 'truncate_request_timeout_in_ms': 1,
> 'write_request_timeout_in_ms': 1}
> dtest: DEBUG: Upgrading to binary:2.0.17
> dtest: DEBUG: Shutting down node: node1
> dtest: DEBUG: Set new cassandra dir for node1: 
> /home/automaton/.ccm/repository/2.0.17
> dtest: DEBUG: Starting node1 on new version (binary:2.0.17)
> - >> end captured logging << -
> Stacktrace
>   File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
>   File "/home/automaton/cassandra-dtest/upgrade_supercolumns_test.py", line 
> 215, in upgrade_with_counters_test
> client.add('Counter1', column_parent, column, ThriftConsistencyLevel.ONE)
>   File "/home/automaton/cassandra-dtest/thrift_bindings/v22/Cassandra.py", 
> line 985, in add
> self.recv_add()
>   File "/home/automaton/cassandra-dtest/thrift_bindings/v22/Cassandra.py", 
> line 1013, in recv_add
> raise result.te
> "TimedOutException(acknowledged_by=0, paxos_in_progress=None, 
> acknowledged_by_batchlog=None)\n >> begin captured 
> logging << \ndtest: DEBUG: cluster ccm directory: 
> /mnt/tmp/dtest-1Fi9qz\ndtest: DEBUG: Custom init_config not found. Setting 
> defaults.\ndtest: DEBUG: Done setting configuration options:\n{   
> 'initial_token': None,\n'num_tokens': '32',\n'phi_convict_threshold': 
> 5,\n'range_request_timeout_in_ms': 1,\n
> 'read_request_timeout_in_ms': 1,\n'request_timeout_in_ms': 1,\n   
>  'truncate_request_timeout_in_ms': 1,\n'write_request_timeout_in_ms': 
> 1}\ndtest: DEBUG: Upgrading to binary:2.0.17\ndtest: DEBUG: Shutting down 
> node: node1\ndtest: DEBUG: Set new cassandra dir for node1: 
> /home/automaton/.ccm/repository/2.0.17\ndtest: DEBUG: Starting node1 on new 
> version (binary:2.0.17)\n- >> end captured logging << 
> -"
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11679) Cassandra Driver returns different number of results depending on fetchsize

[Varun Barala (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264372#comment-15264372
 ] 

Varun Barala commented on CASSANDRA-11679:
--

Yes I tried 100 as a fetch size and it was returning 503 distinct keys. In fact 
for all fetch size <=498 it returns 503 distinct keys. 

I didn't check the impact on the number of total rows returned But If `select 
*` query fetches the results corresponding to all distinct keys then yes It 
will have an impact.

> Cassandra Driver returns different number of results depending on fetchsize
> ---
>
> Key: CASSANDRA-11679
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11679
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Varun Barala
>Assignee: Benjamin Lerer
>
> I'm trying to fetch all distinct keys from a CF using cassandra-driver 
> (2.1.7.1) and I observed some strange behavior :-
> The total distinct rows are 498 so If I perform a query get All distinctKeys 
> It return 503 instead of 498(five keys twice).
> But If I define the fetch size in select statement more than 498 then it 
> returns exact 498 rows. 
> And If I execute same statement on Dev-center it returns 498 rows.
> Some Additional and useful information :- 
> ---
> Cassandra-2.1.13  (C)* version
> Consistency level: ONE 
> local machine(ubuntu 14.04)
> Table Schema:-
> --
> {code:xml}
> CREATE TABLE sample (
>  pk1 text,
>  pk2 text,
> row_id uuid,
> value blob,
> PRIMARY KEY (( pk1,  pk2))
> ) WITH bloom_filter_fp_chance = 0.01
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = ''
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
> AND compression = {'sstable_compression': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
> {code}
> query :-
> 
> {code:xml}
> SELECT DISTINCT  pk2, pk1 FROM sample LIMIT 2147483647;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-11677) Incredibly slow jolokia response times

[Andrew Jorgensen (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264356#comment-15264356
 ] 

Andrew Jorgensen edited comment on CASSANDRA-11677 at 4/29/16 5:14 PM:
---

Not great, much slower than the jolokia 1.2.3 client. So i think there must 
have been a performance degradation in jolokia. I have a ticket open on the 
jolokia github repository to talk about the issue. We ended up being able to 
work around the issue by upgrading to the master version of the diamond jolokia 
collector and kept the updated jolokia agent which seems to have resolved to 
issue.

Thanks so much for your help. I think we can close this issue as not a problem, 
It'll be good for people to be able to search for posterity.


was (Author: ajorgensen):
Not great, much slower than the jolokia 1.2.3 client. So i think there must 
have been a performance degradation in jolokia. I have a ticket open on the 
jolokia github repository to talk about the issue. We ended up being able to 
work around the issue by upgrading to the master version of the diamond jolokia 
collector and kept the updated jolokia agent which seems to have resolved to 
issue.

Thanks so much for your help.

> Incredibly slow jolokia response times
> --
>
> Key: CASSANDRA-11677
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11677
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Andrew Jorgensen
>
> I am seeing some very slow jolokia request times on my Cassandra 3.0 cluster. 
> Specifically when running the following:
> {code}
> curl 127.0.0.1:8778/jolokia/list
> {code}
> on a slightly loaded cluster I am seeing request times around 30-40 seconds 
> and on a more heavily loaded cluster I am seeing request times in the 2 
> minute mark. We are currently using jolokia 1.3.2 and v4 of the diamond 
> collector. I also have a Cassandra 1.1 cluster that has the same load and 
> number of nodes and running the same curl command comes back in about 1 
> second.
> Is there anything I can do to help diagnose this issue to see what is causing 
> the slowdown or has anyone else experience this?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-11677) Incredibly slow jolokia response times

[Andrew Jorgensen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Jorgensen resolved CASSANDRA-11677.
--
Resolution: Not A Problem

> Incredibly slow jolokia response times
> --
>
> Key: CASSANDRA-11677
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11677
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Andrew Jorgensen
>
> I am seeing some very slow jolokia request times on my Cassandra 3.0 cluster. 
> Specifically when running the following:
> {code}
> curl 127.0.0.1:8778/jolokia/list
> {code}
> on a slightly loaded cluster I am seeing request times around 30-40 seconds 
> and on a more heavily loaded cluster I am seeing request times in the 2 
> minute mark. We are currently using jolokia 1.3.2 and v4 of the diamond 
> collector. I also have a Cassandra 1.1 cluster that has the same load and 
> number of nodes and running the same curl command comes back in about 1 
> second.
> Is there anything I can do to help diagnose this issue to see what is causing 
> the slowdown or has anyone else experience this?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11677) Incredibly slow jolokia response times

[Andrew Jorgensen (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264356#comment-15264356
 ] 

Andrew Jorgensen commented on CASSANDRA-11677:
--

Not great, much slower than the jolokia 1.2.3 client. So i think there must 
have been a performance degradation in jolokia. I have a ticket open on the 
jolokia github repository to talk about the issue. We ended up being able to 
work around the issue by upgrading to the master version of the diamond jolokia 
collector and kept the updated jolokia agent which seems to have resolved to 
issue.

Thanks so much for your help.

> Incredibly slow jolokia response times
> --
>
> Key: CASSANDRA-11677
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11677
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Andrew Jorgensen
>
> I am seeing some very slow jolokia request times on my Cassandra 3.0 cluster. 
> Specifically when running the following:
> {code}
> curl 127.0.0.1:8778/jolokia/list
> {code}
> on a slightly loaded cluster I am seeing request times around 30-40 seconds 
> and on a more heavily loaded cluster I am seeing request times in the 2 
> minute mark. We are currently using jolokia 1.3.2 and v4 of the diamond 
> collector. I also have a Cassandra 1.1 cluster that has the same load and 
> number of nodes and running the same curl command comes back in about 1 
> second.
> Is there anything I can do to help diagnose this issue to see what is causing 
> the slowdown or has anyone else experience this?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-11678) cassandra crush when enable hints_compression

[Blake Eggleston (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264347#comment-15264347
 ] 

Blake Eggleston edited comment on CASSANDRA-11678 at 4/29/16 5:10 PM:
--

[~linweiji...@gmail.com] I haven't had any luck duplicating this so far, can 
you post all of the hints settings in your cassandra.yaml? Also, is this an 
isolated incident, or does it happen regularly?


was (Author: bdeggleston):
[~linweiji...@gmail.com] I haven't had any luck duplicating this so far, can 
you post the relevant section of your cassandra.yaml? Also, is this an isolated 
incident, or does it happen regularly?

> cassandra crush when enable hints_compression
> -
>
> Key: CASSANDRA-11678
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11678
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core, Local Write-Read Paths
> Environment: Centos 7
>Reporter: Weijian Lin
>Assignee: Blake Eggleston
>Priority: Critical
>
> When I enable hints_compression and set the compression class to
> LZ4Compressor，the
> cassandra (v3.05, V3.5.0) will crush。That is a bug, or any conf is wrong?
> *Exception   in V 3.5.0  *
> ERROR [HintsDispatcher:2] 2016-04-26 15:02:56,970
> HintsDispatchExecutor.java:225 - Failed to dispatch hints file
> abc4dda2-b551-427e-bb0b-e383d4a392e1-1461654138963-1.hints: file is
> corrupted ({})
> org.apache.cassandra.io.FSReadError: java.io.EOFException
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:284)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:254)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:91)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:259)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:242)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:220)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:199)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_65]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> Caused by: java.io.EOFException: null
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:146)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readPrimitiveSlowly(RebufferingInputStream.java:108)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readInt(RebufferingInputStream.java:188)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNextInternal(HintsReader.java:297)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:280)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 15 common frames omitted
> *Exception   in V 3.0.5  *
> ERROR [HintsDispatcher:2] 2016-04-26 15:54:46,294
> HintsDispatchExecutor.java:225 - Failed to dispatch hints file
> 8603be13-6878-4de3-8bc3-a7a7146b0376-1461657251205-1.hints: file is
> corrupted ({})
> org.apache.cassandra.io.FSReadError: java.io.EOFException
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:282)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:252)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> 

[jira] [Commented] (CASSANDRA-11678) cassandra crush when enable hints_compression

[Blake Eggleston (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264347#comment-15264347
 ] 

Blake Eggleston commented on CASSANDRA-11678:
-

[~linweiji...@gmail.com] I haven't had any luck duplicating this so far, can 
you post the relevant section of your cassandra.yaml? Also, is this an isolated 
incident, or does it happen regularly?

> cassandra crush when enable hints_compression
> -
>
> Key: CASSANDRA-11678
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11678
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core, Local Write-Read Paths
> Environment: Centos 7
>Reporter: Weijian Lin
>Assignee: Blake Eggleston
>Priority: Critical
>
> When I enable hints_compression and set the compression class to
> LZ4Compressor，the
> cassandra (v3.05, V3.5.0) will crush。That is a bug, or any conf is wrong?
> *Exception   in V 3.5.0  *
> ERROR [HintsDispatcher:2] 2016-04-26 15:02:56,970
> HintsDispatchExecutor.java:225 - Failed to dispatch hints file
> abc4dda2-b551-427e-bb0b-e383d4a392e1-1461654138963-1.hints: file is
> corrupted ({})
> org.apache.cassandra.io.FSReadError: java.io.EOFException
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:284)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:254)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:91)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:259)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:242)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:220)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:199)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_65]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> Caused by: java.io.EOFException: null
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:146)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readPrimitiveSlowly(RebufferingInputStream.java:108)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readInt(RebufferingInputStream.java:188)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNextInternal(HintsReader.java:297)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:280)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 15 common frames omitted
> *Exception   in V 3.0.5  *
> ERROR [HintsDispatcher:2] 2016-04-26 15:54:46,294
> HintsDispatchExecutor.java:225 - Failed to dispatch hints file
> 8603be13-6878-4de3-8bc3-a7a7146b0376-1461657251205-1.hints: file is
> corrupted ({})
> org.apache.cassandra.io.FSReadError: java.io.EOFException
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:282)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:252)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137)
> 

[jira] [Commented] (CASSANDRA-11609) Nested UDTs cause error when migrating 2.x schema to trunk

[Benjamin Lerer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264326#comment-15264326
 ] 

Benjamin Lerer commented on CASSANDRA-11609:


+1

> Nested UDTs cause error when migrating 2.x schema to trunk
> --
>
> Key: CASSANDRA-11609
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11609
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Russ Hatch
>Assignee: Tyler Hobbs
> Fix For: 3.x
>
>
> This was found in the upgrades user_types_test.
> Can also be repro'd with ccm.
> To repro using ccm:
> Create a 1 node cluster on 2.2.x
> Create this schema:
> {noformat}
> create keyspace test2 with replication = {'class':'SimpleStrategy', 
> 'replication_factor':1};
> use test2;
> CREATE TYPE address (
>  street text,
>  city text,
>  zip_code int,
>  phones set
>  );
> CREATE TYPE fullname (
>  irstname text,
>  astname text
>  );
> CREATE TABLE users (
>  d uuid PRIMARY KEY,
>  ame frozen,
>  ddresses map
>  );
> {noformat}
> Upgrade the single node to trunk, attempt to start the node up. Start will 
> fail with this exception:
> {noformat}
> ERROR [main] 2016-04-19 11:33:19,218 CassandraDaemon.java:704 - Exception 
> encountered during startup
> org.apache.cassandra.exceptions.InvalidRequestException: Non-frozen UDTs are 
> not allowed inside collections: map
> at 
> org.apache.cassandra.cql3.CQL3Type$Raw$RawCollection.throwNestedNonFrozenError(CQL3Type.java:686)
>  ~[main/:na]
> at 
> org.apache.cassandra.cql3.CQL3Type$Raw$RawCollection.prepare(CQL3Type.java:652)
>  ~[main/:na]
> at 
> org.apache.cassandra.cql3.CQL3Type$Raw$RawCollection.prepareInternal(CQL3Type.java:644)
>  ~[main/:na]
> at 
> org.apache.cassandra.schema.CQLTypeParser.parse(CQLTypeParser.java:53) 
> ~[main/:na]
> at 
> org.apache.cassandra.schema.SchemaKeyspace.createColumnFromRow(SchemaKeyspace.java:1022)
>  ~[main/:na]
> at 
> org.apache.cassandra.schema.SchemaKeyspace.lambda$fetchColumns$12(SchemaKeyspace.java:1006)
>  ~[main/:na]
> at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_77]
> at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchColumns(SchemaKeyspace.java:1006)
>  ~[main/:na]
> at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:960)
>  ~[main/:na]
> at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:939)
>  ~[main/:na]
> at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:902)
>  ~[main/:na]
> at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:879)
>  ~[main/:na]
> at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:867)
>  ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:134) 
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:124) 
> ~[main/:na]
> at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:229) 
> [main/:na]
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:558)
>  [main/:na]
> at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:687) 
> [main/:na]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11679) Cassandra Driver returns different number of results depending on fetchsize

[Benjamin Lerer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264315#comment-15264315
 ] 

Benjamin Lerer commented on CASSANDRA-11679:


If I am not mistaken, the default fetch size in the java driver should be 5000. 
Which is more than the number of rows that you have. Did you use another value 
like 100?
What is the reason for your limit clause? Does it have an impact on the number 
of row returned?


> Cassandra Driver returns different number of results depending on fetchsize
> ---
>
> Key: CASSANDRA-11679
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11679
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Varun Barala
>Assignee: Benjamin Lerer
>
> I'm trying to fetch all distinct keys from a CF using cassandra-driver 
> (2.1.7.1) and I observed some strange behavior :-
> The total distinct rows are 498 so If I perform a query get All distinctKeys 
> It return 503 instead of 498(five keys twice).
> But If I define the fetch size in select statement more than 498 then it 
> returns exact 498 rows. 
> And If I execute same statement on Dev-center it returns 498 rows.
> Some Additional and useful information :- 
> ---
> Cassandra-2.1.13  (C)* version
> Consistency level: ONE 
> local machine(ubuntu 14.04)
> Table Schema:-
> --
> {code:xml}
> CREATE TABLE sample (
>  pk1 text,
>  pk2 text,
> row_id uuid,
> value blob,
> PRIMARY KEY (( pk1,  pk2))
> ) WITH bloom_filter_fp_chance = 0.01
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = ''
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
> AND compression = {'sstable_compression': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
> {code}
> query :-
> 
> {code:xml}
> SELECT DISTINCT  pk2, pk1 FROM sample LIMIT 2147483647;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11678) cassandra crush when enable hints_compression

[Jeremiah Jordan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeremiah Jordan updated CASSANDRA-11678:

Assignee: Blake Eggleston

> cassandra crush when enable hints_compression
> -
>
> Key: CASSANDRA-11678
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11678
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core, Local Write-Read Paths
> Environment: Centos 7
>Reporter: Weijian Lin
>Assignee: Blake Eggleston
>Priority: Critical
>
> When I enable hints_compression and set the compression class to
> LZ4Compressor，the
> cassandra (v3.05, V3.5.0) will crush。That is a bug, or any conf is wrong?
> *Exception   in V 3.5.0  *
> ERROR [HintsDispatcher:2] 2016-04-26 15:02:56,970
> HintsDispatchExecutor.java:225 - Failed to dispatch hints file
> abc4dda2-b551-427e-bb0b-e383d4a392e1-1461654138963-1.hints: file is
> corrupted ({})
> org.apache.cassandra.io.FSReadError: java.io.EOFException
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:284)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:254)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:91)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:259)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:242)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:220)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:199)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_65]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> Caused by: java.io.EOFException: null
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:146)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readPrimitiveSlowly(RebufferingInputStream.java:108)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readInt(RebufferingInputStream.java:188)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNextInternal(HintsReader.java:297)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:280)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 15 common frames omitted
> *Exception   in V 3.0.5  *
> ERROR [HintsDispatcher:2] 2016-04-26 15:54:46,294
> HintsDispatchExecutor.java:225 - Failed to dispatch hints file
> 8603be13-6878-4de3-8bc3-a7a7146b0376-1461657251205-1.hints: file is
> corrupted ({})
> org.apache.cassandra.io.FSReadError: java.io.EOFException
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:282)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:252)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> 

[jira] [Updated] (CASSANDRA-9830) Option to disable bloom filter in highest level of LCS sstables

[Paulo Motta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-9830:
---
Status: Patch Available  (was: Open)

> Option to disable bloom filter in highest level of LCS sstables
> ---
>
> Key: CASSANDRA-9830
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9830
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Compaction
>Reporter: Jonathan Ellis
>Assignee: Paulo Motta
>Priority: Minor
>  Labels: performance
> Fix For: 3.x
>
>
> We expect about 90% of data to be in the highest level of LCS in a fully 
> populated series.  (See also CASSANDRA-9829.)
> Thus if the user is primarily asking for data (partitions) that has actually 
> been inserted, the bloom filter on the highest level only helps reject 
> sstables about 10% of the time.
> We should add an option that suppresses bloom filter creation on top-level 
> sstables.  This will dramatically reduce memory usage for LCS and may even 
> improve performance as we no longer check a low-value filter.
> (This is also an idea from RocksDB.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9830) Option to disable bloom filter in highest level of LCS sstables

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264264#comment-15264264
 ] 

Paulo Motta commented on CASSANDRA-9830:


There a few situations when a previously disabled top-level bloom filter needs 
to be reloaded:
- Anti-compaction causes previously unrepaired top-level sstable drop to L0
- Anti-compaction increases the number of levels in the repaired set (so 
previously top-level repaired sstables are no longer top-level)
- disable_top_level_bloom_filter option is unset
- user changes compaction strategy to other strategy

Given that the main objective of this optimization is to reduce memory usage 
and rebuilding bloom filters is quite expensive, rather than not generating (or 
removing) top-level bloom filters on disk, it's more reasonable to only release 
bloom filters from memory while still keeping them on disk for a potential 
reload in the future.

Another benefit of keeping BFs on disk is to keep most of the logic within 
{{LeveledCompactionStrategy}}, rather than having other sstables consumers 
(such as tools like {{sstablelevelreset}}) being aware that a top-level sstable 
may not have it's bloom filter component if this option is enabled to deal with 
it accordingly.

One caveat is that when a new level L is created, overlapping sstables from L-1 
must have it's bloom filter reloaded to avoid expensive seek when doing new 
compactions. This is automatically done by "organic" compactions when they 
replace compacted sstables from L-1. Since anti-compactions may create new-top 
levels in the repaired set, we must explicitly check for overlapping sstables 
in lower levels to reload their bloom filters if necessary. In order to avoid 
doing this more expensive overlap check for every sstable added, I modified the 
compaction manager to always use the bulk add method (addSSTables) (which is 
overridden by {{LeveledCompactionStrategy}}) so we can perform this check fewer 
times (specially when doing anti-compaction).

I rebased and added unit tests to cover edge cases mentioned above.

||trunk||
|[branch|https://github.com/apache/cassandra/compare/trunk...pauloricardomg:trunk-9830]|
|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-trunk-9830-testall/lastCompletedBuild/testReport/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-trunk-9830-dtest/lastCompletedBuild/testReport/]|

Also resubmitted cstar_perf tests to make sure we're getting consistent results 
(will post results later):
* 
[majors|http://cstar.datastax.com/tests/id/ddf75066-0e23-11e6-979b-0256e416528f]
* 
[minors|http://cstar.datastax.com/tests/id/e86f087c-0e23-11e6-979b-0256e416528f]
* 
[repair|http://cstar.datastax.com/tests/id/da385b14-0e23-11e6-979b-0256e416528f]

> Option to disable bloom filter in highest level of LCS sstables
> ---
>
> Key: CASSANDRA-9830
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9830
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Compaction
>Reporter: Jonathan Ellis
>Assignee: Paulo Motta
>Priority: Minor
>  Labels: performance
> Fix For: 3.x
>
>
> We expect about 90% of data to be in the highest level of LCS in a fully 
> populated series.  (See also CASSANDRA-9829.)
> Thus if the user is primarily asking for data (partitions) that has actually 
> been inserted, the bloom filter on the highest level only helps reject 
> sstables about 10% of the time.
> We should add an option that suppresses bloom filter creation on top-level 
> sstables.  This will dramatically reduce memory usage for LCS and may even 
> improve performance as we no longer check a low-value filter.
> (This is also an idea from RocksDB.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11602) Materialized View Doest Not Have Static Columns

[Ravishankar Rajendran (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264260#comment-15264260
 ] 

Ravishankar Rajendran commented on CASSANDRA-11602:
---

Guys,

Denormalization and Static Columns are the backbone of Cassandra Data Model. If 
you cannot do static columns in MV, that means there is something fundamentally 
wrong in the base design. 

With Static Columns an application gets an option to run one UPDATE CQL instead 
of a million UPDATE CQL. The fundamental requirement of an MV is 
denormalization, which must include static columns too.

By handicapping application developers get the right thing, we are only driving 
away people from using Cassandra. 

By the way "the new comer" wont be scary. He will be happy that finally MV does 
what it really promises. 

-Ravi

> Materialized View Doest Not Have Static Columns
> ---
>
> Key: CASSANDRA-11602
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11602
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Ravishankar Rajendran
>Assignee: Carl Yeksigian
> Fix For: 3.0.x, 3.x
>
>
> {quote}
> CREATE TABLE "team" (teamname text, manager text, location text static, 
> PRIMARY KEY ((teamname), manager));
> INSERT INTO team (teamname, manager, location) VALUES ('Red Bull1', 
> 'Ricciardo11', 'Australian');
> INSERT INTO team (teamname, manager, location) VALUES ('Red Bull2', 
> 'Ricciardo12', 'Australian');
> INSERT INTO team (teamname, manager, location) VALUES ('Red Bull2', 
> 'Ricciardo13', 'Australian');
> select * From team;
> CREATE MATERIALIZED VIEW IF NOT EXISTS "teamMV" AS SELECT "teamname", 
> "manager", "location" FROM "team" WHERE "teamname" IS NOT NULL AND "manager" 
> is NOT NULL AND "location" is NOT NULL PRIMARY KEY("manager", "teamname");  
> select * from "teamMV";
> {quote}
> The teamMV does not have "location" column. Static columns are not getting 
> created in MV.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11626) cqlsh fails and exits on non-ascii chars

[Tyler Hobbs (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tyler Hobbs updated CASSANDRA-11626:

Reproduced In: 3.5, 3.0.4

> cqlsh fails and exits on non-ascii chars
> 
>
> Key: CASSANDRA-11626
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11626
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Robert Stupp
>Assignee: Tyler Hobbs
>Priority: Minor
>
> Just seen on cqlsh on current trunk:
> To repro, copy {{ä}} (german umlaut) to cqlsh and press return.
> cqlsh errors out and immediately exits.
> {code}
> $ bin/cqlsh
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.13-SNAPSHOT | CQL spec 3.2.1 | Native protocol 
> v3]
> Use HELP for help.
> cqlsh> ä
> Invalid syntax at line 1, char 1
> Traceback (most recent call last):
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2636, in 
> 
> main(*read_options(sys.argv[1:], os.environ))
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2625, in main
> shell.cmdloop()
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 1114, in 
> cmdloop
> if self.onecmd(self.statement.getvalue()):
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 1139, in onecmd
> self.printerr('  %s' % statementline)
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2314, in 
> printerr
> self.writeresult(text, color, newline=newline, out=sys.stderr)
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2303, in 
> writeresult
> out.write(self.applycolor(str(text), color) + ('\n' if newline else ''))
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 
> 2: ordinal not in range(128)
> $ 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11352) Include units of metrics in the cassandra-stress tool

[Tyler Hobbs (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tyler Hobbs updated CASSANDRA-11352:

   Resolution: Fixed
Fix Version/s: (was: 3.x)
   3.6
   Status: Resolved  (was: Patch Available)

The test results look good, so +1, committed as 
{{ff4d0f9abe79ef0925c150d92e6123650e07629b}} to trunk.

Thanks!

> Include units of metrics in the cassandra-stress tool 
> --
>
> Key: CASSANDRA-11352
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11352
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Rajath Subramanyam
>Assignee: Giampaolo
>Priority: Minor
>  Labels: lhf
> Fix For: 3.6
>
> Attachments: 
> cassandra-11352-trunk-giampaolo-trapasso@radicalbit-io.patch
>
>
> cassandra-stress in the Results section can have units for the metrics as an 
> improvement to make the tool more usable. 
> {noformat}
> Results:
> op rate   : 14668 [READ:7334, WRITE:7334]
> partition rate: 14668 [READ:7334, WRITE:7334]
> row rate  : 14668 [READ:7334, WRITE:7334]
> latency mean  : 0.7 [READ:0.7, WRITE:0.7]
> latency median: 0.6 [READ:0.6, WRITE:0.6]
> latency 95th percentile   : 0.8 [READ:0.8, WRITE:0.8]
> latency 99th percentile   : 1.2 [READ:1.2, WRITE:1.2]
> latency 99.9th percentile : 8.8 [READ:8.9, WRITE:9.0]
> latency max   : 448.7 [READ:162.3, WRITE:448.7]
> Total partitions  : 105612753 [READ:52805915, WRITE:52806838]
> Total errors  : 0 [READ:0, WRITE:0]
> total gc count: 0
> total gc mb   : 0
> total gc time (s) : 0
> avg gc time(ms)   : NaN
> stdev gc time(ms) : 0
> Total operation time  : 02:00:00
> END
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

cassandra git commit: Add units to stress output

[tylerhobbs]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 99f0ff938 -> ff4d0f9ab


Add units to stress output

Patch by Giampaolo Trapasso; reviewed by Tyler Hobbs for CASSANDRA-11352


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ff4d0f9a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ff4d0f9a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ff4d0f9a

Branch: refs/heads/trunk
Commit: ff4d0f9abe79ef0925c150d92e6123650e07629b
Parents: 99f0ff9
Author: Giampaolo Trapasso 
Authored: Thu Apr 28 15:54:00 2016 -0500
Committer: Tyler Hobbs 
Committed: Fri Apr 29 11:01:24 2016 -0500

--
 CHANGES.txt |  1 +
 .../org/apache/cassandra/utils/FBUtilities.java | 11 --
 .../apache/cassandra/stress/StressMetrics.java  | 33 +
 .../cassandra/stress/util/TimingInterval.java   | 20 +-
 .../cassandra/stress/util/TimingIntervals.java  | 39 +---
 5 files changed, 62 insertions(+), 42 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/ff4d0f9a/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index e11ebd8..8ee39dc 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.6
+ * Add units to stress ouput (CASSANDRA-11352)
  * Fix PER PARTITION LIMIT for single and multi partitions queries 
(CASSANDRA-11603)
  * Add uncompressed chunk cache for RandomAccessReader (CASSANDRA-5863)
  * Clarify ClusteringPrefix hierarchy (CASSANDRA-11213)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ff4d0f9a/src/java/org/apache/cassandra/utils/FBUtilities.java
--
diff --git a/src/java/org/apache/cassandra/utils/FBUtilities.java 
b/src/java/org/apache/cassandra/utils/FBUtilities.java
index f8c82c3..76178ad 100644
--- a/src/java/org/apache/cassandra/utils/FBUtilities.java
+++ b/src/java/org/apache/cassandra/utils/FBUtilities.java
@@ -588,11 +588,16 @@ public class FBUtilities
 
 public static String prettyPrintMemory(long size)
 {
+return prettyPrintMemory(size, false);
+}
+
+public static String prettyPrintMemory(long size, boolean includeSpace)
+{
 if (size >= 1 << 30)
-return String.format("%.3fGiB", size / (double) (1 << 30));
+return String.format("%.3f%sGiB", size / (double) (1 << 30), 
includeSpace ? " " : "");
 if (size >= 1 << 20)
-return String.format("%.3fMiB", size / (double) (1 << 20));
-return String.format("%.3fKiB", size / (double) (1 << 10));
+return String.format("%.3f%sMiB", size / (double) (1 << 20), 
includeSpace ? " " : "");
+return String.format("%.3f%sKiB", size / (double) (1 << 10), 
includeSpace ? " " : "");
 }
 
 public static String prettyPrintMemoryPerSecond(long rate)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ff4d0f9a/tools/stress/src/org/apache/cassandra/stress/StressMetrics.java
--
diff --git a/tools/stress/src/org/apache/cassandra/stress/StressMetrics.java 
b/tools/stress/src/org/apache/cassandra/stress/StressMetrics.java
index 3585a00..fa36716 100644
--- a/tools/stress/src/org/apache/cassandra/stress/StressMetrics.java
+++ b/tools/stress/src/org/apache/cassandra/stress/StressMetrics.java
@@ -33,6 +33,7 @@ import org.apache.cassandra.stress.util.*;
 import org.apache.commons.lang3.time.DurationFormatUtils;
 import org.apache.cassandra.concurrent.NamedThreadFactory;
 import org.apache.cassandra.stress.settings.StressSettings;
+import org.apache.cassandra.utils.FBUtilities;
 
 public class StressMetrics
 {
@@ -217,22 +218,22 @@ public class StressMetrics
 
 TimingIntervals opHistory = timing.getHistory();
 TimingInterval history = 
opHistory.combine(settings.samples.historyCount);
-output.println(String.format("op rate   : %.0f %s", 
history.opRate(), opHistory.opRates()));
-output.println(String.format("partition rate: %.0f %s", 
history.partitionRate(), opHistory.partitionRates()));
-output.println(String.format("row rate  : %.0f %s", 
history.rowRate(), opHistory.rowRates()));
-output.println(String.format("latency mean  : %.1f %s", 
history.meanLatency(), opHistory.meanLatencies()));
-output.println(String.format("latency median: %.1f %s", 
history.medianLatency(), opHistory.medianLatencies()));
-output.println(String.format("latency 95th percentile   : %.1f %s", 
history.rankLatency(.95f), opHistory.rankLatencies(0.95f)));
-

[jira] [Updated] (CASSANDRA-5863) In process (uncompressed) page cache

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-5863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-5863:
-
Fix Version/s: (was: 3.x)
   3.6

> In process (uncompressed) page cache
> 
>
> Key: CASSANDRA-5863
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5863
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: T Jake Luciani
>Assignee: Branimir Lambov
>  Labels: performance
> Fix For: 3.6
>
>
> Currently, for every read, the CRAR reads each compressed chunk into a 
> byte[], sends it to ICompressor, gets back another byte[] and verifies a 
> checksum.  
> This process is where the majority of time is spent in a read request.  
> Before compression, we would have zero-copy of data and could respond 
> directly from the page-cache.
> It would be useful to have some kind of Chunk cache that could speed up this 
> process for hot data, possibly off heap.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (CASSANDRA-11626) cqlsh fails and exits on non-ascii chars

[Tyler Hobbs (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tyler Hobbs reassigned CASSANDRA-11626:
---

Assignee: Tyler Hobbs

> cqlsh fails and exits on non-ascii chars
> 
>
> Key: CASSANDRA-11626
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11626
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Robert Stupp
>Assignee: Tyler Hobbs
>Priority: Minor
>
> Just seen on cqlsh on current trunk:
> To repro, copy {{ä}} (german umlaut) to cqlsh and press return.
> cqlsh errors out and immediately exits.
> {code}
> $ bin/cqlsh
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.13-SNAPSHOT | CQL spec 3.2.1 | Native protocol 
> v3]
> Use HELP for help.
> cqlsh> ä
> Invalid syntax at line 1, char 1
> Traceback (most recent call last):
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2636, in 
> 
> main(*read_options(sys.argv[1:], os.environ))
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2625, in main
> shell.cmdloop()
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 1114, in 
> cmdloop
> if self.onecmd(self.statement.getvalue()):
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 1139, in onecmd
> self.printerr('  %s' % statementline)
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2314, in 
> printerr
> self.writeresult(text, color, newline=newline, out=sys.stderr)
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2303, in 
> writeresult
> out.write(self.applycolor(str(text), color) + ('\n' if newline else ''))
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 
> 2: ordinal not in range(128)
> $ 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11679) Cassandra Driver returns different number of results depending on fetchsize

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-11679:
-
Assignee: Benjamin Lerer

> Cassandra Driver returns different number of results depending on fetchsize
> ---
>
> Key: CASSANDRA-11679
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11679
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Varun Barala
>Assignee: Benjamin Lerer
>
> I'm trying to fetch all distinct keys from a CF using cassandra-driver 
> (2.1.7.1) and I observed some strange behavior :-
> The total distinct rows are 498 so If I perform a query get All distinctKeys 
> It return 503 instead of 498(five keys twice).
> But If I define the fetch size in select statement more than 498 then it 
> returns exact 498 rows. 
> And If I execute same statement on Dev-center it returns 498 rows.
> Some Additional and useful information :- 
> ---
> Cassandra-2.1.13  (C)* version
> Consistency level: ONE 
> local machine(ubuntu 14.04)
> Table Schema:-
> --
> {code:xml}
> CREATE TABLE sample (
>  pk1 text,
>  pk2 text,
> row_id uuid,
> value blob,
> PRIMARY KEY (( pk1,  pk2))
> ) WITH bloom_filter_fp_chance = 0.01
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = ''
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
> AND compression = {'sstable_compression': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
> {code}
> query :-
> 
> {code:xml}
> SELECT DISTINCT  pk2, pk1 FROM sample LIMIT 2147483647;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11655) sstabledump doesn't print out tombstone information for deleted collection column

[Wei Deng (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264233#comment-15264233
 ] 

Wei Deng commented on CASSANDRA-11655:
--

The latest v3 patch looks good to me.

+1

> sstabledump doesn't print out tombstone information for deleted collection 
> column
> -
>
> Key: CASSANDRA-11655
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11655
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Wei Deng
>Assignee: Chris Lohfink
>  Labels: Tools
> Attachments: CASSANDRA-11655.patch, trunk-11655v2.patch, 
> trunk-11655v3.patch
>
>
> Pretty trivial to reproduce.
> {noformat}
> echo "CREATE KEYSPACE IF NOT EXISTS testks WITH replication = {'class': 
> 'SimpleStrategy', 'replication_factor': '1'};" | cqlsh
> echo "CREATE TABLE IF NOT EXISTS testks.testcf ( k int, c text, val0_int int, 
> val1_set_of_int set, PRIMARY KEY (k, c) );" | cqlsh
> echo "INSERT INTO testks.testcf (k, c, val0_int, val1_set_of_int) VALUES (1, 
> 'c1', 100, {1, 2, 3, 4, 5});" | cqlsh
> echo "delete val1_set_of_int from testks.testcf where k=1 and c='c1';" | cqlsh
> echo "select * from testks.testcf;" | cqlsh
> nodetool flush testks testcf
> {noformat}
> Now if you run sstabledump (even after taking the 
> [patch|https://github.com/yukim/cassandra/tree/11654-3.0] for 
> CASSANDRA-11654) against the newly generated SSTable like the following:
> {noformat}
> ~/cassandra-trunk/tools/bin/sstabledump ma-1-big-Data.db
> [
>   {
> "partition" : {
>   "key" : [ "1" ],
>   "position" : 0
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 18,
> "clustering" : [ "c1" ],
> "liveness_info" : { "tstamp" : 1461645231352208 },
> "cells" : [
>   { "name" : "val0_int", "value" : "100" }
> ]
>   }
> ]
>   }
> ]
> {noformat}
> You will see that the collection-level Deletion Info is nowhere to be found, 
> so you will not be able to know "markedForDeleteAt" or "localDeletionTime" 
> for this collection tombstone.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11677) Incredibly slow jolokia response times

[Joshua McKenzie (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264234#comment-15264234
 ] 

Joshua McKenzie commented on CASSANDRA-11677:
-

So how does it look w/jolokia 1.3.0+ against the cassandra-1.1 cluster?

> Incredibly slow jolokia response times
> --
>
> Key: CASSANDRA-11677
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11677
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Andrew Jorgensen
>
> I am seeing some very slow jolokia request times on my Cassandra 3.0 cluster. 
> Specifically when running the following:
> {code}
> curl 127.0.0.1:8778/jolokia/list
> {code}
> on a slightly loaded cluster I am seeing request times around 30-40 seconds 
> and on a more heavily loaded cluster I am seeing request times in the 2 
> minute mark. We are currently using jolokia 1.3.2 and v4 of the diamond 
> collector. I also have a Cassandra 1.1 cluster that has the same load and 
> number of nodes and running the same curl command comes back in about 1 
> second.
> Is there anything I can do to help diagnose this issue to see what is causing 
> the slowdown or has anyone else experience this?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (CASSANDRA-11672) Upgradesstables errors with "CompoundComposite cannot be cast to org.apache.cassandra.db.composites.CellName"

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne reassigned CASSANDRA-11672:


Assignee: Sylvain Lebresne

> Upgradesstables errors with "CompoundComposite cannot be cast to 
> org.apache.cassandra.db.composites.CellName" 
> --
>
> Key: CASSANDRA-11672
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11672
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Simon Ashley
>Assignee: Sylvain Lebresne
>
> Upgradesstables in C* 2.1 fails on thrift tables originally created on C*1.2 
> with the following error:
> {quote}
> $ nodetool upgradesstables -a
> error: org.apache.cassandra.db.composites.CompoundComposite cannot be cast to 
> org.apache.cassandra.db.composites.CellName
> -- StackTrace --
> java.lang.ClassCastException: 
> org.apache.cassandra.db.composites.CompoundComposite cannot be cast to 
> org.apache.cassandra.db.composites.CellName
>   at 
> org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:86)
>   at 
> org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52)
>   at 
> org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46)
>   at 
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>   at 
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>   at 
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:171)
>   at 
> org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:202)
>   at 
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>   at 
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>   at com.google.common.collect.Iterators$7.computeNext(Iterators.java:645)
>   at 
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>   at 
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>   at 
> org.apache.cassandra.db.ColumnIndex$Builder.buildForCompaction(ColumnIndex.java:166)
>   at 
> org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:121)
>   at 
> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:193)
>   at 
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:126)
>   at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:197)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>   at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73)
>   at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
>   at 
> org.apache.cassandra.db.compaction.CompactionManager$4.execute(CompactionManager.java:376)
>   at 
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:304)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {quote}
> This problem is not seen if the thrift table was originally created in C* 
> 2.0.x
> The suspicion is that this is related to the use of a CompositeType 
> comparator. 
> The following schema is an example of a cf that will cause this issue.
> {quote}
> create column family cf1
>   with column_type = 'Standard'
>   and comparator = 
> 'CompositeType(org.apache.cassandra.db.marshal.ReversedType(org.apache.cassandra.db.marshal.DateType),org.apache.cassandra.db.marshal.UUIDType,org.apache.cassandra.db.marshal.AsciiType,org.apache.cassandra.db.marshal.UUIDType,org.apache.cassandra.db.marshal.UUIDType,org.apache.cassandra.db.marshal.AsciiType,org.apache.cassandra.db.marshal.AsciiType,org.apache.cassandra.db.marshal.AsciiType,org.apache.cassandra.db.marshal.AsciiType,org.apache.cassandra.db.marshal.AsciiType,org.apache.cassandra.db.marshal.AsciiType)'
>   and default_validation_class = 'UTF8Type'
>   and key_validation_class = 
> 'CompositeType(org.apache.cassandra.db.marshal.LongType,org.apache.cassandra.db.marshal.IntegerType)'
>   and read_repair_chance = 1.0
>   and dclocal_read_repair_chance = 0.0
>   and populate_io_cache_on_flush = false
>   and gc_grace = 259200
>   and min_compaction_threshold = 4
>   and max_compaction_threshold = 32
>   and replicate_on_write = true
>   and compaction_strategy = 
> 

[jira] [Updated] (CASSANDRA-11655) sstabledump doesn't print out tombstone information for deleted collection column

[Joshua McKenzie (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua McKenzie updated CASSANDRA-11655:

Reviewer: Yuki Morishita

> sstabledump doesn't print out tombstone information for deleted collection 
> column
> -
>
> Key: CASSANDRA-11655
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11655
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Wei Deng
>Assignee: Chris Lohfink
>  Labels: Tools
> Attachments: CASSANDRA-11655.patch, trunk-11655v2.patch, 
> trunk-11655v3.patch
>
>
> Pretty trivial to reproduce.
> {noformat}
> echo "CREATE KEYSPACE IF NOT EXISTS testks WITH replication = {'class': 
> 'SimpleStrategy', 'replication_factor': '1'};" | cqlsh
> echo "CREATE TABLE IF NOT EXISTS testks.testcf ( k int, c text, val0_int int, 
> val1_set_of_int set, PRIMARY KEY (k, c) );" | cqlsh
> echo "INSERT INTO testks.testcf (k, c, val0_int, val1_set_of_int) VALUES (1, 
> 'c1', 100, {1, 2, 3, 4, 5});" | cqlsh
> echo "delete val1_set_of_int from testks.testcf where k=1 and c='c1';" | cqlsh
> echo "select * from testks.testcf;" | cqlsh
> nodetool flush testks testcf
> {noformat}
> Now if you run sstabledump (even after taking the 
> [patch|https://github.com/yukim/cassandra/tree/11654-3.0] for 
> CASSANDRA-11654) against the newly generated SSTable like the following:
> {noformat}
> ~/cassandra-trunk/tools/bin/sstabledump ma-1-big-Data.db
> [
>   {
> "partition" : {
>   "key" : [ "1" ],
>   "position" : 0
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 18,
> "clustering" : [ "c1" ],
> "liveness_info" : { "tstamp" : 1461645231352208 },
> "cells" : [
>   { "name" : "val0_int", "value" : "100" }
> ]
>   }
> ]
>   }
> ]
> {noformat}
> You will see that the collection-level Deletion Info is nowhere to be found, 
> so you will not be able to know "markedForDeleteAt" or "localDeletionTime" 
> for this collection tombstone.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11656) sstabledump has inconsistency in deletion_time printout

[Wei Deng (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264222#comment-15264222
 ] 

Wei Deng commented on CASSANDRA-11656:
--

I like the new output now that we can distinguish markedForDeleteAt and 
localDeletionTime in human-readable format. Using "-t" for epoch format, and by 
default using human-readable format sounds good to me.

+1 for this JIRA.

I'll add my +1 to CASSANDRA-11655 once I'm able to test the code there.

> sstabledump has inconsistency in deletion_time printout
> ---
>
> Key: CASSANDRA-11656
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11656
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Wei Deng
>  Labels: Tools
>
> See the following output (note the deletion info under the second row):
> {noformat}
> [
>   {
> "partition" : {
>   "key" : [ "1" ],
>   "position" : 0
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 18,
> "clustering" : [ "c1" ],
> "liveness_info" : { "tstamp" : 1461646542601774 },
> "cells" : [
>   { "name" : "val0_int", "deletion_time" : 1461647421, "tstamp" : 
> 1461647421344759 },
>   { "name" : "val1_set_of_int", "path" : [ "1" ], "deletion_time" : 
> 1461647320, "tstamp" : 1461647320160261 },
>   { "name" : "val1_set_of_int", "path" : [ "10" ], "value" : "", 
> "tstamp" : 1461647295880444 },
>   { "name" : "val1_set_of_int", "path" : [ "11" ], "value" : "", 
> "tstamp" : 1461647295880444 },
>   { "name" : "val1_set_of_int", "path" : [ "12" ], "value" : "", 
> "tstamp" : 1461647295880444 }
> ]
>   },
>   {
> "type" : "row",
> "position" : 85,
> "clustering" : [ "c2" ],
> "deletion_info" : { "deletion_time" : 1461647588089843, "tstamp" : 
> 1461647588 },
> "cells" : [ ]
>   }
> ]
>   }
> ]
> {noformat}
> To avoid confusion, we need to have consistency in printing out the 
> DeletionTime object. By definition, markedForDeleteAt is in microseconds 
> since epoch and marks the time when the "delete" mutation happens; 
> localDeletionTime is in seconds since epoch and allows GC to collect the 
> tombstone if the current epoch second is greater than localDeletionTime + 
> gc_grace_seconds. I'm ok to use "tstamp" to represent markedForDeleteAt 
> because markedForDeleteAt does represent this delete mutation's timestamp, 
> but we need to be consistent everywhere.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11661) Cassandra 2.0 and later require Java 7u25 or later - jdk 101

[Joshua McKenzie (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua McKenzie updated CASSANDRA-11661:

Assignee: Michael Shuler

> Cassandra 2.0 and later require Java 7u25 or later - jdk 101
> 
>
> Key: CASSANDRA-11661
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11661
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
> Environment: Cassandra Server 2.1.5 and Java jdk1.7.0_101-b14
>Reporter: William Boutin
>Assignee: Michael Shuler
>Priority: Critical
> Fix For: 2.1.x
>
>
> We have been running the cassandr server version 2.1.5. Friday, we applied 
> the latest java patch, Java(TM) SE Runtime Environment (build 1.7.0_101-b14). 
> Cassandra cannot start with this patch. The cassandra log states:  Cassandra 
> 2.0 and later require Java 7u25 or later.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11641) java.lang.IllegalArgumentException: Not enough bytes in system.log

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-11641:
--
Status: Awaiting Feedback  (was: Open)

> java.lang.IllegalArgumentException: Not enough bytes in system.log
> --
>
> Key: CASSANDRA-11641
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11641
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
> Environment: centos 6.5 cassandra2.1.13
>Reporter: peng xiao
> Attachments: system.log
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11475) MV code refactor

[Carl Yeksigian (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264206#comment-15264206
 ] 

Carl Yeksigian commented on CASSANDRA-11475:


{quote}
You're completely right and I don't any obvious way to avoid this since we rely 
on all updates to be on the same batchlog mutation. I guess my point was that 
since we can already have a high footprint, all the more reason to lower it as 
much as we can.
{quote}
Agreed; definitely a huge improvement to reduce our footprint by 2/3. We should 
open a ticket to track this; maybe [~iamaleksey] has some idea how we might be 
able to do this with some batchlog manipulations like writing multiple entries 
but only setting them as writable when all of the entries are written.

{quote}
Is that a naming problem? I don't give that much importance to "patterns" so 
having it not build a ViewUpdate doesn't bother me (it still "builds" updates 
to views). But if you think that's a bit confusing, I don't mind renaming to 
ViewMutationGenerator or something.
{quote}
I think that the {{Builder}}s seem like things you would be using if you were 
interacting with C* from a unit test, or otherwise programmatically building 
these; not necessarily internal usages. I think {{ViewMutationGenerator}} is 
more explanatory; {{generateViewMutations}} and {{build}} don't really describe 
to me how the API is supposed to be used. I would have expected 
{{generateViewMutations}} to return the {{PartitionUpdate}}s; I guess 
{{addRow}} and {{generateMutations}} would be more descriptive to me.

{quote}
I'm not sure to follow what you mean by that.
{quote}
Yeah, I was just saying that it would make more sense to be passing in 2 
{{Partition}}s, one for the old and one for the merged, but it was impractical. 
It ends up being a moot point since it is about the API change that was 
necessary to prevent holding the partitions in memory.


> MV code refactor
> 
>
> Key: CASSANDRA-11475
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11475
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Sylvain Lebresne
>Assignee: Sylvain Lebresne
> Fix For: 3.0.x, 3.x
>
>
> While working on CASSANDRA-5546 I run into a problem with TTLs on MVs, which 
> looking more closely is a bug of the MV code. But one thing leading to 
> another I reviewed a good portion of the MV code and found the following 
> correction problems:
> * If a base row is TTLed then even if an update remove that TTL the view 
> entry remained TTLed and expires, leading to an inconsistency.
> * Due to calling the wrong ctor for {{LivenessInfo}}, when a TTL was set on 
> the base table, the view entry was living twice as long as the TTL. Again 
> leading to a temporary inconsistency.
> * When reading existing data to compute view updates, all deletion 
> informations are completely ignored (the code uses a {{PartitionIterator}} 
> instead of an {{UnfilteredPartitionIterator}}). This is a serious issue since 
> it means some deletions could be totally ignored as far as views are 
> concerned especially when messages are delivered to a replica out of order. 
> I'll note that while the 2 previous points are relatively easy to fix, I 
> didn't find an easy and clean way to fix this one on the current code.
> Further, I think the MV code in general has inefficiencies/code complexities 
> that should be avoidable:
> * {{TemporalRow.Set}} is buffering both everything read and a pretty much 
> complete copy of the updates. That's a potentially high memory requirement. 
> We shouldn't have to copy the updates and we shouldn't buffer all reads but 
> rather work incrementally.
> * {{TemporalRow}}/{{TemporalRow.Set}}/{{TemporalCell}} classes are somewhat 
> re-inventing the wheel. They are really just storing both an update we're 
> doing and the corresponding existing data, but we already have 
> {{Row}}/{{Partition}}/{{Cell}} for that. In practice, those {{Temporal*}} 
> class generates a lot of allocations that we could avoid.
> * The code from CASSANDRA-10060 to avoid multiple reads of the base table 
> with multiple views doesn't work when the update has partition/range 
> tombstones because the code uses {{TemporalRow.Set.setTombstonedExisting()}} 
> to trigger reuse, but the {{TemporalRow.Set.withNewViewPrimaryKey()}} method 
> is used between view and it does not preseve the {{hasTombstonedExisting}} 
> flag.  But that oversight, which is trivial to fix, is kind of a good thing 
> since if you fix it, you're left with a correction problem.
>   The read done when there is a partition deletion depends on the view itself 
> (if there is clustering filters in particular) and so reusing that read for 
> other views is wrong. Which makes that whole reuse code really dodgy imo: the 
> read for existing data is in {{View.java}}, 

[jira] [Updated] (CASSANDRA-11626) cqlsh fails and exits on non-ascii chars

[Joshua McKenzie (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua McKenzie updated CASSANDRA-11626:

Summary: cqlsh fails and exits on non-ascii chars  (was: cqlsh fails and 
exists on non-ascii chars)

> cqlsh fails and exits on non-ascii chars
> 
>
> Key: CASSANDRA-11626
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11626
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Robert Stupp
>Priority: Minor
>
> Just seen on cqlsh on current trunk:
> To repro, copy {{ä}} (german umlaut) to cqlsh and press return.
> cqlsh errors out and immediately exits.
> {code}
> $ bin/cqlsh
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.13-SNAPSHOT | CQL spec 3.2.1 | Native protocol 
> v3]
> Use HELP for help.
> cqlsh> ä
> Invalid syntax at line 1, char 1
> Traceback (most recent call last):
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2636, in 
> 
> main(*read_options(sys.argv[1:], os.environ))
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2625, in main
> shell.cmdloop()
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 1114, in 
> cmdloop
> if self.onecmd(self.statement.getvalue()):
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 1139, in onecmd
> self.printerr('  %s' % statementline)
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2314, in 
> printerr
> self.writeresult(text, color, newline=newline, out=sys.stderr)
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2303, in 
> writeresult
> out.write(self.applycolor(str(text), color) + ('\n' if newline else ''))
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 
> 2: ordinal not in range(128)
> $ 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11604) select on table fails after changing user defined type in map

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264190#comment-15264190
 ] 

Aleksey Yeschenko commented on CASSANDRA-11604:
---

Whoever is gonna deal with the ticket, please have a look into why scrub is 
able to fix the issue (it shouldn't be - likely there is a bug in scrub).

> select on table fails after changing user defined type in map
> -
>
> Key: CASSANDRA-11604
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11604
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Andreas Jaekle
>Assignee: Alex Petrov
> Fix For: 3.x
>
>
> in cassandra 3.5 i get the following exception when i run this cqls:
> {code}
> --DROP KEYSPACE bugtest ;
> CREATE KEYSPACE bugtest
>  WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };
> use bugtest;
> CREATE TYPE tt (
>   a boolean
> );
> create table t1 (
>   k text,
>   v map,
>   PRIMARY KEY(k)
> );
> insert into t1 (k,v) values ('k2',{'mk':{a:false}});
> ALTER TYPE tt ADD b boolean;
> UPDATE t1 SET v['mk'] = { b:true } WHERE k = 'k2';
> select * from t1;  
> {code}
> the last select fails.
> {code}
> WARN  [SharedPool-Worker-5] 2016-04-19 14:18:49,885 
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread 
> Thread[SharedPool-Worker-5,5,main]: {}
> java.lang.AssertionError: null
> at 
> org.apache.cassandra.db.rows.ComplexColumnData$Builder.addCell(ComplexColumnData.java:254)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer.getReduced(Row.java:623)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer.getReduced(Row.java:549)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:217)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:156)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.rows.Row$Merger.merge(Row.java:526) 
> ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:473)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:437)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:217)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:156)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:112) 
> ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.db.transform.UnfilteredRows.isEmpty(UnfilteredRows.java:38)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:64)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:76)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> 

[jira] [Updated] (CASSANDRA-11606) Upgrade from 2.1.9 to 3.0.5 Fails with AssertionError

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-11606:
-
Status: Awaiting Feedback  (was: Open)

> Upgrade from 2.1.9 to 3.0.5 Fails with AssertionError
> -
>
> Key: CASSANDRA-11606
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11606
> Project: Cassandra
>  Issue Type: Bug
> Environment: Fedora 20, Oracle Java 8, Apache Cassandra 2.1.9 -> 3.0.5
>Reporter: Anthony Verslues
> Fix For: 3.0.x
>
>
> I get this error while upgrading sstables. I got the same error when 
> upgrading to 3.0.2 and 3.0.4.
> error: null
> -- StackTrace --
> java.lang.AssertionError
> at 
> org.apache.cassandra.db.LegacyLayout$CellGrouper.addCell(LegacyLayout.java:1167)
> at 
> org.apache.cassandra.db.LegacyLayout$CellGrouper.addAtom(LegacyLayout.java:1142)
> at 
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.readRow(UnfilteredDeserializer.java:444)
> at 
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:423)
> at 
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:289)
> at 
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.readStaticRow(SSTableSimpleIterator.java:133)
> at 
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:57)
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:334)
> at 
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48)
> at 
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:65)
> at 
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:109)
> at 
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:100)
> at 
> org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:442)
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at 
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:150)
> at 
> org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72)
> at 
> org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
> at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
> at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
> at 
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:416)
> at 
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:313)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11604) select on table fails after changing user defined type in map

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-11604:
-
Assignee: Alex Petrov

> select on table fails after changing user defined type in map
> -
>
> Key: CASSANDRA-11604
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11604
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Andreas Jaekle
>Assignee: Alex Petrov
> Fix For: 3.x
>
>
> in cassandra 3.5 i get the following exception when i run this cqls:
> {code}
> --DROP KEYSPACE bugtest ;
> CREATE KEYSPACE bugtest
>  WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };
> use bugtest;
> CREATE TYPE tt (
>   a boolean
> );
> create table t1 (
>   k text,
>   v map,
>   PRIMARY KEY(k)
> );
> insert into t1 (k,v) values ('k2',{'mk':{a:false}});
> ALTER TYPE tt ADD b boolean;
> UPDATE t1 SET v['mk'] = { b:true } WHERE k = 'k2';
> select * from t1;  
> {code}
> the last select fails.
> {code}
> WARN  [SharedPool-Worker-5] 2016-04-19 14:18:49,885 
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread 
> Thread[SharedPool-Worker-5,5,main]: {}
> java.lang.AssertionError: null
> at 
> org.apache.cassandra.db.rows.ComplexColumnData$Builder.addCell(ComplexColumnData.java:254)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer.getReduced(Row.java:623)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer.getReduced(Row.java:549)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:217)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:156)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.rows.Row$Merger.merge(Row.java:526) 
> ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:473)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:437)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:217)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:156)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:112) 
> ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.db.transform.UnfilteredRows.isEmpty(UnfilteredRows.java:38)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:64)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:76)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:289)
>  ~[apache-cassandra-3.5.jar:3.5]
> at 
> 

[jira] [Commented] (CASSANDRA-11391) "class declared as inner class" error when using UDF

[Robert Stupp (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264188#comment-15264188
 ] 

Robert Stupp commented on CASSANDRA-11391:
--

Oops. Some of the unit test classes did not compile anymore. Fixed that. 
[~thobbs], want to take a look at the latest CI results?

> "class declared as inner class" error when using UDF
> 
>
> Key: CASSANDRA-11391
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11391
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: C* 3.4
>Reporter: DOAN DuyHai
>Assignee: Robert Stupp
>Priority: Critical
> Fix For: 3.x
>
>
> {noformat}
> cqlsh:music> CREATE FUNCTION testMapEntry(my_map map)
>  ... CALLED ON NULL INPUT
>  ... RETURNS text
>  ... LANGUAGE java
>  ... AS $$
>  ... String buffer = "";
>  ... for(java.util.Map.Entry entry: 
> my_map.entrySet()) {
>  ... buffer = buffer + entry.getKey() + ": " + 
> entry.getValue() + ", ";
>  ... }
>  ... return buffer;
>  ... $$;
> InvalidRequest: code=2200 [Invalid query] 
> message="Could not compile function 'music.testmapentry' from Java source: 
> org.apache.cassandra.exceptions.InvalidRequestException: 
> Java UDF validation failed: [class declared as inner class]"
> {noformat}
> When I try to decompile the source code into byte code, below is the result:
> {noformat}
>   public java.lang.String test(java.util.Map java.lang.String>);
> Code:
>0: ldc   #2  // String
>2: astore_2
>3: aload_1
>4: invokeinterface #3,  1// InterfaceMethod 
> java/util/Map.entrySet:()Ljava/util/Set;
>9: astore_3
>   10: aload_3
>   11: invokeinterface #4,  1// InterfaceMethod 
> java/util/Set.iterator:()Ljava/util/Iterator;
>   16: astore4
>   18: aload 4
>   20: invokeinterface #5,  1// InterfaceMethod 
> java/util/Iterator.hasNext:()Z
>   25: ifeq  94
>   28: aload 4
>   30: invokeinterface #6,  1// InterfaceMethod 
> java/util/Iterator.next:()Ljava/lang/Object;
>   35: checkcast #7  // class java/util/Map$Entry
>   38: astore5
>   40: new   #8  // class java/lang/StringBuilder
>   43: dup
>   44: invokespecial #9  // Method 
> java/lang/StringBuilder."":()V
>   47: aload_2
>   48: invokevirtual #10 // Method 
> java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
>   51: aload 5
>   53: invokeinterface #11,  1   // InterfaceMethod 
> java/util/Map$Entry.getKey:()Ljava/lang/Object;
>   58: checkcast #12 // class java/lang/String
>   61: invokevirtual #10 // Method 
> java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
>   64: ldc   #13 // String :
>   66: invokevirtual #10 // Method 
> java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
>   69: aload 5
>   71: invokeinterface #14,  1   // InterfaceMethod 
> java/util/Map$Entry.getValue:()Ljava/lang/Object;
>   76: checkcast #12 // class java/lang/String
>   79: invokevirtual #10 // Method 
> java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
>   82: ldc   #15 // String ,
>   84: invokevirtual #10 // Method 
> java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
>   87: invokevirtual #16 // Method 
> java/lang/StringBuilder.toString:()Ljava/lang/String;
>   90: astore_2
>   91: goto  18
>   94: aload_2
>   95: areturn
> {noformat}
>  There is nothing that could trigger inner class creation ...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8844) Change Data Capture (CDC)

[Carl Yeksigian (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264184#comment-15264184
 ] 

Carl Yeksigian commented on CASSANDRA-8844:
---

A few things I think we should nail out before we embark on yet another CDC 
journey:

- How will cdc space be measured? Since we have a compaction process, users 
will want to have two CDC directories, cdc_overflow (where the commit log files 
get moved to), and cdc_compacted (where the compacted files would be written 
to). Also, need to make sure that we are calculating it frequently enough that 
we don't mark it as over size.
- Will you keep the size rejection in the same place as it is now (inside the 
commit log)? We will have to reject all updates, because if someone turns on 
the CDC in the yaml, but doesn't have any daemon running, we'll be moving all 
of the files to the cdc_overflow folder regardless of whether any CFs have CDC 
enabled.
- Not sure how we can handle someone turning off CDC and then restarting their 
daemon; this will probably end up being an edge case we won't handle well. All 
of the updates before they turn off CDC should be included in the compacted 
file, the ones after should not be. However, if we read state from C*, we won't 
be getting the latest values out.
- How will we guarantee atomicity between the overflow and the compacted files? 
Can we compact the same overflow file multiple times (in the case of multiple 
restarts of the daemon)? I'm thinking of the case where we mark the compacted 
file complete, and then fail before we can delete the overflow file.


> Change Data Capture (CDC)
> -
>
> Key: CASSANDRA-8844
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8844
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Coordination, Local Write-Read Paths
>Reporter: Tupshin Harper
>Assignee: Joshua McKenzie
>Priority: Critical
> Fix For: 3.x
>
>
> "In databases, change data capture (CDC) is a set of software design patterns 
> used to determine (and track) the data that has changed so that action can be 
> taken using the changed data. Also, Change data capture (CDC) is an approach 
> to data integration that is based on the identification, capture and delivery 
> of the changes made to enterprise data sources."
> -Wikipedia
> As Cassandra is increasingly being used as the Source of Record (SoR) for 
> mission critical data in large enterprises, it is increasingly being called 
> upon to act as the central hub of traffic and data flow to other systems. In 
> order to try to address the general need, we (cc [~brianmhess]), propose 
> implementing a simple data logging mechanism to enable per-table CDC patterns.
> h2. The goals:
> # Use CQL as the primary ingestion mechanism, in order to leverage its 
> Consistency Level semantics, and in order to treat it as the single 
> reliable/durable SoR for the data.
> # To provide a mechanism for implementing good and reliable 
> (deliver-at-least-once with possible mechanisms for deliver-exactly-once ) 
> continuous semi-realtime feeds of mutations going into a Cassandra cluster.
> # To eliminate the developmental and operational burden of users so that they 
> don't have to do dual writes to other systems.
> # For users that are currently doing batch export from a Cassandra system, 
> give them the opportunity to make that realtime with a minimum of coding.
> h2. The mechanism:
> We propose a durable logging mechanism that functions similar to a commitlog, 
> with the following nuances:
> - Takes place on every node, not just the coordinator, so RF number of copies 
> are logged.
> - Separate log per table.
> - Per-table configuration. Only tables that are specified as CDC_LOG would do 
> any logging.
> - Per DC. We are trying to keep the complexity to a minimum to make this an 
> easy enhancement, but most likely use cases would prefer to only implement 
> CDC logging in one (or a subset) of the DCs that are being replicated to
> - In the critical path of ConsistencyLevel acknowledgment. Just as with the 
> commitlog, failure to write to the CDC log should fail that node's write. If 
> that means the requested consistency level was not met, then clients *should* 
> experience UnavailableExceptions.
> - Be written in a Row-centric manner such that it is easy for consumers to 
> reconstitute rows atomically.
> - Written in a simple format designed to be consumed *directly* by daemons 
> written in non JVM languages
> h2. Nice-to-haves
> I strongly suspect that the following features will be asked for, but I also 
> believe that they can be deferred for a subsequent release, and to guage 
> actual interest.
> - Multiple logs per table. This would make it easy to have multiple 
> "subscribers" to a single table's changes. A workaround 

[jira] [Updated] (CASSANDRA-11596) Add native transport port to system.peers

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-11596:
-
Labels: lhf  (was: )

> Add native transport port to system.peers
> -
>
> Key: CASSANDRA-11596
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11596
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Distributed Metadata
>Reporter: Nico Haller
>Priority: Minor
>  Labels: lhf
>
> Is there any reason why the native transport port is not being stored in 
> system.peers along the rpc broadcast address and transmitted to the connected 
> drivers?
> I would love to have that feature, that would allow me to "hide" my cluster 
> behind a reverse NAT or LB and only consume one external IP address and 
> forward packets based on the port the client is connecting to.
> I guess it makes sense to provide the complete socket information instead of 
> just the address and using a default port setting on the client to complete 
> the connection information.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11589) NodeProbe.checkJobs uses DatabaseDescriptor but should instead use JMX

[Joshua McKenzie (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua McKenzie updated CASSANDRA-11589:

Summary: NodeProbe.checkJobs uses DatabaseDescriptor but should instead use 
JMX  (was: NodeProbe.checkJobs uses DatabaseDescriptor)

> NodeProbe.checkJobs uses DatabaseDescriptor but should instead use JMX
> --
>
> Key: CASSANDRA-11589
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11589
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Robert Stupp
>Priority: Minor
>
> CASSANDRA-11179 introduced the {{-j}} parameter for several nodetool 
> commands. The function {{NodeProbe.checkJobs}} validates the argument against 
> {{DatabaseDescriptor.getConcurrentCompactors()}}, which returns the 
> configured value of the _local_ node - so it might not return the correct 
> value for the node to which nodetool talks.
> So, {{checkJobs()}} should get the configured number of concurrent_compactors 
> via JMX and not using DatabaseDescriptor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11587) Cfstats estimate number of keys should return 0 for empty table

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-11587:
-
Labels: lhf  (was: )

> Cfstats estimate number of keys should return 0 for empty table
> ---
>
> Key: CASSANDRA-11587
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11587
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
> Environment: Cassandra 2.1.13
> Nodeltool
>Reporter: Jane Deng
>Priority: Trivial
>  Labels: lhf
>
> If sstable count is 0, the "estimate number of keys" in cfstats shows -1. 
> {noformat}
> SSTable count: 0
> Number of keys (estimate): -1
> {noformat}
> The initial value of keyCount in SSTableReader is -1. When there is no 
> sstable, nor entry in memtable, the keyCount isn't increased. Cfstats should 
> have some boundary check and return 0 for this case. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11602) Materialized View Doest Not Have Static Columns

[Carl Yeksigian (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264137#comment-15264137
 ] 

Carl Yeksigian commented on CASSANDRA-11602:


Good point; I'll update the ticket to exclude those as well.

> Materialized View Doest Not Have Static Columns
> ---
>
> Key: CASSANDRA-11602
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11602
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Ravishankar Rajendran
>Assignee: Carl Yeksigian
> Fix For: 3.0.x, 3.x
>
>
> {quote}
> CREATE TABLE "team" (teamname text, manager text, location text static, 
> PRIMARY KEY ((teamname), manager));
> INSERT INTO team (teamname, manager, location) VALUES ('Red Bull1', 
> 'Ricciardo11', 'Australian');
> INSERT INTO team (teamname, manager, location) VALUES ('Red Bull2', 
> 'Ricciardo12', 'Australian');
> INSERT INTO team (teamname, manager, location) VALUES ('Red Bull2', 
> 'Ricciardo13', 'Australian');
> select * From team;
> CREATE MATERIALIZED VIEW IF NOT EXISTS "teamMV" AS SELECT "teamname", 
> "manager", "location" FROM "team" WHERE "teamname" IS NOT NULL AND "manager" 
> is NOT NULL AND "location" is NOT NULL PRIMARY KEY("manager", "teamname");  
> select * from "teamMV";
> {quote}
> The teamMV does not have "location" column. Static columns are not getting 
> created in MV.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11602) Materialized View Doest Not Have Static Columns

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264120#comment-15264120
 ] 

Sylvain Lebresne commented on CASSANDRA-11602:
--

bq.  and to warn if it is dropped when using a {{SELECT *}}

I'd actually be in favor of rejecting that too. First because silently ignoring 
columns is bound to surprise/confuse user, and this whether users gets the 
warning or not. If they don't get it it's confusing, but if they get it, it's a 
bit scary for new comer as they won't know if they are doing something wrong or 
not. Second, because if we ever do find a way to include static columns, we'd 
have a breaking change here since user would have been used to them being 
ignored. And third, because the only argument for allowing that is the minor 
convenience of not having to list columns manually, but if you do so you're 
getting a client warning which is annoying/distracting, so you'll kind of end 
up listing columns manually anyway. 

> Materialized View Doest Not Have Static Columns
> ---
>
> Key: CASSANDRA-11602
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11602
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Ravishankar Rajendran
>Assignee: Carl Yeksigian
> Fix For: 3.0.x, 3.x
>
>
> {quote}
> CREATE TABLE "team" (teamname text, manager text, location text static, 
> PRIMARY KEY ((teamname), manager));
> INSERT INTO team (teamname, manager, location) VALUES ('Red Bull1', 
> 'Ricciardo11', 'Australian');
> INSERT INTO team (teamname, manager, location) VALUES ('Red Bull2', 
> 'Ricciardo12', 'Australian');
> INSERT INTO team (teamname, manager, location) VALUES ('Red Bull2', 
> 'Ricciardo13', 'Australian');
> select * From team;
> CREATE MATERIALIZED VIEW IF NOT EXISTS "teamMV" AS SELECT "teamname", 
> "manager", "location" FROM "team" WHERE "teamname" IS NOT NULL AND "manager" 
> is NOT NULL AND "location" is NOT NULL PRIMARY KEY("manager", "teamname");  
> select * from "teamMV";
> {quote}
> The teamMV does not have "location" column. Static columns are not getting 
> created in MV.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11468) Reading from early opened, and compressed, sstable throws CorruptSSTableException

[Blake Eggleston (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264111#comment-15264111
 ] 

Blake Eggleston commented on CASSANDRA-11468:
-

Ah ok, that makes sense. In that case, I'd agree that not fixing is the way to 
go.

> Reading from early opened, and compressed, sstable throws 
> CorruptSSTableException
> -
>
> Key: CASSANDRA-11468
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11468
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Blake Eggleston
>Assignee: Marcus Eriksson
> Fix For: 3.0.x
>
>
> When a compressed sstable is early opened during compaction, reading from it 
> can fail with the following exception:
> {code}
> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: 
> build/test/cassandra/data/ks_1459378971131/early_open_test-893eb3d0f6cb11e59e1b4f343b985d3e/ma-3-big-Data.db
>   at 
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator.(AbstractSSTableIterator.java:130)
>   at 
> org.apache.cassandra.db.columniterator.SSTableIterator.(SSTableIterator.java:46)
>   at 
> org.apache.cassandra.db.columniterator.SSTableIterator.(SSTableIterator.java:36)
>   at 
> org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:62)
>   at 
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndSSTablesInTimestampOrder(SinglePartitionReadCommand.java:715)
>   at 
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:482)
>   at 
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:459)
>   at 
> org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:325)
>   at 
> org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:363)
>   at 
> org.apache.cassandra.db.ReadCommand.executeInternal(ReadCommand.java:393)
>   at 
> org.apache.cassandra.db.SinglePartitionReadCommand$Group.executeInternal(SinglePartitionReadCommand.java:950)
>   at 
> org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:397)
>   at 
> org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:76)
>   at 
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:295)
>   at 
> org.apache.cassandra.io.sstable.EarlyOpenTest.earlyOpen(EarlyOpenTest.java:160)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:44)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
>   at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
>   at org.junit.runner.JUnitCore.run(JUnitCore.java:159)
>   at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:78)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:212)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:68)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
> Caused by: java.io.EOFException
>   at 
> org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:146)
>   at 

[jira] [Resolved] (CASSANDRA-11468) Reading from early opened, and compressed, sstable throws CorruptSSTableException

[Blake Eggleston (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Blake Eggleston resolved CASSANDRA-11468.
-
Resolution: Invalid

> Reading from early opened, and compressed, sstable throws 
> CorruptSSTableException
> -
>
> Key: CASSANDRA-11468
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11468
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Blake Eggleston
>Assignee: Marcus Eriksson
> Fix For: 3.0.x
>
>
> When a compressed sstable is early opened during compaction, reading from it 
> can fail with the following exception:
> {code}
> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: 
> build/test/cassandra/data/ks_1459378971131/early_open_test-893eb3d0f6cb11e59e1b4f343b985d3e/ma-3-big-Data.db
>   at 
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator.(AbstractSSTableIterator.java:130)
>   at 
> org.apache.cassandra.db.columniterator.SSTableIterator.(SSTableIterator.java:46)
>   at 
> org.apache.cassandra.db.columniterator.SSTableIterator.(SSTableIterator.java:36)
>   at 
> org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:62)
>   at 
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndSSTablesInTimestampOrder(SinglePartitionReadCommand.java:715)
>   at 
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:482)
>   at 
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:459)
>   at 
> org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:325)
>   at 
> org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:363)
>   at 
> org.apache.cassandra.db.ReadCommand.executeInternal(ReadCommand.java:393)
>   at 
> org.apache.cassandra.db.SinglePartitionReadCommand$Group.executeInternal(SinglePartitionReadCommand.java:950)
>   at 
> org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:397)
>   at 
> org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:76)
>   at 
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:295)
>   at 
> org.apache.cassandra.io.sstable.EarlyOpenTest.earlyOpen(EarlyOpenTest.java:160)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:44)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
>   at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
>   at org.junit.runner.JUnitCore.run(JUnitCore.java:159)
>   at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:78)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:212)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:68)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
> Caused by: java.io.EOFException
>   at 
> org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:146)
>   at 
> 

[jira] [Commented] (CASSANDRA-11475) MV code refactor

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264106#comment-15264106
 ] 

Sylvain Lebresne commented on CASSANDRA-11475:
--

bq. We are still going to be holding all of the updates that we generate in 
memory; this refactor does make the memory footprint lower, but still includes 
a potential for using up a lot of memory.

You're completely right and I don't any obvious way to avoid this since we rely 
on all updates to be on the same batchlog mutation. I guess my point was that 
since we can already have a high footprint, all the more reason to lower it as 
much as we can.

bq. I think your todo is right about promoting TableViews; I'm not sure if we 
can completely get rid of ViewManager, since we need something at the Keyspace 
level, but maybe that stuff can be integrated into Keyspace again

I started moving {{TableViews}} to its own class with all the methods that deal 
with a single base table but I stopped there. I do think we can and should get 
rid of ViewManager: the 2 only things is does now is dealing with view by names 
(for addition/removal) and holding the locks. This can probably move to 
Keyspace but regarding lock I'm actually wondering if it shouldn't be move to 
the table level. Is there a reason I'm forgetting why it should be at the 
keyspace level?  It will require a bit of refactor of Keyspace.apply() though 
to be clean, which is why I've stopped here and I'd rather left that as a 
follow up (this is only tangentially relevant to the rest of the patch).

bq. I find the UpdateBuilder pattern not to work well here. ViewUpdateBuilder 
should be building ViewUpdate}}s, instead of {{Collection, and 
using the build method at the end seems weird.  

Is that a naming problem? I don't give that much importance to "patterns" so 
having it not build a {{ViewUpdate}} doesn't bother me (it still "builds" 
updates to views). But if you think that's a bit confusing, I don't mind 
renaming to {{ViewMutationGenerator}} or something.

bq. to me, it seems like we'd be better off with not exposing the rows

I'm not sure to follow what you mean by that.

bq. MultiViewUpdateBuilder doesn't really seem necessary; it seems like it 
should be part of ViewManager

I guess the idea was that it could have seen reuse some day and didn't felt 
like a weird concept per-se, but you're right that the verbosity is not that 
justified right now so got rid of it.

bq. In {{TableViews}}, {{removeByName}} is empty

Indeed. Fixed.


> MV code refactor
> 
>
> Key: CASSANDRA-11475
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11475
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Sylvain Lebresne
>Assignee: Sylvain Lebresne
> Fix For: 3.0.x, 3.x
>
>
> While working on CASSANDRA-5546 I run into a problem with TTLs on MVs, which 
> looking more closely is a bug of the MV code. But one thing leading to 
> another I reviewed a good portion of the MV code and found the following 
> correction problems:
> * If a base row is TTLed then even if an update remove that TTL the view 
> entry remained TTLed and expires, leading to an inconsistency.
> * Due to calling the wrong ctor for {{LivenessInfo}}, when a TTL was set on 
> the base table, the view entry was living twice as long as the TTL. Again 
> leading to a temporary inconsistency.
> * When reading existing data to compute view updates, all deletion 
> informations are completely ignored (the code uses a {{PartitionIterator}} 
> instead of an {{UnfilteredPartitionIterator}}). This is a serious issue since 
> it means some deletions could be totally ignored as far as views are 
> concerned especially when messages are delivered to a replica out of order. 
> I'll note that while the 2 previous points are relatively easy to fix, I 
> didn't find an easy and clean way to fix this one on the current code.
> Further, I think the MV code in general has inefficiencies/code complexities 
> that should be avoidable:
> * {{TemporalRow.Set}} is buffering both everything read and a pretty much 
> complete copy of the updates. That's a potentially high memory requirement. 
> We shouldn't have to copy the updates and we shouldn't buffer all reads but 
> rather work incrementally.
> * {{TemporalRow}}/{{TemporalRow.Set}}/{{TemporalCell}} classes are somewhat 
> re-inventing the wheel. They are really just storing both an update we're 
> doing and the corresponding existing data, but we already have 
> {{Row}}/{{Partition}}/{{Cell}} for that. In practice, those {{Temporal*}} 
> class generates a lot of allocations that we could avoid.
> * The code from CASSANDRA-10060 to avoid multiple reads of the base table 
> with multiple views doesn't work when the update has partition/range 
> tombstones because the code uses 

[2/3] cassandra git commit: ninja: use supplied CL in QueryProcessor::execute

[samt]

ninja: use supplied CL in QueryProcessor::execute


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b15983e8
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b15983e8
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b15983e8

Branch: refs/heads/trunk
Commit: b15983e83bd8e7ee7ddb12871da62fdf72401ba2
Parents: 943e732
Author: Sam Tunnicliffe 
Authored: Fri Apr 29 15:02:29 2016 +0100
Committer: Sam Tunnicliffe 
Committed: Fri Apr 29 15:02:29 2016 +0100

--
 src/java/org/apache/cassandra/cql3/QueryProcessor.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b15983e8/src/java/org/apache/cassandra/cql3/QueryProcessor.java
--
diff --git a/src/java/org/apache/cassandra/cql3/QueryProcessor.java 
b/src/java/org/apache/cassandra/cql3/QueryProcessor.java
index 9ee3e17..da146ef 100644
--- a/src/java/org/apache/cassandra/cql3/QueryProcessor.java
+++ b/src/java/org/apache/cassandra/cql3/QueryProcessor.java
@@ -305,7 +305,7 @@ public class QueryProcessor implements QueryHandler
 try
 {
 ParsedStatement.Prepared prepared = prepareInternal(query);
-ResultMessage result = prepared.statement.execute(state, 
makeInternalOptions(prepared, values));
+ResultMessage result = prepared.statement.execute(state, 
makeInternalOptions(prepared, values, cl));
 if (result instanceof ResultMessage.Rows)
 return 
UntypedResultSet.create(((ResultMessage.Rows)result).result);
 else

[3/3] cassandra git commit: Merge branch 'cassandra-3.0' into trunk

[samt]

Merge branch 'cassandra-3.0' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/99f0ff93
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/99f0ff93
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/99f0ff93

Branch: refs/heads/trunk
Commit: 99f0ff9384257c7aab37df027a8b2f3f53963b6e
Parents: b2d8e88 b15983e
Author: Sam Tunnicliffe 
Authored: Fri Apr 29 15:03:14 2016 +0100
Committer: Sam Tunnicliffe 
Committed: Fri Apr 29 15:03:14 2016 +0100

--

--

[1/3] cassandra git commit: ninja: use supplied CL in QueryProcessor::execute

[samt]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 943e732ca -> b15983e83
  refs/heads/trunk b2d8e8821 -> 99f0ff938


ninja: use supplied CL in QueryProcessor::execute


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b15983e8
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b15983e8
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b15983e8

Branch: refs/heads/cassandra-3.0
Commit: b15983e83bd8e7ee7ddb12871da62fdf72401ba2
Parents: 943e732
Author: Sam Tunnicliffe 
Authored: Fri Apr 29 15:02:29 2016 +0100
Committer: Sam Tunnicliffe 
Committed: Fri Apr 29 15:02:29 2016 +0100

--
 src/java/org/apache/cassandra/cql3/QueryProcessor.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b15983e8/src/java/org/apache/cassandra/cql3/QueryProcessor.java
--
diff --git a/src/java/org/apache/cassandra/cql3/QueryProcessor.java 
b/src/java/org/apache/cassandra/cql3/QueryProcessor.java
index 9ee3e17..da146ef 100644
--- a/src/java/org/apache/cassandra/cql3/QueryProcessor.java
+++ b/src/java/org/apache/cassandra/cql3/QueryProcessor.java
@@ -305,7 +305,7 @@ public class QueryProcessor implements QueryHandler
 try
 {
 ParsedStatement.Prepared prepared = prepareInternal(query);
-ResultMessage result = prepared.statement.execute(state, 
makeInternalOptions(prepared, values));
+ResultMessage result = prepared.statement.execute(state, 
makeInternalOptions(prepared, values, cl));
 if (result instanceof ResultMessage.Rows)
 return 
UntypedResultSet.create(((ResultMessage.Rows)result).result);
 else

[jira] [Commented] (CASSANDRA-11593) getFunctions methods produce too much garbage

[Sam Tunnicliffe (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264058#comment-15264058
 ] 

Sam Tunnicliffe commented on CASSANDRA-11593:
-

+1 on the 3.0 patch too.

> getFunctions methods produce too much garbage
> -
>
> Key: CASSANDRA-11593
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11593
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Benjamin Lerer
>Assignee: Benjamin Lerer
> Fix For: 3.x
>
>
> While profiling an heavy write workload on a single machine, I discover that 
> calls to {{getFunctions}} were producing a lot of garbage.
> Internally, the getFunctions method use {{Iterators}} or {{Iterables}} 
> functions that creates new immutable collections at each level.
> As {{getFunctions}} is called for each SELECT, INSERT, UPDATE or DELETE 
> through the {{checkAccess}} method the impact is not neglectable. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11593) getFunctions methods produce too much garbage

[Sam Tunnicliffe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Tunnicliffe updated CASSANDRA-11593:

Status: Ready to Commit  (was: Patch Available)

> getFunctions methods produce too much garbage
> -
>
> Key: CASSANDRA-11593
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11593
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Benjamin Lerer
>Assignee: Benjamin Lerer
> Fix For: 3.x
>
>
> While profiling an heavy write workload on a single machine, I discover that 
> calls to {{getFunctions}} were producing a lot of garbage.
> Internally, the getFunctions method use {{Iterators}} or {{Iterables}} 
> functions that creates new immutable collections at each level.
> As {{getFunctions}} is called for each SELECT, INSERT, UPDATE or DELETE 
> through the {{checkAccess}} method the impact is not neglectable. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-11683) Code refactor for StorageServiceMBean.forceRepair* methods

[Paulo Motta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta resolved CASSANDRA-11683.
-
Resolution: Won't Fix

We appreciate your patch [~peoplehlj]. Unfortunately 2.1 has reached end of 
development and we're only accepting critical bug fixes at this stage.

It seems this was method signature was already fixed on more recent versions 
(2.2+), so I will close this ticket. Thanks!

> Code refactor for StorageServiceMBean.forceRepair* methods
> --
>
> Key: CASSANDRA-11683
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11683
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Lijun Huang
>Priority: Trivial
> Fix For: 2.1.x
>
>
> For the class StorageServiceMBean, it has below methods, 
> {code:title=org/apache/cassandra/service/StorageServiceMBean.java|borderStyle=solid}
> public int forceRepairAsync(String keyspace, boolean isSequential, 
> Collection dataCenters, Collection hosts,  boolean 
> primaryRange, boolean repairedAt, String... columnFamilies) throws 
> IOException;
> public int forceRepairRangeAsync(String beginToken, String endToken, String 
> keyspaceName, boolean isSequential, Collection dataCenters, 
> Collection hosts, boolean repairedAt, String... columnFamilies) 
> throws IOException;
> public int forceRepairRangeAsync(String beginToken, String endToken, String 
> keyspaceName, boolean isSequential, boolean isLocal, boolean repairedAt, 
> String... columnFamilies);
> {code}
> But in the implementation, the arguments are different from this MBean, 
> please notice the last second argument, from *repairedAt* to *fullRepair*, 
> and actually *fullRepair* is more clear for this API.
> {code:title=org/apache/cassandra/service/StorageService.java|borderStyle=solid}
> public int forceRepairAsync(String keyspace, boolean isSequential, 
> Collection dataCenters, Collection hosts, boolean 
> primaryRange, boolean fullRepair, String... columnFamilies) throws 
> IOException{}
> public int forceRepairRangeAsync(String beginToken, String endToken, String 
> keyspaceName, boolean isSequential, Collection dataCenters, 
> Collection hosts, boolean fullRepair, String... columnFamilies) 
> throws IOException{}
> public int forceRepairRangeAsync(String beginToken, String endToken, String 
> keyspaceName, boolean isSequential, Collection dataCenters, 
> Collection hosts, boolean fullRepair, String... columnFamilies) 
> throws IOException{}
> {code}
> This will make developers confused, for example I met an issue that we didn't 
> want to do a "full" repair but from the MBean I didn't know how to pass the 
> argument.
> I will send out a patch soon, and please help to merge it if we want to fix 
> this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11603) PER PARTITION LIMIT does not work properly for SinglePartition

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer updated CASSANDRA-11603:
---
Component/s: CQL

> PER PARTITION LIMIT does not work properly for SinglePartition
> --
>
> Key: CASSANDRA-11603
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11603
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Benjamin Lerer
>Assignee: Benjamin Lerer
> Fix For: 3.6
>
> Attachments: 11603-trunk.txt
>
>
> When the {{PER PARTITION LIMIT}} is greater than the page size the limit is 
> not respected for single or multi partitions queries.
> The problem can be reproduced using the java driver with the following code:
> {code}
> session = cluster.connect();
> session.execute("CREATE KEYSPACE IF NOT EXISTS test WITH REPLICATION 
> = {'class' : 'SimpleStrategy', 'replication_factor' : '1'}");
> session.execute("USE test");
> session.execute("DROP TABLE IF EXISTS test");
> session.execute("CREATE TABLE IF NOT EXISTS test (a int, b int, c 
> int, PRIMARY KEY(a, b))");
> PreparedStatement prepare = session.prepare("INSERT INTO test (a, b, 
> c) VALUES (?, ?, ?);");
> for (int i = 0; i < 5; i++)
> for (int j = 0; j < 10; j++)
> session.execute(prepare.bind(i, j, i + j));
> ResultSet rs = session.execute(session.newSimpleStatement("SELECT * 
> FROM test WHERE a = 1 PER PARTITION LIMIT 3")
>   .setFetchSize(2));
> for (Row row : rs)
> {
> System.out.println(row);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11603) PER PARTITION LIMIT does not work properly for SinglePartition

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer updated CASSANDRA-11603:
---
   Resolution: Fixed
Fix Version/s: (was: 3.x)
   3.6
   Status: Resolved  (was: Ready to Commit)

Committed to trunk at b2d8e88217d0532d54cba0cfae8ab6951df66f35.

> PER PARTITION LIMIT does not work properly for SinglePartition
> --
>
> Key: CASSANDRA-11603
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11603
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Benjamin Lerer
>Assignee: Benjamin Lerer
> Fix For: 3.6
>
> Attachments: 11603-trunk.txt
>
>
> When the {{PER PARTITION LIMIT}} is greater than the page size the limit is 
> not respected for single or multi partitions queries.
> The problem can be reproduced using the java driver with the following code:
> {code}
> session = cluster.connect();
> session.execute("CREATE KEYSPACE IF NOT EXISTS test WITH REPLICATION 
> = {'class' : 'SimpleStrategy', 'replication_factor' : '1'}");
> session.execute("USE test");
> session.execute("DROP TABLE IF EXISTS test");
> session.execute("CREATE TABLE IF NOT EXISTS test (a int, b int, c 
> int, PRIMARY KEY(a, b))");
> PreparedStatement prepare = session.prepare("INSERT INTO test (a, b, 
> c) VALUES (?, ?, ?);");
> for (int i = 0; i < 5; i++)
> for (int j = 0; j < 10; j++)
> session.execute(prepare.bind(i, j, i + j));
> ResultSet rs = session.execute(session.newSimpleStatement("SELECT * 
> FROM test WHERE a = 1 PER PARTITION LIMIT 3")
>   .setFetchSize(2));
> for (Row row : rs)
> {
> System.out.println(row);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11603) PER PARTITION LIMIT does not work properly for SinglePartition

[Benjamin Lerer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263997#comment-15263997
 ] 

Benjamin Lerer commented on CASSANDRA-11603:


Thanks for the review.

> PER PARTITION LIMIT does not work properly for SinglePartition
> --
>
> Key: CASSANDRA-11603
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11603
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Benjamin Lerer
>Assignee: Benjamin Lerer
> Fix For: 3.x
>
> Attachments: 11603-trunk.txt
>
>
> When the {{PER PARTITION LIMIT}} is greater than the page size the limit is 
> not respected for single or multi partitions queries.
> The problem can be reproduced using the java driver with the following code:
> {code}
> session = cluster.connect();
> session.execute("CREATE KEYSPACE IF NOT EXISTS test WITH REPLICATION 
> = {'class' : 'SimpleStrategy', 'replication_factor' : '1'}");
> session.execute("USE test");
> session.execute("DROP TABLE IF EXISTS test");
> session.execute("CREATE TABLE IF NOT EXISTS test (a int, b int, c 
> int, PRIMARY KEY(a, b))");
> PreparedStatement prepare = session.prepare("INSERT INTO test (a, b, 
> c) VALUES (?, ?, ?);");
> for (int i = 0; i < 5; i++)
> for (int j = 0; j < 10; j++)
> session.execute(prepare.bind(i, j, i + j));
> ResultSet rs = session.execute(session.newSimpleStatement("SELECT * 
> FROM test WHERE a = 1 PER PARTITION LIMIT 3")
>   .setFetchSize(2));
> for (Row row : rs)
> {
> System.out.println(row);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11603) PER PARTITION LIMIT does not work properly for SinglePartition

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer updated CASSANDRA-11603:
---
Attachment: 11603-trunk.txt

> PER PARTITION LIMIT does not work properly for SinglePartition
> --
>
> Key: CASSANDRA-11603
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11603
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Benjamin Lerer
>Assignee: Benjamin Lerer
> Fix For: 3.x
>
> Attachments: 11603-trunk.txt
>
>
> When the {{PER PARTITION LIMIT}} is greater than the page size the limit is 
> not respected for single or multi partitions queries.
> The problem can be reproduced using the java driver with the following code:
> {code}
> session = cluster.connect();
> session.execute("CREATE KEYSPACE IF NOT EXISTS test WITH REPLICATION 
> = {'class' : 'SimpleStrategy', 'replication_factor' : '1'}");
> session.execute("USE test");
> session.execute("DROP TABLE IF EXISTS test");
> session.execute("CREATE TABLE IF NOT EXISTS test (a int, b int, c 
> int, PRIMARY KEY(a, b))");
> PreparedStatement prepare = session.prepare("INSERT INTO test (a, b, 
> c) VALUES (?, ?, ?);");
> for (int i = 0; i < 5; i++)
> for (int j = 0; j < 10; j++)
> session.execute(prepare.bind(i, j, i + j));
> ResultSet rs = session.execute(session.newSimpleStatement("SELECT * 
> FROM test WHERE a = 1 PER PARTITION LIMIT 3")
>   .setFetchSize(2));
> for (Row row : rs)
> {
> System.out.println(row);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

cassandra git commit: Fix PER PARTITION LIMIT for single and multi partitions queries

[blerer]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 0e5f2c807 -> b2d8e8821


Fix PER PARTITION LIMIT for single and multi partitions queries

patch by Benjamin Lerer; reviewed by Alex Petrov for CASSANDRA-11603


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b2d8e882
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b2d8e882
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b2d8e882

Branch: refs/heads/trunk
Commit: b2d8e88217d0532d54cba0cfae8ab6951df66f35
Parents: 0e5f2c8
Author: Benjamin Lerer 
Authored: Fri Apr 29 14:41:40 2016 +0200
Committer: Benjamin Lerer 
Committed: Fri Apr 29 14:43:36 2016 +0200

--
 CHANGES.txt| 1 +
 .../org/apache/cassandra/db/SinglePartitionReadCommand.java| 6 +++---
 .../apache/cassandra/service/pager/MultiPartitionPager.java| 2 +-
 .../apache/cassandra/service/pager/SinglePartitionPager.java   | 6 +-
 4 files changed, 10 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b2d8e882/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 0315f1b..e11ebd8 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.6
+ * Fix PER PARTITION LIMIT for single and multi partitions queries 
(CASSANDRA-11603)
  * Add uncompressed chunk cache for RandomAccessReader (CASSANDRA-5863)
  * Clarify ClusteringPrefix hierarchy (CASSANDRA-11213)
  * Always perform collision check before joining ring (CASSANDRA-10134)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b2d8e882/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
--
diff --git a/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java 
b/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
index c5dd7d6..a6d6047 100644
--- a/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
+++ b/src/java/org/apache/cassandra/db/SinglePartitionReadCommand.java
@@ -279,11 +279,11 @@ public class SinglePartitionReadCommand extends 
ReadCommand
  * @param lastReturned the last row returned by the previous page. The 
newly created command
  * will only query row that comes after this (in query order). This can be 
{@code null} if this
  * is the first page.
- * @param pageSize the size to use for the page to query.
+ * @param limits the limits to use for the page to query.
  *
  * @return the newly create command.
  */
-public SinglePartitionReadCommand forPaging(Clustering lastReturned, int 
pageSize)
+public SinglePartitionReadCommand forPaging(Clustering lastReturned, 
DataLimits limits)
 {
 // We shouldn't have set digest yet when reaching that point
 assert !isDigestQuery();
@@ -292,7 +292,7 @@ public class SinglePartitionReadCommand extends ReadCommand
   nowInSec(),
   columnFilter(),
   rowFilter(),
-  limits().forPaging(pageSize),
+  limits,
   partitionKey(),
   lastReturned == null ? clusteringIndexFilter() : 
clusteringIndexFilter.forPaging(metadata().comparator, lastReturned, false));
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b2d8e882/src/java/org/apache/cassandra/service/pager/MultiPartitionPager.java
--
diff --git 
a/src/java/org/apache/cassandra/service/pager/MultiPartitionPager.java 
b/src/java/org/apache/cassandra/service/pager/MultiPartitionPager.java
index 922df2e..57d6c62 100644
--- a/src/java/org/apache/cassandra/service/pager/MultiPartitionPager.java
+++ b/src/java/org/apache/cassandra/service/pager/MultiPartitionPager.java
@@ -88,7 +88,7 @@ public class MultiPartitionPager implements QueryPager
 return null;
 
 PagingState state = pagers[current].state();
-return new PagingState(pagers[current].key(), state == null ? null : 
state.rowMark, remaining, Integer.MAX_VALUE);
+return new PagingState(pagers[current].key(), state == null ? null : 
state.rowMark, remaining, pagers[current].remainingInPartition());
 }
 
 public boolean isExhausted()

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b2d8e882/src/java/org/apache/cassandra/service/pager/SinglePartitionPager.java
--
diff --git 
a/src/java/org/apache/cassandra/service/pager/SinglePartitionPager.java 
b/src/java/org/apache/cassandra/service/pager/SinglePartitionPager.java
index 6f17284..acb55bb 100644
--- 

[jira] [Commented] (CASSANDRA-11593) getFunctions methods produce too much garbage

[Benjamin Lerer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263984#comment-15263984
 ] 

Benjamin Lerer commented on CASSANDRA-11593:


I pushed a new patch for 3.0.x.
|[3.0|https://github.com/blerer/cassandra/tree/11593-3.0]|[utest|http://cassci.datastax.com/view/Dev/view/blerer/job/blerer-11593-3.0-testall/lastCompletedBuild/testReport/]|[dtest|http://cassci.datastax.com/view/Dev/view/blerer/job/blerer-11593-3.0-dtest/2/]|


> getFunctions methods produce too much garbage
> -
>
> Key: CASSANDRA-11593
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11593
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Benjamin Lerer
>Assignee: Benjamin Lerer
> Fix For: 3.x
>
>
> While profiling an heavy write workload on a single machine, I discover that 
> calls to {{getFunctions}} were producing a lot of garbage.
> Internally, the getFunctions method use {{Iterators}} or {{Iterables}} 
> functions that creates new immutable collections at each level.
> As {{getFunctions}} is called for each SELECT, INSERT, UPDATE or DELETE 
> through the {{checkAccess}} method the impact is not neglectable. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)