[jira] [Updated] (CASSANDRA-12379) CQLSH completion test broken by #12236

2016-08-07 Thread Stefania (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefania updated CASSANDRA-12379:
-
Issue Type: Test  (was: Bug)

> CQLSH completion test broken by #12236
> --
>
> Key: CASSANDRA-12379
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12379
> Project: Cassandra
>  Issue Type: Test
>  Components: Tools
>Reporter: Sylvain Lebresne
>Assignee: Stefania
> Fix For: 3.8
>
>
> The commit of CASSANDRA-12236 appears to have broken [cqlsh completion 
> tests|http://cassci.datastax.com/job/cassandra-3.8_cqlsh_tests/6/cython=yes,label=ctool-lab/testReport/junit/cqlshlib.test.test_cqlsh_completion/TestCqlshCompletion/test_complete_in_create_columnfamily/].
>  For the error message I suspect this may have to do with something like the 
> test comparing the completion output to what DESCRIBE shows and the later now 
> doesn't include the {{cdc}} option by default.
> Anyway, I'm not really familiar with cqlsh completion nor it's test so I'm 
> not sure what's the best option. I don't think we want to remove {{cdc}} from 
> completion so I suspect we want to either special case the test somehow (no 
> clue how to do that), or make the test run with cdc enabled so it doesn't 
> complain (which I think mostly apply a change to the CI environment since it 
> seems the tests themselves don't spin up the cluster).
> Anyway, pushing that fix to someone else as I'm not competent here and I 
> have't even be able to run those cqlsh test so far (getting stuck at the test 
> telling me that "No appropriate python interpreter found", even though I 
> totally have an appropriate interpreter and cqlsh works perfectly if I 
> execute it directly). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (CASSANDRA-12379) CQLSH completion test broken by #12236

2016-08-07 Thread Stefania (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefania updated CASSANDRA-12379:
-
   Resolution: Fixed
Fix Version/s: 3.8
   Status: Resolved  (was: Patch Available)

> CQLSH completion test broken by #12236
> --
>
> Key: CASSANDRA-12379
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12379
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Sylvain Lebresne
>Assignee: Stefania
> Fix For: 3.8
>
>
> The commit of CASSANDRA-12236 appears to have broken [cqlsh completion 
> tests|http://cassci.datastax.com/job/cassandra-3.8_cqlsh_tests/6/cython=yes,label=ctool-lab/testReport/junit/cqlshlib.test.test_cqlsh_completion/TestCqlshCompletion/test_complete_in_create_columnfamily/].
>  For the error message I suspect this may have to do with something like the 
> test comparing the completion output to what DESCRIBE shows and the later now 
> doesn't include the {{cdc}} option by default.
> Anyway, I'm not really familiar with cqlsh completion nor it's test so I'm 
> not sure what's the best option. I don't think we want to remove {{cdc}} from 
> completion so I suspect we want to either special case the test somehow (no 
> clue how to do that), or make the test run with cdc enabled so it doesn't 
> complain (which I think mostly apply a change to the CI environment since it 
> seems the tests themselves don't spin up the cluster).
> Anyway, pushing that fix to someone else as I'm not competent here and I 
> have't even be able to run those cqlsh test so far (getting stuck at the test 
> telling me that "No appropriate python interpreter found", even though I 
> totally have an appropriate interpreter and cqlsh works perfectly if I 
> execute it directly). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (CASSANDRA-12379) CQLSH completion test broken by #12236

2016-08-07 Thread Stefania (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefania updated CASSANDRA-12379:
-
Component/s: Tools

> CQLSH completion test broken by #12236
> --
>
> Key: CASSANDRA-12379
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12379
> Project: Cassandra
>  Issue Type: Test
>  Components: Tools
>Reporter: Sylvain Lebresne
>Assignee: Stefania
> Fix For: 3.8
>
>
> The commit of CASSANDRA-12236 appears to have broken [cqlsh completion 
> tests|http://cassci.datastax.com/job/cassandra-3.8_cqlsh_tests/6/cython=yes,label=ctool-lab/testReport/junit/cqlshlib.test.test_cqlsh_completion/TestCqlshCompletion/test_complete_in_create_columnfamily/].
>  For the error message I suspect this may have to do with something like the 
> test comparing the completion output to what DESCRIBE shows and the later now 
> doesn't include the {{cdc}} option by default.
> Anyway, I'm not really familiar with cqlsh completion nor it's test so I'm 
> not sure what's the best option. I don't think we want to remove {{cdc}} from 
> completion so I suspect we want to either special case the test somehow (no 
> clue how to do that), or make the test run with cdc enabled so it doesn't 
> complain (which I think mostly apply a change to the CI environment since it 
> seems the tests themselves don't spin up the cluster).
> Anyway, pushing that fix to someone else as I'm not competent here and I 
> have't even be able to run those cqlsh test so far (getting stuck at the test 
> telling me that "No appropriate python interpreter found", even though I 
> totally have an appropriate interpreter and cqlsh works perfectly if I 
> execute it directly). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (CASSANDRA-12379) CQLSH completion test broken by #12236

2016-08-07 Thread Stefania (JIRA)

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411230#comment-15411230
 ] 

Stefania commented on CASSANDRA-12379:
--

I've edited the cqlsh CI jobs for 
[3.9|http://cassci.datastax.com/view/cassandra-3.9/job/cassandra-3.9_cqlsh_tests/8//]
 and [trunk|http://cassci.datastax.com/view/trunk/job/trunk_cqlsh_tests/34/] 
and relaunched them. They both completed without errors, closing.

> CQLSH completion test broken by #12236
> --
>
> Key: CASSANDRA-12379
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12379
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Sylvain Lebresne
>Assignee: Stefania
>
> The commit of CASSANDRA-12236 appears to have broken [cqlsh completion 
> tests|http://cassci.datastax.com/job/cassandra-3.8_cqlsh_tests/6/cython=yes,label=ctool-lab/testReport/junit/cqlshlib.test.test_cqlsh_completion/TestCqlshCompletion/test_complete_in_create_columnfamily/].
>  For the error message I suspect this may have to do with something like the 
> test comparing the completion output to what DESCRIBE shows and the later now 
> doesn't include the {{cdc}} option by default.
> Anyway, I'm not really familiar with cqlsh completion nor it's test so I'm 
> not sure what's the best option. I don't think we want to remove {{cdc}} from 
> completion so I suspect we want to either special case the test somehow (no 
> clue how to do that), or make the test run with cdc enabled so it doesn't 
> complain (which I think mostly apply a change to the CI environment since it 
> seems the tests themselves don't spin up the cluster).
> Anyway, pushing that fix to someone else as I'm not competent here and I 
> have't even be able to run those cqlsh test so far (getting stuck at the test 
> telling me that "No appropriate python interpreter found", even though I 
> totally have an appropriate interpreter and cqlsh works perfectly if I 
> execute it directly). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (CASSANDRA-11521) Implement streaming for bulk read requests

2016-08-07 Thread Stefania (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefania updated CASSANDRA-11521:
-
Status: Open  (was: Patch Available)

Rebasing on CASSANDRA-10707 is a bit problematic, both patches modified select 
statement and the pagers in a way that is not necessarily compatible. So back 
to in progress.

> Implement streaming for bulk read requests
> --
>
> Key: CASSANDRA-11521
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11521
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Local Write-Read Paths
>Reporter: Stefania
>Assignee: Stefania
>  Labels: client-impacting, protocolv5
> Fix For: 3.x
>
> Attachments: final-patch-jfr-profiles-1.zip
>
>
> Allow clients to stream data from a C* host, bypassing the coordination layer 
> and eliminating the need to query individual pages one by one.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (CASSANDRA-12403) Slow query detecting

2016-08-07 Thread Shogo Hoshii (JIRA)

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411195#comment-15411195
 ] 

Shogo Hoshii commented on CASSANDRA-12403:
--

Hello [~Stefania],

Thank you very much for quick and very detailed comment.
I will look into `Monitorable`, and `MonitoringTask`.

> Slow query detecting
> 
>
> Key: CASSANDRA-12403
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12403
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Shogo Hoshii
>Assignee: Shogo Hoshii
> Attachments: sample.log, slow_query.patch
>
>
> Hello,
> In cassandra production environment, users sometimes build anti-pattern 
> tables and throw queries in inefficient manners.
> So I would like to suggest a feature that enables to log slow query.
> The feature can help cassandra operators to identify bad query patterns.
> Then operators can give advices about queries and data model to users who 
> don't know cassandra so much.
> This ticket is related to CASSANDRA-6226, and I focus on detecting bad query 
> patterns, not aborting them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (CASSANDRA-11701) [windows] dtest failure in cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_reading_with_skip_and_max_rows

2016-08-07 Thread Stefania (JIRA)

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411185#comment-15411185
 ] 

Stefania commented on CASSANDRA-11701:
--

CI results on Linux are good. 

On Windows I multiplexed copy tests 5 times without problems, [build 
#8|http://cassci.datastax.com/job/stef1927-dtest-multiplex-win32/8/] but [build 
#9|http://cassci.datastax.com/job/stef1927-dtest-multiplex-win32/9/] was 
aborted. So I launched [build 
#10|http://cassci.datastax.com/job/stef1927-dtest-multiplex-win32/10/] with 
some additional debug information.

> [windows] dtest failure in 
> cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_reading_with_skip_and_max_rows
> -
>
> Key: CASSANDRA-11701
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11701
> Project: Cassandra
>  Issue Type: Test
>Reporter: Russ Hatch
>Assignee: Stefania
>  Labels: dtest, windows
>
> looks to be an assertion problem, so could be test or cassandra related:
> e.g.:
> {noformat}
> 1 != 331
> {noformat}
> http://cassci.datastax.com/job/trunk_dtest_win32/404/testReport/cqlsh_tests.cqlsh_copy_tests/CqlshCopyTest/test_reading_with_skip_and_max_rows
> Failed on CassCI build trunk_dtest_win32 #404



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Comment Edited] (CASSANDRA-11701) [windows] dtest failure in cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_reading_with_skip_and_max_rows

2016-08-07 Thread Stefania (JIRA)

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408997#comment-15408997
 ] 

Stefania edited comment on CASSANDRA-11701 at 8/8/16 2:10 AM:
--

I rewrote the patch to avoid using a thread lock, since it cannot be pickl-ed 
on Windows. I extracted the pipe out of the channels, so we only transfer pipes 
and not channels to the child processes, which then create the channels after 
forking. To avoid redundant threads, which are only required for sending, I've 
separated receiving from sending channels. Only sending channels create a 
feeding thread, right in the constructor so the lock is no longer required.

I'm not sure if we want this fix in 2.1 or not, it is not a critical bug but it 
is a regression compared to the old cqlsh COPY functionality. It is a rare 
failure but it can occur if the main thread of a child process needs to send an 
error when the receiving thread is already sending results.

||2.1||2.2||3.0||3.9||trunk||
|[patch|https://github.com/stef1927/cassandra/commits/11701-cqlsh-2.1]|[patch|https://github.com/stef1927/cassandra/commits/11701-cqlsh-2.2]|[patch|https://github.com/stef1927/cassandra/commits/11701-cqlsh-3.0]|[patch|https://github.com/stef1927/cassandra/commits/11701-cqlsh-3.9]|[patch|https://github.com/stef1927/cassandra/commits/11701-cqlsh]|
|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11701-cqlsh-2.1-cqlsh-tests/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11701-cqlsh-2.2-cqlsh-tests/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11701-cqlsh-3.0-cqlsh-tests/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11701-cqlsh-3.9-cqlsh-tests/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11701-cqlsh-cqlsh-tests/]|

The windows test results will be available 
[here|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-dtest-multiplex-win32/],
 build -#7 and #8- #8 and #9.


was (Author: stefania):
I rewrote the patch to avoid using a thread lock, since it cannot be pickl-ed 
on Windows. I extracted the pipe out of the channels, so we only transfer pipes 
and not channels to the child processes, which then create the channels after 
forking. To avoid redundant threads, which are only required for sending, I've 
separated receiving from sending channels. Only sending channels create a 
feeding thread, right in the constructor so the lock is no longer required.

I'm not sure if we want this fix in 2.1 or not, it is not a critical bug but it 
is a regression compared to the old cqlsh COPY functionality. It is a rare 
failure but it can occur if the main thread of a child process needs to send an 
error when the receiving thread is already sending results.

||2.1||2.2||3.0||3.9||trunk||
|[patch|https://github.com/stef1927/cassandra/commits/11701-cqlsh-2.1]|[patch|https://github.com/stef1927/cassandra/commits/11701-cqlsh-2.2]|[patch|https://github.com/stef1927/cassandra/commits/11701-cqlsh-3.0]|[patch|https://github.com/stef1927/cassandra/commits/11701-cqlsh-3.9]|[patch|https://github.com/stef1927/cassandra/commits/11701-cqlsh]|
|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11701-cqlsh-2.1-cqlsh-tests/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11701-cqlsh-2.2-cqlsh-tests/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11701-cqlsh-3.0-cqlsh-tests/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11701-cqlsh-3.9-cqlsh-tests/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11701-cqlsh-cqlsh-tests/]|

The windows test results will be available 
[here|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-dtest-multiplex-win32/],
 build #7 and #8.

> [windows] dtest failure in 
> cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_reading_with_skip_and_max_rows
> -
>
> Key: CASSANDRA-11701
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11701
> Project: Cassandra
>  Issue Type: Test
>Reporter: Russ Hatch
>Assignee: Stefania
>  Labels: dtest, windows
>
> looks to be an assertion problem, so could be test or cassandra related:
> e.g.:
> {noformat}
> 1 != 331
> {noformat}
> http://cassci.datastax.com/job/trunk_dtest_win32/404/testReport/cqlsh_tests.cqlsh_copy_tests/CqlshCopyTest/test_reading_with_skip_and_max_rows
> Failed on CassCI build trunk_dtest_win32 #404



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (CASSANDRA-12404) ColumnFamilyStore.getIfExists

2016-08-07 Thread stone (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stone updated CASSANDRA-12404:
--
Description: 
ColumnFamilyStore.getIfExists("ks","cf")
when it does not exist ks.cf,it should return null,but now there is a 
Nullpointexception.
see heartbeat.PNG

  was:
ColumnFamilyStore.getIfExists("ks","cf")
when it does not exist ks.cf,it should return null,but now there is a 
Nullpointexception.


> ColumnFamilyStore.getIfExists
> -
>
> Key: CASSANDRA-12404
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12404
> Project: Cassandra
>  Issue Type: Bug
>  Components: Packaging
>Reporter: stone
> Attachments: heartbeat.PNG
>
>
> ColumnFamilyStore.getIfExists("ks","cf")
> when it does not exist ks.cf,it should return null,but now there is a 
> Nullpointexception.
> see heartbeat.PNG



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (CASSANDRA-12404) ColumnFamilyStore.getIfExists

2016-08-07 Thread stone (JIRA)
stone created CASSANDRA-12404:
-

 Summary: ColumnFamilyStore.getIfExists
 Key: CASSANDRA-12404
 URL: https://issues.apache.org/jira/browse/CASSANDRA-12404
 Project: Cassandra
  Issue Type: Bug
  Components: Packaging
Reporter: stone
 Attachments: heartbeat.PNG

ColumnFamilyStore.getIfExists("ks","cf")
when it does not exist ks.cf,it should return null,but now there is a 
Nullpointexception.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (CASSANDRA-12403) Slow query detecting

2016-08-07 Thread Stefania (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefania updated CASSANDRA-12403:
-
Reviewer: Stefania

> Slow query detecting
> 
>
> Key: CASSANDRA-12403
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12403
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Shogo Hoshii
>Assignee: Shogo Hoshii
> Attachments: sample.log, slow_query.patch
>
>
> Hello,
> In cassandra production environment, users sometimes build anti-pattern 
> tables and throw queries in inefficient manners.
> So I would like to suggest a feature that enables to log slow query.
> The feature can help cassandra operators to identify bad query patterns.
> Then operators can give advices about queries and data model to users who 
> don't know cassandra so much.
> This ticket is related to CASSANDRA-6226, and I focus on detecting bad query 
> patterns, not aborting them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (CASSANDRA-12403) Slow query detecting

2016-08-07 Thread Stefania (JIRA)

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411168#comment-15411168
 ] 

Stefania commented on CASSANDRA-12403:
--

Thank you for the patch. I think this is definitely something that operators 
will find very useful.

CASSANDRA-7392 introduced a poor man's "slow query log" for queries that timed 
out, the code is in 
[{{MonitoringTask}}|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/monitoring/MonitoringTask.java#L144]
 in the 
[db.monitoring|https://github.com/apache/cassandra/tree/trunk/src/java/org/apache/cassandra/db/monitoring]
 package. This is done replica side rather than coordinator side. 

We may want to build on that rather than log slow queries coordinator side. If 
we log coordinator side, it may be that some queries took a long time because a 
replica was overloaded. Replica side we can instead measure the accurate query 
time. Each read command is already monitored, we basically just need to add a 
new state to {{Monitorable}}, and notify {{MonitoringTask}} when this state is 
triggered.

I would continue logging in aggregate and at debug level, like it is currently 
done for queries that timed out. In {{MonitoringTask}}, we would basically 
first log slow queries, if any, then queries that timed out, if any. We can do 
this with two separate queues or by generalizing (and renaming) 
{{FailedOperation}}.


> Slow query detecting
> 
>
> Key: CASSANDRA-12403
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12403
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Shogo Hoshii
>Assignee: Shogo Hoshii
> Attachments: sample.log, slow_query.patch
>
>
> Hello,
> In cassandra production environment, users sometimes build anti-pattern 
> tables and throw queries in inefficient manners.
> So I would like to suggest a feature that enables to log slow query.
> The feature can help cassandra operators to identify bad query patterns.
> Then operators can give advices about queries and data model to users who 
> don't know cassandra so much.
> This ticket is related to CASSANDRA-6226, and I focus on detecting bad query 
> patterns, not aborting them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (CASSANDRA-12274) mx4j not work in 3.0.8

2016-08-07 Thread Edward Ribeiro (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Edward Ribeiro updated CASSANDRA-12274:
---
Attachment: mx4j-error-log.txt

Hi [~snazy],

I've discovered by accident that this regression was indirectly caused by 
CASSANDRA-9402. I am testing on C* 3.9 branch.

I could check that by putting the mx4j-tools.jar in the lib directory then 
commenting/uncommenting this line: 
https://github.com/apache/cassandra/blob/143d16961a67cc9d5608605ee2561253de629d2c/src/java/org/apache/cassandra/service/CassandraDaemon.java#L189

Looks like, the addition of a {{SecurityManager}} makes {{mx4j}} throws an 
exception and show a blank page. I have attached the Runtime error exception. 
{{Mx4jTool}} is the class in charge of loading mx4j if it is in the classpath.

Last but not least, I found a small typo ("panalty") here: 
https://github.com/apache/cassandra/blob/3dcbe90e02440e6ee534f643c7603d50ca08482b/src/java/org/apache/cassandra/cql3/functions/ThreadAwareSecurityManager.java#L36
 

I am sorry for not being able to provide a ready solution, but this 
SecurityManager + ClassLoader + XSLT stuff is beyond me now. :(


> mx4j not work in 3.0.8
> --
>
> Key: CASSANDRA-12274
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12274
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: suse 12
> java 1.8.0_60
> mx4j 3.0.2
>Reporter: Ilya
> Attachments: mx4j-error-log.txt
>
>
> After update from 2.1 to 3.x version mx4j page begin empty
> {code}
> $ curl -i cassandra1:8081
> HTTP/1.0 200 OK
> expires: now
> Server: MX4J-HTTPD/1.0
> Cache-Control: no-cache
> pragma: no-cache
> Content-Type: text/html
> {code}
> There are no errors in the log.
> logs:
> {code}
> ~ $ grep -i mx4j /local/apache-cassandra/logs/system.log | tail -2
> INFO  [main] 2016-07-22 13:48:00,352 CassandraDaemon.java:432 - JVM 
> Arguments: [-Xloggc:/local/apache-cassandra//logs/gc.log, 
> -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, 
> -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/local/tmp, -Xss256k, 
> -XX:StringTableSize=103, -XX:+AlwaysPreTouch, -XX:+UseTLAB, 
> -XX:+ResizeTLAB, -XX:+UseNUMA, -Djava.net.preferIPv4Stack=true, -Xms512M, 
> -Xmx1G, -XX:+UseG1GC, -XX:G1RSetUpdatingPauseTimePercent=5, 
> -XX:MaxGCPauseMillis=500, -XX:InitiatingHeapOccupancyPercent=25, 
> -XX:G1HeapRegionSize=32m, -XX:ParallelGCThreads=16, -XX:+PrintGCDetails, 
> -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, 
> -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, 
> -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=10M, 
> -XX:CompileCommandFile=/local/apache-cassandra//conf/hotspot_compiler, 
> -javaagent:/local/apache-cassandra//lib/jamm-0.3.0.jar, 
> -Djava.rmi.server.hostname=cassandra1.d3, 
> -Dcom.sun.management.jmxremote.port=7199, 
> -Dcom.sun.management.jmxremote.rmi.port=7199, 
> -Dcom.sun.management.jmxremote.ssl=false, 
> -Dcom.sun.management.jmxremote.authenticate=false, 
> -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password,
>  -Djava.library.path=/local/apache-cassandra//lib/sigar-bin, -Dmx4jport=8081, 
> -Dlogback.configurationFile=logback.xml, 
> -Dcassandra.logdir=/local/apache-cassandra//logs, 
> -Dcassandra.storagedir=/local/apache-cassandra//data, 
> -Dcassandra-pidfile=/local/apache-cassandra/run/cassandra.pid]
> INFO  [main] 2016-07-22 13:48:04,045 Mx4jTool.java:63 - mx4j successfuly 
> loaded
> {code}
> {code}
> ~ $ sudo lsof -i:8081
> COMMAND   PID  USER   FD   TYPEDEVICE SIZE/OFF NODE NAME
> java14489 cassandra   86u  IPv4 381043582  0t0  TCP 
> cassandra1.d3:sunproxyadmin (LISTEN)
> {code}
> I checked versions 3.0.8  and 3.5, result the same - not work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (CASSANDRA-12403) Slow query detecting

2016-08-07 Thread Shogo Hoshii (JIRA)

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1542#comment-1542
 ] 

Shogo Hoshii commented on CASSANDRA-12403:
--

Hello [~ifesdjeen]

I appreciate your advice.
It is very useful in the case cassandra operator can access logs in client 
server.
But I cannot do that because in my company my team(operators) has a lot of 
users from different departments and operators do not have permission to client 
servers.
And it is also difficult for my team to let all of users to use specific driver 
because the organisations are separated.
(I guess some of other cassandra operators are in same circumstance as my team 
is.)
So I would like to say we need to log slow query in server side.

> Slow query detecting
> 
>
> Key: CASSANDRA-12403
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12403
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Shogo Hoshii
>Assignee: Shogo Hoshii
> Attachments: sample.log, slow_query.patch
>
>
> Hello,
> In cassandra production environment, users sometimes build anti-pattern 
> tables and throw queries in inefficient manners.
> So I would like to suggest a feature that enables to log slow query.
> The feature can help cassandra operators to identify bad query patterns.
> Then operators can give advices about queries and data model to users who 
> don't know cassandra so much.
> This ticket is related to CASSANDRA-6226, and I focus on detecting bad query 
> patterns, not aborting them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (CASSANDRA-12403) Slow query detecting

2016-08-07 Thread Alex Petrov (JIRA)

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411024#comment-15411024
 ] 

Alex Petrov commented on CASSANDRA-12403:
-

Java Driver has a {{QueryLogger.SLOW}} that might be helpful in such 
circumstances. The big advantage of having it on client side is that you can 
also catch queries that are caused by slowness of client and network latencies 
between client and the server.

> Slow query detecting
> 
>
> Key: CASSANDRA-12403
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12403
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Shogo Hoshii
>Assignee: Shogo Hoshii
> Attachments: sample.log, slow_query.patch
>
>
> Hello,
> In cassandra production environment, users sometimes build anti-pattern 
> tables and throw queries in inefficient manners.
> So I would like to suggest a feature that enables to log slow query.
> The feature can help cassandra operators to identify bad query patterns.
> Then operators can give advices about queries and data model to users who 
> don't know cassandra so much.
> This ticket is related to CASSANDRA-6226, and I focus on detecting bad query 
> patterns, not aborting them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Resolved] (CASSANDRA-12356) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_2_1_x_To_indev_2_2_x.bug_10652_test

2016-08-07 Thread Philip Thompson (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Thompson resolved CASSANDRA-12356.
-
Resolution: Duplicate

> dtest failure in 
> upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_2_1_x_To_indev_2_2_x.bug_10652_test
> -
>
> Key: CASSANDRA-12356
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12356
> Project: Cassandra
>  Issue Type: Test
>Reporter: Craig Kodman
>Assignee: DS Test Eng
>  Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log, 
> node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.2_dtest_upgrade/7/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_2_1_x_To_indev_2_2_x/bug_10652_test



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (CASSANDRA-12403) Slow query detecting

2016-08-07 Thread Shogo Hoshii (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shogo Hoshii updated CASSANDRA-12403:
-
Status: Patch Available  (was: Open)

I attached the patch. I hope someone would review it!

> Slow query detecting
> 
>
> Key: CASSANDRA-12403
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12403
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Shogo Hoshii
>Assignee: Shogo Hoshii
> Attachments: sample.log, slow_query.patch
>
>
> Hello,
> In cassandra production environment, users sometimes build anti-pattern 
> tables and throw queries in inefficient manners.
> So I would like to suggest a feature that enables to log slow query.
> The feature can help cassandra operators to identify bad query patterns.
> Then operators can give advices about queries and data model to users who 
> don't know cassandra so much.
> This ticket is related to CASSANDRA-6226, and I focus on detecting bad query 
> patterns, not aborting them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (CASSANDRA-12403) Slow query detecting

2016-08-07 Thread Shogo Hoshii (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shogo Hoshii updated CASSANDRA-12403:
-
Attachment: sample.log

This is a sample result with setting
- enable_logging_slow_select_query: true
- logging_slow_select_query_threshold_in_ms: 10


> Slow query detecting
> 
>
> Key: CASSANDRA-12403
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12403
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Shogo Hoshii
>Assignee: Shogo Hoshii
> Attachments: sample.log, slow_query.patch
>
>
> Hello,
> In cassandra production environment, users sometimes build anti-pattern 
> tables and throw queries in inefficient manners.
> So I would like to suggest a feature that enables to log slow query.
> The feature can help cassandra operators to identify bad query patterns.
> Then operators can give advices about queries and data model to users who 
> don't know cassandra so much.
> This ticket is related to CASSANDRA-6226, and I focus on detecting bad query 
> patterns, not aborting them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (CASSANDRA-12403) Slow query detecting

2016-08-07 Thread Shogo Hoshii (JIRA)
Shogo Hoshii created CASSANDRA-12403:


 Summary: Slow query detecting
 Key: CASSANDRA-12403
 URL: https://issues.apache.org/jira/browse/CASSANDRA-12403
 Project: Cassandra
  Issue Type: Improvement
Reporter: Shogo Hoshii
Assignee: Shogo Hoshii
 Attachments: slow_query.patch

Hello,

In cassandra production environment, users sometimes build anti-pattern tables 
and throw queries in inefficient manners.
So I would like to suggest a feature that enables to log slow query.
The feature can help cassandra operators to identify bad query patterns.
Then operators can give advices about queries and data model to users who don't 
know cassandra so much.

This ticket is related to CASSANDRA-6226, and I focus on detecting bad query 
patterns, not aborting them.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (CASSANDRA-11906) Unstable JVM due too many files when anticompacting big LCS tables

2016-08-07 Thread Stefano Ortolani (JIRA)

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15410918#comment-15410918
 ] 

Stefano Ortolani commented on CASSANDRA-11906:
--

Some more details: 
* I have many LCS cfs on that cluster
* Some (2/3) cfs have 300 SStables
* While repairing those 2/3 cfs I see the number of SStables rising to 1K (per 
cf) until repairs are done (then it drops back to normal values)

> Unstable JVM due too many files when anticompacting big LCS tables
> --
>
> Key: CASSANDRA-11906
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11906
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Stefano Ortolani
>
> I have recently moved from C* 2.1.x to C* 3.0.6. The setup is quite 
> heavy:
>   - 13 nodes with spinning disks
>   - ~120 GB of data per node
>   - 50% of CFs are LCS and have quite wide rows.
>   - 2/3 CFs with LCS have more than 200 SStables
> Incremental repairs do not seem to play really well with that.
> I have been running some tests node by node by using the -pr option:
> {code:xml}
> nodetool -h localhost repair -pr keyscheme
> {code}
> and to my surprise the whole process takes quite some time (1 hour
> minimum, 8 hours if I haven't been repairing for 5/6 days).
> Yesterday I tried to run the command with the -seq option so to 
> decrease the number of simultanoues compactions. After a while
> the node on which I was running the repair simply died during
> the anticompaction phase with the following
> exception in the logs.
> {code:xml}
> ERROR [metrics-graphite-reporter-1-thread-1] 2016-05-25 21:54:21,868 
> ScheduledReporter.java:119 - RuntimeException thrown from 
> GraphiteReporter#report. Exception was suppressed.
> java.lang.RuntimeException: Failed to list files in 
> /data/cassandra/data/keyschema/columnfamily-3996ce80b7ac11e48a9b6776bf484396
>   at 
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:57)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
>   at 
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:547)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
>   at 
> org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:691)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
>   at 
> org.apache.cassandra.db.Directories$SSTableLister.listFiles(Directories.java:662)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
>   at 
> org.apache.cassandra.db.Directories$TrueFilesSizeVisitor.(Directories.java:981)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
>   at 
> org.apache.cassandra.db.Directories.getTrueAllocatedSizeIn(Directories.java:893)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
>   at 
> org.apache.cassandra.db.Directories.trueSnapshotsSize(Directories.java:883) 
> ~[apache-cassandra-3.0.6.jar:3.0.6]
>   at 
> org.apache.cassandra.db.ColumnFamilyStore.trueSnapshotsSize(ColumnFamilyStore.java:2332)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
>   at 
> org.apache.cassandra.metrics.TableMetrics$32.getValue(TableMetrics.java:637) 
> ~[apache-cassandra-3.0.6.jar:3.0.6]
>   at 
> org.apache.cassandra.metrics.TableMetrics$32.getValue(TableMetrics.java:634) 
> ~[apache-cassandra-3.0.6.jar:3.0.6]
>   at 
> com.codahale.metrics.graphite.GraphiteReporter.reportGauge(GraphiteReporter.java:281)
>  ~[metrics-graphite-3.1.0.jar:3.1.0]
>   at 
> com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:158)
>  ~[metrics-graphite-3.1.0.jar:3.1.0]
>   at 
> com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162) 
> ~[metrics-core-3.1.0.jar:3.1.0]
>   at 
> com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:117) 
> ~[metrics-core-3.1.0.jar:3.1.0]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_91]
>   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
> [na:1.8.0_91]
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>  [na:1.8.0_91]
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>  [na:1.8.0_91]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_91]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_91]
>   at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: java.lang.RuntimeException: java.nio.file.FileSystemException: 
> /data/cassandra/data/keyschema/columnfamily-3996ce80b7ac11e48a9b6776bf484396/ma_txn_anticompactionafterrepair_f20b50d0-22bd-11e6-970f-6f22464f4624.log:
>  Too many open files
>   at 

[jira] [Reopened] (CASSANDRA-11906) Unstable JVM due too many files when anticompacting big LCS tables

2016-08-07 Thread Stefano Ortolani (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefano Ortolani reopened CASSANDRA-11906:
--

> Unstable JVM due too many files when anticompacting big LCS tables
> --
>
> Key: CASSANDRA-11906
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11906
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Stefano Ortolani
>
> I have recently moved from C* 2.1.x to C* 3.0.6. The setup is quite 
> heavy:
>   - 13 nodes with spinning disks
>   - ~120 GB of data per node
>   - 50% of CFs are LCS and have quite wide rows.
>   - 2/3 CFs with LCS have more than 200 SStables
> Incremental repairs do not seem to play really well with that.
> I have been running some tests node by node by using the -pr option:
> {code:xml}
> nodetool -h localhost repair -pr keyscheme
> {code}
> and to my surprise the whole process takes quite some time (1 hour
> minimum, 8 hours if I haven't been repairing for 5/6 days).
> Yesterday I tried to run the command with the -seq option so to 
> decrease the number of simultanoues compactions. After a while
> the node on which I was running the repair simply died during
> the anticompaction phase with the following
> exception in the logs.
> {code:xml}
> ERROR [metrics-graphite-reporter-1-thread-1] 2016-05-25 21:54:21,868 
> ScheduledReporter.java:119 - RuntimeException thrown from 
> GraphiteReporter#report. Exception was suppressed.
> java.lang.RuntimeException: Failed to list files in 
> /data/cassandra/data/keyschema/columnfamily-3996ce80b7ac11e48a9b6776bf484396
>   at 
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:57)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
>   at 
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:547)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
>   at 
> org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:691)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
>   at 
> org.apache.cassandra.db.Directories$SSTableLister.listFiles(Directories.java:662)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
>   at 
> org.apache.cassandra.db.Directories$TrueFilesSizeVisitor.(Directories.java:981)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
>   at 
> org.apache.cassandra.db.Directories.getTrueAllocatedSizeIn(Directories.java:893)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
>   at 
> org.apache.cassandra.db.Directories.trueSnapshotsSize(Directories.java:883) 
> ~[apache-cassandra-3.0.6.jar:3.0.6]
>   at 
> org.apache.cassandra.db.ColumnFamilyStore.trueSnapshotsSize(ColumnFamilyStore.java:2332)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
>   at 
> org.apache.cassandra.metrics.TableMetrics$32.getValue(TableMetrics.java:637) 
> ~[apache-cassandra-3.0.6.jar:3.0.6]
>   at 
> org.apache.cassandra.metrics.TableMetrics$32.getValue(TableMetrics.java:634) 
> ~[apache-cassandra-3.0.6.jar:3.0.6]
>   at 
> com.codahale.metrics.graphite.GraphiteReporter.reportGauge(GraphiteReporter.java:281)
>  ~[metrics-graphite-3.1.0.jar:3.1.0]
>   at 
> com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:158)
>  ~[metrics-graphite-3.1.0.jar:3.1.0]
>   at 
> com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162) 
> ~[metrics-core-3.1.0.jar:3.1.0]
>   at 
> com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:117) 
> ~[metrics-core-3.1.0.jar:3.1.0]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_91]
>   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
> [na:1.8.0_91]
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>  [na:1.8.0_91]
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>  [na:1.8.0_91]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_91]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_91]
>   at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: java.lang.RuntimeException: java.nio.file.FileSystemException: 
> /data/cassandra/data/keyschema/columnfamily-3996ce80b7ac11e48a9b6776bf484396/ma_txn_anticompactionafterrepair_f20b50d0-22bd-11e6-970f-6f22464f4624.log:
>  Too many open files
>   at org.apache.cassandra.io.util.FileUtils.readLines(FileUtils.java:622) 
> ~[apache-cassandra-3.0.6.jar:3.0.6]
>   at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321) 
> ~[na:1.8.0_91]
>   at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) 
> ~[na:1.8.0_91]
>   at 
> 

[jira] [Commented] (CASSANDRA-11906) Unstable JVM due too many files when anticompacting big LCS tables

2016-08-07 Thread Stefano Ortolani (JIRA)

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15410907#comment-15410907
 ] 

Stefano Ortolani commented on CASSANDRA-11906:
--

Unfortunately I might need to reopen this issues since one node died last night 
while doing a sequential repair.

{noformat}
INFO  [STREAM-IN-/10.12.81.6] 2016-08-07 00:07:11,145 
StreamResultFuture.java:169 - [Stream #e276de60-5c32-11e6-b182-e548aa39c6eb 
ID#0] Prepare completed. Receiving 2 files(1974371 bytes), sending 2 
files(2101094 bytes)
INFO  [AntiEntropyStage:1] 2016-08-07 00:07:11,895 RepairSession.java:181 - 
[repair #739ae1c1-5c2e-11e6-b182-e548aa39c6eb] Received merkle tree for 
llkgbsubject2sample from /10.12.33.6
INFO  [AntiEntropyStage:1] 2016-08-07 00:07:11,895 RepairJob.java:213 - 
Validating /10.12.81.6
WARN  [STREAM-IN-/10.12.81.6] 2016-08-07 00:07:11,970 CLibrary.java:280 - 
open(/data/cassandra/data/schema/data-cf-685a935014a311e69af3cdff32bdcb0d, 
O_RDONLY) failed, errno (24).
WARN  [STREAM-IN-/10.12.33.6] 2016-08-07 00:07:11,970 CLibrary.java:280 - 
open(/data/cassandra/data/schema/data-cf-685a935014a311e69af3cdff32bdcb0d, 
O_RDONLY) failed, errno (24).
WARN  [STREAM-IN-/10.12.81.6] 2016-08-07 00:07:11,970 
CompressedStreamReader.java:118 - [Stream e276de60-5c32-11e6-b182-e548aa39c6eb] 
Error while reading partition null from stream on ks='schema' and 
table='data-cf'.
ERROR [STREAM-IN-/10.12.81.6] 2016-08-07 00:07:11,984 StreamSession.java:528 - 
[Stream #e276de60-5c32-11e6-b182-e548aa39c6eb] Streaming error occurred
java.nio.file.FileSystemException: 
/data/cassandra/data/schema/data-cf-685a935014a311e69af3cdff32bdcb0d/mb-4984-big-Index.db:
 Too many open files
at 
sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) 
~[na:1.8.0_91]
at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) 
~[na:1.8.0_91]
at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) 
~[na:1.8.0_91]
at 
sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
 ~[na:1.8.0_91]
at java.nio.channels.FileChannel.open(FileChannel.java:287) 
~[na:1.8.0_91]
at java.nio.channels.FileChannel.open(FileChannel.java:335) 
~[na:1.8.0_91]
at 
org.apache.cassandra.io.util.SequentialWriter.openChannel(SequentialWriter.java:114)
 ~[apache-cassandra-3.0.8.jar:3.0.8]
at 
org.apache.cassandra.io.util.SequentialWriter.(SequentialWriter.java:135) 
~[apache-cassandra-3.0.8.jar:3.0.8]
at 
org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:150) 
~[apache-cassandra-3.0.8.jar:3.0.8]
at 
org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.(BigTableWriter.java:375)
 ~[apache-cassandra-3.0.8.jar:3.0.8]
at 
org.apache.cassandra.io.sstable.format.big.BigTableWriter.(BigTableWriter.java:84)
 ~[apache-cassandra-3.0.8.jar:3.0.8]
at 
org.apache.cassandra.io.sstable.format.big.BigFormat$WriterFactory.open(BigFormat.java:93)
 ~[apache-cassandra-3.0.8.jar:3.0.8]
at 
org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTableWriter.java:96)
 ~[apache-cassandra-3.0.8.jar:3.0.8]
at 
org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.create(SimpleSSTableMultiWriter.java:114)
 ~[apache-cassandra-3.0.8.jar:3.0.8]
at 
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.createSSTableMultiWriter(AbstractCompactionStrategy.java:516)
 ~[apache-cassandra-3.0.8.jar:3.0.8]
at 
org.apache.cassandra.db.compaction.CompactionStrategyManager.createSSTableMultiWriter(CompactionStrategyManager.java:498)
 ~[apache-cassandra-3.0.8.jar:3.0.8]
at 
org.apache.cassandra.db.ColumnFamilyStore.createSSTableMultiWriter(ColumnFamilyStore.java:481)
 ~[apache-cassandra-3.0.8.jar:3.0.8]
at 
org.apache.cassandra.db.ColumnFamilyStore.createSSTableMultiWriter(ColumnFamilyStore.java:476)
 ~[apache-cassandra-3.0.8.jar:3.0.8]
at 
org.apache.cassandra.streaming.StreamReader.createWriter(StreamReader.java:162) 
~[apache-cassandra-3.0.8.jar:3.0.8]
at 
org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:92)
 ~[apache-cassandra-3.0.8.jar:3.0.8]
at 
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
 ~[apache-cassandra-3.0.8.jar:3.0.8]
at 
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
 ~[apache-cassandra-3.0.8.jar:3.0.8]
at 
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
 ~[apache-cassandra-3.0.8.jar:3.0.8]
at 
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:287)
 ~[apache-cassandra-3.0.8.jar:3.0.8]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]