[jira] [Issue Comment Deleted] (CASSANDRA-11713) Add ability to log thread dump when NTR pool is blocked

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-11713:
---
Comment: was deleted

(was: I know this sounds weird, but we've seen clusters that actually LOOK like 
they're behaving DESPITE the high NTR blocks, and I suspect a lot of 
unsuspecting users who would have been happy with that type of situation would 
be hurt severely by this patch because the log output in those clusters would 
be significant - for example, we've had 2-3 hour long stress runs where 
individual nodes have 7 digit ntr-blocked counters, can you imagine dumping 
threads to logs a million times an hour?.

At the very least, I'd personally like to see it guarded with a system property 
or log spam protection. As an operator, I'd strongly suggest we not commit this 
patch until one of those were available.)

> Add ability to log thread dump when NTR pool is blocked
> ---
>
> Key: CASSANDRA-11713
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11713
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability
>Reporter: Paulo Motta
>Assignee: Paulo Motta
>Priority: Minor
>
> Thread dumps are very useful for troubleshooting Native-Transport-Requests 
> contention issues like CASSANDRA-11363 and CASSANDRA-11529.
> While they could be generated externally with {{jstack}}, sometimes the 
> conditions are transient and it's hard to catch the exact moment when they 
> happen, so it could be useful to generate and log them upon user request when 
> certain internal condition happens.
> I propose adding a {{logThreadDumpOnNextContention}} flag to {{SEPExecutor}} 
> that when enabled via JMX generates and logs a single thread dump on the 
> system log when the thread pool queue is full.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11713) Add ability to log thread dump when NTR pool is blocked

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271910#comment-15271910
 ] 

Jeff Jirsa commented on CASSANDRA-11713:


I know this sounds weird, but we've seen clusters that actually LOOK like 
they're behaving DESPITE the high NTR blocks, and I suspect a lot of 
unsuspecting users who would have been happy with that type of situation would 
be hurt severely by this patch because the log output in those clusters would 
be significant - for example, we've had 2-3 hour long stress runs where 
individual nodes have 7 digit ntr-blocked counters, can you imagine dumping 
threads to logs a million times an hour?.

At the very least, I'd personally like to see it guarded with a system property 
or log spam protection. As an operator, I'd strongly suggest we not commit this 
patch until one of those were available.

> Add ability to log thread dump when NTR pool is blocked
> ---
>
> Key: CASSANDRA-11713
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11713
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability
>Reporter: Paulo Motta
>Assignee: Paulo Motta
>Priority: Minor
>
> Thread dumps are very useful for troubleshooting Native-Transport-Requests 
> contention issues like CASSANDRA-11363 and CASSANDRA-11529.
> While they could be generated externally with {{jstack}}, sometimes the 
> conditions are transient and it's hard to catch the exact moment when they 
> happen, so it could be useful to generate and log them upon user request when 
> certain internal condition happens.
> I propose adding a {{logThreadDumpOnNextContention}} flag to {{SEPExecutor}} 
> that when enabled via JMX generates and logs a single thread dump on the 
> system log when the thread pool queue is full.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11715) Make GCInspector's MIN_LOG_DURATION configurable

[Brandon Williams (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Williams updated CASSANDRA-11715:
-
Description: It's common for people to run C* with the G1 collector on 
appropriately-sized heaps.  Quite often, the target pause time is set to 500ms, 
but GCI fires on anything over 200ms.  We can already control the warn 
threshold, but these are acceptable GCs for the configuration and create noise 
at the INFO log level.  (was: It's common for people to run C* with the G1 
collector on appropriately-sized heaps.  Quite often, the target pause time is 
set to 500ms, but GCI fires on anything over 200ms.  We can already control the 
warn threshold, but these are acceptable GCs for the configuring and create 
noise at the INFO log level.)

> Make GCInspector's MIN_LOG_DURATION configurable
> 
>
> Key: CASSANDRA-11715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11715
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Brandon Williams
>  Labels: lhf
>
> It's common for people to run C* with the G1 collector on appropriately-sized 
> heaps.  Quite often, the target pause time is set to 500ms, but GCI fires on 
> anything over 200ms.  We can already control the warn threshold, but these 
> are acceptable GCs for the configuration and create noise at the INFO log 
> level.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11715) Make GCInspector's MIN_LOG_DURATION configurable

[Brandon Williams (JIRA)]

Brandon Williams created CASSANDRA-11715:


 Summary: Make GCInspector's MIN_LOG_DURATION configurable
 Key: CASSANDRA-11715
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11715
 Project: Cassandra
  Issue Type: Improvement
Reporter: Brandon Williams


It's common for people to run C* with the G1 collector on appropriately-sized 
heaps.  Quite often, the target pause time is set to 500ms, but GCI fires on 
anything over 200ms.  We can already control the warn threshold, but these are 
acceptable GCs for the configuring and create noise at the INFO log level.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11715) Make GCInspector's MIN_LOG_DURATION configurable

[Brandon Williams (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Williams updated CASSANDRA-11715:
-
Labels: lhf  (was: )

> Make GCInspector's MIN_LOG_DURATION configurable
> 
>
> Key: CASSANDRA-11715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11715
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Brandon Williams
>  Labels: lhf
>
> It's common for people to run C* with the G1 collector on appropriately-sized 
> heaps.  Quite often, the target pause time is set to 500ms, but GCI fires on 
> anything over 200ms.  We can already control the warn threshold, but these 
> are acceptable GCs for the configuring and create noise at the INFO log level.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11714) shutdown script should wait more than 30 seconds before force kill

[Wei Deng (JIRA)]

Wei Deng created CASSANDRA-11714:


 Summary: shutdown script should wait more than 30 seconds before 
force kill
 Key: CASSANDRA-11714
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11714
 Project: Cassandra
  Issue Type: Improvement
  Components: Tools
Reporter: Wei Deng
Priority: Minor


I'm running with 3.0.6 and it appears that if I let it run for multiple days 
(ingesting a never-ending streaming data from Kafka), then the next time I'm 
restarting the JVM by "service cassandra restart", it will take 9 minutes to 
replay the commit log segments (the commitlog_total_space_in_mb is using the 
default 8GB and I observed close to 8GB commitlog segment files before the 
restart). Here is a fragment of the system.log showing how long it takes:

{noformat}
INFO  [main] 2016-05-04 19:06:54,356  CommitLog.java:168 - Replaying 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248187.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248191.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248195.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248198.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248201.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248205.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248209.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248213.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248217.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248221.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248225.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248230.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248234.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248238.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248242.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248246.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248250.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248254.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248258.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248262.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248266.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248270.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248274.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248278.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248282.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248286.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248290.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248294.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248299.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248303.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248307.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248311.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248315.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248319.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248323.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248327.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248331.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248335.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248340.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248344.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248348.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248352.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248356.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248360.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248364.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248368.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248372.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248376.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248380.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248384.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248388.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248392.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248396.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248397.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248401.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248405.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248409.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248413.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248417.log, 
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248421.log, 

[jira] [Commented] (CASSANDRA-11279) dtest failure in disk_balance_test.TestDiskBalance.disk_balance_bootstrap_test

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271726#comment-15271726
 ] 

Paulo Motta commented on CASSANDRA-11279:
-

Should we maybe keep the asserts while unsetting 
{{allocate_tokens_for_keyspace}} to test that disks are balanced for the random 
allocation bootstrap case?

We can extend these after CASSANDRA-11643 to include the balanced bootstrap 
case.

> dtest failure in disk_balance_test.TestDiskBalance.disk_balance_bootstrap_test
> --
>
> Key: CASSANDRA-11279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11279
> Project: Cassandra
>  Issue Type: Test
>Reporter: Russ Hatch
>Assignee: Marcus Eriksson
>  Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest/1011/testReport/disk_balance_test/TestDiskBalance/disk_balance_bootstrap_test
> Failed on CassCI build trunk_dtest #1011
> This looks likely to be a test issue, perhaps we need to relax the assertion 
> here a bit:
> {noformat}
> values not within 20.00% of the max: (474650, 382235, 513385) (node1)
> {noformat}
> This is flaky with several flaps in the last few weeks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11626) cqlsh fails and exits on non-ascii chars

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271612#comment-15271612
 ] 

Paulo Motta commented on CASSANDRA-11626:
-

LGTM (tested with latin chars), let's hope this is our last juggling with 
encoding on cqlsh.

> cqlsh fails and exits on non-ascii chars
> 
>
> Key: CASSANDRA-11626
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11626
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Robert Stupp
>Assignee: Tyler Hobbs
>Priority: Minor
>  Labels: cqlsh
> Fix For: 2.2.x, 3.0.x, 3.x
>
>
> Just seen on cqlsh on current trunk:
> To repro, copy {{ä}} (german umlaut) to cqlsh and press return.
> cqlsh errors out and immediately exits.
> {code}
> $ bin/cqlsh
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.13-SNAPSHOT | CQL spec 3.2.1 | Native protocol 
> v3]
> Use HELP for help.
> cqlsh> ä
> Invalid syntax at line 1, char 1
> Traceback (most recent call last):
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2636, in 
> 
> main(*read_options(sys.argv[1:], os.environ))
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2625, in main
> shell.cmdloop()
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 1114, in 
> cmdloop
> if self.onecmd(self.statement.getvalue()):
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 1139, in onecmd
> self.printerr('  %s' % statementline)
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2314, in 
> printerr
> self.writeresult(text, color, newline=newline, out=sys.stderr)
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2303, in 
> writeresult
> out.write(self.applycolor(str(text), color) + ('\n' if newline else ''))
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 
> 2: ordinal not in range(128)
> $ 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11626) cqlsh fails and exits on non-ascii chars

[Paulo Motta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-11626:

Status: Ready to Commit  (was: Patch Available)

> cqlsh fails and exits on non-ascii chars
> 
>
> Key: CASSANDRA-11626
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11626
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Robert Stupp
>Assignee: Tyler Hobbs
>Priority: Minor
>  Labels: cqlsh
> Fix For: 2.2.x, 3.0.x, 3.x
>
>
> Just seen on cqlsh on current trunk:
> To repro, copy {{ä}} (german umlaut) to cqlsh and press return.
> cqlsh errors out and immediately exits.
> {code}
> $ bin/cqlsh
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.13-SNAPSHOT | CQL spec 3.2.1 | Native protocol 
> v3]
> Use HELP for help.
> cqlsh> ä
> Invalid syntax at line 1, char 1
> Traceback (most recent call last):
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2636, in 
> 
> main(*read_options(sys.argv[1:], os.environ))
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2625, in main
> shell.cmdloop()
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 1114, in 
> cmdloop
> if self.onecmd(self.statement.getvalue()):
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 1139, in onecmd
> self.printerr('  %s' % statementline)
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2314, in 
> printerr
> self.writeresult(text, color, newline=newline, out=sys.stderr)
>   File "/Users/snazy/devel/cassandra/trunk/bin/cqlsh.py", line 2303, in 
> writeresult
> out.write(self.applycolor(str(text), color) + ('\n' if newline else ''))
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 
> 2: ordinal not in range(128)
> $ 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11713) Add ability to log thread dump when NTR pool is blocked

[Paulo Motta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-11713:

Description: 
Thread dumps are very useful for troubleshooting Native-Transport-Requests 
contention issues like CASSANDRA-11363 and CASSANDRA-11529.

While they could be generated externally with {{jstack}}, sometimes the 
conditions are transient and it's hard to catch the exact moment when they 
happen, so it could be useful to generate and log them upon user request when 
certain internal condition happens.

I propose adding a {{logThreadDumpOnNextContention}} flag to {{SEPExecutor}} 
that when enabled via JMX generates and logs a single thread dump on the system 
log when the thread pool queue is full.

  was:
Thread dumps are very useful for troubleshooting thread pool contention issues 
like CASSANDRA-11363 and CASSANDRA-11529.

While they could be generated externally with {{jstack}}, sometimes the 
conditions are transient and it's hard to catch the exact moment when they 
happen, so it could be useful to generate and log them upon user request when 
certain internal condition happens.

I propose adding a {{logThreadDumpOnNextContention}} flag to {{SEPExecutor}} 
and {{JMXEnabledThreadPoolExecutor}} that when enabled via JMX generates and 
logs a single thread dump on the system log when the thread pool queue is full.

Summary: Add ability to log thread dump when NTR pool is blocked  (was: 
Add ability to log thread dump when thread pool is full/blocked)

> Add ability to log thread dump when NTR pool is blocked
> ---
>
> Key: CASSANDRA-11713
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11713
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability
>Reporter: Paulo Motta
>Assignee: Paulo Motta
>Priority: Minor
>
> Thread dumps are very useful for troubleshooting Native-Transport-Requests 
> contention issues like CASSANDRA-11363 and CASSANDRA-11529.
> While they could be generated externally with {{jstack}}, sometimes the 
> conditions are transient and it's hard to catch the exact moment when they 
> happen, so it could be useful to generate and log them upon user request when 
> certain internal condition happens.
> I propose adding a {{logThreadDumpOnNextContention}} flag to {{SEPExecutor}} 
> that when enabled via JMX generates and logs a single thread dump on the 
> system log when the thread pool queue is full.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11713) Add ability to log thread dump when thread pool is full/blocked

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271568#comment-15271568
 ] 

Paulo Motta commented on CASSANDRA-11713:
-

That's right, thanks for pointing this out.

I initially added that only to {{SEPExecutor}} with the objective of 
troubleshooting {{Native-Transport-Requests}}, but then thought it could be 
useful for other thread pools too without noticing their "unboundedness". :-)

Updated patch to only add this ability only to {{SEPExecutor}}.

> Add ability to log thread dump when thread pool is full/blocked
> ---
>
> Key: CASSANDRA-11713
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11713
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability
>Reporter: Paulo Motta
>Assignee: Paulo Motta
>Priority: Minor
>
> Thread dumps are very useful for troubleshooting thread pool contention 
> issues like CASSANDRA-11363 and CASSANDRA-11529.
> While they could be generated externally with {{jstack}}, sometimes the 
> conditions are transient and it's hard to catch the exact moment when they 
> happen, so it could be useful to generate and log them upon user request when 
> certain internal condition happens.
> I propose adding a {{logThreadDumpOnNextContention}} flag to {{SEPExecutor}} 
> and {{JMXEnabledThreadPoolExecutor}} that when enabled via JMX generates and 
> logs a single thread dump on the system log when the thread pool queue is 
> full.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11713) Add ability to log thread dump when thread pool is full/blocked

[Chris Lohfink (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271545#comment-15271545
 ] 

Chris Lohfink commented on CASSANDRA-11713:
---

Probably worth mentioning that since 2.1 or so the only thread pool where its 
actually really possible to be in a blocked state is the 
{{Native-Transport-Requests}} one. The queues are basically unbounded (MAX_INT) 
so they will all OOM before the rejection policy gets kicked

> Add ability to log thread dump when thread pool is full/blocked
> ---
>
> Key: CASSANDRA-11713
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11713
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability
>Reporter: Paulo Motta
>Assignee: Paulo Motta
>Priority: Minor
>
> Thread dumps are very useful for troubleshooting thread pool contention 
> issues like CASSANDRA-11363 and CASSANDRA-11529.
> While they could be generated externally with {{jstack}}, sometimes the 
> conditions are transient and it's hard to catch the exact moment when they 
> happen, so it could be useful to generate and log them upon user request when 
> certain internal condition happens.
> I propose adding a {{logThreadDumpOnNextContention}} flag to {{SEPExecutor}} 
> and {{JMXEnabledThreadPoolExecutor}} that when enabled via JMX generates and 
> logs a single thread dump on the system log when the thread pool queue is 
> full.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11363) Blocked NTR When Connecting Causing Excessive Load

[Paulo Motta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-11363:

Reproduced In: 3.0.3, 2.1.13, 2.1.12  (was: 2.1.12, 2.1.13, 3.0.3)
 Priority: Major  (was: Critical)

> Blocked NTR When Connecting Causing Excessive Load
> --
>
> Key: CASSANDRA-11363
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11363
> Project: Cassandra
>  Issue Type: Bug
>  Components: Coordination
>Reporter: Russell Bradberry
>Assignee: Paulo Motta
> Attachments: cassandra-102-cms.stack, cassandra-102-g1gc.stack
>
>
> When upgrading from 2.1.9 to 2.1.13, we are witnessing an issue where the 
> machine load increases to very high levels (> 120 on an 8 core machine) and 
> native transport requests get blocked in tpstats.
> I was able to reproduce this in both CMS and G1GC as well as on JVM 7 and 8.
> The issue does not seem to affect the nodes running 2.1.9.
> The issue seems to coincide with the number of connections OR the number of 
> total requests being processed at a given time (as the latter increases with 
> the former in our system)
> Currently there is between 600 and 800 client connections on each machine and 
> each machine is handling roughly 2000-3000 client requests per second.
> Disabling the binary protocol fixes the issue for this node but isn't a 
> viable option cluster-wide.
> Here is the output from tpstats:
> {code}
> Pool NameActive   Pending  Completed   Blocked  All 
> time blocked
> MutationStage 0 88387821 0
>  0
> ReadStage 0 0 355860 0
>  0
> RequestResponseStage  0 72532457 0
>  0
> ReadRepairStage   0 0150 0
>  0
> CounterMutationStage 32   104 897560 0
>  0
> MiscStage 0 0  0 0
>  0
> HintedHandoff 0 0 65 0
>  0
> GossipStage   0 0   2338 0
>  0
> CacheCleanupExecutor  0 0  0 0
>  0
> InternalResponseStage 0 0  0 0
>  0
> CommitLogArchiver 0 0  0 0
>  0
> CompactionExecutor2   190474 0
>  0
> ValidationExecutor0 0  0 0
>  0
> MigrationStage0 0 10 0
>  0
> AntiEntropyStage  0 0  0 0
>  0
> PendingRangeCalculator0 0310 0
>  0
> Sampler   0 0  0 0
>  0
> MemtableFlushWriter   110 94 0
>  0
> MemtablePostFlush 134257 0
>  0
> MemtableReclaimMemory 0 0 94 0
>  0
> Native-Transport-Requests   128   156 38795716
> 278451
> Message type   Dropped
> READ 0
> RANGE_SLICE  0
> _TRACE   0
> MUTATION 0
> COUNTER_MUTATION 0
> BINARY   0
> REQUEST_RESPONSE 0
> PAGED_RANGE  0
> READ_REPAIR  0
> {code}
> Attached is the jstack output for both CMS and G1GC.
> Flight recordings are here:
> https://s3.amazonaws.com/simple-logs/cassandra-102-cms.jfr
> https://s3.amazonaws.com/simple-logs/cassandra-102-g1gc.jfr
> It is interesting to note that while the flight recording was taking place, 
> the load on the machine went back to healthy, and when the flight recording 
> finished the load went back to > 100.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11363) Blocked NTR When Connecting Causing Excessive Load

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271519#comment-15271519
 ] 

Paulo Motta commented on CASSANDRA-11363:
-

Tried reproducing this in a variety of workloads without success in the 
following environment:
* 3 nodes m3.xlarge (m3.xlarge, 4 vcpu, 15GB RAM, 2 x 40)
* 8GB Heap, CMS
* 100M keys (24GB per node)
* C* 3.0.3 with default settings and a variation with 
native_transport_max_threads=10
* no-vnodes

The workloads were based a variation of 
https://raw.githubusercontent.com/mesosphere/cassandra-mesos/master/driver-extensions/cluster-loadtest/cqlstress-example.yaml
 with 100M keys and RF=3 executed from 2 m3.xlarge stress nodes for 1, 2 and 6 
hours with 10, 20 and 30 threads without exhausting the cluster CPU/IO 
capacity. I tried several combinations of the following workloads:
* read-only
* write-only
* range-only
* triggering repairs during the execution
* unthrottling compaction

I recorded and analyzed flight recordings during the tests but didn't find 
anything suspicious. No blocked native transport threads were verified during 
tests with above scenarios, so this might indicate that this condition is not a 
widespread bug like CASSANDRA-11529 but probably some edgy combination of 
workload, environment and bad scheduling that happens in production but is 
harder to reproduce with synthetic workloads.
 
A thread dump of when this condition happens would probably help us detect 
where is the bottleneck or contention, so I created CASSANDRA-11713 to add 
ability of logging a thread dump when the thread pool queue is full. If someone 
could install that patch and enable it in production to capture a thread dump 
when the blockage happens that would probably help us elucidate what's going on 
here.

> Blocked NTR When Connecting Causing Excessive Load
> --
>
> Key: CASSANDRA-11363
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11363
> Project: Cassandra
>  Issue Type: Bug
>  Components: Coordination
>Reporter: Russell Bradberry
>Assignee: Paulo Motta
>Priority: Critical
> Attachments: cassandra-102-cms.stack, cassandra-102-g1gc.stack
>
>
> When upgrading from 2.1.9 to 2.1.13, we are witnessing an issue where the 
> machine load increases to very high levels (> 120 on an 8 core machine) and 
> native transport requests get blocked in tpstats.
> I was able to reproduce this in both CMS and G1GC as well as on JVM 7 and 8.
> The issue does not seem to affect the nodes running 2.1.9.
> The issue seems to coincide with the number of connections OR the number of 
> total requests being processed at a given time (as the latter increases with 
> the former in our system)
> Currently there is between 600 and 800 client connections on each machine and 
> each machine is handling roughly 2000-3000 client requests per second.
> Disabling the binary protocol fixes the issue for this node but isn't a 
> viable option cluster-wide.
> Here is the output from tpstats:
> {code}
> Pool NameActive   Pending  Completed   Blocked  All 
> time blocked
> MutationStage 0 88387821 0
>  0
> ReadStage 0 0 355860 0
>  0
> RequestResponseStage  0 72532457 0
>  0
> ReadRepairStage   0 0150 0
>  0
> CounterMutationStage 32   104 897560 0
>  0
> MiscStage 0 0  0 0
>  0
> HintedHandoff 0 0 65 0
>  0
> GossipStage   0 0   2338 0
>  0
> CacheCleanupExecutor  0 0  0 0
>  0
> InternalResponseStage 0 0  0 0
>  0
> CommitLogArchiver 0 0  0 0
>  0
> CompactionExecutor2   190474 0
>  0
> ValidationExecutor0 0  0 0
>  0
> MigrationStage0 0 10 0
>  0
> AntiEntropyStage  0 0  0 0
>  0
> PendingRangeCalculator0 0310 0
>  0
> Sampler   0 0  0 0
>  0
> MemtableFlushWriter   110 94 0
>

[jira] [Updated] (CASSANDRA-11713) Add ability to log thread dump when thread pool is full/blocked

[Paulo Motta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-11713:

Status: Patch Available  (was: Open)

> Add ability to log thread dump when thread pool is full/blocked
> ---
>
> Key: CASSANDRA-11713
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11713
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability
>Reporter: Paulo Motta
>Assignee: Paulo Motta
>Priority: Minor
>
> Thread dumps are very useful for troubleshooting thread pool contention 
> issues like CASSANDRA-11363 and CASSANDRA-11529.
> While they could be generated externally with {{jstack}}, sometimes the 
> conditions are transient and it's hard to catch the exact moment when they 
> happen, so it could be useful to generate and log them upon user request when 
> certain internal condition happens.
> I propose adding a {{logThreadDumpOnNextContention}} flag to {{SEPExecutor}} 
> and {{JMXEnabledThreadPoolExecutor}} that when enabled via JMX generates and 
> logs a single thread dump on the system log when the thread pool queue is 
> full.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11713) Add ability to log thread dump when thread pool is full/blocked

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271514#comment-15271514
 ] 

Paulo Motta commented on CASSANDRA-11713:
-

Initial patch available 
[here|https://github.com/pauloricardomg/cassandra/tree/thread-dumper] for 
review.

> Add ability to log thread dump when thread pool is full/blocked
> ---
>
> Key: CASSANDRA-11713
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11713
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability
>Reporter: Paulo Motta
>Assignee: Paulo Motta
>Priority: Minor
>
> Thread dumps are very useful for troubleshooting thread pool contention 
> issues like CASSANDRA-11363 and CASSANDRA-11529.
> While they could be generated externally with {{jstack}}, sometimes the 
> conditions are transient and it's hard to catch the exact moment when they 
> happen, so it could be useful to generate and log them upon user request when 
> certain internal condition happens.
> I propose adding a {{logThreadDumpOnNextContention}} flag to {{SEPExecutor}} 
> and {{JMXEnabledThreadPoolExecutor}} that when enabled via JMX generates and 
> logs a single thread dump on the system log when the thread pool queue is 
> full.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11713) Add ability to log thread dump when thread pool is full/blocked

[Paulo Motta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-11713:

Description: 
Thread dumps are very useful for troubleshooting thread pool contention issues 
like CASSANDRA-11363 and CASSANDRA-11529.

While they could be generated externally with {{jstack}}, sometimes the 
conditions are transient and it's hard to catch the exact moment when they 
happen, so it could be useful to generate and log them upon user request when 
certain internal condition happens.

I propose adding a {{logThreadDumpOnNextContention}} flag to {{SEPExecutor}} 
and {{JMXEnabledThreadPoolExecutor}} that when enabled via JMX generates and 
logs a single thread dump on the system log when the thread pool queue is full.

  was:
Thread dumps are very useful for troubleshooting thread pool contention issues 
like CASSANDRA-11363 and CASSANDRA-11529.

While they could be generated externally with {{jstack}}, sometimes the 
conditions are transient and it's hard to catch the exact moment when they 
happen, so it could be useful to generate and log them upon user request when 
certain internal condition happens.

I propose adding a {{logThreadDumpOnNextContention}} JMX flag to 
{{SEPExecutor}} and {{JMXEnabledThreadPoolExecutor}} that when enabled 
generates and logs a single thread dump on the system log when the thread pool 
queue is full.


> Add ability to log thread dump when thread pool is full/blocked
> ---
>
> Key: CASSANDRA-11713
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11713
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability
>Reporter: Paulo Motta
>Assignee: Paulo Motta
>Priority: Minor
>
> Thread dumps are very useful for troubleshooting thread pool contention 
> issues like CASSANDRA-11363 and CASSANDRA-11529.
> While they could be generated externally with {{jstack}}, sometimes the 
> conditions are transient and it's hard to catch the exact moment when they 
> happen, so it could be useful to generate and log them upon user request when 
> certain internal condition happens.
> I propose adding a {{logThreadDumpOnNextContention}} flag to {{SEPExecutor}} 
> and {{JMXEnabledThreadPoolExecutor}} that when enabled via JMX generates and 
> logs a single thread dump on the system log when the thread pool queue is 
> full.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11713) Add ability to log thread dump when thread pool is full/blocked

[Paulo Motta (JIRA)]

Paulo Motta created CASSANDRA-11713:
---

 Summary: Add ability to log thread dump when thread pool is 
full/blocked
 Key: CASSANDRA-11713
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11713
 Project: Cassandra
  Issue Type: Improvement
  Components: Observability
Reporter: Paulo Motta
Assignee: Paulo Motta
Priority: Minor


Thread dumps are very useful for troubleshooting thread pool contention issues 
like CASSANDRA-11363 and CASSANDRA-11529.

While they could be generated externally with {{jstack}}, sometimes the 
conditions are transient and it's hard to catch the exact moment when they 
happen, so it could be useful to generate and log them upon user request when 
certain internal condition happens.

I propose adding a {{logThreadDumpOnNextContention}} JMX flag to 
{{SEPExecutor}} and {{JMXEnabledThreadPoolExecutor}} that when enabled 
generates and logs a single thread dump on the system log when the thread pool 
queue is full.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-9766) Faster Streaming

[T Jake Luciani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

T Jake Luciani updated CASSANDRA-9766:
--
   Resolution: Fixed
Fix Version/s: (was: 3.x)
   3.8
   Status: Resolved  (was: Patch Available)

committed {{1e92ce43a5a730f81d3f6cfd72e7f4b126db788a}}

> Faster Streaming
> 
>
> Key: CASSANDRA-9766
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9766
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Streaming and Messaging
> Environment: Cassandra 2.1.2. more details in the pdf attached 
>Reporter: Alexei K
>Assignee: T Jake Luciani
>  Labels: performance
> Fix For: 3.8
>
> Attachments: problem.pdf
>
>
> I have a cluster in Amazon cloud , its described in detail in the attachment. 
> What I've noticed is that we during bootstrap we never go above 12MB/sec 
> transmission speeds and also those speeds flat line almost like we're hitting 
> some sort of a limit ( this remains true for other tests that I've ran) 
> however during the repair we see much higher,variable sending rates. I've 
> provided network charts in the attachment as well . Is there an explanation 
> for this? Is something wrong with my configuration, or is it a possible bug?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

cassandra git commit: Faster Streaming

[jake]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 594183b03 -> 1e92ce43a


Faster Streaming

Patch by tjake; reviewed by Stefania Alborghetti for CASSANDRA-9766


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1e92ce43
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1e92ce43
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1e92ce43

Branch: refs/heads/trunk
Commit: 1e92ce43a5a730f81d3f6cfd72e7f4b126db788a
Parents: 594183b
Author: T Jake Luciani 
Authored: Sat Apr 9 10:03:32 2016 -0400
Committer: T Jake Luciani 
Committed: Wed May 4 17:04:12 2016 -0400

--
 CHANGES.txt |   1 +
 .../concurrent/NamedThreadFactory.java  |   4 +-
 .../apache/cassandra/concurrent/SEPWorker.java  |   3 +-
 .../org/apache/cassandra/db/ColumnIndex.java|  69 +---
 .../columniterator/SSTableReversedIterator.java |   2 +-
 .../db/commitlog/FileDirectSegment.java |   3 +-
 .../org/apache/cassandra/db/rows/BTreeRow.java  |  30 ++--
 .../cassandra/db/rows/ComplexColumnData.java|  10 +-
 .../cassandra/db/rows/UnfilteredSerializer.java |  40 -
 .../io/sstable/format/big/BigTableWriter.java   |  50 +++---
 .../cassandra/io/util/DataOutputBuffer.java |  38 -
 .../io/util/DataOutputBufferFixed.java  |   4 +-
 .../cassandra/io/util/DataOutputStreamPlus.java |   3 +-
 .../cassandra/io/util/SafeMemoryWriter.java |   2 +-
 .../cassandra/streaming/ConnectionHandler.java  |   4 +-
 .../compress/CompressedInputStream.java |  59 +--
 .../compress/CompressedStreamReader.java|  11 +-
 .../cassandra/tools/SSTableMetadataViewer.java  |  10 +-
 .../org/apache/cassandra/utils/BloomFilter.java |   7 +-
 .../apache/cassandra/utils/FilterFactory.java   |   6 +-
 .../cassandra/utils/StreamingHistogram.java |  84 +
 .../org/apache/cassandra/utils/btree/BTree.java |  67 
 .../apache/cassandra/utils/btree/BTreeSet.java  |   3 -
 .../cassandra/utils/btree/TreeBuilder.java  |  30 +++-
 .../concurrent/WrappedSharedCloseable.java  |   4 +-
 .../cassandra/utils/memory/BufferPool.java  |  16 +-
 .../apache/cassandra/utils/vint/VIntCoding.java |   3 +-
 .../cassandra/streaming/LongStreamingTest.java  | 171 +++
 .../apache/cassandra/db/RowIndexEntryTest.java  |   9 +-
 .../org/apache/cassandra/utils/BTreeTest.java   |  20 ++-
 .../cassandra/utils/StreamingHistogramTest.java |  48 +-
 31 files changed, 597 insertions(+), 214 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/1e92ce43/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 6fe57b2..c2165db 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.8
+ * Faster streaming (CASSANDRA-9766)
 
 3.6
  * Enhanced Compaction Logging (CASSANDRA-10805)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/1e92ce43/src/java/org/apache/cassandra/concurrent/NamedThreadFactory.java
--
diff --git a/src/java/org/apache/cassandra/concurrent/NamedThreadFactory.java 
b/src/java/org/apache/cassandra/concurrent/NamedThreadFactory.java
index 33c80d5..85edf74 100644
--- a/src/java/org/apache/cassandra/concurrent/NamedThreadFactory.java
+++ b/src/java/org/apache/cassandra/concurrent/NamedThreadFactory.java
@@ -20,6 +20,8 @@ package org.apache.cassandra.concurrent;
 import java.util.concurrent.ThreadFactory;
 import java.util.concurrent.atomic.AtomicInteger;
 
+import io.netty.util.concurrent.FastThreadLocalThread;
+
 /**
  * This class is an implementation of the ThreadFactory interface. This
  * is useful to give Java threads meaningful names which is useful when using
@@ -55,7 +57,7 @@ public class NamedThreadFactory implements ThreadFactory
 public Thread newThread(Runnable runnable)
 {
 String name = id + ":" + n.getAndIncrement();
-Thread thread = new Thread(threadGroup, runnable, name);
+Thread thread = new FastThreadLocalThread(threadGroup, runnable, name);
 thread.setPriority(priority);
 thread.setDaemon(true);
 if (contextClassLoader != null)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/1e92ce43/src/java/org/apache/cassandra/concurrent/SEPWorker.java
--
diff --git a/src/java/org/apache/cassandra/concurrent/SEPWorker.java 
b/src/java/org/apache/cassandra/concurrent/SEPWorker.java
index 3b3e7ad..d7c21bc 100644
--- a/src/java/org/apache/cassandra/concurrent/SEPWorker.java
+++ b/src/java/org/apache/cassandra/concurrent/SEPWorker.java
@@ -24,6 +24,7 @@ import java.util.concurrent.locks.LockSupport;

[jira] [Commented] (CASSANDRA-11483) Enhance sstablemetadata

[Yuki Morishita (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271404#comment-15271404
 ] 

Yuki Morishita commented on CASSANDRA-11483:


Alright, let's just fix dtest.
Looks like adding backward compatibility isn't worth the effort at all.

> Enhance sstablemetadata
> ---
>
> Key: CASSANDRA-11483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11483
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability
>Reporter: Chris Lohfink
>Assignee: Chris Lohfink
>Priority: Minor
> Fix For: 3.x
>
> Attachments: CASSANDRA-11483.txt, CASSANDRA-11483v2.txt, 
> CASSANDRA-11483v3.txt, CASSANDRA-11483v4.txt, Screen Shot 2016-04-03 at 
> 11.40.32 PM.png
>
>
> sstablemetadata provides quite a bit of useful information but theres a few 
> hiccups I would like to see addressed:
> * Does not use client mode
> * Units are not provided (or anything for that matter). There is data in 
> micros, millis, seconds as durations and timestamps from epoch. But there is 
> no way to tell what one is without a non-trival code dive
> * in general pretty frustrating to parse



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11503) Need a tool to detect what percentage of SSTables on a node have been repaired when using incremental repairs.

[Yuki Morishita (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271401#comment-15271401
 ] 

Yuki Morishita commented on CASSANDRA-11503:


bq. that sound good?

Yes. One thing to consider is, how do users access global repaired% other than 
direct JMX access?
Per keyspace or table we can use nodetool tablestat, maybe we can use 
{{nodetool info}} for global.

> Need a tool to detect what percentage of SSTables on a node have been 
> repaired when using incremental repairs.
> --
>
> Key: CASSANDRA-11503
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11503
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Sean Usher
>Assignee: Chris Lohfink
>Priority: Minor
> Attachments: CASSANDRA-11503.patch
>
>
> When using incremental repair, we should be able to look at SSTables and 
> understand how many sstables are in the repaired and unrepaired buckets on 
> each machine. This can help us track the repair progress and if we are 
> hitting any issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11657) LongLeveledCompactionStrategyTest broken since CASSANDRA-10099

[Yuki Morishita (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271378#comment-15271378
 ] 

Yuki Morishita commented on CASSANDRA-11657:


I still am not sure what's causing that error (and I still cannot reproduce in 
local).
Looking at the code change around {{LongLeveledCompactionStrategyTest}}, 
background task is disabled and {{CompactionTask}} is created sequentially.



> LongLeveledCompactionStrategyTest broken since CASSANDRA-10099
> --
>
> Key: CASSANDRA-11657
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11657
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
> Fix For: 3.x
>
>
> Seems CASSANDRA-10099 broke LongLeveledCompactionStrategyTest



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[3/3] cassandra git commit: replace deprecated sameThreadExecutor with directExecutor

[jbellis]

replace deprecated sameThreadExecutor with directExecutor


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/594183b0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/594183b0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/594183b0

Branch: refs/heads/trunk
Commit: 594183b0360aa252624f04e0e29fc12815fa39f4
Parents: f4233eb
Author: Jonathan Ellis 
Authored: Wed May 4 10:21:25 2016 -0500
Committer: Jonathan Ellis 
Committed: Wed May 4 14:32:09 2016 -0500

--
 .../org/apache/cassandra/repair/RepairMessageVerbHandler.java| 2 +-
 src/java/org/apache/cassandra/service/ActiveRepairService.java   | 4 ++--
 src/java/org/apache/cassandra/streaming/StreamManager.java   | 4 ++--
 3 files changed, 5 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/594183b0/src/java/org/apache/cassandra/repair/RepairMessageVerbHandler.java
--
diff --git a/src/java/org/apache/cassandra/repair/RepairMessageVerbHandler.java 
b/src/java/org/apache/cassandra/repair/RepairMessageVerbHandler.java
index 519fe5f..b06abb2 100644
--- a/src/java/org/apache/cassandra/repair/RepairMessageVerbHandler.java
+++ b/src/java/org/apache/cassandra/repair/RepairMessageVerbHandler.java
@@ -154,7 +154,7 @@ public class RepairMessageVerbHandler implements 
IVerbHandler
 {
 MessagingService.instance().sendReply(new 
MessageOut(MessagingService.Verb.INTERNAL_RESPONSE), id, message.from);
 }
-}, MoreExecutors.sameThreadExecutor());
+}, MoreExecutors.directExecutor());
 break;
 
 case CLEANUP:

http://git-wip-us.apache.org/repos/asf/cassandra/blob/594183b0/src/java/org/apache/cassandra/service/ActiveRepairService.java
--
diff --git a/src/java/org/apache/cassandra/service/ActiveRepairService.java 
b/src/java/org/apache/cassandra/service/ActiveRepairService.java
index 6c75149..dd196b9 100644
--- a/src/java/org/apache/cassandra/service/ActiveRepairService.java
+++ b/src/java/org/apache/cassandra/service/ActiveRepairService.java
@@ -151,7 +151,7 @@ public class ActiveRepairService
 gossiper.unregister(session);
 sessions.remove(session.getId());
 }
-}, MoreExecutors.sameThreadExecutor());
+}, MoreExecutors.directExecutor());
 session.start(executor);
 return session;
 }
@@ -401,7 +401,7 @@ public class ActiveRepairService
 {
 removeParentRepairSession(parentRepairSession);
 }
-}, MoreExecutors.sameThreadExecutor());
+}, MoreExecutors.directExecutor());
 
 return allAntiCompactionResults;
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/594183b0/src/java/org/apache/cassandra/streaming/StreamManager.java
--
diff --git a/src/java/org/apache/cassandra/streaming/StreamManager.java 
b/src/java/org/apache/cassandra/streaming/StreamManager.java
index dc8ec19..52652c0 100644
--- a/src/java/org/apache/cassandra/streaming/StreamManager.java
+++ b/src/java/org/apache/cassandra/streaming/StreamManager.java
@@ -131,7 +131,7 @@ public class StreamManager implements StreamManagerMBean
 {
 initiatedStreams.remove(result.planId);
 }
-}, MoreExecutors.sameThreadExecutor());
+}, MoreExecutors.directExecutor());
 
 initiatedStreams.put(result.planId, result);
 }
@@ -146,7 +146,7 @@ public class StreamManager implements StreamManagerMBean
 {
 receivingStreams.remove(result.planId);
 }
-}, MoreExecutors.sameThreadExecutor());
+}, MoreExecutors.directExecutor());
 
 receivingStreams.put(result.planId, result);
 }

[1/3] cassandra git commit: replace deprecated Iterables.emptyIterator with Collections.emptyIterator

[jbellis]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 06cd4e824 -> 594183b03


replace deprecated Iterables.emptyIterator with Collections.emptyIterator


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f4233eb7
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f4233eb7
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f4233eb7

Branch: refs/heads/trunk
Commit: f4233eb77c4703c951050b864abec40f4b7b03f2
Parents: e8a0d0a
Author: Jonathan Ellis 
Authored: Wed May 4 10:15:47 2016 -0500
Committer: Jonathan Ellis 
Committed: Wed May 4 14:32:08 2016 -0500

--
 src/java/org/apache/cassandra/db/MutableDeletionInfo.java | 5 +++--
 src/java/org/apache/cassandra/db/Slices.java  | 2 +-
 src/java/org/apache/cassandra/utils/IntervalTree.java | 2 +-
 test/unit/org/apache/cassandra/Util.java  | 2 +-
 4 files changed, 6 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f4233eb7/src/java/org/apache/cassandra/db/MutableDeletionInfo.java
--
diff --git a/src/java/org/apache/cassandra/db/MutableDeletionInfo.java 
b/src/java/org/apache/cassandra/db/MutableDeletionInfo.java
index 1ba77bb..4a2c455 100644
--- a/src/java/org/apache/cassandra/db/MutableDeletionInfo.java
+++ b/src/java/org/apache/cassandra/db/MutableDeletionInfo.java
@@ -17,6 +17,7 @@
  */
 package org.apache.cassandra.db;
 
+import java.util.Collections;
 import java.util.Iterator;
 
 import com.google.common.base.Objects;
@@ -151,12 +152,12 @@ public class MutableDeletionInfo implements DeletionInfo
 // Use sparingly, not the most efficient thing
 public Iterator rangeIterator(boolean reversed)
 {
-return ranges == null ? Iterators.emptyIterator() : 
ranges.iterator(reversed);
+return ranges == null ? Collections.emptyIterator() : 
ranges.iterator(reversed);
 }
 
 public Iterator rangeIterator(Slice slice, boolean 
reversed)
 {
-return ranges == null ? Iterators.emptyIterator() : 
ranges.iterator(slice, reversed);
+return ranges == null ? Collections.emptyIterator() : 
ranges.iterator(slice, reversed);
 }
 
 public RangeTombstone rangeCovering(Clustering name)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f4233eb7/src/java/org/apache/cassandra/db/Slices.java
--
diff --git a/src/java/org/apache/cassandra/db/Slices.java 
b/src/java/org/apache/cassandra/db/Slices.java
index 4fcda59..07da5b2 100644
--- a/src/java/org/apache/cassandra/db/Slices.java
+++ b/src/java/org/apache/cassandra/db/Slices.java
@@ -811,7 +811,7 @@ public abstract class Slices implements Iterable
 
 public Iterator iterator()
 {
-return Iterators.emptyIterator();
+return Collections.emptyIterator();
 }
 
 @Override

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f4233eb7/src/java/org/apache/cassandra/utils/IntervalTree.java
--
diff --git a/src/java/org/apache/cassandra/utils/IntervalTree.java 
b/src/java/org/apache/cassandra/utils/IntervalTree.java
index 4cf1222..f761180 100644
--- a/src/java/org/apache/cassandra/utils/IntervalTree.java
+++ b/src/java/org/apache/cassandra/utils/IntervalTree.java
@@ -114,7 +114,7 @@ public class IntervalTree, 
D, I extends Interval
 public Iterator iterator()
 {
 if (head == null)
-return Iterators.emptyIterator();
+return Collections.emptyIterator();
 
 return new TreeIterator(head);
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f4233eb7/test/unit/org/apache/cassandra/Util.java
--
diff --git a/test/unit/org/apache/cassandra/Util.java 
b/test/unit/org/apache/cassandra/Util.java
index 87a07b0..ed4fe10 100644
--- a/test/unit/org/apache/cassandra/Util.java
+++ b/test/unit/org/apache/cassandra/Util.java
@@ -470,7 +470,7 @@ public class Util
 // moved & refactored from KeyspaceTest in < 3.0
 public static void assertColumns(Row row, String... expectedColumnNames)
 {
-Iterator cells = row == null ? Iterators.emptyIterator() : 
row.cells().iterator();
+Iterator cells = row == null ? Collections.emptyIterator() : 
row.cells().iterator();
 String[] actual = Iterators.toArray(Iterators.transform(cells, new 
Function()
 {
 public String apply(Cell cell)

[2/3] cassandra git commit: update deprecated Objects.toStringHelper -> MoreObjects.toStringHelper

[jbellis]

update deprecated Objects.toStringHelper -> MoreObjects.toStringHelper


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e8a0d0aa
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e8a0d0aa
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e8a0d0aa

Branch: refs/heads/trunk
Commit: e8a0d0aac5bdcd468e334a46b5aade4d4559ef76
Parents: 06cd4e8
Author: Jonathan Ellis 
Authored: Wed May 4 10:00:15 2016 -0500
Committer: Jonathan Ellis 
Committed: Wed May 4 14:32:08 2016 -0500

--
 .../apache/cassandra/config/ColumnDefinition.java  | 13 +++--
 .../apache/cassandra/cql3/ColumnSpecification.java |  9 +
 .../apache/cassandra/cql3/selection/Selection.java | 15 ---
 .../cassandra/cql3/statements/SelectStatement.java | 13 +++--
 .../apache/cassandra/schema/KeyspaceMetadata.java  | 17 +
 .../apache/cassandra/schema/KeyspaceParams.java|  9 +
 .../apache/cassandra/schema/TriggerMetadata.java   |  9 +
 7 files changed, 46 insertions(+), 39 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e8a0d0aa/src/java/org/apache/cassandra/config/ColumnDefinition.java
--
diff --git a/src/java/org/apache/cassandra/config/ColumnDefinition.java 
b/src/java/org/apache/cassandra/config/ColumnDefinition.java
index a18ed3f..a900ce7 100644
--- a/src/java/org/apache/cassandra/config/ColumnDefinition.java
+++ b/src/java/org/apache/cassandra/config/ColumnDefinition.java
@@ -22,6 +22,7 @@ import java.util.*;
 
 import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Function;
+import com.google.common.base.MoreObjects;
 import com.google.common.base.Objects;
 import com.google.common.collect.Collections2;
 
@@ -282,12 +283,12 @@ public class ColumnDefinition extends ColumnSpecification 
implements Comparable<
 @Override
 public String toString()
 {
-return Objects.toStringHelper(this)
-  .add("name", name)
-  .add("type", type)
-  .add("kind", kind)
-  .add("position", position)
-  .toString();
+return MoreObjects.toStringHelper(this)
+  .add("name", name)
+  .add("type", type)
+  .add("kind", kind)
+  .add("position", position)
+  .toString();
 }
 
 public boolean isPrimaryKeyColumn()

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e8a0d0aa/src/java/org/apache/cassandra/cql3/ColumnSpecification.java
--
diff --git a/src/java/org/apache/cassandra/cql3/ColumnSpecification.java 
b/src/java/org/apache/cassandra/cql3/ColumnSpecification.java
index e64f5f9..8cf869b 100644
--- a/src/java/org/apache/cassandra/cql3/ColumnSpecification.java
+++ b/src/java/org/apache/cassandra/cql3/ColumnSpecification.java
@@ -17,6 +17,7 @@
  */
 package org.apache.cassandra.cql3;
 
+import com.google.common.base.MoreObjects;
 import com.google.common.base.Objects;
 
 import org.apache.cassandra.db.marshal.AbstractType;
@@ -96,9 +97,9 @@ public class ColumnSpecification
 @Override
 public String toString()
 {
-return Objects.toStringHelper(this)
-  .add("name", name)
-  .add("type", type)
-  .toString();
+return MoreObjects.toStringHelper(this)
+  .add("name", name)
+  .add("type", type)
+  .toString();
 }
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e8a0d0aa/src/java/org/apache/cassandra/cql3/selection/Selection.java
--
diff --git a/src/java/org/apache/cassandra/cql3/selection/Selection.java 
b/src/java/org/apache/cassandra/cql3/selection/Selection.java
index f337c1a..664fd4f 100644
--- a/src/java/org/apache/cassandra/cql3/selection/Selection.java
+++ b/src/java/org/apache/cassandra/cql3/selection/Selection.java
@@ -20,6 +20,7 @@ package org.apache.cassandra.cql3.selection;
 import java.nio.ByteBuffer;
 import java.util.*;
 
+import com.google.common.base.MoreObjects;
 import com.google.common.base.Objects;
 import com.google.common.base.Predicate;
 import com.google.common.collect.Iterables;
@@ -248,13 +249,13 @@ public abstract class Selection
 @Override
 public String toString()
 {
-return Objects.toStringHelper(this)
-.add("columns", columns)
-.add("columnMapping", columnMapping)
-

[jira] [Commented] (CASSANDRA-7826) support non-frozen, nested collections

[Alex Petrov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271250#comment-15271250
 ] 

Alex Petrov commented on CASSANDRA-7826:


Would the updates and deletes for particular cells be within the scope of the 
ticket?

After talking to [~thobbs] there's also the question if we would like to 
support arbitrary updates or deletes right away, too. That would require new 
syntax for selecting the path for update or delete (maybe something like 
{{mymap['a']['b'][1]}}). Also, {{BufferCell/tombstone}} can most likely handle 
deletes as they are, without slices, as we'll always know at least the partial 
path, but if I understand it correctly, it will require additional logic for 
discarding columns whose partial path matches the path of the tombstone.

Also, most likely [7396|https://issues.apache.org/jira/browse/CASSANDRA-7396] 
will land before this patch, so it we might have to add support for selecting 
individual parts on arbitrary level (or just on the top level at first and 
leave the rest for the next iterations). So the question is what we'd like to 
have on the current step.

Right now, arbitrary nesting of maps is fully working (also, dtests and UDFs 
with nested structures are working). Non-frozen Set and List can be nested 
arbitrarily deep within the map, although nothing can be nested within Set or 
List. As was mentioned above. 

> support non-frozen, nested collections
> --
>
> Key: CASSANDRA-7826
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7826
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CQL
>Reporter: Tupshin Harper
>Assignee: Alex Petrov
>  Labels: ponies
> Fix For: 3.x
>
>
> The inability to nest collections is one of the bigger data modelling 
> limitations we have right now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (CASSANDRA-7826) support non-frozen, nested collections

[Alex Petrov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Petrov updated CASSANDRA-7826:
---
Comment: was deleted

(was: S)

> support non-frozen, nested collections
> --
>
> Key: CASSANDRA-7826
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7826
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CQL
>Reporter: Tupshin Harper
>Assignee: Alex Petrov
>  Labels: ponies
> Fix For: 3.x
>
>
> The inability to nest collections is one of the bigger data modelling 
> limitations we have right now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-7826) support non-frozen, nested collections

[Alex Petrov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271221#comment-15271221
 ] 

Alex Petrov commented on CASSANDRA-7826:


S

> support non-frozen, nested collections
> --
>
> Key: CASSANDRA-7826
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7826
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CQL
>Reporter: Tupshin Harper
>Assignee: Alex Petrov
>  Labels: ponies
> Fix For: 3.x
>
>
> The inability to nest collections is one of the bigger data modelling 
> limitations we have right now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11540) The JVM should exit if jmx fails to bind

[Alex Petrov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271159#comment-15271159
 ] 

Alex Petrov commented on CASSANDRA-11540:
-

oh :) thank you [~beobal] :) looks like I got late with my comment about tests )

> The JVM should exit if jmx fails to bind
> 
>
> Key: CASSANDRA-11540
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11540
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Brandon Williams
>Assignee: Alex Petrov
>  Labels: lhf
> Fix For: 2.2.7, 3.7, 3.0.7
>
>
> If you are already running a cassandra instance, but for some reason try to 
> start another one, this happens:
> {noformat}
> INFO  20:57:09 JNA mlockall successful
> WARN  20:57:09 JMX is not enabled to receive remote connections. Please see 
> cassandra-env.sh for more info.
> ERROR 20:57:10 Error starting local jmx server:
> java.rmi.server.ExportException: Port already in use: 7199; nested exception 
> is:
> java.net.BindException: Address already in use
> at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:340) 
> ~[na:1.7.0_76]
> at 
> sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:248) 
> ~[na:1.7.0_76]
> at 
> sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:411) 
> ~[na:1.7.0_76]
> at sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147) 
> ~[na:1.7.0_76]
> at 
> sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:207) 
> ~[na:1.7.0_76]
> at sun.rmi.registry.RegistryImpl.setup(RegistryImpl.java:122) 
> ~[na:1.7.0_76]
> at sun.rmi.registry.RegistryImpl.(RegistryImpl.java:98) 
> ~[na:1.7.0_76]
> at 
> java.rmi.registry.LocateRegistry.createRegistry(LocateRegistry.java:239) 
> ~[na:1.7.0_76]
> at 
> org.apache.cassandra.service.CassandraDaemon.maybeInitJmx(CassandraDaemon.java:100)
>  [main/:na]
> at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:222) 
> [main/:na]
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:564)
>  [main/:na]
> at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:653) 
> [main/:na]
> Caused by: java.net.BindException: Address already in use
> at java.net.PlainSocketImpl.socketBind(Native Method) ~[na:1.7.0_76]
> at 
> java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376) 
> ~[na:1.7.0_76]
> at java.net.ServerSocket.bind(ServerSocket.java:376) ~[na:1.7.0_76]
> at java.net.ServerSocket.(ServerSocket.java:237) ~[na:1.7.0_76]
> at 
> javax.net.DefaultServerSocketFactory.createServerSocket(ServerSocketFactory.java:231)
>  ~[na:1.7.0_76]
> at 
> org.apache.cassandra.utils.RMIServerSocketFactoryImpl.createServerSocket(RMIServerSocketFactoryImpl.java:13)
>  ~[main/:na]
> at 
> sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(TCPEndpoint.java:666) 
> ~[na:1.7.0_76]
> at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:329) 
> ~[na:1.7.0_76]
> ... 11 common frames omitted
> {noformat}
> However the startup continues, and ends up replaying commitlogs, which is 
> probably not a good thing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11540) The JVM should exit if jmx fails to bind

[Alex Petrov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271157#comment-15271157
 ] 

Alex Petrov commented on CASSANDRA-11540:
-

>From dtests, {{restart_node_localhost_test}} is already reported was 
>introduced before the patch: 
>[11702|https://issues.apache.org/jira/browse/CASSANDRA-11702] (although it 
>looked related. also, passes locally) same with {{test_upgrade_index_summary}} 
>[11127|https://issues.apache.org/jira/browse/CASSANDRA-11127] and 
>{{upgrade_with_wide_partition_reversed_test}} 
>[11663|https://issues.apache.org/jira/browse/CASSANDRA-11663]. The rest of 
>them also look unrelated, although I ran them just in case. 
>{{testJsonThreadSafety}} is failing for me quite often, so I filed the issue, 
>too: [11712|https://issues.apache.org/jira/browse/CASSANDRA-11712].

> The JVM should exit if jmx fails to bind
> 
>
> Key: CASSANDRA-11540
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11540
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Brandon Williams
>Assignee: Alex Petrov
>  Labels: lhf
> Fix For: 2.2.7, 3.7, 3.0.7
>
>
> If you are already running a cassandra instance, but for some reason try to 
> start another one, this happens:
> {noformat}
> INFO  20:57:09 JNA mlockall successful
> WARN  20:57:09 JMX is not enabled to receive remote connections. Please see 
> cassandra-env.sh for more info.
> ERROR 20:57:10 Error starting local jmx server:
> java.rmi.server.ExportException: Port already in use: 7199; nested exception 
> is:
> java.net.BindException: Address already in use
> at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:340) 
> ~[na:1.7.0_76]
> at 
> sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:248) 
> ~[na:1.7.0_76]
> at 
> sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:411) 
> ~[na:1.7.0_76]
> at sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147) 
> ~[na:1.7.0_76]
> at 
> sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:207) 
> ~[na:1.7.0_76]
> at sun.rmi.registry.RegistryImpl.setup(RegistryImpl.java:122) 
> ~[na:1.7.0_76]
> at sun.rmi.registry.RegistryImpl.(RegistryImpl.java:98) 
> ~[na:1.7.0_76]
> at 
> java.rmi.registry.LocateRegistry.createRegistry(LocateRegistry.java:239) 
> ~[na:1.7.0_76]
> at 
> org.apache.cassandra.service.CassandraDaemon.maybeInitJmx(CassandraDaemon.java:100)
>  [main/:na]
> at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:222) 
> [main/:na]
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:564)
>  [main/:na]
> at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:653) 
> [main/:na]
> Caused by: java.net.BindException: Address already in use
> at java.net.PlainSocketImpl.socketBind(Native Method) ~[na:1.7.0_76]
> at 
> java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376) 
> ~[na:1.7.0_76]
> at java.net.ServerSocket.bind(ServerSocket.java:376) ~[na:1.7.0_76]
> at java.net.ServerSocket.(ServerSocket.java:237) ~[na:1.7.0_76]
> at 
> javax.net.DefaultServerSocketFactory.createServerSocket(ServerSocketFactory.java:231)
>  ~[na:1.7.0_76]
> at 
> org.apache.cassandra.utils.RMIServerSocketFactoryImpl.createServerSocket(RMIServerSocketFactoryImpl.java:13)
>  ~[main/:na]
> at 
> sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(TCPEndpoint.java:666) 
> ~[na:1.7.0_76]
> at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:329) 
> ~[na:1.7.0_76]
> ... 11 common frames omitted
> {noformat}
> However the startup continues, and ends up replaying commitlogs, which is 
> probably not a good thing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10962) Cassandra should not create snapshot at restart for compactions_in_progress

[Joel Knighton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Knighton updated CASSANDRA-10962:
--
Status: Ready to Commit  (was: Patch Available)

> Cassandra should not create snapshot at restart for compactions_in_progress
> ---
>
> Key: CASSANDRA-10962
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10962
> Project: Cassandra
>  Issue Type: Bug
> Environment: Ubuntu 14.04.3 LTS
>Reporter: FACORAT
>Assignee: Alex Petrov
>Priority: Minor
> Fix For: 2.2.x
>
>
> If auto_snapshot is set to true in cassandra.yaml, each time you restart 
> Cassandra, a snapshot is created for system.compactions_in_progress as the 
> table is truncated at cassandra start.
> However as datas in this table are temporary, Cassandra should not create 
> snapshot for this table (or maybe even for system.* tables). This will be 
> coherent with the fact that "nodetool listsnapshots" doesn't even list this 
> table.
> Exemple:
> {code}
> $ nodetool listsnapshots | grep compactions
> $ ls -lh 
> system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/snapshots/
> total 16K
> drwxr-xr-x 2 cassandra cassandra 4.0K Nov 30 13:12 
> 1448885530280-compactions_in_progress
> drwxr-xr-x 2 cassandra cassandra 4.0K Dec  7 15:36 
> 1449498977181-compactions_in_progress
> drwxr-xr-x 2 cassandra cassandra 4.0K Dec 14 18:20 
> 1450113621506-compactions_in_progress
> drwxr-xr-x 2 cassandra cassandra 4.0K Jan  4 12:53 
> 1451908396364-compactions_in_progress
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10962) Cassandra should not create snapshot at restart for compactions_in_progress

[Joel Knighton (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271088#comment-15271088
 ] 

Joel Knighton commented on CASSANDRA-10962:
---

+1, LGTM. CI is clean relative to upstream.

As [~ifesdjeen] noted, this should only go to 2.2, since the related subsystem 
was rewritten in 3.0.

> Cassandra should not create snapshot at restart for compactions_in_progress
> ---
>
> Key: CASSANDRA-10962
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10962
> Project: Cassandra
>  Issue Type: Bug
> Environment: Ubuntu 14.04.3 LTS
>Reporter: FACORAT
>Assignee: Alex Petrov
>Priority: Minor
> Fix For: 2.2.x
>
>
> If auto_snapshot is set to true in cassandra.yaml, each time you restart 
> Cassandra, a snapshot is created for system.compactions_in_progress as the 
> table is truncated at cassandra start.
> However as datas in this table are temporary, Cassandra should not create 
> snapshot for this table (or maybe even for system.* tables). This will be 
> coherent with the fact that "nodetool listsnapshots" doesn't even list this 
> table.
> Exemple:
> {code}
> $ nodetool listsnapshots | grep compactions
> $ ls -lh 
> system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/snapshots/
> total 16K
> drwxr-xr-x 2 cassandra cassandra 4.0K Nov 30 13:12 
> 1448885530280-compactions_in_progress
> drwxr-xr-x 2 cassandra cassandra 4.0K Dec  7 15:36 
> 1449498977181-compactions_in_progress
> drwxr-xr-x 2 cassandra cassandra 4.0K Dec 14 18:20 
> 1450113621506-compactions_in_progress
> drwxr-xr-x 2 cassandra cassandra 4.0K Jan  4 12:53 
> 1451908396364-compactions_in_progress
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11540) The JVM should exit if jmx fails to bind

[Sam Tunnicliffe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Tunnicliffe updated CASSANDRA-11540:

   Resolution: Fixed
Fix Version/s: (was: 3.0.x)
   (was: 2.2.x)
   (was: 3.x)
   3.0.7
   3.7
   2.2.7
   Status: Resolved  (was: Patch Available)

CI looks good, so committed to 2.2 in 
{{93c5bc616e21ffa7f31266ad095ca374f2ba73a4}} and merged to 3.0/3.7/trunk.

> The JVM should exit if jmx fails to bind
> 
>
> Key: CASSANDRA-11540
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11540
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Brandon Williams
>Assignee: Alex Petrov
>  Labels: lhf
> Fix For: 2.2.7, 3.7, 3.0.7
>
>
> If you are already running a cassandra instance, but for some reason try to 
> start another one, this happens:
> {noformat}
> INFO  20:57:09 JNA mlockall successful
> WARN  20:57:09 JMX is not enabled to receive remote connections. Please see 
> cassandra-env.sh for more info.
> ERROR 20:57:10 Error starting local jmx server:
> java.rmi.server.ExportException: Port already in use: 7199; nested exception 
> is:
> java.net.BindException: Address already in use
> at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:340) 
> ~[na:1.7.0_76]
> at 
> sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:248) 
> ~[na:1.7.0_76]
> at 
> sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:411) 
> ~[na:1.7.0_76]
> at sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147) 
> ~[na:1.7.0_76]
> at 
> sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:207) 
> ~[na:1.7.0_76]
> at sun.rmi.registry.RegistryImpl.setup(RegistryImpl.java:122) 
> ~[na:1.7.0_76]
> at sun.rmi.registry.RegistryImpl.(RegistryImpl.java:98) 
> ~[na:1.7.0_76]
> at 
> java.rmi.registry.LocateRegistry.createRegistry(LocateRegistry.java:239) 
> ~[na:1.7.0_76]
> at 
> org.apache.cassandra.service.CassandraDaemon.maybeInitJmx(CassandraDaemon.java:100)
>  [main/:na]
> at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:222) 
> [main/:na]
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:564)
>  [main/:na]
> at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:653) 
> [main/:na]
> Caused by: java.net.BindException: Address already in use
> at java.net.PlainSocketImpl.socketBind(Native Method) ~[na:1.7.0_76]
> at 
> java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376) 
> ~[na:1.7.0_76]
> at java.net.ServerSocket.bind(ServerSocket.java:376) ~[na:1.7.0_76]
> at java.net.ServerSocket.(ServerSocket.java:237) ~[na:1.7.0_76]
> at 
> javax.net.DefaultServerSocketFactory.createServerSocket(ServerSocketFactory.java:231)
>  ~[na:1.7.0_76]
> at 
> org.apache.cassandra.utils.RMIServerSocketFactoryImpl.createServerSocket(RMIServerSocketFactoryImpl.java:13)
>  ~[main/:na]
> at 
> sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(TCPEndpoint.java:666) 
> ~[na:1.7.0_76]
> at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:329) 
> ~[na:1.7.0_76]
> ... 11 common frames omitted
> {noformat}
> However the startup continues, and ends up replaying commitlogs, which is 
> probably not a good thing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[02/10] cassandra git commit: Avoid starting Cassandra on JMX bind failure

[samt]

Avoid starting Cassandra on JMX bind failure

Patch by Alex Petrov; reviewed by Sam Tunnicliffe for CASSANDRA-11540


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/93c5bc61
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/93c5bc61
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/93c5bc61

Branch: refs/heads/cassandra-3.0
Commit: 93c5bc616e21ffa7f31266ad095ca374f2ba73a4
Parents: 3d73282
Author: Alex Petrov 
Authored: Mon Apr 18 13:47:59 2016 +0200
Committer: Sam Tunnicliffe 
Committed: Wed May 4 17:55:47 2016 +0100

--
 CHANGES.txt| 1 +
 src/java/org/apache/cassandra/service/CassandraDaemon.java | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/93c5bc61/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 7c7295d..0d9d3e9 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.7
+ * Exit JVM if JMX server fails to startup (CASSANDRA-11540)
  * Produce a heap dump when exiting on OOM (CASSANDRA-9861)
  * Avoid read repairing purgeable tombstones on range slices (CASSANDRA-11427)
  * Restore ability to filter on clustering columns when using a 2i 
(CASSANDRA-11510)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/93c5bc61/src/java/org/apache/cassandra/service/CassandraDaemon.java
--
diff --git a/src/java/org/apache/cassandra/service/CassandraDaemon.java 
b/src/java/org/apache/cassandra/service/CassandraDaemon.java
index 6f38545..7e33e9c 100644
--- a/src/java/org/apache/cassandra/service/CassandraDaemon.java
+++ b/src/java/org/apache/cassandra/service/CassandraDaemon.java
@@ -98,7 +98,7 @@ public class CassandraDaemon
 logger = LoggerFactory.getLogger(CassandraDaemon.class);
 }
 
-private static void maybeInitJmx()
+private void maybeInitJmx()
 {
 if (System.getProperty("com.sun.management.jmxremote.port") != null)
 return;
@@ -119,7 +119,7 @@ public class CassandraDaemon
 }
 catch (IOException e)
 {
-logger.error("Error starting local jmx server: ", e);
+exitOrFail(1, e.getMessage(), e.getCause());
 }
 }
 

[07/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[samt]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5980b336
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5980b336
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5980b336

Branch: refs/heads/trunk
Commit: 5980b336f0511987066ba175b9b52f0eafda77fd
Parents: 2e202d4 93c5bc6
Author: Sam Tunnicliffe 
Authored: Wed May 4 17:58:01 2016 +0100
Committer: Sam Tunnicliffe 
Committed: Wed May 4 17:58:01 2016 +0100

--
 CHANGES.txt| 1 +
 src/java/org/apache/cassandra/service/CassandraDaemon.java | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5980b336/CHANGES.txt
--
diff --cc CHANGES.txt
index 31e46d9,0d9d3e9..860a4c2
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,25 -1,7 +1,26 @@@
 -2.2.7
 +3.0.6
 + * Disallow creating view with a static column (CASSANDRA-11602)
 + * Reduce the amount of object allocations caused by the getFunctions methods 
(CASSANDRA-11593)
 + * Potential error replaying commitlog with smallint/tinyint/date/time types 
(CASSANDRA-11618)
 + * Fix queries with filtering on counter columns (CASSANDRA-11629)
 + * Improve tombstone printing in sstabledump (CASSANDRA-11655)
 + * Fix paging for range queries where all clustering columns are specified 
(CASSANDRA-11669)
 + * Don't require HEAP_NEW_SIZE to be set when using G1 (CASSANDRA-11600)
 + * Fix sstabledump not showing cells after tombstone marker (CASSANDRA-11654)
 + * Ignore all LocalStrategy keyspaces for streaming and other related
 +   operations (CASSANDRA-11627)
 + * Ensure columnfilter covers indexed columns for thrift 2i queries 
(CASSANDRA-11523)
 + * Only open one sstable scanner per sstable (CASSANDRA-11412)
 + * Option to specify ProtocolVersion in cassandra-stress (CASSANDRA-11410)
 + * ArithmeticException in avgFunctionForDecimal (CASSANDRA-11485)
 + * LogAwareFileLister should only use OLD sstable files in current folder to 
determine disk consistency (CASSANDRA-11470)
 + * Notify indexers of expired rows during compaction (CASSANDRA-11329)
 + * Properly respond with ProtocolError when a v1/v2 native protocol
 +   header is received (CASSANDRA-11464)
 + * Validate that num_tokens and initial_token are consistent with one another 
(CASSANDRA-10120)
 +Merged from 2.2:
+  * Exit JVM if JMX server fails to startup (CASSANDRA-11540)
   * Produce a heap dump when exiting on OOM (CASSANDRA-9861)
 - * Avoid read repairing purgeable tombstones on range slices (CASSANDRA-11427)
   * Restore ability to filter on clustering columns when using a 2i 
(CASSANDRA-11510)
   * JSON datetime formatting needs timezone (CASSANDRA-11137)
   * Fix is_dense recalculation for Thrift-updated tables (CASSANDRA-11502)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5980b336/src/java/org/apache/cassandra/service/CassandraDaemon.java
--

[03/10] cassandra git commit: Avoid starting Cassandra on JMX bind failure

[samt]

Avoid starting Cassandra on JMX bind failure

Patch by Alex Petrov; reviewed by Sam Tunnicliffe for CASSANDRA-11540


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/93c5bc61
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/93c5bc61
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/93c5bc61

Branch: refs/heads/cassandra-3.7
Commit: 93c5bc616e21ffa7f31266ad095ca374f2ba73a4
Parents: 3d73282
Author: Alex Petrov 
Authored: Mon Apr 18 13:47:59 2016 +0200
Committer: Sam Tunnicliffe 
Committed: Wed May 4 17:55:47 2016 +0100

--
 CHANGES.txt| 1 +
 src/java/org/apache/cassandra/service/CassandraDaemon.java | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/93c5bc61/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 7c7295d..0d9d3e9 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.7
+ * Exit JVM if JMX server fails to startup (CASSANDRA-11540)
  * Produce a heap dump when exiting on OOM (CASSANDRA-9861)
  * Avoid read repairing purgeable tombstones on range slices (CASSANDRA-11427)
  * Restore ability to filter on clustering columns when using a 2i 
(CASSANDRA-11510)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/93c5bc61/src/java/org/apache/cassandra/service/CassandraDaemon.java
--
diff --git a/src/java/org/apache/cassandra/service/CassandraDaemon.java 
b/src/java/org/apache/cassandra/service/CassandraDaemon.java
index 6f38545..7e33e9c 100644
--- a/src/java/org/apache/cassandra/service/CassandraDaemon.java
+++ b/src/java/org/apache/cassandra/service/CassandraDaemon.java
@@ -98,7 +98,7 @@ public class CassandraDaemon
 logger = LoggerFactory.getLogger(CassandraDaemon.class);
 }
 
-private static void maybeInitJmx()
+private void maybeInitJmx()
 {
 if (System.getProperty("com.sun.management.jmxremote.port") != null)
 return;
@@ -119,7 +119,7 @@ public class CassandraDaemon
 }
 catch (IOException e)
 {
-logger.error("Error starting local jmx server: ", e);
+exitOrFail(1, e.getMessage(), e.getCause());
 }
 }
 

[04/10] cassandra git commit: Avoid starting Cassandra on JMX bind failure

[samt]

Avoid starting Cassandra on JMX bind failure

Patch by Alex Petrov; reviewed by Sam Tunnicliffe for CASSANDRA-11540


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/93c5bc61
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/93c5bc61
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/93c5bc61

Branch: refs/heads/trunk
Commit: 93c5bc616e21ffa7f31266ad095ca374f2ba73a4
Parents: 3d73282
Author: Alex Petrov 
Authored: Mon Apr 18 13:47:59 2016 +0200
Committer: Sam Tunnicliffe 
Committed: Wed May 4 17:55:47 2016 +0100

--
 CHANGES.txt| 1 +
 src/java/org/apache/cassandra/service/CassandraDaemon.java | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/93c5bc61/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 7c7295d..0d9d3e9 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.7
+ * Exit JVM if JMX server fails to startup (CASSANDRA-11540)
  * Produce a heap dump when exiting on OOM (CASSANDRA-9861)
  * Avoid read repairing purgeable tombstones on range slices (CASSANDRA-11427)
  * Restore ability to filter on clustering columns when using a 2i 
(CASSANDRA-11510)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/93c5bc61/src/java/org/apache/cassandra/service/CassandraDaemon.java
--
diff --git a/src/java/org/apache/cassandra/service/CassandraDaemon.java 
b/src/java/org/apache/cassandra/service/CassandraDaemon.java
index 6f38545..7e33e9c 100644
--- a/src/java/org/apache/cassandra/service/CassandraDaemon.java
+++ b/src/java/org/apache/cassandra/service/CassandraDaemon.java
@@ -98,7 +98,7 @@ public class CassandraDaemon
 logger = LoggerFactory.getLogger(CassandraDaemon.class);
 }
 
-private static void maybeInitJmx()
+private void maybeInitJmx()
 {
 if (System.getProperty("com.sun.management.jmxremote.port") != null)
 return;
@@ -119,7 +119,7 @@ public class CassandraDaemon
 }
 catch (IOException e)
 {
-logger.error("Error starting local jmx server: ", e);
+exitOrFail(1, e.getMessage(), e.getCause());
 }
 }
 

[08/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.7

[samt]

Merge branch 'cassandra-3.0' into cassandra-3.7


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/68580c7a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/68580c7a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/68580c7a

Branch: refs/heads/cassandra-3.7
Commit: 68580c7a81fb20ef76abdfbe5eaefcc07b97
Parents: 415bb56 5980b33
Author: Sam Tunnicliffe 
Authored: Wed May 4 18:02:40 2016 +0100
Committer: Sam Tunnicliffe 
Committed: Wed May 4 18:02:40 2016 +0100

--
 CHANGES.txt| 1 +
 src/java/org/apache/cassandra/service/CassandraDaemon.java | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/68580c7a/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/68580c7a/src/java/org/apache/cassandra/service/CassandraDaemon.java
--

[01/10] cassandra git commit: Avoid starting Cassandra on JMX bind failure

[samt]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.2 3d7328232 -> 93c5bc616
  refs/heads/cassandra-3.0 2e202d47d -> 5980b336f
  refs/heads/cassandra-3.7 415bb569d -> 68580c7a8
  refs/heads/trunk d6daf1b66 -> 06cd4e824


Avoid starting Cassandra on JMX bind failure

Patch by Alex Petrov; reviewed by Sam Tunnicliffe for CASSANDRA-11540


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/93c5bc61
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/93c5bc61
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/93c5bc61

Branch: refs/heads/cassandra-2.2
Commit: 93c5bc616e21ffa7f31266ad095ca374f2ba73a4
Parents: 3d73282
Author: Alex Petrov 
Authored: Mon Apr 18 13:47:59 2016 +0200
Committer: Sam Tunnicliffe 
Committed: Wed May 4 17:55:47 2016 +0100

--
 CHANGES.txt| 1 +
 src/java/org/apache/cassandra/service/CassandraDaemon.java | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/93c5bc61/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 7c7295d..0d9d3e9 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.7
+ * Exit JVM if JMX server fails to startup (CASSANDRA-11540)
  * Produce a heap dump when exiting on OOM (CASSANDRA-9861)
  * Avoid read repairing purgeable tombstones on range slices (CASSANDRA-11427)
  * Restore ability to filter on clustering columns when using a 2i 
(CASSANDRA-11510)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/93c5bc61/src/java/org/apache/cassandra/service/CassandraDaemon.java
--
diff --git a/src/java/org/apache/cassandra/service/CassandraDaemon.java 
b/src/java/org/apache/cassandra/service/CassandraDaemon.java
index 6f38545..7e33e9c 100644
--- a/src/java/org/apache/cassandra/service/CassandraDaemon.java
+++ b/src/java/org/apache/cassandra/service/CassandraDaemon.java
@@ -98,7 +98,7 @@ public class CassandraDaemon
 logger = LoggerFactory.getLogger(CassandraDaemon.class);
 }
 
-private static void maybeInitJmx()
+private void maybeInitJmx()
 {
 if (System.getProperty("com.sun.management.jmxremote.port") != null)
 return;
@@ -119,7 +119,7 @@ public class CassandraDaemon
 }
 catch (IOException e)
 {
-logger.error("Error starting local jmx server: ", e);
+exitOrFail(1, e.getMessage(), e.getCause());
 }
 }
 

[06/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[samt]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5980b336
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5980b336
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5980b336

Branch: refs/heads/cassandra-3.0
Commit: 5980b336f0511987066ba175b9b52f0eafda77fd
Parents: 2e202d4 93c5bc6
Author: Sam Tunnicliffe 
Authored: Wed May 4 17:58:01 2016 +0100
Committer: Sam Tunnicliffe 
Committed: Wed May 4 17:58:01 2016 +0100

--
 CHANGES.txt| 1 +
 src/java/org/apache/cassandra/service/CassandraDaemon.java | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5980b336/CHANGES.txt
--
diff --cc CHANGES.txt
index 31e46d9,0d9d3e9..860a4c2
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,25 -1,7 +1,26 @@@
 -2.2.7
 +3.0.6
 + * Disallow creating view with a static column (CASSANDRA-11602)
 + * Reduce the amount of object allocations caused by the getFunctions methods 
(CASSANDRA-11593)
 + * Potential error replaying commitlog with smallint/tinyint/date/time types 
(CASSANDRA-11618)
 + * Fix queries with filtering on counter columns (CASSANDRA-11629)
 + * Improve tombstone printing in sstabledump (CASSANDRA-11655)
 + * Fix paging for range queries where all clustering columns are specified 
(CASSANDRA-11669)
 + * Don't require HEAP_NEW_SIZE to be set when using G1 (CASSANDRA-11600)
 + * Fix sstabledump not showing cells after tombstone marker (CASSANDRA-11654)
 + * Ignore all LocalStrategy keyspaces for streaming and other related
 +   operations (CASSANDRA-11627)
 + * Ensure columnfilter covers indexed columns for thrift 2i queries 
(CASSANDRA-11523)
 + * Only open one sstable scanner per sstable (CASSANDRA-11412)
 + * Option to specify ProtocolVersion in cassandra-stress (CASSANDRA-11410)
 + * ArithmeticException in avgFunctionForDecimal (CASSANDRA-11485)
 + * LogAwareFileLister should only use OLD sstable files in current folder to 
determine disk consistency (CASSANDRA-11470)
 + * Notify indexers of expired rows during compaction (CASSANDRA-11329)
 + * Properly respond with ProtocolError when a v1/v2 native protocol
 +   header is received (CASSANDRA-11464)
 + * Validate that num_tokens and initial_token are consistent with one another 
(CASSANDRA-10120)
 +Merged from 2.2:
+  * Exit JVM if JMX server fails to startup (CASSANDRA-11540)
   * Produce a heap dump when exiting on OOM (CASSANDRA-9861)
 - * Avoid read repairing purgeable tombstones on range slices (CASSANDRA-11427)
   * Restore ability to filter on clustering columns when using a 2i 
(CASSANDRA-11510)
   * JSON datetime formatting needs timezone (CASSANDRA-11137)
   * Fix is_dense recalculation for Thrift-updated tables (CASSANDRA-11502)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5980b336/src/java/org/apache/cassandra/service/CassandraDaemon.java
--

[10/10] cassandra git commit: Merge branch 'cassandra-3.7' into trunk

[samt]

Merge branch 'cassandra-3.7' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/06cd4e82
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/06cd4e82
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/06cd4e82

Branch: refs/heads/trunk
Commit: 06cd4e824539e266305733e5b3ffee65e42fe602
Parents: d6daf1b 68580c7
Author: Sam Tunnicliffe 
Authored: Wed May 4 18:02:49 2016 +0100
Committer: Sam Tunnicliffe 
Committed: Wed May 4 18:02:49 2016 +0100

--
 CHANGES.txt| 1 +
 src/java/org/apache/cassandra/service/CassandraDaemon.java | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/06cd4e82/CHANGES.txt
--

[05/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[samt]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5980b336
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5980b336
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5980b336

Branch: refs/heads/cassandra-3.7
Commit: 5980b336f0511987066ba175b9b52f0eafda77fd
Parents: 2e202d4 93c5bc6
Author: Sam Tunnicliffe 
Authored: Wed May 4 17:58:01 2016 +0100
Committer: Sam Tunnicliffe 
Committed: Wed May 4 17:58:01 2016 +0100

--
 CHANGES.txt| 1 +
 src/java/org/apache/cassandra/service/CassandraDaemon.java | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5980b336/CHANGES.txt
--
diff --cc CHANGES.txt
index 31e46d9,0d9d3e9..860a4c2
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,25 -1,7 +1,26 @@@
 -2.2.7
 +3.0.6
 + * Disallow creating view with a static column (CASSANDRA-11602)
 + * Reduce the amount of object allocations caused by the getFunctions methods 
(CASSANDRA-11593)
 + * Potential error replaying commitlog with smallint/tinyint/date/time types 
(CASSANDRA-11618)
 + * Fix queries with filtering on counter columns (CASSANDRA-11629)
 + * Improve tombstone printing in sstabledump (CASSANDRA-11655)
 + * Fix paging for range queries where all clustering columns are specified 
(CASSANDRA-11669)
 + * Don't require HEAP_NEW_SIZE to be set when using G1 (CASSANDRA-11600)
 + * Fix sstabledump not showing cells after tombstone marker (CASSANDRA-11654)
 + * Ignore all LocalStrategy keyspaces for streaming and other related
 +   operations (CASSANDRA-11627)
 + * Ensure columnfilter covers indexed columns for thrift 2i queries 
(CASSANDRA-11523)
 + * Only open one sstable scanner per sstable (CASSANDRA-11412)
 + * Option to specify ProtocolVersion in cassandra-stress (CASSANDRA-11410)
 + * ArithmeticException in avgFunctionForDecimal (CASSANDRA-11485)
 + * LogAwareFileLister should only use OLD sstable files in current folder to 
determine disk consistency (CASSANDRA-11470)
 + * Notify indexers of expired rows during compaction (CASSANDRA-11329)
 + * Properly respond with ProtocolError when a v1/v2 native protocol
 +   header is received (CASSANDRA-11464)
 + * Validate that num_tokens and initial_token are consistent with one another 
(CASSANDRA-10120)
 +Merged from 2.2:
+  * Exit JVM if JMX server fails to startup (CASSANDRA-11540)
   * Produce a heap dump when exiting on OOM (CASSANDRA-9861)
 - * Avoid read repairing purgeable tombstones on range slices (CASSANDRA-11427)
   * Restore ability to filter on clustering columns when using a 2i 
(CASSANDRA-11510)
   * JSON datetime formatting needs timezone (CASSANDRA-11137)
   * Fix is_dense recalculation for Thrift-updated tables (CASSANDRA-11502)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5980b336/src/java/org/apache/cassandra/service/CassandraDaemon.java
--

[09/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.7

[samt]

Merge branch 'cassandra-3.0' into cassandra-3.7


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/68580c7a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/68580c7a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/68580c7a

Branch: refs/heads/trunk
Commit: 68580c7a81fb20ef76abdfbe5eaefcc07b97
Parents: 415bb56 5980b33
Author: Sam Tunnicliffe 
Authored: Wed May 4 18:02:40 2016 +0100
Committer: Sam Tunnicliffe 
Committed: Wed May 4 18:02:40 2016 +0100

--
 CHANGES.txt| 1 +
 src/java/org/apache/cassandra/service/CassandraDaemon.java | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/68580c7a/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/68580c7a/src/java/org/apache/cassandra/service/CassandraDaemon.java
--

[jira] [Updated] (CASSANDRA-11552) Reduce amount of logging calls from ColumnFamilyStore.selectAndReference

[Robert Stupp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Stupp updated CASSANDRA-11552:
-
Fix Version/s: 3.0.7
   3.7
   2.2.7
   2.1.15

> Reduce amount of logging calls from ColumnFamilyStore.selectAndReference
> 
>
> Key: CASSANDRA-11552
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11552
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Robert Stupp
>Assignee: Robert Stupp
> Fix For: 2.1.15, 2.2.7, 3.7, 3.0.7
>
>
> {{org.apache.cassandra.db.ColumnFamilyStore#selectAndReference}} logs two 
> messages at _info_ level "as fast as it can" if it waits for more than 100ms.
> The following code is executed in a while-true fashion in this case:
> {code}
> logger.info("Spinning trying to capture released readers {}", 
> released);
> logger.info("Spinning trying to capture all readers {}", 
> view.sstables);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11552) Reduce amount of logging calls from ColumnFamilyStore.selectAndReference

[Robert Stupp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Stupp updated CASSANDRA-11552:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Thanks!
Committed as aacd4526309693233f7f41e966e06ffcc0edf4d0 to 2.1 and merged to 2.2, 
3.0, 3.7 and trunk.

> Reduce amount of logging calls from ColumnFamilyStore.selectAndReference
> 
>
> Key: CASSANDRA-11552
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11552
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>
> {{org.apache.cassandra.db.ColumnFamilyStore#selectAndReference}} logs two 
> messages at _info_ level "as fast as it can" if it waits for more than 100ms.
> The following code is executed in a while-true fashion in this case:
> {code}
> logger.info("Spinning trying to capture released readers {}", 
> released);
> logger.info("Spinning trying to capture all readers {}", 
> view.sstables);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[11/15] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[snazy]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2e202d47
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2e202d47
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2e202d47

Branch: refs/heads/cassandra-3.7
Commit: 2e202d47d34a3a0b91670b7fe7bf0e1e35d20b1b
Parents: 7480202 3d73282
Author: Robert Stupp 
Authored: Wed May 4 18:44:52 2016 +0200
Committer: Robert Stupp 
Committed: Wed May 4 18:44:52 2016 +0200

--
 src/java/org/apache/cassandra/db/ColumnFamilyStore.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/2e202d47/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--

[02/15] cassandra git commit: Reduce amount of logging calls from ColumnFamilyStore.selectAndReference

[snazy]

Reduce amount of logging calls from ColumnFamilyStore.selectAndReference

patch by Robert Stupp; reviewed by Benjamin Lerer for CASSANDRA-11552


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/aacd4526
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/aacd4526
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/aacd4526

Branch: refs/heads/cassandra-2.2
Commit: aacd4526309693233f7f41e966e06ffcc0edf4d0
Parents: 5202147
Author: Robert Stupp 
Authored: Wed May 4 18:44:08 2016 +0200
Committer: Robert Stupp 
Committed: Wed May 4 18:44:08 2016 +0200

--
 src/java/org/apache/cassandra/db/ColumnFamilyStore.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/aacd4526/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index edc3564..b64d5de 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -1941,8 +1941,8 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 for (SSTableReader reader : view.sstables)
 if (reader.selfRef().globalCount() == 0)
 released.add(reader);
-logger.info("Spinning trying to capture released readers {}", 
released);
-logger.info("Spinning trying to capture all readers {}", 
view.sstables);
+NoSpamLogger.log(logger, NoSpamLogger.Level.WARN, 1, 
TimeUnit.SECONDS,
+ "Spinning trying to capture readers {}, 
released: {}, ", view.sstables, released);
 failingSince = System.nanoTime();
 }
 }

[10/15] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[snazy]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2e202d47
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2e202d47
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2e202d47

Branch: refs/heads/trunk
Commit: 2e202d47d34a3a0b91670b7fe7bf0e1e35d20b1b
Parents: 7480202 3d73282
Author: Robert Stupp 
Authored: Wed May 4 18:44:52 2016 +0200
Committer: Robert Stupp 
Committed: Wed May 4 18:44:52 2016 +0200

--
 src/java/org/apache/cassandra/db/ColumnFamilyStore.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/2e202d47/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--

[08/15] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2

[snazy]

Merge branch 'cassandra-2.1' into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3d732823
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3d732823
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3d732823

Branch: refs/heads/cassandra-3.0
Commit: 3d7328232552d14daaa9f99d842aa53d92fb51f9
Parents: b189a7f aacd452
Author: Robert Stupp 
Authored: Wed May 4 18:44:35 2016 +0200
Committer: Robert Stupp 
Committed: Wed May 4 18:44:35 2016 +0200

--
 src/java/org/apache/cassandra/db/ColumnFamilyStore.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3d732823/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--

[13/15] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.7

[snazy]

Merge branch 'cassandra-3.0' into cassandra-3.7


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/415bb569
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/415bb569
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/415bb569

Branch: refs/heads/cassandra-3.7
Commit: 415bb569d050c0e15ed8c13bff93602591388f7c
Parents: 9e5161b 2e202d4
Author: Robert Stupp 
Authored: Wed May 4 18:47:04 2016 +0200
Committer: Robert Stupp 
Committed: Wed May 4 18:47:04 2016 +0200

--
 src/java/org/apache/cassandra/db/ColumnFamilyStore.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/415bb569/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--

[12/15] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[snazy]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2e202d47
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2e202d47
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2e202d47

Branch: refs/heads/cassandra-3.0
Commit: 2e202d47d34a3a0b91670b7fe7bf0e1e35d20b1b
Parents: 7480202 3d73282
Author: Robert Stupp 
Authored: Wed May 4 18:44:52 2016 +0200
Committer: Robert Stupp 
Committed: Wed May 4 18:44:52 2016 +0200

--
 src/java/org/apache/cassandra/db/ColumnFamilyStore.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/2e202d47/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--

[05/15] cassandra git commit: Reduce amount of logging calls from ColumnFamilyStore.selectAndReference

[snazy]

Reduce amount of logging calls from ColumnFamilyStore.selectAndReference

patch by Robert Stupp; reviewed by Benjamin Lerer for CASSANDRA-11552


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/aacd4526
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/aacd4526
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/aacd4526

Branch: refs/heads/cassandra-3.7
Commit: aacd4526309693233f7f41e966e06ffcc0edf4d0
Parents: 5202147
Author: Robert Stupp 
Authored: Wed May 4 18:44:08 2016 +0200
Committer: Robert Stupp 
Committed: Wed May 4 18:44:08 2016 +0200

--
 src/java/org/apache/cassandra/db/ColumnFamilyStore.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/aacd4526/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index edc3564..b64d5de 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -1941,8 +1941,8 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 for (SSTableReader reader : view.sstables)
 if (reader.selfRef().globalCount() == 0)
 released.add(reader);
-logger.info("Spinning trying to capture released readers {}", 
released);
-logger.info("Spinning trying to capture all readers {}", 
view.sstables);
+NoSpamLogger.log(logger, NoSpamLogger.Level.WARN, 1, 
TimeUnit.SECONDS,
+ "Spinning trying to capture readers {}, 
released: {}, ", view.sstables, released);
 failingSince = System.nanoTime();
 }
 }

[03/15] cassandra git commit: Reduce amount of logging calls from ColumnFamilyStore.selectAndReference

[snazy]

Reduce amount of logging calls from ColumnFamilyStore.selectAndReference

patch by Robert Stupp; reviewed by Benjamin Lerer for CASSANDRA-11552


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/aacd4526
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/aacd4526
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/aacd4526

Branch: refs/heads/trunk
Commit: aacd4526309693233f7f41e966e06ffcc0edf4d0
Parents: 5202147
Author: Robert Stupp 
Authored: Wed May 4 18:44:08 2016 +0200
Committer: Robert Stupp 
Committed: Wed May 4 18:44:08 2016 +0200

--
 src/java/org/apache/cassandra/db/ColumnFamilyStore.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/aacd4526/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index edc3564..b64d5de 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -1941,8 +1941,8 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 for (SSTableReader reader : view.sstables)
 if (reader.selfRef().globalCount() == 0)
 released.add(reader);
-logger.info("Spinning trying to capture released readers {}", 
released);
-logger.info("Spinning trying to capture all readers {}", 
view.sstables);
+NoSpamLogger.log(logger, NoSpamLogger.Level.WARN, 1, 
TimeUnit.SECONDS,
+ "Spinning trying to capture readers {}, 
released: {}, ", view.sstables, released);
 failingSince = System.nanoTime();
 }
 }

[04/15] cassandra git commit: Reduce amount of logging calls from ColumnFamilyStore.selectAndReference

[snazy]

Reduce amount of logging calls from ColumnFamilyStore.selectAndReference

patch by Robert Stupp; reviewed by Benjamin Lerer for CASSANDRA-11552


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/aacd4526
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/aacd4526
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/aacd4526

Branch: refs/heads/cassandra-3.0
Commit: aacd4526309693233f7f41e966e06ffcc0edf4d0
Parents: 5202147
Author: Robert Stupp 
Authored: Wed May 4 18:44:08 2016 +0200
Committer: Robert Stupp 
Committed: Wed May 4 18:44:08 2016 +0200

--
 src/java/org/apache/cassandra/db/ColumnFamilyStore.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/aacd4526/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index edc3564..b64d5de 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -1941,8 +1941,8 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 for (SSTableReader reader : view.sstables)
 if (reader.selfRef().globalCount() == 0)
 released.add(reader);
-logger.info("Spinning trying to capture released readers {}", 
released);
-logger.info("Spinning trying to capture all readers {}", 
view.sstables);
+NoSpamLogger.log(logger, NoSpamLogger.Level.WARN, 1, 
TimeUnit.SECONDS,
+ "Spinning trying to capture readers {}, 
released: {}, ", view.sstables, released);
 failingSince = System.nanoTime();
 }
 }

[14/15] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.7

[snazy]

Merge branch 'cassandra-3.0' into cassandra-3.7


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/415bb569
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/415bb569
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/415bb569

Branch: refs/heads/trunk
Commit: 415bb569d050c0e15ed8c13bff93602591388f7c
Parents: 9e5161b 2e202d4
Author: Robert Stupp 
Authored: Wed May 4 18:47:04 2016 +0200
Committer: Robert Stupp 
Committed: Wed May 4 18:47:04 2016 +0200

--
 src/java/org/apache/cassandra/db/ColumnFamilyStore.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/415bb569/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--

[09/15] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2

[snazy]

Merge branch 'cassandra-2.1' into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3d732823
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3d732823
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3d732823

Branch: refs/heads/cassandra-3.7
Commit: 3d7328232552d14daaa9f99d842aa53d92fb51f9
Parents: b189a7f aacd452
Author: Robert Stupp 
Authored: Wed May 4 18:44:35 2016 +0200
Committer: Robert Stupp 
Committed: Wed May 4 18:44:35 2016 +0200

--
 src/java/org/apache/cassandra/db/ColumnFamilyStore.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3d732823/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--

[07/15] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2

[snazy]

Merge branch 'cassandra-2.1' into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3d732823
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3d732823
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3d732823

Branch: refs/heads/cassandra-2.2
Commit: 3d7328232552d14daaa9f99d842aa53d92fb51f9
Parents: b189a7f aacd452
Author: Robert Stupp 
Authored: Wed May 4 18:44:35 2016 +0200
Committer: Robert Stupp 
Committed: Wed May 4 18:44:35 2016 +0200

--
 src/java/org/apache/cassandra/db/ColumnFamilyStore.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3d732823/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--

[15/15] cassandra git commit: Merge branch 'cassandra-3.7' into trunk

[snazy]

Merge branch 'cassandra-3.7' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d6daf1b6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d6daf1b6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d6daf1b6

Branch: refs/heads/trunk
Commit: d6daf1b667b584cf30545cdb5cb3fac726a9cf4b
Parents: f5ae572 415bb56
Author: Robert Stupp 
Authored: Wed May 4 18:47:29 2016 +0200
Committer: Robert Stupp 
Committed: Wed May 4 18:47:29 2016 +0200

--
 src/java/org/apache/cassandra/db/ColumnFamilyStore.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--

[01/15] cassandra git commit: Reduce amount of logging calls from ColumnFamilyStore.selectAndReference

[snazy]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.1 5202147d9 -> aacd45263
  refs/heads/cassandra-2.2 b189a7f6c -> 3d7328232
  refs/heads/cassandra-3.0 7480202e5 -> 2e202d47d
  refs/heads/cassandra-3.7 9e5161b67 -> 415bb569d
  refs/heads/trunk f5ae57292 -> d6daf1b66


Reduce amount of logging calls from ColumnFamilyStore.selectAndReference

patch by Robert Stupp; reviewed by Benjamin Lerer for CASSANDRA-11552


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/aacd4526
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/aacd4526
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/aacd4526

Branch: refs/heads/cassandra-2.1
Commit: aacd4526309693233f7f41e966e06ffcc0edf4d0
Parents: 5202147
Author: Robert Stupp 
Authored: Wed May 4 18:44:08 2016 +0200
Committer: Robert Stupp 
Committed: Wed May 4 18:44:08 2016 +0200

--
 src/java/org/apache/cassandra/db/ColumnFamilyStore.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/aacd4526/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index edc3564..b64d5de 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -1941,8 +1941,8 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 for (SSTableReader reader : view.sstables)
 if (reader.selfRef().globalCount() == 0)
 released.add(reader);
-logger.info("Spinning trying to capture released readers {}", 
released);
-logger.info("Spinning trying to capture all readers {}", 
view.sstables);
+NoSpamLogger.log(logger, NoSpamLogger.Level.WARN, 1, 
TimeUnit.SECONDS,
+ "Spinning trying to capture readers {}, 
released: {}, ", view.sstables, released);
 failingSince = System.nanoTime();
 }
 }

[06/15] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2

[snazy]

Merge branch 'cassandra-2.1' into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3d732823
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3d732823
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3d732823

Branch: refs/heads/trunk
Commit: 3d7328232552d14daaa9f99d842aa53d92fb51f9
Parents: b189a7f aacd452
Author: Robert Stupp 
Authored: Wed May 4 18:44:35 2016 +0200
Committer: Robert Stupp 
Committed: Wed May 4 18:44:35 2016 +0200

--
 src/java/org/apache/cassandra/db/ColumnFamilyStore.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3d732823/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--

[jira] [Updated] (CASSANDRA-10309) Avoid always looking up column type

[T Jake Luciani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

T Jake Luciani updated CASSANDRA-10309:
---
Assignee: Carl Yeksigian

> Avoid always looking up column type
> ---
>
> Key: CASSANDRA-10309
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10309
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: T Jake Luciani
>Assignee: Carl Yeksigian
>Priority: Minor
>  Labels: perfomance
> Fix For: 3.x
>
>
> Doing some read profiling I noticed we always seem to look up the type of a 
> column from the schema metadata when we have the type already in the column 
> class.
> This one simple change to SerializationHeader improves read performance 
> non-trivially.
> https://github.com/tjake/cassandra/commit/69b94c389b3f36aa035ac4619fd22d1f62ea80b2
> http://cstar.datastax.com/graph?stats=3fb1ced4-58c7-11e5-9faf-42010af0688f=op_rate=2_read=1_aggregates=true=0=357.94=0=157416.6
> I assume we are looking this up to deal with schema changes. But I'm sure 
> there is a more performant way of doing this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8911) Consider Mutation-based Repairs

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270947#comment-15270947
 ] 

Paulo Motta commented on CASSANDRA-8911:


bq.  the nice thing about this is that it requires no coordination or 
scheduling at all - all nodes should run this (slowly) all the time

is the idea to have one thread per table running continuously, or a fixed-size 
pool for all tables? If the second option we may still need to prioritize which 
table should run MBR next.

> Consider Mutation-based Repairs
> ---
>
> Key: CASSANDRA-8911
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8911
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Tyler Hobbs
>Assignee: Marcus Eriksson
> Fix For: 3.x
>
>
> We should consider a mutation-based repair to replace the existing streaming 
> repair.  While we're at it, we could do away with a lot of the complexity 
> around merkle trees.
> I have not planned this out in detail, but here's roughly what I'm thinking:
>  * Instead of building an entire merkle tree up front, just send the "leaves" 
> one-by-one.  Instead of dealing with token ranges, make the leaves primary 
> key ranges.  The PK ranges would need to be contiguous, so that the start of 
> each range would match the end of the previous range. (The first and last 
> leaves would need to be open-ended on one end of the PK range.) This would be 
> similar to doing a read with paging.
>  * Once one page of data is read, compute a hash of it and send it to the 
> other replicas along with the PK range that it covers and a row count.
>  * When the replicas receive the hash, the perform a read over the same PK 
> range (using a LIMIT of the row count + 1) and compare hashes (unless the row 
> counts don't match, in which case this can be skipped).
>  * If there is a mismatch, the replica will send a mutation covering that 
> page's worth of data (ignoring the row count this time) to the source node.
> Here are the advantages that I can think of:
>  * With the current repair behavior of streaming, vnode-enabled clusters may 
> need to stream hundreds of small SSTables.  This results in increased compact
> ion load on the receiving node.  With the mutation-based approach, memtables 
> would naturally merge these.
>  * It's simple to throttle.  For example, you could give a number of rows/sec 
> that should be repaired.
>  * It's easy to see what PK range has been repaired so far.  This could make 
> it simpler to resume a repair that fails midway.
>  * Inconsistencies start to be repaired almost right away.
>  * Less special code \(?\)
>  * Wide partitions are no longer a problem.
> There are a few problems I can think of:
>  * Counters.  I don't know if this can be made safe, or if they need to be 
> skipped.
>  * To support incremental repair, we need to be able to read from only 
> repaired sstables.  Probably not too difficult to do.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8911) Consider Mutation-based Repairs

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270926#comment-15270926
 ] 

Marcus Eriksson commented on CASSANDRA-8911:


thanks [~pauloricardomg] it is not ready for review yet so finish your other 
things first :)

Regarding CASSANDRA-10070 the nice thing about this is that it requires no 
coordination or scheduling at all - all nodes should run this (slowly) all the 
time. What we need to figure out is how to automatically size the repair pages 
and rate limiting/error handling etc.

> Consider Mutation-based Repairs
> ---
>
> Key: CASSANDRA-8911
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8911
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Tyler Hobbs
>Assignee: Marcus Eriksson
> Fix For: 3.x
>
>
> We should consider a mutation-based repair to replace the existing streaming 
> repair.  While we're at it, we could do away with a lot of the complexity 
> around merkle trees.
> I have not planned this out in detail, but here's roughly what I'm thinking:
>  * Instead of building an entire merkle tree up front, just send the "leaves" 
> one-by-one.  Instead of dealing with token ranges, make the leaves primary 
> key ranges.  The PK ranges would need to be contiguous, so that the start of 
> each range would match the end of the previous range. (The first and last 
> leaves would need to be open-ended on one end of the PK range.) This would be 
> similar to doing a read with paging.
>  * Once one page of data is read, compute a hash of it and send it to the 
> other replicas along with the PK range that it covers and a row count.
>  * When the replicas receive the hash, the perform a read over the same PK 
> range (using a LIMIT of the row count + 1) and compare hashes (unless the row 
> counts don't match, in which case this can be skipped).
>  * If there is a mismatch, the replica will send a mutation covering that 
> page's worth of data (ignoring the row count this time) to the source node.
> Here are the advantages that I can think of:
>  * With the current repair behavior of streaming, vnode-enabled clusters may 
> need to stream hundreds of small SSTables.  This results in increased compact
> ion load on the receiving node.  With the mutation-based approach, memtables 
> would naturally merge these.
>  * It's simple to throttle.  For example, you could give a number of rows/sec 
> that should be repaired.
>  * It's easy to see what PK range has been repaired so far.  This could make 
> it simpler to resume a repair that fails midway.
>  * Inconsistencies start to be repaired almost right away.
>  * Less special code \(?\)
>  * Wide partitions are no longer a problem.
> There are a few problems I can think of:
>  * Counters.  I don't know if this can be made safe, or if they need to be 
> skipped.
>  * To support incremental repair, we need to be able to read from only 
> repaired sstables.  Probably not too difficult to do.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9766) Faster Streaming

[T Jake Luciani (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270910#comment-15270910
 ] 

T Jake Luciani commented on CASSANDRA-9766:
---

I've made those changes, squashed and restarted tests. I'll commit assuming a 
clean CI.

The final streaming improvement is 25%

> Faster Streaming
> 
>
> Key: CASSANDRA-9766
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9766
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Streaming and Messaging
> Environment: Cassandra 2.1.2. more details in the pdf attached 
>Reporter: Alexei K
>Assignee: T Jake Luciani
>  Labels: performance
> Fix For: 3.x
>
> Attachments: problem.pdf
>
>
> I have a cluster in Amazon cloud , its described in detail in the attachment. 
> What I've noticed is that we during bootstrap we never go above 12MB/sec 
> transmission speeds and also those speeds flat line almost like we're hitting 
> some sort of a limit ( this remains true for other tests that I've ran) 
> however during the repair we see much higher,variable sending rates. I've 
> provided network charts in the attachment as well . Is there an explanation 
> for this? Is something wrong with my configuration, or is it a possible bug?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10787) OutOfMemoryError after few hours from node restart

[Piotr Westfalewicz (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270908#comment-15270908
 ] 

Piotr Westfalewicz commented on CASSANDRA-10787:


[~ifesdjeen], I've deleted those big, potentially problematic sstables and the 
problem is gone. Since then I've changed compaction strategy to create smaller 
sstables and the problem never happened again. Hence, it's no longer possible 
to recreate the issue and it can be closed.

> OutOfMemoryError after few hours from node restart
> --
>
> Key: CASSANDRA-10787
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10787
> Project: Cassandra
>  Issue Type: Bug
> Environment: Amazon DataStax Auto-Clustering AMI 2.6.3-1404-pv
> on 2x m1.large instances (2 vCPU, 64-bit, 7.5GB RAM, Raid0 2x420GB Disk)
> [cqlsh 5.0.1 | Cassandra 2.2.3 | CQL spec 3.3.1 | Native protocol v4]
> RF=3
>Reporter: Piotr Westfalewicz
> Fix For: 2.2.x
>
> Attachments: case2_debuglog_head.txt, case2_debuglog_tail.txt, 
> case2_systemlog.txt, case3_debuglog_tail.txt, case3_systemlog_tail.txt, 
> case4_debuglog_tail.txt, case4_systemlog.txt, case5_debuglog.txt, 
> case5_systemlog.txt
>
>
> Cassandra Cluster was operating flawessly for around 3 months. Lately I've 
> got a critical problem with it - after few hours of running clients are 
> disconnected permanently (that may be Datastax C# Driver problem, though), 
> however few more hours later (with smaller load), on all 2 nodes there is 
> thrown an exception (details in files):
> bq. java.lang.OutOfMemoryError: Java heap space
> Cases description:
> Case 2 (heavy load):
> - 2015-11-26 16:09:40,834 Restarted all nodes in cassandra cluster
>   - 2015-11-26 17:03:46,774 First client disconnected permanently
>   - 2015-11-26 22:17:02,327 Node shutdown
>   Case 3 (unknown load, different node):
>   - 2015-11-26 02:19:49,585 Node shutdown (visible only in 
> systemlog, I don't know why not in debug log)
>   Case 4 (low load):
>   - 2015-11-27 13:00:24,994 Node restart
>   - 2015-11-27 22:26:56,131 Node shutdown
> Is that a software issue or I am using too weak Amazon instances? If so, how 
> can the required amount of memory be calculated?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-9766) Faster Streaming

[T Jake Luciani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

T Jake Luciani updated CASSANDRA-9766:
--
Summary: Faster Streaming  (was: Bootstrap outgoing streaming speeds are 
much slower than during repair)

> Faster Streaming
> 
>
> Key: CASSANDRA-9766
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9766
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Streaming and Messaging
> Environment: Cassandra 2.1.2. more details in the pdf attached 
>Reporter: Alexei K
>Assignee: T Jake Luciani
>  Labels: performance
> Fix For: 3.x
>
> Attachments: problem.pdf
>
>
> I have a cluster in Amazon cloud , its described in detail in the attachment. 
> What I've noticed is that we during bootstrap we never go above 12MB/sec 
> transmission speeds and also those speeds flat line almost like we're hitting 
> some sort of a limit ( this remains true for other tests that I've ran) 
> however during the repair we see much higher,variable sending rates. I've 
> provided network charts in the attachment as well . Is there an explanation 
> for this? Is something wrong with my configuration, or is it a possible bug?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8911) Consider Mutation-based Repairs

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270884#comment-15270884
 ] 

Paulo Motta commented on CASSANDRA-8911:


I'd be interested in reviewing this, but since it's quite a big/relevant change 
another reviewer would probably be welcome (specially in the read-path side, 
since I'm not totally comfortable with it). Have some priority things on my 
plate right now, so will have a better look and post comments next week.

For
bq.* continuously run this
we could probably rely on CASSANDRA-10070, which should provide facilities for 
automatizing repairs independent of implementation.

> Consider Mutation-based Repairs
> ---
>
> Key: CASSANDRA-8911
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8911
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Tyler Hobbs
>Assignee: Marcus Eriksson
> Fix For: 3.x
>
>
> We should consider a mutation-based repair to replace the existing streaming 
> repair.  While we're at it, we could do away with a lot of the complexity 
> around merkle trees.
> I have not planned this out in detail, but here's roughly what I'm thinking:
>  * Instead of building an entire merkle tree up front, just send the "leaves" 
> one-by-one.  Instead of dealing with token ranges, make the leaves primary 
> key ranges.  The PK ranges would need to be contiguous, so that the start of 
> each range would match the end of the previous range. (The first and last 
> leaves would need to be open-ended on one end of the PK range.) This would be 
> similar to doing a read with paging.
>  * Once one page of data is read, compute a hash of it and send it to the 
> other replicas along with the PK range that it covers and a row count.
>  * When the replicas receive the hash, the perform a read over the same PK 
> range (using a LIMIT of the row count + 1) and compare hashes (unless the row 
> counts don't match, in which case this can be skipped).
>  * If there is a mismatch, the replica will send a mutation covering that 
> page's worth of data (ignoring the row count this time) to the source node.
> Here are the advantages that I can think of:
>  * With the current repair behavior of streaming, vnode-enabled clusters may 
> need to stream hundreds of small SSTables.  This results in increased compact
> ion load on the receiving node.  With the mutation-based approach, memtables 
> would naturally merge these.
>  * It's simple to throttle.  For example, you could give a number of rows/sec 
> that should be repaired.
>  * It's easy to see what PK range has been repaired so far.  This could make 
> it simpler to resume a repair that fails midway.
>  * Inconsistencies start to be repaired almost right away.
>  * Less special code \(?\)
>  * Wide partitions are no longer a problem.
> There are a few problems I can think of:
>  * Counters.  I don't know if this can be made safe, or if they need to be 
> skipped.
>  * To support incremental repair, we need to be able to read from only 
> repaired sstables.  Probably not too difficult to do.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11552) Reduce amount of logging calls from ColumnFamilyStore.selectAndReference

[Benjamin Lerer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270874#comment-15270874
 ] 

Benjamin Lerer commented on CASSANDRA-11552:


+1

> Reduce amount of logging calls from ColumnFamilyStore.selectAndReference
> 
>
> Key: CASSANDRA-11552
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11552
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>
> {{org.apache.cassandra.db.ColumnFamilyStore#selectAndReference}} logs two 
> messages at _info_ level "as fast as it can" if it waits for more than 100ms.
> The following code is executed in a while-true fashion in this case:
> {code}
> logger.info("Spinning trying to capture released readers {}", 
> released);
> logger.info("Spinning trying to capture all readers {}", 
> view.sstables);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-7056) Add RAMP transactions

[Cyril Scetbon (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-7056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270846#comment-15270846
 ] 

Cyril Scetbon commented on CASSANDRA-7056:
--

[~iamaleksey] Thanks

> Add RAMP transactions
> -
>
> Key: CASSANDRA-7056
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7056
> Project: Cassandra
>  Issue Type: Wish
>Reporter: Tupshin Harper
>Priority: Minor
> Fix For: 3.x
>
>
> We should take a look at 
> [RAMP|http://www.bailis.org/blog/scalable-atomic-visibility-with-ramp-transactions/]
>  transactions, and figure out if they can be used to provide more efficient 
> LWT (or LWT-like) operations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8911) Consider Mutation-based Repairs

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270842#comment-15270842
 ] 

Marcus Eriksson commented on CASSANDRA-8911:


No, it still has a few problems - first, we wont be able to eliminate as many 
sstables during queries based on the min/max clustering values since the all 
cover a large time span, and second, since "all" sstables will contain very old 
data, we can't do the "drop this sstable as it only contains expired 
data"-optimization during compaction since that old data might still be newer 
than the data in other sstables.

> Consider Mutation-based Repairs
> ---
>
> Key: CASSANDRA-8911
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8911
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Tyler Hobbs
>Assignee: Marcus Eriksson
> Fix For: 3.x
>
>
> We should consider a mutation-based repair to replace the existing streaming 
> repair.  While we're at it, we could do away with a lot of the complexity 
> around merkle trees.
> I have not planned this out in detail, but here's roughly what I'm thinking:
>  * Instead of building an entire merkle tree up front, just send the "leaves" 
> one-by-one.  Instead of dealing with token ranges, make the leaves primary 
> key ranges.  The PK ranges would need to be contiguous, so that the start of 
> each range would match the end of the previous range. (The first and last 
> leaves would need to be open-ended on one end of the PK range.) This would be 
> similar to doing a read with paging.
>  * Once one page of data is read, compute a hash of it and send it to the 
> other replicas along with the PK range that it covers and a row count.
>  * When the replicas receive the hash, the perform a read over the same PK 
> range (using a LIMIT of the row count + 1) and compare hashes (unless the row 
> counts don't match, in which case this can be skipped).
>  * If there is a mismatch, the replica will send a mutation covering that 
> page's worth of data (ignoring the row count this time) to the source node.
> Here are the advantages that I can think of:
>  * With the current repair behavior of streaming, vnode-enabled clusters may 
> need to stream hundreds of small SSTables.  This results in increased compact
> ion load on the receiving node.  With the mutation-based approach, memtables 
> would naturally merge these.
>  * It's simple to throttle.  For example, you could give a number of rows/sec 
> that should be repaired.
>  * It's easy to see what PK range has been repaired so far.  This could make 
> it simpler to resume a repair that fails midway.
>  * Inconsistencies start to be repaired almost right away.
>  * Less special code \(?\)
>  * Wide partitions are no longer a problem.
> There are a few problems I can think of:
>  * Counters.  I don't know if this can be made safe, or if they need to be 
> skipped.
>  * To support incremental repair, we need to be able to read from only 
> repaired sstables.  Probably not too difficult to do.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10070) Automatic repair scheduling

[Marcus Olsson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270836#comment-15270836
 ] 

Marcus Olsson commented on CASSANDRA-10070:
---

@[~jbellis]
bq. How closely does this match the design doc from February? Is it worth 
posting an updated design for those of us joining late?
I'd say there have been enough changes for it to be a good idea to update the 
document, so I'll work on that! :)

> Automatic repair scheduling
> ---
>
> Key: CASSANDRA-10070
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10070
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Marcus Olsson
>Assignee: Marcus Olsson
>Priority: Minor
> Fix For: 3.x
>
> Attachments: Distributed Repair Scheduling.doc
>
>
> Scheduling and running repairs in a Cassandra cluster is most often a 
> required task, but this can both be hard for new users and it also requires a 
> bit of manual configuration. There are good tools out there that can be used 
> to simplify things, but wouldn't this be a good feature to have inside of 
> Cassandra? To automatically schedule and run repairs, so that when you start 
> up your cluster it basically maintains itself in terms of normal 
> anti-entropy, with the possibility for manual configuration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11678) cassandra crush when enable hints_compression

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-11678:
--
Reviewer: Aleksey Yeschenko

> cassandra crush when enable hints_compression
> -
>
> Key: CASSANDRA-11678
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11678
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core, Local Write-Read Paths
> Environment: Centos 7
>Reporter: Weijian Lin
>Assignee: Blake Eggleston
>Priority: Critical
>
> When I enable hints_compression and set the compression class to
> LZ4Compressor，the
> cassandra (v3.05, V3.5.0) will crush。That is a bug, or any conf is wrong?
> *Exception in V 3.5.0 *
> {code}
> ERROR [HintsDispatcher:2] 2016-04-26 15:02:56,970
> HintsDispatchExecutor.java:225 - Failed to dispatch hints file
> abc4dda2-b551-427e-bb0b-e383d4a392e1-1461654138963-1.hints: file is
> corrupted ({})
> org.apache.cassandra.io.FSReadError: java.io.EOFException
> at 
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:284)
>  ~[apache-cassandra-3.5.0.jar:3.5.0]
> at 
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:254)
>  ~[apache-cassandra-3.5.0.jar:3.5.0]
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at 
> org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156)
>  ~[apache-cassandra-3.5.0.jar:3.5.0]
> at 
> org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137)
>  ~[apache-cassandra-3.5.0.jar:3.5.0]
> at 
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119) 
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at 
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:91) 
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at 
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:259)
>  [apache-cassandra-3.5.0.jar:3.5.0]
> at 
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:242)
>  [apache-cassandra-3.5.0.jar:3.5.0]
> at 
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:220)
>  [apache-cassandra-3.5.0.jar:3.5.0]
> at 
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:199)
>  [apache-cassandra-3.5.0.jar:3.5.0]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_65]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_65]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_65]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> Caused by: java.io.EOFException: null
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:146)
>  ~[apache-cassandra-3.5.0.jar:3.5.0]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readPrimitiveSlowly(RebufferingInputStream.java:108)
>  ~[apache-cassandra-3.5.0.jar:3.5.0]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readInt(RebufferingInputStream.java:188)
>  ~[apache-cassandra-3.5.0.jar:3.5.0]
> at 
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNextInternal(HintsReader.java:297)
>  ~[apache-cassandra-3.5.0.jar:3.5.0]
> at 
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:280)
>  ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 15 common frames omitted
> {code}
> *Exception in V 3.0.5 *
> {code}
> ERROR [HintsDispatcher:2] 2016-04-26 15:54:46,294
> HintsDispatchExecutor.java:225 - Failed to dispatch hints file
> 8603be13-6878-4de3-8bc3-a7a7146b0376-1461657251205-1.hints: file is
> corrupted ({})
> org.apache.cassandra.io.FSReadError: java.io.EOFException
> at 
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:282)
>  ~[apache-cassandra-3.0.5.jar:3.0.5]
> at 
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:252)
>  ~[apache-cassandra-3.0.5.jar:3.0.5]
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at 
> org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156)
>  ~[apache-cassandra-3.0.5.jar:3.0.5]
> at 
> org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137)
>  ~[apache-cassandra-3.0.5.jar:3.0.5]
> at 
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119) 
> 

[jira] [Commented] (CASSANDRA-11702) dtest failure in pushed_notifications_test.TestPushedNotifications.restart_node_localhost_test

[Joel Knighton (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270835#comment-15270835
 ] 

Joel Knighton commented on CASSANDRA-11702:
---

For another datapoint, same failure on a 2.2 branch that didn't touch related 
functionality 
[here|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-10962-2.2-dtest/2/testReport/junit/pushed_notifications_test/TestPushedNotifications/restart_node_localhost_test/].

> dtest failure in 
> pushed_notifications_test.TestPushedNotifications.restart_node_localhost_test
> --
>
> Key: CASSANDRA-11702
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11702
> Project: Cassandra
>  Issue Type: Test
>Reporter: Russ Hatch
>Assignee: DS Test Eng
>  Labels: dtest
>
> from offheap test job, failure. looks like it could be a routine timeout, but 
> I think I saw the exact same timeout for this test on another job.
> {noformat}
> ('Unable to connect to any servers', {})
> {noformat}
> http://cassci.datastax.com/job/trunk_offheap_dtest/178/testReport/pushed_notifications_test/TestPushedNotifications/restart_node_localhost_test
> Failed on CassCI build trunk_offheap_dtest #178



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-9861) When forcibly exiting due to OOM, we should produce a heap dump

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer updated CASSANDRA-9861:
--
Fix Version/s: (was: 2.2.x)
   3.0.7
   3.7
   2.2.7

> When forcibly exiting due to OOM, we should produce a heap dump
> ---
>
> Key: CASSANDRA-9861
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9861
> Project: Cassandra
>  Issue Type: Bug
>  Components: Lifecycle
>Reporter: Benedict
>Assignee: Benjamin Lerer
>Priority: Minor
>  Labels: lhf
> Fix For: 2.2.7, 3.7, 3.0.7
>
> Attachments: 9861-2.2-V2.txt, 9861-2.2.txt
>
>
> CASSANDRA-7507 introduced earlier termination on encountering an OOM, due to 
> lack of certainty about system state. However a side effect is that we never 
> produce heap dumps on OOM. We should ideally try to produce one forcibly 
> before exiting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-9861) When forcibly exiting due to OOM, we should produce a heap dump

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer updated CASSANDRA-9861:
--
Resolution: Fixed
Status: Resolved  (was: Ready to Commit)

Committed into 2.2 at b189a7f6c955908d8b79d2b4104562a38ea62979 and merged into 
3.0, 3.7 and trunk 

> When forcibly exiting due to OOM, we should produce a heap dump
> ---
>
> Key: CASSANDRA-9861
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9861
> Project: Cassandra
>  Issue Type: Bug
>  Components: Lifecycle
>Reporter: Benedict
>Assignee: Benjamin Lerer
>Priority: Minor
>  Labels: lhf
> Fix For: 2.2.x
>
> Attachments: 9861-2.2-V2.txt, 9861-2.2.txt
>
>
> CASSANDRA-7507 introduced earlier termination on encountering an OOM, due to 
> lack of certainty about system state. However a side effect is that we never 
> produce heap dumps on OOM. We should ideally try to produce one forcibly 
> before exiting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9861) When forcibly exiting due to OOM, we should produce a heap dump

[Benjamin Lerer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270821#comment-15270821
 ] 

Benjamin Lerer commented on CASSANDRA-9861:
---

Thanks for the review.

> When forcibly exiting due to OOM, we should produce a heap dump
> ---
>
> Key: CASSANDRA-9861
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9861
> Project: Cassandra
>  Issue Type: Bug
>  Components: Lifecycle
>Reporter: Benedict
>Assignee: Benjamin Lerer
>Priority: Minor
>  Labels: lhf
> Fix For: 2.2.x
>
> Attachments: 9861-2.2-V2.txt, 9861-2.2.txt
>
>
> CASSANDRA-7507 introduced earlier termination on encountering an OOM, due to 
> lack of certainty about system state. However a side effect is that we never 
> produce heap dumps on OOM. We should ideally try to produce one forcibly 
> before exiting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[4/4] cassandra git commit: Merge branch cassandra-3.7 into trunk

[blerer]

Merge branch cassandra-3.7 into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f5ae5729
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f5ae5729
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f5ae5729

Branch: refs/heads/trunk
Commit: f5ae572929dd6a44dbfa9217b2b5636b327d693b
Parents: 17d30c9 9e5161b
Author: Benjamin Lerer 
Authored: Wed May 4 17:20:02 2016 +0200
Committer: Benjamin Lerer 
Committed: Wed May 4 17:20:14 2016 +0200

--
 CHANGES.txt |   1 +
 .../org/apache/cassandra/utils/HeapUtils.java   | 205 +++
 .../cassandra/utils/JVMStabilityInspector.java  |   3 +
 3 files changed, 209 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f5ae5729/CHANGES.txt
--

[2/4] cassandra git commit: Merge branch cassandra-2.2 into cassandra-3.0

[blerer]

Merge branch cassandra-2.2 into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7480202e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7480202e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7480202e

Branch: refs/heads/trunk
Commit: 7480202e540f86ae4ff02c9a67253d90ad8dde29
Parents: c7d216e b189a7f
Author: Benjamin Lerer 
Authored: Wed May 4 17:17:08 2016 +0200
Committer: Benjamin Lerer 
Committed: Wed May 4 17:17:08 2016 +0200

--
 CHANGES.txt |   1 +
 .../org/apache/cassandra/utils/HeapUtils.java   | 205 +++
 .../cassandra/utils/JVMStabilityInspector.java  |   5 +-
 3 files changed, 209 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/7480202e/CHANGES.txt
--
diff --cc CHANGES.txt
index f947568,7c7295d..31e46d9
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,24 -1,6 +1,25 @@@
 -2.2.7
 +3.0.6
 + * Disallow creating view with a static column (CASSANDRA-11602)
 + * Reduce the amount of object allocations caused by the getFunctions methods 
(CASSANDRA-11593)
 + * Potential error replaying commitlog with smallint/tinyint/date/time types 
(CASSANDRA-11618)
 + * Fix queries with filtering on counter columns (CASSANDRA-11629)
 + * Improve tombstone printing in sstabledump (CASSANDRA-11655)
 + * Fix paging for range queries where all clustering columns are specified 
(CASSANDRA-11669)
 + * Don't require HEAP_NEW_SIZE to be set when using G1 (CASSANDRA-11600)
 + * Fix sstabledump not showing cells after tombstone marker (CASSANDRA-11654)
 + * Ignore all LocalStrategy keyspaces for streaming and other related
 +   operations (CASSANDRA-11627)
 + * Ensure columnfilter covers indexed columns for thrift 2i queries 
(CASSANDRA-11523)
 + * Only open one sstable scanner per sstable (CASSANDRA-11412)
 + * Option to specify ProtocolVersion in cassandra-stress (CASSANDRA-11410)
 + * ArithmeticException in avgFunctionForDecimal (CASSANDRA-11485)
 + * LogAwareFileLister should only use OLD sstable files in current folder to 
determine disk consistency (CASSANDRA-11470)
 + * Notify indexers of expired rows during compaction (CASSANDRA-11329)
 + * Properly respond with ProtocolError when a v1/v2 native protocol
 +   header is received (CASSANDRA-11464)
 + * Validate that num_tokens and initial_token are consistent with one another 
(CASSANDRA-10120)
 +Merged from 2.2:
+  * Produce a heap dump when exiting on OOM (CASSANDRA-9861)
 - * Avoid read repairing purgeable tombstones on range slices (CASSANDRA-11427)
   * Restore ability to filter on clustering columns when using a 2i 
(CASSANDRA-11510)
   * JSON datetime formatting needs timezone (CASSANDRA-11137)
   * Fix is_dense recalculation for Thrift-updated tables (CASSANDRA-11502)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/7480202e/src/java/org/apache/cassandra/utils/JVMStabilityInspector.java
--

[1/4] cassandra git commit: Produce a heap dump when exiting on OO a heap dump when exiting on OOM

[blerer]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 17d30c9e5 -> f5ae57292


Produce a heap dump when exiting on OO a heap dump when exiting on OOM

patch by Benjamin Lerer; reviewed by Aleksey Yeschenko for CASSANDRA-9861


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b189a7f6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b189a7f6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b189a7f6

Branch: refs/heads/trunk
Commit: b189a7f6c955908d8b79d2b4104562a38ea62979
Parents: 8bf453a
Author: Benjamin Lerer 
Authored: Thu Mar 31 15:23:33 2016 +0200
Committer: Benjamin Lerer 
Committed: Wed May 4 17:10:58 2016 +0200

--
 CHANGES.txt |   1 +
 .../org/apache/cassandra/utils/HeapUtils.java   | 205 +++
 .../cassandra/utils/JVMStabilityInspector.java  |   5 +-
 3 files changed, 209 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b189a7f6/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 19e1afe..7c7295d 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.7
+ * Produce a heap dump when exiting on OOM (CASSANDRA-9861)
  * Avoid read repairing purgeable tombstones on range slices (CASSANDRA-11427)
  * Restore ability to filter on clustering columns when using a 2i 
(CASSANDRA-11510)
  * JSON datetime formatting needs timezone (CASSANDRA-11137)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b189a7f6/src/java/org/apache/cassandra/utils/HeapUtils.java
--
diff --git a/src/java/org/apache/cassandra/utils/HeapUtils.java 
b/src/java/org/apache/cassandra/utils/HeapUtils.java
new file mode 100644
index 000..bfc8a0b
--- /dev/null
+++ b/src/java/org/apache/cassandra/utils/HeapUtils.java
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.utils;
+
+import java.io.*;
+import java.lang.management.ManagementFactory;
+import java.lang.management.RuntimeMXBean;
+import java.nio.file.FileSystems;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.commons.lang3.text.StrBuilder;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Utility to generate heap dumps.
+ *
+ */
+public final class HeapUtils
+{
+private static final Logger logger = 
LoggerFactory.getLogger(HeapUtils.class);
+
+/**
+ * Generates a HEAP dump in the directory specified by the 
HeapDumpPath JVM option
+ * or in the CASSANDRA_HOME directory.
+ */
+public static void generateHeapDump()
+{
+Long processId = getProcessId();
+if (processId == null)
+{
+logger.error("The process ID could not be retrieved. Skipping heap 
dump generation.");
+return;
+}
+
+String heapDumpPath = getHeapDumpPathOption();
+if (heapDumpPath == null)
+{
+String cassandraHome = System.getenv("CASSANDRA_HOME");
+if (cassandraHome == null)
+{
+return;
+}
+
+heapDumpPath = cassandraHome;
+}
+
+Path dumpPath = FileSystems.getDefault().getPath(heapDumpPath);
+if (Files.isDirectory(dumpPath))
+{
+dumpPath = dumpPath.resolve("java_pid" + processId + ".hprof");
+}
+
+String jmapPath = getJmapPath();
+
+// The jmap file could not be found. In this case let's default to 
jmap in the hope that it is in the path.
+String jmapCommand = jmapPath == null ? "jmap" : jmapPath;
+
+String[] dumpCommands = new String[] {jmapCommand,
+  "-dump:format=b,file=" + 
dumpPath,
+  processId.toString()};
+
+// 

[3/4] cassandra git commit: Merge branch cassandra-3.0 into cassandra-3.7

[blerer]

Merge branch cassandra-3.0 into cassandra-3.7


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9e5161b6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9e5161b6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9e5161b6

Branch: refs/heads/trunk
Commit: 9e5161b6752611515e3e349a7bc4d3f76f8d15d4
Parents: 54d7fc4 7480202
Author: Benjamin Lerer 
Authored: Wed May 4 17:18:18 2016 +0200
Committer: Benjamin Lerer 
Committed: Wed May 4 17:18:32 2016 +0200

--
 CHANGES.txt |   1 +
 .../org/apache/cassandra/utils/HeapUtils.java   | 205 +++
 .../cassandra/utils/JVMStabilityInspector.java  |   3 +
 3 files changed, 209 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/9e5161b6/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/9e5161b6/src/java/org/apache/cassandra/utils/JVMStabilityInspector.java
--

[2/3] cassandra git commit: Merge branch cassandra-2.2 into cassandra-3.0

[blerer]

Merge branch cassandra-2.2 into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7480202e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7480202e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7480202e

Branch: refs/heads/cassandra-3.7
Commit: 7480202e540f86ae4ff02c9a67253d90ad8dde29
Parents: c7d216e b189a7f
Author: Benjamin Lerer 
Authored: Wed May 4 17:17:08 2016 +0200
Committer: Benjamin Lerer 
Committed: Wed May 4 17:17:08 2016 +0200

--
 CHANGES.txt |   1 +
 .../org/apache/cassandra/utils/HeapUtils.java   | 205 +++
 .../cassandra/utils/JVMStabilityInspector.java  |   5 +-
 3 files changed, 209 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/7480202e/CHANGES.txt
--
diff --cc CHANGES.txt
index f947568,7c7295d..31e46d9
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,24 -1,6 +1,25 @@@
 -2.2.7
 +3.0.6
 + * Disallow creating view with a static column (CASSANDRA-11602)
 + * Reduce the amount of object allocations caused by the getFunctions methods 
(CASSANDRA-11593)
 + * Potential error replaying commitlog with smallint/tinyint/date/time types 
(CASSANDRA-11618)
 + * Fix queries with filtering on counter columns (CASSANDRA-11629)
 + * Improve tombstone printing in sstabledump (CASSANDRA-11655)
 + * Fix paging for range queries where all clustering columns are specified 
(CASSANDRA-11669)
 + * Don't require HEAP_NEW_SIZE to be set when using G1 (CASSANDRA-11600)
 + * Fix sstabledump not showing cells after tombstone marker (CASSANDRA-11654)
 + * Ignore all LocalStrategy keyspaces for streaming and other related
 +   operations (CASSANDRA-11627)
 + * Ensure columnfilter covers indexed columns for thrift 2i queries 
(CASSANDRA-11523)
 + * Only open one sstable scanner per sstable (CASSANDRA-11412)
 + * Option to specify ProtocolVersion in cassandra-stress (CASSANDRA-11410)
 + * ArithmeticException in avgFunctionForDecimal (CASSANDRA-11485)
 + * LogAwareFileLister should only use OLD sstable files in current folder to 
determine disk consistency (CASSANDRA-11470)
 + * Notify indexers of expired rows during compaction (CASSANDRA-11329)
 + * Properly respond with ProtocolError when a v1/v2 native protocol
 +   header is received (CASSANDRA-11464)
 + * Validate that num_tokens and initial_token are consistent with one another 
(CASSANDRA-10120)
 +Merged from 2.2:
+  * Produce a heap dump when exiting on OOM (CASSANDRA-9861)
 - * Avoid read repairing purgeable tombstones on range slices (CASSANDRA-11427)
   * Restore ability to filter on clustering columns when using a 2i 
(CASSANDRA-11510)
   * JSON datetime formatting needs timezone (CASSANDRA-11137)
   * Fix is_dense recalculation for Thrift-updated tables (CASSANDRA-11502)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/7480202e/src/java/org/apache/cassandra/utils/JVMStabilityInspector.java
--

[1/3] cassandra git commit: Produce a heap dump when exiting on OO a heap dump when exiting on OOM

[blerer]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.7 54d7fc4b0 -> 9e5161b67


Produce a heap dump when exiting on OO a heap dump when exiting on OOM

patch by Benjamin Lerer; reviewed by Aleksey Yeschenko for CASSANDRA-9861


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b189a7f6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b189a7f6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b189a7f6

Branch: refs/heads/cassandra-3.7
Commit: b189a7f6c955908d8b79d2b4104562a38ea62979
Parents: 8bf453a
Author: Benjamin Lerer 
Authored: Thu Mar 31 15:23:33 2016 +0200
Committer: Benjamin Lerer 
Committed: Wed May 4 17:10:58 2016 +0200

--
 CHANGES.txt |   1 +
 .../org/apache/cassandra/utils/HeapUtils.java   | 205 +++
 .../cassandra/utils/JVMStabilityInspector.java  |   5 +-
 3 files changed, 209 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b189a7f6/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 19e1afe..7c7295d 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.7
+ * Produce a heap dump when exiting on OOM (CASSANDRA-9861)
  * Avoid read repairing purgeable tombstones on range slices (CASSANDRA-11427)
  * Restore ability to filter on clustering columns when using a 2i 
(CASSANDRA-11510)
  * JSON datetime formatting needs timezone (CASSANDRA-11137)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b189a7f6/src/java/org/apache/cassandra/utils/HeapUtils.java
--
diff --git a/src/java/org/apache/cassandra/utils/HeapUtils.java 
b/src/java/org/apache/cassandra/utils/HeapUtils.java
new file mode 100644
index 000..bfc8a0b
--- /dev/null
+++ b/src/java/org/apache/cassandra/utils/HeapUtils.java
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.utils;
+
+import java.io.*;
+import java.lang.management.ManagementFactory;
+import java.lang.management.RuntimeMXBean;
+import java.nio.file.FileSystems;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.commons.lang3.text.StrBuilder;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Utility to generate heap dumps.
+ *
+ */
+public final class HeapUtils
+{
+private static final Logger logger = 
LoggerFactory.getLogger(HeapUtils.class);
+
+/**
+ * Generates a HEAP dump in the directory specified by the 
HeapDumpPath JVM option
+ * or in the CASSANDRA_HOME directory.
+ */
+public static void generateHeapDump()
+{
+Long processId = getProcessId();
+if (processId == null)
+{
+logger.error("The process ID could not be retrieved. Skipping heap 
dump generation.");
+return;
+}
+
+String heapDumpPath = getHeapDumpPathOption();
+if (heapDumpPath == null)
+{
+String cassandraHome = System.getenv("CASSANDRA_HOME");
+if (cassandraHome == null)
+{
+return;
+}
+
+heapDumpPath = cassandraHome;
+}
+
+Path dumpPath = FileSystems.getDefault().getPath(heapDumpPath);
+if (Files.isDirectory(dumpPath))
+{
+dumpPath = dumpPath.resolve("java_pid" + processId + ".hprof");
+}
+
+String jmapPath = getJmapPath();
+
+// The jmap file could not be found. In this case let's default to 
jmap in the hope that it is in the path.
+String jmapCommand = jmapPath == null ? "jmap" : jmapPath;
+
+String[] dumpCommands = new String[] {jmapCommand,
+  "-dump:format=b,file=" + 
dumpPath,
+  processId.toString()};

[3/3] cassandra git commit: Merge branch cassandra-3.0 into cassandra-3.7

[blerer]

Merge branch cassandra-3.0 into cassandra-3.7


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9e5161b6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9e5161b6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9e5161b6

Branch: refs/heads/cassandra-3.7
Commit: 9e5161b6752611515e3e349a7bc4d3f76f8d15d4
Parents: 54d7fc4 7480202
Author: Benjamin Lerer 
Authored: Wed May 4 17:18:18 2016 +0200
Committer: Benjamin Lerer 
Committed: Wed May 4 17:18:32 2016 +0200

--
 CHANGES.txt |   1 +
 .../org/apache/cassandra/utils/HeapUtils.java   | 205 +++
 .../cassandra/utils/JVMStabilityInspector.java  |   3 +
 3 files changed, 209 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/9e5161b6/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/9e5161b6/src/java/org/apache/cassandra/utils/JVMStabilityInspector.java
--

[1/2] cassandra git commit: Produce a heap dump when exiting on OO a heap dump when exiting on OOM

[blerer]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 c7d216e3f -> 7480202e5


Produce a heap dump when exiting on OO a heap dump when exiting on OOM

patch by Benjamin Lerer; reviewed by Aleksey Yeschenko for CASSANDRA-9861


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b189a7f6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b189a7f6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b189a7f6

Branch: refs/heads/cassandra-3.0
Commit: b189a7f6c955908d8b79d2b4104562a38ea62979
Parents: 8bf453a
Author: Benjamin Lerer 
Authored: Thu Mar 31 15:23:33 2016 +0200
Committer: Benjamin Lerer 
Committed: Wed May 4 17:10:58 2016 +0200

--
 CHANGES.txt |   1 +
 .../org/apache/cassandra/utils/HeapUtils.java   | 205 +++
 .../cassandra/utils/JVMStabilityInspector.java  |   5 +-
 3 files changed, 209 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b189a7f6/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 19e1afe..7c7295d 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.7
+ * Produce a heap dump when exiting on OOM (CASSANDRA-9861)
  * Avoid read repairing purgeable tombstones on range slices (CASSANDRA-11427)
  * Restore ability to filter on clustering columns when using a 2i 
(CASSANDRA-11510)
  * JSON datetime formatting needs timezone (CASSANDRA-11137)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b189a7f6/src/java/org/apache/cassandra/utils/HeapUtils.java
--
diff --git a/src/java/org/apache/cassandra/utils/HeapUtils.java 
b/src/java/org/apache/cassandra/utils/HeapUtils.java
new file mode 100644
index 000..bfc8a0b
--- /dev/null
+++ b/src/java/org/apache/cassandra/utils/HeapUtils.java
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.utils;
+
+import java.io.*;
+import java.lang.management.ManagementFactory;
+import java.lang.management.RuntimeMXBean;
+import java.nio.file.FileSystems;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.commons.lang3.text.StrBuilder;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Utility to generate heap dumps.
+ *
+ */
+public final class HeapUtils
+{
+private static final Logger logger = 
LoggerFactory.getLogger(HeapUtils.class);
+
+/**
+ * Generates a HEAP dump in the directory specified by the 
HeapDumpPath JVM option
+ * or in the CASSANDRA_HOME directory.
+ */
+public static void generateHeapDump()
+{
+Long processId = getProcessId();
+if (processId == null)
+{
+logger.error("The process ID could not be retrieved. Skipping heap 
dump generation.");
+return;
+}
+
+String heapDumpPath = getHeapDumpPathOption();
+if (heapDumpPath == null)
+{
+String cassandraHome = System.getenv("CASSANDRA_HOME");
+if (cassandraHome == null)
+{
+return;
+}
+
+heapDumpPath = cassandraHome;
+}
+
+Path dumpPath = FileSystems.getDefault().getPath(heapDumpPath);
+if (Files.isDirectory(dumpPath))
+{
+dumpPath = dumpPath.resolve("java_pid" + processId + ".hprof");
+}
+
+String jmapPath = getJmapPath();
+
+// The jmap file could not be found. In this case let's default to 
jmap in the hope that it is in the path.
+String jmapCommand = jmapPath == null ? "jmap" : jmapPath;
+
+String[] dumpCommands = new String[] {jmapCommand,
+  "-dump:format=b,file=" + 
dumpPath,
+  processId.toString()};

[jira] [Commented] (CASSANDRA-8911) Consider Mutation-based Repairs

[T Jake Luciani (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270811#comment-15270811
 ] 

T Jake Luciani commented on CASSANDRA-8911:
---

I'll volunteer to write some large scale tests for this feature

> Consider Mutation-based Repairs
> ---
>
> Key: CASSANDRA-8911
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8911
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Tyler Hobbs
>Assignee: Marcus Eriksson
> Fix For: 3.x
>
>
> We should consider a mutation-based repair to replace the existing streaming 
> repair.  While we're at it, we could do away with a lot of the complexity 
> around merkle trees.
> I have not planned this out in detail, but here's roughly what I'm thinking:
>  * Instead of building an entire merkle tree up front, just send the "leaves" 
> one-by-one.  Instead of dealing with token ranges, make the leaves primary 
> key ranges.  The PK ranges would need to be contiguous, so that the start of 
> each range would match the end of the previous range. (The first and last 
> leaves would need to be open-ended on one end of the PK range.) This would be 
> similar to doing a read with paging.
>  * Once one page of data is read, compute a hash of it and send it to the 
> other replicas along with the PK range that it covers and a row count.
>  * When the replicas receive the hash, the perform a read over the same PK 
> range (using a LIMIT of the row count + 1) and compare hashes (unless the row 
> counts don't match, in which case this can be skipped).
>  * If there is a mismatch, the replica will send a mutation covering that 
> page's worth of data (ignoring the row count this time) to the source node.
> Here are the advantages that I can think of:
>  * With the current repair behavior of streaming, vnode-enabled clusters may 
> need to stream hundreds of small SSTables.  This results in increased compact
> ion load on the receiving node.  With the mutation-based approach, memtables 
> would naturally merge these.
>  * It's simple to throttle.  For example, you could give a number of rows/sec 
> that should be repaired.
>  * It's easy to see what PK range has been repaired so far.  This could make 
> it simpler to resume a repair that fails midway.
>  * Inconsistencies start to be repaired almost right away.
>  * Less special code \(?\)
>  * Wide partitions are no longer a problem.
> There are a few problems I can think of:
>  * Counters.  I don't know if this can be made safe, or if they need to be 
> skipped.
>  * To support incremental repair, we need to be able to read from only 
> repaired sstables.  Probably not too difficult to do.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11258) Repair scheduling - Resource locking API

[Marcus Olsson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270807#comment-15270807
 ] 

Marcus Olsson commented on CASSANDRA-11258:
---

I've added the check for if the lease was acquired in the case of a WTE and 
also added the mentioned tests to the same branch as before.

I also did some other changes:
* Added javadoc to package/classes (based on the ongoing proposal in the dev 
mailing list regarding mandatory comments).
* Changed from insert to update when doing CAS for the lease.

The reason for changing from {{insert}} to {{update}} was that with the 
previous {{insert}} a row marker is added with a TTL of 60(default TTL) and it 
doesn't get updated when renewing the lease since that uses an {{update}}. So 
if we would "forget" a lease or are unable to renew it in time we would have a 
row marker that is blocking us from acquiring the lease until the TTL has 
expired. Another alternative would be to do a {{SERIAL}} read and check if the 
host column isn't set and then either delete it and retry or use update with 
CAS. Always using update seemed cleaner for the implementation, but it might be 
that this is a small enough issue that the 60 second wait isn't a problem since 
it's only the first 60 seconds from when the lease was acquired.

bq. While it would be interesting to add dtests to make sure leases are working 
in a distributed environment, we would probably need to expose the LeaseFactory 
interface over JMX, but we want to keep this strictly internal to avoid 
external misuse. So it's probably better to move on and test this more 
extensively on dtests via scheduled repairs, for instance by triggering 
simultaneous scheduled repair requests and modifying the resource lease tables 
manually to test leases in scenarios with tampering, network partitions and 
nodes out of sync, so let's leave this for a future task.
I had the same thoughts about dtest and it makes more sense to wait until we 
have scheduled repairs until we test it like that.

---

Since the comments/implementation details regarding update vs insert shouldn't 
affect the interface as such I'll start implementing the repair scheduling. :)

> Repair scheduling - Resource locking API
> 
>
> Key: CASSANDRA-11258
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11258
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: Marcus Olsson
>Assignee: Marcus Olsson
>Priority: Minor
>
> Create a resource locking API & implementation that is able to lock a 
> resource in a specified data center. It should handle priorities to avoid 
> node starvation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] cassandra git commit: Merge branch cassandra-2.2 into cassandra-3.0

[blerer]

Merge branch cassandra-2.2 into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7480202e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7480202e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7480202e

Branch: refs/heads/cassandra-3.0
Commit: 7480202e540f86ae4ff02c9a67253d90ad8dde29
Parents: c7d216e b189a7f
Author: Benjamin Lerer 
Authored: Wed May 4 17:17:08 2016 +0200
Committer: Benjamin Lerer 
Committed: Wed May 4 17:17:08 2016 +0200

--
 CHANGES.txt |   1 +
 .../org/apache/cassandra/utils/HeapUtils.java   | 205 +++
 .../cassandra/utils/JVMStabilityInspector.java  |   5 +-
 3 files changed, 209 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/7480202e/CHANGES.txt
--
diff --cc CHANGES.txt
index f947568,7c7295d..31e46d9
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,24 -1,6 +1,25 @@@
 -2.2.7
 +3.0.6
 + * Disallow creating view with a static column (CASSANDRA-11602)
 + * Reduce the amount of object allocations caused by the getFunctions methods 
(CASSANDRA-11593)
 + * Potential error replaying commitlog with smallint/tinyint/date/time types 
(CASSANDRA-11618)
 + * Fix queries with filtering on counter columns (CASSANDRA-11629)
 + * Improve tombstone printing in sstabledump (CASSANDRA-11655)
 + * Fix paging for range queries where all clustering columns are specified 
(CASSANDRA-11669)
 + * Don't require HEAP_NEW_SIZE to be set when using G1 (CASSANDRA-11600)
 + * Fix sstabledump not showing cells after tombstone marker (CASSANDRA-11654)
 + * Ignore all LocalStrategy keyspaces for streaming and other related
 +   operations (CASSANDRA-11627)
 + * Ensure columnfilter covers indexed columns for thrift 2i queries 
(CASSANDRA-11523)
 + * Only open one sstable scanner per sstable (CASSANDRA-11412)
 + * Option to specify ProtocolVersion in cassandra-stress (CASSANDRA-11410)
 + * ArithmeticException in avgFunctionForDecimal (CASSANDRA-11485)
 + * LogAwareFileLister should only use OLD sstable files in current folder to 
determine disk consistency (CASSANDRA-11470)
 + * Notify indexers of expired rows during compaction (CASSANDRA-11329)
 + * Properly respond with ProtocolError when a v1/v2 native protocol
 +   header is received (CASSANDRA-11464)
 + * Validate that num_tokens and initial_token are consistent with one another 
(CASSANDRA-10120)
 +Merged from 2.2:
+  * Produce a heap dump when exiting on OOM (CASSANDRA-9861)
 - * Avoid read repairing purgeable tombstones on range slices (CASSANDRA-11427)
   * Restore ability to filter on clustering columns when using a 2i 
(CASSANDRA-11510)
   * JSON datetime formatting needs timezone (CASSANDRA-11137)
   * Fix is_dense recalculation for Thrift-updated tables (CASSANDRA-11502)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/7480202e/src/java/org/apache/cassandra/utils/JVMStabilityInspector.java
--

cassandra git commit: Produce a heap dump when exiting on OO a heap dump when exiting on OOM

[blerer]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.2 8bf453a44 -> b189a7f6c


Produce a heap dump when exiting on OO a heap dump when exiting on OOM

patch by Benjamin Lerer; reviewed by Aleksey Yeschenko for CASSANDRA-9861


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b189a7f6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b189a7f6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b189a7f6

Branch: refs/heads/cassandra-2.2
Commit: b189a7f6c955908d8b79d2b4104562a38ea62979
Parents: 8bf453a
Author: Benjamin Lerer 
Authored: Thu Mar 31 15:23:33 2016 +0200
Committer: Benjamin Lerer 
Committed: Wed May 4 17:10:58 2016 +0200

--
 CHANGES.txt |   1 +
 .../org/apache/cassandra/utils/HeapUtils.java   | 205 +++
 .../cassandra/utils/JVMStabilityInspector.java  |   5 +-
 3 files changed, 209 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b189a7f6/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 19e1afe..7c7295d 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.7
+ * Produce a heap dump when exiting on OOM (CASSANDRA-9861)
  * Avoid read repairing purgeable tombstones on range slices (CASSANDRA-11427)
  * Restore ability to filter on clustering columns when using a 2i 
(CASSANDRA-11510)
  * JSON datetime formatting needs timezone (CASSANDRA-11137)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b189a7f6/src/java/org/apache/cassandra/utils/HeapUtils.java
--
diff --git a/src/java/org/apache/cassandra/utils/HeapUtils.java 
b/src/java/org/apache/cassandra/utils/HeapUtils.java
new file mode 100644
index 000..bfc8a0b
--- /dev/null
+++ b/src/java/org/apache/cassandra/utils/HeapUtils.java
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.utils;
+
+import java.io.*;
+import java.lang.management.ManagementFactory;
+import java.lang.management.RuntimeMXBean;
+import java.nio.file.FileSystems;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.commons.lang3.text.StrBuilder;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Utility to generate heap dumps.
+ *
+ */
+public final class HeapUtils
+{
+private static final Logger logger = 
LoggerFactory.getLogger(HeapUtils.class);
+
+/**
+ * Generates a HEAP dump in the directory specified by the 
HeapDumpPath JVM option
+ * or in the CASSANDRA_HOME directory.
+ */
+public static void generateHeapDump()
+{
+Long processId = getProcessId();
+if (processId == null)
+{
+logger.error("The process ID could not be retrieved. Skipping heap 
dump generation.");
+return;
+}
+
+String heapDumpPath = getHeapDumpPathOption();
+if (heapDumpPath == null)
+{
+String cassandraHome = System.getenv("CASSANDRA_HOME");
+if (cassandraHome == null)
+{
+return;
+}
+
+heapDumpPath = cassandraHome;
+}
+
+Path dumpPath = FileSystems.getDefault().getPath(heapDumpPath);
+if (Files.isDirectory(dumpPath))
+{
+dumpPath = dumpPath.resolve("java_pid" + processId + ".hprof");
+}
+
+String jmapPath = getJmapPath();
+
+// The jmap file could not be found. In this case let's default to 
jmap in the hope that it is in the path.
+String jmapCommand = jmapPath == null ? "jmap" : jmapPath;
+
+String[] dumpCommands = new String[] {jmapCommand,
+  "-dump:format=b,file=" + 
dumpPath,
+  processId.toString()};

[jira] [Commented] (CASSANDRA-8911) Consider Mutation-based Repairs

[Jon Haddad (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270790#comment-15270790
 ] 

Jon Haddad commented on CASSANDRA-8911:
---

Regarding DTCS, wouldn't that have been solved with CASSANDRA-11056, if it had 
been merged?

> Consider Mutation-based Repairs
> ---
>
> Key: CASSANDRA-8911
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8911
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Tyler Hobbs
>Assignee: Marcus Eriksson
> Fix For: 3.x
>
>
> We should consider a mutation-based repair to replace the existing streaming 
> repair.  While we're at it, we could do away with a lot of the complexity 
> around merkle trees.
> I have not planned this out in detail, but here's roughly what I'm thinking:
>  * Instead of building an entire merkle tree up front, just send the "leaves" 
> one-by-one.  Instead of dealing with token ranges, make the leaves primary 
> key ranges.  The PK ranges would need to be contiguous, so that the start of 
> each range would match the end of the previous range. (The first and last 
> leaves would need to be open-ended on one end of the PK range.) This would be 
> similar to doing a read with paging.
>  * Once one page of data is read, compute a hash of it and send it to the 
> other replicas along with the PK range that it covers and a row count.
>  * When the replicas receive the hash, the perform a read over the same PK 
> range (using a LIMIT of the row count + 1) and compare hashes (unless the row 
> counts don't match, in which case this can be skipped).
>  * If there is a mismatch, the replica will send a mutation covering that 
> page's worth of data (ignoring the row count this time) to the source node.
> Here are the advantages that I can think of:
>  * With the current repair behavior of streaming, vnode-enabled clusters may 
> need to stream hundreds of small SSTables.  This results in increased compact
> ion load on the receiving node.  With the mutation-based approach, memtables 
> would naturally merge these.
>  * It's simple to throttle.  For example, you could give a number of rows/sec 
> that should be repaired.
>  * It's easy to see what PK range has been repaired so far.  This could make 
> it simpler to resume a repair that fails midway.
>  * Inconsistencies start to be repaired almost right away.
>  * Less special code \(?\)
>  * Wide partitions are no longer a problem.
> There are a few problems I can think of:
>  * Counters.  I don't know if this can be made safe, or if they need to be 
> skipped.
>  * To support incremental repair, we need to be able to read from only 
> repaired sstables.  Probably not too difficult to do.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8911) Consider Mutation-based Repairs

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270766#comment-15270766
 ] 

Marcus Eriksson commented on CASSANDRA-8911:


Have pushed a bunch new commits to the 
[branch|https://github.com/krummas/cassandra/commits/marcuse/8911]

Apart from a bunch of fixes, it adds a nodetool command to trigger a repair {{$ 
nodetool -p7100 mutationbasedrepair stresscql datatest -w 2000 -r 8000}} where 
-w is the page size and -r is the number of rows per second to repair.

Also pushed a bunch of dtests 
[here|https://github.com/krummas/cassandra-dtest/commits/marcuse/8911]

My plan for this is:
* Add a separate memtable without commitlog, instead we record the last 
repaired page when we flush this memtable, so if we lose the memtable, we will 
start repairing from the point where we flushed the memtable last time. We can 
probably also skip serving reads from this separate memtable.
* Make it impossible to start if you run DTCS
* Clean up code, get it reviewed etc.
* Release it as an "experimental" feature
* Create new tickets:
**  make it incremental
** continuously run this
** investigate how to handle gc grace
** make it safe to use with DTCS

Reason I prefer to release it incrementally like this is that I don't want us 
to waste a lot of time if the approach does not really work in real life

> Consider Mutation-based Repairs
> ---
>
> Key: CASSANDRA-8911
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8911
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Tyler Hobbs
>Assignee: Marcus Eriksson
> Fix For: 3.x
>
>
> We should consider a mutation-based repair to replace the existing streaming 
> repair.  While we're at it, we could do away with a lot of the complexity 
> around merkle trees.
> I have not planned this out in detail, but here's roughly what I'm thinking:
>  * Instead of building an entire merkle tree up front, just send the "leaves" 
> one-by-one.  Instead of dealing with token ranges, make the leaves primary 
> key ranges.  The PK ranges would need to be contiguous, so that the start of 
> each range would match the end of the previous range. (The first and last 
> leaves would need to be open-ended on one end of the PK range.) This would be 
> similar to doing a read with paging.
>  * Once one page of data is read, compute a hash of it and send it to the 
> other replicas along with the PK range that it covers and a row count.
>  * When the replicas receive the hash, the perform a read over the same PK 
> range (using a LIMIT of the row count + 1) and compare hashes (unless the row 
> counts don't match, in which case this can be skipped).
>  * If there is a mismatch, the replica will send a mutation covering that 
> page's worth of data (ignoring the row count this time) to the source node.
> Here are the advantages that I can think of:
>  * With the current repair behavior of streaming, vnode-enabled clusters may 
> need to stream hundreds of small SSTables.  This results in increased compact
> ion load on the receiving node.  With the mutation-based approach, memtables 
> would naturally merge these.
>  * It's simple to throttle.  For example, you could give a number of rows/sec 
> that should be repaired.
>  * It's easy to see what PK range has been repaired so far.  This could make 
> it simpler to resume a repair that fails midway.
>  * Inconsistencies start to be repaired almost right away.
>  * Less special code \(?\)
>  * Wide partitions are no longer a problem.
> There are a few problems I can think of:
>  * Counters.  I don't know if this can be made safe, or if they need to be 
> skipped.
>  * To support incremental repair, we need to be able to read from only 
> repaired sstables.  Probably not too difficult to do.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11712) testJsonThreadSafety is failing / flapping

[Alex Petrov (JIRA)]

Alex Petrov created CASSANDRA-11712:
---

 Summary: testJsonThreadSafety is failing / flapping
 Key: CASSANDRA-11712
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11712
 Project: Cassandra
  Issue Type: Bug
  Components: Testing
Reporter: Alex Petrov
Priority: Minor


{{JsonTest::testJsonThreadSafety}} is failing quite often recently: 
https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-11540-2.2-testall/lastCompletedBuild/testReport/org.apache.cassandra.cql3.validation.entities/JsonTest/testJsonThreadSafety/

Output looks like 

{code}
Stacktrace

java.util.concurrent.TimeoutException
at java.util.concurrent.FutureTask.get(FutureTask.java:201)
at 
org.apache.cassandra.cql3.validation.entities.JsonTest.testJsonThreadSafety(JsonTest.java:1028)

WARN  12:19:23 Small commitlog volume detected at 
build/test/cassandra/commitlog:30; setting commitlog_total_space_in_mb to 1982. 
 You can override this in cassandra.yaml
WARN  12:19:23 Small commitlog volume detected at 
build/test/cassandra/commitlog:30; setting commitlog_total_space_in_mb to 1982. 
 You can override this in cassandra.yaml
WARN  12:19:23 Only 5581 MB free across all data volumes. Consider adding more 
capacity to your cluster or removing obsolete snapshots
WARN  12:19:23 Only 5581 MB free across all data volumes. Consider adding more 
capacity to your cluster or removing obsolete snapshots
WARN  12:19:26 Aggregation query used without partition key
WARN  12:19:26 Aggregation query used without partition key
WARN  12:19:26 Aggregation query used without partition key
WARN  12:19:26 Aggregation query used without partition key
Seed 889742091470
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11711) testJsonThreadSafety is failing / flapping

[Alex Petrov (JIRA)]

Alex Petrov created CASSANDRA-11711:
---

 Summary: testJsonThreadSafety is failing / flapping
 Key: CASSANDRA-11711
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11711
 Project: Cassandra
  Issue Type: Bug
  Components: Testing
Reporter: Alex Petrov
Priority: Minor


{{JsonTest::testJsonThreadSafety}} is failing quite often recently: 
https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-11540-2.2-testall/lastCompletedBuild/testReport/org.apache.cassandra.cql3.validation.entities/JsonTest/testJsonThreadSafety/

Output looks like 

{code}
Stacktrace

java.util.concurrent.TimeoutException
at java.util.concurrent.FutureTask.get(FutureTask.java:201)
at 
org.apache.cassandra.cql3.validation.entities.JsonTest.testJsonThreadSafety(JsonTest.java:1028)

WARN  12:19:23 Small commitlog volume detected at 
build/test/cassandra/commitlog:30; setting commitlog_total_space_in_mb to 1982. 
 You can override this in cassandra.yaml
WARN  12:19:23 Small commitlog volume detected at 
build/test/cassandra/commitlog:30; setting commitlog_total_space_in_mb to 1982. 
 You can override this in cassandra.yaml
WARN  12:19:23 Only 5581 MB free across all data volumes. Consider adding more 
capacity to your cluster or removing obsolete snapshots
WARN  12:19:23 Only 5581 MB free across all data volumes. Consider adding more 
capacity to your cluster or removing obsolete snapshots
WARN  12:19:26 Aggregation query used without partition key
WARN  12:19:26 Aggregation query used without partition key
WARN  12:19:26 Aggregation query used without partition key
WARN  12:19:26 Aggregation query used without partition key
Seed 889742091470
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11679) Cassandra Driver returns different number of results depending on fetchsize

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer updated CASSANDRA-11679:
---
Fix Version/s: 2.1.x

> Cassandra Driver returns different number of results depending on fetchsize
> ---
>
> Key: CASSANDRA-11679
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11679
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Varun Barala
>Assignee: Benjamin Lerer
> Fix For: 2.1.x
>
> Attachments: 11679-2.1.txt
>
>
> I'm trying to fetch all distinct keys from a CF using cassandra-driver 
> (2.1.7.1) and I observed some strange behavior :-
> The total distinct rows are 498 so If I perform a query get All distinctKeys 
> It returns 503 instead of 498(five keys twice).
> But If I define the fetch size in select statement more than 498 then it 
> returns exact 498 rows. 
> And If I execute same statement on Dev-center it returns 498 rows (because 
> the default fetch size is 5000). In `cqlsh` it returns 503 rows (because 
> cqlsh uses fetch size=100).
> Some Additional and useful information :- 
> ---
> Cassandra-2.1.13  (C)* version
> Consistency level: ONE 
> local machine(ubuntu 14.04)
> Table Schema:-
> --
> {code:xml}
> CREATE TABLE sample (
>  pk1 text,
>  pk2 text,
> row_id uuid,
> value blob,
> PRIMARY KEY (( pk1,  pk2))
> ) WITH bloom_filter_fp_chance = 0.01
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = ''
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
> AND compression = {'sstable_compression': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
> {code}
> query :-
> 
> {code:xml}
> SELECT DISTINCT  pk2, pk1 FROM sample LIMIT 2147483647;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11679) Cassandra Driver returns different number of results depending on fetchsize

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer updated CASSANDRA-11679:
---
Status: Patch Available  (was: Open)

> Cassandra Driver returns different number of results depending on fetchsize
> ---
>
> Key: CASSANDRA-11679
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11679
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Varun Barala
>Assignee: Benjamin Lerer
> Attachments: 11679-2.1.txt
>
>
> I'm trying to fetch all distinct keys from a CF using cassandra-driver 
> (2.1.7.1) and I observed some strange behavior :-
> The total distinct rows are 498 so If I perform a query get All distinctKeys 
> It returns 503 instead of 498(five keys twice).
> But If I define the fetch size in select statement more than 498 then it 
> returns exact 498 rows. 
> And If I execute same statement on Dev-center it returns 498 rows (because 
> the default fetch size is 5000). In `cqlsh` it returns 503 rows (because 
> cqlsh uses fetch size=100).
> Some Additional and useful information :- 
> ---
> Cassandra-2.1.13  (C)* version
> Consistency level: ONE 
> local machine(ubuntu 14.04)
> Table Schema:-
> --
> {code:xml}
> CREATE TABLE sample (
>  pk1 text,
>  pk2 text,
> row_id uuid,
> value blob,
> PRIMARY KEY (( pk1,  pk2))
> ) WITH bloom_filter_fp_chance = 0.01
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = ''
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
> AND compression = {'sstable_compression': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
> {code}
> query :-
> 
> {code:xml}
> SELECT DISTINCT  pk2, pk1 FROM sample LIMIT 2147483647;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11679) Cassandra Driver returns different number of results depending on fetchsize

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer updated CASSANDRA-11679:
---
Attachment: 11679-2.1.txt

|[unit 
tests|http://cassci.datastax.com/view/Dev/view/blerer/job/blerer-11679-2.1-testall/2/]|[dtests|http://cassci.datastax.com/view/Dev/view/blerer/job/blerer-11679-2.1-dtest/2/]|
The {{paging_test.TestPagingData.test_paging_for_range_name_queries}} is 
failing because the test is invalid for 2.1. I provided a fix for it.

[~thobbs] could you review? It is a backport of CASSANDRA-10010.

> Cassandra Driver returns different number of results depending on fetchsize
> ---
>
> Key: CASSANDRA-11679
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11679
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Varun Barala
>Assignee: Benjamin Lerer
> Attachments: 11679-2.1.txt
>
>
> I'm trying to fetch all distinct keys from a CF using cassandra-driver 
> (2.1.7.1) and I observed some strange behavior :-
> The total distinct rows are 498 so If I perform a query get All distinctKeys 
> It returns 503 instead of 498(five keys twice).
> But If I define the fetch size in select statement more than 498 then it 
> returns exact 498 rows. 
> And If I execute same statement on Dev-center it returns 498 rows (because 
> the default fetch size is 5000). In `cqlsh` it returns 503 rows (because 
> cqlsh uses fetch size=100).
> Some Additional and useful information :- 
> ---
> Cassandra-2.1.13  (C)* version
> Consistency level: ONE 
> local machine(ubuntu 14.04)
> Table Schema:-
> --
> {code:xml}
> CREATE TABLE sample (
>  pk1 text,
>  pk2 text,
> row_id uuid,
> value blob,
> PRIMARY KEY (( pk1,  pk2))
> ) WITH bloom_filter_fp_chance = 0.01
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = ''
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
> AND compression = {'sstable_compression': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
> {code}
> query :-
> 
> {code:xml}
> SELECT DISTINCT  pk2, pk1 FROM sample LIMIT 2147483647;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10962) Cassandra should not create snapshot at restart for compactions_in_progress

[Alex Petrov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270665#comment-15270665
 ] 

Alex Petrov commented on CASSANDRA-10962:
-

true, my initial patch was to add {{skipSnapshot}}, although logic there was 
getting tricky, so I simplified it. 

Just to give heads up, in 3.0+ this problem does not appear as 
[~fabrice.facorat] mentioned due to 
[7066|https://issues.apache.org/jira/browse/CASSANDRA-7066].

I've checked the failing dtests locally, both are passing. With unit tests, 
it's same. Same with utests...


|[2.2|https://github.com/ifesdjeen/cassandra/tree/10962-2.2]|[utest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-10962-2.2-testall/]|[dtest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-10962-2.2-dtest/]|

> Cassandra should not create snapshot at restart for compactions_in_progress
> ---
>
> Key: CASSANDRA-10962
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10962
> Project: Cassandra
>  Issue Type: Bug
> Environment: Ubuntu 14.04.3 LTS
>Reporter: FACORAT
>Assignee: Alex Petrov
>Priority: Minor
> Fix For: 2.2.x
>
>
> If auto_snapshot is set to true in cassandra.yaml, each time you restart 
> Cassandra, a snapshot is created for system.compactions_in_progress as the 
> table is truncated at cassandra start.
> However as datas in this table are temporary, Cassandra should not create 
> snapshot for this table (or maybe even for system.* tables). This will be 
> coherent with the fact that "nodetool listsnapshots" doesn't even list this 
> table.
> Exemple:
> {code}
> $ nodetool listsnapshots | grep compactions
> $ ls -lh 
> system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/snapshots/
> total 16K
> drwxr-xr-x 2 cassandra cassandra 4.0K Nov 30 13:12 
> 1448885530280-compactions_in_progress
> drwxr-xr-x 2 cassandra cassandra 4.0K Dec  7 15:36 
> 1449498977181-compactions_in_progress
> drwxr-xr-x 2 cassandra cassandra 4.0K Dec 14 18:20 
> 1450113621506-compactions_in_progress
> drwxr-xr-x 2 cassandra cassandra 4.0K Jan  4 12:53 
> 1451908396364-compactions_in_progress
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (CASSANDRA-11710) Cassandra dies with OOM when running stress

[Branimir Lambov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Branimir Lambov reassigned CASSANDRA-11710:
---

Assignee: Branimir Lambov

> Cassandra dies with OOM when running stress
> ---
>
> Key: CASSANDRA-11710
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11710
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Branimir Lambov
>
> Running stress on trunk dies with OOM after about 3.5M ops:
> {code}
> ERROR [CompactionExecutor:1] 2016-05-04 15:01:31,231 
> JVMStabilityInspector.java:137 - JVM state determined to be unstable.  
> Exiting forcefully due to:
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:693) ~[na:1.8.0_91]
> at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) 
> ~[na:1.8.0_91]
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) 
> ~[na:1.8.0_91]
> at 
> org.apache.cassandra.utils.memory.BufferPool.allocateDirectAligned(BufferPool.java:519)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool.access$600(BufferPool.java:46) 
> ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool$GlobalPool.allocateMoreChunks(BufferPool.java:276)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool$GlobalPool.get(BufferPool.java:249)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool$LocalPool.addChunkFromGlobalPool(BufferPool.java:338)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool$LocalPool.get(BufferPool.java:381)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool.maybeTakeFromPool(BufferPool.java:142)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool.takeFromPool(BufferPool.java:114)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool.get(BufferPool.java:84) 
> ~[main/:na]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:135) 
> ~[main/:na]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:19) 
> ~[main/:na]
> at 
> com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:2949)
>  ~[caffeine-2.2.6.jar:na]
> at 
> com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$15(BoundedLocalCache.java:1807)
>  ~[caffeine-2.2.6.jar:na]
> at 
> java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853) 
> ~[na:1.8.0_91]
> at 
> com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1805)
>  ~[caffeine-2.2.6.jar:na]
> at 
> com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1788)
>  ~[caffeine-2.2.6.jar:na]
> at 
> com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:97)
>  ~[caffeine-2.2.6.jar:na]
> at 
> com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:66)
>  ~[caffeine-2.2.6.jar:na]
> at 
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:215)
>  ~[main/:na]
> at 
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:193)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.LimitingRebufferer.rebuffer(LimitingRebufferer.java:34)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RandomAccessReader.reBufferAt(RandomAccessReader.java:78)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:72)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.read(RebufferingInputStream.java:88)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:66)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:400) 
> ~[main/:na]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.readWithVIntLength(ByteBufferUtil.java:338)
>  ~[main/:na]
> at 
> org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:414) 
> ~[main/:na]
> at 
> org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:243) 
> ~[main/:na]
> at 
> org.apache.cassandra.db.rows.UnfilteredSerializer.readSimpleColumn(UnfilteredSerializer.java:473)
>  ~[main/:na]
> at 
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:451)
>  ~[main/:na]
> at 
> 

[jira] [Created] (CASSANDRA-11710) Cassandra dies with OOM when running stress

[Marcus Eriksson (JIRA)]

Marcus Eriksson created CASSANDRA-11710:
---

 Summary: Cassandra dies with OOM when running stress
 Key: CASSANDRA-11710
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11710
 Project: Cassandra
  Issue Type: Bug
Reporter: Marcus Eriksson


Running stress on trunk dies with OOM after about 3.5M ops:

{code}
ERROR [CompactionExecutor:1] 2016-05-04 15:01:31,231 
JVMStabilityInspector.java:137 - JVM state determined to be unstable.  Exiting 
forcefully due to:
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:693) ~[na:1.8.0_91]
at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) 
~[na:1.8.0_91]
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) 
~[na:1.8.0_91]
at 
org.apache.cassandra.utils.memory.BufferPool.allocateDirectAligned(BufferPool.java:519)
 ~[main/:na]
at 
org.apache.cassandra.utils.memory.BufferPool.access$600(BufferPool.java:46) 
~[main/:na]
at 
org.apache.cassandra.utils.memory.BufferPool$GlobalPool.allocateMoreChunks(BufferPool.java:276)
 ~[main/:na]
at 
org.apache.cassandra.utils.memory.BufferPool$GlobalPool.get(BufferPool.java:249)
 ~[main/:na]
at 
org.apache.cassandra.utils.memory.BufferPool$LocalPool.addChunkFromGlobalPool(BufferPool.java:338)
 ~[main/:na]
at 
org.apache.cassandra.utils.memory.BufferPool$LocalPool.get(BufferPool.java:381) 
~[main/:na]
at 
org.apache.cassandra.utils.memory.BufferPool.maybeTakeFromPool(BufferPool.java:142)
 ~[main/:na]
at 
org.apache.cassandra.utils.memory.BufferPool.takeFromPool(BufferPool.java:114) 
~[main/:na]
at org.apache.cassandra.utils.memory.BufferPool.get(BufferPool.java:84) 
~[main/:na]
at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:135) 
~[main/:na]
at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:19) 
~[main/:na]
at 
com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:2949)
 ~[caffeine-2.2.6.jar:na]
at 
com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$15(BoundedLocalCache.java:1807)
 ~[caffeine-2.2.6.jar:na]
at 
java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853) 
~[na:1.8.0_91]
at 
com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1805)
 ~[caffeine-2.2.6.jar:na]
at 
com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1788)
 ~[caffeine-2.2.6.jar:na]
at 
com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:97)
 ~[caffeine-2.2.6.jar:na]
at 
com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:66)
 ~[caffeine-2.2.6.jar:na]
at 
org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:215)
 ~[main/:na]
at 
org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:193)
 ~[main/:na]
at 
org.apache.cassandra.io.util.LimitingRebufferer.rebuffer(LimitingRebufferer.java:34)
 ~[main/:na]
at 
org.apache.cassandra.io.util.RandomAccessReader.reBufferAt(RandomAccessReader.java:78)
 ~[main/:na]
at 
org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:72)
 ~[main/:na]
at 
org.apache.cassandra.io.util.RebufferingInputStream.read(RebufferingInputStream.java:88)
 ~[main/:na]
at 
org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:66)
 ~[main/:na]
at 
org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
 ~[main/:na]
at 
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:400) 
~[main/:na]
at 
org.apache.cassandra.utils.ByteBufferUtil.readWithVIntLength(ByteBufferUtil.java:338)
 ~[main/:na]
at 
org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:414) 
~[main/:na]
at 
org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:243) 
~[main/:na]
at 
org.apache.cassandra.db.rows.UnfilteredSerializer.readSimpleColumn(UnfilteredSerializer.java:473)
 ~[main/:na]
at 
org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:451)
 ~[main/:na]
at 
org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:380)
 ~[main/:na]
at 
org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:87)
 ~[main/:na]
at 
org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:65)
 ~[main/:na]
at 

[jira] [Commented] (CASSANDRA-11349) MerkleTree mismatch when multiple range tombstones exists for the same partition and interval

[Branimir Lambov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270572#comment-15270572
 ] 

Branimir Lambov commented on CASSANDRA-11349:
-

As I see it neither solution will be sufficient. A lot of the visible effects 
of the problem come as a side effect of CASSANDRA-7953, but there are some 
underlying issues that are only really solved in 3.0 by the new tombstone 
handling from CASSANDRA-8099.

Whether we change {{onDiskAtomComparator}} or not, we will still get disordered 
or multiple equal range tombstones from a single source as that's how they are 
written in the sstables. {{MergeIterator}} will not combine equal entries from 
the same source, even if it did and everything was written using the 
{{onDiskAtomComparator}} (which I don't believe to be the case), it is still in 
the wrong order for resolving which tombstones can be deleted without delaying 
their processing.

In other words the problem cannot be solved by changing the reducer; we can, 
however, do it if we change {{update}} to follow closely or, better still, 
_call_ {{IndexBuilder.buildForCompaction}} and make the builder accept a 
prepared atom serializer (or some subinterface) instead of an output file, and 
update the digest in the calls to that serializer.

> MerkleTree mismatch when multiple range tombstones exists for the same 
> partition and interval
> -
>
> Key: CASSANDRA-11349
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11349
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Fabien Rousseau
>Assignee: Stefan Podkowinski
>  Labels: repair
> Fix For: 2.1.x, 2.2.x
>
> Attachments: 11349-2.1-v2.patch, 11349-2.1-v3.patch, 11349-2.1.patch
>
>
> We observed that repair, for some of our clusters, streamed a lot of data and 
> many partitions were "out of sync".
> Moreover, the read repair mismatch ratio is around 3% on those clusters, 
> which is really high.
> After investigation, it appears that, if two range tombstones exists for a 
> partition for the same range/interval, they're both included in the merkle 
> tree computation.
> But, if for some reason, on another node, the two range tombstones were 
> already compacted into a single range tombstone, this will result in a merkle 
> tree difference.
> Currently, this is clearly bad because MerkleTree differences are dependent 
> on compactions (and if a partition is deleted and created multiple times, the 
> only way to ensure that repair "works correctly"/"don't overstream data" is 
> to major compact before each repair... which is not really feasible).
> Below is a list of steps allowing to easily reproduce this case:
> {noformat}
> ccm create test -v 2.1.13 -n 2 -s
> ccm node1 cqlsh
> CREATE KEYSPACE test_rt WITH replication = {'class': 'SimpleStrategy', 
> 'replication_factor': 2};
> USE test_rt;
> CREATE TABLE IF NOT EXISTS table1 (
> c1 text,
> c2 text,
> c3 float,
> c4 float,
> PRIMARY KEY ((c1), c2)
> );
> INSERT INTO table1 (c1, c2, c3, c4) VALUES ( 'a', 'b', 1, 2);
> DELETE FROM table1 WHERE c1 = 'a' AND c2 = 'b';
> ctrl ^d
> # now flush only one of the two nodes
> ccm node1 flush 
> ccm node1 cqlsh
> USE test_rt;
> INSERT INTO table1 (c1, c2, c3, c4) VALUES ( 'a', 'b', 1, 3);
> DELETE FROM table1 WHERE c1 = 'a' AND c2 = 'b';
> ctrl ^d
> ccm node1 repair
> # now grep the log and observe that there was some inconstencies detected 
> between nodes (while it shouldn't have detected any)
> ccm node1 showlog | grep "out of sync"
> {noformat}
> Consequences of this are a costly repair, accumulating many small SSTables 
> (up to thousands for a rather short period of time when using VNodes, the 
> time for compaction to absorb those small files), but also an increased size 
> on disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)