[jira] [Updated] (CASSANDRA-13072) Cassandra failed to run on Linux-aarch64

[Jun He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun He updated CASSANDRA-13072:
---
Priority: Blocker  (was: Major)

> Cassandra failed to run on Linux-aarch64
> 
>
> Key: CASSANDRA-13072
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13072
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Hardware: ARM aarch64
> OS: Ubuntu 16.04.1 LTS
>Reporter: Jun He
>Priority: Blocker
>  Labels: incompatible
> Attachments: compat_report.html
>
>
> Steps to reproduce:
> 1. Download cassandra latest source
> 2. Build it with "ant"
> 3. Run with "./bin/cassandra". Daemon is crashed with following error message:
> {quote}
> INFO  05:30:21 Initializing system.schema_functions
> INFO  05:30:21 Initializing system.schema_aggregates
> ERROR 05:30:22 Exception in thread Thread[MemtableFlushWriter:1,5,main]
> java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
> at 
> org.apache.cassandra.utils.memory.MemoryUtil.allocate(MemoryUtil.java:97) 
> ~[main/:na]
> at org.apache.cassandra.io.util.Memory.(Memory.java:74) 
> ~[main/:na]
> at org.apache.cassandra.io.util.SafeMemory.(SafeMemory.java:32) 
> ~[main/:na]
> at 
> org.apache.cassandra.io.compress.CompressionMetadata$Writer.(CompressionMetadata.java:316)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.compress.CompressionMetadata$Writer.open(CompressionMetadata.java:330)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.compress.CompressedSequentialWriter.(CompressedSequentialWriter.java:76)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:163) 
> ~[main/:na]
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.(BigTableWriter.java:73)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.sstable.format.big.BigFormat$WriterFactory.open(BigFormat.java:93)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTableWriter.java:96)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.create(SimpleSSTableMultiWriter.java:114)
>  ~[main/:na]
> at 
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.createSSTableMultiWriter(AbstractCompactionStrategy.java:519)
>  ~[main/:na]
> at 
> org.apache.cassandra.db.compaction.CompactionStrategyManager.createSSTableMultiWriter(CompactionStrategyManager.java:497)
>  ~[main/:na]
> at 
> org.apache.cassandra.db.ColumnFamilyStore.createSSTableMultiWriter(ColumnFamilyStore.java:480)
>  ~[main/:na]
> at 
> org.apache.cassandra.db.Memtable.createFlushWriter(Memtable.java:439) 
> ~[main/:na]
> at 
> org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:371) 
> ~[main/:na]
> at org.apache.cassandra.db.Memtable.flush(Memtable.java:332) 
> ~[main/:na]
> at 
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1054)
>  ~[main/:na]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  ~[na:1.8.0_111]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  ~[na:1.8.0_111]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_111]
> {quote}
> Analyze:
> This issue is caused by bundled jna-4.0.0.jar which doesn't come with aarch64 
> native support. Replace lib/jna-4.0.0.jar with jna-4.2.0.jar from 
> http://central.maven.org/maven2/net/java/dev/jna/jna/4.2.0/ can fix this 
> problem.
> Attached is the binary compatibility report of jna.jar between 4.0 and 4.2. 
> The result is good (97.4%). So is there possibility to upgrade jna to 4.2.0 
> in upstream? Should there be any kind of tests to execute, please kindly 
> point me. Thanks a lot.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-13133) Unclosed file descriptors when querying SnapshotsSize metric

[Stefania (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefania updated CASSANDRA-13133:
-
Fix Version/s: (was: 3.9)
   3.10

> Unclosed file descriptors when querying SnapshotsSize metric
> 
>
> Key: CASSANDRA-13133
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13133
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths
> Environment: CentOS 7
>Reporter: Nicholas Rushton
>  Labels: jmx, lhf, metrics, newbie
> Fix For: 3.10
>
>
> Started to notice many open file descriptors (100k+) per node, growing at a 
> rate of about 30 per minute in our cluster. After turning off our JMX 
> exporting server(https://github.com/prometheus/jmx_exporter), which gets 
> queried every 30 seconds, the number of file descriptors remained static. 
> Digging a bit further I ran a jmx dump tool over all the cassandra metrics 
> and tracked the number of file descriptors after each query, boiling it down 
> to a single metric causing the number of file descriptors to increase:
> org.apache.cassandra.metrics:keyspace=tpsv1,name=SnapshotsSize,scope=events_by_engagement_id,type=Table
> running a query a few times against this metric shows the file descriptors 
> increasing after each query:
> {code}
> for _ in {0..3} 
> do 
>java -jar jmx-dump-0.4.2-standalone.jar --port 7199 --dump 
> org.apache.cassandra.metrics:keyspace=tpsv1,name=SnapshotsSize,scope=events_by_engagement_id,type=Table
>  > /dev/null; 
>sudo lsof -p `pgrep -f CassandraDaemon` | fgrep "DIR" | awk 
> '{a[$(NF)]+=1}END{for(k in a){print k, a[k]}}' | grep "events_by" 
> done
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> >  33176
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> >  33177
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> >  33178
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> >  33179
> {code}
> it should be noted that the file descriptor is open on a directory, not an 
> actual file



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-13133) Unclosed file descriptors when querying SnapshotsSize metric

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831287#comment-15831287
 ] 

Stefania commented on CASSANDRA-13133:
--

Thanks for confirming!

> Unclosed file descriptors when querying SnapshotsSize metric
> 
>
> Key: CASSANDRA-13133
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13133
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths
> Environment: CentOS 7
>Reporter: Nicholas Rushton
>  Labels: jmx, lhf, metrics, newbie
> Fix For: 3.10
>
>
> Started to notice many open file descriptors (100k+) per node, growing at a 
> rate of about 30 per minute in our cluster. After turning off our JMX 
> exporting server(https://github.com/prometheus/jmx_exporter), which gets 
> queried every 30 seconds, the number of file descriptors remained static. 
> Digging a bit further I ran a jmx dump tool over all the cassandra metrics 
> and tracked the number of file descriptors after each query, boiling it down 
> to a single metric causing the number of file descriptors to increase:
> org.apache.cassandra.metrics:keyspace=tpsv1,name=SnapshotsSize,scope=events_by_engagement_id,type=Table
> running a query a few times against this metric shows the file descriptors 
> increasing after each query:
> {code}
> for _ in {0..3} 
> do 
>java -jar jmx-dump-0.4.2-standalone.jar --port 7199 --dump 
> org.apache.cassandra.metrics:keyspace=tpsv1,name=SnapshotsSize,scope=events_by_engagement_id,type=Table
>  > /dev/null; 
>sudo lsof -p `pgrep -f CassandraDaemon` | fgrep "DIR" | awk 
> '{a[$(NF)]+=1}END{for(k in a){print k, a[k]}}' | grep "events_by" 
> done
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> >  33176
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> >  33177
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> >  33178
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> >  33179
> {code}
> it should be noted that the file descriptor is open on a directory, not an 
> actual file



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-13117) Dump threads when unit test times out

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830910#comment-15830910
 ] 

Jason Brown commented on CASSANDRA-13117:
-

+1. I like this idea!

> Dump threads when unit test times out
> -
>
> Key: CASSANDRA-13117
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13117
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Minor
> Fix For: 3.0.x, 3.x
>
>
> It would be nice to get a thread dump when unit tests time out



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-13137) nodetool disablethrift deadlocks if THsHaDisruptorServer is stopped while a request is being processed

[Sotirios Delimanolis (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830856#comment-15830856
 ] 

Sotirios Delimanolis commented on CASSANDRA-13137:
--

My opinion is that a proper stop/drain would stop the selector threads' 
{{select()}} loop (or really just the read part), but would wait to clean up 
its resources until after the ring buffer was drained.

> nodetool disablethrift deadlocks if THsHaDisruptorServer is stopped while a 
> request is being processed
> --
>
> Key: CASSANDRA-13137
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13137
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: 2.2.9
>Reporter: Sotirios Delimanolis
>
> We are using Thrift with {{rpc_server_type}} set to {{hsha}}. This creates a 
> {{THsHaDisruptorServer}} which is a subclass of 
> [{{TDisruptorServer}}|https://github.com/xedin/disruptor_thrift_server/blob/master/src/main/java/com/thinkaurelius/thrift/TDisruptorServer.java].
> Internally, this spawns {{number_of_cores}} number of selector threads. Each 
> gets a {{RingBuffer}} and {{rpc_max_threads / cores}} number of worker 
> threads (the {{RPC-Thread}} threads). As the server starts receiving 
> requests, each selector thread adds events to its {{RingBuffer}} and the 
> worker threads process them. 
> The _events_ are 
> [{{Message}}|https://github.com/xedin/disruptor_thrift_server/blob/master/src/main/java/com/thinkaurelius/thrift/Message.java]
>  instances, which have preallocated buffers for eventual IO.
> When the thrift server starts up, the corresponding {{ThriftServerThread}} 
> joins on the selector threads, waiting for them to die. It then iterates 
> through all the {{SelectorThread}} objects and calls their {{shutdown}} 
> method which attempts to drain their corresponding {{RingBuffer}}. The [drain 
> ({{drainAndHalt}})|https://github.com/LMAX-Exchange/disruptor/blob/master/src/main/java/com/lmax/disruptor/WorkerPool.java#L147]
>  works by letting the worker pool "consumer" threads catch up to the 
> "producer" index, ie. the selector thread.
> When we execute a {{nodetool disablethrift}}, it attempts to {{stop}} the 
> {{THsHaDisruptorServer}}. That works by setting a {{stopped}} flag to 
> {{true}}. When the selector threads see that, they break from their 
> {{select()}} loop, and clean up their resources, ie. the {{Message}} objects 
> they've created and their buffers. *However*, if one of those {{Message}} 
> objects is currently being used by a worker pool thread to process a request, 
> if it calls [this piece of 
> code|https://github.com/xedin/disruptor_thrift_server/blob/master/src/main/java/com/thinkaurelius/thrift/Message.java#L317],
>  you'll get the following {{NullPointerException}}
> {noformat}
> Jan 18, 2017 6:28:50 PM com.lmax.disruptor.FatalExceptionHandler 
> handleEventException
> SEVERE: Exception processing: 633124 
> com.thinkaurelius.thrift.Message$Invocation@25c9fbeb
> java.lang.NullPointerException
> at 
> com.thinkaurelius.thrift.Message.getInputTransport(Message.java:338)
> at com.thinkaurelius.thrift.Message.invoke(Message.java:308)
> at 
> com.thinkaurelius.thrift.Message$Invocation.execute(Message.java:90)
> at 
> com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:695)
> at 
> com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:689)
> at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:112)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> That fails because it tries to dereference one of the {{Message}} "cleaned 
> up", ie. {{null}}, buffers.
> Because that call is outside the {{try}} block, the exception escapes and 
> basically kills the worker pool thread. This has the side effect of 
> "discarding" one of the consumers of a selector's {{RingBuffer}}. 
> *That* has the side effect of preventing the {{ThriftServerThread}} from 
> draining the {{RingBuffer}} (and dying) since the consumers never catch up to 
> the stopped producer. And that finally has the effect of preventing the 
> {{nodetool disablethrift}} from proceeding since it's trying to {{join}} the 
> {{ThriftServerThread}}. Deadlock!
> The {{ThriftServerThread}} thread looks like
> {noformat}
> "Thread-1" #2234 prio=5 os_prio=0 tid=0x7f4ae6ff1000 nid=0x2eb6 runnable 
> [0x7f4729174000]
>java.lang.Thread.State: RUNNABLE
> at java.lang.Thread.yield(Native Method)
> at 

[jira] [Updated] (CASSANDRA-13137) nodetool disablethrift deadlocks if THsHaDisruptorServer is stopped while a request is being processed

[Sotirios Delimanolis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sotirios Delimanolis updated CASSANDRA-13137:
-
Summary: nodetool disablethrift deadlocks if THsHaDisruptorServer is 
stopped while a request is being processed  (was: nodetool disablethrift 
deadlocks if THsHaDisruptorServer is stopped while a read is going on)

> nodetool disablethrift deadlocks if THsHaDisruptorServer is stopped while a 
> request is being processed
> --
>
> Key: CASSANDRA-13137
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13137
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: 2.2.9
>Reporter: Sotirios Delimanolis
>
> We are using Thrift with {{rpc_server_type}} set to {{hsha}}. This creates a 
> {{THsHaDisruptorServer}} which is a subclass of 
> [{{TDisruptorServer}}|https://github.com/xedin/disruptor_thrift_server/blob/master/src/main/java/com/thinkaurelius/thrift/TDisruptorServer.java].
> Internally, this spawns {{number_of_cores}} number of selector threads. Each 
> gets a {{RingBuffer}} and {{rpc_max_threads / cores}} number of worker 
> threads (the {{RPC-Thread}} threads). As the server starts receiving 
> requests, each selector thread adds events to its {{RingBuffer}} and the 
> worker threads process them. 
> The _events_ are 
> [{{Message}}|https://github.com/xedin/disruptor_thrift_server/blob/master/src/main/java/com/thinkaurelius/thrift/Message.java]
>  instances, which have preallocated buffers for eventual IO.
> When the thrift server starts up, the corresponding {{ThriftServerThread}} 
> joins on the selector threads, waiting for them to die. It then iterates 
> through all the {{SelectorThread}} objects and calls their {{shutdown}} 
> method which attempts to drain their corresponding {{RingBuffer}}. The [drain 
> ({{drainAndHalt}})|https://github.com/LMAX-Exchange/disruptor/blob/master/src/main/java/com/lmax/disruptor/WorkerPool.java#L147]
>  works by letting the worker pool "consumer" threads catch up to the 
> "producer" index, ie. the selector thread.
> When we execute a {{nodetool disablethrift}}, it attempts to {{stop}} the 
> {{THsHaDisruptorServer}}. That works by setting a {{stopped}} flag to 
> {{true}}. When the selector threads see that, they break from their 
> {{select()}} loop, and clean up their resources, ie. the {{Message}} objects 
> they've created and their buffers. *However*, if one of those {{Message}} 
> objects is currently being used by a worker pool thread to process a request, 
> if it calls [this piece of 
> code|https://github.com/xedin/disruptor_thrift_server/blob/master/src/main/java/com/thinkaurelius/thrift/Message.java#L317],
>  you'll get the following {{NullPointerException}}
> {noformat}
> Jan 18, 2017 6:28:50 PM com.lmax.disruptor.FatalExceptionHandler 
> handleEventException
> SEVERE: Exception processing: 633124 
> com.thinkaurelius.thrift.Message$Invocation@25c9fbeb
> java.lang.NullPointerException
> at 
> com.thinkaurelius.thrift.Message.getInputTransport(Message.java:338)
> at com.thinkaurelius.thrift.Message.invoke(Message.java:308)
> at 
> com.thinkaurelius.thrift.Message$Invocation.execute(Message.java:90)
> at 
> com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:695)
> at 
> com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:689)
> at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:112)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> That fails because it tries to dereference one of the {{Message}} "cleaned 
> up", ie. {{null}}, buffers.
> Because that call is outside the {{try}} block, the exception escapes and 
> basically kills the worker pool thread. This has the side effect of 
> "discarding" one of the consumers of a selector's {{RingBuffer}}. 
> *That* has the side effect of preventing the {{ThriftServerThread}} from 
> draining the {{RingBuffer}} (and dying) since the consumers never catch up to 
> the stopped producer. And that finally has the effect of preventing the 
> {{nodetool disablethrift}} from proceeding since it's trying to {{join}} the 
> {{ThriftServerThread}}. Deadlock!
> The {{ThriftServerThread}} thread looks like
> {noformat}
> "Thread-1" #2234 prio=5 os_prio=0 tid=0x7f4ae6ff1000 nid=0x2eb6 runnable 
> [0x7f4729174000]
>java.lang.Thread.State: RUNNABLE
> at java.lang.Thread.yield(Native Method)
> at 

[jira] [Commented] (CASSANDRA-13137) nodetool disablethrift deadlocks if THsHaDisruptorServer is stopped while a read is going on

[Sotirios Delimanolis (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830766#comment-15830766
 ] 

Sotirios Delimanolis commented on CASSANDRA-13137:
--

There's also this {{NullPointerException}} possible

{noformat}
ERROR [RPC-Thread:68] 2017-01-18 18:28:50,879 Message.java:324 - Unexpected 
throwable while invoking!
java.lang.NullPointerException: null
at com.thinkaurelius.thrift.util.mem.Buffer.size(Buffer.java:83) 
~[thrift-server-0.3.7.jar:na]
at 
com.thinkaurelius.thrift.util.mem.FastMemoryOutputTransport.expand(FastMemoryOutputTransport.java:84)
 ~[thrift-server-0.3.7.jar:na]
at 
com.thinkaurelius.thrift.util.mem.FastMemoryOutputTransport.write(FastMemoryOutputTransport.java:167)
 ~[thrift-server-0.3.7.jar:na]
at 
org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:156) 
~[libthrift-0.9.2.jar:0.9.2]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:55) 
~[libthrift-0.9.2.jar:0.9.2]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) 
~[libthrift-0.9.2.jar:0.9.2]
at com.thinkaurelius.thrift.Message.invoke(Message.java:314) 
~[thrift-server-0.3.7.jar:na]
at com.thinkaurelius.thrift.Message$Invocation.execute(Message.java:90) 
[thrift-server-0.3.7.jar:na]
at 
com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:695)
 [thrift-server-0.3.7.jar:na]
at 
com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:689)
 [thrift-server-0.3.7.jar:na]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:112) 
[disruptor-3.0.1.jar:na]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_102]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_102]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
{noformat}

But that happens within the {{invoke}} method's try/catch block which 
essentially just swallows it, but doesn't "kill" the current thread.

> nodetool disablethrift deadlocks if THsHaDisruptorServer is stopped while a 
> read is going on
> 
>
> Key: CASSANDRA-13137
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13137
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: 2.2.9
>Reporter: Sotirios Delimanolis
>
> We are using Thrift with {{rpc_server_type}} set to {{hsha}}. This creates a 
> {{THsHaDisruptorServer}} which is a subclass of 
> [{{TDisruptorServer}}|https://github.com/xedin/disruptor_thrift_server/blob/master/src/main/java/com/thinkaurelius/thrift/TDisruptorServer.java].
> Internally, this spawns {{number_of_cores}} number of selector threads. Each 
> gets a {{RingBuffer}} and {{rpc_max_threads / cores}} number of worker 
> threads (the {{RPC-Thread}} threads). As the server starts receiving 
> requests, each selector thread adds events to its {{RingBuffer}} and the 
> worker threads process them. 
> The _events_ are 
> [{{Message}}|https://github.com/xedin/disruptor_thrift_server/blob/master/src/main/java/com/thinkaurelius/thrift/Message.java]
>  instances, which have preallocated buffers for eventual IO.
> When the thrift server starts up, the corresponding {{ThriftServerThread}} 
> joins on the selector threads, waiting for them to die. It then iterates 
> through all the {{SelectorThread}} objects and calls their {{shutdown}} 
> method which attempts to drain their corresponding {{RingBuffer}}. The [drain 
> ({{drainAndHalt}})|https://github.com/LMAX-Exchange/disruptor/blob/master/src/main/java/com/lmax/disruptor/WorkerPool.java#L147]
>  works by letting the worker pool "consumer" threads catch up to the 
> "producer" index, ie. the selector thread.
> When we execute a {{nodetool disablethrift}}, it attempts to {{stop}} the 
> {{THsHaDisruptorServer}}. That works by setting a {{stopped}} flag to 
> {{true}}. When the selector threads see that, they break from their 
> {{select()}} loop, and clean up their resources, ie. the {{Message}} objects 
> they've created and their buffers. *However*, if one of those {{Message}} 
> objects is currently being used by a worker pool thread to process a request, 
> if it calls [this piece of 
> code|https://github.com/xedin/disruptor_thrift_server/blob/master/src/main/java/com/thinkaurelius/thrift/Message.java#L317],
>  you'll get the following {{NullPointerException}}
> {noformat}
> Jan 18, 2017 6:28:50 PM com.lmax.disruptor.FatalExceptionHandler 
> handleEventException
> SEVERE: Exception processing: 633124 
> com.thinkaurelius.thrift.Message$Invocation@25c9fbeb
> java.lang.NullPointerException
> at 
> 

[jira] [Comment Edited] (CASSANDRA-13137) nodetool disablethrift deadlocks if THsHaDisruptorServer is stopped while a read is going on

[Sotirios Delimanolis (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830766#comment-15830766
 ] 

Sotirios Delimanolis edited comment on CASSANDRA-13137 at 1/19/17 10:57 PM:


It can also cause this {{NullPointerException}} 

{noformat}
ERROR [RPC-Thread:68] 2017-01-18 18:28:50,879 Message.java:324 - Unexpected 
throwable while invoking!
java.lang.NullPointerException: null
at com.thinkaurelius.thrift.util.mem.Buffer.size(Buffer.java:83) 
~[thrift-server-0.3.7.jar:na]
at 
com.thinkaurelius.thrift.util.mem.FastMemoryOutputTransport.expand(FastMemoryOutputTransport.java:84)
 ~[thrift-server-0.3.7.jar:na]
at 
com.thinkaurelius.thrift.util.mem.FastMemoryOutputTransport.write(FastMemoryOutputTransport.java:167)
 ~[thrift-server-0.3.7.jar:na]
at 
org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:156) 
~[libthrift-0.9.2.jar:0.9.2]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:55) 
~[libthrift-0.9.2.jar:0.9.2]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) 
~[libthrift-0.9.2.jar:0.9.2]
at com.thinkaurelius.thrift.Message.invoke(Message.java:314) 
~[thrift-server-0.3.7.jar:na]
at com.thinkaurelius.thrift.Message$Invocation.execute(Message.java:90) 
[thrift-server-0.3.7.jar:na]
at 
com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:695)
 [thrift-server-0.3.7.jar:na]
at 
com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:689)
 [thrift-server-0.3.7.jar:na]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:112) 
[disruptor-3.0.1.jar:na]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_102]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_102]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
{noformat}

But that happens within the {{invoke}} method's try/catch block which 
essentially just swallows it, but doesn't "kill" the current thread.


was (Author: s_delima):
There's also this {{NullPointerException}} possible

{noformat}
ERROR [RPC-Thread:68] 2017-01-18 18:28:50,879 Message.java:324 - Unexpected 
throwable while invoking!
java.lang.NullPointerException: null
at com.thinkaurelius.thrift.util.mem.Buffer.size(Buffer.java:83) 
~[thrift-server-0.3.7.jar:na]
at 
com.thinkaurelius.thrift.util.mem.FastMemoryOutputTransport.expand(FastMemoryOutputTransport.java:84)
 ~[thrift-server-0.3.7.jar:na]
at 
com.thinkaurelius.thrift.util.mem.FastMemoryOutputTransport.write(FastMemoryOutputTransport.java:167)
 ~[thrift-server-0.3.7.jar:na]
at 
org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:156) 
~[libthrift-0.9.2.jar:0.9.2]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:55) 
~[libthrift-0.9.2.jar:0.9.2]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) 
~[libthrift-0.9.2.jar:0.9.2]
at com.thinkaurelius.thrift.Message.invoke(Message.java:314) 
~[thrift-server-0.3.7.jar:na]
at com.thinkaurelius.thrift.Message$Invocation.execute(Message.java:90) 
[thrift-server-0.3.7.jar:na]
at 
com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:695)
 [thrift-server-0.3.7.jar:na]
at 
com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:689)
 [thrift-server-0.3.7.jar:na]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:112) 
[disruptor-3.0.1.jar:na]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_102]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_102]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
{noformat}

But that happens within the {{invoke}} method's try/catch block which 
essentially just swallows it, but doesn't "kill" the current thread.

> nodetool disablethrift deadlocks if THsHaDisruptorServer is stopped while a 
> read is going on
> 
>
> Key: CASSANDRA-13137
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13137
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: 2.2.9
>Reporter: Sotirios Delimanolis
>
> We are using Thrift with {{rpc_server_type}} set to {{hsha}}. This creates a 
> {{THsHaDisruptorServer}} which is a subclass of 
> [{{TDisruptorServer}}|https://github.com/xedin/disruptor_thrift_server/blob/master/src/main/java/com/thinkaurelius/thrift/TDisruptorServer.java].
> 

[jira] [Updated] (CASSANDRA-13137) nodetool disablethrift deadlocks if THsHaDisruptorServer is stopped while a read is going on

[Sotirios Delimanolis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sotirios Delimanolis updated CASSANDRA-13137:
-
Description: 
We are using Thrift with {{rpc_server_type}} set to {{hsha}}. This creates a 
{{THsHaDisruptorServer}} which is a subclass of 
[{{TDisruptorServer}}|https://github.com/xedin/disruptor_thrift_server/blob/master/src/main/java/com/thinkaurelius/thrift/TDisruptorServer.java].

Internally, this spawns {{number_of_cores}} number of selector threads. Each 
gets a {{RingBuffer}} and {{rpc_max_threads / cores}} number of worker threads 
(the {{RPC-Thread}} threads). As the server starts receiving requests, each 
selector thread adds events to its {{RingBuffer}} and the worker threads 
process them. 

The _events_ are 
[{{Message}}|https://github.com/xedin/disruptor_thrift_server/blob/master/src/main/java/com/thinkaurelius/thrift/Message.java]
 instances, which have preallocated buffers for eventual IO.

When the thrift server starts up, the corresponding {{ThriftServerThread}} 
joins on the selector threads, waiting for them to die. It then iterates 
through all the {{SelectorThread}} objects and calls their {{shutdown}} method 
which attempts to drain their corresponding {{RingBuffer}}. The [drain 
({{drainAndHalt}})|https://github.com/LMAX-Exchange/disruptor/blob/master/src/main/java/com/lmax/disruptor/WorkerPool.java#L147]
 works by letting the worker pool "consumer" threads catch up to the "producer" 
index, ie. the selector thread.

When we execute a {{nodetool disablethrift}}, it attempts to {{stop}} the 
{{THsHaDisruptorServer}}. That works by setting a {{stopped}} flag to {{true}}. 
When the selector threads see that, they break from their {{select()}} loop, 
and clean up their resources, ie. the {{Message}} objects they've created and 
their buffers. *However*, if one of those {{Message}} objects is currently 
being used by a worker pool thread to process a request, if it calls [this 
piece of 
code|https://github.com/xedin/disruptor_thrift_server/blob/master/src/main/java/com/thinkaurelius/thrift/Message.java#L317],
 you'll get the following {{NullPointerException}}

{noformat}
Jan 18, 2017 6:28:50 PM com.lmax.disruptor.FatalExceptionHandler 
handleEventException
SEVERE: Exception processing: 633124 
com.thinkaurelius.thrift.Message$Invocation@25c9fbeb
java.lang.NullPointerException
at com.thinkaurelius.thrift.Message.getInputTransport(Message.java:338)
at com.thinkaurelius.thrift.Message.invoke(Message.java:308)
at com.thinkaurelius.thrift.Message$Invocation.execute(Message.java:90)
at 
com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:695)
at 
com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:689)
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:112)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}

That fails because it tries to dereference one of the {{Message}} "cleaned up", 
ie. {{null}}, buffers.

Because that call is outside the {{try}} block, the exception escapes and 
basically kills the worker pool thread. This has the side effect of 
"discarding" one of the consumers of a selector's {{RingBuffer}}. 

That has the side effect of preventing the {{ThriftServerThread}} from draining 
the {{RingBuffer}} (and dying) since the consumers never catch up to the 
stopped producer. And that finally has the effect of preventing the {{nodetool 
disablethrift}} from proceeding since it's trying to {{join}} the 
{{ThriftServerThread}}. Deadlock!

The {{ThriftServerThread}} thread looks like

{noformat}
"Thread-1" #2234 prio=5 os_prio=0 tid=0x7f4ae6ff1000 nid=0x2eb6 runnable 
[0x7f4729174000]
   java.lang.Thread.State: RUNNABLE
at java.lang.Thread.yield(Native Method)
at com.lmax.disruptor.WorkerPool.drainAndHalt(WorkerPool.java:147)
at 
com.thinkaurelius.thrift.TDisruptorServer$SelectorThread.shutdown(TDisruptorServer.java:633)
at 
com.thinkaurelius.thrift.TDisruptorServer.gracefullyShutdownInvokerPool(TDisruptorServer.java:301)
at 
com.thinkaurelius.thrift.TDisruptorServer.waitForShutdown(TDisruptorServer.java:280)
at 
org.apache.thrift.server.AbstractNonblockingServer.serve(AbstractNonblockingServer.java:95)
at 
org.apache.cassandra.thrift.ThriftServer$ThriftServerThread.run(ThriftServer.java:137)
{noformat}

The {{nodetool disablethrift}} thread looks like

{noformat}
"RMI TCP Connection(18183)-127.0.0.1" #12121 daemon prio=5 os_prio=0 
tid=0x7f4ac2c61000 nid=0x5805 in Object.wait() [0x7f4aab7ec000]
   java.lang.Thread.State: WAITING (on object monitor)
at 

[jira] [Updated] (CASSANDRA-13137) nodetool disablethrift deadlocks if THsHaDisruptorServer is stopped while a read is going on

[Sotirios Delimanolis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sotirios Delimanolis updated CASSANDRA-13137:
-
Description: 
We are using Thrift with {{rpc_server_type}} set to {{hsha}}. This creates a 
{{THsHaDisruptorServer}} which is a subclass of 
[{{TDisruptorServer}}|https://github.com/xedin/disruptor_thrift_server/blob/master/src/main/java/com/thinkaurelius/thrift/TDisruptorServer.java].

Internally, this spawns {{number_of_cores}} number of selector threads. Each 
gets a {{RingBuffer}} and {{rpc_max_threads / cores}} number of worker threads 
(the {{RPC-Thread}} threads). As the server starts receiving requests, each 
selector thread adds events to its {{RingBuffer}} and the worker threads 
process them. 

The _events_ are 
[{{Message}}|https://github.com/xedin/disruptor_thrift_server/blob/master/src/main/java/com/thinkaurelius/thrift/Message.java]
 instances, which have preallocated buffers for eventual IO.

When the thrift server starts up, the corresponding {{ThriftServerThread}} 
joins on the selector threads, waiting for them to die. It then iterates 
through all the {{SelectorThread}} objects and calls their {{shutdown}} method 
which attempts to drain their corresponding {{RingBuffer}}. The [drain 
({{drainAndHalt}})|https://github.com/LMAX-Exchange/disruptor/blob/master/src/main/java/com/lmax/disruptor/WorkerPool.java#L147]
 works by letting the worker pool "consumer" threads catch up to the "producer" 
index, ie. the selector thread.

When we execute a {{nodetool disablethrift}}, it attempts to {{stop}} the 
{{THsHaDisruptorServer}}. That works by setting a {{stopped}} flag to {{true}}. 
When the selector threads see that, they break from their {{select()}} loop, 
and clean up their resources, ie. the {{Message}} objects they've created and 
their buffers. *However*, if one of those {{Message}} objects is currently 
being used by a worker pool thread to process a request, if it calls [this 
piece of 
code|https://github.com/xedin/disruptor_thrift_server/blob/master/src/main/java/com/thinkaurelius/thrift/Message.java#L317],
 you'll get the following {{NullPointerException}}

{noformat}
Jan 18, 2017 6:28:50 PM com.lmax.disruptor.FatalExceptionHandler 
handleEventException
SEVERE: Exception processing: 633124 
com.thinkaurelius.thrift.Message$Invocation@25c9fbeb
java.lang.NullPointerException
at com.thinkaurelius.thrift.Message.getInputTransport(Message.java:338)
at com.thinkaurelius.thrift.Message.invoke(Message.java:308)
at com.thinkaurelius.thrift.Message$Invocation.execute(Message.java:90)
at 
com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:695)
at 
com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:689)
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:112)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}

That fails because it tries to dereference one of the {{Message}} "cleaned up", 
ie. {{null}}, buffers.

Because that call is outside the {{try}} block, the exception escapes and 
basically kills the worker pool thread. This has the side effect of 
"discarding" one of the consumers of a selector's {{RingBuffer}}. 

*That* has the side effect of preventing the {{ThriftServerThread}} from 
draining the {{RingBuffer}} (and dying) since the consumers never catch up to 
the stopped producer. And that finally has the effect of preventing the 
{{nodetool disablethrift}} from proceeding since it's trying to {{join}} the 
{{ThriftServerThread}}. Deadlock!

The {{ThriftServerThread}} thread looks like

{noformat}
"Thread-1" #2234 prio=5 os_prio=0 tid=0x7f4ae6ff1000 nid=0x2eb6 runnable 
[0x7f4729174000]
   java.lang.Thread.State: RUNNABLE
at java.lang.Thread.yield(Native Method)
at com.lmax.disruptor.WorkerPool.drainAndHalt(WorkerPool.java:147)
at 
com.thinkaurelius.thrift.TDisruptorServer$SelectorThread.shutdown(TDisruptorServer.java:633)
at 
com.thinkaurelius.thrift.TDisruptorServer.gracefullyShutdownInvokerPool(TDisruptorServer.java:301)
at 
com.thinkaurelius.thrift.TDisruptorServer.waitForShutdown(TDisruptorServer.java:280)
at 
org.apache.thrift.server.AbstractNonblockingServer.serve(AbstractNonblockingServer.java:95)
at 
org.apache.cassandra.thrift.ThriftServer$ThriftServerThread.run(ThriftServer.java:137)
{noformat}

The {{nodetool disablethrift}} thread looks like

{noformat}
"RMI TCP Connection(18183)-127.0.0.1" #12121 daemon prio=5 os_prio=0 
tid=0x7f4ac2c61000 nid=0x5805 in Object.wait() [0x7f4aab7ec000]
   java.lang.Thread.State: WAITING (on object monitor)
at 

[jira] [Updated] (CASSANDRA-13137) nodetool disablethrift deadlocks if THsHaDisruptorServer is stopped while a read is going on

[Sotirios Delimanolis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sotirios Delimanolis updated CASSANDRA-13137:
-
Description: 
We are using Thrift with {{rpc_server_type}} set to {{hsha}}. This creates a 
{{THsHaDisruptorServer}} which is a subclass of 
[{{TDisruptorServer}}|https://github.com/xedin/disruptor_thrift_server/blob/master/src/main/java/com/thinkaurelius/thrift/TDisruptorServer.java].

Internally, this spawns {{number_of_cores}} number of selector threads. Each 
gets a {{RingBuffer}} and {{rpc_max_threads / cores}} number of worker threads 
(the {{RPC-Thread}} threads). As the server starts receiving requests, each 
selector thread adds events to its {{RingBuffer}} and the worker threads 
process them. 

The _events_ are 
[{{Message}}|https://github.com/xedin/disruptor_thrift_server/blob/master/src/main/java/com/thinkaurelius/thrift/Message.java]
 instances, which have preallocated buffers for eventual IO.

When the thrift server starts up, the corresponding {{ThriftServerThread}} 
joins on the selector threads, waiting for them to die. It then iterates 
through all the {{SelectorThread}} objects and calls their {{shutdown}} method 
which attempts to drain their corresponding {{RingBuffer}}. The [drain 
({{drainAndHalt}})|https://github.com/LMAX-Exchange/disruptor/blob/master/src/main/java/com/lmax/disruptor/WorkerPool.java#L147]
 works by letting the worker pool "consumer" threads catch up to the "producer" 
index, ie. the selector thread.

When we execute a {{nodetool disablethrift}}, this attempts to {{stop}} the 
{{THsHaDisruptorServer}}. That works by setting a {{stopped}} flag to {{true}}. 
When the selector threads see that, they break from their {{select()}} loop, 
and clean up their resources, ie. the {{Message}} objects they've created and 
their buffers. *However*, if one of those {{Message}} objects is currently 
being used by a worker pool thread to process a request, if it calls [this 
piece of 
code|https://github.com/xedin/disruptor_thrift_server/blob/master/src/main/java/com/thinkaurelius/thrift/Message.java#L317],
 you'll get the following {{NullPointerException}}

{noformat}
Jan 18, 2017 6:28:50 PM com.lmax.disruptor.FatalExceptionHandler 
handleEventException
SEVERE: Exception processing: 633124 
com.thinkaurelius.thrift.Message$Invocation@25c9fbeb
java.lang.NullPointerException
at com.thinkaurelius.thrift.Message.getInputTransport(Message.java:338)
at com.thinkaurelius.thrift.Message.invoke(Message.java:308)
at com.thinkaurelius.thrift.Message$Invocation.execute(Message.java:90)
at 
com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:695)
at 
com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:689)
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:112)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}

That fails because it tries to dereference one of the {{Message}} "cleaned up", 
ie. {{null}}, buffers.

Because that call is outside the {{try}} block, the exception escapes and 
basically kills the worker pool thread. This has the side effect of 
"discarding" one of the consumers of a selector's {{RingBuffer}}. 

That has the side effect of preventing the {{ThriftServerThread}} from draining 
the {{RingBuffer}} (and dying) since the consumers never catch up to the 
stopped producer. And that finally has the effect of preventing the {{nodetool 
disablethrift}} from proceeding since it's trying to {{join}} the 
{{ThriftServerThread}}. Deadlock!

The {{ThriftServerThread}} thread looks like

{noformat}
"Thread-1" #2234 prio=5 os_prio=0 tid=0x7f4ae6ff1000 nid=0x2eb6 runnable 
[0x7f4729174000]
   java.lang.Thread.State: RUNNABLE
at java.lang.Thread.yield(Native Method)
at com.lmax.disruptor.WorkerPool.drainAndHalt(WorkerPool.java:147)
at 
com.thinkaurelius.thrift.TDisruptorServer$SelectorThread.shutdown(TDisruptorServer.java:633)
at 
com.thinkaurelius.thrift.TDisruptorServer.gracefullyShutdownInvokerPool(TDisruptorServer.java:301)
at 
com.thinkaurelius.thrift.TDisruptorServer.waitForShutdown(TDisruptorServer.java:280)
at 
org.apache.thrift.server.AbstractNonblockingServer.serve(AbstractNonblockingServer.java:95)
at 
org.apache.cassandra.thrift.ThriftServer$ThriftServerThread.run(ThriftServer.java:137)
{noformat}

The {{nodetool disablethrift}} thread looks like

{noformat}
"RMI TCP Connection(18183)-127.0.0.1" #12121 daemon prio=5 os_prio=0 
tid=0x7f4ac2c61000 nid=0x5805 in Object.wait() [0x7f4aab7ec000]
   java.lang.Thread.State: WAITING (on object monitor)
at 

[jira] [Updated] (CASSANDRA-13137) nodetool disablethrift deadlocks if THsHaDisruptorServer is stopped while a read is going on

[Sotirios Delimanolis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sotirios Delimanolis updated CASSANDRA-13137:
-
Description: 
We are using Thrift with {{rpc_server_type}} set to {{hsha}}. This creates a 
{{THsHaDisruptorServer}} which is a subclass of 
[{{TDisruptorServer}}|https://github.com/xedin/disruptor_thrift_server/blob/master/src/main/java/com/thinkaurelius/thrift/TDisruptorServer.java].

Internally, this spawns {{number_of_cores}} number of selector threads. Each 
gets a {{RingBuffer}} and {{rpc_max_threads / cores}} number of worker threads 
(the {{RPC-Thread}} threads). As the server starts receiving requests, each 
selector thread adds events to its {{RingBuffer}} and the worker threads 
process them. 

The _events_ are 
[{{Message}}|https://github.com/xedin/disruptor_thrift_server/blob/master/src/main/java/com/thinkaurelius/thrift/Message.java]
 instances, which have preallocated buffers for eventual IO.

When the thrift server starts up, the corresponding {{ThriftServerThread}} 
joins on the selector threads, waiting for them to die. It then iterates 
through all the {{SelectorThread}} objects and calls their {{shutdown}} method 
which attempts to drain their corresponding {{RingBuffer}}. The [drain 
({{drainAndHalt}})|https://github.com/LMAX-Exchange/disruptor/blob/master/src/main/java/com/lmax/disruptor/WorkerPool.java#L147]
 works by letting the worker pool threads catch up to the "producer" index, ie. 
the selector thread.

When we execute a {{nodetool disablethrift}}, this attempts to {{stop}} the 
{{THsHaDisruptorServer}}. That works by setting a {{stopped}} flag to {{true}}. 
When the selector threads see that, they break from their {{select()}} loop, 
and clean up their resources, ie. the {{Message}} objects they've created and 
their buffers. *However*, if one of those {{Message}} objects is currently 
being used by a worker pool thread to process a request, if it calls [this 
piece of 
code|https://github.com/xedin/disruptor_thrift_server/blob/master/src/main/java/com/thinkaurelius/thrift/Message.java#L317],
 you'll get the following {{NullPointerException}}

{noformat}
Jan 18, 2017 6:28:50 PM com.lmax.disruptor.FatalExceptionHandler 
handleEventException
SEVERE: Exception processing: 633124 
com.thinkaurelius.thrift.Message$Invocation@25c9fbeb
java.lang.NullPointerException
at com.thinkaurelius.thrift.Message.getInputTransport(Message.java:338)
at com.thinkaurelius.thrift.Message.invoke(Message.java:308)
at com.thinkaurelius.thrift.Message$Invocation.execute(Message.java:90)
at 
com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:695)
at 
com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:689)
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:112)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}

That fails because it tries to dereference one of the {{Message}} "cleaned up", 
ie. {{null}}, buffers.

Because that call is outside the {{try}} block, the exception escapes and 
basically kills the worker pool thread. This has the side effect of 
"discarding" one of the consumers of a selector's {{RingBuffer}}. 

That has the side effect of preventing the {{ThriftServerThread}} from draining 
the {{RingBuffer}} (and dying) since the consumers never catch up to the 
stopped producer. And that finally has the effect of preventing the {{nodetool 
disablethrift}} from proceeding since it's trying to {{join}} the 
{{ThriftServerThread}}. Deadlock!

The {{ThriftServerThread}} thread looks like

{noformat}
"Thread-1" #2234 prio=5 os_prio=0 tid=0x7f4ae6ff1000 nid=0x2eb6 runnable 
[0x7f4729174000]
   java.lang.Thread.State: RUNNABLE
at java.lang.Thread.yield(Native Method)
at com.lmax.disruptor.WorkerPool.drainAndHalt(WorkerPool.java:147)
at 
com.thinkaurelius.thrift.TDisruptorServer$SelectorThread.shutdown(TDisruptorServer.java:633)
at 
com.thinkaurelius.thrift.TDisruptorServer.gracefullyShutdownInvokerPool(TDisruptorServer.java:301)
at 
com.thinkaurelius.thrift.TDisruptorServer.waitForShutdown(TDisruptorServer.java:280)
at 
org.apache.thrift.server.AbstractNonblockingServer.serve(AbstractNonblockingServer.java:95)
at 
org.apache.cassandra.thrift.ThriftServer$ThriftServerThread.run(ThriftServer.java:137)
{noformat}

The {{nodetool disablethrift}} thread looks like

{noformat}
"RMI TCP Connection(18183)-127.0.0.1" #12121 daemon prio=5 os_prio=0 
tid=0x7f4ac2c61000 nid=0x5805 in Object.wait() [0x7f4aab7ec000]
   java.lang.Thread.State: WAITING (on object monitor)
at 

[jira] [Created] (CASSANDRA-13137) nodetool disablethrift deadlocks if THsHaDisruptorServer is stopped while a read is going on

[Sotirios Delimanolis (JIRA)]

Sotirios Delimanolis created CASSANDRA-13137:


 Summary: nodetool disablethrift deadlocks if THsHaDisruptorServer 
is stopped while a read is going on
 Key: CASSANDRA-13137
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13137
 Project: Cassandra
  Issue Type: Bug
  Components: Core
 Environment: 2.2.9
Reporter: Sotirios Delimanolis


We are using Thrift with {{rpc_server_type}} set to {{hsha}}. This creates a 
{{THsHaDisruptorServer}} which is a subclass of 
[{{TDisruptorServer}}|https://github.com/xedin/disruptor_thrift_server/blob/master/src/main/java/com/thinkaurelius/thrift/TDisruptorServer.java].

Internally, this spawns {{cores}} number of selector threads. Each gets a 
{{RingBuffer}} and {{rpc_max_threads / cores}} number of worker threads (the 
{{RPC-Thread}} threads). As the server starts receiving requests, each selector 
thread adds events to its {{RingBuffer}} and the worker threads process them. 

The _events_ are 
[{{Message}}|https://github.com/xedin/disruptor_thrift_server/blob/master/src/main/java/com/thinkaurelius/thrift/Message.java]
 instances, which have preallocated buffers for eventual IO.

When the thrift server starts up, the corresponding {{ThriftServerThread}} 
joins on the selector threads, waiting for them to die. It then iterates 
through all the {{SelectorThread}} objects and calls their {{shutdown}} method 
which attempts to drain their corresponding {{RingBuffer}}. The [drain 
({{drainAndHalt}})|https://github.com/LMAX-Exchange/disruptor/blob/master/src/main/java/com/lmax/disruptor/WorkerPool.java#L147]
 works by letting the worker pool threads catch up to the "producer" index, ie. 
the selector thread.

When we execute a {{nodetool disablethrift}}, this attempts to {{stop}} the 
{{THsHaDisruptorServer}}. That works by setting a {{stopped}} flag to {{true}}. 
When the selector threads see that, they break from their {{select()}} loop, 
and clean up their resources, ie. the {{Message}} objects they've created and 
their buffers. *However*, if one of those {{Message}} objects is currently 
being used by a worker pool thread to process a request, if it calls [this 
piece of 
code|https://github.com/xedin/disruptor_thrift_server/blob/master/src/main/java/com/thinkaurelius/thrift/Message.java#L317],
 you'll get the following {{NullPointerException}}

{noformat}
Jan 18, 2017 6:28:50 PM com.lmax.disruptor.FatalExceptionHandler 
handleEventException
SEVERE: Exception processing: 633124 
com.thinkaurelius.thrift.Message$Invocation@25c9fbeb
java.lang.NullPointerException
at com.thinkaurelius.thrift.Message.getInputTransport(Message.java:338)
at com.thinkaurelius.thrift.Message.invoke(Message.java:308)
at com.thinkaurelius.thrift.Message$Invocation.execute(Message.java:90)
at 
com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:695)
at 
com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:689)
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:112)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}

That fails because it tries to dereference one of the {{Message}} "cleaned up", 
ie. {{null}}, buffers.

Because that call is outside the {{try}} block, the exception escapes and 
basically kills the worker pool thread. This has the side effect of 
"discarding" one of the consumers of a selector's {{RingBuffer}}. 

That has the side effect of preventing the {{ThriftServerThread}} from draining 
the {{RingBuffer}} (and dying) since the consumers never catch up to the 
stopped producer. And that finally has the effect of preventing the {{nodetool 
disablethrift}} from proceeding since it's trying to {{join}} the 
{{ThriftServerThread}}. Deadlock!

The {{ThriftServerThread}} thread looks like

{noformat}
"Thread-1" #2234 prio=5 os_prio=0 tid=0x7f4ae6ff1000 nid=0x2eb6 runnable 
[0x7f4729174000]
   java.lang.Thread.State: RUNNABLE
at java.lang.Thread.yield(Native Method)
at com.lmax.disruptor.WorkerPool.drainAndHalt(WorkerPool.java:147)
at 
com.thinkaurelius.thrift.TDisruptorServer$SelectorThread.shutdown(TDisruptorServer.java:633)
at 
com.thinkaurelius.thrift.TDisruptorServer.gracefullyShutdownInvokerPool(TDisruptorServer.java:301)
at 
com.thinkaurelius.thrift.TDisruptorServer.waitForShutdown(TDisruptorServer.java:280)
at 
org.apache.thrift.server.AbstractNonblockingServer.serve(AbstractNonblockingServer.java:95)
at 
org.apache.cassandra.thrift.ThriftServer$ThriftServerThread.run(ThriftServer.java:137)
{noformat}

The {{nodetool disablethrift}} thread looks like

[jira] [Updated] (CASSANDRA-13075) Indexer is not correctly invoked when building indexes over sstables

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown updated CASSANDRA-13075:

   Resolution: Fixed
Fix Version/s: 3.11
   4.0
   3.0.11
   Status: Resolved  (was: Ready to Commit)

committed to 3.0, 3.11, and trunk as sha 
{{7a06df79d829ac073264045eb9420a61c5ba939a}}. Thanks!

> Indexer is not correctly invoked when building indexes over sstables
> 
>
> Key: CASSANDRA-13075
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13075
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Sergio Bossa
>Assignee: Alex Petrov
>Priority: Critical
> Fix For: 3.0.11, 4.0, 3.11
>
> Attachments: CustomIndexTest.java
>
>
> Following CASSANDRA-12796, {{SecondaryIndexManager#indexPartition()}} calls 
> each {{Indexer}} {{begin}} and {{finish}} methods multiple times per 
> partition (depending on the page size), as 
> {{PartitionIterators#getOnlyElement()}} returns an empty partition even when 
> the iterator is exhausted.
> This leads to bugs for {{Indexer}} implementations doing actual work in those 
>  methods, but even worse, it provides the {{Indexer}} the same input of an 
> empty partition containing only a non-live partition deletion, as the 
> {{Indexer#partitionDelete()}} method is *not* actually called.
> My proposed solution:
> 1) Stop the iteration before the empty partition is returned and ingested 
> into the {{Indexer}}.
> 2) Actually call the {{Indexer#partitionDelete()}} method inside 
> {{SecondaryIndexManager#indexPartition()}} (which requires to use a filtered 
> iterator so it actually contains the deletion info).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[jasobrown]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e3d26b6a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e3d26b6a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e3d26b6a

Branch: refs/heads/trunk
Commit: e3d26b6aa18a48a52ec4f5511bdc484a45ec3ce4
Parents: 74559de 7a06df7
Author: Jason Brown 
Authored: Thu Jan 19 14:08:37 2017 -0800
Committer: Jason Brown 
Committed: Thu Jan 19 14:13:21 2017 -0800

--
 CHANGES.txt |   1 +
 .../cassandra/index/SecondaryIndexManager.java  |  86 ++-
 .../service/pager/AbstractQueryPager.java   |  56 --
 .../apache/cassandra/index/CustomIndexTest.java | 106 +++
 .../org/apache/cassandra/index/StubIndex.java   |   4 +
 5 files changed, 216 insertions(+), 37 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e3d26b6a/CHANGES.txt
--
diff --cc CHANGES.txt
index 8b613e6,a85386b..224ef5d
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,121 -1,12 +1,122 @@@
 -3.0.11
 +3.10
 + * Fix TestHintedHandoff.hintedhandoff_decom_test (CASSANDRA-13058)
 + * Fixed query monitoring for range queries (CASSANDRA-13050)
 + * Remove outboundBindAny configuration property (CASSANDRA-12673)
 + * Use correct bounds for all-data range when filtering (CASSANDRA-12666)
 + * Remove timing window in test case (CASSANDRA-12875)
 + * Resolve unit testing without JCE security libraries installed 
(CASSANDRA-12945)
 + * Fix inconsistencies in cassandra-stress load balancing policy 
(CASSANDRA-12919)
 + * Fix validation of non-frozen UDT cells (CASSANDRA-12916)
 + * Don't shut down socket input/output on StreamSession (CASSANDRA-12903)
 + * Fix Murmur3PartitionerTest (CASSANDRA-12858)
 + * Move cqlsh syntax rules into separate module and allow easier 
customization (CASSANDRA-12897)
 + * Fix CommitLogSegmentManagerTest (CASSANDRA-12283)
 + * Fix cassandra-stress truncate option (CASSANDRA-12695)
 + * Fix crossNode value when receiving messages (CASSANDRA-12791)
 + * Don't load MX4J beans twice (CASSANDRA-12869)
 + * Extend native protocol request flags, add versions to SUPPORTED, and 
introduce ProtocolVersion enum (CASSANDRA-12838)
 + * Set JOINING mode when running pre-join tasks (CASSANDRA-12836)
 + * remove net.mintern.primitive library due to license issue (CASSANDRA-12845)
 + * Properly format IPv6 addresses when logging JMX service URL 
(CASSANDRA-12454)
 + * Optimize the vnode allocation for single replica per DC (CASSANDRA-12777)
 + * Use non-token restrictions for bounds when token restrictions are 
overridden (CASSANDRA-12419)
 + * Fix CQLSH auto completion for PER PARTITION LIMIT (CASSANDRA-12803)
 + * Use different build directories for Eclipse and Ant (CASSANDRA-12466)
 + * Avoid potential AttributeError in cqlsh due to no table metadata 
(CASSANDRA-12815)
 + * Fix RandomReplicationAwareTokenAllocatorTest.testExistingCluster 
(CASSANDRA-12812)
 + * Upgrade commons-codec to 1.9 (CASSANDRA-12790)
 + * Make the fanout size for LeveledCompactionStrategy to be configurable 
(CASSANDRA-11550)
 + * Add duration data type (CASSANDRA-11873)
 + * Fix timeout in ReplicationAwareTokenAllocatorTest (CASSANDRA-12784)
 + * Improve sum aggregate functions (CASSANDRA-12417)
 + * Make cassandra.yaml docs for batch_size_*_threshold_in_kb reflect changes 
in CASSANDRA-10876 (CASSANDRA-12761)
 + * cqlsh fails to format collections when using aliases (CASSANDRA-11534)
 + * Check for hash conflicts in prepared statements (CASSANDRA-12733)
 + * Exit query parsing upon first error (CASSANDRA-12598)
 + * Fix cassandra-stress to use single seed in UUID generation 
(CASSANDRA-12729)
 + * CQLSSTableWriter does not allow Update statement (CASSANDRA-12450)
 + * Config class uses boxed types but DD exposes primitive types 
(CASSANDRA-12199)
 + * Add pre- and post-shutdown hooks to Storage Service (CASSANDRA-12461)
 + * Add hint delivery metrics (CASSANDRA-12693)
 + * Remove IndexInfo cache from FileIndexInfoRetriever (CASSANDRA-12731)
 + * ColumnIndex does not reuse buffer (CASSANDRA-12502)
 + * cdc column addition still breaks schema migration tasks (CASSANDRA-12697)
 + * Upgrade metrics-reporter dependencies (CASSANDRA-12089)
 + * Tune compaction thread count via nodetool (CASSANDRA-12248)
 + * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232)
 + * Include repair session IDs in repair start message (CASSANDRA-12532)
 + * Add a blocking task to Index, run before joining the ring (CASSANDRA-12039)
 + * Fix NPE when using CQLSSTableWriter (CASSANDRA-12667)
 + * Support optional backpressure strategies at the coordinator 

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[jasobrown]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d6da7b79
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d6da7b79
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d6da7b79

Branch: refs/heads/trunk
Commit: d6da7b79903183e1b21f447098a1e01034152bb1
Parents: c3d7244 e3d26b6
Author: Jason Brown 
Authored: Thu Jan 19 14:13:40 2017 -0800
Committer: Jason Brown 
Committed: Thu Jan 19 14:14:24 2017 -0800

--
 CHANGES.txt |   1 +
 .../cassandra/index/SecondaryIndexManager.java  |  86 ++-
 .../service/pager/AbstractQueryPager.java   |  56 --
 .../apache/cassandra/index/CustomIndexTest.java | 106 +++
 .../org/apache/cassandra/index/StubIndex.java   |   4 +
 5 files changed, 216 insertions(+), 37 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d6da7b79/CHANGES.txt
--
diff --cc CHANGES.txt
index cc16a8f,224ef5d..ebf161d
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -139,12 -109,14 +139,13 @@@
   * Remove pre-startup check for open JMX port (CASSANDRA-12074)
   * Remove compaction Severity from DynamicEndpointSnitch (CASSANDRA-11738)
   * Restore resumable hints delivery (CASSANDRA-11960)
 - * Properly report LWT contention (CASSANDRA-12626)
 + * Properly record CAS contention (CASSANDRA-12626)
  Merged from 3.0:
+  * Indexer is not correctly invoked when building indexes over sstables 
(CASSANDRA-13075)
 + * Stress daemon help is incorrect (CASSANDRA-12563)
   * Read repair is not blocking repair to finish in foreground repair 
(CASSANDRA-13115)
 - * Stress daemon help is incorrect(CASSANDRA-12563)
 - * Remove ALTER TYPE support (CASSANDRA-12443)
 - * Fix assertion for certain legacy range tombstone pattern (CASSANDRA-12203)
   * Replace empty strings with null values if they cannot be converted 
(CASSANDRA-12794)
 + * Remove support for non-JavaScript UDFs (CASSANDRA-12883)
   * Fix deserialization of 2.x DeletedCells (CASSANDRA-12620)
   * Add parent repair session id to anticompaction log message 
(CASSANDRA-12186)
   * Improve contention handling on failure to acquire MV lock for streaming 
and hints (CASSANDRA-12905)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d6da7b79/src/java/org/apache/cassandra/service/pager/AbstractQueryPager.java
--
diff --cc src/java/org/apache/cassandra/service/pager/AbstractQueryPager.java
index 22ddc84,e937a0d..8a04108
--- a/src/java/org/apache/cassandra/service/pager/AbstractQueryPager.java
+++ b/src/java/org/apache/cassandra/service/pager/AbstractQueryPager.java
@@@ -78,10 -77,49 +77,49 @@@ abstract class AbstractQueryPager imple
  return 
Transformation.apply(nextPageReadCommand(pageSize).executeInternal(executionController),
 pager);
  }
  
- private class Pager extends Transformation
+ public UnfilteredPartitionIterator fetchPageUnfiltered(CFMetaData cfm, 
int pageSize, ReadExecutionController executionController)
+ {
+ if (isExhausted())
 -return EmptyIterators.unfilteredPartition(cfm, false);
++return EmptyIterators.unfilteredPartition(cfm);
+ 
+ pageSize = Math.min(pageSize, remaining);
+ UnfilteredPager pager = new 
UnfilteredPager(limits.forPaging(pageSize), command.nowInSec());
+ 
+ return 
Transformation.apply(nextPageReadCommand(pageSize).executeLocally(executionController),
 pager);
+ }
+ 
+ private class UnfilteredPager extends Pager
+ {
+ 
+ private UnfilteredPager(DataLimits pageLimits, int nowInSec)
+ {
+ super(pageLimits, nowInSec);
+ }
+ 
+ protected BaseRowIterator 
apply(BaseRowIterator partition)
+ {
+ return 
Transformation.apply(counter.applyTo((UnfilteredRowIterator) partition), this);
+ }
+ }
+ 
+ private class RowPager extends Pager
+ {
+ 
+ private RowPager(DataLimits pageLimits, int nowInSec)
+ {
+ super(pageLimits, nowInSec);
+ }
+ 
+ protected BaseRowIterator apply(BaseRowIterator partition)
+ {
+ return Transformation.apply(counter.applyTo((RowIterator) 
partition), this);
+ }
+ }
+ 
+ private abstract class Pager extends 
Transformation
  {
  private final DataLimits pageLimits;
- private final DataLimits.Counter counter;
+ protected final DataLimits.Counter counter;
  private DecoratedKey currentKey;
  private Row lastRow;
  private boolean isFirstPartition = true;

[2/6] cassandra git commit: Use unfiltered iterator for partition indexing

[jasobrown]

Use unfiltered iterator for partition indexing

Patch by Alex Petrov; reviewed by Sergio Bossa for CASSANDRA-13075;


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7a06df79
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7a06df79
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7a06df79

Branch: refs/heads/cassandra-3.11
Commit: 7a06df79d829ac073264045eb9420a61c5ba939a
Parents: 48fed80
Author: Alex Petrov 
Authored: Thu Jan 19 09:49:23 2017 +0100
Committer: Jason Brown 
Committed: Thu Jan 19 10:55:10 2017 -0800

--
 CHANGES.txt |   1 +
 .../cassandra/index/SecondaryIndexManager.java  |  86 ++-
 .../service/pager/AbstractQueryPager.java   |  54 --
 .../apache/cassandra/index/CustomIndexTest.java | 106 +++
 .../org/apache/cassandra/index/StubIndex.java   |   4 +
 5 files changed, 216 insertions(+), 35 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/7a06df79/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 6293cfa..a85386b 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Indexer is not correctly invoked when building indexes over sstables 
(CASSANDRA-13075)
  * Read repair is not blocking repair to finish in foreground repair 
(CASSANDRA-13115)
  * Stress daemon help is incorrect (CASSANDRA-12563)
  * Remove ALTER TYPE support (CASSANDRA-12443)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/7a06df79/src/java/org/apache/cassandra/index/SecondaryIndexManager.java
--
diff --git a/src/java/org/apache/cassandra/index/SecondaryIndexManager.java 
b/src/java/org/apache/cassandra/index/SecondaryIndexManager.java
index a6ed3ba..d39b607 100644
--- a/src/java/org/apache/cassandra/index/SecondaryIndexManager.java
+++ b/src/java/org/apache/cassandra/index/SecondaryIndexManager.java
@@ -48,8 +48,7 @@ import org.apache.cassandra.db.compaction.CompactionManager;
 import org.apache.cassandra.db.filter.RowFilter;
 import org.apache.cassandra.db.lifecycle.SSTableSet;
 import org.apache.cassandra.db.lifecycle.View;
-import org.apache.cassandra.db.partitions.PartitionIterators;
-import org.apache.cassandra.db.partitions.PartitionUpdate;
+import org.apache.cassandra.db.partitions.*;
 import org.apache.cassandra.db.rows.*;
 import org.apache.cassandra.exceptions.InvalidRequestException;
 import org.apache.cassandra.index.internal.CassandraIndex;
@@ -543,35 +542,64 @@ public class SecondaryIndexManager implements 
IndexRegistry
 {
 try (ReadOrderGroup readGroup = cmd.startOrderGroup();
  OpOrder.Group writeGroup = Keyspace.writeOrder.start();
- RowIterator partition =
-
PartitionIterators.getOnlyElement(pager.fetchPageInternal(pageSize,readGroup),
-  cmd))
+ UnfilteredPartitionIterator page = 
pager.fetchPageUnfiltered(baseCfs.metadata, pageSize, readGroup))
 {
-Set indexers = indexes.stream()
- .map(index -> 
index.indexerFor(key,
-   
 partition.columns(),
-   
 nowInSec,
-   
 writeGroup,
-   
 IndexTransaction.Type.UPDATE))
- 
.filter(Objects::nonNull)
- 
.collect(Collectors.toSet());
-
-indexers.forEach(Index.Indexer::begin);
-
-// only process the static row once per partition
-if (!readStatic && !partition.staticRow().isEmpty())
-{
-indexers.forEach(indexer -> 
indexer.insertRow(partition.staticRow()));
-readStatic = true;
+if (!page.hasNext())
+break;
+
+try (UnfilteredRowIterator partition = page.next()) {
+Set indexers = indexes.stream()
+ .map(index -> 
index.indexerFor(key,
+   
 partition.columns(),
+ 

[3/6] cassandra git commit: Use unfiltered iterator for partition indexing

[jasobrown]

Use unfiltered iterator for partition indexing

Patch by Alex Petrov; reviewed by Sergio Bossa for CASSANDRA-13075;


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7a06df79
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7a06df79
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7a06df79

Branch: refs/heads/trunk
Commit: 7a06df79d829ac073264045eb9420a61c5ba939a
Parents: 48fed80
Author: Alex Petrov 
Authored: Thu Jan 19 09:49:23 2017 +0100
Committer: Jason Brown 
Committed: Thu Jan 19 10:55:10 2017 -0800

--
 CHANGES.txt |   1 +
 .../cassandra/index/SecondaryIndexManager.java  |  86 ++-
 .../service/pager/AbstractQueryPager.java   |  54 --
 .../apache/cassandra/index/CustomIndexTest.java | 106 +++
 .../org/apache/cassandra/index/StubIndex.java   |   4 +
 5 files changed, 216 insertions(+), 35 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/7a06df79/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 6293cfa..a85386b 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Indexer is not correctly invoked when building indexes over sstables 
(CASSANDRA-13075)
  * Read repair is not blocking repair to finish in foreground repair 
(CASSANDRA-13115)
  * Stress daemon help is incorrect (CASSANDRA-12563)
  * Remove ALTER TYPE support (CASSANDRA-12443)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/7a06df79/src/java/org/apache/cassandra/index/SecondaryIndexManager.java
--
diff --git a/src/java/org/apache/cassandra/index/SecondaryIndexManager.java 
b/src/java/org/apache/cassandra/index/SecondaryIndexManager.java
index a6ed3ba..d39b607 100644
--- a/src/java/org/apache/cassandra/index/SecondaryIndexManager.java
+++ b/src/java/org/apache/cassandra/index/SecondaryIndexManager.java
@@ -48,8 +48,7 @@ import org.apache.cassandra.db.compaction.CompactionManager;
 import org.apache.cassandra.db.filter.RowFilter;
 import org.apache.cassandra.db.lifecycle.SSTableSet;
 import org.apache.cassandra.db.lifecycle.View;
-import org.apache.cassandra.db.partitions.PartitionIterators;
-import org.apache.cassandra.db.partitions.PartitionUpdate;
+import org.apache.cassandra.db.partitions.*;
 import org.apache.cassandra.db.rows.*;
 import org.apache.cassandra.exceptions.InvalidRequestException;
 import org.apache.cassandra.index.internal.CassandraIndex;
@@ -543,35 +542,64 @@ public class SecondaryIndexManager implements 
IndexRegistry
 {
 try (ReadOrderGroup readGroup = cmd.startOrderGroup();
  OpOrder.Group writeGroup = Keyspace.writeOrder.start();
- RowIterator partition =
-
PartitionIterators.getOnlyElement(pager.fetchPageInternal(pageSize,readGroup),
-  cmd))
+ UnfilteredPartitionIterator page = 
pager.fetchPageUnfiltered(baseCfs.metadata, pageSize, readGroup))
 {
-Set indexers = indexes.stream()
- .map(index -> 
index.indexerFor(key,
-   
 partition.columns(),
-   
 nowInSec,
-   
 writeGroup,
-   
 IndexTransaction.Type.UPDATE))
- 
.filter(Objects::nonNull)
- 
.collect(Collectors.toSet());
-
-indexers.forEach(Index.Indexer::begin);
-
-// only process the static row once per partition
-if (!readStatic && !partition.staticRow().isEmpty())
-{
-indexers.forEach(indexer -> 
indexer.insertRow(partition.staticRow()));
-readStatic = true;
+if (!page.hasNext())
+break;
+
+try (UnfilteredRowIterator partition = page.next()) {
+Set indexers = indexes.stream()
+ .map(index -> 
index.indexerFor(key,
+   
 partition.columns(),
+  

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[jasobrown]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e3d26b6a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e3d26b6a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e3d26b6a

Branch: refs/heads/cassandra-3.11
Commit: e3d26b6aa18a48a52ec4f5511bdc484a45ec3ce4
Parents: 74559de 7a06df7
Author: Jason Brown 
Authored: Thu Jan 19 14:08:37 2017 -0800
Committer: Jason Brown 
Committed: Thu Jan 19 14:13:21 2017 -0800

--
 CHANGES.txt |   1 +
 .../cassandra/index/SecondaryIndexManager.java  |  86 ++-
 .../service/pager/AbstractQueryPager.java   |  56 --
 .../apache/cassandra/index/CustomIndexTest.java | 106 +++
 .../org/apache/cassandra/index/StubIndex.java   |   4 +
 5 files changed, 216 insertions(+), 37 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e3d26b6a/CHANGES.txt
--
diff --cc CHANGES.txt
index 8b613e6,a85386b..224ef5d
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,121 -1,12 +1,122 @@@
 -3.0.11
 +3.10
 + * Fix TestHintedHandoff.hintedhandoff_decom_test (CASSANDRA-13058)
 + * Fixed query monitoring for range queries (CASSANDRA-13050)
 + * Remove outboundBindAny configuration property (CASSANDRA-12673)
 + * Use correct bounds for all-data range when filtering (CASSANDRA-12666)
 + * Remove timing window in test case (CASSANDRA-12875)
 + * Resolve unit testing without JCE security libraries installed 
(CASSANDRA-12945)
 + * Fix inconsistencies in cassandra-stress load balancing policy 
(CASSANDRA-12919)
 + * Fix validation of non-frozen UDT cells (CASSANDRA-12916)
 + * Don't shut down socket input/output on StreamSession (CASSANDRA-12903)
 + * Fix Murmur3PartitionerTest (CASSANDRA-12858)
 + * Move cqlsh syntax rules into separate module and allow easier 
customization (CASSANDRA-12897)
 + * Fix CommitLogSegmentManagerTest (CASSANDRA-12283)
 + * Fix cassandra-stress truncate option (CASSANDRA-12695)
 + * Fix crossNode value when receiving messages (CASSANDRA-12791)
 + * Don't load MX4J beans twice (CASSANDRA-12869)
 + * Extend native protocol request flags, add versions to SUPPORTED, and 
introduce ProtocolVersion enum (CASSANDRA-12838)
 + * Set JOINING mode when running pre-join tasks (CASSANDRA-12836)
 + * remove net.mintern.primitive library due to license issue (CASSANDRA-12845)
 + * Properly format IPv6 addresses when logging JMX service URL 
(CASSANDRA-12454)
 + * Optimize the vnode allocation for single replica per DC (CASSANDRA-12777)
 + * Use non-token restrictions for bounds when token restrictions are 
overridden (CASSANDRA-12419)
 + * Fix CQLSH auto completion for PER PARTITION LIMIT (CASSANDRA-12803)
 + * Use different build directories for Eclipse and Ant (CASSANDRA-12466)
 + * Avoid potential AttributeError in cqlsh due to no table metadata 
(CASSANDRA-12815)
 + * Fix RandomReplicationAwareTokenAllocatorTest.testExistingCluster 
(CASSANDRA-12812)
 + * Upgrade commons-codec to 1.9 (CASSANDRA-12790)
 + * Make the fanout size for LeveledCompactionStrategy to be configurable 
(CASSANDRA-11550)
 + * Add duration data type (CASSANDRA-11873)
 + * Fix timeout in ReplicationAwareTokenAllocatorTest (CASSANDRA-12784)
 + * Improve sum aggregate functions (CASSANDRA-12417)
 + * Make cassandra.yaml docs for batch_size_*_threshold_in_kb reflect changes 
in CASSANDRA-10876 (CASSANDRA-12761)
 + * cqlsh fails to format collections when using aliases (CASSANDRA-11534)
 + * Check for hash conflicts in prepared statements (CASSANDRA-12733)
 + * Exit query parsing upon first error (CASSANDRA-12598)
 + * Fix cassandra-stress to use single seed in UUID generation 
(CASSANDRA-12729)
 + * CQLSSTableWriter does not allow Update statement (CASSANDRA-12450)
 + * Config class uses boxed types but DD exposes primitive types 
(CASSANDRA-12199)
 + * Add pre- and post-shutdown hooks to Storage Service (CASSANDRA-12461)
 + * Add hint delivery metrics (CASSANDRA-12693)
 + * Remove IndexInfo cache from FileIndexInfoRetriever (CASSANDRA-12731)
 + * ColumnIndex does not reuse buffer (CASSANDRA-12502)
 + * cdc column addition still breaks schema migration tasks (CASSANDRA-12697)
 + * Upgrade metrics-reporter dependencies (CASSANDRA-12089)
 + * Tune compaction thread count via nodetool (CASSANDRA-12248)
 + * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232)
 + * Include repair session IDs in repair start message (CASSANDRA-12532)
 + * Add a blocking task to Index, run before joining the ring (CASSANDRA-12039)
 + * Fix NPE when using CQLSSTableWriter (CASSANDRA-12667)
 + * Support optional backpressure strategies at the coordinator 

[1/6] cassandra git commit: Use unfiltered iterator for partition indexing

[jasobrown]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 48fed8016 -> 7a06df79d
  refs/heads/cassandra-3.11 74559de50 -> e3d26b6aa
  refs/heads/trunk c3d724445 -> d6da7b799


Use unfiltered iterator for partition indexing

Patch by Alex Petrov; reviewed by Sergio Bossa for CASSANDRA-13075;


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7a06df79
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7a06df79
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7a06df79

Branch: refs/heads/cassandra-3.0
Commit: 7a06df79d829ac073264045eb9420a61c5ba939a
Parents: 48fed80
Author: Alex Petrov 
Authored: Thu Jan 19 09:49:23 2017 +0100
Committer: Jason Brown 
Committed: Thu Jan 19 10:55:10 2017 -0800

--
 CHANGES.txt |   1 +
 .../cassandra/index/SecondaryIndexManager.java  |  86 ++-
 .../service/pager/AbstractQueryPager.java   |  54 --
 .../apache/cassandra/index/CustomIndexTest.java | 106 +++
 .../org/apache/cassandra/index/StubIndex.java   |   4 +
 5 files changed, 216 insertions(+), 35 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/7a06df79/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 6293cfa..a85386b 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Indexer is not correctly invoked when building indexes over sstables 
(CASSANDRA-13075)
  * Read repair is not blocking repair to finish in foreground repair 
(CASSANDRA-13115)
  * Stress daemon help is incorrect (CASSANDRA-12563)
  * Remove ALTER TYPE support (CASSANDRA-12443)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/7a06df79/src/java/org/apache/cassandra/index/SecondaryIndexManager.java
--
diff --git a/src/java/org/apache/cassandra/index/SecondaryIndexManager.java 
b/src/java/org/apache/cassandra/index/SecondaryIndexManager.java
index a6ed3ba..d39b607 100644
--- a/src/java/org/apache/cassandra/index/SecondaryIndexManager.java
+++ b/src/java/org/apache/cassandra/index/SecondaryIndexManager.java
@@ -48,8 +48,7 @@ import org.apache.cassandra.db.compaction.CompactionManager;
 import org.apache.cassandra.db.filter.RowFilter;
 import org.apache.cassandra.db.lifecycle.SSTableSet;
 import org.apache.cassandra.db.lifecycle.View;
-import org.apache.cassandra.db.partitions.PartitionIterators;
-import org.apache.cassandra.db.partitions.PartitionUpdate;
+import org.apache.cassandra.db.partitions.*;
 import org.apache.cassandra.db.rows.*;
 import org.apache.cassandra.exceptions.InvalidRequestException;
 import org.apache.cassandra.index.internal.CassandraIndex;
@@ -543,35 +542,64 @@ public class SecondaryIndexManager implements 
IndexRegistry
 {
 try (ReadOrderGroup readGroup = cmd.startOrderGroup();
  OpOrder.Group writeGroup = Keyspace.writeOrder.start();
- RowIterator partition =
-
PartitionIterators.getOnlyElement(pager.fetchPageInternal(pageSize,readGroup),
-  cmd))
+ UnfilteredPartitionIterator page = 
pager.fetchPageUnfiltered(baseCfs.metadata, pageSize, readGroup))
 {
-Set indexers = indexes.stream()
- .map(index -> 
index.indexerFor(key,
-   
 partition.columns(),
-   
 nowInSec,
-   
 writeGroup,
-   
 IndexTransaction.Type.UPDATE))
- 
.filter(Objects::nonNull)
- 
.collect(Collectors.toSet());
-
-indexers.forEach(Index.Indexer::begin);
-
-// only process the static row once per partition
-if (!readStatic && !partition.staticRow().isEmpty())
-{
-indexers.forEach(indexer -> 
indexer.insertRow(partition.staticRow()));
-readStatic = true;
+if (!page.hasNext())
+break;
+
+try (UnfilteredRowIterator partition = page.next()) {
+Set indexers = indexes.stream()
+   

[jira] [Commented] (CASSANDRA-12744) Randomness of stress distributions is not good

[T Jake Luciani (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830595#comment-15830595
 ] 

T Jake Luciani commented on CASSANDRA-12744:


There are test failures I haven't looked into yet so no.

> Randomness of stress distributions is not good
> --
>
> Key: CASSANDRA-12744
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12744
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: T Jake Luciani
>Assignee: T Jake Luciani
>Priority: Minor
>  Labels: stress
> Fix For: 3.0.x
>
>
> The randomness of our distributions is pretty bad.  We are using the 
> JDKRandomGenerator() but in testing of uniform(1..3) we see for 100 
> iterations it's only outputting 3.  If you bump it to 10k it hits all 3 
> values. 
> I made a change to just use the default commons math random generator and now 
> see all 3 values for n=10



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-13133) Unclosed file descriptors when querying SnapshotsSize metric

[Nicholas Rushton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicholas Rushton resolved CASSANDRA-13133.
--
Resolution: Fixed

fixed by CASSANDRA-11594

> Unclosed file descriptors when querying SnapshotsSize metric
> 
>
> Key: CASSANDRA-13133
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13133
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths
> Environment: CentOS 7
>Reporter: Nicholas Rushton
>  Labels: jmx, lhf, metrics, newbie
> Fix For: 3.9
>
>
> Started to notice many open file descriptors (100k+) per node, growing at a 
> rate of about 30 per minute in our cluster. After turning off our JMX 
> exporting server(https://github.com/prometheus/jmx_exporter), which gets 
> queried every 30 seconds, the number of file descriptors remained static. 
> Digging a bit further I ran a jmx dump tool over all the cassandra metrics 
> and tracked the number of file descriptors after each query, boiling it down 
> to a single metric causing the number of file descriptors to increase:
> org.apache.cassandra.metrics:keyspace=tpsv1,name=SnapshotsSize,scope=events_by_engagement_id,type=Table
> running a query a few times against this metric shows the file descriptors 
> increasing after each query:
> {code}
> for _ in {0..3} 
> do 
>java -jar jmx-dump-0.4.2-standalone.jar --port 7199 --dump 
> org.apache.cassandra.metrics:keyspace=tpsv1,name=SnapshotsSize,scope=events_by_engagement_id,type=Table
>  > /dev/null; 
>sudo lsof -p `pgrep -f CassandraDaemon` | fgrep "DIR" | awk 
> '{a[$(NF)]+=1}END{for(k in a){print k, a[k]}}' | grep "events_by" 
> done
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> >  33176
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> >  33177
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> >  33178
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> >  33179
> {code}
> it should be noted that the file descriptor is open on a directory, not an 
> actual file



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-13133) Unclosed file descriptors when querying SnapshotsSize metric

[Nicholas Rushton (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830430#comment-15830430
 ] 

Nicholas Rushton commented on CASSANDRA-13133:
--

patch worked, closing this ticket as fixed by CASSANDRA-11594

> Unclosed file descriptors when querying SnapshotsSize metric
> 
>
> Key: CASSANDRA-13133
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13133
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths
> Environment: CentOS 7
>Reporter: Nicholas Rushton
>  Labels: jmx, lhf, metrics, newbie
> Fix For: 3.9
>
>
> Started to notice many open file descriptors (100k+) per node, growing at a 
> rate of about 30 per minute in our cluster. After turning off our JMX 
> exporting server(https://github.com/prometheus/jmx_exporter), which gets 
> queried every 30 seconds, the number of file descriptors remained static. 
> Digging a bit further I ran a jmx dump tool over all the cassandra metrics 
> and tracked the number of file descriptors after each query, boiling it down 
> to a single metric causing the number of file descriptors to increase:
> org.apache.cassandra.metrics:keyspace=tpsv1,name=SnapshotsSize,scope=events_by_engagement_id,type=Table
> running a query a few times against this metric shows the file descriptors 
> increasing after each query:
> {code}
> for _ in {0..3} 
> do 
>java -jar jmx-dump-0.4.2-standalone.jar --port 7199 --dump 
> org.apache.cassandra.metrics:keyspace=tpsv1,name=SnapshotsSize,scope=events_by_engagement_id,type=Table
>  > /dev/null; 
>sudo lsof -p `pgrep -f CassandraDaemon` | fgrep "DIR" | awk 
> '{a[$(NF)]+=1}END{for(k in a){print k, a[k]}}' | grep "events_by" 
> done
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> >  33176
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> >  33177
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> >  33178
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> >  33179
> {code}
> it should be noted that the file descriptor is open on a directory, not an 
> actual file



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12674) [SASI] Confusing AND/OR semantics for StandardAnalyzer

[Alex Petrov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830351#comment-15830351
 ] 

Alex Petrov commented on CASSANDRA-12674:
-

[~xedin] sure! I was working on finalising and testing [CASSANDRA-11990], so 
didn't get a chance continue on this one. Hope to get to it soon. 

> [SASI] Confusing AND/OR semantics for StandardAnalyzer 
> ---
>
> Key: CASSANDRA-12674
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12674
> Project: Cassandra
>  Issue Type: Bug
>  Components: sasi
> Environment: Cassandra 3.7
>Reporter: DOAN DuyHai
>
> {code:sql}
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 3.7 | CQL spec 3.4.2 | Native protocol v4]
> Use HELP for help.
> cqlsh> use test;
> cqlsh:test> CREATE TABLE sasi_bug(id int, clustering int, val text, PRIMARY 
> KEY((id), clustering));
> cqlsh:test> CREATE CUSTOM INDEX ON sasi_bug(val) USING 
> 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = {
> 'mode': 'CONTAINS',
>  'analyzer_class': 
> 'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer',
> 'analyzed': 'true'};
> //1st example SAME PARTITION KEY
> cqlsh:test> INSERT INTO sasi_bug(id, clustering , val ) VALUES(1, 1, 
> 'homeworker');
> cqlsh:test> INSERT INTO sasi_bug(id, clustering , val ) VALUES(1, 2, 
> 'hardworker');
> cqlsh:test> SELECT * FROM sasi_bug WHERE val LIKE '%work home%';
>  id | clustering | val
> ++
>   1 |  1 | homeworker
>   1 |  2 | hardworker
> (2 rows)
> //2nd example DIFFERENT PARTITION KEY
> cqlsh:test> INSERT INTO sasi_bug(id, clustering, val) VALUES(10, 1, 
> 'speedrun');
> cqlsh:test> INSERT INTO sasi_bug(id, clustering, val) VALUES(11, 1, 
> 'longrun');
> cqlsh:test> SELECT * FROM sasi_bug WHERE val LIKE '%long run%';
>  id | clustering | val
> ++-
>  11 |  1 | longrun
> (1 rows)
> {code}
> In the 1st example, both rows belong to the same partition so SASI returns 
> both values. Indeed {{LIKE '%work home%'}} means {{contains 'work' OR 
> 'home'}} so the result makes sense
> In the 2nd example, only one row is returned whereas we expect 2 rows because 
> {{LIKE '%long run%'}} means {{contains 'long' OR 'run'}} so *speedrun* should 
> be returned too.
> So where is the problem ? Explanation:
> When there is only 1 predicate, the root operation type is an *AND*:
> {code:java|title=QueryPlan}
> private Operation analyze()
> {
> try
> {
> Operation.Builder and = new Operation.Builder(OperationType.AND, 
> controller);
> controller.getExpressions().forEach(and::add);
> return and.complete();
> }
>...
> }
> {code}
> During the parsing of {{LIKE '%long run%'}}, SASI creates 2 expressions for 
> the searched term: {{long}} and {{run}}, which corresponds to an *OR* logic. 
> However, this piece of code just ruins the *OR* logic:
> {code:java|title=Operation}
> public Operation complete()
> {
> if (!expressions.isEmpty())
> {
> ListMultimap 
> analyzedExpressions = analyzeGroup(controller, op, expressions);
> RangeIterator.Builder range = 
> controller.getIndexes(op, analyzedExpressions.values());
>  ...
> }
> {code}
> As you can see, we blindly take all the *values* of the MultiMap (which 
> contains a single entry for the {{val}} column with 2 expressions) and pass 
> it to {{controller.getIndexes(...)}}
> {code:java|title=QueryController}
> public RangeIterator.Builder getIndexes(OperationType op, 
> Collection expressions)
> {
> if (resources.containsKey(expressions))
> throw new IllegalArgumentException("Can't process the same 
> expressions multiple times.");
> RangeIterator.Builder builder = op == OperationType.OR
> ? RangeUnionIterator. Token>builder()
> : 
> RangeIntersectionIterator.builder();
> ...
> }
> {code}
> And because the root operation has *AND* type, the 
> {{RangeIntersectionIterator}} will be used on both expressions {{long}} and 
> {{run}}.
> So when data belong to different partitions, we have the *AND* logic that 
> applies and eliminates _speedrun_
> When data belong to the same partition but different row, the 
> {{RangeIntersectionIterator}} returns a single partition and then the rows 
> are filtered further by {{operationTree.satisfiedBy}} and the results are 
> correct
> {code:java|title=QueryPlan}
> while (currentKeys.hasNext())
> {
> DecoratedKey key = 

[jira] [Updated] (CASSANDRA-13136) Debian init script does not return non-zero when jar or config cease to exist

[Andrew Konkol (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Konkol updated CASSANDRA-13136:
--
Description: 
Hey there,
The init.d script included in the debian package returns zero when the 
cassandra jar, yaml, or env files are missing.

branch: https://github.com/akonkol/cassandra/tree/fix_init
commit: 
https://github.com/akonkol/cassandra/commit/d11f9c676c90372f852380104ab3e4abccc29d71

  was:
Hey there,
The init.d script included in the debian package returns zero when the 
cassandra jar, yaml, or env files are missing.

branch: https://github.com/akonkol/cassandra/tree/fix_init
co


> Debian init script does not return non-zero when jar or config cease to exist
> -
>
> Key: CASSANDRA-13136
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13136
> Project: Cassandra
>  Issue Type: Improvement
> Environment: Ubuntu 12.04.5 LTS
>Reporter: Andrew Konkol
>Priority: Trivial
>  Labels: easyfix
> Fix For: 2.1.x
>
>
> Hey there,
> The init.d script included in the debian package returns zero when the 
> cassandra jar, yaml, or env files are missing.
> branch: https://github.com/akonkol/cassandra/tree/fix_init
> commit: 
> https://github.com/akonkol/cassandra/commit/d11f9c676c90372f852380104ab3e4abccc29d71



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-13136) Debian init script does not return non-zero when jar or config cease to exist

[Andrew Konkol (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Konkol updated CASSANDRA-13136:
--
Description: 
Hey there,
The init.d script included in the debian package returns zero when the 
cassandra jar, yaml, or env files are missing.

branch: https://github.com/akonkol/cassandra/tree/fix_init
co

  was:The init.d script included in the debian package returns zero when the 
cassandra jar, yaml, or env files are missing.


> Debian init script does not return non-zero when jar or config cease to exist
> -
>
> Key: CASSANDRA-13136
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13136
> Project: Cassandra
>  Issue Type: Improvement
> Environment: Ubuntu 12.04.5 LTS
>Reporter: Andrew Konkol
>Priority: Trivial
>  Labels: easyfix
> Fix For: 2.1.x
>
>
> Hey there,
> The init.d script included in the debian package returns zero when the 
> cassandra jar, yaml, or env files are missing.
> branch: https://github.com/akonkol/cassandra/tree/fix_init
> co



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (CASSANDRA-12443) Remove alter type support

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer reopened CASSANDRA-12443:

  Assignee: Benjamin Lerer  (was: Carl Yeksigian)

> Remove alter type support
> -
>
> Key: CASSANDRA-12443
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12443
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Carl Yeksigian
>Assignee: Benjamin Lerer
> Fix For: 3.0.11, 3.10
>
>
> Currently, we allow altering of types. However, because we no longer store 
> the length for all types anymore, switching from a fixed-width to 
> variable-width type causes issues. commitlog playback breaking startup, 
> queries currently in flight getting back bad results, and special casing 
> required to handle the changes. In addition, this would solve 
> CASSANDRA-10309, as there is no possibility of the types changing while an 
> SSTableReader is open.
> For fixed-length, compatible types, the alter also doesn't add much over a 
> cast, so users could use that in order to retrieve the altered type.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-13136) Debian init script does not return non-zero when jar or config cease to exist

[Andrew Konkol (JIRA)]

Andrew Konkol created CASSANDRA-13136:
-

 Summary: Debian init script does not return non-zero when jar or 
config cease to exist
 Key: CASSANDRA-13136
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13136
 Project: Cassandra
  Issue Type: Improvement
 Environment: Ubuntu 12.04.5 LTS
Reporter: Andrew Konkol
Priority: Trivial
 Fix For: 2.1.x


The init.d script included in the debian package returns zero when the 
cassandra jar, yaml, or env files are missing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-13123) Draining a node might fail to delete all inactive commitlogs

[JIRA]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830254#comment-15830254
 ] 

Jan Urbański commented on CASSANDRA-13123:
--

[~jasobrown] I haven't had the chance to try this out in production yet, I'll 
try to do that tomorrow. The initial commitlog replay takes up to two minutes 
for each of our nodes right now and if I understand correctly, after a drain 
all commitlogs except for at most two would be deleted, so the initial replay 
phase would be reduced to essentially zero. The shutdown phase might take a bit 
longer, because it'll have to wait for those commitlogs to be deleted, of 
course.

The exact improvement depends on the number of CLs left behind after a drain - 
on machines with heavily contended disks it can be a lot, on lightly loaded 
ones it might be 0.

As to when we're doing drains, it's on every restart (it's part of the restart 
procedure that we have).

> Draining a node might fail to delete all inactive commitlogs
> 
>
> Key: CASSANDRA-13123
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13123
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths
>Reporter: Jan Urbański
>Assignee: Jan Urbański
> Fix For: 3.8
>
> Attachments: 13123-2.2.8.txt, 13123-3.0.10.txt, 13123-3.9.txt, 
> 13123-trunk.txt
>
>
> After issuing a drain command, it's possible that not all of the inactive 
> commitlogs are removed.
> The drain command shuts down the CommitLog instance, which in turn shuts down 
> the CommitLogSegmentManager. This has the effect of discarding any pending 
> management tasks it might have, like the removal of inactive commitlogs.
> This in turn leads to an excessive amount of commitlogs being left behind 
> after a drain and a lengthy recovery after a restart. With a fleet of dozens 
> of nodes, each of them leaving several GB of commitlogs after a drain and 
> taking up to two minutes to recover them on restart, the additional time 
> required to restart the entire fleet becomes noticeable.
> This problem is not present in 3.x or trunk because of the CLSM rewrite done 
> in CASSANDRA-8844.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-13133) Unclosed file descriptors when querying SnapshotsSize metric

[NIcholas Rushton (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830224#comment-15830224
 ] 

NIcholas Rushton commented on CASSANDRA-13133:
--

[~Stefania] thank you, will take a look

> Unclosed file descriptors when querying SnapshotsSize metric
> 
>
> Key: CASSANDRA-13133
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13133
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths
> Environment: CentOS 7
>Reporter: NIcholas Rushton
>  Labels: jmx, lhf, metrics, newbie
> Fix For: 3.9
>
>
> Started to notice many open file descriptors (100k+) per node, growing at a 
> rate of about 30 per minute in our cluster. After turning off our JMX 
> exporting server(https://github.com/prometheus/jmx_exporter), which gets 
> queried every 30 seconds, the number of file descriptors remained static. 
> Digging a bit further I ran a jmx dump tool over all the cassandra metrics 
> and tracked the number of file descriptors after each query, boiling it down 
> to a single metric causing the number of file descriptors to increase:
> org.apache.cassandra.metrics:keyspace=tpsv1,name=SnapshotsSize,scope=events_by_engagement_id,type=Table
> running a query a few times against this metric shows the file descriptors 
> increasing after each query:
> {code}
> for _ in {0..3} 
> do 
>java -jar jmx-dump-0.4.2-standalone.jar --port 7199 --dump 
> org.apache.cassandra.metrics:keyspace=tpsv1,name=SnapshotsSize,scope=events_by_engagement_id,type=Table
>  > /dev/null; 
>sudo lsof -p `pgrep -f CassandraDaemon` | fgrep "DIR" | awk 
> '{a[$(NF)]+=1}END{for(k in a){print k, a[k]}}' | grep "events_by" 
> done
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> >  33176
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> >  33177
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> >  33178
> > /data/cassandra/data/tpsv1/events_by_engagement_id-01d8f450a54911e6917ec93f8a91ec71
> >  33179
> {code}
> it should be noted that the file descriptor is open on a directory, not an 
> actual file



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12978) mx4j -> HTTP 500 -> ConcurrentModificationException

[Edward Ribeiro (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830166#comment-15830166
 ] 

Edward Ribeiro commented on CASSANDRA-12978:


Hi [~Reweavers],

It looks like the problem is because 
{{org.apache.cassandra.utils.StreamingHistogram}} is not *thread-safe*. So, if 
{{sum}} method, for example, is called concurrently with an {{update}} then the 
{{TreeMap}} underneath {{StreamingHistogram}} will get changed and throws the 
{{ConcurrentModificationException}}. One possible 'straightforward' solution 
would be to make the methods {{synchronized}} but this  could have a net impact 
on performance, maybe. This problem didn't pop up before because the usually it 
is used by a single thread, as far as I can quickly see looking at C* code base.

I will /cc [~tjake] 'cause he worked on this class about 1 year ago, so better 
than me to say if it makes sense.


> mx4j -> HTTP 500 -> ConcurrentModificationException
> ---
>
> Key: CASSANDRA-12978
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12978
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
> Environment: Debian, Single cluster, 2 data centres, E5-2620 v3, 
> 16GB, RAID1 SSD Commit log, RAID10 15k HDD data
>Reporter: Rob Emery
>Priority: Critical
> Fix For: 2.1.6
>
>
> We run some checks from our Monitoring software that rely on mx4j.
> The checks typically grab some xml via HTTP request and parse it. For 
> example, CF Stats on 'MyKeySpace' and 'MyColumnFamily' are retrieved 
> using:
> http://cassandra001:8081/mbean?template=identity=org.apache.cassandra.db%3Atype%3DColumnFamilies%2Ckeyspace%3DMyKeySpace%2Ccolumnfamily%3DMyColumnFamily
> The checks run each minute. Periodically they result in a "HTTP 500 internal 
> server error". The HTML body returned is empty.
> Experimentally we ran Cassandra in the foreground on one node and reproduced 
> the problem. this elicited the following stack trace:
> javax.management.RuntimeMBeanException: 
> java.util.ConcurrentModificationException
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
> at 
> com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
> at 
> mx4j.tools.adaptor.http.MBeanCommandProcessor.createMBeanElement(MBeanCommandProcessor.java:119)
> at 
> mx4j.tools.adaptor.http.MBeanCommandProcessor.executeRequest(MBeanCommandProcessor.java:56)
> at 
> mx4j.tools.adaptor.http.HttpAdaptor$HttpClient.run(HttpAdaptor.java:980)
> Caused by: java.util.ConcurrentModificationException
> at 
> java.util.TreeMap$NavigableSubMap$SubMapIterator.nextEntry(TreeMap.java:1594)
> at 
> java.util.TreeMap$NavigableSubMap$SubMapEntryIterator.next(TreeMap.java:1642)
> at 
> java.util.TreeMap$NavigableSubMap$SubMapEntryIterator.next(TreeMap.java:1636)
> at java.util.AbstractMap$2$1.next(AbstractMap.java:385)
> at 
> org.apache.cassandra.utils.StreamingHistogram.sum(StreamingHistogram.java:160)
> at 
> org.apache.cassandra.io.sstable.metadata.StatsMetadata.getDroppableTombstonesBefore(StatsMetadata.java:113)
> at 
> org.apache.cassandra.io.sstable.SSTableReader.getDroppableTombstonesBefore(SSTableReader.java:2004)
> at 
> org.apache.cassandra.db.DataTracker.getDroppableTombstoneRatio(DataTracker.java:507)
> at 
> org.apache.cassandra.db.ColumnFamilyStore.getDroppableTombstoneRatio(ColumnFamilyStore.java:3089)
> at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
> at 
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at 
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at 
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at 
> 

[jira] [Updated] (CASSANDRA-13115) Read repair is not blocking repair to finish in foreground repair

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-13115:
-
   Resolution: Fixed
Fix Version/s: (was: 3.0.x)
   (was: 3.x)
   3.11
   3.0.11
   Status: Resolved  (was: Patch Available)

Committed, thanks.

> Read repair is not blocking repair to finish in foreground repair
> -
>
> Key: CASSANDRA-13115
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13115
> Project: Cassandra
>  Issue Type: Bug
> Environment: ccm on OSX 
>Reporter: Xiaolong Jiang
>Assignee: Sylvain Lebresne
> Fix For: 3.0.11, 3.11
>
>
> The code trying to wait(block) for repair result to come back in 3.X is below:
> {code:title= DataResolver.java|borderStyle=solid}
> public void close()
> {
> try
> {
> FBUtilities.waitOnFutures(repairResults, 
> DatabaseDescriptor.getWriteRpcTimeout());
> }
> catch (TimeoutException ex)
> {
> // We got all responses, but timed out while repairing
> int blockFor = consistency.blockFor(keyspace);
> if (Tracing.isTracing())
> Tracing.trace("Timed out while read-repairing after 
> receiving all {} data and digest responses", blockFor);
> else
> logger.debug("Timeout while read-repairing after 
> receiving all {} data and digest responses", blockFor);
> throw new ReadTimeoutException(consistency, blockFor-1, 
> blockFor, true);
> }
> }
> {code}
> in DataResolver class, but this close method is never called and it's also 
> not auto close(RepairMergeListener is not extending from 
> AutoCloseable/CloseableIterator) which means we never wait for repair to 
> finish before returning final result. 
> The steps to reproduce:
> 1. create some keyspace/table with RF = 2
> 2. start 2 nodes using ccm
> 3. stop node2
> 4. disable node1 hinted hand off
> 5. write some data to node1 with consistency level one
> 6. start node2
> 7. query some data from node1 
> This should trigger read repair. I put some log in above close method, and 
> can not see log print put.
> So this bug will basically violate "monotonic quorum reads " guarantee. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-13075) Indexer is not correctly invoked when building indexes over sstables

[Alex Petrov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Petrov updated CASSANDRA-13075:

Status: Ready to Commit  (was: Patch Available)

> Indexer is not correctly invoked when building indexes over sstables
> 
>
> Key: CASSANDRA-13075
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13075
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Sergio Bossa
>Assignee: Alex Petrov
>Priority: Critical
> Attachments: CustomIndexTest.java
>
>
> Following CASSANDRA-12796, {{SecondaryIndexManager#indexPartition()}} calls 
> each {{Indexer}} {{begin}} and {{finish}} methods multiple times per 
> partition (depending on the page size), as 
> {{PartitionIterators#getOnlyElement()}} returns an empty partition even when 
> the iterator is exhausted.
> This leads to bugs for {{Indexer}} implementations doing actual work in those 
>  methods, but even worse, it provides the {{Indexer}} the same input of an 
> empty partition containing only a non-live partition deletion, as the 
> {{Indexer#partitionDelete()}} method is *not* actually called.
> My proposed solution:
> 1) Stop the iteration before the empty partition is returned and ingested 
> into the {{Indexer}}.
> 2) Actually call the {{Indexer#partitionDelete()}} method inside 
> {{SecondaryIndexManager#indexPartition()}} (which requires to use a filtered 
> iterator so it actually contains the deletion info).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-13075) Indexer is not correctly invoked when building indexes over sstables

[Sergio Bossa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830158#comment-15830158
 ] 

Sergio Bossa commented on CASSANDRA-13075:
--

+1

> Indexer is not correctly invoked when building indexes over sstables
> 
>
> Key: CASSANDRA-13075
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13075
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Sergio Bossa
>Assignee: Alex Petrov
>Priority: Critical
> Attachments: CustomIndexTest.java
>
>
> Following CASSANDRA-12796, {{SecondaryIndexManager#indexPartition()}} calls 
> each {{Indexer}} {{begin}} and {{finish}} methods multiple times per 
> partition (depending on the page size), as 
> {{PartitionIterators#getOnlyElement()}} returns an empty partition even when 
> the iterator is exhausted.
> This leads to bugs for {{Indexer}} implementations doing actual work in those 
>  methods, but even worse, it provides the {{Indexer}} the same input of an 
> empty partition containing only a non-live partition deletion, as the 
> {{Indexer#partitionDelete()}} method is *not* actually called.
> My proposed solution:
> 1) Stop the iteration before the empty partition is returned and ingested 
> into the {{Indexer}}.
> 2) Actually call the {{Indexer#partitionDelete()}} method inside 
> {{SecondaryIndexManager#indexPartition()}} (which requires to use a filtered 
> iterator so it actually contains the deletion info).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[slebresne]

Merge branch 'cassandra-3.0' into cassandra-3.11

* cassandra-3.0:
  Read repair is not blocking repair to finish in foreground repair


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/74559de5
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/74559de5
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/74559de5

Branch: refs/heads/cassandra-3.11
Commit: 74559de50fa6974eba56a00ca37cfc7746134211
Parents: affa68f 48fed80
Author: Sylvain Lebresne 
Authored: Thu Jan 19 16:54:22 2017 +0100
Committer: Sylvain Lebresne 
Committed: Thu Jan 19 16:54:22 2017 +0100

--
 CHANGES.txt |   1 +
 .../UnfilteredPartitionIterators.java   |   1 +
 .../cassandra/service/AsyncRepairCallback.java  |   2 +-
 .../apache/cassandra/service/DataResolver.java  |  14 ++-
 .../cassandra/service/DigestResolver.java   |   9 +-
 .../apache/cassandra/service/ReadCallback.java  |   4 +-
 .../cassandra/service/ResponseResolver.java |  12 ++
 .../cassandra/service/DataResolverTest.java | 117 +--
 8 files changed, 118 insertions(+), 42 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/74559de5/CHANGES.txt
--
diff --cc CHANGES.txt
index 853ca09,6293cfa..8b613e6
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,120 -1,11 +1,121 @@@
 -3.0.11
 +3.10
 + * Fix TestHintedHandoff.hintedhandoff_decom_test (CASSANDRA-13058)
 + * Fixed query monitoring for range queries (CASSANDRA-13050)
 + * Remove outboundBindAny configuration property (CASSANDRA-12673)
 + * Use correct bounds for all-data range when filtering (CASSANDRA-12666)
 + * Remove timing window in test case (CASSANDRA-12875)
 + * Resolve unit testing without JCE security libraries installed 
(CASSANDRA-12945)
 + * Fix inconsistencies in cassandra-stress load balancing policy 
(CASSANDRA-12919)
 + * Fix validation of non-frozen UDT cells (CASSANDRA-12916)
 + * Don't shut down socket input/output on StreamSession (CASSANDRA-12903)
 + * Fix Murmur3PartitionerTest (CASSANDRA-12858)
 + * Move cqlsh syntax rules into separate module and allow easier 
customization (CASSANDRA-12897)
 + * Fix CommitLogSegmentManagerTest (CASSANDRA-12283)
 + * Fix cassandra-stress truncate option (CASSANDRA-12695)
 + * Fix crossNode value when receiving messages (CASSANDRA-12791)
 + * Don't load MX4J beans twice (CASSANDRA-12869)
 + * Extend native protocol request flags, add versions to SUPPORTED, and 
introduce ProtocolVersion enum (CASSANDRA-12838)
 + * Set JOINING mode when running pre-join tasks (CASSANDRA-12836)
 + * remove net.mintern.primitive library due to license issue (CASSANDRA-12845)
 + * Properly format IPv6 addresses when logging JMX service URL 
(CASSANDRA-12454)
 + * Optimize the vnode allocation for single replica per DC (CASSANDRA-12777)
 + * Use non-token restrictions for bounds when token restrictions are 
overridden (CASSANDRA-12419)
 + * Fix CQLSH auto completion for PER PARTITION LIMIT (CASSANDRA-12803)
 + * Use different build directories for Eclipse and Ant (CASSANDRA-12466)
 + * Avoid potential AttributeError in cqlsh due to no table metadata 
(CASSANDRA-12815)
 + * Fix RandomReplicationAwareTokenAllocatorTest.testExistingCluster 
(CASSANDRA-12812)
 + * Upgrade commons-codec to 1.9 (CASSANDRA-12790)
 + * Make the fanout size for LeveledCompactionStrategy to be configurable 
(CASSANDRA-11550)
 + * Add duration data type (CASSANDRA-11873)
 + * Fix timeout in ReplicationAwareTokenAllocatorTest (CASSANDRA-12784)
 + * Improve sum aggregate functions (CASSANDRA-12417)
 + * Make cassandra.yaml docs for batch_size_*_threshold_in_kb reflect changes 
in CASSANDRA-10876 (CASSANDRA-12761)
 + * cqlsh fails to format collections when using aliases (CASSANDRA-11534)
 + * Check for hash conflicts in prepared statements (CASSANDRA-12733)
 + * Exit query parsing upon first error (CASSANDRA-12598)
 + * Fix cassandra-stress to use single seed in UUID generation 
(CASSANDRA-12729)
 + * CQLSSTableWriter does not allow Update statement (CASSANDRA-12450)
 + * Config class uses boxed types but DD exposes primitive types 
(CASSANDRA-12199)
 + * Add pre- and post-shutdown hooks to Storage Service (CASSANDRA-12461)
 + * Add hint delivery metrics (CASSANDRA-12693)
 + * Remove IndexInfo cache from FileIndexInfoRetriever (CASSANDRA-12731)
 + * ColumnIndex does not reuse buffer (CASSANDRA-12502)
 + * cdc column addition still breaks schema migration tasks (CASSANDRA-12697)
 + * Upgrade metrics-reporter dependencies (CASSANDRA-12089)
 + * Tune compaction thread count via nodetool (CASSANDRA-12248)
 + * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232)
 + * Include repair session IDs in 

[1/6] cassandra git commit: Read repair is not blocking repair to finish in foreground repair

[slebresne]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 3f41d7a76 -> 48fed8016
  refs/heads/cassandra-3.11 affa68fd1 -> 74559de50
  refs/heads/trunk 52df6a58d -> c3d724445


Read repair is not blocking repair to finish in foreground repair

patch by Sylvain Lebresne; reviewed by Xiaolong Jiang and Jason Brown for 
CASSANDRA-13115


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/48fed801
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/48fed801
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/48fed801

Branch: refs/heads/cassandra-3.0
Commit: 48fed80162592f741bf29298e2064452d53de4d8
Parents: 3f41d7a
Author: Sylvain Lebresne 
Authored: Thu Jan 12 10:03:11 2017 +0100
Committer: Sylvain Lebresne 
Committed: Thu Jan 19 16:49:14 2017 +0100

--
 CHANGES.txt |   1 +
 .../UnfilteredPartitionIterators.java   |   1 +
 .../cassandra/service/AsyncRepairCallback.java  |   5 +-
 .../apache/cassandra/service/DataResolver.java  |  14 ++-
 .../cassandra/service/DigestResolver.java   |   9 +-
 .../apache/cassandra/service/ReadCallback.java  |   4 +-
 .../cassandra/service/ResponseResolver.java |  12 ++
 .../cassandra/service/DataResolverTest.java | 117 +--
 8 files changed, 119 insertions(+), 44 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/48fed801/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 97d49af..6293cfa 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Read repair is not blocking repair to finish in foreground repair 
(CASSANDRA-13115)
  * Stress daemon help is incorrect (CASSANDRA-12563)
  * Remove ALTER TYPE support (CASSANDRA-12443)
  * Fix assertion for certain legacy range tombstone pattern (CASSANDRA-12203)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/48fed801/src/java/org/apache/cassandra/db/partitions/UnfilteredPartitionIterators.java
--
diff --git 
a/src/java/org/apache/cassandra/db/partitions/UnfilteredPartitionIterators.java 
b/src/java/org/apache/cassandra/db/partitions/UnfilteredPartitionIterators.java
index 41b1424..1abbb19 100644
--- 
a/src/java/org/apache/cassandra/db/partitions/UnfilteredPartitionIterators.java
+++ 
b/src/java/org/apache/cassandra/db/partitions/UnfilteredPartitionIterators.java
@@ -159,6 +159,7 @@ public abstract class UnfilteredPartitionIterators
 public void close()
 {
 merged.close();
+listener.close();
 }
 };
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/48fed801/src/java/org/apache/cassandra/service/AsyncRepairCallback.java
--
diff --git a/src/java/org/apache/cassandra/service/AsyncRepairCallback.java 
b/src/java/org/apache/cassandra/service/AsyncRepairCallback.java
index dec5319..d613f3d 100644
--- a/src/java/org/apache/cassandra/service/AsyncRepairCallback.java
+++ b/src/java/org/apache/cassandra/service/AsyncRepairCallback.java
@@ -17,7 +17,6 @@
  */
 package org.apache.cassandra.service;
 
-import java.io.IOException;
 import java.util.concurrent.atomic.AtomicInteger;
 
 import org.apache.cassandra.concurrent.Stage;
@@ -46,9 +45,9 @@ public class AsyncRepairCallback implements 
IAsyncCallback
 {
 StageManager.getStage(Stage.READ_REPAIR).execute(new 
WrappedRunnable()
 {
-protected void runMayThrow() throws DigestMismatchException, 
IOException
+protected void runMayThrow()
 {
-repairResolver.resolve();
+repairResolver.compareResponses();
 }
 });
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/48fed801/src/java/org/apache/cassandra/service/DataResolver.java
--
diff --git a/src/java/org/apache/cassandra/service/DataResolver.java 
b/src/java/org/apache/cassandra/service/DataResolver.java
index 4e5bfb8..01953e1 100644
--- a/src/java/org/apache/cassandra/service/DataResolver.java
+++ b/src/java/org/apache/cassandra/service/DataResolver.java
@@ -21,6 +21,8 @@ import java.net.InetAddress;
 import java.util.*;
 import java.util.concurrent.TimeoutException;
 
+import com.google.common.annotations.VisibleForTesting;
+
 import org.apache.cassandra.concurrent.Stage;
 import org.apache.cassandra.concurrent.StageManager;
 import org.apache.cassandra.config.CFMetaData;
@@ -40,7 +42,8 @@ import org.apache.cassandra.utils.FBUtilities;
 
 

[3/6] cassandra git commit: Read repair is not blocking repair to finish in foreground repair

[slebresne]

Read repair is not blocking repair to finish in foreground repair

patch by Sylvain Lebresne; reviewed by Xiaolong Jiang and Jason Brown for 
CASSANDRA-13115


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/48fed801
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/48fed801
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/48fed801

Branch: refs/heads/trunk
Commit: 48fed80162592f741bf29298e2064452d53de4d8
Parents: 3f41d7a
Author: Sylvain Lebresne 
Authored: Thu Jan 12 10:03:11 2017 +0100
Committer: Sylvain Lebresne 
Committed: Thu Jan 19 16:49:14 2017 +0100

--
 CHANGES.txt |   1 +
 .../UnfilteredPartitionIterators.java   |   1 +
 .../cassandra/service/AsyncRepairCallback.java  |   5 +-
 .../apache/cassandra/service/DataResolver.java  |  14 ++-
 .../cassandra/service/DigestResolver.java   |   9 +-
 .../apache/cassandra/service/ReadCallback.java  |   4 +-
 .../cassandra/service/ResponseResolver.java |  12 ++
 .../cassandra/service/DataResolverTest.java | 117 +--
 8 files changed, 119 insertions(+), 44 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/48fed801/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 97d49af..6293cfa 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Read repair is not blocking repair to finish in foreground repair 
(CASSANDRA-13115)
  * Stress daemon help is incorrect (CASSANDRA-12563)
  * Remove ALTER TYPE support (CASSANDRA-12443)
  * Fix assertion for certain legacy range tombstone pattern (CASSANDRA-12203)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/48fed801/src/java/org/apache/cassandra/db/partitions/UnfilteredPartitionIterators.java
--
diff --git 
a/src/java/org/apache/cassandra/db/partitions/UnfilteredPartitionIterators.java 
b/src/java/org/apache/cassandra/db/partitions/UnfilteredPartitionIterators.java
index 41b1424..1abbb19 100644
--- 
a/src/java/org/apache/cassandra/db/partitions/UnfilteredPartitionIterators.java
+++ 
b/src/java/org/apache/cassandra/db/partitions/UnfilteredPartitionIterators.java
@@ -159,6 +159,7 @@ public abstract class UnfilteredPartitionIterators
 public void close()
 {
 merged.close();
+listener.close();
 }
 };
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/48fed801/src/java/org/apache/cassandra/service/AsyncRepairCallback.java
--
diff --git a/src/java/org/apache/cassandra/service/AsyncRepairCallback.java 
b/src/java/org/apache/cassandra/service/AsyncRepairCallback.java
index dec5319..d613f3d 100644
--- a/src/java/org/apache/cassandra/service/AsyncRepairCallback.java
+++ b/src/java/org/apache/cassandra/service/AsyncRepairCallback.java
@@ -17,7 +17,6 @@
  */
 package org.apache.cassandra.service;
 
-import java.io.IOException;
 import java.util.concurrent.atomic.AtomicInteger;
 
 import org.apache.cassandra.concurrent.Stage;
@@ -46,9 +45,9 @@ public class AsyncRepairCallback implements 
IAsyncCallback
 {
 StageManager.getStage(Stage.READ_REPAIR).execute(new 
WrappedRunnable()
 {
-protected void runMayThrow() throws DigestMismatchException, 
IOException
+protected void runMayThrow()
 {
-repairResolver.resolve();
+repairResolver.compareResponses();
 }
 });
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/48fed801/src/java/org/apache/cassandra/service/DataResolver.java
--
diff --git a/src/java/org/apache/cassandra/service/DataResolver.java 
b/src/java/org/apache/cassandra/service/DataResolver.java
index 4e5bfb8..01953e1 100644
--- a/src/java/org/apache/cassandra/service/DataResolver.java
+++ b/src/java/org/apache/cassandra/service/DataResolver.java
@@ -21,6 +21,8 @@ import java.net.InetAddress;
 import java.util.*;
 import java.util.concurrent.TimeoutException;
 
+import com.google.common.annotations.VisibleForTesting;
+
 import org.apache.cassandra.concurrent.Stage;
 import org.apache.cassandra.concurrent.StageManager;
 import org.apache.cassandra.config.CFMetaData;
@@ -40,7 +42,8 @@ import org.apache.cassandra.utils.FBUtilities;
 
 public class DataResolver extends ResponseResolver
 {
-private final List repairResults = 
Collections.synchronizedList(new ArrayList<>());
+@VisibleForTesting
+final List repairResults 

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[slebresne]

Merge branch 'cassandra-3.0' into cassandra-3.11

* cassandra-3.0:
  Read repair is not blocking repair to finish in foreground repair


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/74559de5
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/74559de5
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/74559de5

Branch: refs/heads/trunk
Commit: 74559de50fa6974eba56a00ca37cfc7746134211
Parents: affa68f 48fed80
Author: Sylvain Lebresne 
Authored: Thu Jan 19 16:54:22 2017 +0100
Committer: Sylvain Lebresne 
Committed: Thu Jan 19 16:54:22 2017 +0100

--
 CHANGES.txt |   1 +
 .../UnfilteredPartitionIterators.java   |   1 +
 .../cassandra/service/AsyncRepairCallback.java  |   2 +-
 .../apache/cassandra/service/DataResolver.java  |  14 ++-
 .../cassandra/service/DigestResolver.java   |   9 +-
 .../apache/cassandra/service/ReadCallback.java  |   4 +-
 .../cassandra/service/ResponseResolver.java |  12 ++
 .../cassandra/service/DataResolverTest.java | 117 +--
 8 files changed, 118 insertions(+), 42 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/74559de5/CHANGES.txt
--
diff --cc CHANGES.txt
index 853ca09,6293cfa..8b613e6
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,120 -1,11 +1,121 @@@
 -3.0.11
 +3.10
 + * Fix TestHintedHandoff.hintedhandoff_decom_test (CASSANDRA-13058)
 + * Fixed query monitoring for range queries (CASSANDRA-13050)
 + * Remove outboundBindAny configuration property (CASSANDRA-12673)
 + * Use correct bounds for all-data range when filtering (CASSANDRA-12666)
 + * Remove timing window in test case (CASSANDRA-12875)
 + * Resolve unit testing without JCE security libraries installed 
(CASSANDRA-12945)
 + * Fix inconsistencies in cassandra-stress load balancing policy 
(CASSANDRA-12919)
 + * Fix validation of non-frozen UDT cells (CASSANDRA-12916)
 + * Don't shut down socket input/output on StreamSession (CASSANDRA-12903)
 + * Fix Murmur3PartitionerTest (CASSANDRA-12858)
 + * Move cqlsh syntax rules into separate module and allow easier 
customization (CASSANDRA-12897)
 + * Fix CommitLogSegmentManagerTest (CASSANDRA-12283)
 + * Fix cassandra-stress truncate option (CASSANDRA-12695)
 + * Fix crossNode value when receiving messages (CASSANDRA-12791)
 + * Don't load MX4J beans twice (CASSANDRA-12869)
 + * Extend native protocol request flags, add versions to SUPPORTED, and 
introduce ProtocolVersion enum (CASSANDRA-12838)
 + * Set JOINING mode when running pre-join tasks (CASSANDRA-12836)
 + * remove net.mintern.primitive library due to license issue (CASSANDRA-12845)
 + * Properly format IPv6 addresses when logging JMX service URL 
(CASSANDRA-12454)
 + * Optimize the vnode allocation for single replica per DC (CASSANDRA-12777)
 + * Use non-token restrictions for bounds when token restrictions are 
overridden (CASSANDRA-12419)
 + * Fix CQLSH auto completion for PER PARTITION LIMIT (CASSANDRA-12803)
 + * Use different build directories for Eclipse and Ant (CASSANDRA-12466)
 + * Avoid potential AttributeError in cqlsh due to no table metadata 
(CASSANDRA-12815)
 + * Fix RandomReplicationAwareTokenAllocatorTest.testExistingCluster 
(CASSANDRA-12812)
 + * Upgrade commons-codec to 1.9 (CASSANDRA-12790)
 + * Make the fanout size for LeveledCompactionStrategy to be configurable 
(CASSANDRA-11550)
 + * Add duration data type (CASSANDRA-11873)
 + * Fix timeout in ReplicationAwareTokenAllocatorTest (CASSANDRA-12784)
 + * Improve sum aggregate functions (CASSANDRA-12417)
 + * Make cassandra.yaml docs for batch_size_*_threshold_in_kb reflect changes 
in CASSANDRA-10876 (CASSANDRA-12761)
 + * cqlsh fails to format collections when using aliases (CASSANDRA-11534)
 + * Check for hash conflicts in prepared statements (CASSANDRA-12733)
 + * Exit query parsing upon first error (CASSANDRA-12598)
 + * Fix cassandra-stress to use single seed in UUID generation 
(CASSANDRA-12729)
 + * CQLSSTableWriter does not allow Update statement (CASSANDRA-12450)
 + * Config class uses boxed types but DD exposes primitive types 
(CASSANDRA-12199)
 + * Add pre- and post-shutdown hooks to Storage Service (CASSANDRA-12461)
 + * Add hint delivery metrics (CASSANDRA-12693)
 + * Remove IndexInfo cache from FileIndexInfoRetriever (CASSANDRA-12731)
 + * ColumnIndex does not reuse buffer (CASSANDRA-12502)
 + * cdc column addition still breaks schema migration tasks (CASSANDRA-12697)
 + * Upgrade metrics-reporter dependencies (CASSANDRA-12089)
 + * Tune compaction thread count via nodetool (CASSANDRA-12248)
 + * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232)
 + * Include repair session IDs in repair 

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[slebresne]

Merge branch 'cassandra-3.11' into trunk

* cassandra-3.11:
  Read repair is not blocking repair to finish in foreground repair


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c3d72444
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c3d72444
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c3d72444

Branch: refs/heads/trunk
Commit: c3d72444512861c75ec11e17f52770d2613e5364
Parents: 52df6a5 74559de
Author: Sylvain Lebresne 
Authored: Thu Jan 19 16:55:11 2017 +0100
Committer: Sylvain Lebresne 
Committed: Thu Jan 19 16:55:11 2017 +0100

--
 CHANGES.txt |   1 +
 .../UnfilteredPartitionIterators.java   |   1 +
 .../cassandra/service/AsyncRepairCallback.java  |   2 +-
 .../apache/cassandra/service/DataResolver.java  |  14 ++-
 .../cassandra/service/DigestResolver.java   |   9 +-
 .../apache/cassandra/service/ReadCallback.java  |   4 +-
 .../cassandra/service/ResponseResolver.java |  12 ++
 .../cassandra/service/DataResolverTest.java | 117 +--
 8 files changed, 118 insertions(+), 42 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/c3d72444/CHANGES.txt
--
diff --cc CHANGES.txt
index 8f68618,8b613e6..cc16a8f
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -139,11 -109,13 +139,12 @@@
   * Remove pre-startup check for open JMX port (CASSANDRA-12074)
   * Remove compaction Severity from DynamicEndpointSnitch (CASSANDRA-11738)
   * Restore resumable hints delivery (CASSANDRA-11960)
 - * Properly report LWT contention (CASSANDRA-12626)
 + * Properly record CAS contention (CASSANDRA-12626)
  Merged from 3.0:
 + * Stress daemon help is incorrect (CASSANDRA-12563)
+  * Read repair is not blocking repair to finish in foreground repair 
(CASSANDRA-13115)
 - * Stress daemon help is incorrect(CASSANDRA-12563)
 - * Remove ALTER TYPE support (CASSANDRA-12443)
 - * Fix assertion for certain legacy range tombstone pattern (CASSANDRA-12203)
   * Replace empty strings with null values if they cannot be converted 
(CASSANDRA-12794)
 + * Remove support for non-JavaScript UDFs (CASSANDRA-12883)
   * Fix deserialization of 2.x DeletedCells (CASSANDRA-12620)
   * Add parent repair session id to anticompaction log message 
(CASSANDRA-12186)
   * Improve contention handling on failure to acquire MV lock for streaming 
and hints (CASSANDRA-12905)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c3d72444/src/java/org/apache/cassandra/db/partitions/UnfilteredPartitionIterators.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c3d72444/src/java/org/apache/cassandra/service/DataResolver.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c3d72444/src/java/org/apache/cassandra/service/ReadCallback.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c3d72444/test/unit/org/apache/cassandra/service/DataResolverTest.java
--
diff --cc test/unit/org/apache/cassandra/service/DataResolverTest.java
index 3dbc166,7c38224..dba3e95
--- a/test/unit/org/apache/cassandra/service/DataResolverTest.java
+++ b/test/unit/org/apache/cassandra/service/DataResolverTest.java
@@@ -330,14 -353,17 +353,17 @@@ public class DataResolverTes

 .add("c2", "v2")

 .buildUpdate(;
  InetAddress peer2 = peer();
 -resolver.preprocess(readResponseMessage(peer2, 
EmptyIterators.unfilteredPartition(cfm, false)));
 +resolver.preprocess(readResponseMessage(peer2, 
EmptyIterators.unfilteredPartition(cfm)));
  
- try(PartitionIterator data = resolver.resolve();
- RowIterator rows = Iterators.getOnlyElement(data))
+ try(PartitionIterator data = resolver.resolve())
  {
- Row row = Iterators.getOnlyElement(rows);
- assertColumns(row, "c2");
- assertColumn(cfm, row, "c2", "v2", 1);
+ try (RowIterator rows = Iterators.getOnlyElement(data))
+ {
+ Row row = Iterators.getOnlyElement(rows);
+ assertColumns(row, "c2");
+ assertColumn(cfm, row, "c2", "v2", 1);
+ }
+ assertRepairFuture(resolver, 1);
  }
  
  assertEquals(1, 

[2/6] cassandra git commit: Read repair is not blocking repair to finish in foreground repair

[slebresne]

Read repair is not blocking repair to finish in foreground repair

patch by Sylvain Lebresne; reviewed by Xiaolong Jiang and Jason Brown for 
CASSANDRA-13115


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/48fed801
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/48fed801
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/48fed801

Branch: refs/heads/cassandra-3.11
Commit: 48fed80162592f741bf29298e2064452d53de4d8
Parents: 3f41d7a
Author: Sylvain Lebresne 
Authored: Thu Jan 12 10:03:11 2017 +0100
Committer: Sylvain Lebresne 
Committed: Thu Jan 19 16:49:14 2017 +0100

--
 CHANGES.txt |   1 +
 .../UnfilteredPartitionIterators.java   |   1 +
 .../cassandra/service/AsyncRepairCallback.java  |   5 +-
 .../apache/cassandra/service/DataResolver.java  |  14 ++-
 .../cassandra/service/DigestResolver.java   |   9 +-
 .../apache/cassandra/service/ReadCallback.java  |   4 +-
 .../cassandra/service/ResponseResolver.java |  12 ++
 .../cassandra/service/DataResolverTest.java | 117 +--
 8 files changed, 119 insertions(+), 44 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/48fed801/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 97d49af..6293cfa 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Read repair is not blocking repair to finish in foreground repair 
(CASSANDRA-13115)
  * Stress daemon help is incorrect (CASSANDRA-12563)
  * Remove ALTER TYPE support (CASSANDRA-12443)
  * Fix assertion for certain legacy range tombstone pattern (CASSANDRA-12203)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/48fed801/src/java/org/apache/cassandra/db/partitions/UnfilteredPartitionIterators.java
--
diff --git 
a/src/java/org/apache/cassandra/db/partitions/UnfilteredPartitionIterators.java 
b/src/java/org/apache/cassandra/db/partitions/UnfilteredPartitionIterators.java
index 41b1424..1abbb19 100644
--- 
a/src/java/org/apache/cassandra/db/partitions/UnfilteredPartitionIterators.java
+++ 
b/src/java/org/apache/cassandra/db/partitions/UnfilteredPartitionIterators.java
@@ -159,6 +159,7 @@ public abstract class UnfilteredPartitionIterators
 public void close()
 {
 merged.close();
+listener.close();
 }
 };
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/48fed801/src/java/org/apache/cassandra/service/AsyncRepairCallback.java
--
diff --git a/src/java/org/apache/cassandra/service/AsyncRepairCallback.java 
b/src/java/org/apache/cassandra/service/AsyncRepairCallback.java
index dec5319..d613f3d 100644
--- a/src/java/org/apache/cassandra/service/AsyncRepairCallback.java
+++ b/src/java/org/apache/cassandra/service/AsyncRepairCallback.java
@@ -17,7 +17,6 @@
  */
 package org.apache.cassandra.service;
 
-import java.io.IOException;
 import java.util.concurrent.atomic.AtomicInteger;
 
 import org.apache.cassandra.concurrent.Stage;
@@ -46,9 +45,9 @@ public class AsyncRepairCallback implements 
IAsyncCallback
 {
 StageManager.getStage(Stage.READ_REPAIR).execute(new 
WrappedRunnable()
 {
-protected void runMayThrow() throws DigestMismatchException, 
IOException
+protected void runMayThrow()
 {
-repairResolver.resolve();
+repairResolver.compareResponses();
 }
 });
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/48fed801/src/java/org/apache/cassandra/service/DataResolver.java
--
diff --git a/src/java/org/apache/cassandra/service/DataResolver.java 
b/src/java/org/apache/cassandra/service/DataResolver.java
index 4e5bfb8..01953e1 100644
--- a/src/java/org/apache/cassandra/service/DataResolver.java
+++ b/src/java/org/apache/cassandra/service/DataResolver.java
@@ -21,6 +21,8 @@ import java.net.InetAddress;
 import java.util.*;
 import java.util.concurrent.TimeoutException;
 
+import com.google.common.annotations.VisibleForTesting;
+
 import org.apache.cassandra.concurrent.Stage;
 import org.apache.cassandra.concurrent.StageManager;
 import org.apache.cassandra.config.CFMetaData;
@@ -40,7 +42,8 @@ import org.apache.cassandra.utils.FBUtilities;
 
 public class DataResolver extends ResponseResolver
 {
-private final List repairResults = 
Collections.synchronizedList(new ArrayList<>());
+@VisibleForTesting
+final List 

[jira] [Updated] (CASSANDRA-13115) Read repair is not blocking repair to finish in foreground repair

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-13115:
-
Reviewer: Jason Brown

> Read repair is not blocking repair to finish in foreground repair
> -
>
> Key: CASSANDRA-13115
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13115
> Project: Cassandra
>  Issue Type: Bug
> Environment: ccm on OSX 
>Reporter: Xiaolong Jiang
>Assignee: Sylvain Lebresne
> Fix For: 3.0.x, 3.x
>
>
> The code trying to wait(block) for repair result to come back in 3.X is below:
> {code:title= DataResolver.java|borderStyle=solid}
> public void close()
> {
> try
> {
> FBUtilities.waitOnFutures(repairResults, 
> DatabaseDescriptor.getWriteRpcTimeout());
> }
> catch (TimeoutException ex)
> {
> // We got all responses, but timed out while repairing
> int blockFor = consistency.blockFor(keyspace);
> if (Tracing.isTracing())
> Tracing.trace("Timed out while read-repairing after 
> receiving all {} data and digest responses", blockFor);
> else
> logger.debug("Timeout while read-repairing after 
> receiving all {} data and digest responses", blockFor);
> throw new ReadTimeoutException(consistency, blockFor-1, 
> blockFor, true);
> }
> }
> {code}
> in DataResolver class, but this close method is never called and it's also 
> not auto close(RepairMergeListener is not extending from 
> AutoCloseable/CloseableIterator) which means we never wait for repair to 
> finish before returning final result. 
> The steps to reproduce:
> 1. create some keyspace/table with RF = 2
> 2. start 2 nodes using ccm
> 3. stop node2
> 4. disable node1 hinted hand off
> 5. write some data to node1 with consistency level one
> 6. start node2
> 7. query some data from node1 
> This should trigger read repair. I put some log in above close method, and 
> can not see log print put.
> So this bug will basically violate "monotonic quorum reads " guarantee. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-13135) Forced termination of repair session leaves repair jobs running

[Yuki Morishita (JIRA)]

Yuki Morishita created CASSANDRA-13135:
--

 Summary: Forced termination of repair session leaves repair jobs 
running
 Key: CASSANDRA-13135
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13135
 Project: Cassandra
  Issue Type: Bug
Reporter: Yuki Morishita
Assignee: Yuki Morishita
 Fix For: 2.2.x, 3.0.x, 3.x


Forced termination of repair session (by failure detector or jmx) keeps repair 
jobs running that the session created after session is terminated.

This can cause increase in repair time by those unnecessary works left in 
repair job queue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-12443) Remove alter type support

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830081#comment-15830081
 ] 

Aleksey Yeschenko edited comment on CASSANDRA-12443 at 1/19/17 3:26 PM:


Oh, but you aren't taking into account the fact that all that schema code was 
written by drunk baboons. Any change to active {{CFMetaData}} s in the system 
are still done through the {{apply()}} call, until that amazing author of 
CASSANDRA-9425 commits his first patch.


was (Author: iamaleksey):
Oh, but you aren't taking into account the fact that all that schema code was 
written by drunk baboons. Any change to active {{CFMetaData}}s in the system 
are still done through the {{apply()}} call, until that amazing author of 
CASSANDRA-9425 commits his first patch.

> Remove alter type support
> -
>
> Key: CASSANDRA-12443
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12443
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Carl Yeksigian
>Assignee: Carl Yeksigian
> Fix For: 3.0.11, 3.10
>
>
> Currently, we allow altering of types. However, because we no longer store 
> the length for all types anymore, switching from a fixed-width to 
> variable-width type causes issues. commitlog playback breaking startup, 
> queries currently in flight getting back bad results, and special casing 
> required to handle the changes. In addition, this would solve 
> CASSANDRA-10309, as there is no possibility of the types changing while an 
> SSTableReader is open.
> For fixed-length, compatible types, the alter also doesn't add much over a 
> cast, so users could use that in order to retrieve the altered type.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11577) Traces persist for longer than 24 hours

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-11577:
--
Assignee: (was: Aleksey Yeschenko)

> Traces persist for longer than 24 hours
> ---
>
> Key: CASSANDRA-11577
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11577
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Josh Wickman
>Priority: Minor
>
> My deployment currently has clusters on both Cassandra 1.2 (1.2.19) and 2.1 
> (2.1.11) with tracing on.  On 2.1, the trace records persist for longer than 
> the [documented 24 
> hours|https://docs.datastax.com/en/cql/3.3/cql/cql_reference/tracing_r.html]:
> {noformat}
> cqlsh> select started_at from system_traces.sessions limit 10;
>  started_at
> --
>  2016-03-11 23:28:40+
>  2016-03-14 21:09:07+
>  2016-03-14 16:42:25+
>  2016-03-14 16:13:13+
>  2016-03-14 19:12:11+
>  2016-03-14 21:25:57+
>  2016-03-29 22:45:28+
>  2016-03-14 19:56:27+
>  2016-03-09 23:31:41+
>  2016-03-10 23:08:44+
> (10 rows)
> {noformat}
> My systems on 1.2 do not exhibit this problem:
> {noformat}
> cqlsh> select started_at from system_traces.sessions limit 10;
>  started_at
> --
>  2016-04-13 22:49:31+
>  2016-04-14 18:06:45+
>  2016-04-14 07:57:00+
>  2016-04-14 04:35:05+
>  2016-04-14 03:54:20+
>  2016-04-14 10:54:38+
>  2016-04-14 18:34:04+
>  2016-04-14 12:56:57+
>  2016-04-14 01:57:20+
>  2016-04-13 21:36:01+
> {noformat}
> The event records also persist alongside the session records, for example:
> {noformat}
> cqlsh> select session_id, dateOf(event_id) from system_traces.events where 
> session_id = fc8c1e80-e7e0-11e5-a2fb-1968ff3c067b;
>  session_id   | dateOf(event_id)
> --+--
>  fc8c1e80-e7e0-11e5-a2fb-1968ff3c067b | 2016-03-11 23:28:40+
> {noformat}
> Between these versions, the table parameter {{default_time_to_live}} was 
> introduced.  The {{system_traces}} tables report the default value of 0:
> {noformat}
> cqlsh> desc table system_traces.sessions
> CREATE TABLE system_traces.sessions (
> session_id uuid PRIMARY KEY,
> coordinator inet,
> duration int,
> parameters map,
> request text,
> started_at timestamp
> ) WITH bloom_filter_fp_chance = 0.01
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = 'traced sessions'
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
> AND compression = {'sstable_compression': 
> 'org.apache.cassandra.io.compress.SnappyCompressor'}
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 0
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
> {noformat}
> I suspect that {{default_time_to_live}} is superseding the mechanism used in 
> 1.2 to expire the trace records.  Evidently I cannot change this parameter 
> for this table:
> {noformat}
> cqlsh> alter table system_traces.sessions with default_time_to_live = 86400;
> Unauthorized: code=2100 [Unauthorized] message="Cannot ALTER  system_traces.sessions>"
> {noformat}
> I realize Cassandra 1.2 is no longer supported, but the problem is being 
> manifested in Cassandra 2.1 for me (I included 1.2 only for comparison).  
> Since I couldn't find an existing ticket addressing this issue, I'm concerned 
> that it may be present in more recent versions of Cassandra as well, but I 
> have not tested these.
> The persistent trace records are contributing to disk filling, and more 
> importantly, making it more difficult to analyze the trace data.  Is there a 
> workaround for this?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10825) OverloadedException is untested

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830095#comment-15830095
 ] 

Aleksey Yeschenko commented on CASSANDRA-10825:
---

One other issue is that the {{LongAdder}}, used by {{Counter}}, is not a good 
fit for the task. Under normal operation it's rarely being incremented, but 
being read multiple times on each request. {{AtomicLong}} here is actually 
strictly better.

{{LongAdder}} is also not atomic, so the change isn't functionally equivalent 
either.

That said, I'm fine with {{LoadingCache}} here. Not with {{LongAdder}}. Perhaps 
putting {{AtomicLong}} s back, but opting for {{Gauge}} s on the metrics side 
would be a better option?

> OverloadedException is untested
> ---
>
> Key: CASSANDRA-10825
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10825
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths
>Reporter: Ariel Weisberg
>Assignee: Edward Capriolo
> Attachments: jmx-hint.png
>
>
> If you grep test/src and cassandra-dtest you will find that the string 
> OverloadedException doesn't appear anywhere.
> In CASSANDRA-10477 it was found that there were cases where Paxos should 
> back-pressure and throw OverloadedException but didn't.
> If OverloadedException is used for functional purposes then we should test 
> that it is thrown under expected conditions. If there are behaviors driven by 
> catching or tracking OverloadedException we should test those as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12443) Remove alter type support

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830081#comment-15830081
 ] 

Aleksey Yeschenko commented on CASSANDRA-12443:
---

Oh, but you aren't taking into account the fact that all that schema code was 
written by drunk baboons. Any change to active {{CFMetaData}}s in the system 
are still done through the {{apply()}} call, until that amazing author of 
CASSANDRA-9425 commits his first patch.

> Remove alter type support
> -
>
> Key: CASSANDRA-12443
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12443
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Carl Yeksigian
>Assignee: Carl Yeksigian
> Fix For: 3.0.11, 3.10
>
>
> Currently, we allow altering of types. However, because we no longer store 
> the length for all types anymore, switching from a fixed-width to 
> variable-width type causes issues. commitlog playback breaking startup, 
> queries currently in flight getting back bad results, and special casing 
> required to handle the changes. In addition, this would solve 
> CASSANDRA-10309, as there is no possibility of the types changing while an 
> SSTableReader is open.
> For fixed-length, compatible types, the alter also doesn't add much over a 
> cast, so users could use that in order to retrieve the altered type.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-13075) Indexer is not correctly invoked when building indexes over sstables

[Alex Petrov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15829995#comment-15829995
 ] 

Alex Petrov edited comment on CASSANDRA-13075 at 1/19/17 2:15 PM:
--

I've updated patches for all branches and re-ran CI. Unfortunately, the 
failures are also present on trunk (and/or corresponding branch), so they're 
not related to this patch (however situation with tests doesn't make visibility 
better).

|[3.0|https://github.com/ifesdjeen/cassandra/tree/13075-3.0]|[utest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-13075-3.0-testall/]|[dtest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-13075-3.0-dtest/]|
|[3.X|https://github.com/ifesdjeen/cassandra/tree/13075-3.X]|[utest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-13075-3.X-testall/]|[dtest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-13075-3.X-dtest/]|
|[3.11|https://github.com/ifesdjeen/cassandra/tree/13075-3.11]|[utest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-13075-3.11-testall/]|[dtest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-13075-3.11-dtest/]|
|[trunk|https://github.com/ifesdjeen/cassandra/tree/13075-trunk]|[utest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-13075-trunk-testall/]|[dtest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-13075-trunk-dtest/]|

Patch for {{3.0}} is slightly different in terms of passed arguments, {{3.X}} 
and {{3.11}} are similar. trunk's only difference is that there's no thrift so 
calls were thrift boolean was passed are different. 



was (Author: ifesdjeen):
I've updated patches for all branches and re-ran CI. Unfortunately, the 
failures are also present on trunk (and/or corresponding branch), so they're 
not related to this patch (however situation with tests doesn't make visibility 
better).

|[3.0|https://github.com/ifesdjeen/cassandra/tree/13075-3.0]|[utest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-13075-3.0-testall/]|[dtest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-13075-3.0-dtest/]|
|[3.X|https://github.com/ifesdjeen/cassandra/tree/13075-3.X]|[utest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-13075-3.X-testall/]|[dtest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-13075-3.X-dtest/]|
|[3.11|https://github.com/ifesdjeen/cassandra/tree/13075-3.11]|[utest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-13075-3.11-testall/]|[dtest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-13075-3.11-dtest/]|
|[trunk|https://github.com/ifesdjeen/cassandra/tree/13075-trunk]|[utest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-13075-trunk-testall/]|[dtest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-13075-trunk-dtest/]|


> Indexer is not correctly invoked when building indexes over sstables
> 
>
> Key: CASSANDRA-13075
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13075
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Sergio Bossa
>Assignee: Alex Petrov
>Priority: Critical
> Attachments: CustomIndexTest.java
>
>
> Following CASSANDRA-12796, {{SecondaryIndexManager#indexPartition()}} calls 
> each {{Indexer}} {{begin}} and {{finish}} methods multiple times per 
> partition (depending on the page size), as 
> {{PartitionIterators#getOnlyElement()}} returns an empty partition even when 
> the iterator is exhausted.
> This leads to bugs for {{Indexer}} implementations doing actual work in those 
>  methods, but even worse, it provides the {{Indexer}} the same input of an 
> empty partition containing only a non-live partition deletion, as the 
> {{Indexer#partitionDelete()}} method is *not* actually called.
> My proposed solution:
> 1) Stop the iteration before the empty partition is returned and ingested 
> into the {{Indexer}}.
> 2) Actually call the {{Indexer#partitionDelete()}} method inside 
> {{SecondaryIndexManager#indexPartition()}} (which requires to use a filtered 
> iterator so it actually contains the deletion info).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-13075) Indexer is not correctly invoked when building indexes over sstables

[Alex Petrov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15829995#comment-15829995
 ] 

Alex Petrov commented on CASSANDRA-13075:
-

I've updated patches for all branches and re-ran CI. Unfortunately, the 
failures are also present on trunk (and/or corresponding branch), so they're 
not related to this patch (however situation with tests doesn't make visibility 
better).

|[3.0|https://github.com/ifesdjeen/cassandra/tree/13075-3.0]|[utest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-13075-3.0-testall/]|[dtest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-13075-3.0-dtest/]|
|[3.X|https://github.com/ifesdjeen/cassandra/tree/13075-3.X]|[utest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-13075-3.X-testall/]|[dtest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-13075-3.X-dtest/]|
|[3.11|https://github.com/ifesdjeen/cassandra/tree/13075-3.11]|[utest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-13075-3.11-testall/]|[dtest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-13075-3.11-dtest/]|
|[trunk|https://github.com/ifesdjeen/cassandra/tree/13075-trunk]|[utest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-13075-trunk-testall/]|[dtest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-13075-trunk-dtest/]|


> Indexer is not correctly invoked when building indexes over sstables
> 
>
> Key: CASSANDRA-13075
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13075
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Sergio Bossa
>Assignee: Alex Petrov
>Priority: Critical
> Attachments: CustomIndexTest.java
>
>
> Following CASSANDRA-12796, {{SecondaryIndexManager#indexPartition()}} calls 
> each {{Indexer}} {{begin}} and {{finish}} methods multiple times per 
> partition (depending on the page size), as 
> {{PartitionIterators#getOnlyElement()}} returns an empty partition even when 
> the iterator is exhausted.
> This leads to bugs for {{Indexer}} implementations doing actual work in those 
>  methods, but even worse, it provides the {{Indexer}} the same input of an 
> empty partition containing only a non-live partition deletion, as the 
> {{Indexer#partitionDelete()}} method is *not* actually called.
> My proposed solution:
> 1) Stop the iteration before the empty partition is returned and ingested 
> into the {{Indexer}}.
> 2) Actually call the {{Indexer#partitionDelete()}} method inside 
> {{SecondaryIndexManager#indexPartition()}} (which requires to use a filtered 
> iterator so it actually contains the deletion info).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-13115) Read repair is not blocking repair to finish in foreground repair

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15829888#comment-15829888
 ] 

Jason Brown commented on CASSANDRA-13115:
-

+1

> Read repair is not blocking repair to finish in foreground repair
> -
>
> Key: CASSANDRA-13115
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13115
> Project: Cassandra
>  Issue Type: Bug
> Environment: ccm on OSX 
>Reporter: Xiaolong Jiang
>Assignee: Sylvain Lebresne
> Fix For: 3.0.x, 3.x
>
>
> The code trying to wait(block) for repair result to come back in 3.X is below:
> {code:title= DataResolver.java|borderStyle=solid}
> public void close()
> {
> try
> {
> FBUtilities.waitOnFutures(repairResults, 
> DatabaseDescriptor.getWriteRpcTimeout());
> }
> catch (TimeoutException ex)
> {
> // We got all responses, but timed out while repairing
> int blockFor = consistency.blockFor(keyspace);
> if (Tracing.isTracing())
> Tracing.trace("Timed out while read-repairing after 
> receiving all {} data and digest responses", blockFor);
> else
> logger.debug("Timeout while read-repairing after 
> receiving all {} data and digest responses", blockFor);
> throw new ReadTimeoutException(consistency, blockFor-1, 
> blockFor, true);
> }
> }
> {code}
> in DataResolver class, but this close method is never called and it's also 
> not auto close(RepairMergeListener is not extending from 
> AutoCloseable/CloseableIterator) which means we never wait for repair to 
> finish before returning final result. 
> The steps to reproduce:
> 1. create some keyspace/table with RF = 2
> 2. start 2 nodes using ccm
> 3. stop node2
> 4. disable node1 hinted hand off
> 5. write some data to node1 with consistency level one
> 6. start node2
> 7. query some data from node1 
> This should trigger read repair. I put some log in above close method, and 
> can not see log print put.
> So this bug will basically violate "monotonic quorum reads " guarantee. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-13125) Duplicate rows after upgrading from 2.1.16 to 3.0.10/3.9

[Joshua McKenzie (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua McKenzie updated CASSANDRA-13125:

Reproduced In: 3.9, 3.0.10  (was: 3.0.10, 3.9)
 Reviewer: Sylvain Lebresne

> Duplicate rows after upgrading from 2.1.16 to 3.0.10/3.9
> 
>
> Key: CASSANDRA-13125
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13125
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Zhongxiang Zheng
>Assignee: Yasuharu Goto
> Attachments: diff-a.patch, diff-b.patch
>
>
> I found that rows are splitting and duplicated after upgrading the cluster 
> from 2.1.x to 3.0.x.
> I found the way to reproduce the problem as below.
> {code}
> $ ccm create test -v 2.1.16 -n 3 -s   
> 
> Current cluster is now: test
> $ ccm node1 cqlsh  -e "CREATE KEYSPACE test WITH replication = 
> {'class':'SimpleStrategy', 'replication_factor':3}"
> $ ccm node1 cqlsh -e "CREATE TABLE test.test (id text PRIMARY KEY, value1 
> set, value2 set);"
> # Upgrade node1
> $ for i in 1; do ccm node${i} stop; ccm node${i} setdir -v3.0.10; ccm 
> node${i} start;ccm node${i} nodetool upgradesstables; done
> # Insert a row through node1(3.0.10)
> $ ccm node1 cqlsh -e "INSERT INTO test.test (id, value1, value2) values 
> ('aaa', {'aaa', 'bbb'}, {'ccc', 'ddd'});"   
> # Insert a row through node2(2.1.16)
> $ ccm node2 cqlsh -e "INSERT INTO test.test (id, value1, value2) values 
> ('bbb', {'aaa', 'bbb'}, {'ccc', 'ddd'});" 
> # The row inserted from node1 is splitting
> $ ccm node1 cqlsh -e "SELECT * FROM test.test ;"
>  id  | value1 | value2
> -++
>  aaa |   null |   null
>  aaa | {'aaa', 'bbb'} | {'ccc', 'ddd'}
>  bbb | {'aaa', 'bbb'} | {'ccc', 'ddd'}
> $ for i in 1 2; do ccm node${i} nodetool flush; done
> # Results of sstable2json of node2. The row inserted from node1(3.0.10) is 
> different from the row inserted from node2(2.1.16).
> $ ccm node2 json -k test -c test
> running
> ['/home/zzheng/.ccm/test/node2/data0/test/test-5406ee80dbdb11e6a175f57c4c7c85f3/test-test-ka-1-Data.db']
> -- test-test-ka-1-Data.db -
> [
> {"key": "aaa",
>  "cells": [["","",1484564624769577],
>["value1","value2:!",1484564624769576,"t",1484564624],
>["value1:616161","",1484564624769577],
>["value1:626262","",1484564624769577],
>["value2:636363","",1484564624769577],
>["value2:646464","",1484564624769577]]},
> {"key": "bbb",
>  "cells": [["","",1484564634508029],
>["value1:_","value1:!",1484564634508028,"t",1484564634],
>["value1:616161","",1484564634508029],
>["value1:626262","",1484564634508029],
>["value2:_","value2:!",1484564634508028,"t",1484564634],
>["value2:636363","",1484564634508029],
>["value2:646464","",1484564634508029]]}
> ]
> # Upgrade node2,3
> $ for i in `seq 2 3`; do ccm node${i} stop; ccm node${i} setdir -v3.0.10; ccm 
> node${i} start;ccm node${i} nodetool upgradesstables; done
> # After upgrade node2,3, the row inserted from node1 is splitting in node2,3
> $ ccm node2 cqlsh -e "SELECT * FROM test.test ;"  
>   
>  id  | value1 | value2
> -++
>  aaa |   null |   null
>  aaa | {'aaa', 'bbb'} | {'ccc', 'ddd'}
>  bbb | {'aaa', 'bbb'} | {'ccc', 'ddd'}
> (3 rows)
> # Results of sstabledump
> # node1
> [
>   {
> "partition" : {
>   "key" : [ "aaa" ],
>   "position" : 0
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 17,
> "liveness_info" : { "tstamp" : "2017-01-16T11:03:44.769577Z" },
> "cells" : [
>   { "name" : "value1", "deletion_info" : { "marked_deleted" : 
> "2017-01-16T11:03:44.769576Z", "local_delete_time" : "2017-01-16T11:03:44Z" } 
> },
>   { "name" : "value1", "path" : [ "aaa" ], "value" : "" },
>   { "name" : "value1", "path" : [ "bbb" ], "value" : "" },
>   { "name" : "value2", "deletion_info" : { "marked_deleted" : 
> "2017-01-16T11:03:44.769576Z", "local_delete_time" : "2017-01-16T11:03:44Z" } 
> },
>   { "name" : "value2", "path" : [ "ccc" ], "value" : "" },
>   { "name" : "value2", "path" : [ "ddd" ], "value" : "" }
> ]
>   }
> ]
>   },
>   {
> "partition" : {
>   "key" : [ "bbb" ],
>   "position" : 48
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 65,
> "liveness_info" : { "tstamp" : "2017-01-16T11:03:54.508029Z" },
> "cells" : [
>   { "name" : "value1", 

[jira] [Updated] (CASSANDRA-13125) Duplicate rows after upgrading from 2.1.16 to 3.0.10/3.9

[Joshua McKenzie (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua McKenzie updated CASSANDRA-13125:

Assignee: Yasuharu Goto

> Duplicate rows after upgrading from 2.1.16 to 3.0.10/3.9
> 
>
> Key: CASSANDRA-13125
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13125
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Zhongxiang Zheng
>Assignee: Yasuharu Goto
> Attachments: diff-a.patch, diff-b.patch
>
>
> I found that rows are splitting and duplicated after upgrading the cluster 
> from 2.1.x to 3.0.x.
> I found the way to reproduce the problem as below.
> {code}
> $ ccm create test -v 2.1.16 -n 3 -s   
> 
> Current cluster is now: test
> $ ccm node1 cqlsh  -e "CREATE KEYSPACE test WITH replication = 
> {'class':'SimpleStrategy', 'replication_factor':3}"
> $ ccm node1 cqlsh -e "CREATE TABLE test.test (id text PRIMARY KEY, value1 
> set, value2 set);"
> # Upgrade node1
> $ for i in 1; do ccm node${i} stop; ccm node${i} setdir -v3.0.10; ccm 
> node${i} start;ccm node${i} nodetool upgradesstables; done
> # Insert a row through node1(3.0.10)
> $ ccm node1 cqlsh -e "INSERT INTO test.test (id, value1, value2) values 
> ('aaa', {'aaa', 'bbb'}, {'ccc', 'ddd'});"   
> # Insert a row through node2(2.1.16)
> $ ccm node2 cqlsh -e "INSERT INTO test.test (id, value1, value2) values 
> ('bbb', {'aaa', 'bbb'}, {'ccc', 'ddd'});" 
> # The row inserted from node1 is splitting
> $ ccm node1 cqlsh -e "SELECT * FROM test.test ;"
>  id  | value1 | value2
> -++
>  aaa |   null |   null
>  aaa | {'aaa', 'bbb'} | {'ccc', 'ddd'}
>  bbb | {'aaa', 'bbb'} | {'ccc', 'ddd'}
> $ for i in 1 2; do ccm node${i} nodetool flush; done
> # Results of sstable2json of node2. The row inserted from node1(3.0.10) is 
> different from the row inserted from node2(2.1.16).
> $ ccm node2 json -k test -c test
> running
> ['/home/zzheng/.ccm/test/node2/data0/test/test-5406ee80dbdb11e6a175f57c4c7c85f3/test-test-ka-1-Data.db']
> -- test-test-ka-1-Data.db -
> [
> {"key": "aaa",
>  "cells": [["","",1484564624769577],
>["value1","value2:!",1484564624769576,"t",1484564624],
>["value1:616161","",1484564624769577],
>["value1:626262","",1484564624769577],
>["value2:636363","",1484564624769577],
>["value2:646464","",1484564624769577]]},
> {"key": "bbb",
>  "cells": [["","",1484564634508029],
>["value1:_","value1:!",1484564634508028,"t",1484564634],
>["value1:616161","",1484564634508029],
>["value1:626262","",1484564634508029],
>["value2:_","value2:!",1484564634508028,"t",1484564634],
>["value2:636363","",1484564634508029],
>["value2:646464","",1484564634508029]]}
> ]
> # Upgrade node2,3
> $ for i in `seq 2 3`; do ccm node${i} stop; ccm node${i} setdir -v3.0.10; ccm 
> node${i} start;ccm node${i} nodetool upgradesstables; done
> # After upgrade node2,3, the row inserted from node1 is splitting in node2,3
> $ ccm node2 cqlsh -e "SELECT * FROM test.test ;"  
>   
>  id  | value1 | value2
> -++
>  aaa |   null |   null
>  aaa | {'aaa', 'bbb'} | {'ccc', 'ddd'}
>  bbb | {'aaa', 'bbb'} | {'ccc', 'ddd'}
> (3 rows)
> # Results of sstabledump
> # node1
> [
>   {
> "partition" : {
>   "key" : [ "aaa" ],
>   "position" : 0
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 17,
> "liveness_info" : { "tstamp" : "2017-01-16T11:03:44.769577Z" },
> "cells" : [
>   { "name" : "value1", "deletion_info" : { "marked_deleted" : 
> "2017-01-16T11:03:44.769576Z", "local_delete_time" : "2017-01-16T11:03:44Z" } 
> },
>   { "name" : "value1", "path" : [ "aaa" ], "value" : "" },
>   { "name" : "value1", "path" : [ "bbb" ], "value" : "" },
>   { "name" : "value2", "deletion_info" : { "marked_deleted" : 
> "2017-01-16T11:03:44.769576Z", "local_delete_time" : "2017-01-16T11:03:44Z" } 
> },
>   { "name" : "value2", "path" : [ "ccc" ], "value" : "" },
>   { "name" : "value2", "path" : [ "ddd" ], "value" : "" }
> ]
>   }
> ]
>   },
>   {
> "partition" : {
>   "key" : [ "bbb" ],
>   "position" : 48
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 65,
> "liveness_info" : { "tstamp" : "2017-01-16T11:03:54.508029Z" },
> "cells" : [
>   { "name" : "value1", "deletion_info" : { "marked_deleted" : 
> 

[jira] [Updated] (CASSANDRA-13123) Draining a node might fail to delete all inactive commitlogs

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown updated CASSANDRA-13123:

Assignee: Jan Urbański

> Draining a node might fail to delete all inactive commitlogs
> 
>
> Key: CASSANDRA-13123
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13123
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths
>Reporter: Jan Urbański
>Assignee: Jan Urbański
> Fix For: 3.8
>
> Attachments: 13123-2.2.8.txt, 13123-3.0.10.txt, 13123-3.9.txt, 
> 13123-trunk.txt
>
>
> After issuing a drain command, it's possible that not all of the inactive 
> commitlogs are removed.
> The drain command shuts down the CommitLog instance, which in turn shuts down 
> the CommitLogSegmentManager. This has the effect of discarding any pending 
> management tasks it might have, like the removal of inactive commitlogs.
> This in turn leads to an excessive amount of commitlogs being left behind 
> after a drain and a lengthy recovery after a restart. With a fleet of dozens 
> of nodes, each of them leaving several GB of commitlogs after a drain and 
> taking up to two minutes to recover them on restart, the additional time 
> required to restart the entire fleet becomes noticeable.
> This problem is not present in 3.x or trunk because of the CLSM rewrite done 
> in CASSANDRA-8844.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-13123) Draining a node might fail to delete all inactive commitlogs

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown updated CASSANDRA-13123:

Reproduced In: 3.0.10, 2.2.8  (was: 2.2.8, 3.0.10)
 Reviewer: Jason Brown

> Draining a node might fail to delete all inactive commitlogs
> 
>
> Key: CASSANDRA-13123
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13123
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths
>Reporter: Jan Urbański
>Assignee: Jan Urbański
> Fix For: 3.8
>
> Attachments: 13123-2.2.8.txt, 13123-3.0.10.txt, 13123-3.9.txt, 
> 13123-trunk.txt
>
>
> After issuing a drain command, it's possible that not all of the inactive 
> commitlogs are removed.
> The drain command shuts down the CommitLog instance, which in turn shuts down 
> the CommitLogSegmentManager. This has the effect of discarding any pending 
> management tasks it might have, like the removal of inactive commitlogs.
> This in turn leads to an excessive amount of commitlogs being left behind 
> after a drain and a lengthy recovery after a restart. With a fleet of dozens 
> of nodes, each of them leaving several GB of commitlogs after a drain and 
> taking up to two minutes to recover them on restart, the additional time 
> required to restart the entire fleet becomes noticeable.
> This problem is not present in 3.x or trunk because of the CLSM rewrite done 
> in CASSANDRA-8844.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-13123) Draining a node might fail to delete all inactive commitlogs

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15829865#comment-15829865
 ] 

Jason Brown commented on CASSANDRA-13123:
-

[~wulczer] Thanks for the patch. We are at the critical bug fix stage with 2.2, 
so I'll only look at the patch for 3.0 and up. I've taken a quick look and 
things seem legit (need to think about it a bit more), but can you comment on 
any startup improvement time you've observed, if you've deployed this?

Also, when you are issuing a drain? On normal node restarts, or only at 
"special" events, like upgrading a node?

> Draining a node might fail to delete all inactive commitlogs
> 
>
> Key: CASSANDRA-13123
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13123
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths
>Reporter: Jan Urbański
> Fix For: 3.8
>
> Attachments: 13123-2.2.8.txt, 13123-3.0.10.txt, 13123-3.9.txt, 
> 13123-trunk.txt
>
>
> After issuing a drain command, it's possible that not all of the inactive 
> commitlogs are removed.
> The drain command shuts down the CommitLog instance, which in turn shuts down 
> the CommitLogSegmentManager. This has the effect of discarding any pending 
> management tasks it might have, like the removal of inactive commitlogs.
> This in turn leads to an excessive amount of commitlogs being left behind 
> after a drain and a lengthy recovery after a restart. With a fleet of dozens 
> of nodes, each of them leaving several GB of commitlogs after a drain and 
> taking up to two minutes to recover them on restart, the additional time 
> required to restart the entire fleet becomes noticeable.
> This problem is not present in 3.x or trunk because of the CLSM rewrite done 
> in CASSANDRA-8844.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-13015) improved compactions metrics

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown updated CASSANDRA-13015:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

committed as sha {{52df6a58d18200c11d2e9d88fc9276b5ef1ee06a}} to 4.0. thanks!

> improved compactions metrics
> 
>
> Key: CASSANDRA-13015
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13015
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Jon Haddad
>Assignee: Jon Haddad
> Fix For: 4.0
>
>
> Compaction stats is missing some useful metrics.
> * Number of sstables dropped out of compaction due to disk space
> * Number of compactions that had to drop sstables
> * Compactions aborted (due to reduced scope failure)
> There will be more, I'll edit the description with the list as I figure it 
> out.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

cassandra git commit: Added compaction metrics to track failures when running low on disk space

[jasobrown]

Repository: cassandra
Updated Branches:
  refs/heads/trunk d32ae4a60 -> 52df6a58d


Added compaction metrics to track failures when running low on disk space

patch by John Haddad; reviewed by jasobrown for CASSANDRA-13015


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/52df6a58
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/52df6a58
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/52df6a58

Branch: refs/heads/trunk
Commit: 52df6a58d18200c11d2e9d88fc9276b5ef1ee06a
Parents: d32ae4a
Author: Jon Haddad 
Authored: Tue Jan 10 22:32:34 2017 -0800
Committer: Jason Brown 
Committed: Thu Jan 19 04:29:44 2017 -0800

--
 CHANGES.txt |  1 +
 .../cassandra/db/compaction/CompactionManager.java  | 16 
 .../cassandra/db/compaction/CompactionTask.java | 15 +--
 .../apache/cassandra/metrics/CompactionMetrics.java | 15 +++
 4 files changed, 45 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/52df6a58/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index a67cf43..8f68618 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 4.0
+ * Improved compactions metrics (CASSANDRA-13015)
  * Speed-up start-up sequence by avoiding un-needed flushes (CASSANDRA-13031)
  * Use Caffeine (W-TinyLFU) for on-heap caches (CASSANDRA-10855)
  * Thrift removal (CASSANDRA-5)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/52df6a58/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index fb11bad..9c74f62 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -1838,6 +1838,22 @@ public class CompactionManager implements 
CompactionManagerMBean
 void finishCompaction(CompactionInfo.Holder ci);
 }
 
+public void incrementAborted()
+{
+metrics.compactionsAborted.inc();
+}
+
+public void incrementCompactionsReduced()
+{
+metrics.compactionsReduced.inc();
+}
+
+public void incrementSstablesDropppedFromCompactions(long num)
+{
+metrics.sstablesDropppedFromCompactions.inc(num);
+}
+
+
 public List> getCompactions()
 {
 List compactionHolders = CompactionMetrics.getCompactions();

http://git-wip-us.apache.org/repos/asf/cassandra/blob/52df6a58/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionTask.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
index b2e9b8c..62efa3d 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
@@ -27,7 +27,6 @@ import java.util.concurrent.TimeUnit;
 import com.google.common.base.Predicate;
 import com.google.common.collect.Iterables;
 import com.google.common.collect.Sets;
-import com.google.common.primitives.Ints;
 import com.google.common.util.concurrent.RateLimiter;
 
 import org.apache.cassandra.db.Directories;
@@ -333,23 +332,35 @@ public class CompactionTask extends AbstractCompactionTask
 }
 
 CompactionStrategyManager strategy = 
cfs.getCompactionStrategyManager();
-
+int sstablesRemoved = 0;
 while(true)
 {
 long expectedWriteSize = 
cfs.getExpectedCompactedFileSize(transaction.originals(), compactionType);
 long estimatedSSTables = Math.max(1, expectedWriteSize / 
strategy.getMaxSSTableBytes());
 
 if(cfs.getDirectories().hasAvailableDiskSpace(estimatedSSTables, 
expectedWriteSize))
+{
+// we're ok now on space so now we track the failures, if any
+if(sstablesRemoved > 0)
+{
+CompactionManager.instance.incrementCompactionsReduced();
+
CompactionManager.instance.incrementSstablesDropppedFromCompactions(sstablesRemoved);
+}
+
 break;
+}
 
 if (!reduceScopeForLimitedSpace(expectedWriteSize))
 {
 // we end up here if we can't take any more sstables out of 
the compaction.
 // usually means we've run out of disk space
 String msg = String.format("Not 

[jira] [Commented] (CASSANDRA-12443) Remove alter type support

[Benjamin Lerer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15829653#comment-15829653
 ] 

Benjamin Lerer commented on CASSANDRA-12443:


Base on the code of 3.0 it seems that when an {{UDT}}  is altered the 
{{CFMetaData}} are recreated and by consequence the {{keyValidator}} and 
{{comparator}} too.

> Remove alter type support
> -
>
> Key: CASSANDRA-12443
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12443
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Carl Yeksigian
>Assignee: Carl Yeksigian
> Fix For: 3.0.11, 3.10
>
>
> Currently, we allow altering of types. However, because we no longer store 
> the length for all types anymore, switching from a fixed-width to 
> variable-width type causes issues. commitlog playback breaking startup, 
> queries currently in flight getting back bad results, and special casing 
> required to handle the changes. In addition, this would solve 
> CASSANDRA-10309, as there is no possibility of the types changing while an 
> SSTableReader is open.
> For fixed-length, compatible types, the alter also doesn't add much over a 
> cast, so users could use that in order to retrieve the altered type.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-13005) Cassandra TWCS is not removing fully expired tables

[Christian Esken (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christian Esken updated CASSANDRA-13005:

Description: 
I have a table where all columns are stored with TTL of maximum 4 hours. 
Usually TWCS compaction properly removes  expired data via tombstone compaction 
and also removes fully expired tables. The number of SSTables is nearly 
constant since weeks. Good.

The problem:  Suddenly TWCS does not remove old SSTables any longer. They are 
being recreated frequently (judging form the file creation timestamp), but the 
number of tables is growing. Analysis and actions take so far:
- sstablemetadata shows strange data, as if the table is completely empty.
- sstabledump throws an Exception when running it on such a SSTable
- Even triggering a manual major compaction will not remove the old SSTable's. 
To be more precise: They are recreated with new id and timestamp (not sure 
whether they are identical as I cannot inspect content due to the sstabledump 
crash)

{color:blue}edit 2017-01-19: This ticket may be obsolete. See the later 
comments for more information.{color}


  was:
I have a table where all columns are stored with TTL of maximum 4 hours. 
Usually TWCS compaction properly removes  expired data via tombstone compaction 
and also removes fully expired tables. The number of SSTables is nearly 
constant since weeks. Good.

The problem:  Suddenly TWCS does not remove old SSTables any longer. They are 
being recreated frequently (judging form the file creation timestamp), but the 
number of tables is growing. Analysis and actions take so far:
- sstablemetadata shows strange data, as if the table is completely empty.
- sstabledump throws an Exception when running it on such a SSTable
- Even triggering a manual major compaction will not remove the old SSTable's. 
To be more precise: They are recreated with new id and timestamp (not sure 
whether they are identical as I cannot inspect content due to the sstabledump 
crash)

{color:blue}edit 2017-0-19: This ticket may be obsolete. See the later comments 
for more information.{color}



> Cassandra TWCS is not removing fully expired tables
> ---
>
> Key: CASSANDRA-13005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13005
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
> Environment: Cassandra 3.0.9
> Java HotSpot(TM) 64-Bit Server VM version 25.112-b15 (Java version 
> 1.8.0_112-b15)
> Linux 3.16
>Reporter: Christian Esken
>Priority: Minor
>  Labels: twcs
> Attachments: sstablemetadata-empty-type-that-is-3GB.txt
>
>
> I have a table where all columns are stored with TTL of maximum 4 hours. 
> Usually TWCS compaction properly removes  expired data via tombstone 
> compaction and also removes fully expired tables. The number of SSTables is 
> nearly constant since weeks. Good.
> The problem:  Suddenly TWCS does not remove old SSTables any longer. They are 
> being recreated frequently (judging form the file creation timestamp), but 
> the number of tables is growing. Analysis and actions take so far:
> - sstablemetadata shows strange data, as if the table is completely empty.
> - sstabledump throws an Exception when running it on such a SSTable
> - Even triggering a manual major compaction will not remove the old 
> SSTable's. To be more precise: They are recreated with new id and timestamp 
> (not sure whether they are identical as I cannot inspect content due to the 
> sstabledump crash)
> {color:blue}edit 2017-01-19: This ticket may be obsolete. See the later 
> comments for more information.{color}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-13005) Cassandra TWCS is not removing fully expired tables

[Christian Esken (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christian Esken updated CASSANDRA-13005:

Description: 
I have a table where all columns are stored with TTL of maximum 4 hours. 
Usually TWCS compaction properly removes  expired data via tombstone compaction 
and also removes fully expired tables. The number of SSTables is nearly 
constant since weeks. Good.

The problem:  Suddenly TWCS does not remove old SSTables any longer. They are 
being recreated frequently (judging form the file creation timestamp), but the 
number of tables is growing. Analysis and actions take so far:
- sstablemetadata shows strange data, as if the table is completely empty.
- sstabledump throws an Exception when running it on such a SSTable
- Even triggering a manual major compaction will not remove the old SSTable's. 
To be more precise: They are recreated with new id and timestamp (not sure 
whether they are identical as I cannot inspect content due to the sstabledump 
crash)

{color:blue}edit 2017-0-19: This ticket may be obsolete. See the later comments 
for more information.{color}


  was:
I have a table where all columns are stored with TTL of maximum 4 hours. 
Usually TWCS compaction properly removes  expired data via tombstone compaction 
and also removes fully expired tables. The number of SSTables is nearly 
constant since weeks. Good.

The problem:  Suddenly TWCS does not remove old SSTables any longer. They are 
being recreated frequently (judging form the file creation timestamp), but the 
number of tables is growing. Analysis and actions take so far:
- sstablemetadata shows strange data, as if the table is completely empty.
- sstabledump throws an Exception when running it on such a SSTable
- Even triggering a manual major compaction will not remove the old SSTable's. 
To be more precise: They are recreated with new id and timestamp (not sure 
whether they are identical as I cannot inspect content due to the sstabledump 
crash)




> Cassandra TWCS is not removing fully expired tables
> ---
>
> Key: CASSANDRA-13005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13005
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
> Environment: Cassandra 3.0.9
> Java HotSpot(TM) 64-Bit Server VM version 25.112-b15 (Java version 
> 1.8.0_112-b15)
> Linux 3.16
>Reporter: Christian Esken
>Priority: Minor
>  Labels: twcs
> Attachments: sstablemetadata-empty-type-that-is-3GB.txt
>
>
> I have a table where all columns are stored with TTL of maximum 4 hours. 
> Usually TWCS compaction properly removes  expired data via tombstone 
> compaction and also removes fully expired tables. The number of SSTables is 
> nearly constant since weeks. Good.
> The problem:  Suddenly TWCS does not remove old SSTables any longer. They are 
> being recreated frequently (judging form the file creation timestamp), but 
> the number of tables is growing. Analysis and actions take so far:
> - sstablemetadata shows strange data, as if the table is completely empty.
> - sstabledump throws an Exception when running it on such a SSTable
> - Even triggering a manual major compaction will not remove the old 
> SSTable's. To be more precise: They are recreated with new id and timestamp 
> (not sure whether they are identical as I cannot inspect content due to the 
> sstabledump crash)
> {color:blue}edit 2017-0-19: This ticket may be obsolete. See the later 
> comments for more information.{color}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-13005) Cassandra TWCS is not removing fully expired tables

[Christian Esken (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15829587#comment-15829587
 ] 

Christian Esken commented on CASSANDRA-13005:
-

I am not anymore convinced that my bug report is fully accurate.
- It is true that files are missing
- It is true that NullPointerException happens within sstabledump
- OTOH missing files get created after some time. It may be the case that this 
always happens, even though it sometimes takes a long time (I did not yet 
measure this, but I encountered cases where the files were not there even after 
1 minute)
- I inspected some of the data files, after the missing files were created. At 
that point of time they were correct and contained non-expired data.

Thus, this may not be a bug. I will lower priority, but will keep this bug 
report open for some time.


> Cassandra TWCS is not removing fully expired tables
> ---
>
> Key: CASSANDRA-13005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13005
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
> Environment: Cassandra 3.0.9
> Java HotSpot(TM) 64-Bit Server VM version 25.112-b15 (Java version 
> 1.8.0_112-b15)
> Linux 3.16
>Reporter: Christian Esken
>  Labels: twcs
> Attachments: sstablemetadata-empty-type-that-is-3GB.txt
>
>
> I have a table where all columns are stored with TTL of maximum 4 hours. 
> Usually TWCS compaction properly removes  expired data via tombstone 
> compaction and also removes fully expired tables. The number of SSTables is 
> nearly constant since weeks. Good.
> The problem:  Suddenly TWCS does not remove old SSTables any longer. They are 
> being recreated frequently (judging form the file creation timestamp), but 
> the number of tables is growing. Analysis and actions take so far:
> - sstablemetadata shows strange data, as if the table is completely empty.
> - sstabledump throws an Exception when running it on such a SSTable
> - Even triggering a manual major compaction will not remove the old 
> SSTable's. To be more precise: They are recreated with new id and timestamp 
> (not sure whether they are identical as I cannot inspect content due to the 
> sstabledump crash)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-13005) Cassandra TWCS is not removing fully expired tables

[Christian Esken (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christian Esken updated CASSANDRA-13005:

Priority: Minor  (was: Major)

> Cassandra TWCS is not removing fully expired tables
> ---
>
> Key: CASSANDRA-13005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13005
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
> Environment: Cassandra 3.0.9
> Java HotSpot(TM) 64-Bit Server VM version 25.112-b15 (Java version 
> 1.8.0_112-b15)
> Linux 3.16
>Reporter: Christian Esken
>Priority: Minor
>  Labels: twcs
> Attachments: sstablemetadata-empty-type-that-is-3GB.txt
>
>
> I have a table where all columns are stored with TTL of maximum 4 hours. 
> Usually TWCS compaction properly removes  expired data via tombstone 
> compaction and also removes fully expired tables. The number of SSTables is 
> nearly constant since weeks. Good.
> The problem:  Suddenly TWCS does not remove old SSTables any longer. They are 
> being recreated frequently (judging form the file creation timestamp), but 
> the number of tables is growing. Analysis and actions take so far:
> - sstablemetadata shows strange data, as if the table is completely empty.
> - sstabledump throws an Exception when running it on such a SSTable
> - Even triggering a manual major compaction will not remove the old 
> SSTable's. To be more precise: They are recreated with new id and timestamp 
> (not sure whether they are identical as I cannot inspect content due to the 
> sstabledump crash)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)