date:20180406

[jira] [Updated] (CASSANDRA-14369) infinite loop when decommission a node

2018-04-06 Thread Daniel Woo (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Woo updated CASSANDRA-14369:
---
Description: 
I have 6 nodes (N1 to N6)， N2 to N6 are new hardwares with two SSDs on each, N1 
is an old box with spinning disks, and I am trying to decommission N1. Then I 
see two nodes are trying to receive streaming from N1 infinitely. The log 
rotates so quickly that I can only see this:

 

{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,561 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,561 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,561 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}

nodetool tpstats shows some of the compactions are pending:

 

{{Pool Name                         Active   Pending      Completed   Blocked  
All time blocked}}{{ReadStage                              0         0        
1366419         0                 0}}{{MiscStage                              0 
        0              0         0                 0}}{{CompactionExecutor      
               9         9          77739         0                 
0}}{{MutationStage                          0         0        7504702         
0                 0}}{{MemtableReclaimMemory                  0         0       
     327         0                 0}}{{PendingRangeCalculator                 
0         0             20         0                 0}}{{GossipStage           
                 0         0         486365         0                 
0}}{{SecondaryIndexManagement               0         0              0         
0                 0}}

 

This is from the jstack output:

{{"CompactionExecutor:1" #26533 daemon prio=1 os_prio=4 
tid=0x7f971812f170 nid=0x6581 waiting for monitor entry 
[0x7f9990f4a000]}}{{   java.lang.Thread.State: BLOCKED (on object 
monitor)}}{{    at 
org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:310)}}{{
    - waiting to lock <0x0001c14acab0> (a 
org.apache.cassandra.db.compaction.LeveledManifest)}}{{    at 
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:119)}}{{
    at 
org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:119)}}{{
    at

[jira] [Updated] (CASSANDRA-14369) infinite loop when decommission a node

2018-04-06 Thread Daniel Woo (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Woo updated CASSANDRA-14369:
---
Description: 
I have 6 nodes (N1 to N6) and I am trying to decommission N1. Then I see two 
nodes are trying to receive streaming from N1 infinitely. The log rotates so 
quickly that I can only see this:

 

{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,561 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,561 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,561 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}

nodetool tpstats shows some of the compactions are pending:

 

{{Pool Name                         Active   Pending      Completed   Blocked  
All time blocked}}{{ReadStage                              0         0        
1366419         0                 0}}{{MiscStage                              0 
        0              0         0                 0}}{{CompactionExecutor      
               9         9          77739         0                 
0}}{{MutationStage                          0         0        7504702         
0                 0}}{{MemtableReclaimMemory                  0         0       
     327         0                 0}}{{PendingRangeCalculator                 
0         0             20         0                 0}}{{GossipStage           
                 0         0         486365         0                 
0}}{{SecondaryIndexManagement               0         0              0         
0                 0}}

 

This is from the jstack output:

{{"CompactionExecutor:1" #26533 daemon prio=1 os_prio=4 
tid=0x7f971812f170 nid=0x6581 waiting for monitor entry 
[0x7f9990f4a000]}}{{   java.lang.Thread.State: BLOCKED (on object 
monitor)}}{{    at 
org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:310)}}{{
    - waiting to lock <0x0001c14acab0> (a 
org.apache.cassandra.db.compaction.LeveledManifest)}}{{    at 
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:119)}}{{
    at 
org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:119)}}{{
    at 
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:262)}}{{
    at

[jira] [Commented] (CASSANDRA-12728) Handling partially written hint files

2018-04-06 Thread Vineet Ghatge (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429241#comment-16429241
 ] 

Vineet Ghatge commented on CASSANDRA-12728:
---

We are seeing this issue with 14 node cluster. We are currently on 3.11.1. Any 
ideas on what might be wrong?

ERROR [HintsDispatcher:1] 2018-04-06 16:26:44,423 CassandraDaemon.java:228 - 
Exception in thread Thread[HintsDispatcher:1,1,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Digest mismatch 
exception
 at 
org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:298)
 ~[apache-cassandra-3.11.1.jar:3.11.1-SNAPSHOT]
 at 
org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:263)
 ~[apache-cassandra-3.11.1.jar:3.11.1-SNAPSHOT]
 at 
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
~[apache-cassandra-3.11.1.jar:3.11.1-SNAPSHOT]
 at 
org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:169) 
~[apache-cassandra-3.11.1.jar:3.11.1-SNAPSHOT]
 at 
org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:128)
 ~[apache-cassandra-3.11.1.jar:3.11.1-SNAPSHOT]
 at 
org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:113) 
~[apache-cassandra-3.11.1.jar:3.11.1-SNAPSHOT]
 at 
org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:94) 
~[apache-cassandra-3.11.1.jar:3.11.1-SNAPSHOT]
 at 
org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:278)
 ~[apache-cassandra-3.11.1.jar:3.11.1-SNAPSHOT]
 at 
org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:260)
 ~[apache-cassandra-3.11.1.jar:3.11.1-SNAPSHOT]
 at 
org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:238)
 ~[apache-cassandra-3.11.1.jar:3.11.1-SNAPSHOT]
 at 
org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:217)
 ~[apache-cassandra-3.11.1.jar:3.11.1-SNAPSHOT]
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_141]
 at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_141]
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
~[na:1.8.0_141]
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[na:1.8.0_141]
 at 
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
 [apache-cassandra-3.11.1.jar:3.11.1-SNAPSHOT]
 at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_141]
Caused by: java.io.IOException: Digest mismatch exception

> Handling partially written hint files
> -
>
> Key: CASSANDRA-12728
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12728
> Project: Cassandra
>  Issue Type: Bug
>  Components: Hints
>Reporter: Sharvanath Pathak
>Assignee: Garvit Juniwal
>Priority: Major
>  Labels: lhf
> Fix For: 3.0.14, 3.11.0, 4.0
>
> Attachments: CASSANDRA-12728.patch
>
>
> {noformat}
> ERROR [HintsDispatcher:1] 2016-09-28 17:44:43,397 
> HintsDispatchExecutor.java:225 - Failed to dispatch hints file 
> d5d7257c-9f81-49b2-8633-6f9bda6e3dea-1474892654160-1.hints: file is corrupted 
> ({})
> org.apache.cassandra.io.FSReadError: java.io.EOFException
> at 
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:282)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:252)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119) 
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:91) 
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:259)
>  [apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:242)
>  [apache-cassandra-3.0.6.jar:3.0.6]
> at 
>

[jira] [Updated] (CASSANDRA-14369) infinite loop when decommission a node

2018-04-06 Thread Daniel Woo (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Woo updated CASSANDRA-14369:
---
Description: 
I have 6 nodes (N1 to N6) and I am trying to decommission N1. Then I see two 
nodes are trying to receive streaming from N1 infinitely. The log rotates so 
quickly that I can only see this:

 

{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,561 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,561 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,561 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}

 

This is from the jstack output:

{{"CompactionExecutor:1" #26533 daemon prio=1 os_prio=4 
tid=0x7f971812f170 nid=0x6581 waiting for monitor entry 
[0x7f9990f4a000]}}{{   java.lang.Thread.State: BLOCKED (on object 
monitor)}}{{    at 
org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:310)}}{{
    - waiting to lock <0x0001c14acab0> (a 
org.apache.cassandra.db.compaction.LeveledManifest)}}{{    at 
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:119)}}{{
    at 
org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:119)}}{{
    at 
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:262)}}{{
    at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)}}{{    
at java.util.concurrent.FutureTask.run(FutureTask.java:266)}}{{    at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}{{
    at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}{{
    at 
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)}}{{
    at 
org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5/2123444693.run(Unknown
 Source)}}{{    at java.lang.Thread.run(Thread.java:748)}}{{ }}{{   Locked 
ownable synchronizers:}}{{    - <0x0002b48f5ff0> (a 
java.util.concurrent.ThreadPoolExecutor$Worker)}}{{ 
}}{{"CompactionExecutor:16632" #26499 daemon prio=1 os_prio=4 
tid=0x7f970c16c420 nid=0x6553 runnable [0x7f9982714000]}}{{   
java.lang.Thread.State: RUNNABLE}}{{    at 
org.apache.cassandra.db.compaction.LeveledManifest.getLevelSize(LeveledManifest.java:489)}}{{
    -

[jira] [Created] (CASSANDRA-14369) infinite loop when decommission a node

2018-04-06 Thread Daniel Woo (JIRA)

Daniel Woo created CASSANDRA-14369:
--

 Summary: infinite loop when decommission a node
 Key: CASSANDRA-14369
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14369
 Project: Cassandra
  Issue Type: Bug
Reporter: Daniel Woo
 Fix For: 3.11.1


I have 6 nodes (N1 to N6) and I am trying to decommission N1. Then I see two 
nodes are trying to receive streaming from N1 infinitely. The log rotates so 
quickly that I can only see this:

 

{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,560 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,561 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,561 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}{{INFO  [CompactionExecutor:19401] 2018-04-07 13:07:56,561 
LeveledManifest.java:474 - Adding high-level (L3) 
BigTableReader(path='/opt/platform/data1/cassandra/data/data/contract_center_cloud/contract-2f2f9f70cd9911e7bfe87fec03576322/mc-31-big-Data.db')
 to candidates}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13047) Point cqlsh help to the new doc

2018-04-06 Thread Patrick Bannister (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429209#comment-16429209
 ] 

Patrick Bannister edited comment on CASSANDRA-13047 at 4/7/18 3:10 AM:
---

Originally I asked a complicated question about how to handle the new docs, but 
I've found some more information that probably makes all my questions 
irrelevant. I'll continue studying the issue.

Reading doc/README.md, I see we're building our docs at 
[http://cassandra.apache.org/doc/latest/cql/index.html] with the gen-doc target 
in build.xml.

 

-Does this ticket have a hidden subtask to add the online CQL docs at 
[http://cassandra.apache.org/doc/latest/cql/index.html] to the cassandra build?-

-Currently, cqlsh tries to point to help at the following locations:-
 # -$CASSANDRA_DIR/doc/cql3/CQL.html-
 # -/usr/share/doc/cassandra/CQL.html-
 # -A URL, depending on the branch:-
 -3.0 goes to- 
 -[https://cassandra.apache.org/doc/cql3/CQL-3.0.html,] which seems to be the 
same as $CASSANDRA_DIR/doc/cql3/CQL.html-
 -3.11 and trunk go to [https://cassandra.apache.org/doc/cql3/CQL-3.2.html] 
(which currently serves a nice 404 Error)-

-CQL.html at $CASSANDRA_DIR/doc/cql3/CQL.html appears to be built by the ant 
build.xml target generate-cql-html.-

-Nothing uses what seems to be our latest and greatest online CQL docs at 
[http://cassandra.apache.org/doc/latest/cql/index.html,] and it doesn't look 
like that content is part of the current build.-

-As it stands, I can do the following:-
 * -Change 3.11 and trunk to use 
[https://cassandra.apache.org/doc/cql3/CQL-3.0.html] as their fallback URL 
instead of their current broken links-
 * -Double dip by continuing to prefer the filesystem CQL.html files if they're 
present, but falling back to the new online help at 
[http://cassandra.apache.org/doc/latest/cql/index.html], which would also mean 
a rewrite of pylib/cqlshlib/helptopics.py since the structure of the newer 
online help is different-
 * -Add the online CQL docs to our build, remove the old CQL.html, and 
completely favor the new content. This probably merits some more input if this 
is the chosen path, since I think I've seen some discussion on the lists about 
how to host and maintain our docs.-

 


was (Author: ptbannister):
Does this ticket have a hidden subtask to add the online CQL docs at 
[http://cassandra.apache.org/doc/latest/cql/index.html] to the cassandra build?

Currently, cqlsh tries to point to help at the following locations:
 # $CASSANDRA_DIR/doc/cql3/CQL.html
 # /usr/share/doc/cassandra/CQL.html
 # A URL, depending on the branch:
3.0 goes to 
[https://cassandra.apache.org/doc/cql3/CQL-3.0.html,] which seems to be the 
same as $CASSANDRA_DIR/doc/cql3/CQL.html
3.11 and trunk go to [https://cassandra.apache.org/doc/cql3/CQL-3.2.html] 
(which currently serves a nice 404 Error)

CQL.html at $CASSANDRA_DIR/doc/cql3/CQL.html appears to be built by the ant 
build.xml target generate-cql-html.

Nothing uses what seems to be our latest and greatest online CQL docs at 
[http://cassandra.apache.org/doc/latest/cql/index.html,] and it doesn't look 
like that content is part of the current build.

As it stands, I can do the following:
 * Change 3.11 and trunk to use 
https://cassandra.apache.org/doc/cql3/CQL-3.0.html as their fallback URL 
instead of their current broken links
 * Double dip by continuing to prefer the filesystem CQL.html files if they're 
present, but falling back to the new online help at 
[http://cassandra.apache.org/doc/latest/cql/index.html], which would also mean 
a rewrite of pylib/cqlshlib/helptopics.py since the structure of the newer 
online help is different
 * Add the online CQL docs to our build, remove the old CQL.html, and 
completely favor the new content. This probably merits some more input if this 
is the chosen path, since I think I've seen some discussion on the lists about 
how to host and maintain our docs.

 

> Point cqlsh help to the new doc
> ---
>
> Key: CASSANDRA-13047
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13047
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Sylvain Lebresne
>Assignee: Patrick Bannister
>Priority: Major
> Fix For: 4.0
>
>
> Cqlsh still points to the "old" textile CQL doc, but that's not really 
> maintain anymore now that we have the new doc (which include everything the 
> old doc had and more). We should update cqlsh to point to the new doc so we 
> can remove the old one completely.
> Any takers?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail:

[jira] [Commented] (CASSANDRA-13047) Point cqlsh help to the new doc

2018-04-06 Thread Patrick Bannister (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429209#comment-16429209
 ] 

Patrick Bannister commented on CASSANDRA-13047:
---

Does this ticket have a hidden subtask to add the online CQL docs at 
[http://cassandra.apache.org/doc/latest/cql/index.html] to the cassandra build?

Currently, cqlsh tries to point to help at the following locations:
 # $CASSANDRA_DIR/doc/cql3/CQL.html
 # /usr/share/doc/cassandra/CQL.html
 # A URL, depending on the branch:
3.0 goes to 
[https://cassandra.apache.org/doc/cql3/CQL-3.0.html,] which seems to be the 
same as $CASSANDRA_DIR/doc/cql3/CQL.html
3.11 and trunk go to [https://cassandra.apache.org/doc/cql3/CQL-3.2.html] 
(which currently serves a nice 404 Error)

CQL.html at $CASSANDRA_DIR/doc/cql3/CQL.html appears to be built by the ant 
build.xml target generate-cql-html.

Nothing uses what seems to be our latest and greatest online CQL docs at 
[http://cassandra.apache.org/doc/latest/cql/index.html,] and it doesn't look 
like that content is part of the current build.

As it stands, I can do the following:
 * Change 3.11 and trunk to use 
https://cassandra.apache.org/doc/cql3/CQL-3.0.html as their fallback URL 
instead of their current broken links
 * Double dip by continuing to prefer the filesystem CQL.html files if they're 
present, but falling back to the new online help at 
[http://cassandra.apache.org/doc/latest/cql/index.html], which would also mean 
a rewrite of pylib/cqlshlib/helptopics.py since the structure of the newer 
online help is different
 * Add the online CQL docs to our build, remove the old CQL.html, and 
completely favor the new content. This probably merits some more input if this 
is the chosen path, since I think I've seen some discussion on the lists about 
how to host and maintain our docs.

 

> Point cqlsh help to the new doc
> ---
>
> Key: CASSANDRA-13047
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13047
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Sylvain Lebresne
>Assignee: Patrick Bannister
>Priority: Major
> Fix For: 4.0
>
>
> Cqlsh still points to the "old" textile CQL doc, but that's not really 
> maintain anymore now that we have the new doc (which include everything the 
> old doc had and more). We should update cqlsh to point to the new doc so we 
> can remove the old one completely.
> Any takers?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-14303) NetworkTopologyStrategy could have a "default replication" option

2018-04-06 Thread Joseph Lynch (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429187#comment-16429187
 ] 

Joseph Lynch edited comment on CASSANDRA-14303 at 4/7/18 1:24 AM:
--

||trunk||
|[patch|https://github.com/apache/cassandra/compare/trunk...jolynch:CASSANDRA-14303#diff-25adf5676f157a803428726a4e55a8e8]|
|[docs|https://github.com/apache/cassandra/compare/trunk...jolynch:CASSANDRA-14303#diff-911d9ab4d38f8f4be9752d968d414a26]|
|[unit 
tests|https://github.com/apache/cassandra/compare/trunk...jolynch:CASSANDRA-14303#diff-874be227a5ab21298d6a11c7d1f6ce2eR240]
 
[!https://circleci.com/gh/jolynch/cassandra/tree/CASSANDRA-14303.png?circle-token=
 
1102a59698d04899ec971dd36e925928f7b521f5!|https://circleci.com/gh/jolynch/cassandra/tree/CASSANDRA-14303]|

I ended up putting the expansion in {{ReplicationParams}} because after that 
the params appear entirely immutable to the strategies themselves. I think this 
is cleaner than having the map be mutable and allowing the replication 
strategies to manipulate the replication options, but I might be able to 
refactor it so that ARS can call like a static method or something on the 
passed class?

Currently the parameter is called {{default_datacenter_replication}} instead of 
re-using {{replication_factor}} because of where I ended up putting the 
auto-expansion to get around the immutability, but that might not be the best.


was (Author: jolynch):
||trunk||
|[patch|https://github.com/apache/cassandra/compare/trunk...jolynch:CASSANDRA-14303#diff-25adf5676f157a803428726a4e55a8e8]|
|[docs|https://github.com/apache/cassandra/compare/trunk...jolynch:CASSANDRA-14303#diff-911d9ab4d38f8f4be9752d968d414a26]|
|[unit 
tests|https://github.com/apache/cassandra/compare/trunk...jolynch:CASSANDRA-14303#diff-874be227a5ab21298d6a11c7d1f6ce2eR240]
 
[!https://circleci.com/gh/jolynch/cassandra/tree/CASSANDRA-14303.png?circle-token=
 
1102a59698d04899ec971dd36e925928f7b521f5!|https://circleci.com/gh/jolynch/cassandra/tree/CASSANDRA-14303]|

I ended up putting the expansion in {{ReplicationParams}} because after that 
the params appear entirely immutable to the strategies themselves. I think this 
is cleaner than having the map be mutable and allowing the replication 
strategies to manipulate the replication options, but I might be able to 
refactor it so that ARS can call like a static method or something on the 
passed class?

> NetworkTopologyStrategy could have a "default replication" option
> -
>
> Key: CASSANDRA-14303
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14303
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Joseph Lynch
>Assignee: Joseph Lynch
>Priority: Minor
> Fix For: 4.0
>
>
> Right now when creating a keyspace with {{NetworkTopologyStrategy}} the user 
> has to manually specify the datacenters they want their data replicated to 
> with parameters, e.g.:
> {noformat}
>  CREATE KEYSPACE test WITH replication = {'class': 'NetworkTopologyStrategy', 
> 'dc1': 3, 'dc2': 3}{noformat}
> This is a poor user interface because it requires the creator of the keyspace 
> (typically a developer) to know the layout of the Cassandra cluster (which 
> may or may not be controlled by them). Also, at least in my experience, folks 
> typo the datacenters _all_ the time. To work around this I see a number of 
> users creating automation around this where the automation describes the 
> Cassandra cluster and automatically expands out to all the dcs that Cassandra 
> knows about. Why can't Cassandra just do this for us, re-using the previously 
> forbidden {{replication_factor}} option (for backwards compatibility):
> {noformat}
>  CREATE KEYSPACE test WITH replication = {'class': 'NetworkTopologyStrategy', 
> 'replication_factor': 3}{noformat}
> This would automatically replicate this Keyspace to all datacenters that are 
> present in the cluster. If you need to _override_ the default you could 
> supply a datacenter name, e.g.:
> {noformat}
> > CREATE KEYSPACE test WITH replication = {'class': 
> > 'NetworkTopologyStrategy', 'replication_factor': 3, 'dc1': 2}
> > DESCRIBE KEYSPACE test
> CREATE KEYSPACE test WITH replication = {'class': 'NetworkTopologyStrategy', 
> 'dc1': '2', 'dc2': 3} AND durable_writes = true;
> {noformat}
> On the implementation side I think this may be reasonably straightforward to 
> do an auto-expansion at the time of keyspace creation (or alter), where the 
> above would automatically expand to list out the datacenters. We could allow 
> this to be recomputed whenever an AlterKeyspaceStatement runs so that to add 
> datacenters you would just run:
> {noformat}
> ALTER KEYSPACE test WITH replication = {'class':

[jira] [Updated] (CASSANDRA-14303) NetworkTopologyStrategy could have a "default replication" option

2018-04-06 Thread Joseph Lynch (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Lynch updated CASSANDRA-14303:
-
Fix Version/s: 4.0
   Status: Patch Available  (was: Open)

||trunk||
|[patch|https://github.com/apache/cassandra/compare/trunk...jolynch:CASSANDRA-14303#diff-25adf5676f157a803428726a4e55a8e8]|
|[docs|https://github.com/apache/cassandra/compare/trunk...jolynch:CASSANDRA-14303#diff-911d9ab4d38f8f4be9752d968d414a26]|
|[unit 
tests|https://github.com/apache/cassandra/compare/trunk...jolynch:CASSANDRA-14303#diff-874be227a5ab21298d6a11c7d1f6ce2eR240]
 
[!https://circleci.com/gh/jolynch/cassandra/tree/CASSANDRA-14303.png?circle-token=
 
1102a59698d04899ec971dd36e925928f7b521f5!|https://circleci.com/gh/jolynch/cassandra/tree/CASSANDRA-14303]|

I ended up putting the expansion in {{ReplicationParams}} because after that 
the params appear entirely immutable to the strategies themselves. I think this 
is cleaner than having the map be mutable and allowing the replication 
strategies to manipulate the replication options, but I might be able to 
refactor it so that ARS can call like a static method or something on the 
passed class?

> NetworkTopologyStrategy could have a "default replication" option
> -
>
> Key: CASSANDRA-14303
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14303
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Joseph Lynch
>Assignee: Joseph Lynch
>Priority: Minor
> Fix For: 4.0
>
>
> Right now when creating a keyspace with {{NetworkTopologyStrategy}} the user 
> has to manually specify the datacenters they want their data replicated to 
> with parameters, e.g.:
> {noformat}
>  CREATE KEYSPACE test WITH replication = {'class': 'NetworkTopologyStrategy', 
> 'dc1': 3, 'dc2': 3}{noformat}
> This is a poor user interface because it requires the creator of the keyspace 
> (typically a developer) to know the layout of the Cassandra cluster (which 
> may or may not be controlled by them). Also, at least in my experience, folks 
> typo the datacenters _all_ the time. To work around this I see a number of 
> users creating automation around this where the automation describes the 
> Cassandra cluster and automatically expands out to all the dcs that Cassandra 
> knows about. Why can't Cassandra just do this for us, re-using the previously 
> forbidden {{replication_factor}} option (for backwards compatibility):
> {noformat}
>  CREATE KEYSPACE test WITH replication = {'class': 'NetworkTopologyStrategy', 
> 'replication_factor': 3}{noformat}
> This would automatically replicate this Keyspace to all datacenters that are 
> present in the cluster. If you need to _override_ the default you could 
> supply a datacenter name, e.g.:
> {noformat}
> > CREATE KEYSPACE test WITH replication = {'class': 
> > 'NetworkTopologyStrategy', 'replication_factor': 3, 'dc1': 2}
> > DESCRIBE KEYSPACE test
> CREATE KEYSPACE test WITH replication = {'class': 'NetworkTopologyStrategy', 
> 'dc1': '2', 'dc2': 3} AND durable_writes = true;
> {noformat}
> On the implementation side I think this may be reasonably straightforward to 
> do an auto-expansion at the time of keyspace creation (or alter), where the 
> above would automatically expand to list out the datacenters. We could allow 
> this to be recomputed whenever an AlterKeyspaceStatement runs so that to add 
> datacenters you would just run:
> {noformat}
> ALTER KEYSPACE test WITH replication = {'class': 'NetworkTopologyStrategy', 
> 'replication_factor': 3}{noformat}
> and this would check that if the dc's in the current schema are different you 
> add in the new ones (_for safety reasons we'd never remove non explicitly 
> supplied zero dcs when auto-generating dcs_). Removing a datacenter becomes 
> an alter that includes an override for the dc you want to remove (or of 
> course you can always not use the auto-expansion and just use the old way):
> {noformat}
> // Tell it explicitly not to replicate to dc2
> > ALTER KEYSPACE test WITH replication = {'class': 'NetworkTopologyStrategy', 
> > 'replication_factor': 3, 'dc2': 0}
> > DESCRIBE KEYSPACE test
> CREATE KEYSPACE test WITH replication = {'class': 'NetworkTopologyStrategy', 
> 'dc1': '3'} AND durable_writes = true;{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14349) Untracked CDC segment files are not deleted after replay

2018-04-06 Thread Shichao An (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429172#comment-16429172
 ] 

Shichao An commented on CASSANDRA-14349:


I added a dtest, here's the 
[patch|https://github.com/shichao-an/cassandra-dtest/commits/master] on GitHub.

> Untracked CDC segment files are not deleted after replay
> 
>
> Key: CASSANDRA-14349
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14349
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths
>Reporter: Shichao An
>Assignee: Shichao An
>Priority: Minor
>
> When CDC is enabled, a hard link to each commit log file will be created in 
> cdc_raw directory. Those commit logs with CDC mutations will also have cdc 
> index files created along with the hard links; these are intended for the 
> consumer to handle and clean them up.
> However, if we don't produce any CDC traffic, those hard links in cdc_raw 
> will be never cleaned up (because hard links will still be created, without 
> the index files), whereas the real original commit logs are correctly deleted 
> after replay during process startup. This will results in many untracked hard 
> links in cdc_raw if we restart the cassandra process many times. I am able to 
> use CCM to reproduce it in trunk version which has the CASSANDRA-12148 
> changes.
> This seems a bug in handleReplayedSegment of the commit log segment manager 
> which neglects to take care of CDC commit logs. I will attach a patch here.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-14297) Optional startup delay for peers should wait for count rather than percentage

2018-04-06 Thread Joseph Lynch (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16414849#comment-16414849
 ] 

Joseph Lynch edited comment on CASSANDRA-14297 at 4/6/18 11:42 PM:
---

First implementation up on github along with a lot of unit tests. I'll start 
doing some more e2e testing using ccm just to make sure all the edge cases are 
covered but if someone ([~aweisberg] or [~jasobrown] perhaps) wants to review 
that would be excellent.
||trunk||
|[pull request|https://github.com/apache/cassandra/pull/212]|
|[!https://circleci.com/gh/jolynch/cassandra/tree/CASSANDRA-14297.png?circle-token=
 
1102a59698d04899ec971dd36e925928f7b521f5!|https://circleci.com/gh/jolynch/cassandra/tree/CASSANDRA-14297]
 |

 


was (Author: jolynch):
First implementation up on github along with a lot of unit tests. I'll start 
doing some more e2e testing using ccm just to make sure all the edge cases are 
covered but if someone ([~aweisberg] or [~jasobrown] perhaps) wants to review 
that would be excellent.
||trunk||
|[pull request|https://github.com/apache/cassandra/pull/212]|
|!https://circleci.com/gh/jolynch/cassandra/tree/CASSANDRA-14297.png?circle-token=
 1102a59698d04899ec971dd36e925928f7b521f5! |

 

> Optional startup delay for peers should wait for count rather than percentage
> -
>
> Key: CASSANDRA-14297
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14297
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Lifecycle
>Reporter: Joseph Lynch
>Assignee: Joseph Lynch
>Priority: Minor
>  Labels: PatchAvailable
>
> As I commented in CASSANDRA-13993, the current wait for functionality is a 
> great step in the right direction, but I don't think that the current setting 
> (70% of nodes in the cluster) is the right configuration option. First I 
> think this because 70% will not protect against errors as if you wait for 70% 
> of the cluster you could still very easily have {{UnavailableException}} or 
> {{ReadTimeoutException}} exceptions. This is because if you have even two 
> nodes down in different racks in a Cassandra cluster these exceptions are 
> possible (or with the default {{num_tokens}} setting of 256 it is basically 
> guaranteed). Second I think this option is not easy for operators to set, the 
> only setting I could think of that would "just work" is 100%.
> I proposed in that ticket instead of having `block_for_peers_percentage` 
> defaulting to 70%, we instead have `block_for_peers` as a count of nodes that 
> are allowed to be down before the starting node makes itself available as a 
> coordinator. Of course, we would still have the timeout to limit startup time 
> and deal with really extreme situations (whole datacenters down etc).
> I started working on a patch for this change [on 
> github|https://github.com/jasobrown/cassandra/compare/13993...jolynch:13993], 
> and am happy to finish it up with unit tests and such if someone can 
> review/commit it (maybe [~aweisberg]?).
> I think the short version of my proposal is we replace:
> {noformat}
> block_for_peers_percentage: 
> {noformat}
> with either
> {noformat}
> block_for_peers: 
> {noformat}
> or, if we want to do even better imo and enable advanced operators to finely 
> tune this behavior (while still having good defaults that work for almost 
> everyone):
> {noformat}
> block_for_peers_local_dc:  
> block_for_peers_each_dc: 
> block_for_peers_all_dcs: 
> {noformat}
> For example if an operator knows that they must be available at 
> {{LOCAL_QUORUM}} they would set {{block_for_peers_local_dc=1}}, if they use 
> {{EACH_QUOURM}} they would set {{block_for_peers_local_dc=1}}, if they use 
> {{QUORUM}} (RF=3, dcs=2) they would set {{block_for_peers_all_dcs=2}}. 
> Naturally everything would of course have a timeout to prevent startup taking 
> too long.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-14297) Optional startup delay for peers should wait for count rather than percentage

2018-04-06 Thread Joseph Lynch (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16414849#comment-16414849
 ] 

Joseph Lynch edited comment on CASSANDRA-14297 at 4/6/18 11:41 PM:
---

First implementation up on github along with a lot of unit tests. I'll start 
doing some more e2e testing using ccm just to make sure all the edge cases are 
covered but if someone ([~aweisberg] or [~jasobrown] perhaps) wants to review 
that would be excellent.
||trunk||
|[pull request|https://github.com/apache/cassandra/pull/212]|
|!https://circleci.com/gh/jolynch/cassandra/tree/CASSANDRA-14297.png?circle-token=
 1102a59698d04899ec971dd36e925928f7b521f5! |

 


was (Author: jolynch):
First implementation up on github along with a lot of unit tests. I'll start 
doing some more e2e testing using ccm just to make sure all the edge cases are 
covered but if someone ([~aweisberg] or [~jasobrown] perhaps) wants to review 
that would be excellent.
||trunk||
|[pull request|https://github.com/apache/cassandra/pull/212]|
|[utests |https://circleci.com/gh/jolynch/cassandra/tree/CASSANDRA-14297]|

 

> Optional startup delay for peers should wait for count rather than percentage
> -
>
> Key: CASSANDRA-14297
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14297
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Lifecycle
>Reporter: Joseph Lynch
>Assignee: Joseph Lynch
>Priority: Minor
>  Labels: PatchAvailable
>
> As I commented in CASSANDRA-13993, the current wait for functionality is a 
> great step in the right direction, but I don't think that the current setting 
> (70% of nodes in the cluster) is the right configuration option. First I 
> think this because 70% will not protect against errors as if you wait for 70% 
> of the cluster you could still very easily have {{UnavailableException}} or 
> {{ReadTimeoutException}} exceptions. This is because if you have even two 
> nodes down in different racks in a Cassandra cluster these exceptions are 
> possible (or with the default {{num_tokens}} setting of 256 it is basically 
> guaranteed). Second I think this option is not easy for operators to set, the 
> only setting I could think of that would "just work" is 100%.
> I proposed in that ticket instead of having `block_for_peers_percentage` 
> defaulting to 70%, we instead have `block_for_peers` as a count of nodes that 
> are allowed to be down before the starting node makes itself available as a 
> coordinator. Of course, we would still have the timeout to limit startup time 
> and deal with really extreme situations (whole datacenters down etc).
> I started working on a patch for this change [on 
> github|https://github.com/jasobrown/cassandra/compare/13993...jolynch:13993], 
> and am happy to finish it up with unit tests and such if someone can 
> review/commit it (maybe [~aweisberg]?).
> I think the short version of my proposal is we replace:
> {noformat}
> block_for_peers_percentage: 
> {noformat}
> with either
> {noformat}
> block_for_peers: 
> {noformat}
> or, if we want to do even better imo and enable advanced operators to finely 
> tune this behavior (while still having good defaults that work for almost 
> everyone):
> {noformat}
> block_for_peers_local_dc:  
> block_for_peers_each_dc: 
> block_for_peers_all_dcs: 
> {noformat}
> For example if an operator knows that they must be available at 
> {{LOCAL_QUORUM}} they would set {{block_for_peers_local_dc=1}}, if they use 
> {{EACH_QUOURM}} they would set {{block_for_peers_local_dc=1}}, if they use 
> {{QUORUM}} (RF=3, dcs=2) they would set {{block_for_peers_all_dcs=2}}. 
> Naturally everything would of course have a timeout to prevent startup taking 
> too long.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14353) Fix some regressions caused by 14058

2018-04-06 Thread Blake Eggleston (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Blake Eggleston updated CASSANDRA-14353:

Resolution: Fixed
Status: Resolved  (was: Ready to Commit)

bq. A part of me thinks that the extra two catch-rethrows in DataResolver are 
unnecessary

I don't disagree... but there are asserts in that path and I'm already wrapping 
the iterator. I'd feel kinda dumb if I ended having a debug some rare instance 
where they could have helped.

Committed as {{eda2613bcca7e4734b3bb4e04c6a4df39eb59ca2}}

> Fix some regressions caused by 14058
> 
>
> Key: CASSANDRA-14353
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14353
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Major
> Fix For: 4.0
>
>
> As [~iamaleksey] pointed out, CASSANDRA-14058 introduced a few regressions:
> {quote}
> Noticed a couple regressions when merging up CASSANDRA-14330:
> 1. {{DataResolver}} no longer uses 
> {{cassandra.drop_oversized_readrepair_mutations}} prop - and 
> {{DROP_OVERSIZED_READ_REPAIR_MUTATIONS}} constant is now unused, and the 
> feature is missing.
> 2. {{RowIteratorMergeListener}} re-thrown {{AssertionError}} no longer 
> includes the responses. This should be restored, as without it debugging RR 
> issues is an even worse, potentially impossible, nightmare.
> Nit: In {{DataResolver}}, {{repairResults}} field is now unused.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

cassandra git commit: Fix some regressions caused by 14058

2018-04-06 Thread bdeggleston

Repository: cassandra
Updated Branches:
  refs/heads/trunk c5a7fcaa8 -> eda2613bc


Fix some regressions caused by 14058

Patch by Blake Eggleston; Reviewed by Aleksey Yeschenko for CASSANDRA-14353


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/eda2613b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/eda2613b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/eda2613b

Branch: refs/heads/trunk
Commit: eda2613bcca7e4734b3bb4e04c6a4df39eb59ca2
Parents: c5a7fca
Author: Blake Eggleston 
Authored: Wed Mar 28 15:03:52 2018 -0700
Committer: Blake Eggleston 
Committed: Fri Apr 6 16:20:47 2018 -0700

--
 CHANGES.txt |   1 +
 .../cassandra/service/reads/DataResolver.java   | 114 +--
 .../reads/repair/BlockingReadRepair.java|  68 +--
 .../service/reads/repair/NoopReadRepair.java|   1 -
 .../repair/PartitionIteratorMergeListener.java  |   5 -
 .../service/reads/repair/ReadRepair.java|   1 +
 .../reads/repair/RowIteratorMergeListener.java  |  30 -
 .../reads/repair/TestableReadRepair.java|   1 -
 8 files changed, 165 insertions(+), 56 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/eda2613b/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 3db7f27..d6cf162 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 4.0
+ * Fix some regressions caused by 14058 (CASSANDRA-14353)
  * Abstract repair for pluggable storage (CASSANDRA-14116)
  * Add meaningful toString() impls (CASSANDRA-13653)
  * Add sstableloader option to accept target keyspace name (CASSANDRA-13884)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/eda2613b/src/java/org/apache/cassandra/service/reads/DataResolver.java
--
diff --git a/src/java/org/apache/cassandra/service/reads/DataResolver.java 
b/src/java/org/apache/cassandra/service/reads/DataResolver.java
index 11dd083..4c7a6c9 100644
--- a/src/java/org/apache/cassandra/service/reads/DataResolver.java
+++ b/src/java/org/apache/cassandra/service/reads/DataResolver.java
@@ -17,12 +17,17 @@
  */
 package org.apache.cassandra.service.reads;
 
-import java.net.InetAddress;
 import java.util.*;
 
-import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Joiner;
+import com.google.common.collect.Iterables;
 
+import org.apache.cassandra.db.rows.RangeTombstoneMarker;
+import org.apache.cassandra.db.rows.Row;
+import org.apache.cassandra.db.rows.UnfilteredRowIterator;
+import org.apache.cassandra.db.rows.UnfilteredRowIterators;
 import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.schema.TableMetadata;
 import org.apache.cassandra.service.reads.repair.ReadRepair;
 import org.apache.cassandra.db.*;
 import org.apache.cassandra.db.filter.*;
@@ -33,11 +38,6 @@ import org.apache.cassandra.tracing.TraceState;
 
 public class DataResolver extends ResponseResolver
 {
-private static final boolean DROP_OVERSIZED_READ_REPAIR_MUTATIONS =
-Boolean.getBoolean("cassandra.drop_oversized_readrepair_mutations");
-
-@VisibleForTesting
-final List repairResults = 
Collections.synchronizedList(new ArrayList<>());
 private final long queryStartNanoTime;
 private final boolean enforceStrictLiveness;
 
@@ -114,7 +114,7 @@ public class DataResolver extends ResponseResolver
 results.set(i, ShortReadProtection.extend(sources[i], 
results.get(i), command, mergedResultCounter, queryStartNanoTime, 
enforceStrictLiveness));
 }
 
-return UnfilteredPartitionIterators.merge(results, command.nowInSec(), 
readRepair.getMergeListener(sources));
+return UnfilteredPartitionIterators.merge(results, command.nowInSec(), 
wrapMergeListener(readRepair.getMergeListener(sources), sources));
 }
 
 public void evaluateAllResponses()
@@ -130,4 +130,102 @@ public class DataResolver extends ResponseResolver
 {
 evaluateAllResponses();
 }
+
+private String makeResponsesDebugString(DecoratedKey partitionKey)
+{
+return Joiner.on(",\n").join(Iterables.transform(getMessages(), m -> 
m.from + " => " + m.payload.toDebugString(command, partitionKey)));
+}
+
+private UnfilteredPartitionIterators.MergeListener 
wrapMergeListener(UnfilteredPartitionIterators.MergeListener partitionListener, 
InetAddressAndPort[] sources)
+{
+return new UnfilteredPartitionIterators.MergeListener()
+{
+public UnfilteredRowIterators.MergeListener 
getRowMergeListener(DecoratedKey partitionKey, List 
versions)
+{
+

[jira] [Comment Edited] (CASSANDRA-6719) redesign loadnewsstables

2018-04-06 Thread Jordan West (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420655#comment-16420655
 ] 

Jordan West edited comment on CASSANDRA-6719 at 4/6/18 11:22 PM:
-

[~krummas], patch looks great. comments below (mostly minor):
 * Since {{nodetool refresh}} is being deprecated, instead of modified, it 
would be nice to maintain as much of the same functionality as before. 
Additional options could be introduced to address the two areas I noticed 
changes:
 ** FSUtils.handleCorruptSSTable/handleFSError are no longer called
 ** Row cache invalidation was not previously performed — this is a good thing 
regardless, so maybe skip an option for this one. 

 * If using {{nodetool refresh}} with JBOD, the counting keys per boundary work 
is done just to throw it away.
 * Minor/naming nit: consider renaming {{CFS#loadSSTables}}’s dirPath -> 
srcPath and {{findBestDiskAndInvalidCache}}’s path -> srcPath
 * Minor/usability nit: I couldn’t find many cases where 
{{@Option(required=true)}} is used. WDYT about moving the path to a positional 
argument since its required and this command does not take a variable number of 
positional args?
 * Minor/usability nit: Instead of noVerify=true,noVerifyTokens=false being an 
invalid state, make noVerify=true imply noVerifyTokens=true. 
 * The JavaDoc for {{CFS.loadNewSSTables}} should be updated to point to the 
new {{StorageService.loadSSTables}}. 
 * The comment on CFS#L861 is useful but out of place. 
 * Minor/naming nit: The naming of the “allKeys” variable in 
{{ImportTest#testImportInvalidateCache}} is misleading. 
 * Minor nits in {{ImportTest#testBestDisk}}:
 ** Instead of hardcoding token values what about using e.g. 
{{t.compareTo(mock.getDiskBoundaries().positions.get(0).getToken()) <= 0}}?
 ** Are you intentionally leaving the Random seed hardcoded?


was (Author: jrwest):
[~krummas], patch looks great. comments below (mostly minor):
 * Since {{nodetool refresh}} is being deprecated, instead of modified, it 
would be nice to maintain as much of the same functionality as before. 
Additional options could be introduced to address the two areas I noticed 
changes:
 ** FSUtils.handleCorruptSSTable/handleFSError are no longer called
 ** Row cache invalidation was not previously performed — this is a good thing 
regardless, so maybe skip an option for this one. 

 * If using {{nodetool refresh}} with JDOB, the counting keys per boundary work 
is done just to throw it away.
 * Minor/naming nit: consider renaming {{CFS#loadSSTables}}’s dirPath -> 
srcPath and {{findBestDiskAndInvalidCache}}’s path -> srcPath
 * Minor/usability nit: I couldn’t find many cases where 
{{@Option(required=true)}} is used. WDYT about moving the path to a positional 
argument since its required and this command does not take a variable number of 
positional args?
 * Minor/usability nit: Instead of noVerify=true,noVerifyTokens=false being an 
invalid state, make noVerify=true imply noVerifyTokens=true. 
 * The JavaDoc for {{CFS.loadNewSSTables}} should be updated to point to the 
new {{StorageService.loadSSTables}}. 
 * The comment on CFS#L861 is useful but out of place. 
 * Minor/naming nit: The naming of the “allKeys” variable in 
{{ImportTest#testImportInvalidateCache}} is misleading. 
 * Minor nits in {{ImportTest#testBestDisk}}:
 ** Instead of hardcoding token values what about using e.g. 
{{t.compareTo(mock.getDiskBoundaries().positions.get(0).getToken()) <= 0}}?
 ** Are you intentionally leaving the Random seed hardcoded?

> redesign loadnewsstables
> 
>
> Key: CASSANDRA-6719
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6719
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Tools
>Reporter: Jonathan Ellis
>Assignee: Marcus Eriksson
>Priority: Minor
>  Labels: lhf
> Fix For: 4.x
>
> Attachments: 6719.patch
>
>
> CFSMBean.loadNewSSTables scans data directories for new sstables dropped 
> there by an external agent.  This is dangerous because of possible filename 
> conflicts with existing or newly generated sstables.
> Instead, we should support leaving the new sstables in a separate directory 
> (specified by a parameter, or configured as a new location in yaml) and take 
> care of renaming as necessary automagically.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14116) Refactor repair

2018-04-06 Thread Blake Eggleston (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Blake Eggleston updated CASSANDRA-14116:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

committed to trunk as {{c5a7fcaa8e00083d6f74967c40474aef07b0d21a}}

> Refactor repair
> ---
>
> Key: CASSANDRA-14116
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14116
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Repair
>Reporter: Dikang Gu
>Assignee: Blake Eggleston
>Priority: Major
> Fix For: 4.0
>
>
> As part of the pluggable storage engine effort, we'd like to modularize the 
> repair related code, make it to be independent from existing storage engine 
> implementation details.
> For now, refer to 
> https://docs.google.com/document/d/1suZlvhzgB6NIyBNpM9nxoHxz_Ri7qAm-UEO8v8AIFsc
>  for high level designs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[2/2] cassandra git commit: Abstract repair for pluggable storage

2018-04-06 Thread bdeggleston

Abstract repair for pluggable storage

Patch by Blake Eggleston; Reviewed by Jason Brown for CASSANDRA-14116


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c5a7fcaa
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c5a7fcaa
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c5a7fcaa

Branch: refs/heads/trunk
Commit: c5a7fcaa8e00083d6f74967c40474aef07b0d21a
Parents: 478c1a9
Author: Blake Eggleston 
Authored: Sat Mar 10 09:16:46 2018 -0800
Committer: Blake Eggleston 
Committed: Fri Apr 6 16:11:14 2018 -0700

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  |  10 +
 src/java/org/apache/cassandra/db/Keyspace.java  |  10 +
 .../db/compaction/CompactionManager.java| 257 +
 .../repair/CassandraKeyspaceRepairManager.java  |  48 +++
 .../db/repair/CassandraTableRepairManager.java  |  83 +
 .../db/repair/CassandraValidationIterator.java  | 321 
 .../db/repair/PendingAntiCompaction.java| 205 +++
 .../cassandra/repair/KeyspaceRepairManager.java |  42 +++
 .../repair/RepairMessageVerbHandler.java|  19 +-
 .../cassandra/repair/TableRepairManager.java|  64 
 .../cassandra/repair/ValidationManager.java | 163 +
 .../repair/ValidationPartitionIterator.java |  33 ++
 .../repair/consistent/ConsistentSession.java|   8 +-
 .../repair/consistent/LocalSessions.java|  42 +--
 .../consistent/PendingAntiCompaction.java   | 202 --
 .../cassandra/service/ActiveRepairService.java  |  48 +--
 ...tionManagerGetSSTablesForValidationTest.java | 179 -
 .../LeveledCompactionStrategyTest.java  |   4 +-
 ...tionManagerGetSSTablesForValidationTest.java | 182 +
 .../db/repair/PendingAntiCompactionTest.java| 366 +++
 .../apache/cassandra/repair/ValidatorTest.java  |   2 +-
 .../repair/consistent/LocalSessionTest.java |  46 ++-
 .../consistent/PendingAntiCompactionTest.java   | 365 --
 .../service/ActiveRepairServiceTest.java|  12 +-
 25 files changed, 1603 insertions(+), 1109 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/c5a7fcaa/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index b3617c3..3db7f27 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 4.0
+ * Abstract repair for pluggable storage (CASSANDRA-14116)
  * Add meaningful toString() impls (CASSANDRA-13653)
  * Add sstableloader option to accept target keyspace name (CASSANDRA-13884)
  * Move processing of EchoMessage response to gossip stage (CASSANDRA-13713)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c5a7fcaa/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index 4f74667..e4b84fe 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -51,6 +51,7 @@ import org.apache.cassandra.db.compaction.*;
 import org.apache.cassandra.db.filter.ClusteringIndexFilter;
 import org.apache.cassandra.db.filter.DataLimits;
 import org.apache.cassandra.db.streaming.CassandraStreamManager;
+import org.apache.cassandra.db.repair.CassandraTableRepairManager;
 import org.apache.cassandra.db.view.TableViews;
 import org.apache.cassandra.db.lifecycle.*;
 import org.apache.cassandra.db.partitions.CachedPartition;
@@ -75,6 +76,7 @@ import 
org.apache.cassandra.io.sstable.metadata.MetadataCollector;
 import org.apache.cassandra.io.util.FileUtils;
 import org.apache.cassandra.metrics.TableMetrics;
 import org.apache.cassandra.metrics.TableMetrics.Sampler;
+import org.apache.cassandra.repair.TableRepairManager;
 import org.apache.cassandra.schema.*;
 import org.apache.cassandra.schema.CompactionParams.TombstoneOption;
 import org.apache.cassandra.service.CacheService;
@@ -216,6 +218,8 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 
 private final CassandraStreamManager streamManager;
 
+private final TableRepairManager repairManager;
+
 private volatile boolean compactionSpaceCheck = true;
 
 @VisibleForTesting
@@ -447,6 +451,7 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 oldMBeanName= null;
 }
 streamManager = new CassandraStreamManager(this);
+repairManager = new CassandraTableRepairManager(this);
 }
 
 public void updateSpeculationThreshold()
@@ -466,6 +471,11 @@ public class ColumnFamilyStore

[1/2] cassandra git commit: Abstract repair for pluggable storage

2018-04-06 Thread bdeggleston

Repository: cassandra
Updated Branches:
  refs/heads/trunk 478c1a9fd -> c5a7fcaa8


http://git-wip-us.apache.org/repos/asf/cassandra/blob/c5a7fcaa/src/java/org/apache/cassandra/service/ActiveRepairService.java
--
diff --git a/src/java/org/apache/cassandra/service/ActiveRepairService.java 
b/src/java/org/apache/cassandra/service/ActiveRepairService.java
index 950966f..b60088c 100644
--- a/src/java/org/apache/cassandra/service/ActiveRepairService.java
+++ b/src/java/org/apache/cassandra/service/ActiveRepairService.java
@@ -27,7 +27,7 @@ import java.util.concurrent.atomic.AtomicBoolean;
 import javax.management.MBeanServer;
 import javax.management.ObjectName;
 
-import com.google.common.base.Predicate;
+import com.google.common.base.Preconditions;
 import com.google.common.cache.Cache;
 import com.google.common.cache.CacheBuilder;
 import com.google.common.collect.ImmutableSet;
@@ -47,7 +47,7 @@ import org.apache.cassandra.concurrent.ScheduledExecutors;
 import org.apache.cassandra.config.Config;
 import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.db.ColumnFamilyStore;
-import org.apache.cassandra.dht.Bounds;
+import org.apache.cassandra.db.Keyspace;
 import org.apache.cassandra.dht.Range;
 import org.apache.cassandra.dht.Token;
 import org.apache.cassandra.exceptions.RequestFailureReason;
@@ -59,7 +59,6 @@ import org.apache.cassandra.gms.IFailureDetector;
 import org.apache.cassandra.gms.IEndpointStateChangeSubscriber;
 import org.apache.cassandra.gms.IFailureDetectionEventListener;
 import org.apache.cassandra.gms.VersionedValue;
-import org.apache.cassandra.io.sstable.format.SSTableReader;
 import org.apache.cassandra.locator.InetAddressAndPort;
 import org.apache.cassandra.locator.TokenMetadata;
 import org.apache.cassandra.net.IAsyncCallbackWithFailure;
@@ -549,6 +548,7 @@ public class ActiveRepairService implements 
IEndpointStateChangeSubscriber, IFai
  */
 public static class ParentRepairSession
 {
+private final Keyspace keyspace;
 private final Map columnFamilyStores = new 
HashMap<>();
 private final Collection ranges;
 public final boolean isIncremental;
@@ -560,10 +560,16 @@ public class ActiveRepairService implements 
IEndpointStateChangeSubscriber, IFai
 public ParentRepairSession(InetAddressAndPort coordinator, 
List columnFamilyStores, Collection ranges, 
boolean isIncremental, long repairedAt, boolean isGlobal, PreviewKind 
previewKind)
 {
 this.coordinator = coordinator;
+Set keyspaces = new HashSet<>();
 for (ColumnFamilyStore cfs : columnFamilyStores)
 {
+keyspaces.add(cfs.keyspace);
 this.columnFamilyStores.put(cfs.metadata.id, cfs);
 }
+
+Preconditions.checkArgument(keyspaces.size() == 1, "repair 
sessions cannot operate on multiple keyspaces");
+this.keyspace = Iterables.getOnlyElement(keyspaces);
+
 this.ranges = ranges;
 this.repairedAt = repairedAt;
 this.isIncremental = isIncremental;
@@ -576,42 +582,14 @@ public class ActiveRepairService implements 
IEndpointStateChangeSubscriber, IFai
 return previewKind != PreviewKind.NONE;
 }
 
-public Predicate getPreviewPredicate()
-{
-switch (previewKind)
-{
-case ALL:
-return (s) -> true;
-case REPAIRED:
-return (s) -> s.isRepaired();
-case UNREPAIRED:
-return (s) -> !s.isRepaired();
-default:
-throw new RuntimeException("Can't get preview predicate 
for preview kind " + previewKind);
-}
-}
-
-public synchronized void maybeSnapshot(TableId tableId, UUID 
parentSessionId)
+public Collection getColumnFamilyStores()
 {
-String snapshotName = parentSessionId.toString();
-if (!columnFamilyStores.get(tableId).snapshotExists(snapshotName))
-{
-Set snapshottedSSTables = 
columnFamilyStores.get(tableId).snapshot(snapshotName, new 
Predicate()
-{
-public boolean apply(SSTableReader sstable)
-{
-return sstable != null &&
-   (!isIncremental || !sstable.isRepaired()) &&
-   !(sstable.metadata().isIndex()) && // exclude 
SSTables from 2i
-   new Bounds<>(sstable.first.getToken(), 
sstable.last.getToken()).intersects(ranges);
-}
-}, true, false);
-}
+return 
ImmutableSet.builder().addAll(columnFamilyStores.values()).build();
 }
 
-public Collection getColumnFamilyStores()
+

[jira] [Commented] (CASSANDRA-13853) nodetool describecluster should be more informative

2018-04-06 Thread Preetika Tyagi (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429074#comment-16429074
 ] 

Preetika Tyagi commented on CASSANDRA-13853:


[~rustyrazorblade] I have uploaded the dtest source code file that I have 
written for this bug. This is my first attempt at dtests, so please feel free 
to give suggestions/feedback. Thanks!

> nodetool describecluster should be more informative
> ---
>
> Key: CASSANDRA-13853
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13853
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability, Tools
>Reporter: Jon Haddad
>Assignee: Preetika Tyagi
>Priority: Minor
>  Labels: lhf
> Fix For: 4.x
>
> Attachments: cassandra-13853-v2.patch, 
> nodetool_describecluster_test.py
>
>
> Additional information we should be displaying:
> * Total node count
> * List of datacenters, RF, with number of nodes per dc, how many are down, 
> * Version(s)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13853) nodetool describecluster should be more informative

2018-04-06 Thread Preetika Tyagi (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Preetika Tyagi updated CASSANDRA-13853:
---
Attachment: nodetool_describecluster_test.py

> nodetool describecluster should be more informative
> ---
>
> Key: CASSANDRA-13853
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13853
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability, Tools
>Reporter: Jon Haddad
>Assignee: Preetika Tyagi
>Priority: Minor
>  Labels: lhf
> Fix For: 4.x
>
> Attachments: cassandra-13853-v2.patch, 
> nodetool_describecluster_test.py
>
>
> Additional information we should be displaying:
> * Total node count
> * List of datacenters, RF, with number of nodes per dc, how many are down, 
> * Version(s)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14367) prefer Collections.singletonList to Arrays.asList(one_element)

2018-04-06 Thread Dave Brosius (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429015#comment-16429015
 ] 

Dave Brosius commented on CASSANDRA-14367:
--

@jasobrown because the code is wrapped in a new ArrayList() which implies to me 
the list is intended to be modified

> prefer Collections.singletonList to Arrays.asList(one_element)
> --
>
> Key: CASSANDRA-14367
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14367
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Dave Brosius
>Assignee: Dave Brosius
>Priority: Trivial
> Fix For: 4.x
>
> Attachments: 14367.txt
>
>
> small improvement, but Arrays.asList first creates an array, then wraps it 
> with a collections instance, whereas Collections.singletonList just creates 
> one small (one field) bean instance.
> so a small cut down on garbage generated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-14367) prefer Collections.singletonList to Arrays.asList(one_element)

2018-04-06 Thread Aleksey Yeschenko (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428750#comment-16428750
 ] 

Aleksey Yeschenko edited comment on CASSANDRA-14367 at 4/6/18 6:36 PM:
---

And sometimes we mutate those arguments, which you cannot do with the immutable 
{{singletonList()}} for example. Not saying it’s the case with any of this 
substitutions - just please be very careful with these changes.


was (Author: iamaleksey):
And sometimes we mutate those arguments, which you cannot do with the immutable 
{{SingletonList}} for example. Not saying it’s the case with any of this 
substitutions - just please be very careful with these changes.

> prefer Collections.singletonList to Arrays.asList(one_element)
> --
>
> Key: CASSANDRA-14367
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14367
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Dave Brosius
>Assignee: Dave Brosius
>Priority: Trivial
> Fix For: 4.x
>
> Attachments: 14367.txt
>
>
> small improvement, but Arrays.asList first creates an array, then wraps it 
> with a collections instance, whereas Collections.singletonList just creates 
> one small (one field) bean instance.
> so a small cut down on garbage generated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14367) prefer Collections.singletonList to Arrays.asList(one_element)

2018-04-06 Thread Aleksey Yeschenko (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428750#comment-16428750
 ] 

Aleksey Yeschenko commented on CASSANDRA-14367:
---

And sometimes we mutate those arguments, which you cannot do with the immutable 
{{SingletonList}} for example. Not saying it’s the case with any of this 
substitutions - just please be very careful with these changes.

> prefer Collections.singletonList to Arrays.asList(one_element)
> --
>
> Key: CASSANDRA-14367
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14367
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Dave Brosius
>Assignee: Dave Brosius
>Priority: Trivial
> Fix For: 4.x
>
> Attachments: 14367.txt
>
>
> small improvement, but Arrays.asList first creates an array, then wraps it 
> with a collections instance, whereas Collections.singletonList just creates 
> one small (one field) bean instance.
> so a small cut down on garbage generated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14367) prefer Collections.singletonList to Arrays.asList(one_element)

2018-04-06 Thread Aleksey Yeschenko (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428689#comment-16428689
 ] 

Aleksey Yeschenko commented on CASSANDRA-14367:
---

For what it's worth, this is not always an improvement. In some 
performance-sensitive cases I would much rather have a monomorphic call than 
turning things bimorphic or even megamorphic to save on array allocation.

See https://github.com/google/guava/issues/1268 for a similar discussion in 
Guava-land.

> prefer Collections.singletonList to Arrays.asList(one_element)
> --
>
> Key: CASSANDRA-14367
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14367
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Dave Brosius
>Assignee: Dave Brosius
>Priority: Trivial
> Fix For: 4.x
>
> Attachments: 14367.txt
>
>
> small improvement, but Arrays.asList first creates an array, then wraps it 
> with a collections instance, whereas Collections.singletonList just creates 
> one small (one field) bean instance.
> so a small cut down on garbage generated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14367) prefer Collections.singletonList to Arrays.asList(one_element)

2018-04-06 Thread Jason Brown (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428664#comment-16428664
 ] 

Jason Brown commented on CASSANDRA-14367:
-

[~dbrosius] In {{TableHistograms}} you used {{Lists.newArrayList}} instead of 
{{Collections.singletonList}}. Any particular reason why?

> prefer Collections.singletonList to Arrays.asList(one_element)
> --
>
> Key: CASSANDRA-14367
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14367
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Dave Brosius
>Assignee: Dave Brosius
>Priority: Trivial
> Fix For: 4.x
>
> Attachments: 14367.txt
>
>
> small improvement, but Arrays.asList first creates an array, then wraps it 
> with a collections instance, whereas Collections.singletonList just creates 
> one small (one field) bean instance.
> so a small cut down on garbage generated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13910) Remove read_repair_chance/dclocal_read_repair_chance

2018-04-06 Thread Aleksey Yeschenko (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428663#comment-16428663
 ] 

Aleksey Yeschenko commented on CASSANDRA-13910:
---

4.0 branch [here|https://github.com/iamaleksey/cassandra/commits/13910-4.0], CI 
[here|https://circleci.com/workflow-run/ce37b67c-f990-4f99-ac5b-8e252bdc815f].

> Remove read_repair_chance/dclocal_read_repair_chance
> 
>
> Key: CASSANDRA-13910
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13910
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Sylvain Lebresne
>Assignee: Aleksey Yeschenko
>Priority: Minor
> Fix For: 4.0
>
>
> First, let me clarify so this is not misunderstood that I'm not *at all* 
> suggesting to remove the read-repair mechanism of detecting and repairing 
> inconsistencies between read responses: that mechanism is imo fine and 
> useful.  But the {{read_repair_chance}} and {{dclocal_read_repair_chance}} 
> have never been about _enabling_ that mechanism, they are about querying all 
> replicas (even when this is not required by the consistency level) for the 
> sole purpose of maybe read-repairing some of the replica that wouldn't have 
> been queried otherwise. Which btw, bring me to reason 1 for considering their 
> removal: their naming/behavior is super confusing. Over the years, I've seen 
> countless users (and not only newbies) misunderstanding what those options 
> do, and as a consequence misunderstand when read-repair itself was happening.
> But my 2nd reason for suggesting this is that I suspect 
> {{read_repair_chance}}/{{dclocal_read_repair_chance}} are, especially 
> nowadays, more harmful than anything else when enabled. When those option 
> kick in, what you trade-off is additional resources consumption (all nodes 
> have to execute the read) for a _fairly remote chance_ of having some 
> inconsistencies repaired on _some_ replica _a bit faster_ than they would 
> otherwise be. To justify that last part, let's recall that:
> # most inconsistencies are actually fixed by hints in practice; and in the 
> case where a node stay dead for a long time so that hints ends up timing-out, 
> you really should repair the node when it comes back (if not simply 
> re-bootstrapping it).  Read-repair probably don't fix _that_ much stuff in 
> the first place.
> # again, read-repair do happen without those options kicking in. If you do 
> reads at {{QUORUM}}, inconsistencies will eventually get read-repaired all 
> the same.  Just a tiny bit less quickly.
> # I suspect almost everyone use a low "chance" for those options at best 
> (because the extra resources consumption is real), so at the end of the day, 
> it's up to chance how much faster this fixes inconsistencies.
> Overall, I'm having a hard time imagining real cases where that trade-off 
> really make sense. Don't get me wrong, those options had their places a long 
> time ago when hints weren't working all that well, but I think they bring 
> more confusion than benefits now.
> And I think it's sane to reconsider stuffs every once in a while, and to 
> clean up anything that may not make all that much sense anymore, which I 
> think is the case here.
> Tl;dr, I feel the benefits brought by those options are very slim at best and 
> well overshadowed by the confusion they bring, and not worth maintaining the 
> code that supports them (which, to be fair, isn't huge, but getting rid of 
> {{ReadCallback.AsyncRepairRunner}} wouldn't hurt for instance).
> Lastly, if the consensus here ends up being that they can have their use in 
> weird case and that we fill supporting those cases is worth confusing 
> everyone else and maintaining that code, I would still suggest disabling them 
> totally by default.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13910) Remove read_repair_chance/dclocal_read_repair_chance

2018-04-06 Thread Aleksey Yeschenko (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-13910:
--
   Labels:   (was: CommunityFeedbackRequested)
 Reviewer: Blake Eggleston
Fix Version/s: (was: 3.11.x)
   Status: Patch Available  (was: Open)

> Remove read_repair_chance/dclocal_read_repair_chance
> 
>
> Key: CASSANDRA-13910
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13910
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Sylvain Lebresne
>Assignee: Aleksey Yeschenko
>Priority: Minor
> Fix For: 4.0
>
>
> First, let me clarify so this is not misunderstood that I'm not *at all* 
> suggesting to remove the read-repair mechanism of detecting and repairing 
> inconsistencies between read responses: that mechanism is imo fine and 
> useful.  But the {{read_repair_chance}} and {{dclocal_read_repair_chance}} 
> have never been about _enabling_ that mechanism, they are about querying all 
> replicas (even when this is not required by the consistency level) for the 
> sole purpose of maybe read-repairing some of the replica that wouldn't have 
> been queried otherwise. Which btw, bring me to reason 1 for considering their 
> removal: their naming/behavior is super confusing. Over the years, I've seen 
> countless users (and not only newbies) misunderstanding what those options 
> do, and as a consequence misunderstand when read-repair itself was happening.
> But my 2nd reason for suggesting this is that I suspect 
> {{read_repair_chance}}/{{dclocal_read_repair_chance}} are, especially 
> nowadays, more harmful than anything else when enabled. When those option 
> kick in, what you trade-off is additional resources consumption (all nodes 
> have to execute the read) for a _fairly remote chance_ of having some 
> inconsistencies repaired on _some_ replica _a bit faster_ than they would 
> otherwise be. To justify that last part, let's recall that:
> # most inconsistencies are actually fixed by hints in practice; and in the 
> case where a node stay dead for a long time so that hints ends up timing-out, 
> you really should repair the node when it comes back (if not simply 
> re-bootstrapping it).  Read-repair probably don't fix _that_ much stuff in 
> the first place.
> # again, read-repair do happen without those options kicking in. If you do 
> reads at {{QUORUM}}, inconsistencies will eventually get read-repaired all 
> the same.  Just a tiny bit less quickly.
> # I suspect almost everyone use a low "chance" for those options at best 
> (because the extra resources consumption is real), so at the end of the day, 
> it's up to chance how much faster this fixes inconsistencies.
> Overall, I'm having a hard time imagining real cases where that trade-off 
> really make sense. Don't get me wrong, those options had their places a long 
> time ago when hints weren't working all that well, but I think they bring 
> more confusion than benefits now.
> And I think it's sane to reconsider stuffs every once in a while, and to 
> clean up anything that may not make all that much sense anymore, which I 
> think is the case here.
> Tl;dr, I feel the benefits brought by those options are very slim at best and 
> well overshadowed by the confusion they bring, and not worth maintaining the 
> code that supports them (which, to be fair, isn't huge, but getting rid of 
> {{ReadCallback.AsyncRepairRunner}} wouldn't hurt for instance).
> Lastly, if the consensus here ends up being that they can have their use in 
> weird case and that we fill supporting those cases is worth confusing 
> everyone else and maintaining that code, I would still suggest disabling them 
> totally by default.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13910) Remove read_repair_chance/dclocal_read_repair_chance

2018-04-06 Thread Aleksey Yeschenko (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-13910:
--
Summary: Remove read_repair_chance/dclocal_read_repair_chance  (was: 
Consider deprecating (then removing) 
read_repair_chance/dclocal_read_repair_chance)

> Remove read_repair_chance/dclocal_read_repair_chance
> 
>
> Key: CASSANDRA-13910
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13910
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Sylvain Lebresne
>Assignee: Aleksey Yeschenko
>Priority: Minor
>  Labels: CommunityFeedbackRequested
> Fix For: 4.0, 3.11.x
>
>
> First, let me clarify so this is not misunderstood that I'm not *at all* 
> suggesting to remove the read-repair mechanism of detecting and repairing 
> inconsistencies between read responses: that mechanism is imo fine and 
> useful.  But the {{read_repair_chance}} and {{dclocal_read_repair_chance}} 
> have never been about _enabling_ that mechanism, they are about querying all 
> replicas (even when this is not required by the consistency level) for the 
> sole purpose of maybe read-repairing some of the replica that wouldn't have 
> been queried otherwise. Which btw, bring me to reason 1 for considering their 
> removal: their naming/behavior is super confusing. Over the years, I've seen 
> countless users (and not only newbies) misunderstanding what those options 
> do, and as a consequence misunderstand when read-repair itself was happening.
> But my 2nd reason for suggesting this is that I suspect 
> {{read_repair_chance}}/{{dclocal_read_repair_chance}} are, especially 
> nowadays, more harmful than anything else when enabled. When those option 
> kick in, what you trade-off is additional resources consumption (all nodes 
> have to execute the read) for a _fairly remote chance_ of having some 
> inconsistencies repaired on _some_ replica _a bit faster_ than they would 
> otherwise be. To justify that last part, let's recall that:
> # most inconsistencies are actually fixed by hints in practice; and in the 
> case where a node stay dead for a long time so that hints ends up timing-out, 
> you really should repair the node when it comes back (if not simply 
> re-bootstrapping it).  Read-repair probably don't fix _that_ much stuff in 
> the first place.
> # again, read-repair do happen without those options kicking in. If you do 
> reads at {{QUORUM}}, inconsistencies will eventually get read-repaired all 
> the same.  Just a tiny bit less quickly.
> # I suspect almost everyone use a low "chance" for those options at best 
> (because the extra resources consumption is real), so at the end of the day, 
> it's up to chance how much faster this fixes inconsistencies.
> Overall, I'm having a hard time imagining real cases where that trade-off 
> really make sense. Don't get me wrong, those options had their places a long 
> time ago when hints weren't working all that well, but I think they bring 
> more confusion than benefits now.
> And I think it's sane to reconsider stuffs every once in a while, and to 
> clean up anything that may not make all that much sense anymore, which I 
> think is the case here.
> Tl;dr, I feel the benefits brought by those options are very slim at best and 
> well overshadowed by the confusion they bring, and not worth maintaining the 
> code that supports them (which, to be fair, isn't huge, but getting rid of 
> {{ReadCallback.AsyncRepairRunner}} wouldn't hurt for instance).
> Lastly, if the consensus here ends up being that they can have their use in 
> weird case and that we fill supporting those cases is worth confusing 
> everyone else and maintaining that code, I would still suggest disabling them 
> totally by default.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14367) prefer Collections.singletonList to Arrays.asList(one_element)

2018-04-06 Thread Jeff Jirsa (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-14367:
---
Status: Patch Available  (was: Open)

Marking patch available.

> prefer Collections.singletonList to Arrays.asList(one_element)
> --
>
> Key: CASSANDRA-14367
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14367
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Dave Brosius
>Assignee: Dave Brosius
>Priority: Trivial
> Fix For: 4.x
>
> Attachments: 14367.txt
>
>
> small improvement, but Arrays.asList first creates an array, then wraps it 
> with a collections instance, whereas Collections.singletonList just creates 
> one small (one field) bean instance.
> so a small cut down on garbage generated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13993) Add optional startup delay to wait until peers are ready

2018-04-06 Thread Jason Brown (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428638#comment-16428638
 ] 

Jason Brown commented on CASSANDRA-13993:
-

[~iamaleksey] Yup, that''s basically what i meant, as well :)

> Add optional startup delay to wait until peers are ready
> 
>
> Key: CASSANDRA-13993
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13993
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Lifecycle
>Reporter: Jason Brown
>Assignee: Jason Brown
>Priority: Minor
> Fix For: 4.0
>
>
> When bouncing a node in a large cluster, is can take a while to recognize the 
> rest of the cluster as available. This is especially true if using TLS on 
> internode messaging connections. The bouncing node (and any clients connected 
> to it) may see a series of Unavailable or Timeout exceptions until the node 
> is 'warmed up' as connecting to the rest of the cluster is asynchronous from 
> the rest of the startup process.
> There are two aspects that drive a node's ability to successfully communicate 
> with a peer after a bounce:
> - marking the peer as 'alive' (state that is held in gossip). This affects 
> the unavailable exceptions
> - having both open outbound and inbound connections open and ready to each 
> peer. This affects timeouts.
> Details of each of these mechanisms are described in the comments below.
> This ticket proposes adding a mechanism, optional and configurable, to delay 
> opening the client native protocol port until some percentage of the peers in 
> the cluster is marked alive and connected to/from. Thus while we potentially 
> slow down startup (delay opening the client port), we alleviate the chance 
> that queries made by clients don't hit transient unavailable/timeout 
> exceptions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13304) Add checksumming to the native protocol

2018-04-06 Thread Jeff Jirsa (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428628#comment-16428628
 ] 

Jeff Jirsa commented on CASSANDRA-13304:


Marking this as a hard blocker for 4.0, because it's not reasonable not to have 
a way to do this.

 

> Add checksumming to the native protocol
> ---
>
> Key: CASSANDRA-13304
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13304
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Michael Kjellman
>Assignee: Michael Kjellman
>Priority: Blocker
>  Labels: client-impacting
> Fix For: 4.x
>
> Attachments: 13304_v1.diff, boxplot-read-throughput.png, 
> boxplot-write-throughput.png
>
>
> The native binary transport implementation doesn't include checksums. This 
> makes it highly susceptible to silently inserting corrupted data either due 
> to hardware issues causing bit flips on the sender/client side, C*/receiver 
> side, or network in between.
> Attaching an implementation that makes checksum'ing mandatory (assuming both 
> client and server know about a protocol version that supports checksums) -- 
> and also adds checksumming to clients that request compression.
> The serialized format looks something like this:
> {noformat}
>  *  1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
>  *  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |  Number of Compressed Chunks  | Compressed Length (e1)/
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * /  Compressed Length cont. (e1) |Uncompressed Length (e1)   /
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * | Uncompressed Length cont. (e1)| CRC32 Checksum of Lengths (e1)|
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * | Checksum of Lengths cont. (e1)|Compressed Bytes (e1)+//
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |  CRC32 Checksum (e1) ||
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |Compressed Length (e2) |
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |   Uncompressed Length (e2)|
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |CRC32 Checksum of Lengths (e2) |
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * | Compressed Bytes (e2)   +//
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |  CRC32 Checksum (e2) ||
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |Compressed Length (en) |
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |   Uncompressed Length (en)|
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |CRC32 Checksum of Lengths (en) |
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |  Compressed Bytes (en)  +//
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |  CRC32 Checksum (en) ||
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
> {noformat}
> The first pass here adds checksums only to the actual contents of the frame 
> body itself (and doesn't actually checksum lengths and headers). While it 
> would be great to fully add checksuming across the entire protocol, the 
> proposed implementation will ensure we at least catch corrupted data and 
> likely protect ourselves pretty well anyways.
> I didn't go to the trouble of implementing a Snappy Checksum'ed Compressor 
> implementation as it's been deprecated for a while -- is really slow and 
> crappy compared to LZ4 -- and we should do everything in our power to make 
> sure no one in the community is still using it. I left it in (for obvious 
> backwards compatibility aspects) old for clients that don't know about the 
> new protocol.
> The current protocol has a 256MB (max) frame body -- where the serialized 
> contents are simply written in to the frame body.
> If the client sends a compression option in the startup, we will install a 
> FrameCompressor inline. Unfortunately, we went with a decision to treat the 
> frame body separately from the header bits etc in a given message. So, 
> instead we put a compressor

[jira] [Updated] (CASSANDRA-13304) Add checksumming to the native protocol

2018-04-06 Thread Jeff Jirsa (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13304:
---
Fix Version/s: 4.x

> Add checksumming to the native protocol
> ---
>
> Key: CASSANDRA-13304
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13304
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Michael Kjellman
>Assignee: Michael Kjellman
>Priority: Major
>  Labels: client-impacting
> Fix For: 4.x
>
> Attachments: 13304_v1.diff, boxplot-read-throughput.png, 
> boxplot-write-throughput.png
>
>
> The native binary transport implementation doesn't include checksums. This 
> makes it highly susceptible to silently inserting corrupted data either due 
> to hardware issues causing bit flips on the sender/client side, C*/receiver 
> side, or network in between.
> Attaching an implementation that makes checksum'ing mandatory (assuming both 
> client and server know about a protocol version that supports checksums) -- 
> and also adds checksumming to clients that request compression.
> The serialized format looks something like this:
> {noformat}
>  *  1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
>  *  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |  Number of Compressed Chunks  | Compressed Length (e1)/
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * /  Compressed Length cont. (e1) |Uncompressed Length (e1)   /
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * | Uncompressed Length cont. (e1)| CRC32 Checksum of Lengths (e1)|
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * | Checksum of Lengths cont. (e1)|Compressed Bytes (e1)+//
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |  CRC32 Checksum (e1) ||
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |Compressed Length (e2) |
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |   Uncompressed Length (e2)|
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |CRC32 Checksum of Lengths (e2) |
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * | Compressed Bytes (e2)   +//
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |  CRC32 Checksum (e2) ||
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |Compressed Length (en) |
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |   Uncompressed Length (en)|
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |CRC32 Checksum of Lengths (en) |
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |  Compressed Bytes (en)  +//
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |  CRC32 Checksum (en) ||
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
> {noformat}
> The first pass here adds checksums only to the actual contents of the frame 
> body itself (and doesn't actually checksum lengths and headers). While it 
> would be great to fully add checksuming across the entire protocol, the 
> proposed implementation will ensure we at least catch corrupted data and 
> likely protect ourselves pretty well anyways.
> I didn't go to the trouble of implementing a Snappy Checksum'ed Compressor 
> implementation as it's been deprecated for a while -- is really slow and 
> crappy compared to LZ4 -- and we should do everything in our power to make 
> sure no one in the community is still using it. I left it in (for obvious 
> backwards compatibility aspects) old for clients that don't know about the 
> new protocol.
> The current protocol has a 256MB (max) frame body -- where the serialized 
> contents are simply written in to the frame body.
> If the client sends a compression option in the startup, we will install a 
> FrameCompressor inline. Unfortunately, we went with a decision to treat the 
> frame body separately from the header bits etc in a given message. So, 
> instead we put a compressor implementation in the options and then if it's 
> not null, we push the serialized bytes for the frame body *only* thru the 
>

[jira] [Updated] (CASSANDRA-13304) Add checksumming to the native protocol

2018-04-06 Thread Jeff Jirsa (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13304:
---
Priority: Blocker  (was: Major)

> Add checksumming to the native protocol
> ---
>
> Key: CASSANDRA-13304
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13304
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Michael Kjellman
>Assignee: Michael Kjellman
>Priority: Blocker
>  Labels: client-impacting
> Fix For: 4.x
>
> Attachments: 13304_v1.diff, boxplot-read-throughput.png, 
> boxplot-write-throughput.png
>
>
> The native binary transport implementation doesn't include checksums. This 
> makes it highly susceptible to silently inserting corrupted data either due 
> to hardware issues causing bit flips on the sender/client side, C*/receiver 
> side, or network in between.
> Attaching an implementation that makes checksum'ing mandatory (assuming both 
> client and server know about a protocol version that supports checksums) -- 
> and also adds checksumming to clients that request compression.
> The serialized format looks something like this:
> {noformat}
>  *  1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
>  *  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |  Number of Compressed Chunks  | Compressed Length (e1)/
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * /  Compressed Length cont. (e1) |Uncompressed Length (e1)   /
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * | Uncompressed Length cont. (e1)| CRC32 Checksum of Lengths (e1)|
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * | Checksum of Lengths cont. (e1)|Compressed Bytes (e1)+//
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |  CRC32 Checksum (e1) ||
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |Compressed Length (e2) |
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |   Uncompressed Length (e2)|
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |CRC32 Checksum of Lengths (e2) |
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * | Compressed Bytes (e2)   +//
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |  CRC32 Checksum (e2) ||
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |Compressed Length (en) |
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |   Uncompressed Length (en)|
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |CRC32 Checksum of Lengths (en) |
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |  Compressed Bytes (en)  +//
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  * |  CRC32 Checksum (en) ||
>  * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
> {noformat}
> The first pass here adds checksums only to the actual contents of the frame 
> body itself (and doesn't actually checksum lengths and headers). While it 
> would be great to fully add checksuming across the entire protocol, the 
> proposed implementation will ensure we at least catch corrupted data and 
> likely protect ourselves pretty well anyways.
> I didn't go to the trouble of implementing a Snappy Checksum'ed Compressor 
> implementation as it's been deprecated for a while -- is really slow and 
> crappy compared to LZ4 -- and we should do everything in our power to make 
> sure no one in the community is still using it. I left it in (for obvious 
> backwards compatibility aspects) old for clients that don't know about the 
> new protocol.
> The current protocol has a 256MB (max) frame body -- where the serialized 
> contents are simply written in to the frame body.
> If the client sends a compression option in the startup, we will install a 
> FrameCompressor inline. Unfortunately, we went with a decision to treat the 
> frame body separately from the header bits etc in a given message. So, 
> instead we put a compressor implementation in the options and then if it's 
> not null, we push the serialized bytes for the frame body *only*

[jira] [Commented] (CASSANDRA-13993) Add optional startup delay to wait until peers are ready

2018-04-06 Thread Aleksey Yeschenko (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428607#comment-16428607
 ] 

Aleksey Yeschenko commented on CASSANDRA-13993:
---

[~jasobrown] I don't mean backporting this whole ticket - just the ability to 
parse {{PING}} messages. We can just discard them once parsed (:

> Add optional startup delay to wait until peers are ready
> 
>
> Key: CASSANDRA-13993
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13993
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Lifecycle
>Reporter: Jason Brown
>Assignee: Jason Brown
>Priority: Minor
> Fix For: 4.0
>
>
> When bouncing a node in a large cluster, is can take a while to recognize the 
> rest of the cluster as available. This is especially true if using TLS on 
> internode messaging connections. The bouncing node (and any clients connected 
> to it) may see a series of Unavailable or Timeout exceptions until the node 
> is 'warmed up' as connecting to the rest of the cluster is asynchronous from 
> the rest of the startup process.
> There are two aspects that drive a node's ability to successfully communicate 
> with a peer after a bounce:
> - marking the peer as 'alive' (state that is held in gossip). This affects 
> the unavailable exceptions
> - having both open outbound and inbound connections open and ready to each 
> peer. This affects timeouts.
> Details of each of these mechanisms are described in the comments below.
> This ticket proposes adding a mechanism, optional and configurable, to delay 
> opening the client native protocol port until some percentage of the peers in 
> the cluster is marked alive and connected to/from. Thus while we potentially 
> slow down startup (delay opening the client port), we alleviate the chance 
> that queries made by clients don't hit transient unavailable/timeout 
> exceptions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13993) Add optional startup delay to wait until peers are ready

2018-04-06 Thread Jason Brown (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428604#comment-16428604
 ] 

Jason Brown commented on CASSANDRA-13993:
-

On the surface, it seems like backporting Ping-related stuffs is more invasive 
than just skipping some arbitrary bytes in the stream. However, if I think 
understand [~iamaleksey]'s reasoning, skipping some bytes in the stream has 
larger implications and essentially a larger behavior change than simply adding 
a new message. If that's true, then I agree that backporting the Ping message 
is the behavior-wise best route to go.

I'll go ahead and start working on the changes as discussed.

> Add optional startup delay to wait until peers are ready
> 
>
> Key: CASSANDRA-13993
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13993
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Lifecycle
>Reporter: Jason Brown
>Assignee: Jason Brown
>Priority: Minor
> Fix For: 4.0
>
>
> When bouncing a node in a large cluster, is can take a while to recognize the 
> rest of the cluster as available. This is especially true if using TLS on 
> internode messaging connections. The bouncing node (and any clients connected 
> to it) may see a series of Unavailable or Timeout exceptions until the node 
> is 'warmed up' as connecting to the rest of the cluster is asynchronous from 
> the rest of the startup process.
> There are two aspects that drive a node's ability to successfully communicate 
> with a peer after a bounce:
> - marking the peer as 'alive' (state that is held in gossip). This affects 
> the unavailable exceptions
> - having both open outbound and inbound connections open and ready to each 
> peer. This affects timeouts.
> Details of each of these mechanisms are described in the comments below.
> This ticket proposes adding a mechanism, optional and configurable, to delay 
> opening the client native protocol port until some percentage of the peers in 
> the cluster is marked alive and connected to/from. Thus while we potentially 
> slow down startup (delay opening the client port), we alleviate the chance 
> that queries made by clients don't hit transient unavailable/timeout 
> exceptions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13993) Add optional startup delay to wait until peers are ready

2018-04-06 Thread Aleksey Yeschenko (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428460#comment-16428460
 ] 

Aleksey Yeschenko commented on CASSANDRA-13993:
---

I agree with basically everything you said here - except what we should 
backport, so:

bq. if we do get an unknown verb id, skip the payload bytes in MessageIn. This 
leaves the input stream clean to process future messages.

Yes, please, for 4.0+.

bq. Further, I think we can eliminate the whole UNUSED_ verbs thing as that was 
an incomplete defense against unknown verbs, and it didn't account for message 
payload.

Yes please. Keep the five we have - or, four, rather, because one will be 
consumed by {{PING}} - and I'd still say let it be {{UNUSED_4}} or 5, but don't 
introduce any more in 4.0, or after 4.0. We will reclaim the existing ones 
eventually as we EOL older releases.

bq. backport part of CASSANDRA-13283 to get the Verb from a map, not an index 
array offset. This gives us safety for future-proofing against unknown verbs.

Not a bad idea, but we should probably be a bit more conservative re: what we 
backport to 3.0, and especially 2.2 at this point. How about, instead, we just 
backport {{PING}} to 3.11 and 3.0, so in the upgrade scenario there will be no 
harm to connections?

So, TL;DR, maybe do this?
1. Make 4.0 robust against {{null}} verb and skip remainders of messages we 
can't parse. There is precedent for it as well, see 
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/hints/HintMessage.java#L126-L128
2. Stop introducing new {{UNUSED_}} verbs starting with 4.0.
3. Backport {{PING}} to 3.0 and 3.11, so upgraders from recent 3.0 and 3.11 
with the fix will have a smoother experience when going to 4.0.

> Add optional startup delay to wait until peers are ready
> 
>
> Key: CASSANDRA-13993
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13993
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Lifecycle
>Reporter: Jason Brown
>Assignee: Jason Brown
>Priority: Minor
> Fix For: 4.0
>
>
> When bouncing a node in a large cluster, is can take a while to recognize the 
> rest of the cluster as available. This is especially true if using TLS on 
> internode messaging connections. The bouncing node (and any clients connected 
> to it) may see a series of Unavailable or Timeout exceptions until the node 
> is 'warmed up' as connecting to the rest of the cluster is asynchronous from 
> the rest of the startup process.
> There are two aspects that drive a node's ability to successfully communicate 
> with a peer after a bounce:
> - marking the peer as 'alive' (state that is held in gossip). This affects 
> the unavailable exceptions
> - having both open outbound and inbound connections open and ready to each 
> peer. This affects timeouts.
> Details of each of these mechanisms are described in the comments below.
> This ticket proposes adding a mechanism, optional and configurable, to delay 
> opening the client native protocol port until some percentage of the peers in 
> the cluster is marked alive and connected to/from. Thus while we potentially 
> slow down startup (delay opening the client port), we alleviate the chance 
> that queries made by clients don't hit transient unavailable/timeout 
> exceptions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14361) Allow SimpleSeedProvider to resolve multiple IPs per DNS name

2018-04-06 Thread Romain Hardouin (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428457#comment-16428457
 ] 

Romain Hardouin commented on CASSANDRA-14361:
-

{quote}Caching behavior remains the same, given operators relying on hostnames
{quote}
What I meant is that having this feature could motivate operators to use DNS. 
So they must be aware of this setting and set it explicitely. 

I've read Oracle documentation but Java security file is not very explicit:
{noformat}
# default value is forever (FOREVER). For security reasons, this
# caching is made forever when a security manager is set. When a security
# manager is not set, the default behavior in this implementation
# is to cache for 30 seconds.
#
# NOTE: setting this to anything other than the default value can have
#   serious security implications. Do not set it unless
#   you are sure you are not exposed to DNS spoofing attack.
#
#networkaddress.cache.ttl=-1
{noformat}

"{{default value is forever (FOREVER)}}" is misleading.
That's why having CASSANDRA-14364 is nice.

> Allow SimpleSeedProvider to resolve multiple IPs per DNS name
> -
>
> Key: CASSANDRA-14361
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14361
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Ben Bromhead
>Assignee: Ben Bromhead
>Priority: Minor
> Fix For: 4.0
>
>
> Currently SimpleSeedProvider can accept a comma separated string of IPs or 
> hostnames as the set of Cassandra seeds. hostnames are resolved via 
> InetAddress.getByName, which will only return the first IP associated with an 
> A,  or CNAME record.
> By changing to InetAddress.getAllByName, existing behavior is preserved, but 
> now Cassandra can discover multiple IP address per record, allowing seed 
> discovery by DNS to be a little easier.
> Some examples of improved workflows with this change include: 
>  * specify the DNS name of a headless service in Kubernetes which will 
> resolve to all IP addresses of pods within that service. 
>  * seed discovery for multi-region clusters via AWS route53, AzureDNS etc
>  * Other common DNS service discovery mechanisms.
> The only behavior this is likely to impact would be where users are relying 
> on the fact that getByName only returns a single IP address.
> I can't imagine any scenario where that is a sane choice. Even when that 
> choice has been made, it only impacts the first startup of Cassandra and 
> would not be on any critical path.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-14368) Add Spring Data for Apache Cassandra to Java Client listing

2018-04-06 Thread Mark Paluch (JIRA)

Mark Paluch created CASSANDRA-14368:
---

 Summary: Add Spring Data for Apache Cassandra to Java Client 
listing
 Key: CASSANDRA-14368
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14368
 Project: Cassandra
  Issue Type: Task
  Components: Documentation and Website
Reporter: Mark Paluch


I'd like to ask you to add Spring Data for Apache Cassandra to the [List of 
Java 
Clients|http://cassandra.apache.org/doc/latest/getting_started/drivers.html#java]
 your website.


GitHub Repo: [https://github.com/spring-projects/spring-data-cassandra]
Alternatively project site: [http://projects.spring.io/spring-data-cassandra/]

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13993) Add optional startup delay to wait until peers are ready

2018-04-06 Thread Jason Brown (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428361#comment-16428361
 ] 

Jason Brown commented on CASSANDRA-13993:
-

Responding to @aleksey's comments out of order, but hopefully makes sense at 
the end.

bq. we should handle ordinals that are outside of our known range robustly 
instead.

Yeah, I think this is where we should get to.

In {{MessageIn#read()}}, we read the verb id from the stream, and then fetch 
the {{Verb}} instance. In pre-4.0, we literally index into the {{Verb[]}} in 
{{MessagingService}}, so any unknown {{Verb}} s would blow up there with an 
ArrayIndexOutOfBoundsException. With CASSANDRA-13283, committed on trunk, we 
are more intelligently resistant to unknown {{Verb}} s, and would just get a 
null {{Verb}}. Unfortunately, trunk would still have problems with an unknown 
{{Verb}} as it would not know how to deserialize the message (pre-4.0, of 
course, suffers the same problem). It justs reads the basic header data, and 
passes it down, where [it would be dropped by 
{{MessageDeliveryTask}}|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/MessageDeliveryTask.java#L66].
 Unfortunately, if the message had more bytes in the stream which we didn't try 
to deseriliaze, trying to read the next message on the connection would fail 
spectacularly.

It's easy enough to avoid that, though, as we [already know the 
{{payloadSize}}|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/MessageIn.java#L146],
 so we can easily skip over the payload, and leave the incoming stream in a 
clean state after we account for the unknown message. Note: {{payloadSize}} is 
required by the internode messaging protocol, so we are sure to have the 
payload size. Thus, we can just safely skip the stream forward when we don't 
know how to deserialize the message, send it forward, and just discard it at 
{{MessagedeliveryTask}}.

bq. So I was thinking about a major upgrade bounce scenario. Think the first 
ever node to upgrade to 4.0 in a cluster of 3.0 nodes - will send out pings to 
every node, but receive no pongs, correct? So every node until a threshold will 
have a significantly longer bounce. Do we care about this case?

As the {{PingMessage}} contains a one-byte payload, it would leave the stream 
in a bad (unconsumed) state. This is a bug for the upgrade scenario. It's not a 
terrible bug, but it will cause the connection that we tried to eagerly create 
(to the un-upgraded peer) to be thrown away as it will fail on the next 
succeeding message on the connection. See proposal at the end.

bq. As implemented currently, we are going to send PINGs potentially to 
3.11/3.0 - unless we switch to gating by version, which we do sometimes.

So here's the rub: we don't necessarily know the peer's version yet. The ping 
messages are sent on the large/small connections, but we're not guaranteed that 
at least one round of gossip has completed wherein we would learn the version 
of the peers (we're still at in the startup process). The un-upgraded node 
won't know how to respond to the the unkown {{Verb}}, which is acceptable, but 
we shouldn't leave the stream on that connection in a broken state (see above).

Proposal: 
- backport part of CASSANDRA-13283 to get the {{Verb}} from a map, not an index 
array offset. This gives us safety for future-proofing against unknown verbs.  
- if we do get an unknown verb id, skip the payload bytes in {{MessageIn}}. 
This leaves the input stream clean to process future messages.
- Further, I think we can eliminate the whole {{UNUSED_}} verbs thing as that 
was an incomplete defense against unknown verbs, and it didn't account for 
message payload.

I think if we backport this to at least 3.0 (maybe 2.2?) that should be 
sufficient for future-proofing against unknown messages. 

If this sounds reasonable, I'll open a separate ticket for that work.

> Add optional startup delay to wait until peers are ready
> 
>
> Key: CASSANDRA-13993
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13993
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Lifecycle
>Reporter: Jason Brown
>Assignee: Jason Brown
>Priority: Minor
> Fix For: 4.0
>
>
> When bouncing a node in a large cluster, is can take a while to recognize the 
> rest of the cluster as available. This is especially true if using TLS on 
> internode messaging connections. The bouncing node (and any clients connected 
> to it) may see a series of Unavailable or Timeout exceptions until the node 
> is 'warmed up' as connecting to the rest of the cluster is asynchronous from 
> the rest of the startup process.
> There are two aspects that drive a node's ability to

[jira] [Updated] (CASSANDRA-14353) Fix some regressions caused by 14058

2018-04-06 Thread Aleksey Yeschenko (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-14353:
--
Status: Ready to Commit  (was: Patch Available)

> Fix some regressions caused by 14058
> 
>
> Key: CASSANDRA-14353
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14353
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Major
> Fix For: 4.0
>
>
> As [~iamaleksey] pointed out, CASSANDRA-14058 introduced a few regressions:
> {quote}
> Noticed a couple regressions when merging up CASSANDRA-14330:
> 1. {{DataResolver}} no longer uses 
> {{cassandra.drop_oversized_readrepair_mutations}} prop - and 
> {{DROP_OVERSIZED_READ_REPAIR_MUTATIONS}} constant is now unused, and the 
> feature is missing.
> 2. {{RowIteratorMergeListener}} re-thrown {{AssertionError}} no longer 
> includes the responses. This should be restored, as without it debugging RR 
> issues is an even worse, potentially impossible, nightmare.
> Nit: In {{DataResolver}}, {{repairResults}} field is now unused.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14353) Fix some regressions caused by 14058

2018-04-06 Thread Aleksey Yeschenko (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428354#comment-16428354
 ] 

Aleksey Yeschenko commented on CASSANDRA-14353:
---

A part of me thinks that the extra two catch-rethrows in {{DataResolver}} are 
unnecessary - we've only ever hit assertions in notoriously tricky 
{{onMergedRangeTombstoneMarkers()}}, not in 
{{onMergedPartitionLevelDeletion()}} or {{onMergedRows()}}. That part of me 
thinks we should just preserve the old behaviour, which would also get rid of 
triplication.

But overall I'm ambivalent. I feel very strongly about 
{{onMergedRangeTombstoneMarkers()}} catch/retrhow block staying, but the other 
two can go, if you like. It can also stay if you like. Feel free to decide on 
commit.

+1 either way, the change to bring back over oversized mutation handling is 
fine too (made a note to add the missing test for it).

> Fix some regressions caused by 14058
> 
>
> Key: CASSANDRA-14353
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14353
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Major
> Fix For: 4.0
>
>
> As [~iamaleksey] pointed out, CASSANDRA-14058 introduced a few regressions:
> {quote}
> Noticed a couple regressions when merging up CASSANDRA-14330:
> 1. {{DataResolver}} no longer uses 
> {{cassandra.drop_oversized_readrepair_mutations}} prop - and 
> {{DROP_OVERSIZED_READ_REPAIR_MUTATIONS}} constant is now unused, and the 
> feature is missing.
> 2. {{RowIteratorMergeListener}} re-thrown {{AssertionError}} no longer 
> includes the responses. This should be restored, as without it debugging RR 
> issues is an even worse, potentially impossible, nightmare.
> Nit: In {{DataResolver}}, {{repairResults}} field is now unused.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14148) test_no_base_column_in_view_pk_complex_timestamp_with_flush - materialized_views_test.TestMaterializedViews frequently fails in CI

2018-04-06 Thread Paulo Motta (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-14148:

Status: Ready to Commit  (was: Patch Available)

> test_no_base_column_in_view_pk_complex_timestamp_with_flush - 
> materialized_views_test.TestMaterializedViews frequently fails in CI
> --
>
> Key: CASSANDRA-14148
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14148
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Michael Kjellman
>Assignee: Marcus Eriksson
>Priority: Major
>
> test_no_base_column_in_view_pk_complex_timestamp_with_flush - 
> materialized_views_test.TestMaterializedViews frequently fails in CI
> self =  0x7f849b25cf60>
> @since('3.0')
> def test_no_base_column_in_view_pk_complex_timestamp_with_flush(self):
> >   self._test_no_base_column_in_view_pk_complex_timestamp(flush=True)
> materialized_views_test.py:970: 
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
> _ 
> materialized_views_test.py:1066: in 
> _test_no_base_column_in_view_pk_complex_timestamp
> assert_one(session, "SELECT * FROM t", [1, 1, None, None, None, 1])
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
> _ 
> session = 
> query = 'SELECT * FROM t', expected = [1, 1, None, None, None, 1], cl = None
> def assert_one(session, query, expected, cl=None):
> """
> Assert query returns one row.
> @param session Session to use
> @param query Query to run
> @param expected Expected results from query
> @param cl Optional Consistency Level setting. Default ONE
> 
> Examples:
> assert_one(session, "LIST USERS", ['cassandra', True])
> assert_one(session, query, [0, 0])
> """
> simple_query = SimpleStatement(query, consistency_level=cl)
> res = session.execute(simple_query)
> list_res = _rows_to_list(res)
> >   assert list_res == [expected], "Expected {} from {}, but got 
> > {}".format([expected], query, list_res)
> E   AssertionError: Expected [[1, 1, None, None, None, 1]] from SELECT * 
> FROM t, but got []



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14148) test_no_base_column_in_view_pk_complex_timestamp_with_flush - materialized_views_test.TestMaterializedViews frequently fails in CI

2018-04-06 Thread Paulo Motta (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428334#comment-16428334
 ] 

Paulo Motta commented on CASSANDRA-14148:
-

sorry for the delay. fix LGTM, executed multiplexer run and this seems to fix 
flakiness. Marking as ready to commit, thanks!

> test_no_base_column_in_view_pk_complex_timestamp_with_flush - 
> materialized_views_test.TestMaterializedViews frequently fails in CI
> --
>
> Key: CASSANDRA-14148
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14148
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Michael Kjellman
>Assignee: Marcus Eriksson
>Priority: Major
>
> test_no_base_column_in_view_pk_complex_timestamp_with_flush - 
> materialized_views_test.TestMaterializedViews frequently fails in CI
> self =  0x7f849b25cf60>
> @since('3.0')
> def test_no_base_column_in_view_pk_complex_timestamp_with_flush(self):
> >   self._test_no_base_column_in_view_pk_complex_timestamp(flush=True)
> materialized_views_test.py:970: 
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
> _ 
> materialized_views_test.py:1066: in 
> _test_no_base_column_in_view_pk_complex_timestamp
> assert_one(session, "SELECT * FROM t", [1, 1, None, None, None, 1])
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
> _ 
> session = 
> query = 'SELECT * FROM t', expected = [1, 1, None, None, None, 1], cl = None
> def assert_one(session, query, expected, cl=None):
> """
> Assert query returns one row.
> @param session Session to use
> @param query Query to run
> @param expected Expected results from query
> @param cl Optional Consistency Level setting. Default ONE
> 
> Examples:
> assert_one(session, "LIST USERS", ['cassandra', True])
> assert_one(session, query, [0, 0])
> """
> simple_query = SimpleStatement(query, consistency_level=cl)
> res = session.execute(simple_query)
> list_res = _rows_to_list(res)
> >   assert list_res == [expected], "Expected {} from {}, but got 
> > {}".format([expected], query, list_res)
> E   AssertionError: Expected [[1, 1, None, None, None, 1]] from SELECT * 
> FROM t, but got []



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14359) CREATE TABLE fails if there is a column called "default" with Cassandra 3.11.2

2018-04-06 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrés de la Peña updated CASSANDRA-14359:
--
Fix Version/s: 3.11.3
   3.0.17
   4.0
   Status: Patch Available  (was: Open)

> CREATE TABLE fails if there is a column called "default" with Cassandra 3.11.2
> --
>
> Key: CASSANDRA-14359
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14359
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL, Documentation and Website
> Environment: This is using Cassandra 3.11.2. This syntax was accepted 
> in 2.1.20.
>Reporter: Andy Klages
>Assignee: Andrés de la Peña
>Priority: Minor
> Fix For: 4.0, 3.0.17, 3.11.3
>
>
> My project is upgrading from Cassandra 2.1 to 3.11. We have a table whose 
> column name is "default". The Cassandra 3.11.2 is rejecting it. I don't see 
> "default" as a keyword in the CQL spec. 
> To reproduce, try adding the following:
> {code:java}
> CREATE TABLE simple (
> simplekey text PRIMARY KEY,
> default text // THIS IS REJECTED
> );
> {code}
> I get this error:
> {code:java}
> SyntaxException: line 3:4 mismatched input 'default' expecting ')' (...
> simplekey text PRIMARY KEY,[default]...)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14359) CREATE TABLE fails if there is a column called "default" with Cassandra 3.11.2

2018-04-06 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrés de la Peña updated CASSANDRA-14359:
--
Component/s: Documentation and Website

> CREATE TABLE fails if there is a column called "default" with Cassandra 3.11.2
> --
>
> Key: CASSANDRA-14359
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14359
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL, Documentation and Website
> Environment: This is using Cassandra 3.11.2. This syntax was accepted 
> in 2.1.20.
>Reporter: Andy Klages
>Assignee: Andrés de la Peña
>Priority: Minor
>
> My project is upgrading from Cassandra 2.1 to 3.11. We have a table whose 
> column name is "default". The Cassandra 3.11.2 is rejecting it. I don't see 
> "default" as a keyword in the CQL spec. 
> To reproduce, try adding the following:
> {code:java}
> CREATE TABLE simple (
> simplekey text PRIMARY KEY,
> default text // THIS IS REJECTED
> );
> {code}
> I get this error:
> {code:java}
> SyntaxException: line 3:4 mismatched input 'default' expecting ')' (...
> simplekey text PRIMARY KEY,[default]...)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14359) CREATE TABLE fails if there is a column called "default" with Cassandra 3.11.2

2018-04-06 Thread JIRA


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428330#comment-16428330
 ] 

Andrés de la Peña commented on CASSANDRA-14359:
---

{{DEFAULT}} keyword was added by CASSANDRA-11424 and included in 
[{{ReservedKeywords}}|https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/cql3/ReservedKeywords.java#L89]
 by CASSANDRA-14205.

That keyword and some others are indeed missed in the CQL documentation:
* cassandra-3.0: {{IS}}, {{MATERIALIZED}}, {{VIEW}}
* cassandra-3.11 and trunk: {{IS}}, {{CAST}}, {{DEFAULT}}, {{DURATION}}, 
{{GROUP}}, {{LIKE}}, {{MATERIALIZED}}, {{MBEAN}}, {{MBEANS}}, {{PER}}, 
{{PARTITION}}, {{UNSET}}, {{VIEW}}

This patch adds them to the {{CQL.textile}}:
||[3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...adelapena:14359-3.0]||[3.11|https://github.com/apache/cassandra/compare/cassandra-3.11...adelapena:14359-3.11]||[trunk|https://github.com/apache/cassandra/compare/trunk...adelapena:14359-trunk]||

> CREATE TABLE fails if there is a column called "default" with Cassandra 3.11.2
> --
>
> Key: CASSANDRA-14359
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14359
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: This is using Cassandra 3.11.2. This syntax was accepted 
> in 2.1.20.
>Reporter: Andy Klages
>Assignee: Andrés de la Peña
>Priority: Minor
>
> My project is upgrading from Cassandra 2.1 to 3.11. We have a table whose 
> column name is "default". The Cassandra 3.11.2 is rejecting it. I don't see 
> "default" as a keyword in the CQL spec. 
> To reproduce, try adding the following:
> {code:java}
> CREATE TABLE simple (
> simplekey text PRIMARY KEY,
> default text // THIS IS REJECTED
> );
> {code}
> I get this error:
> {code:java}
> SyntaxException: line 3:4 mismatched input 'default' expecting ')' (...
> simplekey text PRIMARY KEY,[default]...)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-14359) CREATE TABLE fails if there is a column called "default" with Cassandra 3.11.2

2018-04-06 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrés de la Peña reassigned CASSANDRA-14359:
-

Assignee: Andrés de la Peña

> CREATE TABLE fails if there is a column called "default" with Cassandra 3.11.2
> --
>
> Key: CASSANDRA-14359
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14359
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: This is using Cassandra 3.11.2. This syntax was accepted 
> in 2.1.20.
>Reporter: Andy Klages
>Assignee: Andrés de la Peña
>Priority: Minor
>
> My project is upgrading from Cassandra 2.1 to 3.11. We have a table whose 
> column name is "default". The Cassandra 3.11.2 is rejecting it. I don't see 
> "default" as a keyword in the CQL spec. 
> To reproduce, try adding the following:
> {code:java}
> CREATE TABLE simple (
> simplekey text PRIMARY KEY,
> default text // THIS IS REJECTED
> );
> {code}
> I get this error:
> {code:java}
> SyntaxException: line 3:4 mismatched input 'default' expecting ')' (...
> simplekey text PRIMARY KEY,[default]...)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-12674) [SASI] Confusing AND/OR semantics for StandardAnalyzer

2018-04-06 Thread DOAN DuyHai (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-12674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428229#comment-16428229
 ] 

DOAN DuyHai commented on CASSANDRA-12674:
-

Look like this ticket id dead.

Because the fix for the bug is far from trivial I don't see any solution soon, 
or ever 

> [SASI] Confusing AND/OR semantics for StandardAnalyzer 
> ---
>
> Key: CASSANDRA-12674
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12674
> Project: Cassandra
>  Issue Type: Bug
>  Components: sasi
> Environment: Cassandra 3.7
>Reporter: DOAN DuyHai
>Assignee: Alex Petrov
>Priority: Major
>
> {code:sql}
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 3.7 | CQL spec 3.4.2 | Native protocol v4]
> Use HELP for help.
> cqlsh> use test;
> cqlsh:test> CREATE TABLE sasi_bug(id int, clustering int, val text, PRIMARY 
> KEY((id), clustering));
> cqlsh:test> CREATE CUSTOM INDEX ON sasi_bug(val) USING 
> 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = {
> 'mode': 'CONTAINS',
>  'analyzer_class': 
> 'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer',
> 'analyzed': 'true'};
> //1st example SAME PARTITION KEY
> cqlsh:test> INSERT INTO sasi_bug(id, clustering , val ) VALUES(1, 1, 
> 'homeworker');
> cqlsh:test> INSERT INTO sasi_bug(id, clustering , val ) VALUES(1, 2, 
> 'hardworker');
> cqlsh:test> SELECT * FROM sasi_bug WHERE val LIKE '%work home%';
>  id | clustering | val
> ++
>   1 |  1 | homeworker
>   1 |  2 | hardworker
> (2 rows)
> //2nd example DIFFERENT PARTITION KEY
> cqlsh:test> INSERT INTO sasi_bug(id, clustering, val) VALUES(10, 1, 
> 'speedrun');
> cqlsh:test> INSERT INTO sasi_bug(id, clustering, val) VALUES(11, 1, 
> 'longrun');
> cqlsh:test> SELECT * FROM sasi_bug WHERE val LIKE '%long run%';
>  id | clustering | val
> ++-
>  11 |  1 | longrun
> (1 rows)
> {code}
> In the 1st example, both rows belong to the same partition so SASI returns 
> both values. Indeed {{LIKE '%work home%'}} means {{contains 'work' OR 
> 'home'}} so the result makes sense
> In the 2nd example, only one row is returned whereas we expect 2 rows because 
> {{LIKE '%long run%'}} means {{contains 'long' OR 'run'}} so *speedrun* should 
> be returned too.
> So where is the problem ? Explanation:
> When there is only 1 predicate, the root operation type is an *AND*:
> {code:java|title=QueryPlan}
> private Operation analyze()
> {
> try
> {
> Operation.Builder and = new Operation.Builder(OperationType.AND, 
> controller);
> controller.getExpressions().forEach(and::add);
> return and.complete();
> }
>...
> }
> {code}
> During the parsing of {{LIKE '%long run%'}}, SASI creates 2 expressions for 
> the searched term: {{long}} and {{run}}, which corresponds to an *OR* logic. 
> However, this piece of code just ruins the *OR* logic:
> {code:java|title=Operation}
> public Operation complete()
> {
> if (!expressions.isEmpty())
> {
> ListMultimap 
> analyzedExpressions = analyzeGroup(controller, op, expressions);
> RangeIterator.Builder range = 
> controller.getIndexes(op, analyzedExpressions.values());
>  ...
> }
> {code}
> As you can see, we blindly take all the *values* of the MultiMap (which 
> contains a single entry for the {{val}} column with 2 expressions) and pass 
> it to {{controller.getIndexes(...)}}
> {code:java|title=QueryController}
> public RangeIterator.Builder getIndexes(OperationType op, 
> Collection expressions)
> {
> if (resources.containsKey(expressions))
> throw new IllegalArgumentException("Can't process the same 
> expressions multiple times.");
> RangeIterator.Builder builder = op == OperationType.OR
> ? RangeUnionIterator. Token>builder()
> : 
> RangeIntersectionIterator.builder();
> ...
> }
> {code}
> And because the root operation has *AND* type, the 
> {{RangeIntersectionIterator}} will be used on both expressions {{long}} and 
> {{run}}.
> So when data belong to different partitions, we have the *AND* logic that 
> applies and eliminates _speedrun_
> When data belong to the same partition but different row, the 
> {{RangeIntersectionIterator}} returns a single partition and then the rows 
> are filtered further by {{operationTree.satisfiedBy}} and the results are 
> correct
> {code:java|title=QueryPlan}
> while (currentKeys.hasNext())
> {
>

[jira] [Commented] (CASSANDRA-14355) Memory leak

2018-04-06 Thread Thomas Steinmaurer (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428135#comment-16428135
 ] 

Thomas Steinmaurer commented on CASSANDRA-14355:


[~zznate]:
{quote}
Also, Thomas Steinmaurer this issue looks different from what you experienced 
in CASSANDRA-13929, but have you all encountered anything like this before?
{quote}
No, sorry. Looks different.

> Memory leak
> ---
>
> Key: CASSANDRA-14355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14355
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Debian Jessie, OpenJDK 1.8.0_151
>Reporter: Eric Evans
>Priority: Major
> Fix For: 3.11.3
>
> Attachments: 01_Screenshot from 2018-04-04 14-24-00.png, 
> 02_Screenshot from 2018-04-04 14-28-33.png, 03_Screenshot from 2018-04-04 
> 14-24-50.png
>
>
> We're seeing regular, frequent {{OutOfMemoryError}} exceptions.  Similar to 
> CASSANDRA-13754, an analysis of the heap dumps shows the heap consumed by the 
> {{threadLocals}} member of the instances of 
> {{io.netty.util.concurrent.FastThreadLocalThread}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-12151) Audit logging for database activity

2018-04-06 Thread Laxmikant Upadhyay (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428102#comment-16428102
 ] 

Laxmikant Upadhyay edited comment on CASSANDRA-12151 at 4/6/18 8:31 AM:


Can we have a configurable property like exitOnAuditFailure ? This fulfills 
requirement of strict auditing .. I mean in case auditing fails for a db 
operation, then the operation should not get executed. 


was (Author: laxmikant99):
Can we have a configurable property like exitOnAuditFailure ? This fulfills 
requirement of strict auditing .. I mean in case auditing fails db operation 
should not get executed. 

> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, 
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done 
> on our server.
> It should show username, remote address, timestamp, action type, keyspace, 
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the 
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every 
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting 
> a specific keyspace and column family.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity

2018-04-06 Thread Laxmikant Upadhyay (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428102#comment-16428102
 ] 

Laxmikant Upadhyay commented on CASSANDRA-12151:


Can we have a configurable property like exitOnAuditFailure ? This fulfills 
requirement of strict auditing .. I mean in case auditing fails db operation 
should not get executed. 

> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, 
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done 
> on our server.
> It should show username, remote address, timestamp, action type, keyspace, 
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the 
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every 
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting 
> a specific keyspace and column family.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

51 matches

Mail list logo