[jira] [Comment Edited] (CASSANDRA-13241) Lower default chunk_length_in_kb from 64kb to 4kb
[
https://issues.apache.org/jira/browse/CASSANDRA-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16656030#comment-16656030
]
Ariel Weisberg edited comment on CASSANDRA-13241 at 10/18/18 11:34 PM:
---
Running
{noformat}
#!/bin/sh
echo "drop keyspace keyspace1;" | ../../bin/cqlsh
./cassandra-stress write no-warmup n=1 -pop seq=1...1 -schema
compression=LZ4Compressor
./cassandra-stress read no-warmup n=1000 -pop dist=UNIFORM\(1...1\)
-rate threads=32
{noformat}
64k load
{noformat}
Results:
Op rate : 42,237 op/s [WRITE: 42,254 op/s]
Partition rate: 42,237 pk/s [WRITE: 42,254 pk/s]
Row rate : 42,237 row/s [WRITE: 42,254 row/s]
Latency mean :4.7 ms [WRITE: 4.7 ms]
Latency median:1.6 ms [WRITE: 1.6 ms]
Latency 95th percentile : 13.2 ms [WRITE: 13.2 ms]
Latency 99th percentile : 85.3 ms [WRITE: 85.3 ms]
Latency 99.9th percentile : 230.0 ms [WRITE: 230.0 ms]
Latency max : 629.1 ms [WRITE: 629.1 ms]
Total partitions : 100,000,000 [WRITE: 100,000,000]
Total errors : 0 [WRITE: 0]
Total GC count: 0
Total GC memory : 0.000 KiB
Total GC time :0.0 seconds
Avg GC time :NaN ms
StdDev GC time:0.0 ms
Total operation time : 00:39:27
{noformat}
64k read
{noformat}
Results:
Op rate :6,576 op/s [READ: 6,576 op/s]
Partition rate:6,576 pk/s [READ: 6,576 pk/s]
Row rate :6,576 row/s [READ: 6,576 row/s]
Latency mean :4.8 ms [READ: 4.8 ms]
Latency median:3.0 ms [READ: 3.0 ms]
Latency 95th percentile : 12.9 ms [READ: 12.9 ms]
Latency 99th percentile : 32.6 ms [READ: 32.6 ms]
Latency 99.9th percentile : 100.8 ms [READ: 100.8 ms]
Latency max : 14982.1 ms [READ: 14,982.1 ms]
Total partitions : 10,000,000 [READ: 10,000,000]
Total errors : 0 [READ: 0]
Total GC count: 0
Total GC memory : 0.000 KiB
Total GC time :0.0 seconds
Avg GC time :NaN ms
StdDev GC time:0.0 ms
Total operation time : 00:25:20
{noformat}
16k write
{noformat}
Results:
Op rate : 42,266 op/s [WRITE: 42,266 op/s]
Partition rate: 42,266 pk/s [WRITE: 42,266 pk/s]
Row rate : 42,266 row/s [WRITE: 42,266 row/s]
Latency mean :4.7 ms [WRITE: 4.7 ms]
Latency median:1.6 ms [WRITE: 1.6 ms]
Latency 95th percentile : 13.1 ms [WRITE: 13.1 ms]
Latency 99th percentile : 83.2 ms [WRITE: 83.2 ms]
Latency 99.9th percentile : 218.1 ms [WRITE: 218.1 ms]
Latency max : 886.0 ms [WRITE: 886.0 ms]
Total partitions : 100,000,000 [WRITE: 100,000,000]
Total errors : 0 [WRITE: 0]
Total GC count: 0
Total GC memory : 0.000 KiB
Total GC time :0.0 seconds
Avg GC time :NaN ms
StdDev GC time:0.0 ms
Total operation time : 00:39:25
{noformat}
16k read
{noformat}
Op rate :8,622 op/s [READ: 8,622 op/s]
Partition rate:8,622 pk/s [READ: 8,622 pk/s]
Row rate :8,622 row/s [READ: 8,622 row/s]
Latency mean :3.7 ms [READ: 3.7 ms]
Latency median:2.6 ms [READ: 2.6 ms]
Latency 95th percentile :9.0 ms [READ: 9.0 ms]
Latency 99th percentile : 22.2 ms [READ: 22.2 ms]
Latency 99.9th percentile : 63.5 ms [READ: 63.5 ms]
Latency max : 256.8 ms [READ: 256.8 ms]
Total partitions : 10,000,000 [READ: 10,000,000]
Total errors : 0 [READ: 0]
Total GC count: 0
Total GC memory : 0.000 KiB
Total GC time :0.0 seconds
Avg GC time :NaN ms
StdDev GC time:0.0 ms
Total operation time : 00:19:19
{noformat}
This read workload is 2x faster with 16k chunks vs 64k chunks.
was (Author: aweisberg):
Running
{noformat}
#!/bin/sh
echo "drop keyspace keyspace1;" | ../../bin/cqlsh
./cassandra-stress write no-warmup n=1 -pop seq=1...1 -schema
compression=LZ4Compressor
./cassandra-stress read no-warmup n=1000 -pop dist=UNIFORM\(1...1\)
-rate threads=32
{noformat}
64k load
{noformat}
Results:
Op rate : 42,237 op/s [WRITE: 42,254 op/s]
Partition rate: 42,237 pk/s [WRITE: 42,254 pk/s]
Row rate : 42,237 row/s [WRITE: 42,254 row/s]
Latency mean :4.7 ms [WRITE: 4.7 ms]
Latency median:1.6 ms [WRITE: 1.6 ms]
Latency 95th percentile : 13.2 ms [WRITE: 13.2 ms]
Latency 99th percentile : 85.3 ms [WRITE: 85.3 ms]
Latency 99.9th
[jira] [Commented] (CASSANDRA-13241) Lower default chunk_length_in_kb from 64kb to 4kb
[
https://issues.apache.org/jira/browse/CASSANDRA-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16656030#comment-16656030
]
Ariel Weisberg commented on CASSANDRA-13241:
Running
{noformat}
#!/bin/sh
echo "drop keyspace keyspace1;" | ../../bin/cqlsh
./cassandra-stress write no-warmup n=1 -pop seq=1...1 -schema
compression=LZ4Compressor
./cassandra-stress read no-warmup n=1000 -pop dist=UNIFORM\(1...1\)
-rate threads=32
{noformat}
64k load
{noformat}
Results:
Op rate : 42,237 op/s [WRITE: 42,254 op/s]
Partition rate: 42,237 pk/s [WRITE: 42,254 pk/s]
Row rate : 42,237 row/s [WRITE: 42,254 row/s]
Latency mean :4.7 ms [WRITE: 4.7 ms]
Latency median:1.6 ms [WRITE: 1.6 ms]
Latency 95th percentile : 13.2 ms [WRITE: 13.2 ms]
Latency 99th percentile : 85.3 ms [WRITE: 85.3 ms]
Latency 99.9th percentile : 230.0 ms [WRITE: 230.0 ms]
Latency max : 629.1 ms [WRITE: 629.1 ms]
Total partitions : 100,000,000 [WRITE: 100,000,000]
Total errors : 0 [WRITE: 0]
Total GC count: 0
Total GC memory : 0.000 KiB
Total GC time :0.0 seconds
Avg GC time :NaN ms
StdDev GC time:0.0 ms
Total operation time : 00:39:27
{noformat}
64k read
{noformat}
Results:
Op rate :6,576 op/s [READ: 6,576 op/s]
Partition rate:6,576 pk/s [READ: 6,576 pk/s]
Row rate :6,576 row/s [READ: 6,576 row/s]
Latency mean :4.8 ms [READ: 4.8 ms]
Latency median:3.0 ms [READ: 3.0 ms]
Latency 95th percentile : 12.9 ms [READ: 12.9 ms]
Latency 99th percentile : 32.6 ms [READ: 32.6 ms]
Latency 99.9th percentile : 100.8 ms [READ: 100.8 ms]
Latency max : 14982.1 ms [READ: 14,982.1 ms]
Total partitions : 10,000,000 [READ: 10,000,000]
Total errors : 0 [READ: 0]
Total GC count: 0
Total GC memory : 0.000 KiB
Total GC time :0.0 seconds
Avg GC time :NaN ms
StdDev GC time:0.0 ms
Total operation time : 00:25:20
16k write
Results:
Op rate : 42,266 op/s [WRITE: 42,266 op/s]
Partition rate: 42,266 pk/s [WRITE: 42,266 pk/s]
Row rate : 42,266 row/s [WRITE: 42,266 row/s]
Latency mean :4.7 ms [WRITE: 4.7 ms]
Latency median:1.6 ms [WRITE: 1.6 ms]
Latency 95th percentile : 13.1 ms [WRITE: 13.1 ms]
Latency 99th percentile : 83.2 ms [WRITE: 83.2 ms]
Latency 99.9th percentile : 218.1 ms [WRITE: 218.1 ms]
Latency max : 886.0 ms [WRITE: 886.0 ms]
Total partitions : 100,000,000 [WRITE: 100,000,000]
Total errors : 0 [WRITE: 0]
Total GC count: 0
Total GC memory : 0.000 KiB
Total GC time :0.0 seconds
Avg GC time :NaN ms
StdDev GC time:0.0 ms
Total operation time : 00:39:25
{noformat}
16k read
{noformat}
Op rate :8,622 op/s [READ: 8,622 op/s]
Partition rate:8,622 pk/s [READ: 8,622 pk/s]
Row rate :8,622 row/s [READ: 8,622 row/s]
Latency mean :3.7 ms [READ: 3.7 ms]
Latency median:2.6 ms [READ: 2.6 ms]
Latency 95th percentile :9.0 ms [READ: 9.0 ms]
Latency 99th percentile : 22.2 ms [READ: 22.2 ms]
Latency 99.9th percentile : 63.5 ms [READ: 63.5 ms]
Latency max : 256.8 ms [READ: 256.8 ms]
Total partitions : 10,000,000 [READ: 10,000,000]
Total errors : 0 [READ: 0]
Total GC count: 0
Total GC memory : 0.000 KiB
Total GC time :0.0 seconds
Avg GC time :NaN ms
StdDev GC time:0.0 ms
Total operation time : 00:19:19
{noformat}
This read workload is 2x faster with 16k chunks vs 64k chunks.
> Lower default chunk_length_in_kb from 64kb to 4kb
> -
>
> Key: CASSANDRA-13241
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13241
> Project: Cassandra
> Issue Type: Wish
> Components: Core
>Reporter: Benjamin Roth
>Assignee: Ariel Weisberg
>Priority: Major
> Attachments: CompactIntegerSequence.java,
> CompactIntegerSequenceBench.java, CompactSummingIntegerSequence.java
>
>
> Having a too low chunk size may result in some wasted disk space. A too high
> chunk size may lead to massive overreads and may have a critical impact on
> overall system performance.
> In my case, the default chunk size lead to peak read IOs of up to 1GB/s and
> avg reads of 200MB/s. After lowering
[jira] [Commented] (CASSANDRA-13241) Lower default chunk_length_in_kb from 64kb to 4kb
[ https://issues.apache.org/jira/browse/CASSANDRA-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16656028#comment-16656028 ] Ariel Weisberg commented on CASSANDRA-13241: For those who were asking about the performance impact of block size on compression I wrote a microbenchmark. https://pastebin.com/RHDNLGdC [java] Benchmark Mode Cnt Score Error Units [java] CompactIntegerSequenceBench.benchCompressLZ4Fast16kthrpt 15 331190055.685 ± 8079758.044 ops/s [java] CompactIntegerSequenceBench.benchCompressLZ4Fast32kthrpt 15 353024925.655 ± 7980400.003 ops/s [java] CompactIntegerSequenceBench.benchCompressLZ4Fast64kthrpt 15 365664477.654 ± 10083336.038 ops/s [java] CompactIntegerSequenceBench.benchCompressLZ4Fast8k thrpt 15 305518114.172 ± 11043705.883 ops/s [java] CompactIntegerSequenceBench.benchDecompressLZ4Fast16k thrpt 15 688369529.911 ± 25620873.933 ops/s [java] CompactIntegerSequenceBench.benchDecompressLZ4Fast32k thrpt 15 703635848.895 ± 5296941.704 ops/s [java] CompactIntegerSequenceBench.benchDecompressLZ4Fast64k thrpt 15 695537044.676 ± 17400763.731 ops/s [java] CompactIntegerSequenceBench.benchDecompressLZ4Fast8k thrpt 15 727725713.128 ± 4252436.331 ops/s To summarize, compression is 8.5% slower and decompression is 1% faster. This is measuring the impact on compression/decompression not the huge impact that would occur if we decompressed data we don't need less often. I didn't test decompression of Snappy and LZ4 high, but I did test compression. Snappy: [java] CompactIntegerSequenceBench.benchCompressSnappy16k thrpt2 196574766.116 ops/s [java] CompactIntegerSequenceBench.benchCompressSnappy32k thrpt2 198538643.844 ops/s [java] CompactIntegerSequenceBench.benchCompressSnappy64k thrpt2 194600497.613 ops/s [java] CompactIntegerSequenceBench.benchCompressSnappy8kthrpt2 186040175.059 ops/s LZ4 high compressor: [java] CompactIntegerSequenceBench.bench16k thrpt2 20822947.578 ops/s [java] CompactIntegerSequenceBench.bench32k thrpt2 12037342.253 ops/s [java] CompactIntegerSequenceBench.bench64k thrpt2 6782534.469 ops/s [java] CompactIntegerSequenceBench.bench8k thrpt2 32254619.594 ops/s LZ4 high is the one instance where block size mattered a lot. It's a bit suspicious really when you look at the ratio of performance to block size being close to 1:1. I couldn't spot a bug in the benchmark though. Compression ratios with LZ4 fast for the text of Alice in Wonderland was: Chunk size 8192, ratio 0.709473 Chunk size 16384, ratio 0.667236 Chunk size 32768, ratio 0.634735 Chunk size 65536, ratio 0.607208 By way of comparison I also ran deflate with maximum compression: Chunk size 8192, ratio 0.426434 Chunk size 16384, ratio 0.402423 Chunk size 32768, ratio 0.381627 Chunk size 65536, ratio 0.364865 > Lower default chunk_length_in_kb from 64kb to 4kb > - > > Key: CASSANDRA-13241 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13241 > Project: Cassandra > Issue Type: Wish > Components: Core >Reporter: Benjamin Roth >Assignee: Ariel Weisberg >Priority: Major > Attachments: CompactIntegerSequence.java, > CompactIntegerSequenceBench.java, CompactSummingIntegerSequence.java > > > Having a too low chunk size may result in some wasted disk space. A too high > chunk size may lead to massive overreads and may have a critical impact on > overall system performance. > In my case, the default chunk size lead to peak read IOs of up to 1GB/s and > avg reads of 200MB/s. After lowering chunksize (of course aligned with read > ahead), the avg read IO went below 20 MB/s, rather 10-15MB/s. > The risk of (physical) overreads is increasing with lower (page cache size) / > (total data size) ratio. > High chunk sizes are mostly appropriate for bigger payloads pre request but > if the model consists rather of small rows or small resultsets, the read > overhead with 64kb chunk size is insanely high. This applies for example for > (small) skinny rows. > Please also see here: > https://groups.google.com/forum/#!topic/scylladb-dev/j_qXSP-6-gY > To give you some insights what a difference it can make (460GB data, 128GB > RAM): > - Latency of a quite large CF: https://cl.ly/1r3e0W0S393L > - Disk throughput: https://cl.ly/2a0Z250S1M3c > - This shows, that the request distribution remained the same, so no "dynamic > snitch magic": https://cl.ly/3E0t1T1z2c0J -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CASSANDRA-14832) Other threads can take all newly allocated BufferPool chunks before original and cause reallocation
[ https://issues.apache.org/jira/browse/CASSANDRA-14832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655798#comment-16655798 ] Abhish Agarwal commented on CASSANDRA-14832: I agree with the approach here. There is a possibility of starvation given that there is no memory available and multiple processes vie for it. Correct way is to deduct the required memory before leaving it to free pool. > Other threads can take all newly allocated BufferPool chunks before original > and cause reallocation > --- > > Key: CASSANDRA-14832 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14832 > Project: Cassandra > Issue Type: Improvement >Reporter: Jon Meredith >Priority: Minor > > When BufferPool does not have any free Chunks to satisfy a request, the > calling thread allocates a new large block of memory which it breaks up into > chunks and adds to the free chunks queue, then pulls from the queue to > satisfy it's own allocation. > If enough other threads request chunks it is possible for the queue to be > exhausted before the original allocating thread is able to pull of it's own > allocation, causing the original allocator to loop and attempt to allocate > more memory. This is unfair to the original caller and may cause it to block > on a system call to allocate more memory. > Instead of the current behavior, allocateMoreChunks could hold back one of > the chunks and return it to the caller instead so that it will at most call > allocate once. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-11710) Cassandra dies with OOM when running stress
[
https://issues.apache.org/jira/browse/CASSANDRA-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated CASSANDRA-11710:
---
Labels: pull-request-available (was: )
> Cassandra dies with OOM when running stress
> ---
>
> Key: CASSANDRA-11710
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11710
> Project: Cassandra
> Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Branimir Lambov
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.6
>
>
> Running stress on trunk dies with OOM after about 3.5M ops:
> {code}
> ERROR [CompactionExecutor:1] 2016-05-04 15:01:31,231
> JVMStabilityInspector.java:137 - JVM state determined to be unstable.
> Exiting forcefully due to:
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:693) ~[na:1.8.0_91]
> at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123)
> ~[na:1.8.0_91]
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
> ~[na:1.8.0_91]
> at
> org.apache.cassandra.utils.memory.BufferPool.allocateDirectAligned(BufferPool.java:519)
> ~[main/:na]
> at
> org.apache.cassandra.utils.memory.BufferPool.access$600(BufferPool.java:46)
> ~[main/:na]
> at
> org.apache.cassandra.utils.memory.BufferPool$GlobalPool.allocateMoreChunks(BufferPool.java:276)
> ~[main/:na]
> at
> org.apache.cassandra.utils.memory.BufferPool$GlobalPool.get(BufferPool.java:249)
> ~[main/:na]
> at
> org.apache.cassandra.utils.memory.BufferPool$LocalPool.addChunkFromGlobalPool(BufferPool.java:338)
> ~[main/:na]
> at
> org.apache.cassandra.utils.memory.BufferPool$LocalPool.get(BufferPool.java:381)
> ~[main/:na]
> at
> org.apache.cassandra.utils.memory.BufferPool.maybeTakeFromPool(BufferPool.java:142)
> ~[main/:na]
> at
> org.apache.cassandra.utils.memory.BufferPool.takeFromPool(BufferPool.java:114)
> ~[main/:na]
> at
> org.apache.cassandra.utils.memory.BufferPool.get(BufferPool.java:84)
> ~[main/:na]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:135)
> ~[main/:na]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:19)
> ~[main/:na]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:2949)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$15(BoundedLocalCache.java:1807)
> ~[caffeine-2.2.6.jar:na]
> at
> java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
> ~[na:1.8.0_91]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1805)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1788)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:97)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:66)
> ~[caffeine-2.2.6.jar:na]
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:215)
> ~[main/:na]
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:193)
> ~[main/:na]
> at
> org.apache.cassandra.io.util.LimitingRebufferer.rebuffer(LimitingRebufferer.java:34)
> ~[main/:na]
> at
> org.apache.cassandra.io.util.RandomAccessReader.reBufferAt(RandomAccessReader.java:78)
> ~[main/:na]
> at
> org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:72)
> ~[main/:na]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.read(RebufferingInputStream.java:88)
> ~[main/:na]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:66)
> ~[main/:na]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
> ~[main/:na]
> at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:400)
> ~[main/:na]
> at
> org.apache.cassandra.utils.ByteBufferUtil.readWithVIntLength(ByteBufferUtil.java:338)
> ~[main/:na]
> at
> org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:414)
> ~[main/:na]
> at
> org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:243)
> ~[main/:na]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.readSimpleColumn(UnfilteredSerializer.java:473)
> ~[main/:na]
> at
>
[jira] [Created] (CASSANDRA-14832) Other threads can take all newly allocated BufferPool chunks before original and cause reallocation
Jon Meredith created CASSANDRA-14832: Summary: Other threads can take all newly allocated BufferPool chunks before original and cause reallocation Key: CASSANDRA-14832 URL: https://issues.apache.org/jira/browse/CASSANDRA-14832 Project: Cassandra Issue Type: Improvement Reporter: Jon Meredith When BufferPool does not have any free Chunks to satisfy a request, the calling thread allocates a new large block of memory which it breaks up into chunks and adds to the free chunks queue, then pulls from the queue to satisfy it's own allocation. If enough other threads request chunks it is possible for the queue to be exhausted before the original allocating thread is able to pull of it's own allocation, causing the original allocator to loop and attempt to allocate more memory. This is unfair to the original caller and may cause it to block on a system call to allocate more memory. Instead of the current behavior, allocateMoreChunks could hold back one of the chunks and return it to the caller instead so that it will at most call allocate once. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14721) sstabledump displays incorrect value for "position" key
[
https://issues.apache.org/jira/browse/CASSANDRA-14721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Lohfink updated CASSANDRA-14721:
--
Reviewers: Chris Lohfink
> sstabledump displays incorrect value for "position" key
> ---
>
> Key: CASSANDRA-14721
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14721
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
>Reporter: Damien Stevenson
>Assignee: Cameron Zemek
>Priority: Minor
> Attachments: cassandra-dump.patch
>
>
> When partitions with multiple rows are displayed using sstabledump, the
> "position" value the first row of each partition is incorrect.
> For example:
> {code:java}
> sstabledump mc-1-big-Data.db
> [
> {
> "partition" : {
> "key" : [ "1", "24" ],
> "position" : 0
> },
> "rows" : [
> {
> "type" : "row",
> "position" : 66,
> "clustering" : [ "2013-12-10 00:00:00.000Z" ],
> "liveness_info" : { "tstamp" : "2018-09-12T05:01:09.290086Z" },
> "cells" : [
> { "name" : "centigrade", "value" : 8 },
> { "name" : "chanceofrain", "value" : 0.1 },
> { "name" : "feelslike", "value" : 8 },
> { "name" : "humidity", "value" : 0.76 },
> { "name" : "wind", "value" : 10.0 }
> ]
> },
> {
> "type" : "row",
> "position" : 66,
> "clustering" : [ "2013-12-11 00:00:00.000Z" ],
> "liveness_info" : { "tstamp" : "2018-09-12T05:01:09.295369Z" },
> "cells" : [
> { "name" : "centigrade", "value" : 4 },
> { "name" : "chanceofrain", "value" : 0.3 },
> { "name" : "feelslike", "value" : 4 },
> { "name" : "humidity", "value" : 0.9 },
> { "name" : "wind", "value" : 12.0 }
> ]
> },
> {
> "type" : "row",
> "position" : 105,
> "clustering" : [ "2013-12-12 00:00:00.000Z" ],
> "liveness_info" : { "tstamp" : "2018-09-12T05:01:09.300841Z" },
> "cells" : [
> { "name" : "centigrade", "value" : 3 },
> { "name" : "chanceofrain", "value" : 0.2 },
> { "name" : "feelslike", "value" : 3 },
> { "name" : "humidity", "value" : 0.68 },
> { "name" : "wind", "value" : 6.0 }
> ]
> }
> ]
> }
> ]
> {code}
> The expected output is:
> {code:java}
> [
> {
> "partition" : {
> "key" : [ "1", "24" ],
> "position" : 0
> },
> "rows" : [
> {
> "type" : "row",
> "position" : 28,
> "clustering" : [ "2013-12-10 00:00:00.000Z" ],
> "liveness_info" : { "tstamp" : "2018-09-12T05:01:09.290086Z" },
> "cells" : [
> { "name" : "centigrade", "value" : 8 },
> { "name" : "chanceofrain", "value" : 0.1 },
> { "name" : "feelslike", "value" : 8 },
> { "name" : "humidity", "value" : 0.76 },
> { "name" : "wind", "value" : 10.0 }
> ]
> },
> {
> "type" : "row",
> "position" : 66,
> "clustering" : [ "2013-12-11 00:00:00.000Z" ],
> "liveness_info" : { "tstamp" : "2018-09-12T05:01:09.295369Z" },
> "cells" : [
> { "name" : "centigrade", "value" : 4 },
> { "name" : "chanceofrain", "value" : 0.3 },
> { "name" : "feelslike", "value" : 4 },
> { "name" : "humidity", "value" : 0.9 },
> { "name" : "wind", "value" : 12.0 }
> ]
> },
> {
> "type" : "row",
> "position" : 105,
> "clustering" : [ "2013-12-12 00:00:00.000Z" ],
> "liveness_info" : { "tstamp" : "2018-09-12T05:01:09.300841Z" },
> "cells" : [
> { "name" : "centigrade", "value" : 3 },
> { "name" : "chanceofrain", "value" : 0.2 },
> { "name" : "feelslike", "value" : 3 },
> { "name" : "humidity", "value" : 0.68 },
> { "name" : "wind", "value" : 6.0 }
> ]
> }
> ]
> }
> ]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14820) Upgrade to 4.0 fails with NullPointerException
[
https://issues.apache.org/jira/browse/CASSANDRA-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655332#comment-16655332
]
C. Scott Andreas commented on CASSANDRA-14820:
--
Thanks [~aweisberg] and [~djoshi3].
[~tommy_s], thank you very much for this report!
> Upgrade to 4.0 fails with NullPointerException
> --
>
> Key: CASSANDRA-14820
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14820
> Project: Cassandra
> Issue Type: Bug
> Components: 4.0
>Reporter: Tommy Stendahl
>Assignee: Ariel Weisberg
>Priority: Major
>
> I tested to upgrade an existing cluster to latest 4.0 but it fails with a
> NullPointerException, I upgraded from 3.0.15 but upgrading from any 3.0.x or
> 3.11.x to 4.0 will give the same fault.
> {noformat}
>
> 2018-10-12T11:27:02.261+0200 ERROR [main] CassandraDaemon.java:251 Error
> while loading schema:
> java.lang.NullPointerException: null
> at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:156)
> at
> org.apache.cassandra.serializers.AbstractTextSerializer.deserialize(AbstractTextSerializer.java:41)
> at
> org.apache.cassandra.serializers.AbstractTextSerializer.deserialize(AbstractTextSerializer.java:28)
> at
> org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:116)
> at
> org.apache.cassandra.cql3.UntypedResultSet$Row.getString(UntypedResultSet.java:267)
> at
> org.apache.cassandra.schema.SchemaKeyspace.createTableParamsFromRow(SchemaKeyspace.java:997)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:973)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:927)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:886)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:877)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:865)
> at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102)
> at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:590)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:679)
> {noformat}
> The problem seams to be line 997 in SchemaKeyspace.java
>
> {noformat}
> .speculativeWriteThreshold(SpeculativeRetryPolicy.fromString(row.getString("speculative_write_threshold"{noformat}
> speculative_write_threshold is a new table option introduced in
> CASSANDRA-14404, when upgrading the table option is missing and we get a
> NullPointerException on this line.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14787) nodetool status "Load" columns has wrong width
[
https://issues.apache.org/jira/browse/CASSANDRA-14787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Lohfink updated CASSANDRA-14787:
--
Reviewers: Chris Lohfink
> nodetool status "Load" columns has wrong width
> --
>
> Key: CASSANDRA-14787
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14787
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
>Reporter: Lapo Luchini
>Assignee: Kirk True
>Priority: Trivial
> Labels: lhf, nodetool, pull-request-available
> Fix For: 4.x
>
> Attachments: CASSANDRA-14787.txt
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Using Cassandra 3.11.2 on FreeBSD, I get:
> {code:java}
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens Owns (effective) Host ID ...
> UN server1.andxor.it 11.11 MiB 256 39.6% ...
> UN server2.andxor.it 32.04 MiB 256 41.8% ...
> UN server3.andxor.it 519.33 KiB 256 40.0% ...
> UN server4.andxor.it 10.95 MiB 256 40.3% ...
> UN server5.andxor.it 11.03 MiB 256 38.4% ...
> {code}
> AFAICT this is caused by {{"%-9s"}} in
> [Status.java:292|https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/tools/nodetool/Status.java#L292]
> which should be probably a 10 instead of 9.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14781) Log message when mutation passed to CommitLog#add(Mutation) is too large is not descriptive enough
[ https://issues.apache.org/jira/browse/CASSANDRA-14781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Lohfink updated CASSANDRA-14781: -- Reviewer: Chris Lohfink > Log message when mutation passed to CommitLog#add(Mutation) is too large is > not descriptive enough > -- > > Key: CASSANDRA-14781 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14781 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths >Reporter: Jordan West >Assignee: Tom Petracca >Priority: Major > Fix For: 3.0.x, 3.11.x, 4.0.x > > Attachments: CASSANDRA-14781.patch, CASSANDRA-14781_3.0.patch > > > When hitting > [https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/commitlog/CommitLog.java#L256-L257], > the log message produced does not help the operator track down what data is > being written. At a minimum the keyspace and cfIds involved would be useful > (and are available) – more detail might not be reasonable to include. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14781) Log message when mutation passed to CommitLog#add(Mutation) is too large is not descriptive enough
[ https://issues.apache.org/jira/browse/CASSANDRA-14781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655317#comment-16655317 ] Chris Lohfink commented on CASSANDRA-14781: --- To cover extreme cases, can you add a limit of like 10 tables then add a " ... and X more" or something? Also can you include the partition keys with a similar cap to prevent the string from getting too large. For the common case where theres only 1 large mutation or a bunch to a single partition that would be monumentally helpful. > Log message when mutation passed to CommitLog#add(Mutation) is too large is > not descriptive enough > -- > > Key: CASSANDRA-14781 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14781 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths >Reporter: Jordan West >Assignee: Tom Petracca >Priority: Major > Fix For: 3.0.x, 3.11.x, 4.0.x > > Attachments: CASSANDRA-14781.patch, CASSANDRA-14781_3.0.patch > > > When hitting > [https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/commitlog/CommitLog.java#L256-L257], > the log message produced does not help the operator track down what data is > being written. At a minimum the keyspace and cfIds involved would be useful > (and are available) – more detail might not be reasonable to include. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14787) nodetool status "Load" columns has wrong width
[
https://issues.apache.org/jira/browse/CASSANDRA-14787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655302#comment-16655302
]
Chris Lohfink commented on CASSANDRA-14787:
---
The offset should be from the max width instead of hard coded since it can vary
so much. The other nodetool outputs have been moving to use {{TableBuilder}} so
that it always aligns properly. Perhaps this table can be changed to use it as
well.
> nodetool status "Load" columns has wrong width
> --
>
> Key: CASSANDRA-14787
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14787
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
>Reporter: Lapo Luchini
>Assignee: Kirk True
>Priority: Trivial
> Labels: lhf, nodetool, pull-request-available
> Fix For: 4.x
>
> Attachments: CASSANDRA-14787.txt
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Using Cassandra 3.11.2 on FreeBSD, I get:
> {code:java}
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens Owns (effective) Host ID ...
> UN server1.andxor.it 11.11 MiB 256 39.6% ...
> UN server2.andxor.it 32.04 MiB 256 41.8% ...
> UN server3.andxor.it 519.33 KiB 256 40.0% ...
> UN server4.andxor.it 10.95 MiB 256 40.3% ...
> UN server5.andxor.it 11.03 MiB 256 38.4% ...
> {code}
> AFAICT this is caused by {{"%-9s"}} in
> [Status.java:292|https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/src/java/org/apache/cassandra/tools/nodetool/Status.java#L292]
> which should be probably a 10 instead of 9.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14690) Add dtests for fqltool replay/compare
[
https://issues.apache.org/jira/browse/CASSANDRA-14690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654986#comment-16654986
]
Stefan Podkowinski commented on CASSANDRA-14690:
There's also {{node.clear()}} as a possible option, instead of deleting data
dirs in your test.
+1 from my side
> Add dtests for fqltool replay/compare
> -
>
> Key: CASSANDRA-14690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14690
> Project: Cassandra
> Issue Type: Test
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Major
> Labels: fqltool
>
> We should add some basic round-trip dtests for {{fqltool replay}} and
> {{compare}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14790) LongBufferPoolTest burn test fails assertion
[
https://issues.apache.org/jira/browse/CASSANDRA-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654946#comment-16654946
]
Benedict commented on CASSANDRA-14790:
--
For good hygiene, it's generally considered better not to commit changes that
are orthogonal to the ticket in question, especially when the ticket is only
fixing a failing test. But it's such a tiny change it doesn't really warrant
its own JIRA, and people definitely don't always stick to the scope of the
JIRA. So, whatever you prefer.
> LongBufferPoolTest burn test fails assertion
>
>
> Key: CASSANDRA-14790
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14790
> Project: Cassandra
> Issue Type: Test
> Components: Testing
> Environment: Run under macOS 10.13.6, with patch (attached, but also
> https://github.com/jonmeredith/cassandra/tree/failing-burn-test)
>Reporter: Jon Meredith
>Assignee: Jon Meredith
>Priority: Major
> Labels: pull-request-available
> Attachments: 0001-Add-burn-testsome-target-to-build.xml.patch,
> 0002-Initialize-before-running-LongBufferPoolTest.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> The LongBufferPoolTest from the burn tests fails with an assertion error. I
> added a build target to run individual burn tests, and \{jasobrown} gave a
> fix for the uninitialized test setup (attached), however the test now fails
> on an assertion about recycling buffers.
> To reproduce (with patch applied)
> {{ant burn-testsome
> -Dtest.name=org.apache.cassandra.utils.memory.LongBufferPoolTest
> -Dtest.methods=testAllocate}}
> Output
> {{ [junit] Testcase:
> testAllocate(org.apache.cassandra.utils.memory.LongBufferPoolTest): FAILED}}
> {{ [junit] null}}
> {{ [junit] junit.framework.AssertionFailedError}}
> {{ [junit] at
> org.apache.cassandra.utils.memory.BufferPool$Debug.check(BufferPool.java:204)}}
> {{ [junit] at
> org.apache.cassandra.utils.memory.BufferPool.assertAllRecycled(BufferPool.java:181)}}
> {{ [junit] at
> org.apache.cassandra.utils.memory.LongBufferPoolTest.testAllocate(LongBufferPoolTest.java:350)}}
> {{ [junit] at
> org.apache.cassandra.utils.memory.LongBufferPoolTest.testAllocate(LongBufferPoolTest.java:54)}}
> All major branches from 3.0 and later have issues, however the trunk branch
> also warns about references not being released before the reference is
> garbage collected.
> {{[junit] ERROR [Reference-Reaper:1] 2018-09-25 13:59:54,089 Ref.java:224 -
> LEAK DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@7f58d19a) to @623704362 was
> not released before the reference was garbage collected}}
> {{ [junit] ERROR [Reference-Reaper:1] 2018-09-25 13:59:54,089 Ref.java:255 -
> Allocate trace org.apache.cassandra.utils.concurrent.Ref$State@7f58d19a:}}
> {{ [junit] Thread[pool-2-thread-24,5,main]}}
> {{ [junit] at java.lang.Thread.getStackTrace(Thread.java:1559)}}
> {{ [junit] at
> org.apache.cassandra.utils.concurrent.Ref$Debug.(Ref.java:245)}}
> {{ [junit] at
> org.apache.cassandra.utils.concurrent.Ref$State.(Ref.java:175)}}
> {{ [junit] at org.apache.cassandra.utils.concurrent.Ref.(Ref.java:97)}}
> {{ [junit] at
> org.apache.cassandra.utils.memory.BufferPool$Chunk.setAttachment(BufferPool.java:663)}}
> {{ [junit] at
> org.apache.cassandra.utils.memory.BufferPool$Chunk.get(BufferPool.java:803)}}
> {{ [junit] at
> org.apache.cassandra.utils.memory.BufferPool$Chunk.get(BufferPool.java:793)}}
> {{ [junit] at
> org.apache.cassandra.utils.memory.BufferPool$LocalPool.get(BufferPool.java:388)}}
> {{ [junit] at
> org.apache.cassandra.utils.memory.BufferPool.maybeTakeFromPool(BufferPool.java:143)}}
> {{ [junit] at
> org.apache.cassandra.utils.memory.BufferPool.takeFromPool(BufferPool.java:115)}}
> {{ [junit] at
> org.apache.cassandra.utils.memory.BufferPool.get(BufferPool.java:85)}}
> {{ [junit] at
> org.apache.cassandra.utils.memory.LongBufferPoolTest$3.allocate(LongBufferPoolTest.java:296)}}
> {{ [junit] at
> org.apache.cassandra.utils.memory.LongBufferPoolTest$3.testOne(LongBufferPoolTest.java:246)}}
> {{ [junit] at
> org.apache.cassandra.utils.memory.LongBufferPoolTest$TestUntil.call(LongBufferPoolTest.java:399)}}
> {{ [junit] at
> org.apache.cassandra.utils.memory.LongBufferPoolTest$TestUntil.call(LongBufferPoolTest.java:379)}}
> {{ [junit] at java.util.concurrent.FutureTask.run(FutureTask.java:266)}}
> {{ [junit] at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}
> {{ [junit] at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}
> {{ [junit] at java.lang.Thread.run(Thread.java:748)}}
>
> Perhaps the environment is not being set up correctly for the tests.
>
--
This message was sent by
[jira] [Commented] (CASSANDRA-14690) Add dtests for fqltool replay/compare
[
https://issues.apache.org/jira/browse/CASSANDRA-14690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654898#comment-16654898
]
Marcus Eriksson commented on CASSANDRA-14690:
-
pushed new commits with the comments fixed
test run: https://circleci.com/workflow-run/e394d48d-0f07-4345-9fd1-ed597b1a0a2c
> Add dtests for fqltool replay/compare
> -
>
> Key: CASSANDRA-14690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14690
> Project: Cassandra
> Issue Type: Test
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Major
> Labels: fqltool
>
> We should add some basic round-trip dtests for {{fqltool replay}} and
> {{compare}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14829) Make stop-server.bat wait for Cassandra to terminate
[ https://issues.apache.org/jira/browse/CASSANDRA-14829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Georg Dietrich updated CASSANDRA-14829: --- Description: While administering a single node Cassandra on Windows, I noticed that the stop-server.bat script returns before the cassandra process has actually terminated. For use cases like creating a script "shut down & create backup of data directory without having to worry about open files, then restart", it would be good to make stop-server.bat wait for Cassandra to terminate. All that is needed for that is to change in apache-cassandra-3.11.3\bin\stop-server.bat "start /B powershell /file ..." to "start /WAIT /B powershell /file ..." (additional /WAIT parameter). Does this sound reasonable? Here is the pull request: https://github.com/apache/cassandra/pull/287 was: While administering a single node Cassandra on Windows, I noticed that the stop-server.bat script returns before the cassandra process has actually terminated. For use cases like creating a script "shut down & create backup of data directory without having to worry about open files, then restart", it would be good to make stop-server.bat wait for Cassandra to terminate. All that is needed for that is to change in apache-cassandra-3.11.3\bin\stop-server.bat "start /B powershell /file ..." to "start /WAIT /B powershell /file ..." (additional /WAIT parameter). Does this sound reasonable? If yes, I'll create a pull request. > Make stop-server.bat wait for Cassandra to terminate > > > Key: CASSANDRA-14829 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14829 > Project: Cassandra > Issue Type: Improvement > Components: Packaging > Environment: Windows 10 >Reporter: Georg Dietrich >Priority: Minor > Labels: easyfix, windows > Fix For: 3.11.x, 4.x, 4.0.x > > > While administering a single node Cassandra on Windows, I noticed that the > stop-server.bat script returns before the cassandra process has actually > terminated. For use cases like creating a script "shut down & create backup > of data directory without having to worry about open files, then restart", it > would be good to make stop-server.bat wait for Cassandra to terminate. > All that is needed for that is to change in > apache-cassandra-3.11.3\bin\stop-server.bat "start /B powershell /file ..." > to "start /WAIT /B powershell /file ..." (additional /WAIT parameter). > Does this sound reasonable? > Here is the pull request: https://github.com/apache/cassandra/pull/287 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13458) Diag. Events: Add unit testing support
[ https://issues.apache.org/jira/browse/CASSANDRA-13458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-13458: --- Attachment: 0001-Add-unit-testing-support-based-on-diag.-events.patch > Diag. Events: Add unit testing support > -- > > Key: CASSANDRA-13458 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13458 > Project: Cassandra > Issue Type: Sub-task > Components: Testing >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Major > Attachments: 0001-Add-unit-testing-support-based-on-diag.-events.patch > > > Diagnostic events will improve unit testing by > * providing test execution control instances based on CompletableFutures (see > [PendingRangeCalculatorServiceTest.java|https://github.com/spodkowinski/cassandra/blob/WIP-13458/test/unit/org/apache/cassandra/gms/PendingRangeCalculatorServiceTest.java]) > > * validate state and behavior by allowing you to inspect generated events > (see > [HintsServiceEventsTest.java|https://github.com/spodkowinski/cassandra/blob/WIP-13458/test/unit/org/apache/cassandra/hints/HintsServiceEventsTest.java]) > See included > [testing.rst|https://github.com/spodkowinski/cassandra/blob/WIP-13458/doc/source/development/testing.rst#diagnostic-events-40] > draft for more details. Let me know if this would be useful for you as a > developer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14806) CircleCI workflow improvements and Java 11 support
[
https://issues.apache.org/jira/browse/CASSANDRA-14806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-14806:
---
Status: Patch Available (was: In Progress)
> CircleCI workflow improvements and Java 11 support
> --
>
> Key: CASSANDRA-14806
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14806
> Project: Cassandra
> Issue Type: Improvement
> Components: Build, Testing
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
>
> The current CircleCI config could use some cleanup and improvements. First of
> all, the config has been made more modular by using the new CircleCI 2.1
> executors and command elements. Based on CASSANDRA-14713, there's now also a
> Java 11 executor that will allow running tests under Java 11. The {{build}}
> step will be done using Java 11 in all cases, so we can catch any regressions
> for that and also test the Java 11 multi-jar artifact during dtests, that
> we'd also create during the release process.
> The job workflow has now also been changed to make use of the [manual job
> approval|https://circleci.com/docs/2.0/workflows/#holding-a-workflow-for-a-manual-approval]
> feature, which now allows running dtest jobs only on request and not
> automatically with every commit. The Java8 unit tests still do, but that
> could also be easily changed if needed. See [example
> workflow|https://circleci.com/workflow-run/be25579d-3cbb-4258-9e19-b1f571873850]
> with start_ jobs being triggers needed manual approval for starting the
> actual jobs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-12540) Password Management: Hardcoded Password
[
https://issues.apache.org/jira/browse/CASSANDRA-12540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski reassigned CASSANDRA-12540:
--
Assignee: (was: Stefan Podkowinski)
> Password Management: Hardcoded Password
> ---
>
> Key: CASSANDRA-12540
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12540
> Project: Cassandra
> Issue Type: Sub-task
>Reporter: Eduardo Aguinaga
>Priority: Major
> Attachments: 12540-trunk.patch
>
>
> Overview:
> In May through June of 2016 a static analysis was performed on version 3.0.5
> of the Cassandra source code. The analysis included an automated analysis
> using HP Fortify v4.21 SCA and a manual analysis utilizing SciTools
> Understand v4. The results of that analysis includes the issue below.
> Issue:
> In the file EncryptionOptions.java there are hard coded passwords on lines 23
> and 25.
> {code:java}
> EncryptionOptions.java, lines 20-30:
> 20 public abstract class EncryptionOptions
> 21 {
> 22 public String keystore = "conf/.keystore";
> 23 public String keystore_password = "cassandra";
> 24 public String truststore = "conf/.truststore";
> 25 public String truststore_password = "cassandra";
> 26 public String[] cipher_suites = {
> 27 "TLS_RSA_WITH_AES_128_CBC_SHA", "TLS_RSA_WITH_AES_256_CBC_SHA",
> 28 "TLS_DHE_RSA_WITH_AES_128_CBC_SHA",
> "TLS_DHE_RSA_WITH_AES_256_CBC_SHA",
> 29 "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA",
> "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA"
> 30 };
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14107) Dynamic key rotation support for transparent data encryption
[ https://issues.apache.org/jira/browse/CASSANDRA-14107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-14107: --- Attachment: patches-14107.tar > Dynamic key rotation support for transparent data encryption > > > Key: CASSANDRA-14107 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14107 > Project: Cassandra > Issue Type: New Feature >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Minor > Labels: encryption > Fix For: 4.x > > Attachments: patches-14107.tar > > > Handling of encryption keys as introduced in CASSANDRA-9945 takes place by > referencing a key alias in either cassandra.yaml, or the header of the > (commitlog/hints) file that has been encrypted. Using the alias as literal > value will work, but requires some attention when rotating keys. > Currently each time a key is rotated (i.e. adding a new key to the keystore > while preserving the previous version), the alias in cassandra.yaml has to be > update as well and the node needs to be restarted. It would be more > convenient to use a symbolic reference instead. My suggestion here would be > to use ":latest" for referring to the latest version. In this case > Cassandra always picks the key with the highest version in > ":". > The non-trivial part of this suggestion is how the "latest" key is referenced > in the file header. If we use "latest", e.g. for the commit log header, and > the key gets rotated, we'd now try do decrypt the file with the new key, > instead of the key it has been created with. Therefor we'd have to introduce > an extra step that will resolve the canonical version for "latest" and refer > to that one during any encrypt operation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-14107) Dynamic key rotation support for transparent data encryption
[ https://issues.apache.org/jira/browse/CASSANDRA-14107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski reassigned CASSANDRA-14107: -- Assignee: (was: Stefan Podkowinski) > Dynamic key rotation support for transparent data encryption > > > Key: CASSANDRA-14107 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14107 > Project: Cassandra > Issue Type: New Feature >Reporter: Stefan Podkowinski >Priority: Minor > Labels: encryption > Fix For: 4.x > > Attachments: patches-14107.tar > > > Handling of encryption keys as introduced in CASSANDRA-9945 takes place by > referencing a key alias in either cassandra.yaml, or the header of the > (commitlog/hints) file that has been encrypted. Using the alias as literal > value will work, but requires some attention when rotating keys. > Currently each time a key is rotated (i.e. adding a new key to the keystore > while preserving the previous version), the alias in cassandra.yaml has to be > update as well and the node needs to be restarted. It would be more > convenient to use a symbolic reference instead. My suggestion here would be > to use ":latest" for referring to the latest version. In this case > Cassandra always picks the key with the highest version in > ":". > The non-trivial part of this suggestion is how the "latest" key is referenced > in the file header. If we use "latest", e.g. for the commit log header, and > the key gets rotated, we'd now try do decrypt the file with the new key, > instead of the key it has been created with. Therefor we'd have to introduce > an extra step that will resolve the canonical version for "latest" and refer > to that one during any encrypt operation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-14102) Vault support for transparent data encryption
[
https://issues.apache.org/jira/browse/CASSANDRA-14102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski reassigned CASSANDRA-14102:
--
Assignee: (was: Stefan Podkowinski)
> Vault support for transparent data encryption
> -
>
> Key: CASSANDRA-14102
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14102
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Stefan Podkowinski
>Priority: Major
> Labels: encryption, security
> Fix For: 4.x
>
> Attachments: patches-14102.tar
>
>
> Transparent data encryption provided by CASSANDRA-9945 can currently be used
> for commitlog and hints. The default {{KeyProvider}} implementation that we
> ship allows to use a local keystore for storing and retrieving keys. Thanks
> to the pluggable handling of the {{KeyStore}} provider and basic Vault
> related classes introduced in CASSANDRA-13971, a Vault based implementation
> can be provided as {{KeyProvider}} as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14102) Vault support for transparent data encryption
[
https://issues.apache.org/jira/browse/CASSANDRA-14102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-14102:
---
Attachment: patches-14102.tar
> Vault support for transparent data encryption
> -
>
> Key: CASSANDRA-14102
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14102
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
> Labels: encryption, security
> Fix For: 4.x
>
> Attachments: patches-14102.tar
>
>
> Transparent data encryption provided by CASSANDRA-9945 can currently be used
> for commitlog and hints. The default {{KeyProvider}} implementation that we
> ship allows to use a local keystore for storing and retrieving keys. Thanks
> to the pluggable handling of the {{KeyStore}} provider and basic Vault
> related classes introduced in CASSANDRA-13971, a Vault based implementation
> can be provided as {{KeyProvider}} as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-14064) Allow using file based certificates instead of keystores
[ https://issues.apache.org/jira/browse/CASSANDRA-14064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski reassigned CASSANDRA-14064: -- Assignee: (was: Stefan Podkowinski) > Allow using file based certificates instead of keystores > > > Key: CASSANDRA-14064 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14064 > Project: Cassandra > Issue Type: New Feature > Components: Streaming and Messaging >Reporter: Stefan Podkowinski >Priority: Major > Labels: security > Fix For: 4.x > > Attachments: 0001-Add-keystore-less-ssl-handling.patch > > > The requirement of having to use a secure archive (keystore) for your > certificates and keys is not very common outside the Java ecosystem. Most > servers will accept individual certificate and key files and will come with > instructions how to generate those using openssl. This should also be an > option for Cassandra for users who see no reason in additionally having to > deal with keystores. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14064) Allow using file based certificates instead of keystores
[ https://issues.apache.org/jira/browse/CASSANDRA-14064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-14064: --- Attachment: 0001-Add-keystore-less-ssl-handling.patch > Allow using file based certificates instead of keystores > > > Key: CASSANDRA-14064 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14064 > Project: Cassandra > Issue Type: New Feature > Components: Streaming and Messaging >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Major > Labels: security > Fix For: 4.x > > Attachments: 0001-Add-keystore-less-ssl-handling.patch > > > The requirement of having to use a secure archive (keystore) for your > certificates and keys is not very common outside the Java ecosystem. Most > servers will accept individual certificate and key files and will come with > instructions how to generate those using openssl. This should also be an > option for Cassandra for users who see no reason in additionally having to > deal with keystores. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-13971) Automatic certificate management using Vault
[ https://issues.apache.org/jira/browse/CASSANDRA-13971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski reassigned CASSANDRA-13971: -- Assignee: (was: Stefan Podkowinski) > Automatic certificate management using Vault > > > Key: CASSANDRA-13971 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13971 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging >Reporter: Stefan Podkowinski >Priority: Major > Labels: security > Fix For: 4.x > > Attachments: patches-13971.tar, start_vault_ssl.sh > > > We've been adding security features during the last years to enable users to > secure their clusters, if they are willing to use them and do so correctly. > Some features are powerful and easy to work with, such as role based > authorization. Other features that require to manage a local keystore are > rather painful to deal with. Think about setting up SSL.. > To be fair, keystore related issues and certificate handling hasn't been > invented by us. We're just following Java standards there. But that doesn't > mean that we absolutely have to, if there are better options. I'd like to > give it a shoot and find out if we can automate certificate/key handling > (PKI) by using external APIs. In this case, the implementation will be based > on [Vault|https://vaultproject.io]. But certificate management services > offered by cloud providers may also be able to handle the use-case and I > intend to create a generic, pluggable API for that. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13971) Automatic certificate management using Vault
[ https://issues.apache.org/jira/browse/CASSANDRA-13971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-13971: --- Attachment: patches-13971.tar > Automatic certificate management using Vault > > > Key: CASSANDRA-13971 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13971 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Major > Labels: security > Fix For: 4.x > > Attachments: patches-13971.tar, start_vault_ssl.sh > > > We've been adding security features during the last years to enable users to > secure their clusters, if they are willing to use them and do so correctly. > Some features are powerful and easy to work with, such as role based > authorization. Other features that require to manage a local keystore are > rather painful to deal with. Think about setting up SSL.. > To be fair, keystore related issues and certificate handling hasn't been > invented by us. We're just following Java standards there. But that doesn't > mean that we absolutely have to, if there are better options. I'd like to > give it a shoot and find out if we can automate certificate/key handling > (PKI) by using external APIs. In this case, the implementation will be based > on [Vault|https://vaultproject.io]. But certificate management services > offered by cloud providers may also be able to handle the use-case and I > intend to create a generic, pluggable API for that. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14498) Audit log does not include statements on some system keyspaces
[
https://issues.apache.org/jira/browse/CASSANDRA-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654777#comment-16654777
]
Vinay Chella commented on CASSANDRA-14498:
--
[~eperott] Attached the patch to remove {{system}}, {{system_schema}} keyspaces
exclusion from AuditLogManager. This patch allows the user to enable the audit
log for system keyspaces.
> Audit log does not include statements on some system keyspaces
> --
>
> Key: CASSANDRA-14498
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14498
> Project: Cassandra
> Issue Type: Bug
> Components: Auth
>Reporter: Per Otterström
>Assignee: Vinay Chella
>Priority: Major
> Labels: audit, lhf, security
> Fix For: 4.0
>
> Attachments: 14498-trunk.txt
>
>
> Audit logs does not include statements on the "system" and "system_schema"
> keyspace.
> It may be a common use case to whitelist queries on these keyspaces, but
> Cassandra should not make assumptions. Users who don't want these statements
> in their audit log are still able to whitelist them with configuration.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14498) Audit log does not include statements on some system keyspaces
[ https://issues.apache.org/jira/browse/CASSANDRA-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay Chella updated CASSANDRA-14498: - Attachment: 14498-trunk.txt > Audit log does not include statements on some system keyspaces > -- > > Key: CASSANDRA-14498 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14498 > Project: Cassandra > Issue Type: Bug > Components: Auth >Reporter: Per Otterström >Assignee: Vinay Chella >Priority: Major > Labels: audit, lhf, security > Fix For: 4.0 > > Attachments: 14498-trunk.txt > > > Audit logs does not include statements on the "system" and "system_schema" > keyspace. > It may be a common use case to whitelist queries on these keyspaces, but > Cassandra should not make assumptions. Users who don't want these statements > in their audit log are still able to whitelist them with configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
