[jira] [Commented] (CASSANDRA-8621) For streaming operations, when a socket is closed/reset, we should retry/reinitiate that stream
[ https://issues.apache.org/jira/browse/CASSANDRA-8621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279774#comment-14279774 ] Jonathan Shook commented on CASSANDRA-8621: --- As well, there were no TCP level errors showing for the receiving side. So it is unclear whether exceptions are being omitted, or whether there was something really strange occurring with the network. > For streaming operations, when a socket is closed/reset, we should > retry/reinitiate that stream > --- > > Key: CASSANDRA-8621 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8621 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Jeremy Hanna >Assignee: Yuki Morishita > > Currently we have a setting (streaming_socket_timeout_in_ms) that will > timeout and retry the stream operation in the case where tcp is idle for a > period of time. However in the case where the socket is closed or reset, we > do not retry the operation. This can happen for a number of reasons, > including when a firewall sends a reset message on a socket during a > streaming operation, such as nodetool rebuild necessarily across DCs or > repairs. > Doing a retry would make the streaming operations more resilient. It would > be good to log the retry clearly as well (with the stream session ID and node > address). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8621) For streaming operations, when a socket is closed/reset, we should retry/reinitiate that stream
[ https://issues.apache.org/jira/browse/CASSANDRA-8621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279768#comment-14279768 ] Jonathan Shook commented on CASSANDRA-8621: --- For the scenario that prompted this ticket, it appeared that the streaming process was completely stalled. One side of the stream (the sender side) had an exception that appeared to be a connection reset. The receiving side appeared to think that the connection was still active, at least in terms of the netstats reported by nodetool. We were unable to verify whether this was specifically the case in terms of connected sockets due to the fact that there were multiple streams for those peers, and there is no simple way to correlate a specific stream to a tcp session. [~yukim] If there is a diagnostic method that we can use to provide more information about specific stalled streams, please let us know so that we can approach the user to get more data. > For streaming operations, when a socket is closed/reset, we should > retry/reinitiate that stream > --- > > Key: CASSANDRA-8621 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8621 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Jeremy Hanna >Assignee: Yuki Morishita > > Currently we have a setting (streaming_socket_timeout_in_ms) that will > timeout and retry the stream operation in the case where tcp is idle for a > period of time. However in the case where the socket is closed or reset, we > do not retry the operation. This can happen for a number of reasons, > including when a firewall sends a reset message on a socket during a > streaming operation, such as nodetool rebuild necessarily across DCs or > repairs. > Doing a retry would make the streaming operations more resilient. It would > be good to log the retry clearly as well (with the stream session ID and node > address). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8603) Cut tombstone memory footprint in half for cql deletes
[ https://issues.apache.org/jira/browse/CASSANDRA-8603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279632#comment-14279632 ] Dominic Letz commented on CASSANDRA-8603: - I used 2.1.2 for this testing. > Cut tombstone memory footprint in half for cql deletes > -- > > Key: CASSANDRA-8603 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8603 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Dominic Letz > Labels: tombstone > Attachments: cassandra-2.0.11-8603.txt, cassandra-2.1-8603.txt, > system.log > > > As CQL does not yet support range deletes every delete from CQL results in a > "Semi-RangeTombstone" which actually has the same start and end values - but > until today they are copies. Effectively doubling the required heap memory to > store the RangeTombstone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8603) Cut tombstone memory footprint in half for cql deletes
[
https://issues.apache.org/jira/browse/CASSANDRA-8603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dominic Letz updated CASSANDRA-8603:
Attachment: system.log
Sorry for the long delay here. I did testing before but your comment made me
look deeper and indeed something is wrong.
First off the attached patches do not catch the situation that they were
intended to catch - but they do catch something and that mislead me. So maybe
there is a bug as you pointed out.
The test case I'm using is like this:
{code}
CREATE KEYSPACE test
WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 3 };
CREATE TABLE test.timeseries (name text, timestamp int, value text, PRIMARY KEY
((name), timestamp))
WITH compaction = {'class': 'SizeTieredCompactionStrategy'} AND CLUSTERING
ORDER BY (timestamp DESC);
INSERT INTO test.timeseries (name, timestamp, value) VALUES('a', 1, 'hello');
INSERT INTO test.timeseries (name, timestamp, value) VALUES('a', 2, 'world');
INSERT INTO test.timeseries (name, timestamp, value) VALUES('a', 5, 'hello
world');
DELETE FROM test.timeseries WHERE name = 'a' AND timestamp = 1;
DELETE FROM test.timeseries WHERE name = 'a' AND timestamp = 2;
DELETE FROM test.timeseries WHERE name = 'a' AND timestamp = 5;
{code}
Plus adding this log line next to the initial patch content:
{code}
public RangeTombstone(Composite start, Composite stop, DeletionTime delTime)
{
super(start, stop.equals(start) ? start : stop, delTime);
LoggerFactory.getLogger(RangeTombstone.class).error(String.format("Test:
start.equals(stop) %s (%s == %s) %s, ", start.equals(stop), new
String(Hex.encodeHex(start.toByteBuffer().array())), new
String(Hex.encodeHex(stop.toByteBuffer().array())),
start.toByteBuffer().compareTo(stop.toByteBuffer(;
}
{/code}
Then I wiped the cassandra data directory and started from scratch, and once
started ran the said cql. The resulting system.log is attached
It seems that just during start-up there are a couple of hits into the new
logic where the tombstone .start and .end are actually exactly the same as
shown by the debug output:
{code}
ERROR [CompactionExecutor:1] 2015-01-16 08:50:57,476 RangeTombstone.java:48 -
Test: start.equals(stop) true (00 == 00) 0,
ERROR [CompactionExecutor:1] 2015-01-16 08:50:57,478 RangeTombstone.java:48 -
Test: start.equals(stop) false (0006746f6b656e73ff == 0006746f6b656e7301) -2,
ERROR [CompactionExecutor:1] 2015-01-16 08:50:57,480 RangeTombstone.java:48 -
Test: start.equals(stop) true (000c626f6f74737472617070656400 ==
000c626f6f74737472617070656400) 0,
ERROR [CompactionExecutor:1] 2015-01-16 08:50:57,481 RangeTombstone.java:48 -
Test: start.equals(stop) true (000c636c75737465725f6e616d6500 ==
000c636c75737465725f6e616d6500) 0,
...
{code}
But then further down in the log when the actual test cql code is executed
these messages pop up:
{code}
ERROR [Thrift:1] 2015-01-16 08:51:14,089 RangeTombstone.java:48 - Test:
start.equals(stop) false (00040001ff == 0004000101) -2,
ERROR [Thrift:1] 2015-01-16 08:51:14,092 RangeTombstone.java:48 - Test:
start.equals(stop) false (00040002ff == 0004000201) -2,
ERROR [Thrift:1] 2015-01-16 08:51:14,094 RangeTombstone.java:48 - Test:
start.equals(stop) false (00040005ff == 0004000501) -2,
{code}
Show that even though they keys match up until the provided sample value (1, 2,
5) there is an additional byte that is creating a difference and thus the code
is not catching. It seems I wasn't deep enough into the code to actually make
my proposed change successfully as I don't know the meaning of that last byte -
but the concept might still hold?
> Cut tombstone memory footprint in half for cql deletes
> --
>
> Key: CASSANDRA-8603
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8603
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Dominic Letz
> Labels: tombstone
> Attachments: cassandra-2.0.11-8603.txt, cassandra-2.1-8603.txt,
> system.log
>
>
> As CQL does not yet support range deletes every delete from CQL results in a
> "Semi-RangeTombstone" which actually has the same start and end values - but
> until today they are copies. Effectively doubling the required heap memory to
> store the RangeTombstone.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-8635) STCS cold sstable omission does not handle overwrites without reads
Tyler Hobbs created CASSANDRA-8635: -- Summary: STCS cold sstable omission does not handle overwrites without reads Key: CASSANDRA-8635 URL: https://issues.apache.org/jira/browse/CASSANDRA-8635 Project: Cassandra Issue Type: Bug Components: Core Reporter: Tyler Hobbs Assignee: Marcus Eriksson Priority: Critical Fix For: 2.1.3 In 2.1, STCS may omit cold SSTables from compaction (CASSANDRA-6109). If data is regularly overwritten or deleted (but not enough to trigger a single-sstable tombstone purging compaction), data size on disk may continuously grow if: * The table receives very few reads * The reads only touch the newest SSTables Basically, if the overwritten data is never read and there aren't many tombstones, STCS has no incentive to compact the sstables. We should take sstable overlap into consideration as well as coldness to address this case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8537) ConcurrentModificationException while executing 'nodetool cleanup'
[ https://issues.apache.org/jira/browse/CASSANDRA-8537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279576#comment-14279576 ] Sebastian Estevez commented on CASSANDRA-8537: -- @marcus just to be clear. This affects concurrent compactions not multithreaded compactions which we anyway removed in 2.1. As a temporary workaround until this is released, just set concurrent compactors to 1 in the yaml. > ConcurrentModificationException while executing 'nodetool cleanup' > -- > > Key: CASSANDRA-8537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8537 > Project: Cassandra > Issue Type: Bug > Components: Tools > Environment: Debian 7.7, Oracle JRE 1.7.0_72 >Reporter: Noureddine Chatti >Assignee: Marcus Eriksson > Fix For: 2.1.3 > > Attachments: > 0001-don-t-use-the-same-cleanupstrategy-for-all-sstables.patch > > > After adding a new node to an existing cluster (7 already started nodes), and > waiting a few minutes to be sure that data migration to the new node is > completed, I began to use the command nodetool cleanup sequentially on each > old node. When I issued this command on the third node, after a few minutes I > got a ConcurrentModificationException. > ~$ nodetool cleanup > error: null > -- StackTrace -- > java.util.ConcurrentModificationException > at java.util.ArrayList$Itr.checkForComodification(Unknown Source) > at java.util.ArrayList$Itr.next(Unknown Source) > at > org.apache.cassandra.db.index.SecondaryIndexManager.deleteFromIndexes(SecondaryIndexManager.java:476) > at > org.apache.cassandra.db.compaction.CompactionManager$CleanupStrategy$Full.cleanup(CompactionManager.java:833) > at > org.apache.cassandra.db.compaction.CompactionManager.doCleanupOne(CompactionManager.java:704) > at > org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:97) > at > org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:370) > at > org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:267) > at java.util.concurrent.FutureTask.run(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8464) Support direct buffer decompression for reads
[
https://issues.apache.org/jira/browse/CASSANDRA-8464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279474#comment-14279474
]
Branimir Lambov commented on CASSANDRA-8464:
I wonder why you still insist on not doing the length read in
{{LZ4Compressor.uncompress()}} the straightforward way. Are you worried about
the performance of ByteBuffer.get(position)? You shouldn't be, on a direct
buffer (including mapped ones) that's practically as fast as byte array access,
and even on a heap one it certainly isn't slow enough to warrant copying to an
array and the extra code.
Just to verify, I resurrected and ran a test I did of LZ4 decompression in safe
mode (which uses ByteBuffer.get/put), you can see the results
[here|http://pastebin.com/SFEqMbdJ].
> Support direct buffer decompression for reads
> -
>
> Key: CASSANDRA-8464
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8464
> Project: Cassandra
> Issue Type: Improvement
>Reporter: T Jake Luciani
>Assignee: T Jake Luciani
> Labels: performance
> Fix For: 3.0
>
> Attachments: compression_direct.png
>
>
> Currently when we read a compressed sstable we copy the data on heap then
> send it to be de-compressed to another on heap buffer (albeit pooled).
> But now both snappy and lz4 (with CASSANDRA-7039) allow decompression of
> direct byte buffers. This lets us mmap the data and decompress completely
> off heap (and avoids moving bytes over JNI).
> One issue is performing the checksum offheap but the Adler32 does support in
> java 8 (it's also in java 7 but marked private?!)
> This change yields a > 10% boost in read performance on cstar. Locally I see
> upto 30% improvement.
> http://cstar.datastax.com/graph?stats=5ebcdd70-816b-11e4-aed6-42010af0688f&metric=op_rate&operation=2_read&smoothing=1&show_aggregates=true&xmin=0&xmax=200.09&ymin=0&ymax=135908.3
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8630) Faster sequencial IO (on compaction, streaming, etc)
[ https://issues.apache.org/jira/browse/CASSANDRA-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279415#comment-14279415 ] Oleg Anastasyev commented on CASSANDRA-8630: Unfortunatedly I dont have numbers ready to share on this. Will try to perform tests without RAR then. > Faster sequencial IO (on compaction, streaming, etc) > > > Key: CASSANDRA-8630 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8630 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Oleg Anastasyev >Assignee: Oleg Anastasyev > Labels: performance > Attachments: 8630-FasterSequencialReadsAndWrites.txt, cpu_load.png > > > When node is doing a lot of sequencial IO (streaming, compacting, etc) a lot > of CPU is lost in calls to RAF's int read() and DataOutputStream's write(int). > This is because default implementations of readShort,readLong, etc as well as > their matching write* are implemented with numerous calls of byte by byte > read and write. > This makes a lot of syscalls as well. > A quick microbench shows than just reimplementation of these methods in > either way gives 8x speed increase. > A patch attached implements RandomAccessReader.read and > SequencialWriter.write methods in more efficient way. > I also eliminated some extra byte copies in CompositeType.split and > ColumnNameHelper.maxComponents, which were on my profiler's hotspot method > list during tests. > A stress tests on my laptop show that this patch makes compaction 25-30% > faster on uncompressed sstables and 15% faster for compressed ones. > A deployment to production shows much less CPU load for compaction. > (I attached a cpu load graph from one of our production, orange is niced CPU > load - i.e. compaction; yellow is user - i.e. not compaction related tasks) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8634) Multi-thread commit log compression
[ https://issues.apache.org/jira/browse/CASSANDRA-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ariel Weisberg updated CASSANDRA-8634: -- Description: The initial commit log compression implementation can do about 240 megabytes/second (uncompressed input) on a fast (500 megabytes/sec) SSD. This is basically what I measured as the throughput of LZ4 against the same data. Assuming a 1.5x compression ratio that is 160 megabytes/second of output. Enough to saturate a spinning disk, but not an SSD. was: The initial commit log compression implementation can do about 240 megabytes/second (uncompressed input) on a fast (500 megabytes/sec) SSD. Assuming a 1.5x compression ratio that is 160 megabytes/second of output. Enough to saturate a spinning disk, but not an SSD. > Multi-thread commit log compression > --- > > Key: CASSANDRA-8634 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8634 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Ariel Weisberg > Labels: performance > > The initial commit log compression implementation can do about 240 > megabytes/second (uncompressed input) on a fast (500 megabytes/sec) SSD. > This is basically what I measured as the throughput of LZ4 against the same > data. > Assuming a 1.5x compression ratio that is 160 megabytes/second of output. > Enough to saturate a spinning disk, but not an SSD. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-6809) Compressed Commit Log
[ https://issues.apache.org/jira/browse/CASSANDRA-6809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ariel Weisberg updated CASSANDRA-6809: -- Attachment: ComitLogStress.java I made some improvements to ComitLogStress as part of my tests. * Compressible data is taken from a text file. I used alice in wonderland. * No shared state between load generating threads. * The commit log is given the opportunity to reclaim old segments. * Reports the average since the beginning. * Can specify a rate limit although it is per thread > Compressed Commit Log > - > > Key: CASSANDRA-6809 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6809 > Project: Cassandra > Issue Type: Improvement >Reporter: Benedict >Assignee: Branimir Lambov >Priority: Minor > Labels: performance > Fix For: 3.0 > > Attachments: ComitLogStress.java, logtest.txt > > > It seems an unnecessary oversight that we don't compress the commit log. > Doing so should improve throughput, but some care will need to be taken to > ensure we use as much of a segment as possible. I propose decoupling the > writing of the records from the segments. Basically write into a (queue of) > DirectByteBuffer, and have the sync thread compress, say, ~64K chunks every X > MB written to the CL (where X is ordinarily CLS size), and then pack as many > of the compressed chunks into a CLS as possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8630) Faster sequencial IO (on compaction, streaming, etc)
[
https://issues.apache.org/jira/browse/CASSANDRA-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279362#comment-14279362
]
Robert Stupp commented on CASSANDRA-8630:
-
[~m0nstermind] did you try the patch without the changes in
{{RandomAccessReader}} - i.e. just the changes to {{SequentialWriter}}? RAR is
already buffered - so the overhead reading primitives should be not that big as
indicated by the CPU graph. But SW seems expensive in 2.0 (SW is buffered in
2.1 + trunk - so seem not to be affected by this ticket).
> Faster sequencial IO (on compaction, streaming, etc)
>
>
> Key: CASSANDRA-8630
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8630
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Oleg Anastasyev
>Assignee: Oleg Anastasyev
> Labels: performance
> Attachments: 8630-FasterSequencialReadsAndWrites.txt, cpu_load.png
>
>
> When node is doing a lot of sequencial IO (streaming, compacting, etc) a lot
> of CPU is lost in calls to RAF's int read() and DataOutputStream's write(int).
> This is because default implementations of readShort,readLong, etc as well as
> their matching write* are implemented with numerous calls of byte by byte
> read and write.
> This makes a lot of syscalls as well.
> A quick microbench shows than just reimplementation of these methods in
> either way gives 8x speed increase.
> A patch attached implements RandomAccessReader.read and
> SequencialWriter.write methods in more efficient way.
> I also eliminated some extra byte copies in CompositeType.split and
> ColumnNameHelper.maxComponents, which were on my profiler's hotspot method
> list during tests.
> A stress tests on my laptop show that this patch makes compaction 25-30%
> faster on uncompressed sstables and 15% faster for compressed ones.
> A deployment to production shows much less CPU load for compaction.
> (I attached a cpu load graph from one of our production, orange is niced CPU
> load - i.e. compaction; yellow is user - i.e. not compaction related tasks)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-8634) Multi-thread commit log compression
Ariel Weisberg created CASSANDRA-8634: - Summary: Multi-thread commit log compression Key: CASSANDRA-8634 URL: https://issues.apache.org/jira/browse/CASSANDRA-8634 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Ariel Weisberg The initial commit log compression implementation can do about 240 megabytes/second (uncompressed input) on a fast (500 megabytes/sec) SSD. Assuming a 1.5x compression ratio that is 160 megabytes/second of output. Enough to saturate a spinning disk, but not an SSD. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8630) Faster sequencial IO (on compaction, streaming, etc)
[ https://issues.apache.org/jira/browse/CASSANDRA-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279354#comment-14279354 ] Oleg Anastasyev commented on CASSANDRA-8630: This does not help. They are final in java.io.RAF and java.io.DataInputStream. > Faster sequencial IO (on compaction, streaming, etc) > > > Key: CASSANDRA-8630 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8630 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Oleg Anastasyev >Assignee: Oleg Anastasyev > Labels: performance > Attachments: 8630-FasterSequencialReadsAndWrites.txt, cpu_load.png > > > When node is doing a lot of sequencial IO (streaming, compacting, etc) a lot > of CPU is lost in calls to RAF's int read() and DataOutputStream's write(int). > This is because default implementations of readShort,readLong, etc as well as > their matching write* are implemented with numerous calls of byte by byte > read and write. > This makes a lot of syscalls as well. > A quick microbench shows than just reimplementation of these methods in > either way gives 8x speed increase. > A patch attached implements RandomAccessReader.read and > SequencialWriter.write methods in more efficient way. > I also eliminated some extra byte copies in CompositeType.split and > ColumnNameHelper.maxComponents, which were on my profiler's hotspot method > list during tests. > A stress tests on my laptop show that this patch makes compaction 25-30% > faster on uncompressed sstables and 15% faster for compressed ones. > A deployment to production shows much less CPU load for compaction. > (I attached a cpu load graph from one of our production, orange is niced CPU > load - i.e. compaction; yellow is user - i.e. not compaction related tasks) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8630) Faster sequencial IO (on compaction, streaming, etc)
[ https://issues.apache.org/jira/browse/CASSANDRA-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279354#comment-14279354 ] Oleg Anastasyev edited comment on CASSANDRA-8630 at 1/15/15 9:55 PM: - This does not help. They are final in java.io.RAF and java.io.DataOutputStream. was (Author: m0nstermind): This does not help. They are final in java.io.RAF and java.io.DataInputStream. > Faster sequencial IO (on compaction, streaming, etc) > > > Key: CASSANDRA-8630 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8630 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Oleg Anastasyev >Assignee: Oleg Anastasyev > Labels: performance > Attachments: 8630-FasterSequencialReadsAndWrites.txt, cpu_load.png > > > When node is doing a lot of sequencial IO (streaming, compacting, etc) a lot > of CPU is lost in calls to RAF's int read() and DataOutputStream's write(int). > This is because default implementations of readShort,readLong, etc as well as > their matching write* are implemented with numerous calls of byte by byte > read and write. > This makes a lot of syscalls as well. > A quick microbench shows than just reimplementation of these methods in > either way gives 8x speed increase. > A patch attached implements RandomAccessReader.read and > SequencialWriter.write methods in more efficient way. > I also eliminated some extra byte copies in CompositeType.split and > ColumnNameHelper.maxComponents, which were on my profiler's hotspot method > list during tests. > A stress tests on my laptop show that this patch makes compaction 25-30% > faster on uncompressed sstables and 15% faster for compressed ones. > A deployment to production shows much less CPU load for compaction. > (I attached a cpu load graph from one of our production, orange is niced CPU > load - i.e. compaction; yellow is user - i.e. not compaction related tasks) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8622) All of pig-test is failing in trunk
[
https://issues.apache.org/jira/browse/CASSANDRA-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279348#comment-14279348
]
Brandon Williams commented on CASSANDRA-8622:
-
Actually, not quite... that fixes ECS, but now pig-test fails to find its
logger.
> All of pig-test is failing in trunk
> ---
>
> Key: CASSANDRA-8622
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8622
> Project: Cassandra
> Issue Type: Test
> Components: Hadoop
>Reporter: Philip Thompson
>Assignee: Brandon Williams
> Fix For: 3.0
>
>
> See http://cassci.datastax.com/job/trunk_pigtest/330/testReport/
> Every test in the ant target {{ant pig-test}} has been failing on trunk for a
> while now.
> {code}
> java.lang.ExceptionInInitializerError
> at org.apache.log4j.Logger.getLogger(Logger.java:40)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:48)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:44)
> at org.hyperic.sigar.SigarLog.debug(SigarLog.java:60)
> at org.hyperic.sigar.Sigar.(Sigar.java:108)
> at org.apache.cassandra.utils.SigarLibrary.(SigarLibrary.java:45)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:206)
> at
> org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:408)
> at
> org.apache.cassandra.service.EmbeddedCassandraService.start(EmbeddedCassandraService.java:52)
> at
> org.apache.cassandra.pig.PigTestBase.startCassandra(PigTestBase.java:96)
> at
> org.apache.cassandra.pig.CqlRecordReaderTest.setup(CqlRecordReaderTest.java:63)
> at
> org.apache.log4j.Log4jLoggerFactory.(Log4jLoggerFactory.java:50)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8622) All of pig-test is failing in trunk
[
https://issues.apache.org/jira/browse/CASSANDRA-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279342#comment-14279342
]
Brandon Williams commented on CASSANDRA-8622:
-
Removing {noformat}build/lib/jars/slf4j-log4j12-1.7.2.jar{noformat} solves this
problem. We may need to build with this jar, but I don't see a reason to run
with it since it creates multiple slf4j bindings and we have logback.
> All of pig-test is failing in trunk
> ---
>
> Key: CASSANDRA-8622
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8622
> Project: Cassandra
> Issue Type: Test
> Components: Hadoop
>Reporter: Philip Thompson
>Assignee: Brandon Williams
> Fix For: 3.0
>
>
> See http://cassci.datastax.com/job/trunk_pigtest/330/testReport/
> Every test in the ant target {{ant pig-test}} has been failing on trunk for a
> while now.
> {code}
> java.lang.ExceptionInInitializerError
> at org.apache.log4j.Logger.getLogger(Logger.java:40)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:48)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:44)
> at org.hyperic.sigar.SigarLog.debug(SigarLog.java:60)
> at org.hyperic.sigar.Sigar.(Sigar.java:108)
> at org.apache.cassandra.utils.SigarLibrary.(SigarLibrary.java:45)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:206)
> at
> org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:408)
> at
> org.apache.cassandra.service.EmbeddedCassandraService.start(EmbeddedCassandraService.java:52)
> at
> org.apache.cassandra.pig.PigTestBase.startCassandra(PigTestBase.java:96)
> at
> org.apache.cassandra.pig.CqlRecordReaderTest.setup(CqlRecordReaderTest.java:63)
> at
> org.apache.log4j.Log4jLoggerFactory.(Log4jLoggerFactory.java:50)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8630) Faster sequencial IO (on compaction, streaming, etc)
[ https://issues.apache.org/jira/browse/CASSANDRA-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279335#comment-14279335 ] Benedict commented on CASSANDRA-8630: - I have access to a backspace key ;) > Faster sequencial IO (on compaction, streaming, etc) > > > Key: CASSANDRA-8630 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8630 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Oleg Anastasyev >Assignee: Oleg Anastasyev > Labels: performance > Attachments: 8630-FasterSequencialReadsAndWrites.txt, cpu_load.png > > > When node is doing a lot of sequencial IO (streaming, compacting, etc) a lot > of CPU is lost in calls to RAF's int read() and DataOutputStream's write(int). > This is because default implementations of readShort,readLong, etc as well as > their matching write* are implemented with numerous calls of byte by byte > read and write. > This makes a lot of syscalls as well. > A quick microbench shows than just reimplementation of these methods in > either way gives 8x speed increase. > A patch attached implements RandomAccessReader.read and > SequencialWriter.write methods in more efficient way. > I also eliminated some extra byte copies in CompositeType.split and > ColumnNameHelper.maxComponents, which were on my profiler's hotspot method > list during tests. > A stress tests on my laptop show that this patch makes compaction 25-30% > faster on uncompressed sstables and 15% faster for compressed ones. > A deployment to production shows much less CPU load for compaction. > (I attached a cpu load graph from one of our production, orange is niced CPU > load - i.e. compaction; yellow is user - i.e. not compaction related tasks) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8630) Faster sequencial IO (on compaction, streaming, etc)
[ https://issues.apache.org/jira/browse/CASSANDRA-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279331#comment-14279331 ] Oleg Anastasyev commented on CASSANDRA-8630: Sure, but you cannot override them. They are final. > Faster sequencial IO (on compaction, streaming, etc) > > > Key: CASSANDRA-8630 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8630 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Oleg Anastasyev >Assignee: Oleg Anastasyev > Labels: performance > Attachments: 8630-FasterSequencialReadsAndWrites.txt, cpu_load.png > > > When node is doing a lot of sequencial IO (streaming, compacting, etc) a lot > of CPU is lost in calls to RAF's int read() and DataOutputStream's write(int). > This is because default implementations of readShort,readLong, etc as well as > their matching write* are implemented with numerous calls of byte by byte > read and write. > This makes a lot of syscalls as well. > A quick microbench shows than just reimplementation of these methods in > either way gives 8x speed increase. > A patch attached implements RandomAccessReader.read and > SequencialWriter.write methods in more efficient way. > I also eliminated some extra byte copies in CompositeType.split and > ColumnNameHelper.maxComponents, which were on my profiler's hotspot method > list during tests. > A stress tests on my laptop show that this patch makes compaction 25-30% > faster on uncompressed sstables and 15% faster for compressed ones. > A deployment to production shows much less CPU load for compaction. > (I attached a cpu load graph from one of our production, orange is niced CPU > load - i.e. compaction; yellow is user - i.e. not compaction related tasks) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8633) cassandra-stress writes don't respect 'select' parameter
[
https://issues.apache.org/jira/browse/CASSANDRA-8633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279285#comment-14279285
]
Aleksey Yeschenko commented on CASSANDRA-8633:
--
W/ the attached partition4.yaml (1/100/10K) stress just consumes all the cores
w/ generating the batches, and getting nowhere.
> cassandra-stress writes don't respect 'select' parameter
>
>
> Key: CASSANDRA-8633
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8633
> Project: Cassandra
> Issue Type: Bug
>Reporter: Aleksey Yeschenko
>Assignee: Benedict
> Fix For: 2.1.3
>
> Attachments: partition1.yaml, partition2.yaml, partition3.yaml,
> partition4.yaml
>
>
> W/ the attached profile (partition1.yaml), which has 3 clustering columns,
> each w/ clustering: fixed(100), and select: fixed(1)/1M, stress generates
> huge batches (whereas should only generate single-row mutations).
> {noformat}
> WARN 20:37:59 Batch of prepared statements for [extreme.extreme] is of size
> 269973, exceeding specified threshold of 5120 by 264853.
> {noformat}
> It can be better or worth depending on the distribution of those clustering-s
> (w/ product fixed at 1M).
> W/ 10/1000/100 (partition2.yaml):
> {noformat}
> WARN 20:47:22 Batch of prepared statements for [extreme.extreme] is of size
> 1769445, exceeding specified threshold of 5120 by 1764325.
> {noformat}
> W/ 10k/100/1 (partition3.yaml):
> {noformat}
> WARN 20:50:19 Batch of prepared statements for [extreme.extreme] is of size
> 5373, exceeding specified threshold of 5120 by 253.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8633) cassandra-stress writes don't respect 'select' parameter
[
https://issues.apache.org/jira/browse/CASSANDRA-8633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksey Yeschenko updated CASSANDRA-8633:
-
Attachment: partition4.yaml
> cassandra-stress writes don't respect 'select' parameter
>
>
> Key: CASSANDRA-8633
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8633
> Project: Cassandra
> Issue Type: Bug
>Reporter: Aleksey Yeschenko
>Assignee: Benedict
> Fix For: 2.1.3
>
> Attachments: partition1.yaml, partition2.yaml, partition3.yaml,
> partition4.yaml
>
>
> W/ the attached profile (partition1.yaml), which has 3 clustering columns,
> each w/ clustering: fixed(100), and select: fixed(1)/1M, stress generates
> huge batches (whereas should only generate single-row mutations).
> {noformat}
> WARN 20:37:59 Batch of prepared statements for [extreme.extreme] is of size
> 269973, exceeding specified threshold of 5120 by 264853.
> {noformat}
> It can be better or worth depending on the distribution of those clustering-s
> (w/ product fixed at 1M).
> W/ 10/1000/100 (partition2.yaml):
> {noformat}
> WARN 20:47:22 Batch of prepared statements for [extreme.extreme] is of size
> 1769445, exceeding specified threshold of 5120 by 1764325.
> {noformat}
> W/ 10k/100/1 (partition3.yaml):
> {noformat}
> WARN 20:50:19 Batch of prepared statements for [extreme.extreme] is of size
> 5373, exceeding specified threshold of 5120 by 253.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8633) cassandra-stress writes don't respect 'select' parameter
[
https://issues.apache.org/jira/browse/CASSANDRA-8633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksey Yeschenko updated CASSANDRA-8633:
-
Attachment: partition1.yaml
> cassandra-stress writes don't respect 'select' parameter
>
>
> Key: CASSANDRA-8633
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8633
> Project: Cassandra
> Issue Type: Bug
>Reporter: Aleksey Yeschenko
>Assignee: Benedict
> Fix For: 2.1.3
>
> Attachments: partition1.yaml, partition2.yaml, partition3.yaml
>
>
> W/ the attached profile (partition1.yaml), which has 3 clustering columns,
> each w/ clustering: fixed(100), and select: fixed(1)/1M, stress generates
> huge batches (whereas should only generate single-row mutations).
> {noformat}
> WARN 20:37:59 Batch of prepared statements for [extreme.extreme] is of size
> 269973, exceeding specified threshold of 5120 by 264853.
> {noformat}
> It can be better or worth depending on the distribution of those clustering-s
> (w/ product fixed at 1M).
> W/ 10/1000/100 (partition2.yaml):
> {noformat}
> WARN 20:47:22 Batch of prepared statements for [extreme.extreme] is of size
> 1769445, exceeding specified threshold of 5120 by 1764325.
> {noformat}
> W/ 10k/100/1 (partition3.yaml):
> {noformat}
> WARN 20:50:19 Batch of prepared statements for [extreme.extreme] is of size
> 5373, exceeding specified threshold of 5120 by 253.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8633) cassandra-stress writes don't respect 'select' parameter
[
https://issues.apache.org/jira/browse/CASSANDRA-8633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksey Yeschenko updated CASSANDRA-8633:
-
Attachment: partition2.yaml
> cassandra-stress writes don't respect 'select' parameter
>
>
> Key: CASSANDRA-8633
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8633
> Project: Cassandra
> Issue Type: Bug
>Reporter: Aleksey Yeschenko
>Assignee: Benedict
> Fix For: 2.1.3
>
> Attachments: partition1.yaml, partition2.yaml, partition3.yaml
>
>
> W/ the attached profile (partition1.yaml), which has 3 clustering columns,
> each w/ clustering: fixed(100), and select: fixed(1)/1M, stress generates
> huge batches (whereas should only generate single-row mutations).
> {noformat}
> WARN 20:37:59 Batch of prepared statements for [extreme.extreme] is of size
> 269973, exceeding specified threshold of 5120 by 264853.
> {noformat}
> It can be better or worth depending on the distribution of those clustering-s
> (w/ product fixed at 1M).
> W/ 10/1000/100 (partition2.yaml):
> {noformat}
> WARN 20:47:22 Batch of prepared statements for [extreme.extreme] is of size
> 1769445, exceeding specified threshold of 5120 by 1764325.
> {noformat}
> W/ 10k/100/1 (partition3.yaml):
> {noformat}
> WARN 20:50:19 Batch of prepared statements for [extreme.extreme] is of size
> 5373, exceeding specified threshold of 5120 by 253.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8633) cassandra-stress writes don't respect 'select' parameter
[
https://issues.apache.org/jira/browse/CASSANDRA-8633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksey Yeschenko updated CASSANDRA-8633:
-
Attachment: partition3.yaml
> cassandra-stress writes don't respect 'select' parameter
>
>
> Key: CASSANDRA-8633
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8633
> Project: Cassandra
> Issue Type: Bug
>Reporter: Aleksey Yeschenko
>Assignee: Benedict
> Fix For: 2.1.3
>
> Attachments: partition1.yaml, partition2.yaml, partition3.yaml
>
>
> W/ the attached profile (partition1.yaml), which has 3 clustering columns,
> each w/ clustering: fixed(100), and select: fixed(1)/1M, stress generates
> huge batches (whereas should only generate single-row mutations).
> {noformat}
> WARN 20:37:59 Batch of prepared statements for [extreme.extreme] is of size
> 269973, exceeding specified threshold of 5120 by 264853.
> {noformat}
> It can be better or worth depending on the distribution of those clustering-s
> (w/ product fixed at 1M).
> W/ 10/1000/100 (partition2.yaml):
> {noformat}
> WARN 20:47:22 Batch of prepared statements for [extreme.extreme] is of size
> 1769445, exceeding specified threshold of 5120 by 1764325.
> {noformat}
> W/ 10k/100/1 (partition3.yaml):
> {noformat}
> WARN 20:50:19 Batch of prepared statements for [extreme.extreme] is of size
> 5373, exceeding specified threshold of 5120 by 253.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-8633) cassandra-stress writes don't respect 'select' parameter
Aleksey Yeschenko created CASSANDRA-8633:
Summary: cassandra-stress writes don't respect 'select' parameter
Key: CASSANDRA-8633
URL: https://issues.apache.org/jira/browse/CASSANDRA-8633
Project: Cassandra
Issue Type: Bug
Reporter: Aleksey Yeschenko
Assignee: Benedict
Fix For: 2.1.3
W/ the attached profile (partition1.yaml), which has 3 clustering columns, each
w/ clustering: fixed(100), and select: fixed(1)/1M, stress generates huge
batches (whereas should only generate single-row mutations).
{noformat}
WARN 20:37:59 Batch of prepared statements for [extreme.extreme] is of size
269973, exceeding specified threshold of 5120 by 264853.
{noformat}
It can be better or worth depending on the distribution of those clustering-s
(w/ product fixed at 1M).
W/ 10/1000/100 (partition2.yaml):
{noformat}
WARN 20:47:22 Batch of prepared statements for [extreme.extreme] is of size
1769445, exceeding specified threshold of 5120 by 1764325.
{noformat}
W/ 10k/100/1 (partition3.yaml):
{noformat}
WARN 20:50:19 Batch of prepared statements for [extreme.extreme] is of size
5373, exceeding specified threshold of 5120 by 253.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-8632) cassandra-stress only generating a single unique row
[
https://issues.apache.org/jira/browse/CASSANDRA-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benedict resolved CASSANDRA-8632.
-
Resolution: Duplicate
Already found based off your IRC report and filed a bug fix, which is under
review.
> cassandra-stress only generating a single unique row
>
>
> Key: CASSANDRA-8632
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8632
> Project: Cassandra
> Issue Type: Bug
>Reporter: Aleksey Yeschenko
>Assignee: Benedict
> Fix For: 2.1.3
>
> Attachments: partition.yaml
>
>
> Most likely caused by CASSANDRA-7964, since 2.1.2 is fine, and only 2.1-head
> is broken.
> Using the attached profile, and "user profile=partition.yaml ops\(insert=1\)
> n=100 -rate threads=50" as the command, I'm only getting a single row
> as the result:
> {noformat}
> cqlsh:extreme> select * from extreme;
> partition_key | clustering_col0 | clustering_col1 | clustering_col2 | val
> ---+-+-+-+-
> 1 | -1024190305 | 840771439 | -43718277 |
> -1410971548
> (1 rows)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8632) cassandra-stress only generating a single unique row
[
https://issues.apache.org/jira/browse/CASSANDRA-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksey Yeschenko updated CASSANDRA-8632:
-
Attachment: partition.yaml
> cassandra-stress only generating a single unique row
>
>
> Key: CASSANDRA-8632
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8632
> Project: Cassandra
> Issue Type: Bug
>Reporter: Aleksey Yeschenko
>Assignee: Benedict
> Fix For: 2.1.3
>
> Attachments: partition.yaml
>
>
> Most likely caused by CASSANDRA-7964, since 2.1.2 is fine, and only 2.1-head
> is broken.
> Using the attached profile, and "user profile=partition.yaml ops\(insert=1\)
> n=100 -rate threads=50" as the command, I'm only getting a single row
> as the result:
> {noformat}
> cqlsh:extreme> select * from extreme;
> partition_key | clustering_col0 | clustering_col1 | clustering_col2 | val
> ---+-+-+-+-
> 1 | -1024190305 | 840771439 | -43718277 |
> -1410971548
> (1 rows)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8464) Support direct buffer decompression for reads
[ https://issues.apache.org/jira/browse/CASSANDRA-8464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279256#comment-14279256 ] T Jake Luciani commented on CASSANDRA-8464: --- Rebased and fixed nits. bq. Does it work with DeflateCompressor now? Yes. It copies the input buffer on heap if it's mmapped. > Support direct buffer decompression for reads > - > > Key: CASSANDRA-8464 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8464 > Project: Cassandra > Issue Type: Improvement >Reporter: T Jake Luciani >Assignee: T Jake Luciani > Labels: performance > Fix For: 3.0 > > Attachments: compression_direct.png > > > Currently when we read a compressed sstable we copy the data on heap then > send it to be de-compressed to another on heap buffer (albeit pooled). > But now both snappy and lz4 (with CASSANDRA-7039) allow decompression of > direct byte buffers. This lets us mmap the data and decompress completely > off heap (and avoids moving bytes over JNI). > One issue is performing the checksum offheap but the Adler32 does support in > java 8 (it's also in java 7 but marked private?!) > This change yields a > 10% boost in read performance on cstar. Locally I see > upto 30% improvement. > http://cstar.datastax.com/graph?stats=5ebcdd70-816b-11e4-aed6-42010af0688f&metric=op_rate&operation=2_read&smoothing=1&show_aggregates=true&xmin=0&xmax=200.09&ymin=0&ymax=135908.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-8632) cassandra-stress only generating a single unique row
Aleksey Yeschenko created CASSANDRA-8632:
Summary: cassandra-stress only generating a single unique row
Key: CASSANDRA-8632
URL: https://issues.apache.org/jira/browse/CASSANDRA-8632
Project: Cassandra
Issue Type: Bug
Reporter: Aleksey Yeschenko
Assignee: Benedict
Fix For: 2.1.3
Most likely caused by CASSANDRA-7964, since 2.1.2 is fine, and only 2.1-head is
broken.
Using the attached profile, and "user profile=partition.yaml ops\(insert=1\)
n=100 -rate threads=50" as the command, I'm only getting a single row
as the result:
{noformat}
cqlsh:extreme> select * from extreme;
partition_key | clustering_col0 | clustering_col1 | clustering_col2 | val
---+-+-+-+-
1 | -1024190305 | 840771439 | -43718277 |
-1410971548
(1 rows)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8630) Faster sequencial IO (on compaction, streaming, etc)
[ https://issues.apache.org/jira/browse/CASSANDRA-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Shuler updated CASSANDRA-8630: -- Labels: performance (was: ) > Faster sequencial IO (on compaction, streaming, etc) > > > Key: CASSANDRA-8630 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8630 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Oleg Anastasyev >Assignee: Oleg Anastasyev > Labels: performance > Attachments: 8630-FasterSequencialReadsAndWrites.txt, cpu_load.png > > > When node is doing a lot of sequencial IO (streaming, compacting, etc) a lot > of CPU is lost in calls to RAF's int read() and DataOutputStream's write(int). > This is because default implementations of readShort,readLong, etc as well as > their matching write* are implemented with numerous calls of byte by byte > read and write. > This makes a lot of syscalls as well. > A quick microbench shows than just reimplementation of these methods in > either way gives 8x speed increase. > A patch attached implements RandomAccessReader.read and > SequencialWriter.write methods in more efficient way. > I also eliminated some extra byte copies in CompositeType.split and > ColumnNameHelper.maxComponents, which were on my profiler's hotspot method > list during tests. > A stress tests on my laptop show that this patch makes compaction 25-30% > faster on uncompressed sstables and 15% faster for compressed ones. > A deployment to production shows much less CPU load for compaction. > (I attached a cpu load graph from one of our production, orange is niced CPU > load - i.e. compaction; yellow is user - i.e. not compaction related tasks) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8631) long-test MeteredFlusherTest fails with heap OOM
[
https://issues.apache.org/jira/browse/CASSANDRA-8631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Shuler updated CASSANDRA-8631:
--
Component/s: Tests
> long-test MeteredFlusherTest fails with heap OOM
>
>
> Key: CASSANDRA-8631
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8631
> Project: Cassandra
> Issue Type: Test
> Components: Tests
> Environment: cassandra-2.0 commit 90780b5
>Reporter: Michael Shuler
>Priority: Minor
> Labels: unit-test
>
> I attempted to find a spot where this test actually passes, and I was unable
> to do so. I tried back to 1.1 HEAD and various release tags under jdk7 and
> jdk6 for <= 1.2 tags. In 1.2.0 with jdk6, the test times out with
> java.lang.OutOfMemoryError: GC overhead limit exceeded, so it's similar.
> {noformat}
> [junit] Testcase:
> testManyMemtables(org.apache.cassandra.db.MeteredFlusherTest):Caused an
> ERROR
> [junit] Java heap space
> [junit] java.lang.OutOfMemoryError: Java heap space
> [junit] at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
> [junit] at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
> [junit] at
> org.apache.cassandra.utils.SlabAllocator$Region.(SlabAllocator.java:157)
> [junit] at
> org.apache.cassandra.utils.SlabAllocator$Region.(SlabAllocator.java:131)
> [junit] at
> org.apache.cassandra.utils.SlabAllocator.getRegion(SlabAllocator.java:101)
> [junit] at
> org.apache.cassandra.utils.SlabAllocator.allocate(SlabAllocator.java:73)
> [junit] at
> org.apache.cassandra.utils.Allocator.clone(Allocator.java:30)
> [junit] at org.apache.cassandra.db.Column.localCopy(Column.java:277)
> [junit] at org.apache.cassandra.db.Memtable$1.apply(Memtable.java:114)
> [junit] at org.apache.cassandra.db.Memtable$1.apply(Memtable.java:111)
> [junit] at
> org.apache.cassandra.db.AtomicSortedColumns.addAllWithSizeDelta(AtomicSortedColumns.java:194)
> [junit] at org.apache.cassandra.db.Memtable.resolve(Memtable.java:218)
> [junit] at org.apache.cassandra.db.Memtable.put(Memtable.java:165)
> [junit] at
> org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:901)
> [junit] at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:374)
> [junit] at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:339)
> [junit] at
> org.apache.cassandra.db.RowMutation.applyUnsafe(RowMutation.java:216)
> [junit] at
> org.apache.cassandra.db.MeteredFlusherTest.testManyMemtables(MeteredFlusherTest.java:59)
> [junit]
> [junit]
> [junit] Test org.apache.cassandra.db.MeteredFlusherTest FAILED
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-8631) long-test MeteredFlusherTest fails with heap OOM
Michael Shuler created CASSANDRA-8631:
-
Summary: long-test MeteredFlusherTest fails with heap OOM
Key: CASSANDRA-8631
URL: https://issues.apache.org/jira/browse/CASSANDRA-8631
Project: Cassandra
Issue Type: Test
Environment: cassandra-2.0 commit 90780b5
Reporter: Michael Shuler
Priority: Minor
I attempted to find a spot where this test actually passes, and I was unable to
do so. I tried back to 1.1 HEAD and various release tags under jdk7 and jdk6
for <= 1.2 tags. In 1.2.0 with jdk6, the test times out with
java.lang.OutOfMemoryError: GC overhead limit exceeded, so it's similar.
{noformat}
[junit] Testcase:
testManyMemtables(org.apache.cassandra.db.MeteredFlusherTest):Caused an
ERROR
[junit] Java heap space
[junit] java.lang.OutOfMemoryError: Java heap space
[junit] at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
[junit] at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
[junit] at
org.apache.cassandra.utils.SlabAllocator$Region.(SlabAllocator.java:157)
[junit] at
org.apache.cassandra.utils.SlabAllocator$Region.(SlabAllocator.java:131)
[junit] at
org.apache.cassandra.utils.SlabAllocator.getRegion(SlabAllocator.java:101)
[junit] at
org.apache.cassandra.utils.SlabAllocator.allocate(SlabAllocator.java:73)
[junit] at org.apache.cassandra.utils.Allocator.clone(Allocator.java:30)
[junit] at org.apache.cassandra.db.Column.localCopy(Column.java:277)
[junit] at org.apache.cassandra.db.Memtable$1.apply(Memtable.java:114)
[junit] at org.apache.cassandra.db.Memtable$1.apply(Memtable.java:111)
[junit] at
org.apache.cassandra.db.AtomicSortedColumns.addAllWithSizeDelta(AtomicSortedColumns.java:194)
[junit] at org.apache.cassandra.db.Memtable.resolve(Memtable.java:218)
[junit] at org.apache.cassandra.db.Memtable.put(Memtable.java:165)
[junit] at
org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:901)
[junit] at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:374)
[junit] at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:339)
[junit] at
org.apache.cassandra.db.RowMutation.applyUnsafe(RowMutation.java:216)
[junit] at
org.apache.cassandra.db.MeteredFlusherTest.testManyMemtables(MeteredFlusherTest.java:59)
[junit]
[junit]
[junit] Test org.apache.cassandra.db.MeteredFlusherTest FAILED
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7321) retire jython paging tests and port over to python driver
[ https://issues.apache.org/jira/browse/CASSANDRA-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279222#comment-14279222 ] Russ Hatch commented on CASSANDRA-7321: --- These were ported to cassandra-dtest a while back. Old jenkins jobs have been disabled as the paging tests now run as part of the main cassandra-dtest runs. > retire jython paging tests and port over to python driver > - > > Key: CASSANDRA-7321 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7321 > Project: Cassandra > Issue Type: Test >Reporter: Russ Hatch >Assignee: Russ Hatch > > The latest version of the python driver now supports paging, so we have no > need to test paging with jython anymore. > We should either create another paging test project to supplant > https://github.com/riptano/cassandra-dtest-jython, or better yet get these > tests incorporated into cassandra-dtest since we are soon going to port dtest > over to the new python driver (away from dbapi2). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8630) Faster sequencial IO (on compaction, streaming, etc)
[ https://issues.apache.org/jira/browse/CASSANDRA-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279113#comment-14279113 ] Benedict commented on CASSANDRA-8630: - Ha. I'm kind of amazed we've not spotted this before. I'm not convinced by the exact tack you've taken, but this should definitely be fixed. The best fix is probably to explicitly override getLong, getInt, etc, since all of the readers have a buffer that can be used (if it crosses a buffer boundary, we can simply rebuffer slightly less than a full buffer), thus keeping garbage to a minimum > Faster sequencial IO (on compaction, streaming, etc) > > > Key: CASSANDRA-8630 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8630 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Oleg Anastasyev > Attachments: 8630-FasterSequencialReadsAndWrites.txt, cpu_load.png > > > When node is doing a lot of sequencial IO (streaming, compacting, etc) a lot > of CPU is lost in calls to RAF's int read() and DataOutputStream's write(int). > This is because default implementations of readShort,readLong, etc as well as > their matching write* are implemented with numerous calls of byte by byte > read and write. > This makes a lot of syscalls as well. > A quick microbench shows than just reimplementation of these methods in > either way gives 8x speed increase. > A patch attached implements RandomAccessReader.read and > SequencialWriter.write methods in more efficient way. > I also eliminated some extra byte copies in CompositeType.split and > ColumnNameHelper.maxComponents, which were on my profiler's hotspot method > list during tests. > A stress tests on my laptop show that this patch makes compaction 25-30% > faster on uncompressed sstables and 15% faster for compressed ones. > A deployment to production shows much less CPU load for compaction. > (I attached a cpu load graph from one of our production, orange is niced CPU > load - i.e. compaction; yellow is user - i.e. not compaction related tasks) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8630) Faster sequencial IO (on compaction, streaming, etc)
[ https://issues.apache.org/jira/browse/CASSANDRA-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-8630: Assignee: Oleg Anastasyev > Faster sequencial IO (on compaction, streaming, etc) > > > Key: CASSANDRA-8630 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8630 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Oleg Anastasyev >Assignee: Oleg Anastasyev > Attachments: 8630-FasterSequencialReadsAndWrites.txt, cpu_load.png > > > When node is doing a lot of sequencial IO (streaming, compacting, etc) a lot > of CPU is lost in calls to RAF's int read() and DataOutputStream's write(int). > This is because default implementations of readShort,readLong, etc as well as > their matching write* are implemented with numerous calls of byte by byte > read and write. > This makes a lot of syscalls as well. > A quick microbench shows than just reimplementation of these methods in > either way gives 8x speed increase. > A patch attached implements RandomAccessReader.read and > SequencialWriter.write methods in more efficient way. > I also eliminated some extra byte copies in CompositeType.split and > ColumnNameHelper.maxComponents, which were on my profiler's hotspot method > list during tests. > A stress tests on my laptop show that this patch makes compaction 25-30% > faster on uncompressed sstables and 15% faster for compressed ones. > A deployment to production shows much less CPU load for compaction. > (I attached a cpu load graph from one of our production, orange is niced CPU > load - i.e. compaction; yellow is user - i.e. not compaction related tasks) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8630) Faster sequencial IO (on compaction, streaming, etc)
[ https://issues.apache.org/jira/browse/CASSANDRA-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Anastasyev updated CASSANDRA-8630: --- Description: When node is doing a lot of sequencial IO (streaming, compacting, etc) a lot of CPU is lost in calls to RAF's int read() and DataOutputStream's write(int). This is because default implementations of readShort,readLong, etc as well as their matching write* are implemented with numerous calls of byte by byte read and write. This makes a lot of syscalls as well. A quick microbench shows than just reimplementation of these methods in either way gives 8x speed increase. A patch attached implements RandomAccessReader.read and SequencialWriter.write methods in more efficient way. I also eliminated some extra byte copies in CompositeType.split and ColumnNameHelper.maxComponents, which were on my profiler's hotspot method list during tests. A stress tests on my laptop show that this patch makes compaction 25-30% faster on uncompressed sstables and 15% faster for compressed ones. A deployment to production shows much less CPU load for compaction. (I attached a cpu load graph from one of our production, orange is niced CPU load - i.e. compaction; yellow is user - i.e. not compaction related tasks) was: When node is doing a lot of sequencial IO (streaming, compacting, etc) a lot of CPU is lost in calls to RAF's int read() and DataOutputStream's write(int). This is because default implementations of readShort,readLong, etc as well as their matching write* are implemented with numerous calls of byte by byte read and write. This makes a lot of syscalls as well. A quick microbench shows than just reimplementation of these methods in either way gives 8x speed increase. A patch attached implements RandomAccessReader.read and SequencialWriter.write methods in more efficient way. I also eliminated some extra byte copies in CompositeType.split and ColumnNameHelper.maxComponents, which was on my profiler's hotspot method list during tests. A stress tests on my laptop show that this patch makes compaction 25-30% faster on uncompressed sstables and 15% faster for compressed ones. A deployment to production shows much less CPU load for compaction. (I attached a cpu load graph from one of our production, orange is niced CPU load - i.e. compaction; yellow is user - i.e. not compaction related tasks) > Faster sequencial IO (on compaction, streaming, etc) > > > Key: CASSANDRA-8630 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8630 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Oleg Anastasyev > Attachments: 8630-FasterSequencialReadsAndWrites.txt, cpu_load.png > > > When node is doing a lot of sequencial IO (streaming, compacting, etc) a lot > of CPU is lost in calls to RAF's int read() and DataOutputStream's write(int). > This is because default implementations of readShort,readLong, etc as well as > their matching write* are implemented with numerous calls of byte by byte > read and write. > This makes a lot of syscalls as well. > A quick microbench shows than just reimplementation of these methods in > either way gives 8x speed increase. > A patch attached implements RandomAccessReader.read and > SequencialWriter.write methods in more efficient way. > I also eliminated some extra byte copies in CompositeType.split and > ColumnNameHelper.maxComponents, which were on my profiler's hotspot method > list during tests. > A stress tests on my laptop show that this patch makes compaction 25-30% > faster on uncompressed sstables and 15% faster for compressed ones. > A deployment to production shows much less CPU load for compaction. > (I attached a cpu load graph from one of our production, orange is niced CPU > load - i.e. compaction; yellow is user - i.e. not compaction related tasks) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8630) Faster sequencial IO (on compaction, streaming, etc)
[ https://issues.apache.org/jira/browse/CASSANDRA-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Anastasyev updated CASSANDRA-8630: --- Description: When node is doing a lot of sequencial IO (streaming, compacting, etc) a lot of CPU is lost in calls to RAF's int read() and DataOutputStream's write(int). This is because default implementations of readShort,readLong, etc as well as their matching write* are implemented with numerous calls of byte by byte read and write. This makes a lot of syscalls as well. A quick microbench shows than just reimplementation of these methods in either way gives 8x speed increase. A patch attached implements RandomAccessReader.read and SequencialWriter.write methods in more efficient way. I also eliminated some extra byte copies in CompositeType.split and ColumnNameHelper.maxComponents, which was on my profiler's hotspot method list during tests. A stress tests on my laptop show that this patch makes compaction 25-30% faster on uncompressed sstables and 15% faster for compressed ones. A deployment to production shows much less CPU load for compaction. (I attached a cpu load graph from one of our production, orange is niced CPU load - i.e. compaction; yellow is user - i.e. not compaction related tasks) was: When node is doing a lot of sequencial IO (streaming, compacting, etc) a lot of CPU is lost in calls to RAF's int read() and DataOutputStream's write(int). This is because default implementations of readShort,readLong, etc as well as their matching write* are implemented with numerous calls of byte by byte read and write. This makes a lot of syscalls as well. A quick microbench shows than just reimplementation of these methods in either way gives 8x speed increase. A patch attached implements RandomAccessReader.read and SequencialWriter.write methods in more efficient way. I also eliminated some extra byte copies in CompositeType.split and ColumnNameHelper.maxComponents, which was on my profiler's hotspots during tests. A stress tests on my laptop show that this patch makes compaction 25-30% faster on uncompressed sstables and 15% faster for compressed ones. A deployment to production shows much less CPU load for compaction. (I attached a cpu load graph from one of our production, orange is niced CPU load - i.e. compaction; yellow is user - i.e. not compaction related tasks) > Faster sequencial IO (on compaction, streaming, etc) > > > Key: CASSANDRA-8630 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8630 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Oleg Anastasyev > Attachments: 8630-FasterSequencialReadsAndWrites.txt, cpu_load.png > > > When node is doing a lot of sequencial IO (streaming, compacting, etc) a lot > of CPU is lost in calls to RAF's int read() and DataOutputStream's write(int). > This is because default implementations of readShort,readLong, etc as well as > their matching write* are implemented with numerous calls of byte by byte > read and write. > This makes a lot of syscalls as well. > A quick microbench shows than just reimplementation of these methods in > either way gives 8x speed increase. > A patch attached implements RandomAccessReader.read and > SequencialWriter.write methods in more efficient way. > I also eliminated some extra byte copies in CompositeType.split and > ColumnNameHelper.maxComponents, which was on my profiler's hotspot method > list during tests. > A stress tests on my laptop show that this patch makes compaction 25-30% > faster on uncompressed sstables and 15% faster for compressed ones. > A deployment to production shows much less CPU load for compaction. > (I attached a cpu load graph from one of our production, orange is niced CPU > load - i.e. compaction; yellow is user - i.e. not compaction related tasks) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8630) Faster sequencial IO (on compaction, streaming, etc)
[ https://issues.apache.org/jira/browse/CASSANDRA-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279106#comment-14279106 ] Oleg Anastasyev commented on CASSANDRA-8630: PS. We discussed this issue in London with marcus as related only to high CPU load during streaming, but it turned out than this patch could be applied for all sequencial io. > Faster sequencial IO (on compaction, streaming, etc) > > > Key: CASSANDRA-8630 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8630 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Oleg Anastasyev > Attachments: 8630-FasterSequencialReadsAndWrites.txt, cpu_load.png > > > When node is doing a lot of sequencial IO (streaming, compacting, etc) a lot > of CPU is lost in calls to RAF's int read() and DataOutputStream's write(int). > This is because default implementations of readShort,readLong, etc as well as > their matching write* are implemented with numerous calls of byte by byte > read and write. > This makes a lot of syscalls as well. > A quick microbench shows than just reimplementation of these methods in > either way gives 8x speed increase. > A patch attached implements RandomAccessReader.read and > SequencialWriter.write methods in more efficient way. > I also eliminated some extra byte copies in CompositeType.split and > ColumnNameHelper.maxComponents, which was on my profiler's hotspots during > tests. > A stress tests on my laptop show that this patch makes compaction 25-30% > faster on uncompressed sstables and 15% faster for compressed ones. > A deployment to production shows much less CPU load for compaction. > (I attached a cpu load graph from one of our production, orange is niced CPU > load - i.e. compaction; yellow is user - i.e. not compaction related tasks) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8630) Faster sequencial IO (on compaction, streaming, etc)
[ https://issues.apache.org/jira/browse/CASSANDRA-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Anastasyev updated CASSANDRA-8630: --- Attachment: cpu_load.png a cpu load graph. 2.0.53 arrow is deploy of patched version. > Faster sequencial IO (on compaction, streaming, etc) > > > Key: CASSANDRA-8630 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8630 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Oleg Anastasyev > Attachments: 8630-FasterSequencialReadsAndWrites.txt, cpu_load.png > > > When node is doing a lot of sequencial IO (streaming, compacting, etc) a lot > of CPU is lost in calls to RAF's int read() and DataOutputStream's write(int). > This is because default implementations of readShort,readLong, etc as well as > their matching write* are implemented with numerous calls of byte by byte > read and write. > This makes a lot of syscalls as well. > A quick microbench shows than just reimplementation of these methods in > either way gives 8x speed increase. > A patch attached implements RandomAccessReader.read and > SequencialWriter.write methods in more efficient way. > I also eliminated some extra byte copies in CompositeType.split and > ColumnNameHelper.maxComponents, which was on my profiler's hotspots during > tests. > A stress tests on my laptop show that this patch makes compaction 25-30% > faster on uncompressed sstables and 15% faster for compressed ones. > A deployment to production shows much less CPU load for compaction. > (I attached a cpu load graph from one of our production, orange is niced CPU > load - i.e. compaction; yellow is user - i.e. not compaction related tasks) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8630) Faster sequencial IO (on compaction, streaming, etc)
[ https://issues.apache.org/jira/browse/CASSANDRA-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279101#comment-14279101 ] Oleg Anastasyev edited comment on CASSANDRA-8630 at 1/15/15 6:52 PM: - attached a cpu load graph. 2.0.53 arrow is deploy of patched version. was (Author: m0nstermind): a cpu load graph. 2.0.53 arrow is deploy of patched version. > Faster sequencial IO (on compaction, streaming, etc) > > > Key: CASSANDRA-8630 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8630 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Oleg Anastasyev > Attachments: 8630-FasterSequencialReadsAndWrites.txt, cpu_load.png > > > When node is doing a lot of sequencial IO (streaming, compacting, etc) a lot > of CPU is lost in calls to RAF's int read() and DataOutputStream's write(int). > This is because default implementations of readShort,readLong, etc as well as > their matching write* are implemented with numerous calls of byte by byte > read and write. > This makes a lot of syscalls as well. > A quick microbench shows than just reimplementation of these methods in > either way gives 8x speed increase. > A patch attached implements RandomAccessReader.read and > SequencialWriter.write methods in more efficient way. > I also eliminated some extra byte copies in CompositeType.split and > ColumnNameHelper.maxComponents, which was on my profiler's hotspots during > tests. > A stress tests on my laptop show that this patch makes compaction 25-30% > faster on uncompressed sstables and 15% faster for compressed ones. > A deployment to production shows much less CPU load for compaction. > (I attached a cpu load graph from one of our production, orange is niced CPU > load - i.e. compaction; yellow is user - i.e. not compaction related tasks) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8630) Faster sequencial IO (on compaction, streaming, etc)
[ https://issues.apache.org/jira/browse/CASSANDRA-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279096#comment-14279096 ] Oleg Anastasyev edited comment on CASSANDRA-8630 at 1/15/15 6:52 PM: - attached a patch on cassandra-2.0 branch, commit id is a3234e191f661cfbbb27df748205fb1ac67101de was (Author: m0nstermind): a patch on cassandra-2.0 branch, commit id is a3234e191f661cfbbb27df748205fb1ac67101de > Faster sequencial IO (on compaction, streaming, etc) > > > Key: CASSANDRA-8630 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8630 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Oleg Anastasyev > Attachments: 8630-FasterSequencialReadsAndWrites.txt, cpu_load.png > > > When node is doing a lot of sequencial IO (streaming, compacting, etc) a lot > of CPU is lost in calls to RAF's int read() and DataOutputStream's write(int). > This is because default implementations of readShort,readLong, etc as well as > their matching write* are implemented with numerous calls of byte by byte > read and write. > This makes a lot of syscalls as well. > A quick microbench shows than just reimplementation of these methods in > either way gives 8x speed increase. > A patch attached implements RandomAccessReader.read and > SequencialWriter.write methods in more efficient way. > I also eliminated some extra byte copies in CompositeType.split and > ColumnNameHelper.maxComponents, which was on my profiler's hotspots during > tests. > A stress tests on my laptop show that this patch makes compaction 25-30% > faster on uncompressed sstables and 15% faster for compressed ones. > A deployment to production shows much less CPU load for compaction. > (I attached a cpu load graph from one of our production, orange is niced CPU > load - i.e. compaction; yellow is user - i.e. not compaction related tasks) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8630) Faster sequencial IO (on compaction, streaming, etc)
[ https://issues.apache.org/jira/browse/CASSANDRA-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Anastasyev updated CASSANDRA-8630: --- Attachment: 8630-FasterSequencialReadsAndWrites.txt a patch on cassandra-2.0 branch, commit id is a3234e191f661cfbbb27df748205fb1ac67101de > Faster sequencial IO (on compaction, streaming, etc) > > > Key: CASSANDRA-8630 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8630 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Oleg Anastasyev > Attachments: 8630-FasterSequencialReadsAndWrites.txt > > > When node is doing a lot of sequencial IO (streaming, compacting, etc) a lot > of CPU is lost in calls to RAF's int read() and DataOutputStream's write(int). > This is because default implementations of readShort,readLong, etc as well as > their matching write* are implemented with numerous calls of byte by byte > read and write. > This makes a lot of syscalls as well. > A quick microbench shows than just reimplementation of these methods in > either way gives 8x speed increase. > A patch attached implements RandomAccessReader.read and > SequencialWriter.write methods in more efficient way. > I also eliminated some extra byte copies in CompositeType.split and > ColumnNameHelper.maxComponents, which was on my profiler's hotspots during > tests. > A stress tests on my laptop show that this patch makes compaction 25-30% > faster on uncompressed sstables and 15% faster for compressed ones. > A deployment to production shows much less CPU load for compaction. > (I attached a cpu load graph from one of our production, orange is niced CPU > load - i.e. compaction; yellow is user - i.e. not compaction related tasks) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-8630) Faster sequencial IO (on compaction, streaming, etc)
Oleg Anastasyev created CASSANDRA-8630: -- Summary: Faster sequencial IO (on compaction, streaming, etc) Key: CASSANDRA-8630 URL: https://issues.apache.org/jira/browse/CASSANDRA-8630 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Oleg Anastasyev When node is doing a lot of sequencial IO (streaming, compacting, etc) a lot of CPU is lost in calls to RAF's int read() and DataOutputStream's write(int). This is because default implementations of readShort,readLong, etc as well as their matching write* are implemented with numerous calls of byte by byte read and write. This makes a lot of syscalls as well. A quick microbench shows than just reimplementation of these methods in either way gives 8x speed increase. A patch attached implements RandomAccessReader.read and SequencialWriter.write methods in more efficient way. I also eliminated some extra byte copies in CompositeType.split and ColumnNameHelper.maxComponents, which was on my profiler's hotspots during tests. A stress tests on my laptop show that this patch makes compaction 25-30% faster on uncompressed sstables and 15% faster for compressed ones. A deployment to production shows much less CPU load for compaction. (I attached a cpu load graph from one of our production, orange is niced CPU load - i.e. compaction; yellow is user - i.e. not compaction related tasks) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8622) All of pig-test is failing in trunk
[
https://issues.apache.org/jira/browse/CASSANDRA-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279067#comment-14279067
]
Brandon Williams commented on CASSANDRA-8622:
-
Well the good news is, if you check out f44110c7f (the parent of c832a36f25c)
and apply CASSANDRA-8577 everything passes, so we just need to figure out what
the deal is with c832a36f25c.
> All of pig-test is failing in trunk
> ---
>
> Key: CASSANDRA-8622
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8622
> Project: Cassandra
> Issue Type: Test
> Components: Hadoop
>Reporter: Philip Thompson
>Assignee: Brandon Williams
> Fix For: 3.0
>
>
> See http://cassci.datastax.com/job/trunk_pigtest/330/testReport/
> Every test in the ant target {{ant pig-test}} has been failing on trunk for a
> while now.
> {code}
> java.lang.ExceptionInInitializerError
> at org.apache.log4j.Logger.getLogger(Logger.java:40)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:48)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:44)
> at org.hyperic.sigar.SigarLog.debug(SigarLog.java:60)
> at org.hyperic.sigar.Sigar.(Sigar.java:108)
> at org.apache.cassandra.utils.SigarLibrary.(SigarLibrary.java:45)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:206)
> at
> org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:408)
> at
> org.apache.cassandra.service.EmbeddedCassandraService.start(EmbeddedCassandraService.java:52)
> at
> org.apache.cassandra.pig.PigTestBase.startCassandra(PigTestBase.java:96)
> at
> org.apache.cassandra.pig.CqlRecordReaderTest.setup(CqlRecordReaderTest.java:63)
> at
> org.apache.log4j.Log4jLoggerFactory.(Log4jLoggerFactory.java:50)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-8629) Exceptions in cassandra-stress
Aleksey Yeschenko created CASSANDRA-8629:
Summary: Exceptions in cassandra-stress
Key: CASSANDRA-8629
URL: https://issues.apache.org/jira/browse/CASSANDRA-8629
Project: Cassandra
Issue Type: Bug
Reporter: Aleksey Yeschenko
Assignee: Benedict
Priority: Trivial
cassandra-stress when run with tiny n, throws
org.apache.commons.math3.exception.NotStrictlyPositiveException.
Now, n=1 doesn't really make any sense, w/ 50k writes used just for warmup, but
an exception is still an exception. Labeled w/ priority: Trivial.
Profile used: http://pastebin.com/raw.php?i=9U5EMdVq
{noformat}
tools/bin/cassandra-stress user profile=partition.yaml ops\(insert=1\) n=1
-rate threads=50
INFO 18:21:59 Using data-center name 'datacenter1' for DCAwareRoundRobinPolicy
(if this is incorrect, please provide the correct datacenter name with
DCAwareRoundRobinPolicy constructor)
Connected to cluster: Test Cluster
Datatacenter: datacenter1; Host: localhost/127.0.0.1; Rack: rack1
INFO 18:21:59 New Cassandra host localhost/127.0.0.1:9042 added
Created schema. Sleeping 1s for propagation.
Exception in thread "main"
org.apache.commons.math3.exception.NotStrictlyPositiveException: standard
deviation (0)
at
org.apache.commons.math3.distribution.NormalDistribution.(NormalDistribution.java:108)
at
org.apache.cassandra.stress.settings.OptionDistribution$GaussianFactory.get(OptionDistribution.java:418)
at
org.apache.cassandra.stress.generate.SeedManager.(SeedManager.java:59)
at
org.apache.cassandra.stress.settings.SettingsCommandUser.getFactory(SettingsCommandUser.java:78)
at org.apache.cassandra.stress.StressAction.run(StressAction.java:61)
at org.apache.cassandra.stress.Stress.main(Stress.java:109)
{noformat}
On cassandra-2.1 HEAD, I cannot reproduce it, but get a different exception,
with n=10:
{noformat}
Exception in thread "Thread-13" java.lang.AssertionError
at
org.apache.cassandra.stress.util.DynamicList.remove(DynamicList.java:156)
at org.apache.cassandra.stress.generate.Seed.remove(Seed.java:83)
at
org.apache.cassandra.stress.generate.SeedManager.markLastWrite(SeedManager.java:115)
at
org.apache.cassandra.stress.generate.PartitionIterator$MultiRowIterator.setHasNext(PartitionIterator.java:561)
at
org.apache.cassandra.stress.generate.PartitionIterator$MultiRowIterator.seek(PartitionIterator.java:333)
at
org.apache.cassandra.stress.generate.PartitionIterator$MultiRowIterator.reset(PartitionIterator.java:242)
at
org.apache.cassandra.stress.generate.PartitionIterator.reset(PartitionIterator.java:99)
at org.apache.cassandra.stress.Operation.ready(Operation.java:110)
at
org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:288)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8596) Display datacenter/rack info for offline nodes - PropertyFileSnitch
[ https://issues.apache.org/jira/browse/CASSANDRA-8596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279059#comment-14279059 ] Vova commented on CASSANDRA-8596: - Actually I proposed this patch just as a hint. I think reading from database can be done at higher level, common for all snitches, not only for PropertyFileSnitch. > Display datacenter/rack info for offline nodes - PropertyFileSnitch > --- > > Key: CASSANDRA-8596 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8596 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Vova >Priority: Minor > Attachments: status.diff > > > When using GossipPropertyFileSnitch "nodetool status" shows default (from > cassandra-topology.properties ) datacenter/rack for offline nodes. > It happens because offline nodes are not in endpointMap, and thus > getRawEndpointInfo returns default DC/rack is returned > (PropertyFileSnitch.java). > I suggest to take info for those nodes from system.peers tables - just like > SELECT data_center,rack FROM system.peers WHERE peer='10.0.0.1' > Patch attached. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8622) All of pig-test is failing in trunk
[
https://issues.apache.org/jira/browse/CASSANDRA-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279022#comment-14279022
]
Brandon Williams commented on CASSANDRA-8622:
-
[~dbrosius] can you take a look at this?
> All of pig-test is failing in trunk
> ---
>
> Key: CASSANDRA-8622
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8622
> Project: Cassandra
> Issue Type: Test
> Components: Hadoop
>Reporter: Philip Thompson
>Assignee: Brandon Williams
> Fix For: 3.0
>
>
> See http://cassci.datastax.com/job/trunk_pigtest/330/testReport/
> Every test in the ant target {{ant pig-test}} has been failing on trunk for a
> while now.
> {code}
> java.lang.ExceptionInInitializerError
> at org.apache.log4j.Logger.getLogger(Logger.java:40)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:48)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:44)
> at org.hyperic.sigar.SigarLog.debug(SigarLog.java:60)
> at org.hyperic.sigar.Sigar.(Sigar.java:108)
> at org.apache.cassandra.utils.SigarLibrary.(SigarLibrary.java:45)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:206)
> at
> org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:408)
> at
> org.apache.cassandra.service.EmbeddedCassandraService.start(EmbeddedCassandraService.java:52)
> at
> org.apache.cassandra.pig.PigTestBase.startCassandra(PigTestBase.java:96)
> at
> org.apache.cassandra.pig.CqlRecordReaderTest.setup(CqlRecordReaderTest.java:63)
> at
> org.apache.log4j.Log4jLoggerFactory.(Log4jLoggerFactory.java:50)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8622) All of pig-test is failing in trunk
[
https://issues.apache.org/jira/browse/CASSANDRA-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279020#comment-14279020
]
Brandon Williams commented on CASSANDRA-8622:
-
All d286ac7d07 did was rename the license from .jar to .txt
> All of pig-test is failing in trunk
> ---
>
> Key: CASSANDRA-8622
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8622
> Project: Cassandra
> Issue Type: Test
> Components: Hadoop
>Reporter: Philip Thompson
>Assignee: Brandon Williams
> Fix For: 3.0
>
>
> See http://cassci.datastax.com/job/trunk_pigtest/330/testReport/
> Every test in the ant target {{ant pig-test}} has been failing on trunk for a
> while now.
> {code}
> java.lang.ExceptionInInitializerError
> at org.apache.log4j.Logger.getLogger(Logger.java:40)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:48)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:44)
> at org.hyperic.sigar.SigarLog.debug(SigarLog.java:60)
> at org.hyperic.sigar.Sigar.(Sigar.java:108)
> at org.apache.cassandra.utils.SigarLibrary.(SigarLibrary.java:45)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:206)
> at
> org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:408)
> at
> org.apache.cassandra.service.EmbeddedCassandraService.start(EmbeddedCassandraService.java:52)
> at
> org.apache.cassandra.pig.PigTestBase.startCassandra(PigTestBase.java:96)
> at
> org.apache.cassandra.pig.CqlRecordReaderTest.setup(CqlRecordReaderTest.java:63)
> at
> org.apache.log4j.Log4jLoggerFactory.(Log4jLoggerFactory.java:50)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[1/2] cassandra git commit: Update error message in cqlsh test to match latest
Repository: cassandra Updated Branches: refs/heads/trunk 0c0891ac6 -> a6ae07f16 Update error message in cqlsh test to match latest Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a3234e19 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a3234e19 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a3234e19 Branch: refs/heads/trunk Commit: a3234e191f661cfbbb27df748205fb1ac67101de Parents: a6e953c Author: Tyler Hobbs Authored: Thu Jan 15 11:57:00 2015 -0600 Committer: Tyler Hobbs Committed: Thu Jan 15 11:57:00 2015 -0600 -- pylib/cqlshlib/test/test_cqlsh_output.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/a3234e19/pylib/cqlshlib/test/test_cqlsh_output.py -- diff --git a/pylib/cqlshlib/test/test_cqlsh_output.py b/pylib/cqlshlib/test/test_cqlsh_output.py index baf72c1..60ef3a4 100644 --- a/pylib/cqlshlib/test/test_cqlsh_output.py +++ b/pylib/cqlshlib/test/test_cqlsh_output.py @@ -544,7 +544,7 @@ class TestCqlshOutput(BaseTestCase): self.assertEqual(outputlines[2], 'cqlsh:system> ') midline = ColoredText(outputlines[1]) self.assertEqual(midline.plain(), - 'code=2200 [Invalid query] message="Keyspace \'nonexistentkeyspace\' does not exist"') + 'InvalidRequest: code=2200 [Invalid query] message="Keyspace \'nonexistentkeyspace\' does not exist"') self.assertColorFromTags(midline, "RR")
cassandra git commit: Update error message in cqlsh test to match latest
Repository: cassandra Updated Branches: refs/heads/cassandra-2.1 a6e953c8c -> a3234e191 Update error message in cqlsh test to match latest Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a3234e19 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a3234e19 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a3234e19 Branch: refs/heads/cassandra-2.1 Commit: a3234e191f661cfbbb27df748205fb1ac67101de Parents: a6e953c Author: Tyler Hobbs Authored: Thu Jan 15 11:57:00 2015 -0600 Committer: Tyler Hobbs Committed: Thu Jan 15 11:57:00 2015 -0600 -- pylib/cqlshlib/test/test_cqlsh_output.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/a3234e19/pylib/cqlshlib/test/test_cqlsh_output.py -- diff --git a/pylib/cqlshlib/test/test_cqlsh_output.py b/pylib/cqlshlib/test/test_cqlsh_output.py index baf72c1..60ef3a4 100644 --- a/pylib/cqlshlib/test/test_cqlsh_output.py +++ b/pylib/cqlshlib/test/test_cqlsh_output.py @@ -544,7 +544,7 @@ class TestCqlshOutput(BaseTestCase): self.assertEqual(outputlines[2], 'cqlsh:system> ') midline = ColoredText(outputlines[1]) self.assertEqual(midline.plain(), - 'code=2200 [Invalid query] message="Keyspace \'nonexistentkeyspace\' does not exist"') + 'InvalidRequest: code=2200 [Invalid query] message="Keyspace \'nonexistentkeyspace\' does not exist"') self.assertColorFromTags(midline, "RR")
[2/2] cassandra git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a6ae07f1 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a6ae07f1 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a6ae07f1 Branch: refs/heads/trunk Commit: a6ae07f1687dea674523ed901c271fb926acb51b Parents: 0c0891a a3234e1 Author: Tyler Hobbs Authored: Thu Jan 15 11:57:35 2015 -0600 Committer: Tyler Hobbs Committed: Thu Jan 15 11:57:35 2015 -0600 -- pylib/cqlshlib/test/test_cqlsh_output.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --
[jira] [Commented] (CASSANDRA-8622) All of pig-test is failing in trunk
[
https://issues.apache.org/jira/browse/CASSANDRA-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279002#comment-14279002
]
Philip Thompson commented on CASSANDRA-8622:
[~brandon.williams], when applying a reverse patch, did you reverse both
c832a36f25c and d286ac7d07? The second commit fixed some sort of compilation
issue.
> All of pig-test is failing in trunk
> ---
>
> Key: CASSANDRA-8622
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8622
> Project: Cassandra
> Issue Type: Test
> Components: Hadoop
>Reporter: Philip Thompson
>Assignee: Brandon Williams
> Fix For: 3.0
>
>
> See http://cassci.datastax.com/job/trunk_pigtest/330/testReport/
> Every test in the ant target {{ant pig-test}} has been failing on trunk for a
> while now.
> {code}
> java.lang.ExceptionInInitializerError
> at org.apache.log4j.Logger.getLogger(Logger.java:40)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:48)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:44)
> at org.hyperic.sigar.SigarLog.debug(SigarLog.java:60)
> at org.hyperic.sigar.Sigar.(Sigar.java:108)
> at org.apache.cassandra.utils.SigarLibrary.(SigarLibrary.java:45)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:206)
> at
> org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:408)
> at
> org.apache.cassandra.service.EmbeddedCassandraService.start(EmbeddedCassandraService.java:52)
> at
> org.apache.cassandra.pig.PigTestBase.startCassandra(PigTestBase.java:96)
> at
> org.apache.cassandra.pig.CqlRecordReaderTest.setup(CqlRecordReaderTest.java:63)
> at
> org.apache.log4j.Log4jLoggerFactory.(Log4jLoggerFactory.java:50)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8628) Stalling rebuild of new 2.1.2 node from cluster running 2.0.9
[
https://issues.apache.org/jira/browse/CASSANDRA-8628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278976#comment-14278976
]
Brandon Williams commented on CASSANDRA-8628:
-
It has always been the expected behavior when mixing major versions.
> Stalling rebuild of new 2.1.2 node from cluster running 2.0.9
> -
>
> Key: CASSANDRA-8628
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8628
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Ubuntu 14.04, 4 core machines, 8 and 16 GB RAM
> Cluster running cassandra 2.0.9, new node running 2.1.2,
> inter-node encryption: all
>Reporter: Lennart Liberg
>
> We want to add a new datacenter to our existing cluster consisting of two
> datacenters containing 2 nodes and 1 node respectively, running version 2.0.9
> We configured a node running cassandra 2.1.2 in the new datacenter
> (auto_bootstrap: false) and verified that it managed to join the cluster as
> expected (`nodetool status` saying its status was UN).
> Upon executing `nodetool rebuild ams1`, the system.log output below is
> observed, with a new UnknownColumnFamilyException and NullPointerException
> every few minutes, and no data being transfered.
> After downgrading to cassandra 2.0.11, the rebuild operation worked
> flawlessly, using a cassandra.yaml configured as identical as possible to the
> 2.1.2 yaml file (a lot of new items in 2.1.x).
> My reading of the upgrading compatibility document leads me to think that
> there should not be any problem in performing this kind of rebuild.
> http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeDebC_t.html
> system.log output:
> {noformat}
> INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,228
> StorageService.java:958 - rebuild from dc: ams1
> INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,305
> StreamResultFuture.java:86 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
> Executing streaming plan for Rebuild
> INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,306
> StreamSession.java:213 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
> Starting streaming to 37.xx.xx.1/37.xx.xx.1
> INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,610
> StreamCoordinator.java:209 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342,
> ID#0] Beginning stream session with 37.xx.xx.1/37.xx.xx.1
> WARN [Thread-28935] 2015-01-15 16:03:40,458 IncomingTcpConnection.java:91 -
> UnknownColumnFamilyException reading from socket; closing
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=d1ea009c-feba-3d75-aa64-d022270842c4
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> ERROR [Thread-28974] 2015-01-15 16:05:54,164 CassandraDaemon.java:153 -
> Exception in thread Thread[Thread-28974,5,main]
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.c
[jira] [Commented] (CASSANDRA-8622) All of pig-test is failing in trunk
[
https://issues.apache.org/jira/browse/CASSANDRA-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278965#comment-14278965
]
Brandon Williams commented on CASSANDRA-8622:
-
Of course now I get failures from CASSANDRA-8577.
> All of pig-test is failing in trunk
> ---
>
> Key: CASSANDRA-8622
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8622
> Project: Cassandra
> Issue Type: Test
> Components: Hadoop
>Reporter: Philip Thompson
>Assignee: Brandon Williams
> Fix For: 3.0
>
>
> See http://cassci.datastax.com/job/trunk_pigtest/330/testReport/
> Every test in the ant target {{ant pig-test}} has been failing on trunk for a
> while now.
> {code}
> java.lang.ExceptionInInitializerError
> at org.apache.log4j.Logger.getLogger(Logger.java:40)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:48)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:44)
> at org.hyperic.sigar.SigarLog.debug(SigarLog.java:60)
> at org.hyperic.sigar.Sigar.(Sigar.java:108)
> at org.apache.cassandra.utils.SigarLibrary.(SigarLibrary.java:45)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:206)
> at
> org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:408)
> at
> org.apache.cassandra.service.EmbeddedCassandraService.start(EmbeddedCassandraService.java:52)
> at
> org.apache.cassandra.pig.PigTestBase.startCassandra(PigTestBase.java:96)
> at
> org.apache.cassandra.pig.CqlRecordReaderTest.setup(CqlRecordReaderTest.java:63)
> at
> org.apache.log4j.Log4jLoggerFactory.(Log4jLoggerFactory.java:50)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8622) All of pig-test is failing in trunk
[
https://issues.apache.org/jira/browse/CASSANDRA-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278960#comment-14278960
]
Brandon Williams commented on CASSANDRA-8622:
-
Actually, you're right, checking out the parent gets rid of the sigar problems.
I was applying a reverse patch of that commit and I guess something went wrong.
> All of pig-test is failing in trunk
> ---
>
> Key: CASSANDRA-8622
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8622
> Project: Cassandra
> Issue Type: Test
> Components: Hadoop
>Reporter: Philip Thompson
>Assignee: Brandon Williams
> Fix For: 3.0
>
>
> See http://cassci.datastax.com/job/trunk_pigtest/330/testReport/
> Every test in the ant target {{ant pig-test}} has been failing on trunk for a
> while now.
> {code}
> java.lang.ExceptionInInitializerError
> at org.apache.log4j.Logger.getLogger(Logger.java:40)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:48)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:44)
> at org.hyperic.sigar.SigarLog.debug(SigarLog.java:60)
> at org.hyperic.sigar.Sigar.(Sigar.java:108)
> at org.apache.cassandra.utils.SigarLibrary.(SigarLibrary.java:45)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:206)
> at
> org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:408)
> at
> org.apache.cassandra.service.EmbeddedCassandraService.start(EmbeddedCassandraService.java:52)
> at
> org.apache.cassandra.pig.PigTestBase.startCassandra(PigTestBase.java:96)
> at
> org.apache.cassandra.pig.CqlRecordReaderTest.setup(CqlRecordReaderTest.java:63)
> at
> org.apache.log4j.Log4jLoggerFactory.(Log4jLoggerFactory.java:50)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8628) Stalling rebuild of new 2.1.2 node from cluster running 2.0.9
[
https://issues.apache.org/jira/browse/CASSANDRA-8628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278953#comment-14278953
]
Lennart Liberg commented on CASSANDRA-8628:
---
Thank you for the response. Is this the intended behavior when the "new" node
is on a higher version, or is it specific to 2.0.x --> 2.1.x?
> Stalling rebuild of new 2.1.2 node from cluster running 2.0.9
> -
>
> Key: CASSANDRA-8628
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8628
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Ubuntu 14.04, 4 core machines, 8 and 16 GB RAM
> Cluster running cassandra 2.0.9, new node running 2.1.2,
> inter-node encryption: all
>Reporter: Lennart Liberg
>
> We want to add a new datacenter to our existing cluster consisting of two
> datacenters containing 2 nodes and 1 node respectively, running version 2.0.9
> We configured a node running cassandra 2.1.2 in the new datacenter
> (auto_bootstrap: false) and verified that it managed to join the cluster as
> expected (`nodetool status` saying its status was UN).
> Upon executing `nodetool rebuild ams1`, the system.log output below is
> observed, with a new UnknownColumnFamilyException and NullPointerException
> every few minutes, and no data being transfered.
> After downgrading to cassandra 2.0.11, the rebuild operation worked
> flawlessly, using a cassandra.yaml configured as identical as possible to the
> 2.1.2 yaml file (a lot of new items in 2.1.x).
> My reading of the upgrading compatibility document leads me to think that
> there should not be any problem in performing this kind of rebuild.
> http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeDebC_t.html
> system.log output:
> {noformat}
> INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,228
> StorageService.java:958 - rebuild from dc: ams1
> INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,305
> StreamResultFuture.java:86 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
> Executing streaming plan for Rebuild
> INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,306
> StreamSession.java:213 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
> Starting streaming to 37.xx.xx.1/37.xx.xx.1
> INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,610
> StreamCoordinator.java:209 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342,
> ID#0] Beginning stream session with 37.xx.xx.1/37.xx.xx.1
> WARN [Thread-28935] 2015-01-15 16:03:40,458 IncomingTcpConnection.java:91 -
> UnknownColumnFamilyException reading from socket; closing
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=d1ea009c-feba-3d75-aa64-d022270842c4
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> ERROR [Thread-28974] 2015-01-15 16:05:54,164 CassandraDaemon.java:153 -
> Exception in thread Thread[Thread-28974,5,main]
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
[jira] [Commented] (CASSANDRA-8628) Stalling rebuild of new 2.1.2 node from cluster running 2.0.9
[
https://issues.apache.org/jira/browse/CASSANDRA-8628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278951#comment-14278951
]
Lennart Liberg commented on CASSANDRA-8628:
---
Thank you for the response. Is this the intended behavior when the "new" node
is on a higher version, or is it specific to 2.0.x --> 2.1.x?
> Stalling rebuild of new 2.1.2 node from cluster running 2.0.9
> -
>
> Key: CASSANDRA-8628
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8628
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Ubuntu 14.04, 4 core machines, 8 and 16 GB RAM
> Cluster running cassandra 2.0.9, new node running 2.1.2,
> inter-node encryption: all
>Reporter: Lennart Liberg
>
> We want to add a new datacenter to our existing cluster consisting of two
> datacenters containing 2 nodes and 1 node respectively, running version 2.0.9
> We configured a node running cassandra 2.1.2 in the new datacenter
> (auto_bootstrap: false) and verified that it managed to join the cluster as
> expected (`nodetool status` saying its status was UN).
> Upon executing `nodetool rebuild ams1`, the system.log output below is
> observed, with a new UnknownColumnFamilyException and NullPointerException
> every few minutes, and no data being transfered.
> After downgrading to cassandra 2.0.11, the rebuild operation worked
> flawlessly, using a cassandra.yaml configured as identical as possible to the
> 2.1.2 yaml file (a lot of new items in 2.1.x).
> My reading of the upgrading compatibility document leads me to think that
> there should not be any problem in performing this kind of rebuild.
> http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeDebC_t.html
> system.log output:
> {noformat}
> INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,228
> StorageService.java:958 - rebuild from dc: ams1
> INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,305
> StreamResultFuture.java:86 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
> Executing streaming plan for Rebuild
> INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,306
> StreamSession.java:213 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
> Starting streaming to 37.xx.xx.1/37.xx.xx.1
> INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,610
> StreamCoordinator.java:209 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342,
> ID#0] Beginning stream session with 37.xx.xx.1/37.xx.xx.1
> WARN [Thread-28935] 2015-01-15 16:03:40,458 IncomingTcpConnection.java:91 -
> UnknownColumnFamilyException reading from socket; closing
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=d1ea009c-feba-3d75-aa64-d022270842c4
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> ERROR [Thread-28974] 2015-01-15 16:05:54,164 CassandraDaemon.java:153 -
> Exception in thread Thread[Thread-28974,5,main]
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99
[jira] [Issue Comment Deleted] (CASSANDRA-8628) Stalling rebuild of new 2.1.2 node from cluster running 2.0.9
[
https://issues.apache.org/jira/browse/CASSANDRA-8628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lennart Liberg updated CASSANDRA-8628:
--
Comment: was deleted
(was: Thank you for the response. Is this the intended behavior when the "new"
node is on a higher version, or is it specific to 2.0.x --> 2.1.x?
)
> Stalling rebuild of new 2.1.2 node from cluster running 2.0.9
> -
>
> Key: CASSANDRA-8628
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8628
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Ubuntu 14.04, 4 core machines, 8 and 16 GB RAM
> Cluster running cassandra 2.0.9, new node running 2.1.2,
> inter-node encryption: all
>Reporter: Lennart Liberg
>
> We want to add a new datacenter to our existing cluster consisting of two
> datacenters containing 2 nodes and 1 node respectively, running version 2.0.9
> We configured a node running cassandra 2.1.2 in the new datacenter
> (auto_bootstrap: false) and verified that it managed to join the cluster as
> expected (`nodetool status` saying its status was UN).
> Upon executing `nodetool rebuild ams1`, the system.log output below is
> observed, with a new UnknownColumnFamilyException and NullPointerException
> every few minutes, and no data being transfered.
> After downgrading to cassandra 2.0.11, the rebuild operation worked
> flawlessly, using a cassandra.yaml configured as identical as possible to the
> 2.1.2 yaml file (a lot of new items in 2.1.x).
> My reading of the upgrading compatibility document leads me to think that
> there should not be any problem in performing this kind of rebuild.
> http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeDebC_t.html
> system.log output:
> {noformat}
> INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,228
> StorageService.java:958 - rebuild from dc: ams1
> INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,305
> StreamResultFuture.java:86 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
> Executing streaming plan for Rebuild
> INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,306
> StreamSession.java:213 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
> Starting streaming to 37.xx.xx.1/37.xx.xx.1
> INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,610
> StreamCoordinator.java:209 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342,
> ID#0] Beginning stream session with 37.xx.xx.1/37.xx.xx.1
> WARN [Thread-28935] 2015-01-15 16:03:40,458 IncomingTcpConnection.java:91 -
> UnknownColumnFamilyException reading from socket; closing
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=d1ea009c-feba-3d75-aa64-d022270842c4
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> ERROR [Thread-28974] 2015-01-15 16:05:54,164 CassandraDaemon.java:153 -
> Exception in thread Thread[Thread-28974,5,main]
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2
[jira] [Commented] (CASSANDRA-8624) Cassandra Cluster's Status Inconsistency Strangely
[ https://issues.apache.org/jira/browse/CASSANDRA-8624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278943#comment-14278943 ] Brandon Williams commented on CASSANDRA-8624: - This sounds like some kind of network issue in your environment. > Cassandra Cluster's Status Inconsistency Strangely > -- > > Key: CASSANDRA-8624 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8624 > Project: Cassandra > Issue Type: Bug > Components: Tools > Environment: Cassandra 1.2.11 >Reporter: ZhongYu >Priority: Minor > Attachments: QQ截图20150115125254.png > > > We found a strange phenomenon about Cassandra Cluster's status that all the > nodes in the cluster found other node's status inconsistency. Especially, the > inconsistency has an interesting patten. See the following example: > There are 5 nodes (pc17, pc19, pc21, pc23, pc25) in the cluster. Their seeds > configuration are all "pc17, pc19, pc21, pc23, pc25". In a moment, > pc17 found others UP; > pc19 found pc17 DN, others UP; > pc21 found pc17, pc19 DN, others UP; > pc23 found pc17, pc19, pc21 DN, others UP; > pc25 found pc17, pc19, pc21, pc23 DN, only self UP; > See attachments as screen's snapshot. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[1/3] cassandra git commit: Shutdown CommitLog executor
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.1 d406bab50 -> a6e953c8c
refs/heads/trunk 076b5bdd1 -> 0c0891ac6
Shutdown CommitLog executor
This also reverts previous patch for CASSANDRA-8281
patch by Benjamin Lerer; reviewed by yukim for CASSANDRA-8281
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a6e953c8
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a6e953c8
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a6e953c8
Branch: refs/heads/cassandra-2.1
Commit: a6e953c8c71caa1724d15dc31069aa073a9fe9ee
Parents: d406bab
Author: Benjamin Lerer
Authored: Thu Jan 15 10:22:19 2015 -0600
Committer: Yuki Morishita
Committed: Thu Jan 15 10:22:19 2015 -0600
--
.../apache/cassandra/io/sstable/CQLSSTableWriter.java | 13 +++--
1 file changed, 11 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/a6e953c8/src/java/org/apache/cassandra/io/sstable/CQLSSTableWriter.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/CQLSSTableWriter.java
b/src/java/org/apache/cassandra/io/sstable/CQLSSTableWriter.java
index a42c480..85e54ab 100644
--- a/src/java/org/apache/cassandra/io/sstable/CQLSSTableWriter.java
+++ b/src/java/org/apache/cassandra/io/sstable/CQLSSTableWriter.java
@@ -34,6 +34,7 @@ import org.apache.cassandra.cql3.statements.*;
import org.apache.cassandra.cql3.*;
import org.apache.cassandra.config.*;
import org.apache.cassandra.db.*;
+import org.apache.cassandra.db.commitlog.CommitLog;
import org.apache.cassandra.db.composites.Composite;
import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.dht.IPartitioner;
@@ -78,8 +79,8 @@ public class CQLSSTableWriter implements Closeable
{
static
{
-// The CQLSSTableWriter must always be used in client mode.
-Config.setClientMode(true);
+// The Keyspace need to be initialized before we can call Keyspace.open
+Keyspace.setInitialized();
}
private final AbstractSSTableSimpleWriter writer;
@@ -269,6 +270,14 @@ public class CQLSSTableWriter implements Closeable
public void close() throws IOException
{
writer.close();
+try
+{
+CommitLog.instance.shutdownBlocking();
+}
+catch (InterruptedException e)
+{
+Thread.currentThread().interrupt();
+}
}
/**
[3/3] cassandra git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk
Conflicts:
src/java/org/apache/cassandra/io/sstable/CQLSSTableWriter.java
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0c0891ac
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0c0891ac
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0c0891ac
Branch: refs/heads/trunk
Commit: 0c0891ac65b395610752b98abd21343d1b2902a8
Parents: 076b5bd a6e953c
Author: Yuki Morishita
Authored: Thu Jan 15 11:01:00 2015 -0600
Committer: Yuki Morishita
Committed: Thu Jan 15 11:01:00 2015 -0600
--
.../io/sstable/AbstractSSTableSimpleWriter.java | 16 +
.../cassandra/io/sstable/CQLSSTableWriter.java | 34 ++--
.../io/sstable/SSTableSimpleUnsortedWriter.java | 22 +++--
.../cassandra/db/ColumnFamilyStoreTest.java | 2 +-
4 files changed, 49 insertions(+), 25 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/0c0891ac/src/java/org/apache/cassandra/io/sstable/AbstractSSTableSimpleWriter.java
--
diff --cc
src/java/org/apache/cassandra/io/sstable/AbstractSSTableSimpleWriter.java
index 2854308,67d160d..988923e
--- a/src/java/org/apache/cassandra/io/sstable/AbstractSSTableSimpleWriter.java
+++ b/src/java/org/apache/cassandra/io/sstable/AbstractSSTableSimpleWriter.java
@@@ -56,20 -53,21 +56,25 @@@ public abstract class AbstractSSTableSi
DatabaseDescriptor.setPartitioner(partitioner);
}
+protected void setSSTableFormatType(SSTableFormat.Type type)
+{
+this.formatType = type;
+}
+
protected SSTableWriter getWriter()
{
- return
SSTableWriter.create(Descriptor.fromFilename(makeFilename(directory,
metadata.ksName, metadata.cfName, formatType)), 0,
ActiveRepairService.UNREPAIRED_SSTABLE);
-return new SSTableWriter(
-makeFilename(directory, metadata.ksName, metadata.cfName),
-0, // We don't care about the bloom filter
-ActiveRepairService.UNREPAIRED_SSTABLE,
-metadata,
-DatabaseDescriptor.getPartitioner(),
-new MetadataCollector(metadata.comparator));
++return SSTableWriter.create(createDescriptor(directory,
metadata.ksName, metadata.cfName, formatType), 0,
ActiveRepairService.UNREPAIRED_SSTABLE);
}
--// find available generation and pick up filename from that
- protected static String makeFilename(File directory, final String
keyspace, final String columnFamily, final SSTableFormat.Type fmt)
-protected static String makeFilename(File directory, final String
keyspace, final String columnFamily)
++protected static Descriptor createDescriptor(File directory, final String
keyspace, final String columnFamily, final SSTableFormat.Type fmt)
{
--final Set existing = new HashSet();
++int maxGen = getNextGeneration(directory, columnFamily);
++return new Descriptor(directory, keyspace, columnFamily, maxGen + 1,
Descriptor.Type.TEMP, fmt);
++}
++
++private static int getNextGeneration(File directory, final String
columnFamily)
++{
++final Set existing = new HashSet<>();
directory.list(new FilenameFilter()
{
public boolean accept(File dir, String name)
@@@ -93,8 -91,8 +98,7 @@@
maxGen = generation.getAndIncrement();
}
}
--
- return new Descriptor(directory, keyspace, columnFamily, maxGen + 1,
Descriptor.Type.TEMP, fmt).filenameFor(Component.DATA);
-return new Descriptor(directory, keyspace, columnFamily, maxGen + 1,
Descriptor.Type.TEMP).filenameFor(Component.DATA);
++return maxGen;
}
/**
http://git-wip-us.apache.org/repos/asf/cassandra/blob/0c0891ac/src/java/org/apache/cassandra/io/sstable/CQLSSTableWriter.java
--
diff --cc src/java/org/apache/cassandra/io/sstable/CQLSSTableWriter.java
index ec590f3,85e54ab..a019d41
--- a/src/java/org/apache/cassandra/io/sstable/CQLSSTableWriter.java
+++ b/src/java/org/apache/cassandra/io/sstable/CQLSSTableWriter.java
@@@ -21,19 -21,20 +21,22 @@@ import java.io.Closeable
import java.io.File;
import java.io.IOException;
import java.nio.ByteBuffer;
--import java.util.Arrays;
--import java.util.ArrayList;
--import java.util.Collections;
--import java.util.List;
--import java.util.Map;
++import java.util.*;
import com.google.common.collect.ImmutableMap;
--import com.google.common.collect.Iterables;
--import org.apache.cassandra.cql3.statements.*;
++import org.apache.cassandra.config.CFMetaData;
++import org.apache.cassandra.config.KSMetaData;
++import org.apache.cassandra.co
[2/3] cassandra git commit: Shutdown CommitLog executor
Shutdown CommitLog executor
This also reverts previous patch for CASSANDRA-8281
patch by Benjamin Lerer; reviewed by yukim for CASSANDRA-8281
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a6e953c8
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a6e953c8
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a6e953c8
Branch: refs/heads/trunk
Commit: a6e953c8c71caa1724d15dc31069aa073a9fe9ee
Parents: d406bab
Author: Benjamin Lerer
Authored: Thu Jan 15 10:22:19 2015 -0600
Committer: Yuki Morishita
Committed: Thu Jan 15 10:22:19 2015 -0600
--
.../apache/cassandra/io/sstable/CQLSSTableWriter.java | 13 +++--
1 file changed, 11 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/a6e953c8/src/java/org/apache/cassandra/io/sstable/CQLSSTableWriter.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/CQLSSTableWriter.java
b/src/java/org/apache/cassandra/io/sstable/CQLSSTableWriter.java
index a42c480..85e54ab 100644
--- a/src/java/org/apache/cassandra/io/sstable/CQLSSTableWriter.java
+++ b/src/java/org/apache/cassandra/io/sstable/CQLSSTableWriter.java
@@ -34,6 +34,7 @@ import org.apache.cassandra.cql3.statements.*;
import org.apache.cassandra.cql3.*;
import org.apache.cassandra.config.*;
import org.apache.cassandra.db.*;
+import org.apache.cassandra.db.commitlog.CommitLog;
import org.apache.cassandra.db.composites.Composite;
import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.dht.IPartitioner;
@@ -78,8 +79,8 @@ public class CQLSSTableWriter implements Closeable
{
static
{
-// The CQLSSTableWriter must always be used in client mode.
-Config.setClientMode(true);
+// The Keyspace need to be initialized before we can call Keyspace.open
+Keyspace.setInitialized();
}
private final AbstractSSTableSimpleWriter writer;
@@ -269,6 +270,14 @@ public class CQLSSTableWriter implements Closeable
public void close() throws IOException
{
writer.close();
+try
+{
+CommitLog.instance.shutdownBlocking();
+}
+catch (InterruptedException e)
+{
+Thread.currentThread().interrupt();
+}
}
/**
[Cassandra Wiki] Update of "Committers" by RobertStupp
Dear Wiki user,
You have subscribed to a wiki page or wiki category on "Cassandra Wiki" for
change notification.
The "Committers" page has been changed by RobertStupp:
https://wiki.apache.org/cassandra/Committers?action=diff&rev1=49&rev2=50
||Tyler Hobbs ||March 2014 ||Datastax || ||
||Benedict Elliott Smith ||May 2014 ||Datastax || ||
||Josh Mckenzie ||July 2014 ||Datastax || ||
+ ||Robert Stupp ||Jan 2015 ||contentteam || ||
{{https://c.statcounter.com/9397521/0/fe557aad/1/|stats}}
[jira] [Resolved] (CASSANDRA-8628) Stalling rebuild of new 2.1.2 node from cluster running 2.0.9
[
https://issues.apache.org/jira/browse/CASSANDRA-8628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams resolved CASSANDRA-8628.
-
Resolution: Not a Problem
> Stalling rebuild of new 2.1.2 node from cluster running 2.0.9
> -
>
> Key: CASSANDRA-8628
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8628
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Ubuntu 14.04, 4 core machines, 8 and 16 GB RAM
> Cluster running cassandra 2.0.9, new node running 2.1.2,
> inter-node encryption: all
>Reporter: Lennart Liberg
>
> We want to add a new datacenter to our existing cluster consisting of two
> datacenters containing 2 nodes and 1 node respectively, running version 2.0.9
> We configured a node running cassandra 2.1.2 in the new datacenter
> (auto_bootstrap: false) and verified that it managed to join the cluster as
> expected (`nodetool status` saying its status was UN).
> Upon executing `nodetool rebuild ams1`, the system.log output below is
> observed, with a new UnknownColumnFamilyException and NullPointerException
> every few minutes, and no data being transfered.
> After downgrading to cassandra 2.0.11, the rebuild operation worked
> flawlessly, using a cassandra.yaml configured as identical as possible to the
> 2.1.2 yaml file (a lot of new items in 2.1.x).
> My reading of the upgrading compatibility document leads me to think that
> there should not be any problem in performing this kind of rebuild.
> http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeDebC_t.html
> system.log output:
> {noformat}
> INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,228
> StorageService.java:958 - rebuild from dc: ams1
> INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,305
> StreamResultFuture.java:86 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
> Executing streaming plan for Rebuild
> INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,306
> StreamSession.java:213 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
> Starting streaming to 37.xx.xx.1/37.xx.xx.1
> INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,610
> StreamCoordinator.java:209 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342,
> ID#0] Beginning stream session with 37.xx.xx.1/37.xx.xx.1
> WARN [Thread-28935] 2015-01-15 16:03:40,458 IncomingTcpConnection.java:91 -
> UnknownColumnFamilyException reading from socket; closing
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=d1ea009c-feba-3d75-aa64-d022270842c4
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> ERROR [Thread-28974] 2015-01-15 16:05:54,164 CassandraDaemon.java:153 -
> Exception in thread Thread[Thread-28974,5,main]
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
> ~[apache-c
[jira] [Commented] (CASSANDRA-8628) Stalling rebuild of new 2.1.2 node from cluster running 2.0.9
[
https://issues.apache.org/jira/browse/CASSANDRA-8628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278936#comment-14278936
]
Brandon Williams commented on CASSANDRA-8628:
-
bq. My reading of the upgrading compatibility document leads me to think that
there should not be any problem in performing this kind of rebuild.
Actually, there is. Since the new node is 2.1, it won't pull schema from 2.0,
which leads to your "Couldn't find cfId" problems. You should start the new
node on 2.0, let it get the schema, then upgrade it to 2.1.
> Stalling rebuild of new 2.1.2 node from cluster running 2.0.9
> -
>
> Key: CASSANDRA-8628
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8628
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Ubuntu 14.04, 4 core machines, 8 and 16 GB RAM
> Cluster running cassandra 2.0.9, new node running 2.1.2,
> inter-node encryption: all
>Reporter: Lennart Liberg
>
> We want to add a new datacenter to our existing cluster consisting of two
> datacenters containing 2 nodes and 1 node respectively, running version 2.0.9
> We configured a node running cassandra 2.1.2 in the new datacenter
> (auto_bootstrap: false) and verified that it managed to join the cluster as
> expected (`nodetool status` saying its status was UN).
> Upon executing `nodetool rebuild ams1`, the system.log output below is
> observed, with a new UnknownColumnFamilyException and NullPointerException
> every few minutes, and no data being transfered.
> After downgrading to cassandra 2.0.11, the rebuild operation worked
> flawlessly, using a cassandra.yaml configured as identical as possible to the
> 2.1.2 yaml file (a lot of new items in 2.1.x).
> My reading of the upgrading compatibility document leads me to think that
> there should not be any problem in performing this kind of rebuild.
> http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeDebC_t.html
> system.log output:
> {noformat}
> INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,228
> StorageService.java:958 - rebuild from dc: ams1
> INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,305
> StreamResultFuture.java:86 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
> Executing streaming plan for Rebuild
> INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,306
> StreamSession.java:213 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
> Starting streaming to 37.xx.xx.1/37.xx.xx.1
> INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,610
> StreamCoordinator.java:209 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342,
> ID#0] Beginning stream session with 37.xx.xx.1/37.xx.xx.1
> WARN [Thread-28935] 2015-01-15 16:03:40,458 IncomingTcpConnection.java:91 -
> UnknownColumnFamilyException reading from socket; closing
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=d1ea009c-feba-3d75-aa64-d022270842c4
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> ERROR [Thread-28974] 2015-01-15 16:05:54,164 CassandraDaemon.java:153 -
> Exception in thread Thread[Thread-28974,5,main]
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
> ~[apache-cass
[jira] [Updated] (CASSANDRA-8628) Stalling rebuild of new 2.1.2 node from cluster running 2.0.9
[
https://issues.apache.org/jira/browse/CASSANDRA-8628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lennart Liberg updated CASSANDRA-8628:
--
Description:
We want to add a new datacenter to our existing cluster consisting of two
datacenters containing 2 nodes and 1 node respectively, running version 2.0.9
We configured a node running cassandra 2.1.2 in the new datacenter
(auto_bootstrap: false) and verified that it managed to join the cluster as
expected (`nodetool status` saying its status was UN).
Upon executing `nodetool rebuild ams1`, the system.log output below is
observed, with a new UnknownColumnFamilyException and NullPointerException
every few minutes, and no data being transfered.
After downgrading to cassandra 2.0.11, the rebuild operation worked flawlessly,
using a cassandra.yaml configured as identical as possible to the 2.1.2 yaml
file (a lot of new items in 2.1.x).
My reading of the upgrading compatibility document leads me to think that there
should not be any problem in performing this kind of rebuild.
http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeDebC_t.html
system.log output:
{noformat}
INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,228
StorageService.java:958 - rebuild from dc: ams1
INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,305
StreamResultFuture.java:86 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
Executing streaming plan for Rebuild
INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,306
StreamSession.java:213 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
Starting streaming to 37.xx.xx.1/37.xx.xx.1
INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,610
StreamCoordinator.java:209 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342,
ID#0] Beginning stream session with 37.xx.xx.1/37.xx.xx.1
WARN [Thread-28935] 2015-01-15 16:03:40,458 IncomingTcpConnection.java:91 -
UnknownColumnFamilyException reading from socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
cfId=d1ea009c-feba-3d75-aa64-d022270842c4
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
~[apache-cassandra-2.1.2.jar:2.1.2]
ERROR [Thread-28974] 2015-01-15 16:05:54,164 CassandraDaemon.java:153 -
Exception in thread Thread[Thread-28974,5,main]
java.lang.NullPointerException: null
at
org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131)
~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
~[apache-cassandra-2.1.2.jar:2.1.2]
ERROR [Thread-29011] 2015-01-15 16:07:58,349 CassandraDaemon.java:153 -
Exception in thread Thread[Thread-29011,5,main]
java.lang.NullPointerException: null
at
org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
[jira] [Updated] (CASSANDRA-8628) Stalling rebuild of new 2.1.2 node from cluster running 2.0.9
[
https://issues.apache.org/jira/browse/CASSANDRA-8628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lennart Liberg updated CASSANDRA-8628:
--
Summary: Stalling rebuild of new 2.1.2 node from cluster running 2.0.9
(was: Stalling rebuild of new 2.1.12 node from cluster running 2.0.9)
> Stalling rebuild of new 2.1.2 node from cluster running 2.0.9
> -
>
> Key: CASSANDRA-8628
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8628
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Ubuntu 14.04, 4 core machines, 8 and 16 GB RAM
> Cluster running cassandra 2.0.9, new node running 2.1.2,
> inter-node encryption: all
>Reporter: Lennart Liberg
>
> We want to add a new datacenter to our existing cluster consisting of two
> datacenters containing 2 nodes and 1 node respectively, running version 2.0.9
> We configured a node running cassandra 2.1.2 in the new datacenter
> (auto_bootstrap: false) and verified that it managed to join the cluster as
> expected (`nodetool status` saying its status was UN).
> Upon executing `nodetool rebuild ams1`, the system.log output below is
> observed, with a new UnknownColumnFamilyException and NullPointerException
> every few minutes, and no data being transfered.
> After downgrading to cassandra 2.0.11, the rebuild operation worked flawless,
> using a cassandra.yaml configured as identical as possible to the 2.1.2 yaml
> file (a lot of new items in 2.1.x).
> My reading of the upgrading compatibility document leads me to think that
> there should not be any problem in performing this kind of rebuild.
> http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeDebC_t.html
> system.log output:
> {noformat}
> INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,228
> StorageService.java:958 - rebuild from dc: ams1
> INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,305
> StreamResultFuture.java:86 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
> Executing streaming plan for Rebuild
> INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,306
> StreamSession.java:213 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
> Starting streaming to 37.xx.xx.1/37.xx.xx.1
> INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,610
> StreamCoordinator.java:209 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342,
> ID#0] Beginning stream session with 37.xx.xx.1/37.xx.xx.1
> WARN [Thread-28935] 2015-01-15 16:03:40,458 IncomingTcpConnection.java:91 -
> UnknownColumnFamilyException reading from socket; closing
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=d1ea009c-feba-3d75-aa64-d022270842c4
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> ERROR [Thread-28974] 2015-01-15 16:05:54,164 CassandraDaemon.java:153 -
> Exception in thread Thread[Thread-28974,5,main]
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
[jira] [Updated] (CASSANDRA-5657) remove deprecated metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko updated CASSANDRA-5657: - Reviewer: Aleksey Yeschenko (was: Jonathan Ellis) > remove deprecated metrics > - > > Key: CASSANDRA-5657 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5657 > Project: Cassandra > Issue Type: Task > Components: Tools >Reporter: Jonathan Ellis >Assignee: T Jake Luciani > Labels: technical_debt > Fix For: 3.0 > > Attachments: 5657.txt, 5657v2.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8622) All of pig-test is failing in trunk
[
https://issues.apache.org/jira/browse/CASSANDRA-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278926#comment-14278926
]
Brandon Williams commented on CASSANDRA-8622:
-
As I suspected, the EmbeddedCassandraServiceTest unit test also fails the same
way, so something is either wrong with ECS or sigar.
> All of pig-test is failing in trunk
> ---
>
> Key: CASSANDRA-8622
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8622
> Project: Cassandra
> Issue Type: Test
> Components: Hadoop
>Reporter: Philip Thompson
>Assignee: Brandon Williams
> Fix For: 3.0
>
>
> See http://cassci.datastax.com/job/trunk_pigtest/330/testReport/
> Every test in the ant target {{ant pig-test}} has been failing on trunk for a
> while now.
> {code}
> java.lang.ExceptionInInitializerError
> at org.apache.log4j.Logger.getLogger(Logger.java:40)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:48)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:44)
> at org.hyperic.sigar.SigarLog.debug(SigarLog.java:60)
> at org.hyperic.sigar.Sigar.(Sigar.java:108)
> at org.apache.cassandra.utils.SigarLibrary.(SigarLibrary.java:45)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:206)
> at
> org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:408)
> at
> org.apache.cassandra.service.EmbeddedCassandraService.start(EmbeddedCassandraService.java:52)
> at
> org.apache.cassandra.pig.PigTestBase.startCassandra(PigTestBase.java:96)
> at
> org.apache.cassandra.pig.CqlRecordReaderTest.setup(CqlRecordReaderTest.java:63)
> at
> org.apache.log4j.Log4jLoggerFactory.(Log4jLoggerFactory.java:50)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8628) Stalling rebuild of new 2.1.12 node from cluster running 2.0.9
[
https://issues.apache.org/jira/browse/CASSANDRA-8628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lennart Liberg updated CASSANDRA-8628:
--
Description:
We want to add a new datacenter to our existing cluster consisting of two
datacenters containing 2 nodes and 1 node respectively, running version 2.0.9
We configured a node running cassandra 2.1.2 in the new datacenter
(auto_bootstrap: false) and verified that it managed to join the cluster as
expected (`nodetool status` saying its status was UN).
Upon executing `nodetool rebuild ams1`, the system.log output below is
observed, with a new UnknownColumnFamilyException and NullPointerException
every few minutes, and no data being transfered.
After downgrading to cassandra 2.0.11, the rebuild operation worked flawless,
using a cassandra.yaml configured as identical as possible to the 2.1.2 yaml
file (a lot of new items in 2.1.x).
My reading of the upgrading compatibility document leads me to think that there
should not be any problem in performing this kind of rebuild.
http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeDebC_t.html
system.log output:
{noformat}
INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,228
StorageService.java:958 - rebuild from dc: ams1
INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,305
StreamResultFuture.java:86 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
Executing streaming plan for Rebuild
INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,306
StreamSession.java:213 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
Starting streaming to 37.xx.xx.1/37.xx.xx.1
INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,610
StreamCoordinator.java:209 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342,
ID#0] Beginning stream session with 37.xx.xx.1/37.xx.xx.1
WARN [Thread-28935] 2015-01-15 16:03:40,458 IncomingTcpConnection.java:91 -
UnknownColumnFamilyException reading from socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
cfId=d1ea009c-feba-3d75-aa64-d022270842c4
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
~[apache-cassandra-2.1.2.jar:2.1.2]
ERROR [Thread-28974] 2015-01-15 16:05:54,164 CassandraDaemon.java:153 -
Exception in thread Thread[Thread-28974,5,main]
java.lang.NullPointerException: null
at
org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131)
~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
~[apache-cassandra-2.1.2.jar:2.1.2]
{noformat}
was:
We wanted to add a new datacenter to our cluster consisting of two datacenters
containing 2 nodes and 1 node respectively, running version 2.0.9
We configured a node running cassandra 2.1.2 in the new datacenter
(auto_bootstrap: false) and verified that it managed to join the cluster as
expected (`nodetool status` saying its status was UN).
Upon executing `nodetool rebuild a
[jira] [Updated] (CASSANDRA-8628) Stalling rebuild of new 2.1.12 node from cluster running 2.0.9
[
https://issues.apache.org/jira/browse/CASSANDRA-8628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lennart Liberg updated CASSANDRA-8628:
--
Description:
We want to add a new datacenter to our existing cluster consisting of two
datacenters containing 2 nodes and 1 node respectively, running version 2.0.9
We configured a node running cassandra 2.1.2 in the new datacenter
(auto_bootstrap: false) and verified that it managed to join the cluster as
expected (`nodetool status` saying its status was UN).
Upon executing `nodetool rebuild ams1`, the system.log output below is
observed, with a new UnknownColumnFamilyException and NullPointerException
every few minutes, and no data being transfered.
After downgrading to cassandra 2.0.11, the rebuild operation worked flawless,
using a cassandra.yaml configured as identical as possible to the 2.1.2 yaml
file (a lot of new items in 2.1.x).
My reading of the upgrading compatibility document leads me to think that there
should not be any problem in performing this kind of rebuild.
http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeDebC_t.html
system.log output:
{noformat}
INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,228
StorageService.java:958 - rebuild from dc: ams1
INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,305
StreamResultFuture.java:86 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
Executing streaming plan for Rebuild
INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,306
StreamSession.java:213 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
Starting streaming to 37.xx.xx.1/37.xx.xx.1
INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,610
StreamCoordinator.java:209 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342,
ID#0] Beginning stream session with 37.xx.xx.1/37.xx.xx.1
WARN [Thread-28935] 2015-01-15 16:03:40,458 IncomingTcpConnection.java:91 -
UnknownColumnFamilyException reading from socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
cfId=d1ea009c-feba-3d75-aa64-d022270842c4
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
~[apache-cassandra-2.1.2.jar:2.1.2]
ERROR [Thread-28974] 2015-01-15 16:05:54,164 CassandraDaemon.java:153 -
Exception in thread Thread[Thread-28974,5,main]
java.lang.NullPointerException: null
at
org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131)
~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
~[apache-cassandra-2.1.2.jar:2.1.2]
ERROR [Thread-29011] 2015-01-15 16:07:58,349 CassandraDaemon.java:153 -
Exception in thread Thread[Thread-29011,5,main]
java.lang.NullPointerException: null
at
org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
~[
[jira] [Updated] (CASSANDRA-8628) Stalling rebuild of new 2.1.12 node from cluster running 2.0.9
[
https://issues.apache.org/jira/browse/CASSANDRA-8628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lennart Liberg updated CASSANDRA-8628:
--
Description:
We wanted to add a new datacenter to our cluster consisting of two datacenters
containing 2 nodes and 1 node respectively, running version 2.0.9
We configured a node running cassandra 2.1.2 in the new datacenter
(auto_bootstrap: false) and verified that it managed to join the cluster as
expected (`nodetool status` saying its status was UN).
Upon executing `nodetool rebuild ams1`, the system.log output is observed, with
a new UnknownColumnFamilyException and NullPointerException every few minutes,
and no data being transfered.
After downgrading to cassandra 2.0.11, the rebuild operation worked flawless,
using a cassandra.yaml configured as identical as possible to the 2.1.2 yaml
file (a lot of new items in 2.1.x).
My reading of the upgrading compatibility document does not lead me to think
there ought to be any problem in performing this kind of rebuild.
http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeDebC_t.html
system.log output:
{noformat}
INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,228
StorageService.java:958 - rebuild from dc: ams1
INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,305
StreamResultFuture.java:86 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
Executing streaming plan for Rebuild
INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,306
StreamSession.java:213 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
Starting streaming to 37.xx.xx.1/37.xx.xx.1
INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,610
StreamCoordinator.java:209 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342,
ID#0] Beginning stream session with 37.xx.xx.1/37.xx.xx.1
WARN [Thread-28935] 2015-01-15 16:03:40,458 IncomingTcpConnection.java:91 -
UnknownColumnFamilyException reading from socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
cfId=d1ea009c-feba-3d75-aa64-d022270842c4
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
~[apache-cassandra-2.1.2.jar:2.1.2]
ERROR [Thread-28974] 2015-01-15 16:05:54,164 CassandraDaemon.java:153 -
Exception in thread Thread[Thread-28974,5,main]
java.lang.NullPointerException: null
at
org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131)
~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
~[apache-cassandra-2.1.2.jar:2.1.2]
{noformat}
was:
We wanted to add a new datacenter to our cluster consisting of two datacenters
containing 2 nodes and 1 node respectively, running version 2.0.9
We configured a node running cassandra 2.1.2 in the new datacenter
(auto_bootstrap: false) and verified that it managed to join the cluster as
expected (`nodetool status` saying its status was UN).
Upon executing `nodetool rebuild ams1`, the sy
[jira] [Updated] (CASSANDRA-8628) Stalling rebuild of new 2.1.12 node from cluster running 2.0.9
[
https://issues.apache.org/jira/browse/CASSANDRA-8628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lennart Liberg updated CASSANDRA-8628:
--
Description:
We wanted to add a new datacenter to our cluster consisting of two datacenters
containing 2 nodes and 1 node respectively, running version 2.0.9
We configured a node running cassandra 2.1.2 in the new datacenter
(auto_bootstrap: false) and verified that it managed to join the cluster as
expected (`nodetool status` saying its status was UN).
Upon executing `nodetool rebuild ams1`, the system.log output is observed, with
a new UnknownColumnFamilyException and NullPointerException every few minutes,
and no data being transfered.
After downgrading to cassandra 2.0.11, the rebuild operation worked flawless,
using a cassandra.yaml configured as identical as possible to the 2.1.2 yaml
file (a lot of new items in 2.1.x).
My reading of the upgrading compatibility document leads me to think that there
should not be any problem in performing this kind of rebuild.
http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeDebC_t.html
system.log output:
{noformat}
INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,228
StorageService.java:958 - rebuild from dc: ams1
INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,305
StreamResultFuture.java:86 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
Executing streaming plan for Rebuild
INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,306
StreamSession.java:213 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
Starting streaming to 37.xx.xx.1/37.xx.xx.1
INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,610
StreamCoordinator.java:209 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342,
ID#0] Beginning stream session with 37.xx.xx.1/37.xx.xx.1
WARN [Thread-28935] 2015-01-15 16:03:40,458 IncomingTcpConnection.java:91 -
UnknownColumnFamilyException reading from socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
cfId=d1ea009c-feba-3d75-aa64-d022270842c4
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
~[apache-cassandra-2.1.2.jar:2.1.2]
ERROR [Thread-28974] 2015-01-15 16:05:54,164 CassandraDaemon.java:153 -
Exception in thread Thread[Thread-28974,5,main]
java.lang.NullPointerException: null
at
org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131)
~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
~[apache-cassandra-2.1.2.jar:2.1.2]
{noformat}
was:
We wanted to add a new datacenter to our cluster consisting of two datacenters
containing 2 nodes and 1 node respectively, running version 2.0.9
We configured a node running cassandra 2.1.2 in the new datacenter
(auto_bootstrap: false) and verified that it managed to join the cluster as
expected (`nodetool status` saying its status was UN).
Upon executing `nodetool rebuild ams1`, the sys
[jira] [Created] (CASSANDRA-8628) Stalling rebuild of new 2.1.12 node from cluster running 2.0.9
Lennart Liberg created CASSANDRA-8628:
-
Summary: Stalling rebuild of new 2.1.12 node from cluster running
2.0.9
Key: CASSANDRA-8628
URL: https://issues.apache.org/jira/browse/CASSANDRA-8628
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Ubuntu 14.04, 4 core machines, 8 and 16 GB RAM
Cluster running cassandra 2.0.9, new node running 2.1.2,
inter-node encryption: all
Reporter: Lennart Liberg
We wanted to add a new datacenter to our cluster consisting of two datacenters
containing 2 nodes and 1 node respectively, running version 2.0.9
We configured a node running cassandra 2.1.2 in the new datacenter
(auto_bootstrap: false) and verified that it managed to join the cluster as
expected (`nodetool status` saying its status was UN).
Upon executing `nodetool rebuild ams1`, the system.log output is observed, with
a new UnknownColumnFamilyException and NullPointerException every few minutes,
and no data being transfered.
After downgrading to cassandra 2.0.11, the rebuild operation worked flawless,
using a cassandra.yaml configured as identical as possible to the 2.1.2 yaml
file (a lot of new items in 2.1.x).
My reading of the upgrading compatibility document does not lead me to think
there does not ought to be any problem in performing this kind of rebuild.
http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeDebC_t.html
system.log output:
{noformat}
INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,228
StorageService.java:958 - rebuild from dc: ams1
INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,305
StreamResultFuture.java:86 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
Executing streaming plan for Rebuild
INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,306
StreamSession.java:213 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342]
Starting streaming to 37.xx.xx.1/37.xx.xx.1
INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,610
StreamCoordinator.java:209 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342,
ID#0] Beginning stream session with 37.xx.xx.1/37.xx.xx.1
WARN [Thread-28935] 2015-01-15 16:03:40,458 IncomingTcpConnection.java:91 -
UnknownColumnFamilyException reading from socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
cfId=d1ea009c-feba-3d75-aa64-d022270842c4
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
~[apache-cassandra-2.1.2.jar:2.1.2]
ERROR [Thread-28974] 2015-01-15 16:05:54,164 CassandraDaemon.java:153 -
Exception in thread Thread[Thread-28974,5,main]
java.lang.NullPointerException: null
at
org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131)
~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
~[apache-cassandra-2.1.2.jar:2.1.2]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (CASSANDRA-8622) All of pig-test is failing in trunk
[
https://issues.apache.org/jira/browse/CASSANDRA-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philip Thompson updated CASSANDRA-8622:
---
Comment: was deleted
(was: Okay. When checking out the parent commit of c832a36f, I get a different
error, but I will solve my local problem first.)
> All of pig-test is failing in trunk
> ---
>
> Key: CASSANDRA-8622
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8622
> Project: Cassandra
> Issue Type: Test
> Components: Hadoop
>Reporter: Philip Thompson
>Assignee: Brandon Williams
> Fix For: 3.0
>
>
> See http://cassci.datastax.com/job/trunk_pigtest/330/testReport/
> Every test in the ant target {{ant pig-test}} has been failing on trunk for a
> while now.
> {code}
> java.lang.ExceptionInInitializerError
> at org.apache.log4j.Logger.getLogger(Logger.java:40)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:48)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:44)
> at org.hyperic.sigar.SigarLog.debug(SigarLog.java:60)
> at org.hyperic.sigar.Sigar.(Sigar.java:108)
> at org.apache.cassandra.utils.SigarLibrary.(SigarLibrary.java:45)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:206)
> at
> org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:408)
> at
> org.apache.cassandra.service.EmbeddedCassandraService.start(EmbeddedCassandraService.java:52)
> at
> org.apache.cassandra.pig.PigTestBase.startCassandra(PigTestBase.java:96)
> at
> org.apache.cassandra.pig.CqlRecordReaderTest.setup(CqlRecordReaderTest.java:63)
> at
> org.apache.log4j.Log4jLoggerFactory.(Log4jLoggerFactory.java:50)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8627) Support Total/Recent latency histogram metrics for range slices
[ https://issues.apache.org/jira/browse/CASSANDRA-8627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko updated CASSANDRA-8627: - Assignee: Sam Tunnicliffe Priority: Major (was: Minor) > Support Total/Recent latency histogram metrics for range slices > --- > > Key: CASSANDRA-8627 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8627 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Chris Lohfink >Assignee: Sam Tunnicliffe > Labels: metrics > Fix For: 2.1.3 > > > The Metrics histogram is pretty bad at non-normal data like latencies and > (empirically tested and theoretically) is untrustworthy at 99th percentile. > For applications that care about the percentiles having the more > statistically accurate version is beneficial. Adding the deprecated methods > like other latency histograms for CASSANDRA-7338 temporarily would help. > This is just for 2.1 branch. CASSANDRA-5657 solves everything in 3.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8622) All of pig-test is failing in trunk
[
https://issues.apache.org/jira/browse/CASSANDRA-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278906#comment-14278906
]
Philip Thompson commented on CASSANDRA-8622:
Okay. When checking out the parent commit of c832a36f, I get a different error,
but I will solve my local problem first.
> All of pig-test is failing in trunk
> ---
>
> Key: CASSANDRA-8622
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8622
> Project: Cassandra
> Issue Type: Test
> Components: Hadoop
>Reporter: Philip Thompson
>Assignee: Brandon Williams
> Fix For: 3.0
>
>
> See http://cassci.datastax.com/job/trunk_pigtest/330/testReport/
> Every test in the ant target {{ant pig-test}} has been failing on trunk for a
> while now.
> {code}
> java.lang.ExceptionInInitializerError
> at org.apache.log4j.Logger.getLogger(Logger.java:40)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:48)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:44)
> at org.hyperic.sigar.SigarLog.debug(SigarLog.java:60)
> at org.hyperic.sigar.Sigar.(Sigar.java:108)
> at org.apache.cassandra.utils.SigarLibrary.(SigarLibrary.java:45)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:206)
> at
> org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:408)
> at
> org.apache.cassandra.service.EmbeddedCassandraService.start(EmbeddedCassandraService.java:52)
> at
> org.apache.cassandra.pig.PigTestBase.startCassandra(PigTestBase.java:96)
> at
> org.apache.cassandra.pig.CqlRecordReaderTest.setup(CqlRecordReaderTest.java:63)
> at
> org.apache.log4j.Log4jLoggerFactory.(Log4jLoggerFactory.java:50)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8622) All of pig-test is failing in trunk
[
https://issues.apache.org/jira/browse/CASSANDRA-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278904#comment-14278904
]
Philip Thompson commented on CASSANDRA-8622:
Okay. When checking out the parent commit of c832a36f, I get a different error,
but I will solve my local problem first.
> All of pig-test is failing in trunk
> ---
>
> Key: CASSANDRA-8622
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8622
> Project: Cassandra
> Issue Type: Test
> Components: Hadoop
>Reporter: Philip Thompson
>Assignee: Brandon Williams
> Fix For: 3.0
>
>
> See http://cassci.datastax.com/job/trunk_pigtest/330/testReport/
> Every test in the ant target {{ant pig-test}} has been failing on trunk for a
> while now.
> {code}
> java.lang.ExceptionInInitializerError
> at org.apache.log4j.Logger.getLogger(Logger.java:40)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:48)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:44)
> at org.hyperic.sigar.SigarLog.debug(SigarLog.java:60)
> at org.hyperic.sigar.Sigar.(Sigar.java:108)
> at org.apache.cassandra.utils.SigarLibrary.(SigarLibrary.java:45)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:206)
> at
> org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:408)
> at
> org.apache.cassandra.service.EmbeddedCassandraService.start(EmbeddedCassandraService.java:52)
> at
> org.apache.cassandra.pig.PigTestBase.startCassandra(PigTestBase.java:96)
> at
> org.apache.cassandra.pig.CqlRecordReaderTest.setup(CqlRecordReaderTest.java:63)
> at
> org.apache.log4j.Log4jLoggerFactory.(Log4jLoggerFactory.java:50)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8627) Support Total/Recent latency histogram metrics for range slices
[ https://issues.apache.org/jira/browse/CASSANDRA-8627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko updated CASSANDRA-8627: - Reviewer: Aleksey Yeschenko Fix Version/s: 2.1.3 > Support Total/Recent latency histogram metrics for range slices > --- > > Key: CASSANDRA-8627 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8627 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Chris Lohfink >Assignee: Sam Tunnicliffe > Labels: metrics > Fix For: 2.1.3 > > > The Metrics histogram is pretty bad at non-normal data like latencies and > (empirically tested and theoretically) is untrustworthy at 99th percentile. > For applications that care about the percentiles having the more > statistically accurate version is beneficial. Adding the deprecated methods > like other latency histograms for CASSANDRA-7338 temporarily would help. > This is just for 2.1 branch. CASSANDRA-5657 solves everything in 3.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8622) All of pig-test is failing in trunk
[
https://issues.apache.org/jira/browse/CASSANDRA-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278903#comment-14278903
]
Brandon Williams edited comment on CASSANDRA-8622 at 1/15/15 4:29 PM:
--
I backed c832a36f25c3b out and still get the same error from the description,
so I don't think that's it. :(
was (Author: brandon.williams):
I backed c832a36f25c3b and still get the same error from the description, so I
don't think that's it. :(
> All of pig-test is failing in trunk
> ---
>
> Key: CASSANDRA-8622
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8622
> Project: Cassandra
> Issue Type: Test
> Components: Hadoop
>Reporter: Philip Thompson
>Assignee: Brandon Williams
> Fix For: 3.0
>
>
> See http://cassci.datastax.com/job/trunk_pigtest/330/testReport/
> Every test in the ant target {{ant pig-test}} has been failing on trunk for a
> while now.
> {code}
> java.lang.ExceptionInInitializerError
> at org.apache.log4j.Logger.getLogger(Logger.java:40)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:48)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:44)
> at org.hyperic.sigar.SigarLog.debug(SigarLog.java:60)
> at org.hyperic.sigar.Sigar.(Sigar.java:108)
> at org.apache.cassandra.utils.SigarLibrary.(SigarLibrary.java:45)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:206)
> at
> org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:408)
> at
> org.apache.cassandra.service.EmbeddedCassandraService.start(EmbeddedCassandraService.java:52)
> at
> org.apache.cassandra.pig.PigTestBase.startCassandra(PigTestBase.java:96)
> at
> org.apache.cassandra.pig.CqlRecordReaderTest.setup(CqlRecordReaderTest.java:63)
> at
> org.apache.log4j.Log4jLoggerFactory.(Log4jLoggerFactory.java:50)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8622) All of pig-test is failing in trunk
[
https://issues.apache.org/jira/browse/CASSANDRA-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278903#comment-14278903
]
Brandon Williams commented on CASSANDRA-8622:
-
I backed c832a36f25c3b and still get the same error from the description, so I
don't think that's it. :(
> All of pig-test is failing in trunk
> ---
>
> Key: CASSANDRA-8622
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8622
> Project: Cassandra
> Issue Type: Test
> Components: Hadoop
>Reporter: Philip Thompson
>Assignee: Brandon Williams
> Fix For: 3.0
>
>
> See http://cassci.datastax.com/job/trunk_pigtest/330/testReport/
> Every test in the ant target {{ant pig-test}} has been failing on trunk for a
> while now.
> {code}
> java.lang.ExceptionInInitializerError
> at org.apache.log4j.Logger.getLogger(Logger.java:40)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:48)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:44)
> at org.hyperic.sigar.SigarLog.debug(SigarLog.java:60)
> at org.hyperic.sigar.Sigar.(Sigar.java:108)
> at org.apache.cassandra.utils.SigarLibrary.(SigarLibrary.java:45)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:206)
> at
> org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:408)
> at
> org.apache.cassandra.service.EmbeddedCassandraService.start(EmbeddedCassandraService.java:52)
> at
> org.apache.cassandra.pig.PigTestBase.startCassandra(PigTestBase.java:96)
> at
> org.apache.cassandra.pig.CqlRecordReaderTest.setup(CqlRecordReaderTest.java:63)
> at
> org.apache.log4j.Log4jLoggerFactory.(Log4jLoggerFactory.java:50)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8627) Support Total/Recent latency histogram metrics for range slices
[ https://issues.apache.org/jira/browse/CASSANDRA-8627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Lohfink updated CASSANDRA-8627: - Priority: Minor (was: Major) > Support Total/Recent latency histogram metrics for range slices > --- > > Key: CASSANDRA-8627 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8627 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Chris Lohfink >Priority: Minor > Labels: metrics > > The Metrics histogram is pretty bad at non-normal data like latencies and > (empirically tested and theoretically) is untrustworthy at 99th percentile. > For applications that care about the percentiles having the more > statistically accurate version is beneficial. Adding the deprecated methods > like other latency histograms for CASSANDRA-7338 temporarily would help. > This is just for 2.1 branch. CASSANDRA-5657 solves everything in 3.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-8627) Support Total/Recent latency histogram metrics for range slices
Chris Lohfink created CASSANDRA-8627: Summary: Support Total/Recent latency histogram metrics for range slices Key: CASSANDRA-8627 URL: https://issues.apache.org/jira/browse/CASSANDRA-8627 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Chris Lohfink The Metrics histogram is pretty bad at non-normal data like latencies and (empirically tested and theoretically) is untrustworthy at 99th percentile. For applications that care about the percentiles having the more statistically accurate version is beneficial. Adding the deprecated methods like other latency histograms for CASSANDRA-7338 temporarily would help. This is just for 2.1 branch. CASSANDRA-5657 solves everything in 3.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8596) Display datacenter/rack info for offline nodes - PropertyFileSnitch
[
https://issues.apache.org/jira/browse/CASSANDRA-8596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278889#comment-14278889
]
Brandon Williams commented on CASSANDRA-8596:
-
I can't apply this patch:
{noformat}
patch: malformed patch at line 12: @@ -33,7 +34,11 @@ import
org.apache.cassandra.service.StorageService;
{noformat}
> Display datacenter/rack info for offline nodes - PropertyFileSnitch
> ---
>
> Key: CASSANDRA-8596
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8596
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Vova
>Priority: Minor
> Attachments: status.diff
>
>
> When using GossipPropertyFileSnitch "nodetool status" shows default (from
> cassandra-topology.properties ) datacenter/rack for offline nodes.
> It happens because offline nodes are not in endpointMap, and thus
> getRawEndpointInfo returns default DC/rack is returned
> (PropertyFileSnitch.java).
> I suggest to take info for those nodes from system.peers tables - just like
> SELECT data_center,rack FROM system.peers WHERE peer='10.0.0.1'
> Patch attached.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-5657) remove deprecated metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] T Jake Luciani updated CASSANDRA-5657: -- Attachment: 5657v2.txt Updated version that accomplishes a few things based on previous comments and other feedback. Github branch http://github.com/tjake/cassandra/tree/5657 * Removes all deprecated metrics * Replaces default histogram/timer reservoir with our EstimatedHistogram * Upgrade metrics project to 3.1.0 (required for second part) The new metrics has pretty a poor JMXReporter so I had to create a CassandraMetricsRegistry for us that keeps the old style mbeans we had in place. > remove deprecated metrics > - > > Key: CASSANDRA-5657 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5657 > Project: Cassandra > Issue Type: Task > Components: Tools >Reporter: Jonathan Ellis >Assignee: T Jake Luciani > Labels: technical_debt > Fix For: 3.0 > > Attachments: 5657.txt, 5657v2.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-8479) Timeout Exception on Node Failure in Remote Data Center
[ https://issues.apache.org/jira/browse/CASSANDRA-8479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe resolved CASSANDRA-8479. Resolution: Not a Problem > Timeout Exception on Node Failure in Remote Data Center > --- > > Key: CASSANDRA-8479 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8479 > Project: Cassandra > Issue Type: Bug > Components: API, Core, Tools > Environment: Unix, Cassandra 2.0.11 >Reporter: Amit Singh Chowdhery >Assignee: Sam Tunnicliffe >Priority: Minor > Attachments: TRACE_LOGS.zip > > > Issue Faced : > We have a Geo-red setup with 2 Data centers having 3 nodes each. When we > bring down a single Cassandra node down in DC2 by kill -9 , > reads fail on DC1 with TimedOutException for a brief amount of time (15-20 > sec~). > Reference : > Already a ticket has been opened/resolved and link is provided below : > https://issues.apache.org/jira/browse/CASSANDRA-8352 > Activity Done as per Resolution Provided : > Upgraded to Cassandra 2.0.11 . > We have two 3 node clusters in two different DCs and if one or more of the > nodes go down in one Data Center , ~5-10% traffic failure is observed on the > other. > CL: LOCAL_QUORUM > RF=3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7409) Allow multiple overlapping sstables in L1
[ https://issues.apache.org/jira/browse/CASSANDRA-7409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278871#comment-14278871 ] Carl Yeksigian commented on CASSANDRA-7409: --- It works in much the same way as was described before; except that: - When promoting to a level which is not overlapping, the selection will take all overlapping sstables to compact, rather than just a single one. This is how it was supposed to work initially, but was actually hitting a different case instead. - When selecting the level which needs to be compacted, it starts from 0 going up to MOLO - 1, then from the max level down to MOLO. This means that we try to compact any overlapping levels first, and if we don't find anything, then we'll compact according to our previous ways of compacting. I'm a little suspicious of the MOLO=0 results, now that I've specified the changes; it should more closely mirror the results I got from LCS w/o STCS. It is sensitive to the length of the test; L5 allows for as many overlaps as necessary in this test. The levels shouldn't have any overlap, though; it won't stop compactions until none of the levels have overlap. The L2 runs slower because it is doing more I/O because of the non-overlapping. > Allow multiple overlapping sstables in L1 > - > > Key: CASSANDRA-7409 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7409 > Project: Cassandra > Issue Type: Improvement >Reporter: Carl Yeksigian >Assignee: Carl Yeksigian > Labels: compaction > Fix For: 3.0 > > > Currently, when a normal L0 compaction takes place (not STCS), we take up to > MAX_COMPACTING_L0 L0 sstables and all of the overlapping L1 sstables and > compact them together. If we didn't have to deal with the overlapping L1 > tables, we could compact a higher number of L0 sstables together into a set > of non-overlapping L1 sstables. > This could be done by delaying the invariant that L1 has no overlapping > sstables. Going from L1 to L2, we would be compacting fewer sstables together > which overlap. > When reading, we will not have the same one sstable per level (except L0) > guarantee, but this can be bounded (once we have too many sets of sstables, > either compact them back into the same level, or compact them up to the next > level). > This could be generalized to allow any level to be the maximum for this > overlapping strategy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8622) All of pig-test is failing in trunk
[
https://issues.apache.org/jira/browse/CASSANDRA-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278860#comment-14278860
]
Philip Thompson commented on CASSANDRA-8622:
I think I've busted my environment somehow. I get this same exception no matter
how far back on trunk I go, and on cassandra-2.1. It does still seem that the
sigar change at c832a36f25c3b broke pig-test though.
> All of pig-test is failing in trunk
> ---
>
> Key: CASSANDRA-8622
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8622
> Project: Cassandra
> Issue Type: Test
> Components: Hadoop
>Reporter: Philip Thompson
>Assignee: Brandon Williams
> Fix For: 3.0
>
>
> See http://cassci.datastax.com/job/trunk_pigtest/330/testReport/
> Every test in the ant target {{ant pig-test}} has been failing on trunk for a
> while now.
> {code}
> java.lang.ExceptionInInitializerError
> at org.apache.log4j.Logger.getLogger(Logger.java:40)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:48)
> at org.hyperic.sigar.SigarLog.getLogger(SigarLog.java:44)
> at org.hyperic.sigar.SigarLog.debug(SigarLog.java:60)
> at org.hyperic.sigar.Sigar.(Sigar.java:108)
> at org.apache.cassandra.utils.SigarLibrary.(SigarLibrary.java:45)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:206)
> at
> org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:408)
> at
> org.apache.cassandra.service.EmbeddedCassandraService.start(EmbeddedCassandraService.java:52)
> at
> org.apache.cassandra.pig.PigTestBase.startCassandra(PigTestBase.java:96)
> at
> org.apache.cassandra.pig.CqlRecordReaderTest.setup(CqlRecordReaderTest.java:63)
> at
> org.apache.log4j.Log4jLoggerFactory.(Log4jLoggerFactory.java:50)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8626) Support for TTL on GRANT
[ https://issues.apache.org/jira/browse/CASSANDRA-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278849#comment-14278849 ] Sam Tunnicliffe commented on CASSANDRA-8626: This is really not a good idea because security is an area where you want to be explicit. If, as an admin, you want to grant temporary access to a resource, make the grant then revoke it when you're done. > Support for TTL on GRANT > > > Key: CASSANDRA-8626 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8626 > Project: Cassandra > Issue Type: Improvement >Reporter: Johnny Miller >Priority: Trivial > > There are a variety of situations where named users are only allowed to be > granted temporary permissions to query Cassandra. > It would be useful if we supported the ability to attach a TTL to permissions > so they automatically expire e.g GRANT SELECT ON ALL KEYSPACES TO johnny > USING TTL 86400; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-8626) Support for TTL on GRANT
[ https://issues.apache.org/jira/browse/CASSANDRA-8626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe resolved CASSANDRA-8626. Resolution: Won't Fix > Support for TTL on GRANT > > > Key: CASSANDRA-8626 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8626 > Project: Cassandra > Issue Type: Improvement >Reporter: Johnny Miller >Priority: Trivial > > There are a variety of situations where named users are only allowed to be > granted temporary permissions to query Cassandra. > It would be useful if we supported the ability to attach a TTL to permissions > so they automatically expire e.g GRANT SELECT ON ALL KEYSPACES TO johnny > USING TTL 86400; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8479) Timeout Exception on Node Failure in Remote Data Center
[ https://issues.apache.org/jira/browse/CASSANDRA-8479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278846#comment-14278846 ] Sam Tunnicliffe commented on CASSANDRA-8479: 1. read_repair_chance=1 means that on every request, digest requests will be sent to all nodes, local and remote so this is almost certainly not what you want. As you're reading and writing at LOCAL_QUORUM, you don't need to worry about read repair. A full read will be done by 1 node in the local dc, and a digest request sent to 1 other local node. This is enough to provide the consistency you're looking for, as should the digest not match the data a full read will be done on both nodes and the data resolved according to timestamp. Since you wrote at LOCAL_QUORUM, at least one of those nodes has the latest value. You should note that this is not exactly "strong consistency" as you describe it - that would only be the case with a single client performing both reads and writes & with no failures. But terminology aside, I believe you can just set read repair chance to 0. 2. What's more relevant than the simple rate is the workload and the delay between writing a row and reading it back. It's certainly possibly for a write to be ack'd by 2 nodes, satisfying LOCAL_QUORUM but not yet be processed by the third node. If a read request targets that third node, either for the full data or a digest you'll get a mismatch. 3. Yes, read_repair_chance=1 means perform a global read repair for 100% of read requests, which is usually overkill (see CASSANDRA-7320). The logs don't show which node(s) returned a non-matching digests, only that the coordinator received 3 responses. Those are likely to have been from the nodes in the same dc, and your experiment with dclocal_read_repair_chance seems to bear that out, but it isn't guaranteed. 4. Whether read repair happens is only determined by read_repair_chance/dclocal_read_repair_chance in 2.0, so CL isn't relevant (and has been removed from the documentation - the version you linked is for Cassandra 1.1). Note, this all disregards any additional data reads being performed for [Rapid Read Protection|http://www.datastax.com/dev/blog/rapid-read-protection-in-cassandra-2-0-2] because from the logs it seems as though you have that set to NONE for this table. > Timeout Exception on Node Failure in Remote Data Center > --- > > Key: CASSANDRA-8479 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8479 > Project: Cassandra > Issue Type: Bug > Components: API, Core, Tools > Environment: Unix, Cassandra 2.0.11 >Reporter: Amit Singh Chowdhery >Assignee: Sam Tunnicliffe >Priority: Minor > Attachments: TRACE_LOGS.zip > > > Issue Faced : > We have a Geo-red setup with 2 Data centers having 3 nodes each. When we > bring down a single Cassandra node down in DC2 by kill -9 , > reads fail on DC1 with TimedOutException for a brief amount of time (15-20 > sec~). > Reference : > Already a ticket has been opened/resolved and link is provided below : > https://issues.apache.org/jira/browse/CASSANDRA-8352 > Activity Done as per Resolution Provided : > Upgraded to Cassandra 2.0.11 . > We have two 3 node clusters in two different DCs and if one or more of the > nodes go down in one Data Center , ~5-10% traffic failure is observed on the > other. > CL: LOCAL_QUORUM > RF=3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7409) Allow multiple overlapping sstables in L1
[ https://issues.apache.org/jira/browse/CASSANDRA-7409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278798#comment-14278798 ] Jonathan Ellis commented on CASSANDRA-7409: --- Also, how sensitive is this to the test duration? Is MOLO=5 the magic number because we generate about 5 L1's worth of data before the test ends? > Allow multiple overlapping sstables in L1 > - > > Key: CASSANDRA-7409 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7409 > Project: Cassandra > Issue Type: Improvement >Reporter: Carl Yeksigian >Assignee: Carl Yeksigian > Labels: compaction > Fix For: 3.0 > > > Currently, when a normal L0 compaction takes place (not STCS), we take up to > MAX_COMPACTING_L0 L0 sstables and all of the overlapping L1 sstables and > compact them together. If we didn't have to deal with the overlapping L1 > tables, we could compact a higher number of L0 sstables together into a set > of non-overlapping L1 sstables. > This could be done by delaying the invariant that L1 has no overlapping > sstables. Going from L1 to L2, we would be compacting fewer sstables together > which overlap. > When reading, we will not have the same one sstable per level (except L0) > guarantee, but this can be bounded (once we have too many sets of sstables, > either compact them back into the same level, or compact them up to the next > level). > This could be generalized to allow any level to be the maximum for this > overlapping strategy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8622) All of pig-test is failing in trunk
[
https://issues.apache.org/jira/browse/CASSANDRA-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278795#comment-14278795
]
Philip Thompson commented on CASSANDRA-8622:
It appears to have first been broken by
{code}
commit c832a36f25c3b54f2a12a865455368e5243682bd
Author: Dave Brosius
Date: Sat Nov 8 23:12:04 2014 -0500
route sigar's use of log4j thru slf4j
commit d286ac7d072fe0a2672baebb0f95b9c91676e8f7
Author: Dave Brosius
Date: Sun Nov 9 00:38:09 2014 -0500
fix license extension
{code}
Before that however, the tests were erroring with "Cannot get schema" errors
such as:
{code}
[junit] Error during parsing. Cannot get schema from loadFunc
org.apache.cassandra.hadoop.pig.CassandraStorage
[junit] org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000:
Error during parsing. Cannot get schema from loadFunc
org.apache.cassandra.hadoop.pig.CassandraStorage
[junit] at
org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1691)
[junit] at
org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1625)
[junit] at org.apache.pig.PigServer.registerQuery(PigServer.java:577)
[junit] at org.apache.pig.PigServer.registerQuery(PigServer.java:590)
[junit] at
org.apache.cassandra.pig.ThriftColumnFamilyTest.testCassandraStorageFullCopy(ThriftColumnFamilyTest.java:349)
[junit] Caused by: Failed to parse: Can not retrieve schema from loader
org.apache.cassandra.hadoop.pig.CassandraStorage@4a853012
[junit] at
org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:198)
[junit] at
org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1678)
[junit] Caused by: java.lang.RuntimeException: Can not retrieve schema from
loader org.apache.cassandra.hadoop.pig.CassandraStorage@4a853012
[junit] at
org.apache.pig.newplan.logical.relational.LOLoad.(LOLoad.java:91)
[junit] at
org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:886)
[junit] at
org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568)
[junit] at
org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625)
[junit] at
org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
[junit] at
org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
[junit] at
org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
[junit] at
org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188)
[junit] Caused by: org.apache.pig.impl.logicalLayer.FrontendException:
ERROR 2245: Cannot get schema from loadFunc
org.apache.cassandra.hadoop.pig.CassandraStorage
[junit] at
org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:179)
[junit] at
org.apache.pig.newplan.logical.relational.LOLoad.(LOLoad.java:89)
[junit] Caused by: java.io.IOException: java.io.IOException: Unable to
connect to server localhost:9160
[junit] at
org.apache.cassandra.hadoop.pig.AbstractCassandraStorage.initSchema(AbstractCassandraStorage.java:537)
[junit] at
org.apache.cassandra.hadoop.pig.CassandraStorage.setLocation(CassandraStorage.java:339)
[junit] at
org.apache.cassandra.hadoop.pig.CassandraStorage.getSchema(CassandraStorage.java:388)
[junit] at
org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:175)
[junit] Caused by: java.io.IOException: Unable to connect to server
localhost:9160
[junit] at
org.apache.cassandra.hadoop.ConfigHelper.createConnection(ConfigHelper.java:574)
[junit] at
org.apache.cassandra.hadoop.ConfigHelper.getClientFromAddressList(ConfigHelper.java:545)
[junit] at
org.apache.cassandra.hadoop.ConfigHelper.getClientFromInputAddressList(ConfigHelper.java:529)
[junit] at
org.apache.cassandra.hadoop.pig.AbstractCassandraStorage.initSchema(AbstractCassandraStorage.java:497)
[junit] Caused by: org.apache.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused
[junit] at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
[junit] at
org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
[junit] at
org.apache.cassandra.thrift.TFramedTransportFactory.openTransport(TFramedTransportFactory.java:41)
[junit] at
org.apache.cassandra.hadoop.ConfigHelper.createConnection(ConfigHelper.java:569)
[junit] Caused by: java.net.ConnectException: Connection refused
[junit] at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit] at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit] at
java.net.AbstractPlainSocketImpl.connectToAddress
[jira] [Commented] (CASSANDRA-7409) Allow multiple overlapping sstables in L1
[ https://issues.apache.org/jira/browse/CASSANDRA-7409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278788#comment-14278788 ] Jonathan Ellis commented on CASSANDRA-7409: --- What does your updated algorithm look like now? > Allow multiple overlapping sstables in L1 > - > > Key: CASSANDRA-7409 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7409 > Project: Cassandra > Issue Type: Improvement >Reporter: Carl Yeksigian >Assignee: Carl Yeksigian > Labels: compaction > Fix For: 3.0 > > > Currently, when a normal L0 compaction takes place (not STCS), we take up to > MAX_COMPACTING_L0 L0 sstables and all of the overlapping L1 sstables and > compact them together. If we didn't have to deal with the overlapping L1 > tables, we could compact a higher number of L0 sstables together into a set > of non-overlapping L1 sstables. > This could be done by delaying the invariant that L1 has no overlapping > sstables. Going from L1 to L2, we would be compacting fewer sstables together > which overlap. > When reading, we will not have the same one sstable per level (except L0) > guarantee, but this can be bounded (once we have too many sets of sstables, > either compact them back into the same level, or compact them up to the next > level). > This could be generalized to allow any level to be the maximum for this > overlapping strategy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[2/2] cassandra git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/076b5bdd Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/076b5bdd Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/076b5bdd Branch: refs/heads/trunk Commit: 076b5bdd10e14f4da5cc46863d69410900042b06 Parents: e245ba7 d406bab Author: Aleksey Yeschenko Authored: Thu Jan 15 17:51:44 2015 +0300 Committer: Aleksey Yeschenko Committed: Thu Jan 15 17:51:44 2015 +0300 -- CHANGES.txt | 1 + bin/cqlsh | 5 - 2 files changed, 5 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/076b5bdd/CHANGES.txt -- diff --cc CHANGES.txt index 30a741e,94a2dbd..37630e6 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,55 -1,5 +1,56 @@@ +3.0 + * Add role based access control (CASSANDRA-7653) + * Group sstables for anticompaction correctly (CASSANDRA-8578) + * Add ReadFailureException to native protocol, respond + immediately when replicas encounter errors while handling + a read request (CASSANDRA-7886) + * Switch CommitLogSegment from RandomAccessFile to nio (CASSANDRA-8308) + * Allow mixing token and partition key restrictions (CASSANDRA-7016) + * Support index key/value entries on map collections (CASSANDRA-8473) + * Modernize schema tables (CASSANDRA-8261) + * Support for user-defined aggregation functions (CASSANDRA-8053) + * Fix NPE in SelectStatement with empty IN values (CASSANDRA-8419) + * Refactor SelectStatement, return IN results in natural order instead + of IN value list order (CASSANDRA-7981) + * Support UDTs, tuples, and collections in user-defined + functions (CASSANDRA-7563) + * Fix aggregate fn results on empty selection, result column name, + and cqlsh parsing (CASSANDRA-8229) + * Mark sstables as repaired after full repair (CASSANDRA-7586) + * Extend Descriptor to include a format value and refactor reader/writer + APIs (CASSANDRA-7443) + * Integrate JMH for microbenchmarks (CASSANDRA-8151) + * Keep sstable levels when bootstrapping (CASSANDRA-7460) + * Add Sigar library and perform basic OS settings check on startup (CASSANDRA-7838) + * Support for aggregation functions (CASSANDRA-4914) + * Remove cassandra-cli (CASSANDRA-7920) + * Accept dollar quoted strings in CQL (CASSANDRA-7769) + * Make assassinate a first class command (CASSANDRA-7935) + * Support IN clause on any clustering column (CASSANDRA-4762) + * Improve compaction logging (CASSANDRA-7818) + * Remove YamlFileNetworkTopologySnitch (CASSANDRA-7917) + * Do anticompaction in groups (CASSANDRA-6851) + * Support user-defined functions (CASSANDRA-7395, 7526, 7562, 7740, 7781, 7929, + 7924, 7812, 8063, 7813, 7708) + * Permit configurable timestamps with cassandra-stress (CASSANDRA-7416) + * Move sstable RandomAccessReader to nio2, which allows using the + FILE_SHARE_DELETE flag on Windows (CASSANDRA-4050) + * Remove CQL2 (CASSANDRA-5918) + * Add Thrift get_multi_slice call (CASSANDRA-6757) + * Optimize fetching multiple cells by name (CASSANDRA-6933) + * Allow compilation in java 8 (CASSANDRA-7028) + * Make incremental repair default (CASSANDRA-7250) + * Enable code coverage thru JaCoCo (CASSANDRA-7226) + * Switch external naming of 'column families' to 'tables' (CASSANDRA-4369) + * Shorten SSTable path (CASSANDRA-6962) + * Use unsafe mutations for most unit tests (CASSANDRA-6969) + * Fix race condition during calculation of pending ranges (CASSANDRA-7390) + * Fail on very large batch sizes (CASSANDRA-8011) + * Improve concurrency of repair (CASSANDRA-6455, 8208) + + 2.1.3 + * (cqlsh) Escape clqshrc passwords properly (CASSANDRA-8618) * Fix NPE when passing wrong argument in ALTER TABLE statement (CASSANDRA-8355) * Pig: Refactor and deprecate CqlStorage (CASSANDRA-8599) * Don't reuse the same cleanup strategy for all sstables (CASSANDRA-8537) http://git-wip-us.apache.org/repos/asf/cassandra/blob/076b5bdd/bin/cqlsh --
[1/2] cassandra git commit: (cqlsh) Escape clqshrc passwords properly
Repository: cassandra Updated Branches: refs/heads/trunk e245ba7af -> 076b5bdd1 (cqlsh) Escape clqshrc passwords properly patch by Johnny Miller; reviewed by Aleksey Yeschenko for CASSANDRA-8618 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d406bab5 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d406bab5 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d406bab5 Branch: refs/heads/trunk Commit: d406bab504f558296cba0ccfb2011f55b894b0e8 Parents: 2ff9137 Author: Johnny Miller Authored: Thu Jan 15 17:49:31 2015 +0300 Committer: Aleksey Yeschenko Committed: Thu Jan 15 17:50:35 2015 +0300 -- CHANGES.txt | 1 + bin/cqlsh | 5 - 2 files changed, 5 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/d406bab5/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index d2802ad..94a2dbd 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 2.1.3 + * (cqlsh) Escape clqshrc passwords properly (CASSANDRA-8618) * Fix NPE when passing wrong argument in ALTER TABLE statement (CASSANDRA-8355) * Pig: Refactor and deprecate CqlStorage (CASSANDRA-8599) * Don't reuse the same cleanup strategy for all sstables (CASSANDRA-8537) http://git-wip-us.apache.org/repos/asf/cassandra/blob/d406bab5/bin/cqlsh -- diff --git a/bin/cqlsh b/bin/cqlsh index 363a4f6..0254fad 100755 --- a/bin/cqlsh +++ b/bin/cqlsh @@ -1905,9 +1905,12 @@ def read_options(cmdlineargs, environment): configs = ConfigParser.SafeConfigParser() configs.read(CONFIG_FILE) +rawconfigs = ConfigParser.RawConfigParser() +rawconfigs.read(CONFIG_FILE) + optvalues = optparse.Values() optvalues.username = option_with_default(configs.get, 'authentication', 'username') -optvalues.password = option_with_default(configs.get, 'authentication', 'password') +optvalues.password = option_with_default(rawconfigs.get, 'authentication', 'password') optvalues.keyspace = option_with_default(configs.get, 'authentication', 'keyspace') optvalues.completekey = option_with_default(configs.get, 'ui', 'completekey', DEFAULT_COMPLETEKEY)
cassandra git commit: (cqlsh) Escape clqshrc passwords properly
Repository: cassandra Updated Branches: refs/heads/cassandra-2.1 2ff913767 -> d406bab50 (cqlsh) Escape clqshrc passwords properly patch by Johnny Miller; reviewed by Aleksey Yeschenko for CASSANDRA-8618 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d406bab5 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d406bab5 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d406bab5 Branch: refs/heads/cassandra-2.1 Commit: d406bab504f558296cba0ccfb2011f55b894b0e8 Parents: 2ff9137 Author: Johnny Miller Authored: Thu Jan 15 17:49:31 2015 +0300 Committer: Aleksey Yeschenko Committed: Thu Jan 15 17:50:35 2015 +0300 -- CHANGES.txt | 1 + bin/cqlsh | 5 - 2 files changed, 5 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/d406bab5/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index d2802ad..94a2dbd 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 2.1.3 + * (cqlsh) Escape clqshrc passwords properly (CASSANDRA-8618) * Fix NPE when passing wrong argument in ALTER TABLE statement (CASSANDRA-8355) * Pig: Refactor and deprecate CqlStorage (CASSANDRA-8599) * Don't reuse the same cleanup strategy for all sstables (CASSANDRA-8537) http://git-wip-us.apache.org/repos/asf/cassandra/blob/d406bab5/bin/cqlsh -- diff --git a/bin/cqlsh b/bin/cqlsh index 363a4f6..0254fad 100755 --- a/bin/cqlsh +++ b/bin/cqlsh @@ -1905,9 +1905,12 @@ def read_options(cmdlineargs, environment): configs = ConfigParser.SafeConfigParser() configs.read(CONFIG_FILE) +rawconfigs = ConfigParser.RawConfigParser() +rawconfigs.read(CONFIG_FILE) + optvalues = optparse.Values() optvalues.username = option_with_default(configs.get, 'authentication', 'username') -optvalues.password = option_with_default(configs.get, 'authentication', 'password') +optvalues.password = option_with_default(rawconfigs.get, 'authentication', 'password') optvalues.keyspace = option_with_default(configs.get, 'authentication', 'keyspace') optvalues.completekey = option_with_default(configs.get, 'ui', 'completekey', DEFAULT_COMPLETEKEY)
[jira] [Commented] (CASSANDRA-8594) pidfile is never filled by cassandra
[
https://issues.apache.org/jira/browse/CASSANDRA-8594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278742#comment-14278742
]
Cyril Scetbon commented on CASSANDRA-8594:
--
[~JoshuaMcKenzie] you're right, the line you referenced creates the pid.
However, I just discovered that my debian script is using the foreground option
:
{code}
( nohup start-stop-daemon -S -c cassandra -a /usr/sbin/cassandra -p "$PIDFILE"
--umask "30" -- \
-f -p "$PIDFILE" -H "$heap_dump_f" -E "$error_log_f" >
$CASSANDRA_LOG_DIR/output.log 2>&1 || return 2 ) &
{code}
and that's why the pid is not created. I need to track why mine is using this
option and where it comes from. Thank you guys
> pidfile is never filled by cassandra
>
>
> Key: CASSANDRA-8594
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8594
> Project: Cassandra
> Issue Type: Bug
>Reporter: Cyril Scetbon
>Assignee: Cyril Scetbon
> Fix For: 2.0.13
>
>
> The pid file is never filled by cassandra. there is only a File object that
> is created with [those
> lines|https://github.com/cscetbon/cassandra/blob/cassandra-2.0.10/src/java/org/apache/cassandra/service/CassandraDaemon.java#L498-L501]
> Here is a
> [fix|https://github.com/cscetbon/cassandra/commit/d0c5e0c9be00e48e6d0cd0de208c53274f1919c0.patch]
> that writes the current PID into the pidfile
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8421) Cassandra 2.1.1 & Cassandra 2.1.2 UDT not returning value for LIST type as UDT
[
https://issues.apache.org/jira/browse/CASSANDRA-8421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278688#comment-14278688
]
Benjamin Lerer commented on CASSANDRA-8421:
---
Good news: I can reproduce the problem using cqlsh.
> Cassandra 2.1.1 & Cassandra 2.1.2 UDT not returning value for LIST type as UDT
> --
>
> Key: CASSANDRA-8421
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8421
> Project: Cassandra
> Issue Type: Bug
> Components: API
> Environment: single node cassandra
>Reporter: madheswaran
>Assignee: Benjamin Lerer
> Fix For: 3.0, 2.1.3
>
> Attachments: 8421-unittest.txt, entity_data.csv
>
>
> I using List and its data type is UDT.
> UDT:
> {code}
> CREATE TYPE
> fieldmap (
> key text,
> value text
> );
> {code}
> TABLE:
> {code}
> CREATE TABLE entity (
> entity_id uuid PRIMARY KEY,
> begining int,
> domain text,
> domain_type text,
> entity_template_name text,
> field_values list,
> global_entity_type text,
> revision_time timeuuid,
> status_key int,
> status_name text,
> uuid timeuuid
> ) {code}
> INDEX:
> {code}
> CREATE INDEX entity_domain_idx_1 ON galaxy_dev.entity (domain);
> CREATE INDEX entity_field_values_idx_1 ON galaxy_dev.entity (field_values);
> CREATE INDEX entity_global_entity_type_idx_1 ON galaxy_dev.entity (gen_type );
> {code}
> QUERY
> {code}
> SELECT * FROM entity WHERE status_key < 3 and field_values contains {key:
> 'userName', value: 'Sprint5_22'} and gen_type = 'USER' and domain =
> 'S4_1017.abc.com' allow filtering;
> {code}
> The above query return value for some row and not for many rows but those
> rows and data's are exist.
> Observation:
> If I execute query with other than field_maps, then it returns value. I
> suspect the problem with LIST with UDT.
> I have single node cassadra DB. Please let me know why this strange behavior
> from cassandra.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8479) Timeout Exception on Node Failure in Remote Data Center
[ https://issues.apache.org/jira/browse/CASSANDRA-8479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278685#comment-14278685 ] Anuj commented on CASSANDRA-8479: - You are correct. These logs are from 2.0.3. As suggested in CASSANDRA 8352, we upgraded to 2.0.11 and tested the issue. Same issue was observed there. We are using read_repair_chance=1,dclocal_read_repair_chance=0. When we made read_repair_chance=0, killing node in local and remote DC didn't lead to any read failures:) We need your help in understanding the following points: 1. We are using strong consistency i.e. LOCAL_QUORUM for reads & writes. So, even if one of the replicas is having obsolete value, we will read latest value next time we read data. Does that mean read_repair_chance=1 is not required when LOCAL_QUORUM is used for both reads and writes? We must have read_repair_chance=0, that will give us better performance without sacrificing consistency? What is your recommendation? 2. We are writing to Cassandra at high speeds. Is that the reason we are getting digest mismatch during read repair? And that's when Cassandra goes for CL.ALL irrespective of the fact that we are using CL.LOCAL_QUORUM? 3. I think read_repair is comparing digests from replicas in remote DC also? Isn't that a performance hit? We are currently using Cassandra in Active-Passive mode so updating remote DC fast is not our priority. What's recommended? I tried setting dclocal_read_repair_chance=1 and read_repair_chance=0 in order to make sure that read repairs are only executed within the DC. But I noticed that killing local node didn't caused any read failures. Does that mean the digest mismatch problem occurs with node at remote DC rather than digest of third node which didn't participated in read LOCAL_QUORUM? 4.Documentation at http://www.datastax.com/docs/1.1/configuration/storage_configuration says that read_repair_chance specifies the probability with which read repairs should be invoked on "non-quorum reads". What is the significance of "non-Quorum" here? We are using LOCAL_QUORUM and still read repair is coming into picture. Yes. We misunderstood Tracing. Now that you have identified the issue, do you still need Tracing? > Timeout Exception on Node Failure in Remote Data Center > --- > > Key: CASSANDRA-8479 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8479 > Project: Cassandra > Issue Type: Bug > Components: API, Core, Tools > Environment: Unix, Cassandra 2.0.11 >Reporter: Amit Singh Chowdhery >Assignee: Sam Tunnicliffe >Priority: Minor > Attachments: TRACE_LOGS.zip > > > Issue Faced : > We have a Geo-red setup with 2 Data centers having 3 nodes each. When we > bring down a single Cassandra node down in DC2 by kill -9 , > reads fail on DC1 with TimedOutException for a brief amount of time (15-20 > sec~). > Reference : > Already a ticket has been opened/resolved and link is provided below : > https://issues.apache.org/jira/browse/CASSANDRA-8352 > Activity Done as per Resolution Provided : > Upgraded to Cassandra 2.0.11 . > We have two 3 node clusters in two different DCs and if one or more of the > nodes go down in one Data Center , ~5-10% traffic failure is observed on the > other. > CL: LOCAL_QUORUM > RF=3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Git Push Summary
Repository: cassandra Updated Tags: refs/tags/2.0.12-tentative [created] 90780b550
