Re: [Cassandra 3.0.9] Cannot allocate memory

2017-03-22 Thread Nate McCall 
On Thu, Mar 23, 2017 at 11:18 AM, Abhishek Kumar Maheshwari <
abhishek.maheshw...@timesinternet.in> wrote:

> JVM config is as below:
>
>
>
> -Xms16G
>
> -Xmx16G
>
> -Xmn3000M
>
>
>

I don't think it is the cause, but you need to remove Xmn when using G1GC.

RE: [Cassandra 3.0.9] Cannot allocate memory

2017-03-22 Thread Abhishek Kumar Maheshwari 
JVM config is as below:

-Xms16G
-Xmx16G
-Xmn3000M

What I need to check in dmesg?

From: Thakrar, Jayesh [mailto:jthak...@conversantmedia.com]
Sent: 23 March 2017 03:39
To: Abhishek Kumar Maheshwari ; 
user@cassandra.apache.org
Subject: RE: [Cassandra 3.0.9] Cannot allocate memory


And what is the configured max heap?
Sometimes you may also be able to see some useful messages in "dmesg" output.

Jayesh


From: Abhishek Kumar Maheshwari 
>
Sent: Wednesday, March 22, 2017 5:05:14 PM
To: Thakrar, Jayesh; user@cassandra.apache.org
Subject: RE: [Cassandra 3.0.9] Cannot allocate memory

No only Cassandra is running on these servers.

From: Thakrar, Jayesh [mailto:jthak...@conversantmedia.com]
Sent: 22 March 2017 22:27
To: Abhishek Kumar Maheshwari 
>;
 user@cassandra.apache.org
Subject: Re: [Cassandra 3.0.9] Cannot allocate memory

Is/are the Cassandra server(s) shared?
E.g. do they run mesos + spark?

From: Abhishek Kumar Maheshwari 
>
Date: Wednesday, March 22, 2017 at 12:45 AM
To: "user@cassandra.apache.org" 
>
Subject: [Cassandra 3.0.9] Cannot allocate memory

Hi all,

I am using Cassandra 3.0.9. while I am adding new server after some time I am 
getting below exception. JVM option is attaches.
Hardware info:
Ram 64 GB.
Core: 40


Java HotSpot(TM) 64-Bit Server VM warning: INFO: 
os::commit_memory(0x7fe9c44ee000, 12288, 0) failed; error='Cannot allocate 
memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing 
reserved memory.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: 
os::commit_memory(0x7f5c056ab000, 12288, 0) failed; error='Cannot allocate 
memory' (errno=12)
[thread 140033204860672 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: 
os::commit_memory(0x7f5c0566a000, 12288, 0) failed; error='Cannot allocate 
memory' (errno=12)
[thread 140033204594432 also had an error]Java HotSpot(TM) 64-Bit Server VM 
warning:
INFO: os::commit_memory(0x7fe9c420c000, 12288, 0) failed; error='Cannot 
allocate memory' (errno=12)
Java HotSpot(TM) 64-Bit Server VM warning: [thread 140641994852096 also had an 
error]INFO: os::commit_memory(0x7f5c055a7000, 12288, 0) failed; 
error='Cannot allocate memory' (errno=12)

Please let me know what I miss?

Thanks & Regards,
Abhishek Kumar Maheshwari
+91- 805591 (Mobile)
Times Internet Ltd. | A Times of India Group Company
FC - 6, Sector 16A, Film City,  Noida,  U.P. 201301 | INDIA
P Please do not print this email unless it is absolutely necessary. Spread 
environmental awareness.

YES Bank & The Economic Times Global Business Summit (GBS) is back on 27-28 
March. The 3rd edition of GBS will see participation of 2000+ delegates from 
over 20 countries as they bear witness to leaders sharing insights into how to 
best navigate a dynamic future. Visit www.et-gbs.com

RE: [Cassandra 3.0.9] Cannot allocate memory

2017-03-22 Thread Abhishek Kumar Maheshwari 
The exception is as below:

INFO  17:42:37 Index build of 
til_lineitem_productsku_item_id,til_lineitem_productsku_status complete
WARN  17:43:28 G1 Old Generation GC in 2906ms.  G1 Eden Space: 1560281088 -> 0; 
G1 Old Gen: 3033393144 -> 1127339984; G1 Survivor Space
: 150994944 -> 0;
INFO  17:43:28 Pool NameActive   Pending  Completed   
Blocked  All Time Blocked
Java HotSpot(TM) 64-Bit Server VM warning: INFO: 
os::commit_memory(0x7f9f31a55000, 12288, 0) failed; error='Cannot allocate 
memory'
(errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing 
reserved memory.
# An error report file with more information is saved as:
# /opt/apache-cassandra-3.0.9/bin/hs_err_pid8264.log
INFO  17:43:28 MutationStage 1 0   55409726 
0 0

INFO  17:43:28 ViewMutationStage 0 0  0 
0 0

INFO  17:43:28 ReadStage 0 0  0 
0 0

INFO  17:43:28 RequestResponseStage  0 0 17 
0 0

Java HotSpot(TM) 64-Bit Server VM warning: INFO: 
os::commit_memory(0x7f9f830d8000, 65536, 1) failed; error='Cannot allocate 
memory' (errno=12)INFO  17:43:28 ReadRepairStage   0 0  
0 0 0


[thread 140321932023552 also had an error]
[thread 140321811961600 also had an error]
ERROR 17:43:28 Exception in thread Thread[CompactionExecutor:2482,1,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:156) 
~[apache-cassandra-3.0.9.jar:3.0.9]
at 
org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280) 
~[apache-cassandra-3.0.9.jar:3.0.9]
at 
org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
 ~[apache-cassandra-3.0.9.jar:3.0.9]

From: Abhishek Kumar Maheshwari
Sent: 22 March 2017 11:15
To: user@cassandra.apache.org
Subject: [Cassandra 3.0.9] Cannot allocate memory

Hi all,

I am using Cassandra 3.0.9. while I am adding new server after some time I am 
getting below exception. JVM option is attaches.
Hardware info:
Ram 64 GB.
Core: 40


Java HotSpot(TM) 64-Bit Server VM warning: INFO: 
os::commit_memory(0x7fe9c44ee000, 12288, 0) failed; error='Cannot allocate 
memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing 
reserved memory.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: 
os::commit_memory(0x7f5c056ab000, 12288, 0) failed; error='Cannot allocate 
memory' (errno=12)
[thread 140033204860672 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: 
os::commit_memory(0x7f5c0566a000, 12288, 0) failed; error='Cannot allocate 
memory' (errno=12)
[thread 140033204594432 also had an error]Java HotSpot(TM) 64-Bit Server VM 
warning:
INFO: os::commit_memory(0x7fe9c420c000, 12288, 0) failed; error='Cannot 
allocate memory' (errno=12)
Java HotSpot(TM) 64-Bit Server VM warning: [thread 140641994852096 also had an 
error]INFO: os::commit_memory(0x7f5c055a7000, 12288, 0) failed; 
error='Cannot allocate memory' (errno=12)

Please let me know what I miss?

Thanks & Regards,
Abhishek Kumar Maheshwari
+91- 805591 (Mobile)
Times Internet Ltd. | A Times of India Group Company
FC - 6, Sector 16A, Film City,  Noida,  U.P. 201301 | INDIA
P Please do not print this email unless it is absolutely necessary. Spread 
environmental awareness.

Re: Assertions being hit on Cassandra 3.5 cluster (UnfilteredRowIterators.concat)

2017-03-22 Thread Edward Capriolo 
On Wed, Mar 22, 2017 at 4:34 PM, Daniel Miranda  wrote:

> I found out the problem is conditioned to having the row cache enabled.
> Whenever a query would return an empty result set in a particular table, it
> would fail instead with the exception being thrown in all all nodes.
>
> Disabling the row cache for that particular table fixes the problem.
> Re-enabling has not caused the issue again yet. I do not have row cache
> saving enabled, and the issue persisted between node restarts, which it's
> somewhat strange.
>
> I couldn't reproduce the issue with any other tables. I can produce a
> sanitized dump of the SSTable if a developer desires.
>
> --
> Regards,
> Daniel
>
> On Wed, Mar 22, 2017 at 2:33 PM Daniel Miranda  wrote:
>
>> Thank you for the pointer Michael, I'll try to investigate if this is the
>> same bug I am seeing.
>>
>> I am afraid it might not be, since I'm observing the error periodically,
>> not just during compactions, and the traceback seems different.
>>
>> Regards,
>> Daniel
>>
>> On Wed, Mar 22, 2017 at 1:27 PM Michael Shuler 
>> wrote:
>>
>> Possibly https://issues.apache.org/jira/browse/CASSANDRA-12336, which
>> shows fixed in 3.0.9, 3.8. There are a couple related bug reports listed
>> on there, which you might investigate, as well.
>>
>> --
>> Kind regards,
>> Michael
>>
>> On 03/22/2017 11:21 AM, Daniel Miranda wrote:
>> > Greetings,
>> >
>> > Recently I've started to see the an assertion (traceback follows at the
>> > end of the message) causing exceptions in a 3-node Cassandra 3.5 cluster
>> > (running on Ubuntu 14.04 on Amazon EC2). It seems to happen in all
>> > nodes. Repairs run fine without indicating any errors.
>> >
>> > I can't seem to find any information about it from someone else or any
>> > bug reports.
>> >
>> > Should I bother running an SSTable scrub? Is it a known issue that is
>> > fixed in subsequent versions?
>> >
>> > Thanks in advance,
>> > Daniel
>> >
>> > ---
>> > WARN  [SharedPool-Worker-1] 2017-03-16 18:54:35,587
>> > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on
>> > thread Thread[SharedPool-Worker-1,5,main]: {}
>> > java.lang.AssertionError: null
>> > at
>> > org.apache.cassandra.db.rows.UnfilteredRowIterators.concat(
>> UnfilteredRowIterators.java:157)
>> > ~[apache-cassandra-3.5.jar:3.5]
>> > at
>> > org.apache.cassandra.db.SinglePartitionReadCommand.getThroughCache(
>> SinglePartitionReadCommand.java:420)
>> > ~[apache-cassandra-3.5.jar:3.5]
>> > at
>> > org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(
>> SinglePartitionReadCommand.java:324)
>> > ~[apache-cassandra-3.5.jar:3.5]
>> > at
>> > org.apache.cassandra.db.ReadCommand.executeLocally(
>> ReadCommand.java:366)
>> > ~[apache-cassandra-3.5.jar:3.5]
>> > at
>> > org.apache.cassandra.service.StorageProxy$
>> LocalReadRunnable.runMayThrow(StorageProxy.java:1797)
>> > ~[apache-cassandra-3.5.jar:3.5]
>> > at
>> > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(
>> StorageProxy.java:2466)
>> > ~[apache-cassandra-3.5.jar:3.5]
>> > at
>> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> > ~[na:1.8.0_111]
>> > at
>> > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorServ
>> ice$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
>> > ~[apache-cassandra-3.5.jar:3.5]
>> > at
>> > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorServ
>> ice$LocalSessionFutureTask.run(AbstractLocalAwareExecutorServ
>> ice.java:136)
>> > [apache-cassandra-3.5.jar:3.5]
>> > at
>> > org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>> > [apache-cassandra-3.5.jar:3.5]
>> > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
>> > --
>> > *Daniel Miranda*
>> >
>> > DevOps Engineering Intern
>> > (11) 991959845
>> > www.cobli.co 
>>
>>
I would strongly advice not using the rowcache in the tick-tock series. It
is not on by default and as a result I believe not heavily used. I can not
find reference to it but a few months ago on this list someone pointed out
that it was not doing anything in there version at all. More trouble then
it is worth IMHO just use the standard caching.

Altering of types is not allowed

2017-03-22 Thread Ryan Flynn 
Hi,

I’m fairly new to this and am attempting a pretty basic “change column type” 
example.  I’m receiving an error saying InvalidRequest: Error from server: 
code=2200 [Invalid query] message="Altering of types is not allowed" upon 
attempting to change any column type.

For example, from the DataStax ALTER TABLE CQL Reference 
Page,
 (assuming already USEing a particular keyspace)

CREATE TABLE users (user_name varchar PRIMARY KEY, bio ascii);

Then,

ALTER TABLE users ALTER bio TYPE text;

Results in the error mentioned above.  I saw this JIRA 
Issue, and it’s the only 
thing I can find related to this problem, although that use case is specific to 
changing time-related types.  Could someone tell me if I’m doing something 
wrong or if this is a known issue?  I originally saw this on version 3.10, then 
tried using 3.0.12, and the issue is present there as well.  And I’m just 
running these via cqlsh, if it makes any difference.

Thanks in advance!

Ryan Flynn

RE: [Cassandra 3.0.9] Cannot allocate memory

2017-03-22 Thread Thakrar, Jayesh 
And what is the configured max heap?
Sometimes you may also be able to see some useful messages in "dmesg" output.

Jayesh



From: Abhishek Kumar Maheshwari 
Sent: Wednesday, March 22, 2017 5:05:14 PM
To: Thakrar, Jayesh; user@cassandra.apache.org
Subject: RE: [Cassandra 3.0.9] Cannot allocate memory

No only Cassandra is running on these servers.

From: Thakrar, Jayesh [mailto:jthak...@conversantmedia.com]
Sent: 22 March 2017 22:27
To: Abhishek Kumar Maheshwari ; 
user@cassandra.apache.org
Subject: Re: [Cassandra 3.0.9] Cannot allocate memory

Is/are the Cassandra server(s) shared?
E.g. do they run mesos + spark?

From: Abhishek Kumar Maheshwari 
>
Date: Wednesday, March 22, 2017 at 12:45 AM
To: "user@cassandra.apache.org" 
>
Subject: [Cassandra 3.0.9] Cannot allocate memory

Hi all,

I am using Cassandra 3.0.9. while I am adding new server after some time I am 
getting below exception. JVM option is attaches.
Hardware info:
Ram 64 GB.
Core: 40


Java HotSpot(TM) 64-Bit Server VM warning: INFO: 
os::commit_memory(0x7fe9c44ee000, 12288, 0) failed; error='Cannot allocate 
memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing 
reserved memory.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: 
os::commit_memory(0x7f5c056ab000, 12288, 0) failed; error='Cannot allocate 
memory' (errno=12)
[thread 140033204860672 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: 
os::commit_memory(0x7f5c0566a000, 12288, 0) failed; error='Cannot allocate 
memory' (errno=12)
[thread 140033204594432 also had an error]Java HotSpot(TM) 64-Bit Server VM 
warning:
INFO: os::commit_memory(0x7fe9c420c000, 12288, 0) failed; error='Cannot 
allocate memory' (errno=12)
Java HotSpot(TM) 64-Bit Server VM warning: [thread 140641994852096 also had an 
error]INFO: os::commit_memory(0x7f5c055a7000, 12288, 0) failed; 
error='Cannot allocate memory' (errno=12)

Please let me know what I miss?

Thanks & Regards,
Abhishek Kumar Maheshwari
+91- 805591 (Mobile)
Times Internet Ltd. | A Times of India Group Company
FC - 6, Sector 16A, Film City,  Noida,  U.P. 201301 | INDIA
P Please do not print this email unless it is absolutely necessary. Spread 
environmental awareness.

YES Bank & The Economic Times Global Business Summit (GBS) is back on 27-28 
March. The 3rd edition of GBS will see participation of 2000+ delegates from 
over 20 countries as they bear witness to leaders sharing insights into how to 
best navigate a dynamic future. Visit www.et-gbs.com

RE: [Cassandra 3.0.9] Cannot allocate memory

2017-03-22 Thread Abhishek Kumar Maheshwari 
No only Cassandra is running on these servers.

From: Thakrar, Jayesh [mailto:jthak...@conversantmedia.com]
Sent: 22 March 2017 22:27
To: Abhishek Kumar Maheshwari ; 
user@cassandra.apache.org
Subject: Re: [Cassandra 3.0.9] Cannot allocate memory

Is/are the Cassandra server(s) shared?
E.g. do they run mesos + spark?

From: Abhishek Kumar Maheshwari 
>
Date: Wednesday, March 22, 2017 at 12:45 AM
To: "user@cassandra.apache.org" 
>
Subject: [Cassandra 3.0.9] Cannot allocate memory

Hi all,

I am using Cassandra 3.0.9. while I am adding new server after some time I am 
getting below exception. JVM option is attaches.
Hardware info:
Ram 64 GB.
Core: 40


Java HotSpot(TM) 64-Bit Server VM warning: INFO: 
os::commit_memory(0x7fe9c44ee000, 12288, 0) failed; error='Cannot allocate 
memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing 
reserved memory.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: 
os::commit_memory(0x7f5c056ab000, 12288, 0) failed; error='Cannot allocate 
memory' (errno=12)
[thread 140033204860672 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: 
os::commit_memory(0x7f5c0566a000, 12288, 0) failed; error='Cannot allocate 
memory' (errno=12)
[thread 140033204594432 also had an error]Java HotSpot(TM) 64-Bit Server VM 
warning:
INFO: os::commit_memory(0x7fe9c420c000, 12288, 0) failed; error='Cannot 
allocate memory' (errno=12)
Java HotSpot(TM) 64-Bit Server VM warning: [thread 140641994852096 also had an 
error]INFO: os::commit_memory(0x7f5c055a7000, 12288, 0) failed; 
error='Cannot allocate memory' (errno=12)

Please let me know what I miss?

Thanks & Regards,
Abhishek Kumar Maheshwari
+91- 805591 (Mobile)
Times Internet Ltd. | A Times of India Group Company
FC - 6, Sector 16A, Film City,  Noida,  U.P. 201301 | INDIA
P Please do not print this email unless it is absolutely necessary. Spread 
environmental awareness.

YES Bank & The Economic Times Global Business Summit (GBS) is back on 27-28 
March. The 3rd edition of GBS will see participation of 2000+ delegates from 
over 20 countries as they bear witness to leaders sharing insights into how to 
best navigate a dynamic future. Visit www.et-gbs.com

RE: [Cassandra 3.0.9] Cannot allocate memory

2017-03-22 Thread Abhishek Kumar Maheshwari 
Hi Abhishek,

In sysctl.conf file we have below setting:

vm.swappiness = 10
vm.dirty_ratio = 60
vm.dirty_background_ratio = 2
vm.max_map_count = 1048575

so I need to apply patch for same?

From: Abhishek Verma [mailto:ve...@uber.com]
Sent: 22 March 2017 23:04
To: user@cassandra.apache.org
Cc: Abhishek Kumar Maheshwari 
Subject: Re: [Cassandra 3.0.9] Cannot allocate memory

Just a shot in the dark, but what is your setting of vm.max_map_count in 
/etc/sysctl.conf ?

It is recommended to set it to:
vm.max_map_count = 1048575

Source: 
https://docs.datastax.com/en/landing_page/doc/landing_page/recommendedSettingsLinux.html

We saw a similar problem in the past where mmap failed and we added a check to 
emit warning as a part of https://issues.apache.org/jira/browse/CASSANDRA-13008.


On Wed, Mar 22, 2017 at 9:57 AM, Thakrar, Jayesh 
> wrote:
Is/are the Cassandra server(s) shared?
E.g. do they run mesos + spark?

From: Abhishek Kumar Maheshwari 
>
Date: Wednesday, March 22, 2017 at 12:45 AM
To: "user@cassandra.apache.org" 
>
Subject: [Cassandra 3.0.9] Cannot allocate memory

Hi all,

I am using Cassandra 3.0.9. while I am adding new server after some time I am 
getting below exception. JVM option is attaches.
Hardware info:
Ram 64 GB.
Core: 40


Java HotSpot(TM) 64-Bit Server VM warning: INFO: 
os::commit_memory(0x7fe9c44ee000, 12288, 0) failed; error='Cannot allocate 
memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing 
reserved memory.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: 
os::commit_memory(0x7f5c056ab000, 12288, 0) failed; error='Cannot allocate 
memory' (errno=12)
[thread 140033204860672 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: 
os::commit_memory(0x7f5c0566a000, 12288, 0) failed; error='Cannot allocate 
memory' (errno=12)
[thread 140033204594432 also had an error]Java HotSpot(TM) 64-Bit Server VM 
warning:
INFO: os::commit_memory(0x7fe9c420c000, 12288, 0) failed; error='Cannot 
allocate memory' (errno=12)
Java HotSpot(TM) 64-Bit Server VM warning: [thread 140641994852096 also had an 
error]INFO: os::commit_memory(0x7f5c055a7000, 12288, 0) failed; 
error='Cannot allocate memory' (errno=12)

Please let me know what I miss?

Thanks & Regards,
Abhishek Kumar Maheshwari
+91- 805591 (Mobile)
Times Internet Ltd. | A Times of India Group Company
FC - 6, Sector 16A, Film City,  Noida,  U.P. 201301 | INDIA
P Please do not print this email unless it is absolutely necessary. Spread 
environmental awareness.

YES Bank & The Economic Times Global Business Summit (GBS) is back on 27-28 
March. The 3rd edition of GBS will see participation of 2000+ delegates from 
over 20 countries as they bear witness to leaders sharing insights into how to 
best navigate a dynamic future. Visit www.et-gbs.com

Re: Assertions being hit on Cassandra 3.5 cluster (UnfilteredRowIterators.concat)

2017-03-22 Thread Daniel Miranda 
I found out the problem is conditioned to having the row cache enabled.
Whenever a query would return an empty result set in a particular table, it
would fail instead with the exception being thrown in all all nodes.

Disabling the row cache for that particular table fixes the problem.
Re-enabling has not caused the issue again yet. I do not have row cache
saving enabled, and the issue persisted between node restarts, which it's
somewhat strange.

I couldn't reproduce the issue with any other tables. I can produce a
sanitized dump of the SSTable if a developer desires.

--
Regards,
Daniel

On Wed, Mar 22, 2017 at 2:33 PM Daniel Miranda  wrote:

> Thank you for the pointer Michael, I'll try to investigate if this is the
> same bug I am seeing.
>
> I am afraid it might not be, since I'm observing the error periodically,
> not just during compactions, and the traceback seems different.
>
> Regards,
> Daniel
>
> On Wed, Mar 22, 2017 at 1:27 PM Michael Shuler 
> wrote:
>
> Possibly https://issues.apache.org/jira/browse/CASSANDRA-12336, which
> shows fixed in 3.0.9, 3.8. There are a couple related bug reports listed
> on there, which you might investigate, as well.
>
> --
> Kind regards,
> Michael
>
> On 03/22/2017 11:21 AM, Daniel Miranda wrote:
> > Greetings,
> >
> > Recently I've started to see the an assertion (traceback follows at the
> > end of the message) causing exceptions in a 3-node Cassandra 3.5 cluster
> > (running on Ubuntu 14.04 on Amazon EC2). It seems to happen in all
> > nodes. Repairs run fine without indicating any errors.
> >
> > I can't seem to find any information about it from someone else or any
> > bug reports.
> >
> > Should I bother running an SSTable scrub? Is it a known issue that is
> > fixed in subsequent versions?
> >
> > Thanks in advance,
> > Daniel
> >
> > ---
> > WARN  [SharedPool-Worker-1] 2017-03-16 18:54:35,587
> > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on
> > thread Thread[SharedPool-Worker-1,5,main]: {}
> > java.lang.AssertionError: null
> > at
> >
> org.apache.cassandra.db.rows.UnfilteredRowIterators.concat(UnfilteredRowIterators.java:157)
> > ~[apache-cassandra-3.5.jar:3.5]
> > at
> >
> org.apache.cassandra.db.SinglePartitionReadCommand.getThroughCache(SinglePartitionReadCommand.java:420)
> > ~[apache-cassandra-3.5.jar:3.5]
> > at
> >
> org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:324)
> > ~[apache-cassandra-3.5.jar:3.5]
> > at
> > org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:366)
> > ~[apache-cassandra-3.5.jar:3.5]
> > at
> >
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1797)
> > ~[apache-cassandra-3.5.jar:3.5]
> > at
> >
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2466)
> > ~[apache-cassandra-3.5.jar:3.5]
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > ~[na:1.8.0_111]
> > at
> >
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> > ~[apache-cassandra-3.5.jar:3.5]
> > at
> >
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> > [apache-cassandra-3.5.jar:3.5]
> > at
> > org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> > [apache-cassandra-3.5.jar:3.5]
> > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
> > --
> > *Daniel Miranda*
> >
> > DevOps Engineering Intern
> > (11) 991959845
> > www.cobli.co 
>
>

Re: A Single Dropped Node Fails Entire Read Queries

2017-03-22 Thread Shalom Sagges 
Upgrading to 3.0.12 solved the issue.

Thanks a lot for the help Joel!


Shalom Sagges
DBA
T: +972-74-700-4035
 
 We Create Meaningful Connections



On Tue, Mar 14, 2017 at 10:44 AM, Shalom Sagges 
wrote:

> Thanks a lot Joel!
>
> I'll go ahead and upgrade.
>
> Thanks again!
>
>
> Shalom Sagges
> DBA
> T: +972-74-700-4035 <+972%2074-700-4035>
>  
>  We Create Meaningful Connections
> 
>
>
> On Mon, Mar 13, 2017 at 7:27 PM, Joel Knighton  > wrote:
>
>> It's possible that you're hitting https://issues.apache.
>> org/jira/browse/CASSANDRA-13009 .
>>
>> In (simplified) summary, the read query picks the right number of
>> endpoints fairly early in its execution. Because the down node has not been
>> detected as down yet, it may be one of the nodes. When this node doesn't
>> answer, it is likely that speculative retry will kick in after a certain
>> amount of time and query an up node. This feature is present and working in
>> the earlier releases you tested. Unfortunately, percentile-based
>> speculative retry wasn't working as intended in 2.2+ until fixed in
>> CASSANDRA-13009, which went into 2.2.9/3.0.11+.
>>
>> It may be worth evaluating the latest 3.0.x release.
>>
>> On Mon, Mar 13, 2017 at 11:48 AM, Shalom Sagges 
>> wrote:
>>
>>> Just some more info, I've tried the same scenario on 2.0.14 and 2.1.15
>>> and didn't encounter such errors.
>>> What I did find is that the timeout errors appear only until the node is
>>> discovered as "DN" in nodetool status. Once the node is in DN status, the
>>> errors stop and the data is retrieved.
>>>
>>> Could this be a bug in 3.0.9? Or some sort of misconfiguration I missed?
>>>
>>> Thanks!
>>>
>>>
>>>
>>> Shalom Sagges
>>> DBA
>>> T: +972-74-700-4035 <+972%2074-700-4035>
>>>  
>>>  We Create Meaningful Connections
>>> 
>>>
>>>
>>> On Sun, Mar 12, 2017 at 10:21 AM, Shalom Sagges 
>>> wrote:
>>>
 Hi Michael,

 If a node suddenly fails, and there are other replicas that can still
 satisfy the consistency level, shouldn't the request succeed regardless of
 the failed node?

 Thanks!





 Shalom Sagges
 DBA
 T: +972-74-700-4035 <+972%2074-700-4035>
 
   We
 Create Meaningful Connections
 


 On Fri, Mar 10, 2017 at 6:25 PM, Michael Shuler  wrote:

> I may be mistaken on the exact configuration option for the timeout
> you're hitting, but I believe this may be the general
> `request_timeout_in_ms: 1` in conf/cassandra.yaml.
>
> A reasonable timeout for a "node down" discovery/processing is needed
> to
> prevent random flapping of nodes with a super short timeout interval.
> Applications should also retry on a host unavailable exception like
> this, because in the long run, this should be expected from time to
> time
> for network partitions, node failure, maintenance cycles, etc.
>
> --
> Kind regards,
> Michael
>
> On 03/10/2017 04:07 AM, Shalom Sagges wrote:
> > Hi daniel,
> >
> > I don't think that's a network issue, because ~10 seconds after the
> node
> > stopped, the queries were successful again without any timeout
> issues.
> >
> > Thanks!
> >
> >
> > Shalom Sagges
> > DBA
> > T: +972-74-700-4035
> > 
> >     PersonInc>
> >
> >   We Create Meaningful Connections
> >
> > 
> >
> >
> >
> > On Fri, Mar 10, 2017 at 12:01 PM, Daniel Hölbling-Inzko
> >  > > wrote:
> >
> > Could there be network issues in connecting between the nodes? If
> > node a gets To be the query coordinator but can't reach b and c
> is
> > obviously down it won't get a quorum.
> >
> > Greetings
> >
> > Shalom Sagges  > > schrieb am Fr. 10. März 2017
> um 10:55:
> >
> > @Ryan, my keyspace replication settings are as follows:

Re: [Cassandra 3.0.9] Cannot allocate memory

2017-03-22 Thread Abhishek Verma 
Just a shot in the dark, but what is your setting of
vm.max_map_count in /etc/sysctl.conf ?

It is recommended to set it to:
vm.max_map_count = 1048575

Source:
https://docs.datastax.com/en/landing_page/doc/landing_page/recommendedSettingsLinux.html

We saw a similar problem in the past where mmap failed and we added a check
to emit warning as a part of
https://issues.apache.org/jira/browse/CASSANDRA-13008.


On Wed, Mar 22, 2017 at 9:57 AM, Thakrar, Jayesh <
jthak...@conversantmedia.com> wrote:

> Is/are the Cassandra server(s) shared?
>
> E.g. do they run mesos + spark?
>
>
>
> *From: *Abhishek Kumar Maheshwari 
> *Date: *Wednesday, March 22, 2017 at 12:45 AM
> *To: *"user@cassandra.apache.org" 
> *Subject: *[Cassandra 3.0.9] Cannot allocate memory
>
>
>
> Hi all,
>
>
>
> I am using Cassandra 3.0.9. while I am adding new server after some time I
> am getting below exception. JVM option is attaches.
>
> Hardware info:
>
> Ram 64 GB.
>
> Core: 40
>
>
>
>
>
> Java HotSpot(TM) 64-Bit Server VM warning: INFO: 
> os::commit_memory(0x7fe9c44ee000,
> 12288, 0) failed; error='Cannot allocate memory' (errno=12)
>
> #
>
> # There is insufficient memory for the Java Runtime Environment to
> continue.
>
> # Native memory allocation (mmap) failed to map 12288 bytes for committing
> reserved memory.
>
> Java HotSpot(TM) 64-Bit Server VM warning: INFO: 
> os::commit_memory(0x7f5c056ab000,
> 12288, 0) failed; error='Cannot allocate memory' (errno=12)
>
> [thread 140033204860672 also had an error]
>
> Java HotSpot(TM) 64-Bit Server VM warning: INFO: 
> os::commit_memory(0x7f5c0566a000,
> 12288, 0) failed; error='Cannot allocate memory' (errno=12)
>
> [thread 140033204594432 also had an error]Java HotSpot(TM) 64-Bit Server
> VM warning:
>
> INFO: os::commit_memory(0x7fe9c420c000, 12288, 0) failed;
> error='Cannot allocate memory' (errno=12)
>
> Java HotSpot(TM) 64-Bit Server VM warning: [thread 140641994852096 also
> had an error]INFO: os::commit_memory(0x7f5c055a7000, 12288, 0)
> failed; error='Cannot allocate memory' (errno=12)
>
>
>
> Please let me know what I miss?
>
>
>
> *Thanks & Regards,*
> *Abhishek Kumar Maheshwari*
> *+91- 805591 <+91%208%2005591> (Mobile)*
>
> Times Internet Ltd. | A Times of India Group Company
>
> FC - 6, Sector 16A, Film City,  Noida,  U.P. 201301 | INDIA
>
> *P** Please do not print this email unless it is absolutely necessary.
> Spread environmental awareness.*
>
>
>
> YES Bank & The Economic Times Global Business Summit (GBS) is back on
> 27-28 March. The 3rd edition of GBS will see participation of 2000+
> delegates from over 20 countries as they bear witness to leaders sharing
> insights into how to best navigate a dynamic future. Visit www.et-gbs.com
>

Re: Assertions being hit on Cassandra 3.5 cluster (UnfilteredRowIterators.concat)

2017-03-22 Thread Daniel Miranda 
Thank you for the pointer Michael, I'll try to investigate if this is the
same bug I am seeing.

I am afraid it might not be, since I'm observing the error periodically,
not just during compactions, and the traceback seems different.

Regards,
Daniel

On Wed, Mar 22, 2017 at 1:27 PM Michael Shuler 
wrote:

Possibly https://issues.apache.org/jira/browse/CASSANDRA-12336, which
shows fixed in 3.0.9, 3.8. There are a couple related bug reports listed
on there, which you might investigate, as well.

--
Kind regards,
Michael

On 03/22/2017 11:21 AM, Daniel Miranda wrote:
> Greetings,
>
> Recently I've started to see the an assertion (traceback follows at the
> end of the message) causing exceptions in a 3-node Cassandra 3.5 cluster
> (running on Ubuntu 14.04 on Amazon EC2). It seems to happen in all
> nodes. Repairs run fine without indicating any errors.
>
> I can't seem to find any information about it from someone else or any
> bug reports.
>
> Should I bother running an SSTable scrub? Is it a known issue that is
> fixed in subsequent versions?
>
> Thanks in advance,
> Daniel
>
> ---
> WARN  [SharedPool-Worker-1] 2017-03-16 18:54:35,587
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on
> thread Thread[SharedPool-Worker-1,5,main]: {}
> java.lang.AssertionError: null
> at
>
org.apache.cassandra.db.rows.UnfilteredRowIterators.concat(UnfilteredRowIterators.java:157)
> ~[apache-cassandra-3.5.jar:3.5]
> at
>
org.apache.cassandra.db.SinglePartitionReadCommand.getThroughCache(SinglePartitionReadCommand.java:420)
> ~[apache-cassandra-3.5.jar:3.5]
> at
>
org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:324)
> ~[apache-cassandra-3.5.jar:3.5]
> at
> org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:366)
> ~[apache-cassandra-3.5.jar:3.5]
> at
>
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1797)
> ~[apache-cassandra-3.5.jar:3.5]
> at
>
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2466)
> ~[apache-cassandra-3.5.jar:3.5]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_111]
> at
>
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> ~[apache-cassandra-3.5.jar:3.5]
> at
>
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> [apache-cassandra-3.5.jar:3.5]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.5.jar:3.5]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
> --
> *Daniel Miranda*
>
> DevOps Engineering Intern
> (11) 991959845
> www.cobli.co

Re: [Cassandra 3.0.9] Cannot allocate memory

2017-03-22 Thread Thakrar, Jayesh 
Is/are the Cassandra server(s) shared?
E.g. do they run mesos + spark?

From: Abhishek Kumar Maheshwari 
Date: Wednesday, March 22, 2017 at 12:45 AM
To: "user@cassandra.apache.org" 
Subject: [Cassandra 3.0.9] Cannot allocate memory

Hi all,

I am using Cassandra 3.0.9. while I am adding new server after some time I am 
getting below exception. JVM option is attaches.
Hardware info:
Ram 64 GB.
Core: 40


Java HotSpot(TM) 64-Bit Server VM warning: INFO: 
os::commit_memory(0x7fe9c44ee000, 12288, 0) failed; error='Cannot allocate 
memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing 
reserved memory.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: 
os::commit_memory(0x7f5c056ab000, 12288, 0) failed; error='Cannot allocate 
memory' (errno=12)
[thread 140033204860672 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: 
os::commit_memory(0x7f5c0566a000, 12288, 0) failed; error='Cannot allocate 
memory' (errno=12)
[thread 140033204594432 also had an error]Java HotSpot(TM) 64-Bit Server VM 
warning:
INFO: os::commit_memory(0x7fe9c420c000, 12288, 0) failed; error='Cannot 
allocate memory' (errno=12)
Java HotSpot(TM) 64-Bit Server VM warning: [thread 140641994852096 also had an 
error]INFO: os::commit_memory(0x7f5c055a7000, 12288, 0) failed; 
error='Cannot allocate memory' (errno=12)

Please let me know what I miss?

Thanks & Regards,
Abhishek Kumar Maheshwari
+91- 805591 (Mobile)
Times Internet Ltd. | A Times of India Group Company
FC - 6, Sector 16A, Film City,  Noida,  U.P. 201301 | INDIA
P Please do not print this email unless it is absolutely necessary. Spread 
environmental awareness.

YES Bank & The Economic Times Global Business Summit (GBS) is back on 27-28 
March. The 3rd edition of GBS will see participation of 2000+ delegates from 
over 20 countries as they bear witness to leaders sharing insights into how to 
best navigate a dynamic future. Visit www.et-gbs.com

Most used token or partitions

2017-03-22 Thread D. Salvatore 
Hi,
I am looking for a way to retrieve some statistic about the usage of the
data on my Cassandra cluster.
Ideally I would like to retrieve a list of the most used token ranges or
partitions over the time through JMX or other similar ways.

I found that nodetool as the option "toppartition" to retrieve the most
used partitions over a sample period. However, it return me this error:

 $ nodetool toppartitions ycsb usertable 100
  nodetool: Argument value's index, calculated according to this
TabularData   instance's tabularType, already refers to a value in this
table.
  See 'nodetool help' or 'nodetool help '.

Am I doing something wrong? Do you know better ways than nodetool?

Thanks in advance
Salvatore

Re: ONE has much higher latency than LOCAL_ONE

2017-03-22 Thread Eric Plowe 
Yes, your request from the client is going to the LocalDC that you've
defined for the data center aware load balancing policy, but with a
consistency level of ONE, there is a chance for the coordinator (the node
your client has connected to) to route the request across DC's.

Please see:
https://docs.datastax.com/en/cassandra/2.1/cassandra/dml/dmlClientRequestsRead.html#dmlClientRequestsRead__two-dc-one

"A two datacenter cluster with a consistency level of ONE
"In a multiple datacenter cluster with a replication factor of 3, and a
read consistency of ONE, the closest replica for the given row, regardless
of datacenter, is contacted to fulfill the read request. In the background
a read repair is potentially initiated, based on the read_repair_chance setting
of the table, for the other replicas."


A two datacenter cluster with a consistency level of LOCAL_ONE


In a multiple datacenter cluster with a replication factor of 3, and a read
consistency of LOCAL_ONE, the closest replica for the given row in the same
datacenter as the coordinator node is contacted to fulfill the read
request. In the background a read repair is potentially initiated, based on
the read_repair_chance setting of the table, for the other replicas."


Dynamic snitching also comes into play with reads. Just because your client
is using TokenAware, and should connect to the appropriate replica node
(which now is your coordinator) it can route your read request away from
what it believes to be poorly performing nodes to another replica which
could be in the other DC with CL = ONE. Read more about dynamic snitch
here:
https://docs.datastax.com/en/cassandra/2.1/cassandra/architecture/architectureSnitchDynamic_c.html


Regards,

Eric Plowe







On Wed, Mar 22, 2017 at 12:21 PM Shannon Carey  wrote:

I understand all that, but it doesn't explain why the latency increases.
The requests are not going to a remote DC. I know this because currently
all requests are going to the client in one particular DC. The read request
rate of the Cassandra nodes in the other DC remained flat (near zero) the
whole time, compared to ~200read/s on the Cassandra nodes in the DC local
to the client doing the reads. This is expected, because the
DCAwareRoundRobinPolicy will cause local nodes to be used preferentially
whenever possible. What's not expected is the dramatic latency increase.
Btw this client is read-only: no writes.


From: Eric Plowe 
Reply-To: "user@cassandra.apache.org" 
Date: Tuesday, March 21, 2017 at 7:23 PM

To: "user@cassandra.apache.org" 
Subject: Re: ONE has much higher latency than LOCAL_ONE

ONE means at least one replica node to ack the write, but doesn't require
that the coordinator route the request to a node in the local data center.

LOCAL_ONE was introduced to handle the case of when you have multiple data
centers and cross data center traffic is not desirable.

In multiple datacenter clusters, a consistency level of ONE is often
desirable, but cross-DC traffic is not. LOCAL_ONEaccomplishes this. For
security and quality reasons, you can use this consistency level in an
offline datacenter to prevent automatic connection to online nodes in other
datacenters if an offline node goes down.

From:
https://docs.datastax.com/en/cassandra/2.1/cassandra/dml/dml_config_consistency_c.html

Regards,

Eric

On Tue, Mar 21, 2017 at 7:49 PM Shannon Carey  wrote:

The cluster is in two DCs, and yes the client is deployed locally to each
DC.

From: Matija Gobec 
Reply-To: "user@cassandra.apache.org" 
Date: Tuesday, March 21, 2017 at 2:56 PM
To: "user@cassandra.apache.org" 
Subject: Re: ONE has much higher latency than LOCAL_ONE

Are you running a multi DC cluster? If yes do you have application in both
data centers/regions ?

On Tue, Mar 21, 2017 at 8:07 PM, Shannon Carey  wrote:

I am seeing unexpected behavior: consistency level ONE increases read
latency 99th percentile to ~108ms (95th percentile to 5ms-90ms) up from
~5ms (99th percentile) when using LOCAL_ONE.

I am using DSE 5.0 with Datastax client 3.0.0. The client is configured
with a TokenAwarePolicy wrapping a DCAwareRoundRobinPolicy with
usedHostsPerRemoteDc set to a very high number. Cassandra cluster has two
datacenters.

I would expect that when the cluster is operating normally (all local nodes
reachable), ONE would behave the same as LOCAL_ONE. The  Does anyone know
why this is not the case?

Re: Assertions being hit on Cassandra 3.5 cluster (UnfilteredRowIterators.concat)

2017-03-22 Thread Michael Shuler 
Possibly https://issues.apache.org/jira/browse/CASSANDRA-12336, which
shows fixed in 3.0.9, 3.8. There are a couple related bug reports listed
on there, which you might investigate, as well.

-- 
Kind regards,
Michael

On 03/22/2017 11:21 AM, Daniel Miranda wrote:
> Greetings,
> 
> Recently I've started to see the an assertion (traceback follows at the
> end of the message) causing exceptions in a 3-node Cassandra 3.5 cluster
> (running on Ubuntu 14.04 on Amazon EC2). It seems to happen in all
> nodes. Repairs run fine without indicating any errors.
> 
> I can't seem to find any information about it from someone else or any
> bug reports.
> 
> Should I bother running an SSTable scrub? Is it a known issue that is
> fixed in subsequent versions?
> 
> Thanks in advance,
> Daniel
> 
> ---
> WARN  [SharedPool-Worker-1] 2017-03-16 18:54:35,587
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on
> thread Thread[SharedPool-Worker-1,5,main]: {}
> java.lang.AssertionError: null
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators.concat(UnfilteredRowIterators.java:157)
> ~[apache-cassandra-3.5.jar:3.5]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.getThroughCache(SinglePartitionReadCommand.java:420)
> ~[apache-cassandra-3.5.jar:3.5]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:324)
> ~[apache-cassandra-3.5.jar:3.5]
> at
> org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:366)
> ~[apache-cassandra-3.5.jar:3.5]
> at
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1797)
> ~[apache-cassandra-3.5.jar:3.5]
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2466)
> ~[apache-cassandra-3.5.jar:3.5]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_111]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> ~[apache-cassandra-3.5.jar:3.5]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> [apache-cassandra-3.5.jar:3.5]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.5.jar:3.5]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
> -- 
> *Daniel Miranda*
> 
> DevOps Engineering Intern
> (11) 991959845
> www.cobli.co

Assertions being hit on Cassandra 3.5 cluster (UnfilteredRowIterators.concat)

2017-03-22 Thread Daniel Miranda 
Greetings,

Recently I've started to see the an assertion (traceback follows at the end
of the message) causing exceptions in a 3-node Cassandra 3.5 cluster
(running on Ubuntu 14.04 on Amazon EC2). It seems to happen in all nodes.
Repairs run fine without indicating any errors.

I can't seem to find any information about it from someone else or any bug
reports.

Should I bother running an SSTable scrub? Is it a known issue that is fixed
in subsequent versions?

Thanks in advance,
Daniel

---
WARN  [SharedPool-Worker-1] 2017-03-16 18:54:35,587
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
Thread[SharedPool-Worker-1,5,main]: {}
java.lang.AssertionError: null
at
org.apache.cassandra.db.rows.UnfilteredRowIterators.concat(UnfilteredRowIterators.java:157)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.SinglePartitionReadCommand.getThroughCache(SinglePartitionReadCommand.java:420)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:324)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:366)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1797)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2466)
~[apache-cassandra-3.5.jar:3.5]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_111]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-3.5.jar:3.5]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
-- 
*Daniel Miranda*

DevOps Engineering Intern
(11) 991959845
www.cobli.co

Re: ONE has much higher latency than LOCAL_ONE

2017-03-22 Thread Shannon Carey 
I understand all that, but it doesn't explain why the latency increases. The 
requests are not going to a remote DC. I know this because currently all 
requests are going to the client in one particular DC. The read request rate of 
the Cassandra nodes in the other DC remained flat (near zero) the whole time, 
compared to ~200read/s on the Cassandra nodes in the DC local to the client 
doing the reads. This is expected, because the DCAwareRoundRobinPolicy will 
cause local nodes to be used preferentially whenever possible. What's not 
expected is the dramatic latency increase. Btw this client is read-only: no 
writes.


From: Eric Plowe >
Reply-To: "user@cassandra.apache.org" 
>
Date: Tuesday, March 21, 2017 at 7:23 PM
To: "user@cassandra.apache.org" 
>
Subject: Re: ONE has much higher latency than LOCAL_ONE

ONE means at least one replica node to ack the write, but doesn't require that 
the coordinator route the request to a node in the local data center.

LOCAL_ONE was introduced to handle the case of when you have multiple data 
centers and cross data center traffic is not desirable.

In multiple datacenter clusters, a consistency level of ONE is often desirable, 
but cross-DC traffic is not. LOCAL_ONEaccomplishes this. For security and 
quality reasons, you can use this consistency level in an offline datacenter to 
prevent automatic connection to online nodes in other datacenters if an offline 
node goes down.

From: 
https://docs.datastax.com/en/cassandra/2.1/cassandra/dml/dml_config_consistency_c.html

Regards,

Eric

On Tue, Mar 21, 2017 at 7:49 PM Shannon Carey 
> wrote:
The cluster is in two DCs, and yes the client is deployed locally to each DC.

From: Matija Gobec >
Reply-To: "user@cassandra.apache.org" 
>
Date: Tuesday, March 21, 2017 at 2:56 PM
To: "user@cassandra.apache.org" 
>
Subject: Re: ONE has much higher latency than LOCAL_ONE

Are you running a multi DC cluster? If yes do you have application in both data 
centers/regions ?

On Tue, Mar 21, 2017 at 8:07 PM, Shannon Carey 
> wrote:
I am seeing unexpected behavior: consistency level ONE increases read latency 
99th percentile to ~108ms (95th percentile to 5ms-90ms) up from ~5ms (99th 
percentile) when using LOCAL_ONE.

I am using DSE 5.0 with Datastax client 3.0.0. The client is configured with a 
TokenAwarePolicy wrapping a DCAwareRoundRobinPolicy with usedHostsPerRemoteDc 
set to a very high number. Cassandra cluster has two datacenters.

I would expect that when the cluster is operating normally (all local nodes 
reachable), ONE would behave the same as LOCAL_ONE. The  Does anyone know why 
this is not the case?

Re: ONE has much higher latency than LOCAL_ONE

2017-03-22 Thread Shannon Carey 
Yes, as I mentioned in my other thread, LOCAL_ONE does not allow the retry 
policy to take action if all local nodes are down.

Yes, I am using withLocalDc(); Here's the code (Scala):

  def getClusterBuilder: Builder = {
val pool = new PoolingOptions
pool.setConnectionsPerHost(HostDistance.LOCAL, 
config.coreConnectionsPerHost, config.maxConnectionsPerHost)

val codecRegistry: CodecRegistry = new CodecRegistry()
.register(InstantCodec.instance)
.register(SimpleTimestampCodec.instance)

// By specifying MaxValue here, we allow any & all hosts in remote DCs to 
be used by queries when necessary. That
// allows TokenAwarePolicy to choose the appropriate nodes in remote DCs.
val maxHostsToUsePerRemoteDc = Int.MaxValue

// We have nodes from the remote DC in the initial list (so that we can 
tolerate DC failover), so we have to
// specify the local DC explicitly.
val dcAwarePolicy = DCAwareRoundRobinPolicy.builder()
.withLocalDc(config.localDc)
.withUsedHostsPerRemoteDc(maxHostsToUsePerRemoteDc)
.build()

val shuffleReplicas = true
val builder = Cluster.builder()
.withClusterName(config.clusterName)
.withPoolingOptions(pool)
.withLoadBalancingPolicy(new TokenAwarePolicy(dcAwarePolicy, 
shuffleReplicas))
.withRetryPolicy(DowngradingConsistencyRetryPolicy.INSTANCE)
.withCodecRegistry(codecRegistry)

val contactPoints = config.contactPointsProvider.getContactPoints.asScala
contactPoints.foreach(builder.addContactPoint)

builder
  }

I will have to turn up the logging in order to see that log message you refer 
to. But it seems to me that the DC config is the same regardless of whether I 
use ONE or LOCAL_ONE so I don't think it would make a difference. From what 
I've seen, I'd expect all the non-local nodes to be listed in that message. But 
I'll see what I can.

Thanks for your responses! I posted to the other list as you suggested: 
https://groups.google.com/a/lists.datastax.com/forum/#!topic/java-driver-user/o0GVBjFCHCA

From: Nate McCall >
Reply-To: "user@cassandra.apache.org" 
>
Date: Tuesday, March 21, 2017 at 7:16 PM
To: Cassandra Users 
>
Subject: Re: ONE has much higher latency than LOCAL_ONE



On Wed, Mar 22, 2017 at 1:11 PM, Nate McCall 
> wrote:


On Wed, Mar 22, 2017 at 12:48 PM, Shannon Carey 
> wrote:
>
> The cluster is in two DCs, and yes the client is deployed locally to each DC.

First off, what is the goal of using ONE instead of LOCAL_ONE? If it's 
failover, this could be addressed with a RetryPolicy starting wth LOCAL_ONE and 
falling back to ONE.


Just read your previous thread about this. That's pretty un-intuitive and 
counter to the way I remember that working (though admittedly, it's been a 
while).

Do please open a thread on the driver mailing list, i'm curious about the 
response.