Re: MUTATION messages dropped

[Jared Biel]

I can't comment on your specific issue, but I don't know if running 2.0.0
in production is a good idea. At the very least I'd try upgrading to the
latest 2.0.X (currently 2.0.3.)

 https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/

On 20 December 2013 06:08, Alexander Shutyaev  wrote:

> Thanks for you answers.
>
> *srmore*,
>
> We are using v2.0.0. As for GC I guess it does not correlate in our case,
> because we had cassandra running 9 days under production load and no
> dropped messages and I guess that during this time there were a lot of GCs.
>
> *Ken*,
>
> I've checked the values you indicated. Here they are:
>
> node1 6498
> node2 6476
> node3 6642
>
> I guess this is not good :) What can we do to fix this problem?
>
>
> 2013/12/19 Ken Hancock 
>
>> We had issues where the number of CF families that were being flushed
>> would align and then block writes for a very brief period. If that happened
>> when a bunch of writes came in, we'd see a spike in Mutation drops.
>>
>> Check nodetool tpstats for FlushWriter all time blocked.
>>
>>
>> On Thu, Dec 19, 2013 at 7:12 AM, Alexander Shutyaev 
>> wrote:
>>
>>> Hi all!
>>>
>>> We've had a problem with cassandra recently. We had 2 one-minute periods
>>> when we got a lot of timeouts on the client side (the only timeouts during
>>> 9 days we are using cassandra in production). In the logs we've found
>>> corresponding messages saying something about MUTATION messages dropped.
>>>
>>> Now, the official faq [1] says that this is an indicator that the load
>>> is too high. We've checked our monitoring and found out that 1-minute
>>> average cpu load had a local peak at the time of the problem, but it was
>>> like 0.8 against 0.2 usual which I guess is nothing for a 2 core virtual
>>> machine. We've also checked java threads - there was no peak there and
>>> their count was reasonable ~240-250.
>>>
>>> Can anyone give us a hint - what should we monitor to see this "high
>>> load" and what should we tune to make it acceptable?
>>>
>>> Thanks in advance,
>>> Alexander
>>>
>>> [1] http://wiki.apache.org/cassandra/FAQ#dropped_messages
>>>
>>
>>
>>
>> --
>>  *Ken Hancock *| System Architect, Advanced Advertising
>> SeaChange International
>> 50 Nagog Park
>> Acton, Massachusetts 01720
>> ken.hanc...@schange.com | www.schange.com | 
>> NASDAQ:SEAC
>>
>> Office: +1 (978) 889-3329 | [image: Google Talk:] ken.hanc...@schange.com
>>  | [image: Skype:]hancockks | [image: Yahoo IM:]hancockks [image:
>> LinkedIn] 
>>
>> [image: SeaChange International]
>>  This e-mail and any attachments may contain
>> information which is SeaChange International confidential. The information
>> enclosed is intended only for the addressees herein and may not be copied
>> or forwarded without permission from SeaChange International.
>>
>
>

Re: MUTATION messages dropped

[Alexander Shutyaev]

Thanks for you answers.

*srmore*,

We are using v2.0.0. As for GC I guess it does not correlate in our case,
because we had cassandra running 9 days under production load and no
dropped messages and I guess that during this time there were a lot of GCs.

*Ken*,

I've checked the values you indicated. Here they are:

node1 6498
node2 6476
node3 6642

I guess this is not good :) What can we do to fix this problem?


2013/12/19 Ken Hancock 

> We had issues where the number of CF families that were being flushed
> would align and then block writes for a very brief period. If that happened
> when a bunch of writes came in, we'd see a spike in Mutation drops.
>
> Check nodetool tpstats for FlushWriter all time blocked.
>
>
> On Thu, Dec 19, 2013 at 7:12 AM, Alexander Shutyaev wrote:
>
>> Hi all!
>>
>> We've had a problem with cassandra recently. We had 2 one-minute periods
>> when we got a lot of timeouts on the client side (the only timeouts during
>> 9 days we are using cassandra in production). In the logs we've found
>> corresponding messages saying something about MUTATION messages dropped.
>>
>> Now, the official faq [1] says that this is an indicator that the load is
>> too high. We've checked our monitoring and found out that 1-minute average
>> cpu load had a local peak at the time of the problem, but it was like 0.8
>> against 0.2 usual which I guess is nothing for a 2 core virtual machine.
>> We've also checked java threads - there was no peak there and their count
>> was reasonable ~240-250.
>>
>> Can anyone give us a hint - what should we monitor to see this "high
>> load" and what should we tune to make it acceptable?
>>
>> Thanks in advance,
>> Alexander
>>
>> [1] http://wiki.apache.org/cassandra/FAQ#dropped_messages
>>
>
>
>
> --
> *Ken Hancock *| System Architect, Advanced Advertising
> SeaChange International
> 50 Nagog Park
> Acton, Massachusetts 01720
> ken.hanc...@schange.com | www.schange.com | 
> NASDAQ:SEAC
>
> Office: +1 (978) 889-3329 | [image: Google Talk:] ken.hanc...@schange.com
>  | [image: Skype:]hancockks | [image: Yahoo IM:]hancockks [image:
> LinkedIn] 
>
> [image: SeaChange International]
>  This e-mail and any attachments may contain
> information which is SeaChange International confidential. The information
> enclosed is intended only for the addressees herein and may not be copied
> or forwarded without permission from SeaChange International.
>

Re: Occasional NPE using DataStax Java driver

[David Tinker]

Done. https://datastax-oss.atlassian.net/browse/JAVA-231

On Thu, Dec 19, 2013 at 10:42 AM, Sylvain Lebresne  wrote:
> Mind opening a ticket on https://datastax-oss.atlassian.net/browse/JAVA?
> It's almost surely a bug.
>
> --
> Sylvain
>
>
> On Thu, Dec 19, 2013 at 8:21 AM, David Tinker 
> wrote:
>>
>> We are using Cassandra 2.0.3-1 installed on Ubuntu 12.04 from the
>> DataStax repo with the DataStax Java driver version 2.0.0-rc1. Every
>> now and then we get the following exception:
>>
>> 2013-12-19 06:56:34,619 [sql-2-t15] ERROR core.RequestHandler  -
>> Unexpected error while querying /x.x.x.x
>> java.lang.NullPointerException
>> at
>> com.datastax.driver.core.HostConnectionPool.waitForConnection(HostConnectionPool.java:203)
>> at
>> com.datastax.driver.core.HostConnectionPool.borrowConnection(HostConnectionPool.java:107)
>> at com.datastax.driver.core.RequestHandler.query(RequestHandler.java:112)
>> at
>> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:93)
>> at com.datastax.driver.core.Session$Manager.execute(Session.java:513)
>> at com.datastax.driver.core.Session$Manager.executeQuery(Session.java:549)
>> at com.datastax.driver.core.Session.executeAsync(Session.java:172)
>>
>> This happens during a big data load process which will do up to 256
>> executeAsync's in parallel.
>>
>> Any ideas? Its not causing huge problems because the operation is just
>> retried by our code but it would be nice to eliminate it.
>
>



-- 
http://qdb.io/ Persistent Message Queues With Replay and #RabbitMQ Integration

RE: Issue upgrading from 1.2 to 2.0.3

[Parag Patel]

Thanks for that link.

Our 1.2 version is 1.2.12

Our 2.0.3 nodes were restarted once.  Before restart, it was the 1.2.12 binary, 
after it was the 2.0.3.  Immediately after the node was back in the cluster, we 
ran nodetool upgradesstables.  We haven't restarted since.

Is a restart required for each node?

From: Robert Coli [mailto:rc...@eventbrite.com]
Sent: Thursday, December 19, 2013 4:17 PM
To: user@cassandra.apache.org
Subject: Re: Issue upgrading from 1.2 to 2.0.3

On Thu, Dec 19, 2013 at 1:03 PM, Parag Patel 
mailto:parag.pa...@fusionts.com>> wrote:
We are in the process of upgrading 1.2 to 2.0.3.
 ...
Please help as this will prevent us from pushing into production.

(as a general commentary : 
https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ )

specific feedback on your question :

Did the 2.0.3 nodes see the 1.2.x (which 1.2.x?) nodes after the first restart?

=Rob

Re: Improving write performance in Cassandra and a few related issues...

[Aaron Morton]

> Thanks for the reply.  By packet drops I mean, the client is not able to read 
> the shared memory as fast as the software switch is writing into it..
> 
> 
What is the error you are getting on the client ? 

>Also, I would like to know if in general , distribution of write requests 
> to different Casaandra nodes instead of only to one, leads to increased write 
> performance in Cassandra.
> 
In general yes, clients should distribute their writes. 

>Is there any particular way in which write performance can be measured, 
> preferably from the client???
> 
> 
Logging at the client level ? 

Cheers


-
Aaron Morton
New Zealand
@aaronmorton

Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

On 18/12/2013, at 5:02 pm, Krishna Chaitanya  wrote:

> Thanks for the reply.  By packet drops I mean, the client is not able to read 
> the shared memory as fast as the software switch is writing into it..
>   I doubt its the issue with the client but can you in particular issues 
> that could cause this type of scenario?
>Also, I would like to know if in general , distribution of write requests 
> to different Casaandra nodes instead of only to one, leads to increased write 
> performance in Cassandra.
>Is there any particular way in which write performance can be measured, 
> preferably from the client???
> 
> On Dec 18, 2013 8:30 AM, "Aaron Morton"  wrote:
>> rite throughput is remaining at around 460 pkts/sec or sometimes even 
>> falling below that rate as against the expected rate of around 920 pkts/sec. 
>> Is it some kind of limitation of Cassandra or am I doing something wrong??? 
> There is nothing in cassandra that would make that happen. Double check your 
> client. 
> 
>>I also see an 
>> increase in packet drops when I try to store the packets from both the hosts 
>> into the same keyspace. The packets are getting collected properly followed 
>> by intervals in which they are being dropped in both the systems, at the 
>> same time. Could this be some kind of a buffer issue??? 
> What do you mean by packet drops ? 
> 
> Do you mean dropped messages in cassandra ? 
> 
>> Also, can write throughput be increased by distributing the write requests 
>> between the 2 Cassandra nodes instead of sending the requests to a single 
>> node? Currently, I dont see any improvement even  if I distribute the write 
>> requests to different hosts. How can I improve the write performance overall?
> Normally we expect 3k to 4k non counter writes per core per node, if you are 
> not seeing that it may be configuration or the client. 
> 
> Hope that helps. 
>  
> -
> Aaron Morton
> New Zealand
> @aaronmorton
> 
> Co-Founder & Principal Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
> 
> On 15/12/2013, at 7:51 pm, Krishna Chaitanya  wrote:
> 
>> Hello,
>>I am a newbie to the Cassandra world and have a few doubts which I 
>> wanted to clarify. I am having a software switch that stores netflow packets 
>> into a shared memory segment and a daemon that reads that memory segment and 
>> stores them into a 2-node Cassandra cluster. Currently, I am storing the 
>> packets from 2 hosts into 2 different keyspaces, hence only writes and no 
>> reads. The write throughput is coming to around 460 pkts/sec in each of the 
>> keyspaces. But, when I try to store the packets into the same keyspace, I 
>> observe that the write throughput is remaining at around 460 pkts/sec or 
>> sometimes even falling below that rate as against the expected rate of 
>> around 920 pkts/sec. Is it some kind of limitation of Cassandra or am I 
>> doing something wrong??? 
>>I also see an 
>> increase in packet drops when I try to store the packets from both the hosts 
>> into the same keyspace. The packets are getting collected properly followed 
>> by intervals in which they are being dropped in both the systems, at the 
>> same time. Could this be some kind of a buffer issue??? 
>>   The write requests from both the systems are 
>> sent to the same node which is also the seed node. I am mostly using the 
>> default Cassandra configuration with replication_factor set to 1 and without 
>> durable_writes. The systems are i5s with 4 gb RAM. The data model is: each 
>> second is the row key with all the packets collected in that second as the 
>> columns. Also, can write throughput be increased by distributing the write 
>> requests between the 2 Cassandra nodes instead of sending the requests to a 
>> single node? Currently, I dont see any improvement even  if I distribute the 
>> write requests to different hosts. How can I improve the write performance 
>> overall?
>> 
>> Thanks.
>> -- 
>> Regards,
>> BNSK.
> 

Re: Issue upgrading from 1.2 to 2.0.3

[Robert Coli]

On Thu, Dec 19, 2013 at 1:03 PM, Parag Patel wrote:

>  We are in the process of upgrading 1.2 to 2.0.3.
>
 ...

> Please help as this will prevent us from pushing into production.
>

(as a general commentary :
https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ )

specific feedback on your question :

Did the 2.0.3 nodes see the 1.2.x (which 1.2.x?) nodes after the first
restart?

=Rob

WriteTimeoutException on Lightweight transactions

[Demian Berjman]

Hi. I am using Cassandra 2.0.3 with Datastax Java client.

I execute an insert query:

Insert insert = QueryBuilder.insertInto("demo_cl","demo_table").value("id",
id).value("col1", "transactions").ifNotExists();

session.execute(insert.setConsistencyLevel(ConsistencyLevel.QUORUM);

Then, i force a shutdown on one node and get:

com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra
timeout during write query at consistency SERIAL (2 replica were required
but only 1 acknowledged the write)

Then i read the row and i got not results. It seems that it was not
inserted. What happened to the "1 acknowledged the write"? It's lost? It's
like a rollback?

Thanks,

Issue upgrading from 1.2 to 2.0.3

[Parag Patel]

Hi,

We are in the process of upgrading 1.2 to 2.0.3.  We have a four node cluster 
and we're upgrading one node at a time.  After upgrading two of the nodes, we 
encountered a problem.  We observed that if we run nodetool status on the 2.0.3 
hosts, they would show 2 nodes down and 2 nodes up.  If we run nodetool status 
on the 1.2 hosts, it would show all nodes up.

Has anyone encountered this?  Perhaps I'm missing a step in my upgrade 
procedure?  Please help as this will prevent us from pushing into production.

Thanks,
Parag

Re: Cassandra pytho pagination

[Kumar Ranjan]

Rob - I got a question following your advice. This is how, I define my
column family

validators = {

'approved':'UTF8Type',

'tid': 'UTF8Type',

'iid': 'UTF8Type',

'score':   'IntegerType',

'likes':   'IntegerType',

'retweet': 'IntegerType',

'favorite':'IntegerType',

'screen_name': 'UTF8Type',

'created_date':'UTF8Type',

'expanded_url':'UTF8Type',

'embedly_data':'BytesType',

}

SYSTEM_MANAGER.create_column_family('KeySpaceNNN', 'Twitter_Instagram',
default_validation_class='UTF8Type', super=True, comparator='UTF8Type',
key_validation_class='UTF8Type', column_validation_classes=validator)

Actual data representation:

'row_key': {'1234555665_53323232': {'approved': 'false', 'tid':
123,  'iid': 34, 'score': 2, likes: 50, retweets: 45, favorite: 34,
screen_name:'goodname'},

'2344555665_53323232': {'approved': 'false', 'tid':
134,  'iid': 34, 'score': 2, likes: 50, retweets: 45, favorite: 34,
screen_name:'newname'}.

.

   }

Is there something wrong with it? Here 1234555665_53323232 and
2344555665_53323232 are super columns. Also, If I have to represent this
data with new composite comparator, How will I accomplish that?


Please let me know.


Regards.


On Wed, Dec 18, 2013 at 5:32 PM, Robert Coli  wrote:

> On Wed, Dec 18, 2013 at 1:28 PM, Kumar Ranjan wrote:
>
>> Second approach ( I used in production ):
>> - fetch all super columns for a row key
>>
>
> Stock response mentioning that super columns are anti-advised for use,
> especially in brand new code.
>
> =Rob
>
>

Re: Paging error after upgrade from C* 2.0.1 to 2.0.3 , Driver from 2.0.0-rc1 to 2.0.0-rc2

[Sylvain Lebresne]

> >
> > I'll note that while far from idea, you can also disable assertion with
> 2.0.3.
>
> Is there a way to do that without patching/recompiling?? You make it sound
> as if there is...
>
>
Comment the following line in cassandra-env.sh:
   JVM_OPTS="$JVM_OPTS -ea"

Re: MUTATION messages dropped

[Ken Hancock]

We had issues where the number of CF families that were being flushed would
align and then block writes for a very brief period. If that happened when
a bunch of writes came in, we'd see a spike in Mutation drops.

Check nodetool tpstats for FlushWriter all time blocked.


On Thu, Dec 19, 2013 at 7:12 AM, Alexander Shutyaev wrote:

> Hi all!
>
> We've had a problem with cassandra recently. We had 2 one-minute periods
> when we got a lot of timeouts on the client side (the only timeouts during
> 9 days we are using cassandra in production). In the logs we've found
> corresponding messages saying something about MUTATION messages dropped.
>
> Now, the official faq [1] says that this is an indicator that the load is
> too high. We've checked our monitoring and found out that 1-minute average
> cpu load had a local peak at the time of the problem, but it was like 0.8
> against 0.2 usual which I guess is nothing for a 2 core virtual machine.
> We've also checked java threads - there was no peak there and their count
> was reasonable ~240-250.
>
> Can anyone give us a hint - what should we monitor to see this "high load"
> and what should we tune to make it acceptable?
>
> Thanks in advance,
> Alexander
>
> [1] http://wiki.apache.org/cassandra/FAQ#dropped_messages
>



-- 
*Ken Hancock *| System Architect, Advanced Advertising
SeaChange International
50 Nagog Park
Acton, Massachusetts 01720
ken.hanc...@schange.com | www.schange.com |
NASDAQ:SEAC

Office: +1 (978) 889-3329 | [image: Google Talk:]
ken.hanc...@schange.com | [image:
Skype:]hancockks | [image: Yahoo IM:]hancockks [image:
LinkedIn]

[image: SeaChange International]
 This e-mail and any attachments may contain
information which is SeaChange International confidential. The information
enclosed is intended only for the addressees herein and may not be copied
or forwarded without permission from SeaChange International.

Re: Paging error after upgrade from C* 2.0.1 to 2.0.3 , Driver from 2.0.0-rc1 to 2.0.0-rc2

[Jan Algermissen]

On 19.12.2013, at 14:09, Sylvain Lebresne  wrote:

> 
> 
> Is there anything I can do except waiting for a fix?
> 
> Disable paging (which might imply having to do some manual paging if the 
> result set you were querying were really big).

Yes, that was my original code - might do that.

>  
> I'll note that while far from idea, you can also disable assertion with 2.0.3.

Is there a way to do that without patching/recompiling?? You make it sound as 
if there is...


> If you do so, the code won't crash, but you might get twice the same row 
> during some paging.

No big deal - easily done in my code and far better than lossing around 5% (in 
my case) of the rows with 2.0.1

> That's still probably better than going back to 2.0.1 though.
> And of course, another option could be to apply manually the patch that is on 
> the issue and/or use the current tip of the cassandra-1.2 branch.

As I am on Debian packages, I'd rather not :-)

Thanks for the quick insights!

Jan

> 
> --
> Sylvain
> 
> 
> On 19.12.2013, at 11:39, Sylvain Lebresne  wrote:
> 
> > https://issues.apache.org/jira/browse/CASSANDRA-6447
> >
> >
> > On Thu, Dec 19, 2013 at 11:16 AM, Jan Algermissen 
> >  wrote:
> > Hi all,
> >
> > after upgrading C* and the java-driver I am running into problems with 
> > paging. Maybe someone can provide a quick clue.
> >
> > Upgrading was
> >   C* from 2.0.1 to 2.0.3
> >   Java Driver from 2.0.0-rc1 to 2.0.0-rc2
> >
> >
> >
> > Client side, I get the following messages (apparently during a call to 
> > resultSet.one() ):
> >
> >
> > com.datastax.driver.core.exceptions.DriverInternalError: An unexpected 
> > error occured server side on /37.139.24.133: java.l
> > ang.AssertionError
> > at 
> > com.datastax.driver.core.exceptions.DriverInternalError.copy(DriverInternalError.java:42)
> > at 
> > com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:271)
> > at 
> > com.datastax.driver.core.ResultSet.fetchMoreResultsBlocking(ResultSet.java:252)
> > at com.datastax.driver.core.ResultSet.one(ResultSet.java:166)
> >
> >
> >
> > Server Side:
> >
> >  INFO [HANDSHAKE-/37.139.3.70] 2013-12-19 09:55:11,277 
> > OutboundTcpConnection.java (line 386) Handshaking version with /37.139.3.70
> >  INFO [HANDSHAKE-/37.139.24.133] 2013-12-19 09:55:11,284 
> > OutboundTcpConnection.java (line 386) Handshaking version with 
> > /37.139.24.133
> >  INFO [HANDSHAKE-/37.139.24.133] 2013-12-19 09:55:11,309 
> > OutboundTcpConnection.java (line 386) Handshaking version with 
> > /37.139.24.133
> >  INFO [HANDSHAKE-/146.185.135.226] 2013-12-19 10:00:10,077 
> > OutboundTcpConnection.java (line 386) Handshaking version with 
> > /146.185.135.226
> >  WARN [ReadStage:87] 2013-12-19 10:00:16,490 SliceQueryFilter.java (line 
> > 209) Read 111 live and 1776 tombstoned cells (see tombstone_warn_threshold)
> >  WARN [ReadStage:87] 2013-12-19 10:00:16,976 SliceQueryFilter.java (line 
> > 209) Read 48 live and 1056 tombstoned cells (see tombstone_warn_threshold)
> >  WARN [ReadStage:87] 2013-12-19 10:00:18,588 SliceQueryFilter.java (line 
> > 209) Read 80 live and 1160 tombstoned cells (see tombstone_warn_threshold)
> >  WARN [ReadStage:88] 2013-12-19 10:00:24,675 SliceQueryFilter.java (line 
> > 209) Read 48 live and 1056 tombstoned cells (see tombstone_warn_threshold)
> >  WARN [ReadStage:88] 2013-12-19 10:00:25,715 SliceQueryFilter.java (line 
> > 209) Read 80 live and 1160 tombstoned cells (see tombstone_warn_threshold)
> >  WARN [ReadStage:89] 2013-12-19 10:00:31,406 SliceQueryFilter.java (line 
> > 209) Read 300 live and 6300 tombstoned cells (see tombstone_warn_threshold)
> >  WARN [ReadStage:89] 2013-12-19 10:00:32,075 SliceQueryFilter.java (line 
> > 209) Read 65 live and 1040 tombstoned cells (see tombstone_warn_threshold)
> >  WARN [ReadStage:89] 2013-12-19 10:00:33,207 SliceQueryFilter.java (line 
> > 209) Read 72 live and 1224 tombstoned cells (see tombstone_warn_threshold)
> >  WARN [ReadStage:90] 2013-12-19 10:00:37,183 SliceQueryFilter.java (line 
> > 209) Read 135 live and 1782 tombstoned cells (see tombstone_warn_threshold)
> >  INFO [ScheduledTasks:1] 2013-12-19 10:00:58,523 GCInspector.java (line 
> > 116) GC for ParNew: 213 ms for 1 collections, 720697792 used; max is 
> > 2057306112
> > ERROR [Native-Transport-Requests:216] 2013-12-19 10:00:58,913 
> > ErrorMessage.java (line 222) Unexpected exception during request
> > java.lang.AssertionError
> > at 
> > org.apache.cassandra.service.pager.AbstractQueryPager.discardFirst(AbstractQueryPager.java:183)
> > at 
> > org.apache.cassandra.service.pager.AbstractQueryPager.fetchPage(AbstractQueryPager.java:102)
> > at 
> > org.apache.cassandra.service.pager.RangeSliceQueryPager.fetchPage(RangeSliceQueryPager.java:36)
> > at 
> > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:171)
> > at 
> > org.apache.cassandra.cql3.st

Re: MUTATION messages dropped

[srmore]

What version of Cassandra are you running ? I used to see them a lot with
1.2.9, I could correlate the dropped messages with the heap usage almost
every time, so check in the logs whether you are getting GC'd. In this
respect 1.2.12 appears to be more stable. Moving to 1.2.12 took care of
this for us.

Thanks,
Sandeep


On Thu, Dec 19, 2013 at 6:12 AM, Alexander Shutyaev wrote:

> Hi all!
>
> We've had a problem with cassandra recently. We had 2 one-minute periods
> when we got a lot of timeouts on the client side (the only timeouts during
> 9 days we are using cassandra in production). In the logs we've found
> corresponding messages saying something about MUTATION messages dropped.
>
> Now, the official faq [1] says that this is an indicator that the load is
> too high. We've checked our monitoring and found out that 1-minute average
> cpu load had a local peak at the time of the problem, but it was like 0.8
> against 0.2 usual which I guess is nothing for a 2 core virtual machine.
> We've also checked java threads - there was no peak there and their count
> was reasonable ~240-250.
>
> Can anyone give us a hint - what should we monitor to see this "high load"
> and what should we tune to make it acceptable?
>
> Thanks in advance,
> Alexander
>
> [1] http://wiki.apache.org/cassandra/FAQ#dropped_messages
>

Re: Paging error after upgrade from C* 2.0.1 to 2.0.3 , Driver from 2.0.0-rc1 to 2.0.0-rc2

[Sylvain Lebresne]

>
> Is there anything I can do except waiting for a fix?
>

Disable paging (which might imply having to do some manual paging if the
result set you were querying were really big).


>
> I moved to 2.0.3 because I think I experienced missing rows in 2.0.1
> paging - is this related to the 2.0.3 bug? Meaning: going back to 2.0.1
> will fix the exception, but leave me with the faulty situation the
> assertion is there to detect?
>

Yeah basically paging is more broken in 2.0.1 than in 2.0.3 in general, but
still not fully functional because of this bug.

I'll note that while far from idea, you can also disable assertion with
2.0.3. If you do so, the code won't crash, but you might get twice the same
row during some paging. That's still probably better than going back to
2.0.1 though.
And of course, another option could be to apply manually the patch that is
on the issue and/or use the current tip of the cassandra-1.2 branch.

--
Sylvain


On 19.12.2013, at 11:39, Sylvain Lebresne  wrote:

> https://issues.apache.org/jira/browse/CASSANDRA-6447
>
>
> On Thu, Dec 19, 2013 at 11:16 AM, Jan Algermissen <
jan.algermis...@nordsc.com> wrote:
> Hi all,
>
> after upgrading C* and the java-driver I am running into problems with
paging. Maybe someone can provide a quick clue.
>
> Upgrading was
>   C* from 2.0.1 to 2.0.3
>   Java Driver from 2.0.0-rc1 to 2.0.0-rc2
>
>
>
> Client side, I get the following messages (apparently during a call to
resultSet.one() ):
>
>
> com.datastax.driver.core.exceptions.DriverInternalError: An unexpected
error occured server side on /37.139.24.133: java.l
> ang.AssertionError
> at
com.datastax.driver.core.exceptions.DriverInternalError.copy(DriverInternalError.java:42)
> at
com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:271)
> at
com.datastax.driver.core.ResultSet.fetchMoreResultsBlocking(ResultSet.java:252)
> at com.datastax.driver.core.ResultSet.one(ResultSet.java:166)
>
>
>
> Server Side:
>
>  INFO [HANDSHAKE-/37.139.3.70] 2013-12-19 09:55:11,277
OutboundTcpConnection.java (line 386) Handshaking version with /37.139.3.70
>  INFO [HANDSHAKE-/37.139.24.133] 2013-12-19 09:55:11,284
OutboundTcpConnection.java (line 386) Handshaking version with /
37.139.24.133
>  INFO [HANDSHAKE-/37.139.24.133] 2013-12-19 09:55:11,309
OutboundTcpConnection.java (line 386) Handshaking version with /
37.139.24.133
>  INFO [HANDSHAKE-/146.185.135.226] 2013-12-19 10:00:10,077
OutboundTcpConnection.java (line 386) Handshaking version with /
146.185.135.226
>  WARN [ReadStage:87] 2013-12-19 10:00:16,490 SliceQueryFilter.java (line
209) Read 111 live and 1776 tombstoned cells (see tombstone_warn_threshold)
>  WARN [ReadStage:87] 2013-12-19 10:00:16,976 SliceQueryFilter.java (line
209) Read 48 live and 1056 tombstoned cells (see tombstone_warn_threshold)
>  WARN [ReadStage:87] 2013-12-19 10:00:18,588 SliceQueryFilter.java (line
209) Read 80 live and 1160 tombstoned cells (see tombstone_warn_threshold)
>  WARN [ReadStage:88] 2013-12-19 10:00:24,675 SliceQueryFilter.java (line
209) Read 48 live and 1056 tombstoned cells (see tombstone_warn_threshold)
>  WARN [ReadStage:88] 2013-12-19 10:00:25,715 SliceQueryFilter.java (line
209) Read 80 live and 1160 tombstoned cells (see tombstone_warn_threshold)
>  WARN [ReadStage:89] 2013-12-19 10:00:31,406 SliceQueryFilter.java (line
209) Read 300 live and 6300 tombstoned cells (see tombstone_warn_threshold)
>  WARN [ReadStage:89] 2013-12-19 10:00:32,075 SliceQueryFilter.java (line
209) Read 65 live and 1040 tombstoned cells (see tombstone_warn_threshold)
>  WARN [ReadStage:89] 2013-12-19 10:00:33,207 SliceQueryFilter.java (line
209) Read 72 live and 1224 tombstoned cells (see tombstone_warn_threshold)
>  WARN [ReadStage:90] 2013-12-19 10:00:37,183 SliceQueryFilter.java (line
209) Read 135 live and 1782 tombstoned cells (see tombstone_warn_threshold)
>  INFO [ScheduledTasks:1] 2013-12-19 10:00:58,523 GCInspector.java (line
116) GC for ParNew: 213 ms for 1 collections, 720697792 used; max is
2057306112
> ERROR [Native-Transport-Requests:216] 2013-12-19 10:00:58,913
ErrorMessage.java (line 222) Unexpected exception during request
> java.lang.AssertionError
> at
org.apache.cassandra.service.pager.AbstractQueryPager.discardFirst(AbstractQueryPager.java:183)
> at
org.apache.cassandra.service.pager.AbstractQueryPager.fetchPage(AbstractQueryPager.java:102)
> at
org.apache.cassandra.service.pager.RangeSliceQueryPager.fetchPage(RangeSliceQueryPager.java:36)
> at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:171)
> at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:58)
> at
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188)
> at
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:222)
> at
org.apache.cassandra.transpor

MUTATION messages dropped

[Alexander Shutyaev]

Hi all!

We've had a problem with cassandra recently. We had 2 one-minute periods
when we got a lot of timeouts on the client side (the only timeouts during
9 days we are using cassandra in production). In the logs we've found
corresponding messages saying something about MUTATION messages dropped.

Now, the official faq [1] says that this is an indicator that the load is
too high. We've checked our monitoring and found out that 1-minute average
cpu load had a local peak at the time of the problem, but it was like 0.8
against 0.2 usual which I guess is nothing for a 2 core virtual machine.
We've also checked java threads - there was no peak there and their count
was reasonable ~240-250.

Can anyone give us a hint - what should we monitor to see this "high load"
and what should we tune to make it acceptable?

Thanks in advance,
Alexander

[1] http://wiki.apache.org/cassandra/FAQ#dropped_messages

Re: cassandra performance problems

[Alexander Shutyaev]

Thanks all for your responses. We've downgraded from 2.0.3 to 2.0.0 and
everything became normal.


2013/12/8 Nate McCall 

> If you are really set on using Cassandra as a cache, I would recommend
> disabling durable writes for the keyspace(s)[0]. This will bypass the
> commitlog (the flushing/rotation of which my be a good-sized portion of
> your performance problems given the number of tables).
>
> [0]
> http://www.datastax.com/documentation/cql/3.0/webhelp/index.html#cql/cql_reference/alter_keyspace_r.html
>
>
> On Fri, Dec 6, 2013 at 12:42 PM, J. Ryan Earl  wrote:
>
>> On Thu, Dec 5, 2013 at 6:33 AM, Alexander Shutyaev wrote:
>>
>>> We've plugged it into our production environment as a cache in front of
>>> postgres. Everything worked fine, we even stressed it by explicitly
>>> propagating about 30G (10G/node) data from postgres to cassandra.
>>>
>>
>> If you just want a caching layer, why wouldn't you use Memcached or Redis
>> instead?  Cassandra is designed to be a persist store and not so much
>> designed as caching layer.  If you were replacing your use of Postgres
>> completely, that would be appropriate.
>>
>
>
>
> --
> -
> Nate McCall
> Austin, TX
> @zznate
>
> Co-Founder & Sr. Technical Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>

Re: Paging error after upgrade from C* 2.0.1 to 2.0.3 , Driver from 2.0.0-rc1 to 2.0.0-rc2

[Jan Algermissen]

Sylvain,

thanks.

Is there anything I can do except waiting for a fix?

Could I do something to my data? Or data model?

I moved to 2.0.3 because I think I experienced missing rows in 2.0.1 paging - 
is this related to the 2.0.3 bug? Meaning: going back to 2.0.1 will fix the 
exception, but leave me with the faulty situation the assertion is there to 
detect?

Jan


On 19.12.2013, at 11:39, Sylvain Lebresne  wrote:

> https://issues.apache.org/jira/browse/CASSANDRA-6447
> 
> 
> On Thu, Dec 19, 2013 at 11:16 AM, Jan Algermissen 
>  wrote:
> Hi all,
> 
> after upgrading C* and the java-driver I am running into problems with 
> paging. Maybe someone can provide a quick clue.
> 
> Upgrading was
>   C* from 2.0.1 to 2.0.3
>   Java Driver from 2.0.0-rc1 to 2.0.0-rc2
> 
> 
> 
> Client side, I get the following messages (apparently during a call to 
> resultSet.one() ):
> 
> 
> com.datastax.driver.core.exceptions.DriverInternalError: An unexpected error 
> occured server side on /37.139.24.133: java.l
> ang.AssertionError
> at 
> com.datastax.driver.core.exceptions.DriverInternalError.copy(DriverInternalError.java:42)
> at 
> com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:271)
> at 
> com.datastax.driver.core.ResultSet.fetchMoreResultsBlocking(ResultSet.java:252)
> at com.datastax.driver.core.ResultSet.one(ResultSet.java:166)
>
> 
> 
> Server Side:
> 
>  INFO [HANDSHAKE-/37.139.3.70] 2013-12-19 09:55:11,277 
> OutboundTcpConnection.java (line 386) Handshaking version with /37.139.3.70
>  INFO [HANDSHAKE-/37.139.24.133] 2013-12-19 09:55:11,284 
> OutboundTcpConnection.java (line 386) Handshaking version with /37.139.24.133
>  INFO [HANDSHAKE-/37.139.24.133] 2013-12-19 09:55:11,309 
> OutboundTcpConnection.java (line 386) Handshaking version with /37.139.24.133
>  INFO [HANDSHAKE-/146.185.135.226] 2013-12-19 10:00:10,077 
> OutboundTcpConnection.java (line 386) Handshaking version with 
> /146.185.135.226
>  WARN [ReadStage:87] 2013-12-19 10:00:16,490 SliceQueryFilter.java (line 209) 
> Read 111 live and 1776 tombstoned cells (see tombstone_warn_threshold)
>  WARN [ReadStage:87] 2013-12-19 10:00:16,976 SliceQueryFilter.java (line 209) 
> Read 48 live and 1056 tombstoned cells (see tombstone_warn_threshold)
>  WARN [ReadStage:87] 2013-12-19 10:00:18,588 SliceQueryFilter.java (line 209) 
> Read 80 live and 1160 tombstoned cells (see tombstone_warn_threshold)
>  WARN [ReadStage:88] 2013-12-19 10:00:24,675 SliceQueryFilter.java (line 209) 
> Read 48 live and 1056 tombstoned cells (see tombstone_warn_threshold)
>  WARN [ReadStage:88] 2013-12-19 10:00:25,715 SliceQueryFilter.java (line 209) 
> Read 80 live and 1160 tombstoned cells (see tombstone_warn_threshold)
>  WARN [ReadStage:89] 2013-12-19 10:00:31,406 SliceQueryFilter.java (line 209) 
> Read 300 live and 6300 tombstoned cells (see tombstone_warn_threshold)
>  WARN [ReadStage:89] 2013-12-19 10:00:32,075 SliceQueryFilter.java (line 209) 
> Read 65 live and 1040 tombstoned cells (see tombstone_warn_threshold)
>  WARN [ReadStage:89] 2013-12-19 10:00:33,207 SliceQueryFilter.java (line 209) 
> Read 72 live and 1224 tombstoned cells (see tombstone_warn_threshold)
>  WARN [ReadStage:90] 2013-12-19 10:00:37,183 SliceQueryFilter.java (line 209) 
> Read 135 live and 1782 tombstoned cells (see tombstone_warn_threshold)
>  INFO [ScheduledTasks:1] 2013-12-19 10:00:58,523 GCInspector.java (line 116) 
> GC for ParNew: 213 ms for 1 collections, 720697792 used; max is 2057306112
> ERROR [Native-Transport-Requests:216] 2013-12-19 10:00:58,913 
> ErrorMessage.java (line 222) Unexpected exception during request
> java.lang.AssertionError
> at 
> org.apache.cassandra.service.pager.AbstractQueryPager.discardFirst(AbstractQueryPager.java:183)
> at 
> org.apache.cassandra.service.pager.AbstractQueryPager.fetchPage(AbstractQueryPager.java:102)
> at 
> org.apache.cassandra.service.pager.RangeSliceQueryPager.fetchPage(RangeSliceQueryPager.java:36)
> at 
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:171)
> at 
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:58)
> at 
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188)
> at 
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:222)
> at 
> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:119)
> at 
> org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304)
> at 
> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
> at 
> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.co

Re: Paging error after upgrade from C* 2.0.1 to 2.0.3 , Driver from 2.0.0-rc1 to 2.0.0-rc2

[Sylvain Lebresne]

https://issues.apache.org/jira/browse/CASSANDRA-6447


On Thu, Dec 19, 2013 at 11:16 AM, Jan Algermissen <
jan.algermis...@nordsc.com> wrote:

> Hi all,
>
> after upgrading C* and the java-driver I am running into problems with
> paging. Maybe someone can provide a quick clue.
>
> Upgrading was
>   C* from 2.0.1 to 2.0.3
>   Java Driver from 2.0.0-rc1 to 2.0.0-rc2
>
>
>
> Client side, I get the following messages (apparently during a call to
> resultSet.one() ):
>
>
> com.datastax.driver.core.exceptions.DriverInternalError: An unexpected
> error occured server side on /37.139.24.133: java.l
> ang.AssertionError
> at
> com.datastax.driver.core.exceptions.DriverInternalError.copy(DriverInternalError.java:42)
> at
> com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:271)
> at
> com.datastax.driver.core.ResultSet.fetchMoreResultsBlocking(ResultSet.java:252)
> at com.datastax.driver.core.ResultSet.one(ResultSet.java:166)
>
>
>
> Server Side:
>
>  INFO [HANDSHAKE-/37.139.3.70] 2013-12-19 09:55:11,277
> OutboundTcpConnection.java (line 386) Handshaking version with /
> 37.139.3.70
>  INFO [HANDSHAKE-/37.139.24.133] 2013-12-19 09:55:11,284
> OutboundTcpConnection.java (line 386) Handshaking version with /
> 37.139.24.133
>  INFO [HANDSHAKE-/37.139.24.133] 2013-12-19 09:55:11,309
> OutboundTcpConnection.java (line 386) Handshaking version with /
> 37.139.24.133
>  INFO [HANDSHAKE-/146.185.135.226] 2013-12-19 10:00:10,077
> OutboundTcpConnection.java (line 386) Handshaking version with /
> 146.185.135.226
>  WARN [ReadStage:87] 2013-12-19 10:00:16,490 SliceQueryFilter.java (line
> 209) Read 111 live and 1776 tombstoned cells (see tombstone_warn_threshold)
>  WARN [ReadStage:87] 2013-12-19 10:00:16,976 SliceQueryFilter.java (line
> 209) Read 48 live and 1056 tombstoned cells (see tombstone_warn_threshold)
>  WARN [ReadStage:87] 2013-12-19 10:00:18,588 SliceQueryFilter.java (line
> 209) Read 80 live and 1160 tombstoned cells (see tombstone_warn_threshold)
>  WARN [ReadStage:88] 2013-12-19 10:00:24,675 SliceQueryFilter.java (line
> 209) Read 48 live and 1056 tombstoned cells (see tombstone_warn_threshold)
>  WARN [ReadStage:88] 2013-12-19 10:00:25,715 SliceQueryFilter.java (line
> 209) Read 80 live and 1160 tombstoned cells (see tombstone_warn_threshold)
>  WARN [ReadStage:89] 2013-12-19 10:00:31,406 SliceQueryFilter.java (line
> 209) Read 300 live and 6300 tombstoned cells (see tombstone_warn_threshold)
>  WARN [ReadStage:89] 2013-12-19 10:00:32,075 SliceQueryFilter.java (line
> 209) Read 65 live and 1040 tombstoned cells (see tombstone_warn_threshold)
>  WARN [ReadStage:89] 2013-12-19 10:00:33,207 SliceQueryFilter.java (line
> 209) Read 72 live and 1224 tombstoned cells (see tombstone_warn_threshold)
>  WARN [ReadStage:90] 2013-12-19 10:00:37,183 SliceQueryFilter.java (line
> 209) Read 135 live and 1782 tombstoned cells (see tombstone_warn_threshold)
>  INFO [ScheduledTasks:1] 2013-12-19 10:00:58,523 GCInspector.java (line
> 116) GC for ParNew: 213 ms for 1 collections, 720697792 used; max is
> 2057306112
> ERROR [Native-Transport-Requests:216] 2013-12-19 10:00:58,913
> ErrorMessage.java (line 222) Unexpected exception during request
> java.lang.AssertionError
> at
> org.apache.cassandra.service.pager.AbstractQueryPager.discardFirst(AbstractQueryPager.java:183)
> at
> org.apache.cassandra.service.pager.AbstractQueryPager.fetchPage(AbstractQueryPager.java:102)
> at
> org.apache.cassandra.service.pager.RangeSliceQueryPager.fetchPage(RangeSliceQueryPager.java:36)
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:171)
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:58)
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188)
> at
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:222)
> at
> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:119)
> at
> org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304)
> at
> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
> at
> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
>
>
>
>
> Jan

org.apache.thrift.TApplicationException: get_range_slices failed: out of sequence response

[Jason Wee]

Hi,

In regards to recv_get_range_slices(), in my cassandra client code, it
always throw new
org.apache.thrift.TApplicationException(org.apache.thrift.TApplicationException.BAD_SEQUENCE_ID,
"get_range_slices failed: out of sequence response");


Full source code here

https://raw.github.com/apache/cassandra/cassandra-1.0.12/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java

Can anybody give some clue as to why the exception is always thrown?
My cassandra use is version 0.7.10 and the code checking msg.seqid
remain until version 1.0.12. I know it is ancient cassandra but just
wanna learn it. I have checked cassandra 1.1 branch, the checking
condition if (msg.seqid != seqid_) { was removed.

Thank you and any clue or indication will be great.

/Jason

Paging error after upgrade from C* 2.0.1 to 2.0.3 , Driver from 2.0.0-rc1 to 2.0.0-rc2

[Jan Algermissen]

Hi all,

after upgrading C* and the java-driver I am running into problems with paging. 
Maybe someone can provide a quick clue.

Upgrading was 
  C* from 2.0.1 to 2.0.3
  Java Driver from 2.0.0-rc1 to 2.0.0-rc2



Client side, I get the following messages (apparently during a call to 
resultSet.one() ):


com.datastax.driver.core.exceptions.DriverInternalError: An unexpected error 
occured server side on /37.139.24.133: java.l
ang.AssertionError
at 
com.datastax.driver.core.exceptions.DriverInternalError.copy(DriverInternalError.java:42)
at 
com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:271)
at 
com.datastax.driver.core.ResultSet.fetchMoreResultsBlocking(ResultSet.java:252)
at com.datastax.driver.core.ResultSet.one(ResultSet.java:166)
   


Server Side:

 INFO [HANDSHAKE-/37.139.3.70] 2013-12-19 09:55:11,277 
OutboundTcpConnection.java (line 386) Handshaking version with /37.139.3.70
 INFO [HANDSHAKE-/37.139.24.133] 2013-12-19 09:55:11,284 
OutboundTcpConnection.java (line 386) Handshaking version with /37.139.24.133
 INFO [HANDSHAKE-/37.139.24.133] 2013-12-19 09:55:11,309 
OutboundTcpConnection.java (line 386) Handshaking version with /37.139.24.133
 INFO [HANDSHAKE-/146.185.135.226] 2013-12-19 10:00:10,077 
OutboundTcpConnection.java (line 386) Handshaking version with /146.185.135.226
 WARN [ReadStage:87] 2013-12-19 10:00:16,490 SliceQueryFilter.java (line 209) 
Read 111 live and 1776 tombstoned cells (see tombstone_warn_threshold)
 WARN [ReadStage:87] 2013-12-19 10:00:16,976 SliceQueryFilter.java (line 209) 
Read 48 live and 1056 tombstoned cells (see tombstone_warn_threshold)
 WARN [ReadStage:87] 2013-12-19 10:00:18,588 SliceQueryFilter.java (line 209) 
Read 80 live and 1160 tombstoned cells (see tombstone_warn_threshold)
 WARN [ReadStage:88] 2013-12-19 10:00:24,675 SliceQueryFilter.java (line 209) 
Read 48 live and 1056 tombstoned cells (see tombstone_warn_threshold)
 WARN [ReadStage:88] 2013-12-19 10:00:25,715 SliceQueryFilter.java (line 209) 
Read 80 live and 1160 tombstoned cells (see tombstone_warn_threshold)
 WARN [ReadStage:89] 2013-12-19 10:00:31,406 SliceQueryFilter.java (line 209) 
Read 300 live and 6300 tombstoned cells (see tombstone_warn_threshold)
 WARN [ReadStage:89] 2013-12-19 10:00:32,075 SliceQueryFilter.java (line 209) 
Read 65 live and 1040 tombstoned cells (see tombstone_warn_threshold)
 WARN [ReadStage:89] 2013-12-19 10:00:33,207 SliceQueryFilter.java (line 209) 
Read 72 live and 1224 tombstoned cells (see tombstone_warn_threshold)
 WARN [ReadStage:90] 2013-12-19 10:00:37,183 SliceQueryFilter.java (line 209) 
Read 135 live and 1782 tombstoned cells (see tombstone_warn_threshold)
 INFO [ScheduledTasks:1] 2013-12-19 10:00:58,523 GCInspector.java (line 116) GC 
for ParNew: 213 ms for 1 collections, 720697792 used; max is 2057306112
ERROR [Native-Transport-Requests:216] 2013-12-19 10:00:58,913 ErrorMessage.java 
(line 222) Unexpected exception during request
java.lang.AssertionError
at 
org.apache.cassandra.service.pager.AbstractQueryPager.discardFirst(AbstractQueryPager.java:183)
at 
org.apache.cassandra.service.pager.AbstractQueryPager.fetchPage(AbstractQueryPager.java:102)
at 
org.apache.cassandra.service.pager.RangeSliceQueryPager.fetchPage(RangeSliceQueryPager.java:36)
at 
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:171)
at 
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:58)
at 
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188)
at 
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:222)
at 
org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:119)
at 
org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304)
at 
org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
at 
org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)




Jan

Re: Occasional NPE using DataStax Java driver

[Mikhail Stepura]

I would suggest to file an issue at 
https://datastax-oss.atlassian.net/browse/JAVA


-Mikhail

On 12/18/13, 23:21, David Tinker wrote:

We are using Cassandra 2.0.3-1 installed on Ubuntu 12.04 from the
DataStax repo with the DataStax Java driver version 2.0.0-rc1. Every
now and then we get the following exception:

2013-12-19 06:56:34,619 [sql-2-t15] ERROR core.RequestHandler  -
Unexpected error while querying /x.x.x.x
java.lang.NullPointerException
at 
com.datastax.driver.core.HostConnectionPool.waitForConnection(HostConnectionPool.java:203)
at 
com.datastax.driver.core.HostConnectionPool.borrowConnection(HostConnectionPool.java:107)
at com.datastax.driver.core.RequestHandler.query(RequestHandler.java:112)
at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:93)
at com.datastax.driver.core.Session$Manager.execute(Session.java:513)
at com.datastax.driver.core.Session$Manager.executeQuery(Session.java:549)
at com.datastax.driver.core.Session.executeAsync(Session.java:172)

This happens during a big data load process which will do up to 256
executeAsync's in parallel.

Any ideas? Its not causing huge problems because the operation is just
retried by our code but it would be nice to eliminate it.

Re: Occasional NPE using DataStax Java driver

[Mikhail Stepura]

I would suggest to file an issue at 
https://datastax-oss.atlassian.net/browse/JAVA


-Mikhail

On 12/18/13, 23:21, David Tinker wrote:

We are using Cassandra 2.0.3-1 installed on Ubuntu 12.04 from the
DataStax repo with the DataStax Java driver version 2.0.0-rc1. Every
now and then we get the following exception:

2013-12-19 06:56:34,619 [sql-2-t15] ERROR core.RequestHandler  -
Unexpected error while querying /x.x.x.x
java.lang.NullPointerException
at 
com.datastax.driver.core.HostConnectionPool.waitForConnection(HostConnectionPool.java:203)
at 
com.datastax.driver.core.HostConnectionPool.borrowConnection(HostConnectionPool.java:107)
at com.datastax.driver.core.RequestHandler.query(RequestHandler.java:112)
at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:93)
at com.datastax.driver.core.Session$Manager.execute(Session.java:513)
at com.datastax.driver.core.Session$Manager.executeQuery(Session.java:549)
at com.datastax.driver.core.Session.executeAsync(Session.java:172)

This happens during a big data load process which will do up to 256
executeAsync's in parallel.

Any ideas? Its not causing huge problems because the operation is just
retried by our code but it would be nice to eliminate it.

Re: Occasional NPE using DataStax Java driver

[Sylvain Lebresne]

Mind opening a ticket on https://datastax-oss.atlassian.net/browse/JAVA?
It's almost surely a bug.

--
Sylvain


On Thu, Dec 19, 2013 at 8:21 AM, David Tinker wrote:

> We are using Cassandra 2.0.3-1 installed on Ubuntu 12.04 from the
> DataStax repo with the DataStax Java driver version 2.0.0-rc1. Every
> now and then we get the following exception:
>
> 2013-12-19 06:56:34,619 [sql-2-t15] ERROR core.RequestHandler  -
> Unexpected error while querying /x.x.x.x
> java.lang.NullPointerException
> at
> com.datastax.driver.core.HostConnectionPool.waitForConnection(HostConnectionPool.java:203)
> at
> com.datastax.driver.core.HostConnectionPool.borrowConnection(HostConnectionPool.java:107)
> at com.datastax.driver.core.RequestHandler.query(RequestHandler.java:112)
> at
> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:93)
> at com.datastax.driver.core.Session$Manager.execute(Session.java:513)
> at com.datastax.driver.core.Session$Manager.executeQuery(Session.java:549)
> at com.datastax.driver.core.Session.executeAsync(Session.java:172)
>
> This happens during a big data load process which will do up to 256
> executeAsync's in parallel.
>
> Any ideas? Its not causing huge problems because the operation is just
> retried by our code but it would be nice to eliminate it.
>

Re: Setting up Cassandra to store on a specific node and not replicate

[Sylvain Lebresne]

On Wed, Dec 18, 2013 at 7:31 PM, Robert Coli  wrote:

> On Wed, Dec 18, 2013 at 2:44 AM, Sylvain Lebresne wrote:
>
>> As Janne said, you could still have hint being written by other nodes if
>> the one storage node is dead, but you can use the system
>> property cassandra.maxHintTTL to 0 to disable hints.
>>
>
> If one uses a Token Aware client with RF=1, that would seem to preclude
> hinting even without disabling HH for the entire system; if the coordinator
> is always the single replica, why would it send a copy anywhere else?
>

Colin explicitly said that he would several nodes and I said I wasn't going
to judge, so I implicitly assumed there was a reason for having multiple
nodes.

If you're going to always ever hit one node, then using a token aware
client is over-complicating it. Just use a one node cluster and you'll have
nothing to worry about or to configure.

That being said, Colin, do be aware that as far as I can tell there is
indeed relatively little benefit to having a multi-node cluster on which
all data is on one node (in particular, there is no cache at the
coordinator level, so that even if your client hit other nodes, everything
will still be forwarded to the one node that stores it all, the other nodes
won't store anything really, not even in memory).

--
Sylvain

Re: Setting up Cassandra to store on a specific node and not replicate

[Janne Jalkanen]

Probably yes, if you also disabled any sort of failovers from the token-aware 
client…

(Talking about this makes you realize how many failsafes Cassandra has. And 
still you can lose data… :-P)

/Janne

On 18 Dec 2013, at 20:31, Robert Coli  wrote:

> On Wed, Dec 18, 2013 at 2:44 AM, Sylvain Lebresne  
> wrote:
> As Janne said, you could still have hint being written by other nodes if the 
> one storage node is dead, but you can use the system property 
> cassandra.maxHintTTL to 0 to disable hints.
> 
> If one uses a Token Aware client with RF=1, that would seem to preclude 
> hinting even without disabling HH for the entire system; if the coordinator 
> is always the single replica, why would it send a copy anywhere else?
> 
> =Rob