NPE during schema upgrade from 2.2.6 -> 3.0.6

[Max C]

Hi Everyone,

I’m getting a NullPointerException when I start up 3.0.6 for the first time 
with data from 2.2.6.  Any ideas for how to fix this, or other troubleshooting 
strategies?  

This is just a single-node development box.  

Originally I tried upgrading from 2.1.13 to 3.0.6, but I ran into the same 
error; so I then went from 2.1.13 to 2.2.6, ran “nodetool upgradestables" 
(which succeeded without issue) and then to 3.0.6.

Thanks for any assistance.  :-)

- Max

INFO  [main] 2016-05-27 16:02:01,169 ColumnFamilyStore.java:381 - Initializing 
system.peers
INFO  [main] 2016-05-27 16:02:01,178 ColumnFamilyStore.java:381 - Initializing 
system.peer_events
INFO  [main] 2016-05-27 16:02:01,182 ColumnFamilyStore.java:381 - Initializing 
system.range_xfers
INFO  [main] 2016-05-27 16:02:01,188 ColumnFamilyStore.java:381 - Initializing 
system.compaction_history
INFO  [main] 2016-05-27 16:02:01,200 ColumnFamilyStore.java:381 - Initializing 
system.sstable_activity
INFO  [main] 2016-05-27 16:02:01,210 ColumnFamilyStore.java:381 - Initializing 
system.size_estimates
INFO  [main] 2016-05-27 16:02:01,218 ColumnFamilyStore.java:381 - Initializing 
system.available_ranges
INFO  [main] 2016-05-27 16:02:01,223 ColumnFamilyStore.java:381 - Initializing 
system.views_builds_in_progress
INFO  [main] 2016-05-27 16:02:01,227 ColumnFamilyStore.java:381 - Initializing 
system.built_views
INFO  [main] 2016-05-27 16:02:01,230 ColumnFamilyStore.java:381 - Initializing 
system.hints
INFO  [main] 2016-05-27 16:02:01,234 ColumnFamilyStore.java:381 - Initializing 
system.batchlog
INFO  [main] 2016-05-27 16:02:01,238 ColumnFamilyStore.java:381 - Initializing 
system.schema_keyspaces
INFO  [main] 2016-05-27 16:02:01,245 ColumnFamilyStore.java:381 - Initializing 
system.schema_columnfamilies
INFO  [main] 2016-05-27 16:02:01,252 ColumnFamilyStore.java:381 - Initializing 
system.schema_columns
INFO  [main] 2016-05-27 16:02:01,260 ColumnFamilyStore.java:381 - Initializing 
system.schema_triggers
INFO  [main] 2016-05-27 16:02:01,268 ColumnFamilyStore.java:381 - Initializing 
system.schema_usertypes
INFO  [main] 2016-05-27 16:02:01,275 ColumnFamilyStore.java:381 - Initializing 
system.schema_functions
INFO  [main] 2016-05-27 16:02:01,282 ColumnFamilyStore.java:381 - Initializing 
system.schema_aggregates
INFO  [main] 2016-05-27 16:02:01,421 SystemKeyspace.java:1284 - Detected 
version upgrade from 2.2.6 to 3.0.6, snapshotting system keyspace
WARN  [main] 2016-05-27 16:02:01,711 CompressionParams.java:382 - The 
sstable_compression option has been deprecated. You should use class instead
ERROR [main] 2016-05-27 16:02:01,833 CassandraDaemon.java:692 - Exception 
encountered during startup
java.lang.NullPointerException: null
at 
org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:156) 
~[apache-cassandra-3.0.6.jar:3.0.6]
at 
org.apache.cassandra.serializers.AbstractTextSerializer.deserialize(AbstractTextSerializer.java:41)
 ~[apache-cassandra-3.0.6.jar:3.0.6]
at 
org.apache.cassandra.serializers.AbstractTextSerializer.deserialize(AbstractTextSerializer.java:28)
 ~[apache-cassandra-3.0.6.jar:3.0.6]
at 
org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:114) 
~[apache-cassandra-3.0.6.jar:3.0.6]
at 
org.apache.cassandra.cql3.UntypedResultSet$Row.getString(UntypedResultSet.java:267)
 ~[apache-cassandra-3.0.6.jar:3.0.6]
at 
org.apache.cassandra.schema.LegacySchemaMigrator.isEmptyCompactValueColumn(LegacySchemaMigrator.java:553)
 ~[apache-cassandra-3.0.6.jar:3.0.6]
at 
org.apache.cassandra.schema.LegacySchemaMigrator.createColumnsFromColumnRows(LegacySchemaMigrator.java:638)
 ~[apache-cassandra-3.0.6.jar:3.0.6]
at 
org.apache.cassandra.schema.LegacySchemaMigrator.decodeTableMetadata(LegacySchemaMigrator.java:316)
 ~[apache-cassandra-3.0.6.jar:3.0.6]
at 
org.apache.cassandra.schema.LegacySchemaMigrator.readTableMetadata(LegacySchemaMigrator.java:273)
 ~[apache-cassandra-3.0.6.jar:3.0.6]
at 
org.apache.cassandra.schema.LegacySchemaMigrator.readTable(LegacySchemaMigrator.java:244)
 ~[apache-cassandra-3.0.6.jar:3.0.6]
at 
org.apache.cassandra.schema.LegacySchemaMigrator.lambda$readTables$243(LegacySchemaMigrator.java:237)
 ~[apache-cassandra-3.0.6.jar:3.0.6]
at java.util.ArrayList.forEach(ArrayList.java:1249) ~[na:1.8.0_74]
at 
org.apache.cassandra.schema.LegacySchemaMigrator.readTables(LegacySchemaMigrator.java:237)
 ~[apache-cassandra-3.0.6.jar:3.0.6]
at 
org.apache.cassandra.schema.LegacySchemaMigrator.readKeyspace(LegacySchemaMigrator.java:186)
 ~[apache-cassandra-3.0.6.jar:3.0.6]
at 
org.apache.cassandra.schema.LegacySchemaMigrator.lambda$readSchema$240(LegacySchemaMigrator.java:177)
 ~[apache-cassandra-3.0.6.jar:3.0.6]
at java.util.ArrayList.forEach(ArrayList.java:1249) ~[na:1.8.0_74]
at 
org.apache.cassandra.schema.LegacySchemaMigrator.readSchema(LegacySchemaMigrator.java:177)
 ~[apache-cassandra-3.0.6.jar:3.0.6]
at 

C* working with Open JDK but not Oracle JDK

[Drew Davis]

I am running C* 2.1.4 on an Ubuntu vagrant VM. When I use OpenJDK 8, C* works 
fine. However using Oracle JDK 8, I get the errors listed below. I have read 
that it is not recommended to use Open JDK so I would like to be able to use 
the Oracle JDK. I am stumped on this issue and haven’t been able to find much 
help online. If anyone could point me in the right direction it would help a 
lot. 

vagrant@vagrant-ubuntu-wily-64:/specx$ rake cassandra:create
/usr/local/rvm/gems/ruby-2.2.1/gems/activesupport-4.1.0/lib/active_support/values/time_zone.rb:285:
 warning: circular argument reference - now
/usr/local/rvm/gems/ruby-2.2.1/gems/activerecord-4.1.0/lib/active_record/associations/has_many_association.rb:74:
 warning: circular argument reference - reflection
/usr/local/rvm/gems/ruby-2.2.1/gems/activerecord-4.1.0/lib/active_record/associations/has_many_association.rb:78:
 warning: circular argument reference - reflection
/usr/local/rvm/gems/ruby-2.2.1/gems/activerecord-4.1.0/lib/active_record/associations/has_many_association.rb:82:
 warning: circular argument reference - reflection
/usr/local/rvm/gems/ruby-2.2.1/gems/activerecord-4.1.0/lib/active_record/associations/has_many_association.rb:101:
 warning: circular argument reference - reflection
Keyspace 'specx_dev' does not exist : Keyspace 'specx_dev' does not exist
rake aborted!
CassandraMigrations::Errors::UnexistingKeyspaceError: Keyspace specx_dev does 
not exist. Run rake cassandra:create. 
/usr/local/rvm/gems/ruby-2.2.1/bundler/gems/cassandra_migrations-f5f35feeb972/lib/cassandra_migrations/cassandra/keyspace_operations.rb:30:in
 `rescue in drop_keyspace!'
/usr/local/rvm/gems/ruby-2.2.1/bundler/gems/cassandra_migrations-f5f35feeb972/lib/cassandra_migrations/cassandra/keyspace_operations.rb:27:in
 `drop_keyspace!'
/usr/local/rvm/gems/ruby-2.2.1/bundler/gems/cassandra_migrations-f5f35feeb972/lib/cassandra_migrations/cassandra/keyspace_operations.rb:20:in
 `rescue in create_keyspace!'
/usr/local/rvm/gems/ruby-2.2.1/bundler/gems/cassandra_migrations-f5f35feeb972/lib/cassandra_migrations/cassandra/keyspace_operations.rb:10:in
 `create_keyspace!'
/usr/local/rvm/gems/ruby-2.2.1/bundler/gems/cassandra_migrations-f5f35feeb972/lib/cassandra_migrations/railtie/tasks.rake:15:in
 `rescue in block (2 levels) in '
/usr/local/rvm/gems/ruby-2.2.1/bundler/gems/cassandra_migrations-f5f35feeb972/lib/cassandra_migrations/railtie/tasks.rake:11:in
 `block (2 levels) in '
/usr/local/rvm/gems/ruby-2.2.1/bin/ruby_executable_hooks:15:in `eval'
/usr/local/rvm/gems/ruby-2.2.1/bin/ruby_executable_hooks:15:in `'
Cassandra::Errors::ConfigurationError: Cannot drop non existing keyspace 
'specx_dev'.
/usr/local/rvm/gems/ruby-2.2.1/gems/cassandra-driver-2.1.4/lib/cassandra/future.rb:570:in
 `get'
/usr/local/rvm/gems/ruby-2.2.1/gems/cassandra-driver-2.1.4/lib/cassandra/future.rb:363:in
 `get'
/usr/local/rvm/gems/ruby-2.2.1/gems/cassandra-driver-2.1.4/lib/cassandra/session.rb:118:in
 `execute'
/usr/local/rvm/gems/ruby-2.2.1/bundler/gems/cassandra_migrations-f5f35feeb972/lib/cassandra_migrations/cql-rb-wrapper.rb:69:in
 `execute'
/usr/local/rvm/gems/ruby-2.2.1/bundler/gems/cassandra_migrations-f5f35feeb972/lib/cassandra_migrations/cassandra.rb:63:in
 `execute'
/usr/local/rvm/gems/ruby-2.2.1/bundler/gems/cassandra_migrations-f5f35feeb972/lib/cassandra_migrations/cassandra/keyspace_operations.rb:28:in
 `drop_keyspace!'
/usr/local/rvm/gems/ruby-2.2.1/bundler/gems/cassandra_migrations-f5f35feeb972/lib/cassandra_migrations/cassandra/keyspace_operations.rb:20:in
 `rescue in create_keyspace!'
/usr/local/rvm/gems/ruby-2.2.1/bundler/gems/cassandra_migrations-f5f35feeb972/lib/cassandra_migrations/cassandra/keyspace_operations.rb:10:in
 `create_keyspace!'
/usr/local/rvm/gems/ruby-2.2.1/bundler/gems/cassandra_migrations-f5f35feeb972/lib/cassandra_migrations/railtie/tasks.rake:15:in
 `rescue in block (2 levels) in '
/usr/local/rvm/gems/ruby-2.2.1/bundler/gems/cassandra_migrations-f5f35feeb972/lib/cassandra_migrations/railtie/tasks.rake:11:in
 `block (2 levels) in '
/usr/local/rvm/gems/ruby-2.2.1/bin/ruby_executable_hooks:15:in `eval'
/usr/local/rvm/gems/ruby-2.2.1/bin/ruby_executable_hooks:15:in `'
Cassandra::Errors::NoHostsAvailable: All attempted hosts failed: 127.0.0.1 
(Cassandra::Errors::ServerError: java.lang.RuntimeException: 
java.util.concurrent.ExecutionException: java.lang.AssertionError)
/usr/local/rvm/gems/ruby-2.2.1/gems/cassandra-driver-2.1.4/lib/cassandra/future.rb:570:in
 `get'
/usr/local/rvm/gems/ruby-2.2.1/gems/cassandra-driver-2.1.4/lib/cassandra/future.rb:363:in
 `get'
/usr/local/rvm/gems/ruby-2.2.1/gems/cassandra-driver-2.1.4/lib/cassandra/session.rb:118:in
 `execute'
/usr/local/rvm/gems/ruby-2.2.1/bundler/gems/cassandra_migrations-f5f35feeb972/lib/cassandra_migrations/cql-rb-wrapper.rb:69:in
 `execute'
/usr/local/rvm/gems/ruby-2.2.1/bundler/gems/cassandra_migrations-f5f35feeb972/lib/cassandra_migrations/cassandra.rb:63:in
 `execute'

Per node limit for Disk Space

[Anshu Vajpayee]

Hi All,
I have question regarding max disk space limit  on a node.

As per Data stax, We can have 1TB max disk space for rotational disks and
up to 5 TB for SSDs on a node.

Could you please suggest per your experience what would be limit for space
on a single node with out causing so much stress on a  node?





*​Thanks,​*

Re: Error while rebuilding a node: Stream failed

[George Sigletos]

I am trying once more using more aggressive tcp settings, as recommended
here


sudo sysctl -w net.ipv4.tcp_keepalive_time=60
net.ipv4.tcp_keepalive_probes=3 net.ipv4.tcp_keepalive_intvl=10

(added to /etc/sysctl.conf and run sysctl -p /etc/sysctl.conf on all nodes)

Let's see what happens. I don't know what else to try. I have even further
increased streaming_socket_timeout_in_ms



On Fri, May 27, 2016 at 4:56 PM, Paulo Motta 
wrote:

> I'm afraid raising streaming_socket_timeout_in_ms won't help much in this
> case because the incoming connection on the source node is timing out on
> the network layer, and streaming_socket_timeout_in_ms controls the socket
> timeout in the app layer and throws SocketTimeoutException (not 
> java.io.IOException:
> Connection timed out). So you should probably use more aggressive tcp
> keep-alive settings (net.ipv4.tcp_keepalive_*) on both hosts, did you try
> tuning that? Even that might not be sufficient as some routers tend to
> ignore tcp keep-alives and just kill idle connections.
>
> As said before, this will ultimately be fixed by adding keep-alive to the
> app layer on CASSANDRA-11841. If tuning tcp keep-alives does not help, one
> extreme approach would be to backport this to 2.1 (unless some experienced
> operator out there has a more creative approach).
>
> @eevans, I'm not sure he is using a mixed version cluster, it seem he
> finished the upgrade from 2.1.13 to 2.1.14 before performing the rebuild.
>
> 2016-05-27 11:39 GMT-03:00 Eric Evans :
>
>> From the various stacktraces in this thread, it's obvious you are
>> mixing versions 2.1.13 and 2.1.14.  Topology changes like this aren't
>> supported with mixed Cassandra versions.  Sometimes it will work,
>> sometimes it won't (and it will definitely not work in this instance).
>>
>> You should either upgrade your 2.1.13 nodes to 2.1.14 first, or add
>> the new nodes using 2.1.13, and upgrade after.
>>
>> On Fri, May 27, 2016 at 8:41 AM, George Sigletos 
>> wrote:
>>
>>  ERROR [STREAM-IN-/192.168.1.141] 2016-05-26 09:08:05,027
>>  StreamSession.java:505 - [Stream
>> #74c57bc0-231a-11e6-a698-1b05ac77baf9]
>>  Streaming error occurred
>>  java.lang.RuntimeException: Outgoing stream handler has been closed
>>  at
>> 
>> org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138)
>>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>>  at
>> 
>> org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:568)
>>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>>  at
>> 
>> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:457)
>>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>>  at
>> 
>> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:263)
>>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>>  at java.lang.Thread.run(Unknown Source) [na:1.7.0_79]
>> 
>>  And this is from the source node:
>> 
>>  ERROR [STREAM-OUT-/172.31.22.104] 2016-05-26 11:08:05,097
>>  StreamSession.java:505 - [Stream
>> #74c57bc0-231a-11e6-a698-1b05ac77baf9]
>>  Streaming error occurred
>>  java.io.IOException: Broken pipe
>>  at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
>>  ~[na:1.7.0_79]
>>  at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown
>> Source)
>>  ~[na:1.7.0_79]
>>  at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source)
>>  ~[na:1.7.0_79]
>>  at
>> 
>> org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:84)
>>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>>  at
>> 
>> org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:88)
>>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>>  at
>> 
>> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:49)
>>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>>  at
>> 
>> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:41)
>>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>>  at
>> 
>> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:45)
>>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>>  at
>> 
>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:358)
>>  [apache-cassandra-2.1.14.jar:2.1.14]
>>  at
>> 
>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:330)
>>  [apache-cassandra-2.1.14.jar:2.1.14]
>>
>>
>> >>> ERROR 

Re: Internal Handling of Map Updates

[Eric Stevens]

If you aren't removing elements from the map, you should instead be able to
use an UPDATE statement and append the map. It will have the same effect as
overwriting it, because all the new keys will take precedence over the
existing keys. But it'll happen without generating a tombstone first.

If you do have to remove elements from the collection during this process,
you are either facing tombstones or having to surgically figure out which
elements ought to be removed (which also involves tombstones, though at
least not range tombstones, so a bit cheaper).

On Fri, May 27, 2016, 5:39 AM Matthias Niehoff <
matthias.nieh...@codecentric.de> wrote:

> We are processing events in Spark and store the resulting entries
> (containing a map) in Cassandra. The results can be new (no entry for this
> key in Cassandra) or an Update (there is already an entry with this key in
> Cassandra). We use the spark-cassandra-connector to store the data in
> Cassandra.
>
> The connector will always do an insert of the data and will rely on the
> upsert capabilities of cassandra. So every time an event is updated the
> complete map is replaced with all the problems of tombstones.
> Seems like we have to implement our own persist logic in which we check if
> an element already exists and if yes update the map manually. that would
> require a read before write which would be nasty. Another option would be
> not to use a collection but (clustering) columns. Do you have another idea
> of doing this?
>
> (the conclusion of this whole thing for me would be: use upsert, but do
> specific updates on collections as an upsert might replace the whole
> collection and generate thumbstones)
>
> 2016-05-25 17:37 GMT+02:00 Tyler Hobbs :
>
>> If you replace an entire collection, whether it's a map, set, or list, a
>> range tombstone will be inserted followed by the new collection.  If you
>> only update a single element, no tombstones are generated.
>>
>> On Wed, May 25, 2016 at 9:48 AM, Matthias Niehoff <
>> matthias.nieh...@codecentric.de> wrote:
>>
>>> Hi,
>>>
>>> we have a table with a Map Field. We do not delete anything in this
>>> table, but to updates on the values including the Map Field (most of the
>>> time a new value for an existing key, Rarely adding new keys). We now
>>> encounter a huge amount of thumbstones for this Table.
>>>
>>> We used sstable2json to take a look into the sstables:
>>>
>>>
>>> {"key": "Betty_StoreCatalogLines:7",
>>>
>>>  "cells": [["276-1-6MPQ0RI-276110031802001001:","",1463820040628001],
>>>
>>>["276-1-6MPQ0RI-276110031802001001:last_modified","2016-05-21 
>>> 08:40Z",1463820040628001],
>>>
>>>
>>> ["276-1-6MPQ0RI-276110031802001001:last_modified_by_source:_","276-1-6MPQ0RI-276110031802001001:last_modified_by_source:!",1463040069753999,"t",1463040069],
>>>
>>>
>>> ["276-1-6MPQ0RI-276110031802001001:last_modified_by_source:_","276-1-6MPQ0RI-276110031802001001:last_modified_by_source:!",1463120708590002,"t",1463120708],
>>>
>>>
>>> ["276-1-6MPQ0RI-276110031802001001:last_modified_by_source:_","276-1-6MPQ0RI-276110031802001001:last_modified_by_source:!",1463145700735007,"t",1463145700],
>>>
>>>
>>> ["276-1-6MPQ0RI-276110031802001001:last_modified_by_source:_","276-1-6MPQ0RI-276110031802001001:last_modified_by_source:!",1463157430862000,"t",1463157430],
>>>
>>>
>>> [„276-1-6MPQ0RI-276110031802001001:last_modified_by_source:_“,“276-1-6MPQ0RI-276110031802001001:last_modified_by_source:!“,1463164595291002,"t",1463164595],
>>>
>>> . . .
>>>
>>>   
>>> ["276-1-6MPQ0RI-276110031802001001:last_modified_by_source:_","276-1-6MPQ0RI-276110031802001001:last_modified_by_source:!",1463820040628000,"t",1463820040],
>>>
>>>
>>> ["276-1-6MPQ0RI-276110031802001001:last_modified_by_source:62657474795f73746f72655f636174616c6f675f6c696e6573","0154d265c6b0",1463820040628001],
>>>
>>>
>>> [„276-1-6MPQ0RI-276110031802001001:payload“,"{\"payload\":{\"Article 
>>> Id\":\"276110031802001001\",\"Row Id\":\"1-6MPQ0RI\",\"Article 
>>> #\":\"31802001001\",\"Quote Item Id\":\"1-6MPWPVC\",\"Country 
>>> Code\":\"276\"}}",1463820040628001]
>>>
>>>
>>>
>>> Looking at the SStables it seem like every update of a value in a Map
>>> breaks down to a delete and insert in the corresponding SSTable (see all
>>> the thumbstone flags „t“ in the extract of sstable2json above).
>>>
>>> We are using Cassandra 2.2.5.
>>>
>>> Can you confirm this behavior?
>>>
>>> Thanks!
>>> --
>>> Matthias Niehoff | IT-Consultant | Agile Software Factory  | Consulting
>>> codecentric AG | Zeppelinstr 2 | 76185 Karlsruhe | Deutschland
>>> tel: +49 (0) 721.9595-681 | fax: +49 (0) 721.9595-666 | mobil: +49 (0)
>>> 172.1702676
>>> www.codecentric.de | blog.codecentric.de | www.meettheexperts.de |
>>> www.more4fi.de
>>>
>>> Sitz der Gesellschaft: Solingen | HRB 25917| Amtsgericht Wuppertal
>>> Vorstand: Michael Hochgürtel . Mirko Novakovic . Rainer Vehns
>>> 

Re: cqlsh problem

[Patrick McFadin]

Can you do a netstat -lepunt and show the output? If Cassandra is running
you aren't trying to connect to the ip/port it's bound to.

Patrick

On Monday, May 23, 2016, joseph gao  wrote:

> I used to think it's firewall/network issues too. So I make ufw to be
> inactive. I really don't what's the reason.
>
> 2016-05-09 19:01 GMT+08:00 kurt Greaves  >:
>
>> Don't be fooled, despite saying tcp6 and :::*, it still listens on IPv4.
>> As far as I'm aware this happens on all 2.1 Cassandra nodes, and may just
>> be an oddity of netstat. It would be unrelated to your connection timeout
>> issues, that's most likely related to firewall/network issues.
>>
>> On 9 May 2016 at 09:59, joseph gao > > wrote:
>>
>>> It doesn't work ,still using ipv6 [image: 内嵌图片 1]
>>>
>>> And I already set [image: 内嵌图片 2]
>>>
>>> Now I'm using 4.1.1 using 9160 port instead of 5.x.x。
>>>
>>> Hopefully this could be resolved, Thanks!
>>>
>>> 2016-03-30 22:13 GMT+08:00 Alain RODRIGUEZ >> >:
>>>
 Hi Joseph,

 why cassandra using tcp6 for 9042 port like :
> tcp6   0  0 0.0.0.0:9042:::*
>  LISTEN
>

 if I remember correctly, in 2.1 and higher, cqlsh uses native
 transport, port 9042  (instead of thrift port 9160) and your clients (if
 any) are also probably using native transport (port 9042). So yes, this
 could be an issue indeed.

 You should have something like:

 tcp0  0  1.2.3.4:9042   :::*LISTEN

 You are using IPv6 and no rpc address. Try setting it to the listen
 address and using IPv4.

 C*heers,

 ---

 Alain Rodriguez - al...@thelastpickle.com
 

 France

 The Last Pickle - Apache Cassandra Consulting

 http://www.thelastpickle.com

 2016-03-30 6:09 GMT+02:00 joseph gao >:

> why cassandra using tcp6 for 9042 port like :
> tcp6   0  0 0.0.0.0:9042:::*
>  LISTEN
> would this be the problem
>
> 2016-03-30 11:34 GMT+08:00 joseph gao  >:
>
>> still have not fixed it . cqlsh: error: no such option:
>> --connect-timeout
>> cqlsh version 5.0.1
>>
>>
>>
>> 2016-03-25 16:46 GMT+08:00 Alain RODRIGUEZ > >:
>>
>>> Hi Joseph.
>>>
>>> As I can't reproduce here, I believe you are having network issue of
>>> some kind.
>>>
>>> MacBook-Pro:~ alain$ cqlsh --version
>>> cqlsh 5.0.1
>>> MacBook-Pro:~ alain$ echo 'DESCRIBE KEYSPACES;' | cqlsh
>>> --connect-timeout=5 --request-timeout=10
>>> system_traces  system
>>> MacBook-Pro:~ alain$
>>>
>>> It's been a few days, did you manage to fix it ?
>>>
>>> C*heers,
>>> ---
>>> Alain Rodriguez - al...@thelastpickle.com
>>> 
>>> France
>>>
>>> The Last Pickle - Apache Cassandra Consulting
>>> http://www.thelastpickle.com
>>>
>>> 2016-03-21 9:59 GMT+01:00 joseph gao >> >:
>>>
 cqlsh version 5.0.1. nodetool tpstats looks good, log looks good.
 And I used specified port 9042. And it immediately returns fail (less 
 than
 3 seconds). By the way where should I use '--connect-timeout', cqlsh 
 seems
 don't have such parameters.

 2016-03-18 17:29 GMT+08:00 Alain RODRIGUEZ >:

> Is the node fully healthy or rejecting some requests ?
>
> What are the outputs for "grep -i "ERROR"
> /var/log/cassandra/system.log" and "nodetool tpstats"?
>
> Any error? Any pending / blocked or dropped messages?
>
> Also did you try using distinct ports (9160 for thrift, 9042 for
> native) - out of curiosity, not sure this will help.
>
> What is your version of cqlsh "cqlsh --version" ?
>
> doesn't work most times. But some time it just work fine
>>
>
> Do you fill like this is due to a timeout (query being too big,
> cluster being to busy)? Try setting this higher:
>
> --connect-timeout=CONNECT_TIMEOUT
>
> 

Re: Error while rebuilding a node: Stream failed

[Paulo Motta]

I'm afraid raising streaming_socket_timeout_in_ms won't help much in this
case because the incoming connection on the source node is timing out on
the network layer, and streaming_socket_timeout_in_ms controls the socket
timeout in the app layer and throws SocketTimeoutException (not
java.io.IOException:
Connection timed out). So you should probably use more aggressive tcp
keep-alive settings (net.ipv4.tcp_keepalive_*) on both hosts, did you try
tuning that? Even that might not be sufficient as some routers tend to
ignore tcp keep-alives and just kill idle connections.

As said before, this will ultimately be fixed by adding keep-alive to the
app layer on CASSANDRA-11841. If tuning tcp keep-alives does not help, one
extreme approach would be to backport this to 2.1 (unless some experienced
operator out there has a more creative approach).

@eevans, I'm not sure he is using a mixed version cluster, it seem he
finished the upgrade from 2.1.13 to 2.1.14 before performing the rebuild.

2016-05-27 11:39 GMT-03:00 Eric Evans :

> From the various stacktraces in this thread, it's obvious you are
> mixing versions 2.1.13 and 2.1.14.  Topology changes like this aren't
> supported with mixed Cassandra versions.  Sometimes it will work,
> sometimes it won't (and it will definitely not work in this instance).
>
> You should either upgrade your 2.1.13 nodes to 2.1.14 first, or add
> the new nodes using 2.1.13, and upgrade after.
>
> On Fri, May 27, 2016 at 8:41 AM, George Sigletos 
> wrote:
>
>  ERROR [STREAM-IN-/192.168.1.141] 2016-05-26 09:08:05,027
>  StreamSession.java:505 - [Stream
> #74c57bc0-231a-11e6-a698-1b05ac77baf9]
>  Streaming error occurred
>  java.lang.RuntimeException: Outgoing stream handler has been closed
>  at
> 
> org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138)
>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>  at
> 
> org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:568)
>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>  at
> 
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:457)
>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>  at
> 
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:263)
>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>  at java.lang.Thread.run(Unknown Source) [na:1.7.0_79]
> 
>  And this is from the source node:
> 
>  ERROR [STREAM-OUT-/172.31.22.104] 2016-05-26 11:08:05,097
>  StreamSession.java:505 - [Stream
> #74c57bc0-231a-11e6-a698-1b05ac77baf9]
>  Streaming error occurred
>  java.io.IOException: Broken pipe
>  at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
>  ~[na:1.7.0_79]
>  at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown
> Source)
>  ~[na:1.7.0_79]
>  at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source)
>  ~[na:1.7.0_79]
>  at
> 
> org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:84)
>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>  at
> 
> org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:88)
>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>  at
> 
> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:49)
>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>  at
> 
> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:41)
>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>  at
> 
> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:45)
>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>  at
> 
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:358)
>  [apache-cassandra-2.1.14.jar:2.1.14]
>  at
> 
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:330)
>  [apache-cassandra-2.1.14.jar:2.1.14]
>
>
> >>> ERROR [STREAM-IN-/192.168.1.140] 2016-05-24 22:44:57,704
> >>> StreamSession.java:620 - [Stream
> #2c290460-20d4-11e6-930f-1b05ac77baf9]
> >>> Remote peer 192.168.1.140 failed stream session.
> >>> ERROR [STREAM-OUT-/192.168.1.140] 2016-05-24 22:44:57,705
> >>> StreamSession.java:505 - [Stream
> #2c290460-20d4-11e6-930f-1b05ac77baf9]
> >>> Streaming error occurred
> >>> java.io.IOException: Connection timed out
> >>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
> >>> ~[na:1.7.0_79]
> >>> at sun.nio.ch.SocketDispatcher.write(Unknown Source)
> >>> ~[na:1.7.0_79]
> >>>   

Re: Error while rebuilding a node: Stream failed

[Sebastian Estevez]

Check ifconfig for dripped tpc messages. Let's rule out your network.

all the best,

Sebastián
On May 27, 2016 10:45 AM, "George Sigletos"  wrote:

> Hello,
>
> No there is no version mix. The first stack traces were indeed from
> 2.1.13. Then I upgraded all nodes to 2.1.14. Still getting the same errors
>
>
> On Fri, May 27, 2016 at 4:39 PM, Eric Evans 
> wrote:
>
>> From the various stacktraces in this thread, it's obvious you are
>> mixing versions 2.1.13 and 2.1.14.  Topology changes like this aren't
>> supported with mixed Cassandra versions.  Sometimes it will work,
>> sometimes it won't (and it will definitely not work in this instance).
>>
>> You should either upgrade your 2.1.13 nodes to 2.1.14 first, or add
>> the new nodes using 2.1.13, and upgrade after.
>>
>> On Fri, May 27, 2016 at 8:41 AM, George Sigletos 
>> wrote:
>>
>>  ERROR [STREAM-IN-/192.168.1.141] 2016-05-26 09:08:05,027
>>  StreamSession.java:505 - [Stream
>> #74c57bc0-231a-11e6-a698-1b05ac77baf9]
>>  Streaming error occurred
>>  java.lang.RuntimeException: Outgoing stream handler has been closed
>>  at
>> 
>> org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138)
>>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>>  at
>> 
>> org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:568)
>>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>>  at
>> 
>> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:457)
>>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>>  at
>> 
>> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:263)
>>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>>  at java.lang.Thread.run(Unknown Source) [na:1.7.0_79]
>> 
>>  And this is from the source node:
>> 
>>  ERROR [STREAM-OUT-/172.31.22.104] 2016-05-26 11:08:05,097
>>  StreamSession.java:505 - [Stream
>> #74c57bc0-231a-11e6-a698-1b05ac77baf9]
>>  Streaming error occurred
>>  java.io.IOException: Broken pipe
>>  at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
>>  ~[na:1.7.0_79]
>>  at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown
>> Source)
>>  ~[na:1.7.0_79]
>>  at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source)
>>  ~[na:1.7.0_79]
>>  at
>> 
>> org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:84)
>>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>>  at
>> 
>> org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:88)
>>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>>  at
>> 
>> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:49)
>>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>>  at
>> 
>> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:41)
>>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>>  at
>> 
>> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:45)
>>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>>  at
>> 
>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:358)
>>  [apache-cassandra-2.1.14.jar:2.1.14]
>>  at
>> 
>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:330)
>>  [apache-cassandra-2.1.14.jar:2.1.14]
>>
>>
>> >>> ERROR [STREAM-IN-/192.168.1.140] 2016-05-24 22:44:57,704
>> >>> StreamSession.java:620 - [Stream
>> #2c290460-20d4-11e6-930f-1b05ac77baf9]
>> >>> Remote peer 192.168.1.140 failed stream session.
>> >>> ERROR [STREAM-OUT-/192.168.1.140] 2016-05-24 22:44:57,705
>> >>> StreamSession.java:505 - [Stream
>> #2c290460-20d4-11e6-930f-1b05ac77baf9]
>> >>> Streaming error occurred
>> >>> java.io.IOException: Connection timed out
>> >>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>> >>> ~[na:1.7.0_79]
>> >>> at sun.nio.ch.SocketDispatcher.write(Unknown Source)
>> >>> ~[na:1.7.0_79]
>> >>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown
>> >>> Source) ~[na:1.7.0_79]
>> >>> at sun.nio.ch.IOUtil.write(Unknown Source)
>> ~[na:1.7.0_79]
>> >>> at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
>> >>> ~[na:1.7.0_79]
>> >>> at
>> >>>
>> org.apache.cassandra.io.util.DataOutputStreamAndChannel.write(DataOutputStreamAndChannel.java:48)
>> >>> ~[apache-cassandra-2.1.13.jar:2.1.13]
>> >>> at
>> >>>
>> 

Re: Error while rebuilding a node: Stream failed

[George Sigletos]

Hello,

No there is no version mix. The first stack traces were indeed from 2.1.13.
Then I upgraded all nodes to 2.1.14. Still getting the same errors


On Fri, May 27, 2016 at 4:39 PM, Eric Evans 
wrote:

> From the various stacktraces in this thread, it's obvious you are
> mixing versions 2.1.13 and 2.1.14.  Topology changes like this aren't
> supported with mixed Cassandra versions.  Sometimes it will work,
> sometimes it won't (and it will definitely not work in this instance).
>
> You should either upgrade your 2.1.13 nodes to 2.1.14 first, or add
> the new nodes using 2.1.13, and upgrade after.
>
> On Fri, May 27, 2016 at 8:41 AM, George Sigletos 
> wrote:
>
>  ERROR [STREAM-IN-/192.168.1.141] 2016-05-26 09:08:05,027
>  StreamSession.java:505 - [Stream
> #74c57bc0-231a-11e6-a698-1b05ac77baf9]
>  Streaming error occurred
>  java.lang.RuntimeException: Outgoing stream handler has been closed
>  at
> 
> org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138)
>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>  at
> 
> org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:568)
>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>  at
> 
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:457)
>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>  at
> 
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:263)
>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>  at java.lang.Thread.run(Unknown Source) [na:1.7.0_79]
> 
>  And this is from the source node:
> 
>  ERROR [STREAM-OUT-/172.31.22.104] 2016-05-26 11:08:05,097
>  StreamSession.java:505 - [Stream
> #74c57bc0-231a-11e6-a698-1b05ac77baf9]
>  Streaming error occurred
>  java.io.IOException: Broken pipe
>  at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
>  ~[na:1.7.0_79]
>  at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown
> Source)
>  ~[na:1.7.0_79]
>  at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source)
>  ~[na:1.7.0_79]
>  at
> 
> org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:84)
>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>  at
> 
> org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:88)
>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>  at
> 
> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:49)
>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>  at
> 
> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:41)
>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>  at
> 
> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:45)
>  ~[apache-cassandra-2.1.14.jar:2.1.14]
>  at
> 
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:358)
>  [apache-cassandra-2.1.14.jar:2.1.14]
>  at
> 
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:330)
>  [apache-cassandra-2.1.14.jar:2.1.14]
>
>
> >>> ERROR [STREAM-IN-/192.168.1.140] 2016-05-24 22:44:57,704
> >>> StreamSession.java:620 - [Stream
> #2c290460-20d4-11e6-930f-1b05ac77baf9]
> >>> Remote peer 192.168.1.140 failed stream session.
> >>> ERROR [STREAM-OUT-/192.168.1.140] 2016-05-24 22:44:57,705
> >>> StreamSession.java:505 - [Stream
> #2c290460-20d4-11e6-930f-1b05ac77baf9]
> >>> Streaming error occurred
> >>> java.io.IOException: Connection timed out
> >>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
> >>> ~[na:1.7.0_79]
> >>> at sun.nio.ch.SocketDispatcher.write(Unknown Source)
> >>> ~[na:1.7.0_79]
> >>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown
> >>> Source) ~[na:1.7.0_79]
> >>> at sun.nio.ch.IOUtil.write(Unknown Source)
> ~[na:1.7.0_79]
> >>> at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
> >>> ~[na:1.7.0_79]
> >>> at
> >>>
> org.apache.cassandra.io.util.DataOutputStreamAndChannel.write(DataOutputStreamAndChannel.java:48)
> >>> ~[apache-cassandra-2.1.13.jar:2.1.13]
> >>> at
> >>>
> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44)
> >>> ~[apache-cassandra-2.1.13.jar:2.1.13]
> >>> at
> >>>
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:351)
> >>> 

Re: Error while rebuilding a node: Stream failed

[Eric Evans]

>From the various stacktraces in this thread, it's obvious you are
mixing versions 2.1.13 and 2.1.14.  Topology changes like this aren't
supported with mixed Cassandra versions.  Sometimes it will work,
sometimes it won't (and it will definitely not work in this instance).

You should either upgrade your 2.1.13 nodes to 2.1.14 first, or add
the new nodes using 2.1.13, and upgrade after.

On Fri, May 27, 2016 at 8:41 AM, George Sigletos  wrote:

 ERROR [STREAM-IN-/192.168.1.141] 2016-05-26 09:08:05,027
 StreamSession.java:505 - [Stream #74c57bc0-231a-11e6-a698-1b05ac77baf9]
 Streaming error occurred
 java.lang.RuntimeException: Outgoing stream handler has been closed
 at
 org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138)
 ~[apache-cassandra-2.1.14.jar:2.1.14]
 at
 org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:568)
 ~[apache-cassandra-2.1.14.jar:2.1.14]
 at
 org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:457)
 ~[apache-cassandra-2.1.14.jar:2.1.14]
 at
 org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:263)
 ~[apache-cassandra-2.1.14.jar:2.1.14]
 at java.lang.Thread.run(Unknown Source) [na:1.7.0_79]

 And this is from the source node:

 ERROR [STREAM-OUT-/172.31.22.104] 2016-05-26 11:08:05,097
 StreamSession.java:505 - [Stream #74c57bc0-231a-11e6-a698-1b05ac77baf9]
 Streaming error occurred
 java.io.IOException: Broken pipe
 at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
 ~[na:1.7.0_79]
 at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown Source)
 ~[na:1.7.0_79]
 at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source)
 ~[na:1.7.0_79]
 at
 org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:84)
 ~[apache-cassandra-2.1.14.jar:2.1.14]
 at
 org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:88)
 ~[apache-cassandra-2.1.14.jar:2.1.14]
 at
 org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:49)
 ~[apache-cassandra-2.1.14.jar:2.1.14]
 at
 org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:41)
 ~[apache-cassandra-2.1.14.jar:2.1.14]
 at
 org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:45)
 ~[apache-cassandra-2.1.14.jar:2.1.14]
 at
 org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:358)
 [apache-cassandra-2.1.14.jar:2.1.14]
 at
 org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:330)
 [apache-cassandra-2.1.14.jar:2.1.14]


>>> ERROR [STREAM-IN-/192.168.1.140] 2016-05-24 22:44:57,704
>>> StreamSession.java:620 - [Stream 
>>> #2c290460-20d4-11e6-930f-1b05ac77baf9]
>>> Remote peer 192.168.1.140 failed stream session.
>>> ERROR [STREAM-OUT-/192.168.1.140] 2016-05-24 22:44:57,705
>>> StreamSession.java:505 - [Stream 
>>> #2c290460-20d4-11e6-930f-1b05ac77baf9]
>>> Streaming error occurred
>>> java.io.IOException: Connection timed out
>>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>> ~[na:1.7.0_79]
>>> at sun.nio.ch.SocketDispatcher.write(Unknown Source)
>>> ~[na:1.7.0_79]
>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown
>>> Source) ~[na:1.7.0_79]
>>> at sun.nio.ch.IOUtil.write(Unknown Source) ~[na:1.7.0_79]
>>> at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
>>> ~[na:1.7.0_79]
>>> at
>>> org.apache.cassandra.io.util.DataOutputStreamAndChannel.write(DataOutputStreamAndChannel.java:48)
>>> ~[apache-cassandra-2.1.13.jar:2.1.13]
>>> at
>>> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44)
>>> ~[apache-cassandra-2.1.13.jar:2.1.13]
>>> at
>>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:351)
>>> [apache-cassandra-2.1.13.jar:2.1.13]
>>> at
>>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:323)
>>> [apache-cassandra-2.1.13.jar:2.1.13]
>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_79]
>>> INFO  [STREAM-IN-/192.168.1.140] 2016-05-24 22:44:58,625
>>> StreamResultFuture.java:180 - [Stream 
>>> 

Re: Error while rebuilding a node: Stream failed

[George Sigletos]

Still failing.

Should I maybe set a higher value for streaming_socket_timeout_in_ms? Maybe
2-3 days?

Source: node
ERROR [STREAM-OUT-/54.172.235.227] 2016-05-27 14:30:34,401
StreamSession.java:505 - [Stream #45017970-234c-11e6-9452-1b05ac77baf9]
Streaming error occurred
java.lang.AssertionError: Memory was freed
at
org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:97)
~[apache-cassandra-2.1.14.jar:2.1.14]
at org.apache.cassandra.io.util.Memory.getLong(Memory.java:249)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.io.compress.CompressionMetadata.getTotalSizeForSections(CompressionMetadata.java:247)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.streaming.messages.FileMessageHeader.size(FileMessageHeader.java:112)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.streaming.StreamSession.fileSent(StreamSession.java:546)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:50)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:41)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:45)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:358)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:338)
~[apache-cassandra-2.1.14.jar:2.1.14]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_79]

(Previous error on source node 6 hours before)
ERROR [STREAM-IN-/54.172.235.227] 2016-05-27 08:31:45,868
StreamSession.java:505 - [Stream #45017970-234c-11e6-9452-1b05ac77baf9]
Streaming error occurred
java.io.IOException: Connection timed out
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.7.0_79]
at sun.nio.ch.SocketDispatcher.read(Unknown Source) ~[na:1.7.0_79]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
~[na:1.7.0_79]
at sun.nio.ch.IOUtil.read(Unknown Source) ~[na:1.7.0_79]
at sun.nio.ch.SocketChannelImpl.read(Unknown Source) ~[na:1.7.0_79]
at sun.nio.ch.SocketAdaptor$SocketInputStream.read(Unknown Source)
~[na:1.7.0_79]
at sun.nio.ch.ChannelInputStream.read(Unknown Source) ~[na:1.7.0_79]
at java.nio.channels.Channels$ReadableByteChannelImpl.read(Unknown
Source) ~[na:1.7.0_79]
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:51)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:257)
~[apache-cassandra-2.1.14.jar:2.1.14]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_79]


Destination node (amazon):
ERROR [STREAM-OUT-/192.168.3.4] 2016-05-27 12:30:37,116
StreamSession.java:505 - [Stream #45017970-234c-11e6-9452-1b05ac77baf9]
Streaming error occurred
java.io.IOException: Connection timed out
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
~[na:1.7.0_79]
at sun.nio.ch.SocketDispatcher.write(Unknown Source) ~[na:1.7.0_79]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
~[na:1.7.0_79]
at sun.nio.ch.IOUtil.write(Unknown Source) ~[na:1.7.0_79]
at sun.nio.ch.SocketChannelImpl.write(Unknown Source) ~[na:1.7.0_79]
at
org.apache.cassandra.io.util.DataOutputStreamAndChannel.write(DataOutputStreamAndChannel.java:48)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:358)
[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:330)
[apache-cassandra-2.1.14.jar:2.1.14]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_79]
INFO  [STREAM-IN-/192.168.3.4] 2016-05-27 12:30:37,593
StreamResultFuture.java:180 - [Stream
#45017970-234c-11e6-9452-1b05ac77baf9] Session with /192.168.3.4 is complete
ERROR [STREAM-OUT-/192.168.3.4] 2016-05-27 12:30:37,594
StreamSession.java:505 - [Stream #45017970-234c-11e6-9452-1b05ac77baf9]
Streaming error occurred
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
~[na:1.7.0_79]
at sun.nio.ch.SocketDispatcher.write(Unknown Source) ~[na:1.7.0_79]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
~[na:1.7.0_79]
at sun.nio.ch.IOUtil.write(Unknown Source) ~[na:1.7.0_79]
at sun.nio.ch.SocketChannelImpl.write(Unknown Source) 

Re: Out of memory issues

[Kai Wang]

Paolo,

try a few things in cassandra-env.sh
1. HEAP_NEWSIZE="2G". "The 100mb/core commentary in cassandra-env.sh for
setting HEAP_NEWSIZE is *wrong*" (
https://tobert.github.io/pages/als-cassandra-21-tuning-guide.html)
2. MaxTenuringThreshold=8
3. enable GC logging (under "# GC logging options -- uncomment to enable"
section) to compare GC behaviors on good and bad nodes.


On Fri, May 27, 2016 at 5:36 AM, Paolo Crosato <
paolo.cros...@targaubiest.com> wrote:

> Hi,
>
> thanks for the answer. There were no large insertions and the saved_caches
> dir had a resonable size. I tried to delete the cashes and set
> key_cache_size_in_mb to zero, but it didn't help.
> Today our virtual hardware provided raised cpus to 4, memory to 32GB and
> doubled the disk size, and the nodes are stable again. So it was probably
> an issue of severe lack of resources.
> About HEAP_NEWSIZE, your suggestion is quite intriguing. I thought it was
> better to set it 100mb*#cores, so in my case I set it to 200 and now I
> should set it to 400. Do larger values help without being harmful?
>
> Regards,
>
> Paolo
>
>
> Il 27/05/2016 03:05, Mike Yeap ha scritto:
>
> Hi Paolo,
>
> a) was there any large insertion done?
> b) are the a lot of files in the saved_caches directory?
> c) would you consider to increase the HEAP_NEWSIZE to, say, 1200M?
>
>
> Regards,
> Mike Yeap
>
> On Fri, May 27, 2016 at 12:39 AM, Paolo Crosato <
> paolo.cros...@targaubiest.com> wrote:
>
>> Hi,
>>
>> we are running a cluster of 4 nodes, each one has the same sizing: 2
>> cores, 16G ram and 1TB of disk space.
>>
>> On every node we are running cassandra 2.0.17, oracle java version
>> "1.7.0_45", centos 6 with this kernel version 2.6.32-431.17.1.el6.x86_64
>>
>> Two nodes are running just fine, the other two have started to go OOM at
>> every start.
>>
>> This is the error we get:
>>
>> INFO [ScheduledTasks:1] 2016-05-26 18:15:58,460 StatusLogger.java (line
>> 70) ReadRepairStage   0 0116
>> 0 0
>>  INFO [ScheduledTasks:1] 2016-05-26 18:15:58,462 StatusLogger.java (line
>> 70) MutationStage31  1369  20526
>> 0 0
>>  INFO [ScheduledTasks:1] 2016-05-26 18:15:58,590 StatusLogger.java (line
>> 70) ReplicateOnWriteStage 0 0  0
>> 0 0
>>  INFO [ScheduledTasks:1] 2016-05-26 18:15:58,591 StatusLogger.java (line
>> 70) GossipStage   0 0335
>> 0 0
>>  INFO [ScheduledTasks:1] 2016-05-26 18:16:04,195 StatusLogger.java (line
>> 70) CacheCleanupExecutor  0 0  0
>> 0 0
>>  INFO [ScheduledTasks:1] 2016-05-26 18:16:06,526 StatusLogger.java (line
>> 70) MigrationStage0 0  0
>> 0 0
>>  INFO [ScheduledTasks:1] 2016-05-26 18:16:06,527 StatusLogger.java (line
>> 70) MemoryMeter   1 4 26
>> 0 0
>>  INFO [ScheduledTasks:1] 2016-05-26 18:16:06,527 StatusLogger.java (line
>> 70) ValidationExecutor0 0  0
>> 0 0
>> DEBUG [MessagingService-Outgoing-/10.255.235.19] 2016-05-26 18:16:06,518
>> OutboundTcpConnection.java (line 290) attempting to connect to /
>> 10.255.235.19
>>  INFO [GossipTasks:1] 2016-05-26 18:16:22,912 Gossiper.java (line 992)
>> InetAddress /10.255.235.28 is now DOWN
>>  INFO [ScheduledTasks:1] 2016-05-26 18:16:22,952 StatusLogger.java (line
>> 70) FlushWriter   1 5 47
>> 025
>>  INFO [ScheduledTasks:1] 2016-05-26 18:16:22,953 StatusLogger.java (line
>> 70) InternalResponseStage 0 0  0
>> 0 0
>> ERROR [ReadStage:27] 2016-05-26 18:16:29,250 CassandraDaemon.java (line
>> 258) Exception in thread Thread[ReadStage:27,5,main]
>> java.lang.OutOfMemoryError: Java heap space
>> at
>> org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:347)
>> at
>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:392)
>> at
>> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355)
>> at
>> org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:124)
>> at
>> org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:85)
>> at org.apache.cassandra.db.Column$1.computeNext(Column.java:75)
>> at org.apache.cassandra.db.Column$1.computeNext(Column.java:64)
>> at
>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>> at
>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>> at
>> com.google.common.collect.AbstractIterator.next(AbstractIterator.java:153)
>> at
>> 

Re: Internal Handling of Map Updates

[Matthias Niehoff]

We are processing events in Spark and store the resulting entries
(containing a map) in Cassandra. The results can be new (no entry for this
key in Cassandra) or an Update (there is already an entry with this key in
Cassandra). We use the spark-cassandra-connector to store the data in
Cassandra.

The connector will always do an insert of the data and will rely on the
upsert capabilities of cassandra. So every time an event is updated the
complete map is replaced with all the problems of tombstones.
Seems like we have to implement our own persist logic in which we check if
an element already exists and if yes update the map manually. that would
require a read before write which would be nasty. Another option would be
not to use a collection but (clustering) columns. Do you have another idea
of doing this?

(the conclusion of this whole thing for me would be: use upsert, but do
specific updates on collections as an upsert might replace the whole
collection and generate thumbstones)

2016-05-25 17:37 GMT+02:00 Tyler Hobbs :

> If you replace an entire collection, whether it's a map, set, or list, a
> range tombstone will be inserted followed by the new collection.  If you
> only update a single element, no tombstones are generated.
>
> On Wed, May 25, 2016 at 9:48 AM, Matthias Niehoff <
> matthias.nieh...@codecentric.de> wrote:
>
>> Hi,
>>
>> we have a table with a Map Field. We do not delete anything in this
>> table, but to updates on the values including the Map Field (most of the
>> time a new value for an existing key, Rarely adding new keys). We now
>> encounter a huge amount of thumbstones for this Table.
>>
>> We used sstable2json to take a look into the sstables:
>>
>>
>> {"key": "Betty_StoreCatalogLines:7",
>>
>>  "cells": [["276-1-6MPQ0RI-276110031802001001:","",1463820040628001],
>>
>>["276-1-6MPQ0RI-276110031802001001:last_modified","2016-05-21 
>> 08:40Z",1463820040628001],
>>
>>
>> ["276-1-6MPQ0RI-276110031802001001:last_modified_by_source:_","276-1-6MPQ0RI-276110031802001001:last_modified_by_source:!",1463040069753999,"t",1463040069],
>>
>>
>> ["276-1-6MPQ0RI-276110031802001001:last_modified_by_source:_","276-1-6MPQ0RI-276110031802001001:last_modified_by_source:!",1463120708590002,"t",1463120708],
>>
>>
>> ["276-1-6MPQ0RI-276110031802001001:last_modified_by_source:_","276-1-6MPQ0RI-276110031802001001:last_modified_by_source:!",1463145700735007,"t",1463145700],
>>
>>
>> ["276-1-6MPQ0RI-276110031802001001:last_modified_by_source:_","276-1-6MPQ0RI-276110031802001001:last_modified_by_source:!",1463157430862000,"t",1463157430],
>>
>>
>> [„276-1-6MPQ0RI-276110031802001001:last_modified_by_source:_“,“276-1-6MPQ0RI-276110031802001001:last_modified_by_source:!“,1463164595291002,"t",1463164595],
>>
>> . . .
>>
>>   
>> ["276-1-6MPQ0RI-276110031802001001:last_modified_by_source:_","276-1-6MPQ0RI-276110031802001001:last_modified_by_source:!",1463820040628000,"t",1463820040],
>>
>>
>> ["276-1-6MPQ0RI-276110031802001001:last_modified_by_source:62657474795f73746f72655f636174616c6f675f6c696e6573","0154d265c6b0",1463820040628001],
>>
>>
>> [„276-1-6MPQ0RI-276110031802001001:payload“,"{\"payload\":{\"Article 
>> Id\":\"276110031802001001\",\"Row Id\":\"1-6MPQ0RI\",\"Article 
>> #\":\"31802001001\",\"Quote Item Id\":\"1-6MPWPVC\",\"Country 
>> Code\":\"276\"}}",1463820040628001]
>>
>>
>>
>> Looking at the SStables it seem like every update of a value in a Map
>> breaks down to a delete and insert in the corresponding SSTable (see all
>> the thumbstone flags „t“ in the extract of sstable2json above).
>>
>> We are using Cassandra 2.2.5.
>>
>> Can you confirm this behavior?
>>
>> Thanks!
>> --
>> Matthias Niehoff | IT-Consultant | Agile Software Factory  | Consulting
>> codecentric AG | Zeppelinstr 2 | 76185 Karlsruhe | Deutschland
>> tel: +49 (0) 721.9595-681 | fax: +49 (0) 721.9595-666 | mobil: +49 (0)
>> 172.1702676
>> www.codecentric.de | blog.codecentric.de | www.meettheexperts.de |
>> www.more4fi.de
>>
>> Sitz der Gesellschaft: Solingen | HRB 25917| Amtsgericht Wuppertal
>> Vorstand: Michael Hochgürtel . Mirko Novakovic . Rainer Vehns
>> Aufsichtsrat: Patric Fedlmeier (Vorsitzender) . Klaus Jäger . Jürgen
>> Schütz
>>
>> Diese E-Mail einschließlich evtl. beigefügter Dateien enthält
>> vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht
>> der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben,
>> informieren Sie bitte sofort den Absender und löschen Sie diese E-Mail und
>> evtl. beigefügter Dateien umgehend. Das unerlaubte Kopieren, Nutzen oder
>> Öffnen evtl. beigefügter Dateien sowie die unbefugte Weitergabe dieser
>> E-Mail ist nicht gestattet
>>
>
>
>
> --
> Tyler Hobbs
> DataStax 
>



-- 
Matthias Niehoff | IT-Consultant | Agile Software Factory  | Consulting
codecentric AG | Zeppelinstr 2 | 76185 Karlsruhe | Deutschland
tel: 

Re: Out of memory issues

[Paolo Crosato]

Hi,

thanks for the answer. There were no large insertions and the 
saved_caches dir had a resonable size. I tried to delete the cashes and 
set key_cache_size_in_mb to zero, but it didn't help.
Today our virtual hardware provided raised cpus to 4, memory to 32GB and 
doubled the disk size, and the nodes are stable again. So it was 
probably an issue of severe lack of resources.
About HEAP_NEWSIZE, your suggestion is quite intriguing. I thought it 
was better to set it 100mb*#cores, so in my case I set it to 200 and now 
I should set it to 400. Do larger values help without being harmful?


Regards,

Paolo

Il 27/05/2016 03:05, Mike Yeap ha scritto:

Hi Paolo,

a) was there any large insertion done?
b) are the a lot of files in the saved_caches directory?
c) would you consider to increase the HEAP_NEWSIZE to, say, 1200M?


Regards,
Mike Yeap

On Fri, May 27, 2016 at 12:39 AM, Paolo Crosato 
> 
wrote:


Hi,

we are running a cluster of 4 nodes, each one has the same sizing:
2 cores, 16G ram and 1TB of disk space.

On every node we are running cassandra 2.0.17, oracle java version
"1.7.0_45", centos 6 with this kernel version
2.6.32-431.17.1.el6.x86_64

Two nodes are running just fine, the other two have started to go
OOM at every start.

This is the error we get:

INFO [ScheduledTasks:1] 2016-05-26 18:15:58,460 StatusLogger.java
(line 70) ReadRepairStage   0 0
116 0 0
 INFO [ScheduledTasks:1] 2016-05-26 18:15:58,462 StatusLogger.java
(line 70) MutationStage31  1369
20526 0 0
 INFO [ScheduledTasks:1] 2016-05-26 18:15:58,590 StatusLogger.java
(line 70) ReplicateOnWriteStage 0 0 0
0 0

 INFO [ScheduledTasks:1] 2016-05-26 18:15:58,591 StatusLogger.java
(line 70) GossipStage   0 0
335 0 0
 INFO [ScheduledTasks:1] 2016-05-26 18:16:04,195 StatusLogger.java
(line 70) CacheCleanupExecutor  0 0 0
0 0

 INFO [ScheduledTasks:1] 2016-05-26 18:16:06,526 StatusLogger.java
(line 70) MigrationStage0 0 0
0 0

 INFO [ScheduledTasks:1] 2016-05-26 18:16:06,527 StatusLogger.java
(line 70) MemoryMeter   1 4 26
0 0

 INFO [ScheduledTasks:1] 2016-05-26 18:16:06,527 StatusLogger.java
(line 70) ValidationExecutor0 0 0
0 0

DEBUG [MessagingService-Outgoing-/10.255.235.19
] 2016-05-26 18:16:06,518
OutboundTcpConnection.java (line 290) attempting to connect to
/10.255.235.19 
 INFO [GossipTasks:1] 2016-05-26 18:16:22,912 Gossiper.java (line
992) InetAddress /10.255.235.28  is now DOWN
 INFO [ScheduledTasks:1] 2016-05-26 18:16:22,952 StatusLogger.java
(line 70) FlushWriter   1 5 47
025

 INFO [ScheduledTasks:1] 2016-05-26 18:16:22,953 StatusLogger.java
(line 70) InternalResponseStage 0 0 0
0 0

ERROR [ReadStage:27] 2016-05-26 18:16:29,250 CassandraDaemon.java
(line 258) Exception in thread Thread[ReadStage:27,5,main]
java.lang.OutOfMemoryError: Java heap space
at

org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:347)
at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:392)
at

org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355)
at

org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:124)
at

org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:85)
at org.apache.cassandra.db.Column$1.computeNext(Column.java:75)
at org.apache.cassandra.db.Column$1.computeNext(Column.java:64)
at

com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at

com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
com.google.common.collect.AbstractIterator.next(AbstractIterator.java:153)
at

org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:434)
at

org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.fetchMoreData(IndexedSliceReader.java:387)
at

org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:145)
at

Re: Thrift client creates massive amounts of network packets

[Ralf Steppacher]

Thanks Eric. Indeed it is the way Titan works during graph-traversal: 
https://groups.google.com/forum/#!topic/aureliusgraphs/IwzRMNB0zzM 
. Newer 
Titan versions attempt to batch the requests and reduce the number of network 
roundtrips that way. I will give that a shot.

Ralf


> On 26.05.2016, at 18:36, Eric Stevens  wrote:
> 
> If it's a single node cluster, then it's not consistency level related as all 
> consistencies are essentially the same.  This looks instead like a usage 
> pattern that's entirely driven by Titan's read pattern which appears to be 
> lots of tiny reads (probably not too surprising for a graph database).  In 
> that case you probably want to go to the Titan community to see what their 
> recommendations are WRT performance.
> 
> On Thu, May 26, 2016 at 1:18 AM Ralf Steppacher  > wrote:
> Eric,
> 
> thanks for the hint. Titan 0.5.4 uses ONE, not LOCAL_ONE. I can try and patch 
> the version. Given that it is a single node cluster for the time being, would 
> your remarks apply to that particular setup?
> 
> 
> Thanks again!
> Ralf
> 
> 
>> On 24.05.2016, at 19:18, Eric Stevens > > wrote:
>> 
>> I'm not familiar with Titan's usage patterns for Cassandra, but I wonder if 
>> this is because of the consistency level it's querying Cassandra at - i.e. 
>> if CL isn't LOCAL_[something], then this might just be lots of little 
>> checksums required to satisfy consistency requirements.
>> 
>> On Mon, May 23, 2016 at 7:22 AM Ralf Steppacher > > wrote:
>> I remembered that Titan treats edges (and vertices?) as immutable and 
>> deletes the entity and re-creates it on every change.
>> So I set the gc_grace_seconds to 0 for every table in the Titan keyspace and 
>> ran a major compaction. However, this made the situation worse. Instead of 
>> roughly 2’700 tcp packets per user request before the compaction, the same 
>> request now results in 5’400 packets. Which is suspiciously close to a 
>> factor or 2. But I have no idea wha to make of it.
>> 
>> Ralf
>> 
>> 
>> > On 20.05.2016, at 15:11, Ralf Steppacher > > > wrote:
>> >
>> > Hi all,
>> >
>> > tl:dr
>> > The Titan 0.5.4 cassandrathrift client + C* 2.0.8/2.2.6 create massive 
>> > amounts of network packets for multiget_slice queries. Is there a way to 
>> > avoid the “packet storm”?
>> >
>> >
>> > Details...
>> >
>> > We are using Titan 0.5.4 with its cassandrathrift storage engine to 
>> > connect to a single node cluster running C* 2.2.6 (we also tried 2.0.8, 
>> > which is the version in Titans dependencies). When moving to a 
>> > multi-datacenter setup with the client in one DC and the C* server in the 
>> > other, we ran into the problem that response times from Cassandra/the 
>> > graph became unacceptable (>30s vs. 0.2s within datacenter). Looking at 
>> > the network traffic we saw that the client and server exchange a massive 
>> > number of very small packets.
>> > The user action we were tracing yields three packets of type “REPLY 
>> > multiget_slice”. Per such a reply we see about 1’000 of packet pairs like 
>> > this going back and forth between client and server:
>> >
>> > 968   09:45:55.354613   x.x.x.30 x.x.x.98 TCP   181   54406 → 9160 [PSH, 
>> > ACK] Seq=53709 Ack=39558 Win=1002 Len=115 TSval=4169130400 TSecr=4169119527
>> >    00 50 56 a7 d6 0d 00 0c 29 d1 a4 5e 08 00 45 00  .PV.)..^..E.
>> > 0010   00 a7 e3 6d 40 00 40 06 fe 3c ac 13 00 1e ac 13  ...m@.@..<..
>> > 0020   00 62 d4 86 23 c8 2c 30 4e 45 1b 4b 0b 55 80 18  .b..#.,0NE.K.U..
>> > 0030   03 ea 59 40 00 00 01 01 08 0a f8 7f e1 a0 f8 7f  ..Y@
>> > 0040   b7 27 00 00 00 6f 80 01 00 01 00 00 00 0e 6d 75  .'...omu
>> > 0050   6c 74 69 67 65 74 5f 73 6c 69 63 65 00 00 3a 38  ltiget_slice..:8
>> > 0060   0f 00 01 0b 00 00 00 01 00 00 00 08 00 00 00 00  
>> > 0070   00 00 ab 00 0c 00 02 0b 00 03 00 00 00 09 65 64  ..ed
>> > 0080   67 65 73 74 6f 72 65 00 0c 00 03 0c 00 02 0b 00  gestore.
>> > 0090   01 00 00 00 02 72 c0 0b 00 02 00 00 00 02 72 c1  .rr.
>> > 00a0   02 00 03 00 08 00 04 7f ff ff ff 00 00 08 00 04  
>> > 00b0   00 00 00 01 00   .
>> >
>> > 969   09:45:55.354825   x.x.x.98 x.x.x.30 TCP   123   9160 → 54406 [PSH, 
>> > ACK] Seq=39558 Ack=53824 Win=1540 Len=57 TSval=4169119546 TSecr=4169130400
>> >    00 0c 29 d1 a4 5e 00 50 56 a7 d6 0d 08 00 45 00  ..)..^.PV.E.
>> > 0010   00 6d 19 dd 40 00 40 06 c8 07 ac 13 00 62 ac 13  .m..@.@..b..
>> > 0020   00 1e 23 c8 d4 86 1b 4b 0b 55 2c 30 4e b8 80 18  ..#K.U,0N...
>> > 0030   06 04 3b d6 00 00 01 01 08 0a f8 7f b7 3a f8 7f  ..;..:..
>> > 0040   e1