Re: Cassandra is not running

[Irtiza Ali]

Thankyou everyone

On 13 Feb 2018 8:50 p.m., "Michael Shuler"  wrote:

> On 02/13/2018 09:45 AM, Jürgen Albersdorfer wrote:
> > 1.8.0_161 is not yet supported - try 1.8.0_151
> >
> >> Am 13.02.2018 um 16:44 schrieb Irtiza Ali  >> >:
> >>
> >> 1.8.0_161
> >
>
> Or install the 3.11.2 tentative release, which should work fine on
> 1.8.0_161.
>
> https://lists.apache.org/thread.html/6c38262e25cfca27b4e697aa429fe1
> c51cb83b8fa341ec8a860c272b@%3Cdev.cassandra.apache.org%3E
>
> Looks like a packaged install, so you can find those at:
> https://people.apache.org/~mshuler/
>
> --
> Kind regards,
> Michael
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: Cassandra is not running

[Michael Shuler]

On 02/13/2018 09:45 AM, Jürgen Albersdorfer wrote:
> 1.8.0_161 is not yet supported - try 1.8.0_151
> 
>> Am 13.02.2018 um 16:44 schrieb Irtiza Ali > >:
>>
>> 1.8.0_161
> 

Or install the 3.11.2 tentative release, which should work fine on
1.8.0_161.

https://lists.apache.org/thread.html/6c38262e25cfca27b4e697aa429fe1c51cb83b8fa341ec8a860c272b@%3Cdev.cassandra.apache.org%3E

Looks like a packaged install, so you can find those at:
https://people.apache.org/~mshuler/

-- 
Kind regards,
Michael

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

RE: Cassandra is not running

[Haro, Vanessa (Nokia - US/Naperville)]

Adding known issue reference and fix versions:
CASSANDRA-14173<https://issues.apache.org/jira/browse/CASSANDRA-14173>
Fix Version/s: 3.11.2/4.0

From: Jürgen Albersdorfer [mailto:jalbersdor...@gmail.com]
Sent: Tuesday, February 13, 2018 9:45 AM
To: user@cassandra.apache.org
Subject: Re: Cassandra is not running

1.8.0_161 is not yet supported - try 1.8.0_151


Am 13.02.2018 um 16:44 schrieb Irtiza Ali <i...@an10.io<mailto:i...@an10.io>>:

1.8.0_161

Re: Cassandra is not running

[@Nandan@]

Downgrade your Java version please

On Feb 13, 2018 11:46 PM, "Irtiza Ali"  wrote:

> Thank you
>
> On 13 Feb 2018 20:45, "Jürgen Albersdorfer" 
> wrote:
>
>> 1.8.0_161 is not yet supported - try 1.8.0_151
>>
>> Am 13.02.2018 um 16:44 schrieb Irtiza Ali :
>>
>> 1.8.0_161
>>
>>
>>

Re: Cassandra is not running

[Irtiza Ali]

Thank you

On 13 Feb 2018 20:45, "Jürgen Albersdorfer"  wrote:

> 1.8.0_161 is not yet supported - try 1.8.0_151
>
> Am 13.02.2018 um 16:44 schrieb Irtiza Ali :
>
> 1.8.0_161
>
>
>

Re: Cassandra is not running

[Jürgen Albersdorfer]

1.8.0_161 is not yet supported - try 1.8.0_151

> Am 13.02.2018 um 16:44 schrieb Irtiza Ali :
> 
> 1.8.0_161

Re: Cassandra is not running

[Irtiza Ali]

output of the cassandra system log file:

INFO  [main] 2018-02-13 20:25:40,973 DatabaseDescriptor.java:367 -
DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO  [main] 2018-02-13 20:25:40,973 DatabaseDescriptor.java:421 - Global
memtable on-heap threshold is enabled at 480MB
INFO  [main] 2018-02-13 20:25:40,974 DatabaseDescriptor.java:425 - Global
memtable off-heap threshold is enabled at 480MB
INFO  [main] 2018-02-13 20:25:41,294 RateBasedBackPressure.java:123 -
Initialized back-pressure with high ratio: 0.9, factor: 5, flow: FAST,
window size: 2000.
INFO  [main] 2018-02-13 20:25:41,295 DatabaseDescriptor.java:725 -
Back-pressure is disabled with strategy
org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5,
flow=FAST}.
ERROR [main] 2018-02-13 20:25:41,546 CassandraDaemon.java:706 - Exception
encountered during startup
java.lang.AbstractMethodError:
org.apache.cassandra.utils.JMXServerUtils$Exporter.exportObject(Ljava/rmi/Remote;ILjava/rmi/server/RMIClientSocketFactory;Ljava/rmi/server/RMIServerSocketFactory;Lsun/misc/ObjectInputFilter;)Ljava/rmi/Remote;
at
javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:150)
~[na:1.8.0_161]
at
javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:135)
~[na:1.8.0_161]
at
javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:405)
~[na:1.8.0_161]
at
org.apache.cassandra.utils.JMXServerUtils.createJMXServer(JMXServerUtils.java:104)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.service.CassandraDaemon.maybeInitJmx(CassandraDaemon.java:143)
[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:188)
[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:600)
[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:689)
[apache-cassandra-3.11.1.jar:3.11.1]
INFO  [main] 2018-02-13 20:29:58,199 YamlConfigurationLoader.java:89 -
Configuration location: file:/etc/cassandra/cassandra.yaml



On Tue, Feb 13, 2018 at 8:37 PM, @Nandan@ 
wrote:

> What error message or WARN you got in system.log file and also check
> output.log file ..
>
>
> On Feb 13, 2018 11:34 PM, "Irtiza Ali"  wrote:
>
>> What should I do now?
>>
>> On 13 Feb 2018 20:21, "Irtiza Ali"  wrote:
>>
>>> Thank you i will check it
>>>
>>>
>>> On 13 Feb 2018 20:16, "Nicolas Guyomar" 
>>> wrote:
>>>
 Hi,

 Such a quick failure might indicate that you are trying to start
 Cassandra with the latest JDK which is not yet supported.

 You should have a look at the /var/log/system.log

 On 13 February 2018 at 16:03, Irtiza Ali  wrote:

> Hello everyone:
>
> Whenever I try to run the Cassandra it stop with status:
>
> result of [sudo service cassandra status] command:
>
> ● cassandra.service - LSB: distributed storage system for structured
> data
>Loaded: loaded (/etc/init.d/cassandra; bad; vendor preset: enabled)
>Active: active (exited) since 2018-02-13 19:57:51 PKT; 31s ago
>  Docs: man:systemd-sysv-generator(8)
>   Process: 28844 ExecStop=/etc/init.d/cassandra stop (code=exited,
> status=0/SUCCESS)
>   Process: 28929 ExecStart=/etc/init.d/cassandra start (code=exited,
> status=0/SUCCESS)
>
> Anyone knows why cassandra is not running properly?
>
> With Regards
> Irtiza Ali
>

Re: Cassandra is not running

[@Nandan@]

What error message or WARN you got in system.log file and also check
output.log file ..


On Feb 13, 2018 11:34 PM, "Irtiza Ali"  wrote:

> What should I do now?
>
> On 13 Feb 2018 20:21, "Irtiza Ali"  wrote:
>
>> Thank you i will check it
>>
>>
>> On 13 Feb 2018 20:16, "Nicolas Guyomar" 
>> wrote:
>>
>>> Hi,
>>>
>>> Such a quick failure might indicate that you are trying to start
>>> Cassandra with the latest JDK which is not yet supported.
>>>
>>> You should have a look at the /var/log/system.log
>>>
>>> On 13 February 2018 at 16:03, Irtiza Ali  wrote:
>>>
 Hello everyone:

 Whenever I try to run the Cassandra it stop with status:

 result of [sudo service cassandra status] command:

 ● cassandra.service - LSB: distributed storage system for structured
 data
Loaded: loaded (/etc/init.d/cassandra; bad; vendor preset: enabled)
Active: active (exited) since 2018-02-13 19:57:51 PKT; 31s ago
  Docs: man:systemd-sysv-generator(8)
   Process: 28844 ExecStop=/etc/init.d/cassandra stop (code=exited,
 status=0/SUCCESS)
   Process: 28929 ExecStart=/etc/init.d/cassandra start (code=exited,
 status=0/SUCCESS)

 Anyone knows why cassandra is not running properly?

 With Regards
 Irtiza Ali

>>>
>>>

Re: Cassandra is not running

[Irtiza Ali]

What should I do now?

On 13 Feb 2018 20:21, "Irtiza Ali"  wrote:

> Thank you i will check it
>
>
> On 13 Feb 2018 20:16, "Nicolas Guyomar"  wrote:
>
>> Hi,
>>
>> Such a quick failure might indicate that you are trying to start
>> Cassandra with the latest JDK which is not yet supported.
>>
>> You should have a look at the /var/log/system.log
>>
>> On 13 February 2018 at 16:03, Irtiza Ali  wrote:
>>
>>> Hello everyone:
>>>
>>> Whenever I try to run the Cassandra it stop with status:
>>>
>>> result of [sudo service cassandra status] command:
>>>
>>> ● cassandra.service - LSB: distributed storage system for structured data
>>>Loaded: loaded (/etc/init.d/cassandra; bad; vendor preset: enabled)
>>>Active: active (exited) since 2018-02-13 19:57:51 PKT; 31s ago
>>>  Docs: man:systemd-sysv-generator(8)
>>>   Process: 28844 ExecStop=/etc/init.d/cassandra stop (code=exited,
>>> status=0/SUCCESS)
>>>   Process: 28929 ExecStart=/etc/init.d/cassandra start (code=exited,
>>> status=0/SUCCESS)
>>>
>>> Anyone knows why cassandra is not running properly?
>>>
>>> With Regards
>>> Irtiza Ali
>>>
>>
>>

Re: Cassandra is not running

[Irtiza Ali]

Thank you i will check it


On 13 Feb 2018 20:16, "Nicolas Guyomar"  wrote:

> Hi,
>
> Such a quick failure might indicate that you are trying to start Cassandra
> with the latest JDK which is not yet supported.
>
> You should have a look at the /var/log/system.log
>
> On 13 February 2018 at 16:03, Irtiza Ali  wrote:
>
>> Hello everyone:
>>
>> Whenever I try to run the Cassandra it stop with status:
>>
>> result of [sudo service cassandra status] command:
>>
>> ● cassandra.service - LSB: distributed storage system for structured data
>>Loaded: loaded (/etc/init.d/cassandra; bad; vendor preset: enabled)
>>Active: active (exited) since 2018-02-13 19:57:51 PKT; 31s ago
>>  Docs: man:systemd-sysv-generator(8)
>>   Process: 28844 ExecStop=/etc/init.d/cassandra stop (code=exited,
>> status=0/SUCCESS)
>>   Process: 28929 ExecStart=/etc/init.d/cassandra start (code=exited,
>> status=0/SUCCESS)
>>
>> Anyone knows why cassandra is not running properly?
>>
>> With Regards
>> Irtiza Ali
>>
>
>

Re: Cassandra is not running

[Nicolas Guyomar]

Hi,

Such a quick failure might indicate that you are trying to start Cassandra
with the latest JDK which is not yet supported.

You should have a look at the /var/log/system.log

On 13 February 2018 at 16:03, Irtiza Ali  wrote:

> Hello everyone:
>
> Whenever I try to run the Cassandra it stop with status:
>
> result of [sudo service cassandra status] command:
>
> ● cassandra.service - LSB: distributed storage system for structured data
>Loaded: loaded (/etc/init.d/cassandra; bad; vendor preset: enabled)
>Active: active (exited) since 2018-02-13 19:57:51 PKT; 31s ago
>  Docs: man:systemd-sysv-generator(8)
>   Process: 28844 ExecStop=/etc/init.d/cassandra stop (code=exited,
> status=0/SUCCESS)
>   Process: 28929 ExecStart=/etc/init.d/cassandra start (code=exited,
> status=0/SUCCESS)
>
> Anyone knows why cassandra is not running properly?
>
> With Regards
> Irtiza Ali
>

Re: Cassandra 2.1: replace running node without streaming

[Oleksandr Shulgin]

On Sat, Feb 3, 2018 at 11:23 AM, Kyrylo Lebediev 
wrote:

> Just tested on 3.11.1 and it worked for me (you may see the logs below).
>
> Just comprehended that there is one important prerequisite this method to
> work: new node MUST be located in the same rack (in terms of C*) as the old
> one. Otherwise correct replicas placement order will be violated (I mean
> when replicas of the same token range should be placed in different racks).
>

Correct.

Anyway, even having successful run of node replacement in sandbox I'm still
> in doubt.
>
> Just wondering why this procedure which seems to be much easier than
> [add/remove node] or [replace a node] which are documented ways for live
> node replacement, has never been included into documentation.
>
> Does anybody in the ML know the reason for this?
>

There are a number of reasons why one would need to replace a node.  Losing
a disk would be the most frequent one, I guess.  In that case using
replace_address is the way to go, since it allows you to avoid any
ownership changes.

At the same time on EC2 you might be replacing nodes in order to apply
security updates to your base machine image, etc.  In this case it is
possible to apply the described procedure to migrate the data to the new
node.  However, given that your nodes are small enough, simply using
replace_address seems like a more straightforward way to me.

Also, for some reason in his article Carlos drops files of system keyspace
> (which contains system.local table):
>
> In the new node, delete all system tables except for the schema ones. This
> will ensure that the new Cassandra node will not have any corrupt or
> previous configuration assigned.
>
>1. sudo cd /var/lib/cassandra/data/system && sudo ls | grep -v schema
>| xargs -I {} sudo rm -rf {}
>
>
Ah, this sounds like a wrong thing to do.  That would remove system.local
keyspace, which I expect makes the node forget its tokens.

I wouldn't do that: the node's state on disk should be just like after a
normal restart.

--
Alex

Re: Cassandra 2.1: replace running node without streaming

[Jürgen Albersdorfer]

Good Point about the Rack - Kyrill! This makes total sense to me.
Deleting the System Keyspace not really, If this contains all sensitive 
information about the node.
Maybe this makes sense in conjunction with replace_node(at_first_boot) Option.
Some comments from devs about this would be great.

Regards,
Jürgen 

> Am 03.02.2018 um 16:42 schrieb Kyrylo Lebediev <kyrylo_lebed...@epam.com>:
> 
> I've found modified Carlos' article (more recent than that I was referring 
> to) and this one contains the same method as you described, Oleksandr:
> https://mrcalonso.com/2016/01/26/cassandra-instantaneous-in-place-node-replacement
> 
> Thank you for your readiness to help!
> 
> Kind Regards, 
> Kyrill
> From: Kyrylo Lebediev <kyrylo_lebed...@epam.com>
> Sent: Saturday, February 3, 2018 12:23:15 PM
> To: User
> Subject: Re: Cassandra 2.1: replace running node without streaming
>  
> Thank you Oleksandr,
> Just tested on 3.11.1 and it worked for me (you may see the logs below).
> Just comprehended that there is one important prerequisite this method to 
> work: new node MUST be located in the same rack (in terms of C*) as the old 
> one. Otherwise correct replicas placement order will be violated (I mean when 
> replicas of the same token range should be placed in different racks). 
> 
> Anyway, even having successful run of node replacement in sandbox I'm still 
> in doubt. 
> Just wondering why this procedure which seems to be much easier than 
> [add/remove node] or [replace a node] which are documented ways for live node 
> replacement, has never been included into documentation. 
> Does anybody in the ML know the reason for this?
> 
> Also, for some reason in his article Carlos drops files of system keyspace 
> (which contains system.local table):
> In the new node, delete all system tables except for the schema ones. This 
> will ensure that the new Cassandra node will not have any corrupt or previous 
> configuration assigned.
> sudo cd /var/lib/cassandra/data/system && sudo ls | grep -v schema | xargs -I 
> {} sudo rm -rf {}
> 
> http://engineering.mydrivesolutions.com/posts/cassandra_nodes_replacement/
> [Carlos, if you are here might you, please, comment ]
> 
> So still a mystery to me. 
> 
> -
> Logs for 3.1.11
> -
> 
> == Before:
> --  Address   Load   Tokens   Owns (effective)  Host ID   
> Rack
> UN  10.10.10.222  256.61 KiB  3100.0%
> bd504008-5ff0-4b6c-a3a6-a07049e61c31  rack1
> UN  10.10.10.223  225.65 KiB  3100.0%
> c562263f-4126-4935-b9f7-f4e7d0dc70b4  rack1 <<<<<<
> UN  10.10.10.221  187.39 KiB  3100.0%
> d312c083-8808-4c98-a3ab-72a7cd18b31f  rack1
> 
> === After:
> --  Address   Load   Tokens   Owns (effective)  Host ID   
> Rack
> UN  10.10.10.222  245.84 KiB  3100.0%
> bd504008-5ff0-4b6c-a3a6-a07049e61c31  rack1
> UN  10.10.10.221  192.8 KiB  3100.0%
> d312c083-8808-4c98-a3ab-72a7cd18b31f  rack1
> UN  10.10.10.224  266.61 KiB  3100.0%
> c562263f-4126-4935-b9f7-f4e7d0dc70b4  rack1  <<<<< 
> 
> 
> 
> == Logs from another node (10.10.10.221):
> INFO  [HANDSHAKE-/10.10.10.224] 2018-02-03 11:33:01,397 
> OutboundTcpConnection.java:560 - Handshaking version with /10.10.10.224
> INFO  [GossipStage:1] 2018-02-03 11:33:01,431 Gossiper.java:1067 - Node 
> /10.10.10.224 is now part of the cluster
> INFO  [RequestResponseStage-1] 2018-02-03 11:33:02,190 Gossiper.java:1031 - 
> InetAddress /10.10.10.224 is now UP
> INFO  [RequestResponseStage-1] 2018-02-03 11:33:02,190 Gossiper.java:1031 - 
> InetAddress /10.10.10.224 is now UP
> WARN  [GossipStage:1] 2018-02-03 11:33:08,375 StorageService.java:2313 - Host 
> ID collision for c562263f-4126-4935-b9f7-f4e7d0dc70b4 between /10.10.10.223 
> and /10.10.10.224; /10.10.10.224 is the new owner
> INFO  [GossipTasks:1] 2018-02-03 11:33:08,806 Gossiper.java:810 - FatClient 
> /10.10.10.223 has been silent for 3ms, removing from gossip
> 
> == Logs from new node:
> INFO  [main] 2018-02-03 11:33:01,926 StorageService.java:1442 - JOINING: 
> Finish joining ring
> INFO  [GossipStage:1] 2018-02-03 11:33:02,659 Gossiper.java:1067 - Node 
> /10.10.10.223 is now part of the cluster
> WARN  [GossipStage:1] 2018-02-03 11:33:02,676 StorageService.java:2307 - Not 
> updating host ID c562263f-4126-4935-b9f7-f4e7d0dc70b4 for /10.10.10.223 
> because it's mine
> INFO  [GossipStage:1] 2018-02-03 11:33:02,683 StorageService.java:2365 - 
> Nodes /10.10.10.223 and /10.10.10.224 have the same toke

Re: Cassandra 2.1: replace running node without streaming

[Kyrylo Lebediev]

I've found modified Carlos' article (more recent than that I was referring to) 
and this one contains the same method as you described, Oleksandr:

https://mrcalonso.com/2016/01/26/cassandra-instantaneous-in-place-node-replacement


Thank you for your readiness to help!

Kind Regards,

Kyrill


From: Kyrylo Lebediev <kyrylo_lebed...@epam.com>
Sent: Saturday, February 3, 2018 12:23:15 PM
To: User
Subject: Re: Cassandra 2.1: replace running node without streaming


Thank you Oleksandr,

Just tested on 3.11.1 and it worked for me (you may see the logs below).

Just comprehended that there is one important prerequisite this method to work: 
new node MUST be located in the same rack (in terms of C*) as the old one. 
Otherwise correct replicas placement order will be violated (I mean when 
replicas of the same token range should be placed in different racks).

Anyway, even having successful run of node replacement in sandbox I'm still in 
doubt.

Just wondering why this procedure which seems to be much easier than 
[add/remove node] or [replace a node] which are documented ways for live node 
replacement, has never been included into documentation.

Does anybody in the ML know the reason for this?


Also, for some reason in his article Carlos drops files of system keyspace 
(which contains system.local table):

In the new node, delete all system tables except for the schema ones. This will 
ensure that the new Cassandra node will not have any corrupt or previous 
configuration assigned.

  1.  sudo cd /var/lib/cassandra/data/system && sudo ls | grep -v schema | 
xargs -I {} sudo rm -rf {}


http://engineering.mydrivesolutions.com/posts/cassandra_nodes_replacement/
[Carlos, if you are here might you, please, comment ]

So still a mystery to me.

-
Logs for 3.1.11

-

== Before:

--  Address   Load   Tokens   Owns (effective)  Host ID 
  Rack
UN  10.10.10.222  256.61 KiB  3100.0%
bd504008-5ff0-4b6c-a3a6-a07049e61c31  rack1
UN  10.10.10.223  225.65 KiB  3100.0%
c562263f-4126-4935-b9f7-f4e7d0dc70b4  rack1 <<<<<<
UN  10.10.10.221  187.39 KiB  3100.0%
d312c083-8808-4c98-a3ab-72a7cd18b31f  rack1

=== After:
--  Address   Load   Tokens   Owns (effective)  Host ID 
  Rack
UN  10.10.10.222  245.84 KiB  3100.0%
bd504008-5ff0-4b6c-a3a6-a07049e61c31  rack1
UN  10.10.10.221  192.8 KiB  3100.0%
d312c083-8808-4c98-a3ab-72a7cd18b31f  rack1
UN  10.10.10.224  266.61 KiB  3100.0%
c562263f-4126-4935-b9f7-f4e7d0dc70b4  rack1  <<<<<



== Logs from another node (10.10.10.221):
INFO  [HANDSHAKE-/10.10.10.224] 2018-02-03 11:33:01,397 
OutboundTcpConnection.java:560 - Handshaking version with /10.10.10.224
INFO  [GossipStage:1] 2018-02-03 11:33:01,431 Gossiper.java:1067 - Node 
/10.10.10.224 is now part of the cluster
INFO  [RequestResponseStage-1] 2018-02-03 11:33:02,190 Gossiper.java:1031 - 
InetAddress /10.10.10.224 is now UP
INFO  [RequestResponseStage-1] 2018-02-03 11:33:02,190 Gossiper.java:1031 - 
InetAddress /10.10.10.224 is now UP
WARN  [GossipStage:1] 2018-02-03 11:33:08,375 StorageService.java:2313 - Host 
ID collision for c562263f-4126-4935-b9f7-f4e7d0dc70b4 between /10.10.10.223 and 
/10.10.10.224; /10.10.10.224 is the new owner
INFO  [GossipTasks:1] 2018-02-03 11:33:08,806 Gossiper.java:810 - FatClient 
/10.10.10.223 has been silent for 3ms, removing from gossip

== Logs from new node:
INFO  [main] 2018-02-03 11:33:01,926 StorageService.java:1442 - JOINING: Finish 
joining ring
INFO  [GossipStage:1] 2018-02-03 11:33:02,659 Gossiper.java:1067 - Node 
/10.10.10.223 is now part of the cluster
WARN  [GossipStage:1] 2018-02-03 11:33:02,676 StorageService.java:2307 - Not 
updating host ID c562263f-4126-4935-b9f7-f4e7d0dc70b4 for /10.10.10.223 because 
it's mine
INFO  [GossipStage:1] 2018-02-03 11:33:02,683 StorageService.java:2365 - Nodes 
/10.10.10.223 and /10.10.10.224 have the same token -7774421781914237508.  
Ignoring /10.10.10.223
INFO  [GossipStage:1] 2018-02-03 11:33:02,686 StorageService.java:2365 - Nodes 
/10.10.10.223 and /10.10.10.224 have the same token 2257660731441815305.  
Ignoring /10.10.10.223
INFO  [GossipStage:1] 2018-02-03 11:33:02,692 StorageService.java:2365 - Nodes 
/10.10.10.223 and /10.10.10.224 have the same token 51879124242594885.  
Ignoring /10.10.10.223
WARN  [GossipTasks:1] 2018-02-03 11:33:03,985 Gossiper.java:789 - Gossip stage 
has 5 pending tasks; skipping status check (no nodes will be marked down)
INFO  [main] 2018-02-03 11:33:04,394 SecondaryIndexManager.java:509 - Executing 
pre-join tasks for: CFS(Keyspace='test', ColumnFamily='usr')
WARN  [GossipTasks:1] 2018-02-03 11:33:05,088 Gossiper.java:789 - Gossip stage 
has 7 pending tasks; skipping status check (no 

Re: Cassandra 2.1: replace running node without streaming

[Kyrylo Lebediev]

.10.222 is now DOWN<<<<< have no idea why this appeared in logs
INFO  [main] 2018-02-03 11:33:20,566 NativeTransportService.java:70 - Netty 
using native Epoll event loop
INFO  [HANDSHAKE-/10.10.10.222] 2018-02-03 11:33:20,714 
OutboundTcpConnection.java:560 - Handshaking version with /10.10.10.222




Kind Regards,
Kyrill



From: Oleksandr Shulgin <oleksandr.shul...@zalando.de>
Sent: Saturday, February 3, 2018 10:44:26 AM
To: User
Subject: Re: Cassandra 2.1: replace running node without streaming

On 3 Feb 2018 08:49, "Jürgen Albersdorfer" 
<jalbersdor...@gmail.com<mailto:jalbersdor...@gmail.com>> wrote:
Cool, good to know. Do you know this is still true for 3.11.1?

Well, I've never tried with that specific version, but this is pretty 
fundamental, so I would expect it to work the same way. Test in isolation if 
you want to be sure, though.

I don't think this is documented anywhere, however, since I had the same doubts 
before seeing it worked for the first time.

--
Alex

Am 03.02.2018 um 08:19 schrieb Oleksandr Shulgin 
<oleksandr.shul...@zalando.de<mailto:oleksandr.shul...@zalando.de>>:

On 3 Feb 2018 02:42, "Kyrylo Lebediev" 
<kyrylo_lebed...@epam.com<mailto:kyrylo_lebed...@epam.com>> wrote:

Thanks, Oleksandr,
In my case I'll need to replace all nodes in the cluster (one-by-one), so 
streaming will introduce perceptible overhead.
My question is not about data movement/copy itself, but more about all this 
token magic.

Okay, let's say we stopped old node, moved data to new node.
Once it's started with auto_bootstrap=false it will be added to the cluster 
like an usual node, just skipping streaming stage, right?

For a cluster with vnodes enabled, during addition of new node its token ranges 
are calculated automatically by C* on startup.

So, how will C* know that this new node must be responsible for exactly the 
same token ranges as the old node was?
How would the rest of nodes in the cluster ('peers') figure out that old node 
should be replaced in ring by the new one?

Do you know about some  limitation for this process in case of C* 2.1.x with 
vnodes enabled?

A node stores its tokens and host id in the system.local table. Next time it 
starts up, it will use the same tokens as previously and the host id allows the 
rest of the cluster to see that it is the same node and ignore the IP address 
change. This happens regardless of auto_bootstrap setting.

Try "select * from system.local" to see what is recorded for the old node. When 
the new node starts up it should log "Using saved tokens" with the list of 
numbers. Other nodes should log something like "ignoring IP address change" for 
the affected node addresses.

Be careful though, to make sure that you put the data directory exactly where 
the new node expects to find it: otherwise it might just join as a brand new 
one, allocating new tokens. As a precaution it helps to ensure that the system 
user running the Cassandra process has no permission to create the data 
directory: this should stop the startup in case of misconfiguration.

Cheers,
--
Alex

Re: Cassandra 2.1: replace running node without streaming

[Oleksandr Shulgin]

On 3 Feb 2018 08:49, "Jürgen Albersdorfer"  wrote:

Cool, good to know. Do you know this is still true for 3.11.1?


Well, I've never tried with that specific version, but this is pretty
fundamental, so I would expect it to work the same way. Test in isolation
if you want to be sure, though.

I don't think this is documented anywhere, however, since I had the same
doubts before seeing it worked for the first time.

--
Alex

Am 03.02.2018 um 08:19 schrieb Oleksandr Shulgin <
oleksandr.shul...@zalando.de>:

On 3 Feb 2018 02:42, "Kyrylo Lebediev"  wrote:

Thanks, Oleksandr,
In my case I'll need to replace all nodes in the cluster (one-by-one), so
streaming will introduce perceptible overhead.
My question is not about data movement/copy itself, but more about all this
token magic.

Okay, let's say we stopped old node, moved data to new node.
Once it's started with auto_bootstrap=false it will be added to the cluster
like an usual node, just skipping streaming stage, right?

For a cluster with vnodes enabled, during addition of new node its token
ranges are calculated automatically by C* on startup.

So, how will C* know that this new node must be responsible for exactly the
same token ranges as the old node was?
How would the rest of nodes in the cluster ('peers') figure out that old
node should be replaced in ring by the new one?

Do you know about some  limitation for this process in case of C* 2.1.x
with vnodes enabled?


A node stores its tokens and host id in the system.local table. Next time
it starts up, it will use the same tokens as previously and the host id
allows the rest of the cluster to see that it is the same node and ignore
the IP address change. This happens regardless of auto_bootstrap setting.

Try "select * from system.local" to see what is recorded for the old node.
When the new node starts up it should log "Using saved tokens" with the
list of numbers. Other nodes should log something like "ignoring IP address
change" for the affected node addresses.

Be careful though, to make sure that you put the data directory exactly
where the new node expects to find it: otherwise it might just join as a
brand new one, allocating new tokens. As a precaution it helps to ensure
that the system user running the Cassandra process has no permission to
create the data directory: this should stop the startup in case of
misconfiguration.

Cheers,
--
Alex

Re: Cassandra 2.1: replace running node without streaming

[Jürgen Albersdorfer]

Cool, good to know. Do you know this is still true for 3.11.1?

> Am 03.02.2018 um 08:19 schrieb Oleksandr Shulgin 
> :
> 
> On 3 Feb 2018 02:42, "Kyrylo Lebediev"  wrote:
> Thanks, Oleksandr, 
> In my case I'll need to replace all nodes in the cluster (one-by-one), so 
> streaming will introduce perceptible overhead.
> My question is not about data movement/copy itself, but more about all this 
> token magic. 
> 
> Okay, let's say we stopped old node, moved data to new node.
> Once it's started with auto_bootstrap=false it will be added to the cluster 
> like an usual node, just skipping streaming stage, right?
> For a cluster with vnodes enabled, during addition of new node its token 
> ranges are calculated automatically by C* on startup.
> 
> So, how will C* know that this new node must be responsible for exactly the 
> same token ranges as the old node was?
> How would the rest of nodes in the cluster ('peers') figure out that old node 
> should be replaced in ring by the new one?
> Do you know about some  limitation for this process in case of C* 2.1.x with 
> vnodes enabled?
> 
> A node stores its tokens and host id in the system.local table. Next time it 
> starts up, it will use the same tokens as previously and the host id allows 
> the rest of the cluster to see that it is the same node and ignore the IP 
> address change. This happens regardless of auto_bootstrap setting.
> 
> Try "select * from system.local" to see what is recorded for the old node. 
> When the new node starts up it should log "Using saved tokens" with the list 
> of numbers. Other nodes should log something like "ignoring IP address 
> change" for the affected node addresses.
> 
> Be careful though, to make sure that you put the data directory exactly where 
> the new node expects to find it: otherwise it might just join as a brand new 
> one, allocating new tokens. As a precaution it helps to ensure that the 
> system user running the Cassandra process has no permission to create the 
> data directory: this should stop the startup in case of misconfiguration.
> 
> Cheers,
> --
> Alex
> 

Re: Cassandra 2.1: replace running node without streaming

[Oleksandr Shulgin]

On 3 Feb 2018 02:42, "Kyrylo Lebediev"  wrote:

Thanks, Oleksandr,
In my case I'll need to replace all nodes in the cluster (one-by-one), so
streaming will introduce perceptible overhead.
My question is not about data movement/copy itself, but more about all this
token magic.

Okay, let's say we stopped old node, moved data to new node.
Once it's started with auto_bootstrap=false it will be added to the cluster
like an usual node, just skipping streaming stage, right?

For a cluster with vnodes enabled, during addition of new node its token
ranges are calculated automatically by C* on startup.

So, how will C* know that this new node must be responsible for exactly the
same token ranges as the old node was?
How would the rest of nodes in the cluster ('peers') figure out that old
node should be replaced in ring by the new one?

Do you know about some  limitation for this process in case of C* 2.1.x
with vnodes enabled?


A node stores its tokens and host id in the system.local table. Next time
it starts up, it will use the same tokens as previously and the host id
allows the rest of the cluster to see that it is the same node and ignore
the IP address change. This happens regardless of auto_bootstrap setting.

Try "select * from system.local" to see what is recorded for the old node.
When the new node starts up it should log "Using saved tokens" with the
list of numbers. Other nodes should log something like "ignoring IP address
change" for the affected node addresses.

Be careful though, to make sure that you put the data directory exactly
where the new node expects to find it: otherwise it might just join as a
brand new one, allocating new tokens. As a precaution it helps to ensure
that the system user running the Cassandra process has no permission to
create the data directory: this should stop the startup in case of
misconfiguration.

Cheers,
--
Alex

Re: Cassandra 2.1: replace running node without streaming

[Kyrylo Lebediev]

Thanks, Oleksandr,
In my case I'll need to replace all nodes in the cluster (one-by-one), so 
streaming will introduce perceptible overhead.
My question is not about data movement/copy itself, but more about all this 
token magic.

Okay, let's say we stopped old node, moved data to new node.
Once it's started with auto_bootstrap=false it will be added to the cluster 
like an usual node, just skipping streaming stage, right?

For a cluster with vnodes enabled, during addition of new node its token ranges 
are calculated automatically by C* on startup.

So, how will C* know that this new node must be responsible for exactly the 
same token ranges as the old node was?
How would the rest of nodes in the cluster ('peers') figure out that old node 
should be replaced in ring by the new one?

Do you know about some  limitation for this process in case of C* 2.1.x with 
vnodes enabled?

Regards,
Kyrill


From: Oleksandr Shulgin <oleksandr.shul...@zalando.de>
Sent: Friday, February 2, 2018 4:26:30 PM
To: User
Subject: Re: Cassandra 2.1: replace running node without streaming

On Fri, Feb 2, 2018 at 3:15 PM, Kyrylo Lebediev 
<kyrylo_lebed...@epam.com<mailto:kyrylo_lebed...@epam.com>> wrote:

Hello All!

I've got a pretty standard task - to replace a running C* node [version 2.1.15, 
vnodes=256, Ec2Snitch] (IP address will change after replacement, have no 
control over it).

There are 2 ways stated in C* documentation how this can be done:

1) Add a new node, than 'nodetool decommission' [ = 2 data streaming + 2 token 
range recalculations],

2) Stop the node then replace it by setting -Dcassandra.replace_address [ = 1 
data streaming]
Unfortunately, both these methods imply data streaming.

Is there a supported solution how to replace a live healthy node without data 
streaming / bootstrapping?
Something like: "Stop old node, copy data to new node, start new node with 
auto_bootstrap=false etc..."

On EC2, if you're using EBS it's pretty easy: drain and stop the old node, 
attach the volume to the new one and start it.
If not using EBS, then you have to copy the data to the new node before it is 
started.


I was able to find a couple manuals on the Internet, like this one: 
http://engineering.mydrivesolutions.com/posts/cassandra_nodes_replacement/, but 
not having understanding of C* internals, I don't know if such hacks are safe.

More or less like that: rsync while the old node is still running, then stop 
the node and rsync again.

But given all the hassle, streaming with replace_address doesn't sound too 
costly to me.

Cheers,
--
Alex

Re: Cassandra 2.1: replace running node without streaming

[Oleksandr Shulgin]

On Fri, Feb 2, 2018 at 3:15 PM, Kyrylo Lebediev 
wrote:

> Hello All!
>
> I've got a pretty standard task - to replace a running C* node [version
> 2.1.15, vnodes=256, Ec2Snitch] (IP address will change after replacement,
> have no control over it).
>
> There are 2 ways stated in C* documentation how this can be done:
>
> 1) Add a new node, than 'nodetool decommission' [ = 2 data streaming + 2
> token range recalculations],
>
> 2) Stop the node then replace it by setting -Dcassandra.replace_address [
> = 1 data streaming]
> Unfortunately, both these methods imply data streaming.
>
> Is there a supported solution how to replace a live healthy node without
> data streaming / bootstrapping?
> Something like: "Stop old node, copy data to new node, start new node with
> auto_bootstrap=false etc..."
>

On EC2, if you're using EBS it's pretty easy: drain and stop the old node,
attach the volume to the new one and start it.
If not using EBS, then you have to copy the data to the new node before it
is started.


> I was able to find a couple manuals on the Internet, like this one:
> http://engineering.mydrivesolutions.com/posts/cassandra_nodes_replacement/,
> but not having understanding of C* internals, I don't know if such hacks
> are safe.
>

More or less like that: rsync while the old node is still running, then
stop the node and rsync again.

But given all the hassle, streaming with replace_address doesn't sound too
costly to me.

Cheers,
--
Alex

Re: Cassandra tuning for running test on a desktop

[Nicolas Labrot]

I have try 1400M, and Cassandra OOM too.

Is there another solution ? My data isn't very big.

It seems that is the merge of the db


On Wed, Apr 21, 2010 at 2:14 PM, Mark Greene green...@gmail.com wrote:

 Trying increasing Xmx. 1G is probably not enough for the amount of inserts
 you are doing.


 On Wed, Apr 21, 2010 at 8:10 AM, Nicolas Labrot nith...@gmail.com wrote:

 Hello,

 For my first message I will first thanks Cassandra contributors for their
 great works.

 I have a parameter issue with Cassandra (I hope it's just a parameter
 issue). I'm using Cassandra 6.0.1 with Hector client on my desktop. It's a
 simple dual core with 4GB of RAM on WinXP. I have keep the default JVM
 option inside cassandra.bat (Xmx1G)

 I'm trying to insert 3 millions of SC with 6 Columns each inside 1 CF
 (named Super1). The insertion go to 1 millions of SC (without slowdown) and
 Cassandra crash because of an OOM. (I store an average of 100 bytes per SC
 with a max of 10kB).
 I have aggressively decreased all the memories parameters without any
 respect to the consistency (My config is here [1]), the cache is turn off
 but Cassandra still go to OOM. I have joined the last line of the Cassandra
 life [2].

 What can I do to fix my issue ?  Is there another solution than increasing
 the Xmx ?

 Thanks for your help,

 Nicolas





 [1]
   Keyspaces
 Keyspace Name=Keyspace1
   ColumnFamily Name=Super1
 ColumnType=Super
 CompareWith=BytesType
 CompareSubcolumnsWith=BytesType /

 ReplicaPlacementStrategyorg.apache.cassandra.locator.RackUnawareStrategy/ReplicaPlacementStrategy
   ReplicationFactor1/ReplicationFactor

 EndPointSnitchorg.apache.cassandra.locator.EndPointSnitch/EndPointSnitch
 /Keyspace
   /Keyspaces
   CommitLogRotationThresholdInMB32/CommitLogRotationThresholdInMB

   DiskAccessModeauto/DiskAccessMode
   RowWarningThresholdInMB64/RowWarningThresholdInMB
   SlicedBufferSizeInKB64/SlicedBufferSizeInKB
   FlushDataBufferSizeInMB16/FlushDataBufferSizeInMB
   FlushIndexBufferSizeInMB4/FlushIndexBufferSizeInMB
   ColumnIndexSizeInKB64/ColumnIndexSizeInKB

   MemtableThroughputInMB16/MemtableThroughputInMB
   BinaryMemtableThroughputInMB32/BinaryMemtableThroughputInMB
   MemtableOperationsInMillions0.01/MemtableOperationsInMillions
   MemtableObjectCountInMillions0.01/MemtableObjectCountInMillions
   MemtableFlushAfterMinutes60/MemtableFlushAfterMinutes
   ConcurrentReads4/ConcurrentReads
   ConcurrentWrites8/ConcurrentWrites
 /Storage


 [2]
  INFO 13:36:41,062 Super1 has reached its threshold; switching in a fresh
 Memtable at
 CommitLogContext(file='d:/cassandra/commitlog\CommitLog-1271849783703.log',
 position=5417524)
  INFO 13:36:41,062 Enqueuing flush of Memtable(Super1)@15385755
  INFO 13:36:41,062 Writing Memtable(Super1)@15385755
  INFO 13:36:42,062 Completed flushing
 d:\cassandra\data\Keyspace1\Super1-711-Data.db
  INFO 13:36:45,781 Super1 has reached its threshold; switching in a fresh
 Memtable at
 CommitLogContext(file='d:/cassandra/commitlog\CommitLog-1271849783703.log',
 position=6065637)
  INFO 13:36:45,781 Enqueuing flush of Memtable(Super1)@15578910
  INFO 13:36:45,796 Writing Memtable(Super1)@15578910
  INFO 13:36:46,109 Completed flushing
 d:\cassandra\data\Keyspace1\Super1-712-Data.db
  INFO 13:36:54,296 GC for ConcurrentMarkSweep: 7149 ms, 58337240 reclaimed
 leaving 922392600 used; max is 1174208512
  INFO 13:36:54,593 Super1 has reached its threshold; switching in a fresh
 Memtable at
 CommitLogContext(file='d:/cassandra/commitlog\CommitLog-1271849783703.log',
 position=6722241)
  INFO 13:36:54,593 Enqueuing flush of Memtable(Super1)@24468872
  INFO 13:36:54,593 Writing Memtable(Super1)@24468872
  INFO 13:36:55,421 Completed flushing
 d:\cassandra\data\Keyspace1\Super1-713-Data.dbjava.lang.OutOfMemoryError:
 Java heap space
  INFO 13:37:08,281 GC for ConcurrentMarkSweep: 5561 ms, 9432 reclaimed
 leaving 971904520 used; max is 1174208512

Re: Cassandra tuning for running test on a desktop

[Mark Greene]

RAM doesn't necessarily need to be proportional but I would say the number
of nodes does. You can't just throw a bazillion inserts at one node. This is
the main benefit of Cassandra is that if you start hitting your capacity,
you add more machines and distribute the keys across more machines.

On Wed, Apr 21, 2010 at 9:07 AM, Nicolas Labrot nith...@gmail.com wrote:

 So does it means the RAM needed is proportionnal with the data handled ?

 Or Cassandra need a minimum amount or RAM when dataset is big?

 I must confess this OOM behaviour is strange.


 On Wed, Apr 21, 2010 at 2:54 PM, Mark Jones mjo...@imagehawk.com wrote:

  On my 4GB machine I’m giving it 3GB and having no trouble with 60+
 million 500 byte columns



 *From:* Nicolas Labrot [mailto:nith...@gmail.com]
 *Sent:* Wednesday, April 21, 2010 7:47 AM
 *To:* user@cassandra.apache.org
 *Subject:* Re: Cassandra tuning for running test on a desktop



 I have try 1400M, and Cassandra OOM too.

 Is there another solution ? My data isn't very big.

 It seems that is the merge of the db

  On Wed, Apr 21, 2010 at 2:14 PM, Mark Greene green...@gmail.com wrote:

 Trying increasing Xmx. 1G is probably not enough for the amount of inserts
 you are doing.



 On Wed, Apr 21, 2010 at 8:10 AM, Nicolas Labrot nith...@gmail.com
 wrote:

 Hello,

 For my first message I will first thanks Cassandra contributors for their
 great works.

 I have a parameter issue with Cassandra (I hope it's just a parameter
 issue). I'm using Cassandra 6.0.1 with Hector client on my desktop. It's a
 simple dual core with 4GB of RAM on WinXP. I have keep the default JVM
 option inside cassandra.bat (Xmx1G)

 I'm trying to insert 3 millions of SC with 6 Columns each inside 1 CF
 (named Super1). The insertion go to 1 millions of SC (without slowdown) and
 Cassandra crash because of an OOM. (I store an average of 100 bytes per SC
 with a max of 10kB).
 I have aggressively decreased all the memories parameters without any
 respect to the consistency (My config is here [1]), the cache is turn off
 but Cassandra still go to OOM. I have joined the last line of the Cassandra
 life [2].

 What can I do to fix my issue ?  Is there another solution than increasing
 the Xmx ?

 Thanks for your help,

 Nicolas





 [1]
   Keyspaces
 Keyspace Name=Keyspace1
   ColumnFamily Name=Super1
 ColumnType=Super
 CompareWith=BytesType
 CompareSubcolumnsWith=BytesType /

 ReplicaPlacementStrategyorg.apache.cassandra.locator.RackUnawareStrategy/ReplicaPlacementStrategy
   ReplicationFactor1/ReplicationFactor

 EndPointSnitchorg.apache.cassandra.locator.EndPointSnitch/EndPointSnitch
 /Keyspace
   /Keyspaces
   CommitLogRotationThresholdInMB32/CommitLogRotationThresholdInMB

   DiskAccessModeauto/DiskAccessMode
   RowWarningThresholdInMB64/RowWarningThresholdInMB
   SlicedBufferSizeInKB64/SlicedBufferSizeInKB
   FlushDataBufferSizeInMB16/FlushDataBufferSizeInMB
   FlushIndexBufferSizeInMB4/FlushIndexBufferSizeInMB
   ColumnIndexSizeInKB64/ColumnIndexSizeInKB

   MemtableThroughputInMB16/MemtableThroughputInMB
   BinaryMemtableThroughputInMB32/BinaryMemtableThroughputInMB
   MemtableOperationsInMillions0.01/MemtableOperationsInMillions
   MemtableObjectCountInMillions0.01/MemtableObjectCountInMillions
   MemtableFlushAfterMinutes60/MemtableFlushAfterMinutes
   ConcurrentReads4/ConcurrentReads
   ConcurrentWrites8/ConcurrentWrites
 /Storage


 [2]
  INFO 13:36:41,062 Super1 has reached its threshold; switching in a fresh
 Memtable at
 CommitLogContext(file='d:/cassandra/commitlog\CommitLog-1271849783703.log',
 position=5417524)
  INFO 13:36:41,062 Enqueuing flush of Memtable(Super1)@15385755
  INFO 13:36:41,062 Writing Memtable(Super1)@15385755
  INFO 13:36:42,062 Completed flushing
 d:\cassandra\data\Keyspace1\Super1-711-Data.db
  INFO 13:36:45,781 Super1 has reached its threshold; switching in a fresh
 Memtable at
 CommitLogContext(file='d:/cassandra/commitlog\CommitLog-1271849783703.log',
 position=6065637)
  INFO 13:36:45,781 Enqueuing flush of Memtable(Super1)@15578910
  INFO 13:36:45,796 Writing Memtable(Super1)@15578910
  INFO 13:36:46,109 Completed flushing
 d:\cassandra\data\Keyspace1\Super1-712-Data.db
  INFO 13:36:54,296 GC for ConcurrentMarkSweep: 7149 ms, 58337240 reclaimed
 leaving 922392600 used; max is 1174208512
  INFO 13:36:54,593 Super1 has reached its threshold; switching in a fresh
 Memtable at
 CommitLogContext(file='d:/cassandra/commitlog\CommitLog-1271849783703.log',
 position=6722241)
  INFO 13:36:54,593 Enqueuing flush of Memtable(Super1)@24468872
  INFO 13:36:54,593 Writing Memtable(Super1)@24468872
  INFO 13:36:55,421 Completed flushing
 d:\cassandra\data\Keyspace1\Super1-713-Data.dbjava.lang.OutOfMemoryError:
 Java heap space
  INFO 13:37:08,281 GC for ConcurrentMarkSweep: 5561 ms, 9432 reclaimed
 leaving 971904520 used; max is 1174208512

Re: Cassandra tuning for running test on a desktop

[Mark Greene]

Hit send to early

That being said a lot of people running Cassandra in production are using
4-6GB max heaps on 8GB machines, don't know if that helps but hopefully
gives you some perspective.

On Wed, Apr 21, 2010 at 10:39 AM, Mark Greene green...@gmail.com wrote:

 RAM doesn't necessarily need to be proportional but I would say the number
 of nodes does. You can't just throw a bazillion inserts at one node. This is
 the main benefit of Cassandra is that if you start hitting your capacity,
 you add more machines and distribute the keys across more machines.


 On Wed, Apr 21, 2010 at 9:07 AM, Nicolas Labrot nith...@gmail.com wrote:

 So does it means the RAM needed is proportionnal with the data handled ?

 Or Cassandra need a minimum amount or RAM when dataset is big?

 I must confess this OOM behaviour is strange.


 On Wed, Apr 21, 2010 at 2:54 PM, Mark Jones mjo...@imagehawk.com wrote:

  On my 4GB machine I’m giving it 3GB and having no trouble with 60+
 million 500 byte columns



 *From:* Nicolas Labrot [mailto:nith...@gmail.com]
 *Sent:* Wednesday, April 21, 2010 7:47 AM
 *To:* user@cassandra.apache.org
 *Subject:* Re: Cassandra tuning for running test on a desktop



 I have try 1400M, and Cassandra OOM too.

 Is there another solution ? My data isn't very big.

 It seems that is the merge of the db

  On Wed, Apr 21, 2010 at 2:14 PM, Mark Greene green...@gmail.com
 wrote:

 Trying increasing Xmx. 1G is probably not enough for the amount of
 inserts you are doing.



 On Wed, Apr 21, 2010 at 8:10 AM, Nicolas Labrot nith...@gmail.com
 wrote:

 Hello,

 For my first message I will first thanks Cassandra contributors for their
 great works.

 I have a parameter issue with Cassandra (I hope it's just a parameter
 issue). I'm using Cassandra 6.0.1 with Hector client on my desktop. It's a
 simple dual core with 4GB of RAM on WinXP. I have keep the default JVM
 option inside cassandra.bat (Xmx1G)

 I'm trying to insert 3 millions of SC with 6 Columns each inside 1 CF
 (named Super1). The insertion go to 1 millions of SC (without slowdown) and
 Cassandra crash because of an OOM. (I store an average of 100 bytes per SC
 with a max of 10kB).
 I have aggressively decreased all the memories parameters without any
 respect to the consistency (My config is here [1]), the cache is turn off
 but Cassandra still go to OOM. I have joined the last line of the Cassandra
 life [2].

 What can I do to fix my issue ?  Is there another solution than
 increasing the Xmx ?

 Thanks for your help,

 Nicolas





 [1]
   Keyspaces
 Keyspace Name=Keyspace1
   ColumnFamily Name=Super1
 ColumnType=Super
 CompareWith=BytesType
 CompareSubcolumnsWith=BytesType /

 ReplicaPlacementStrategyorg.apache.cassandra.locator.RackUnawareStrategy/ReplicaPlacementStrategy
   ReplicationFactor1/ReplicationFactor

 EndPointSnitchorg.apache.cassandra.locator.EndPointSnitch/EndPointSnitch
 /Keyspace
   /Keyspaces
   CommitLogRotationThresholdInMB32/CommitLogRotationThresholdInMB

   DiskAccessModeauto/DiskAccessMode
   RowWarningThresholdInMB64/RowWarningThresholdInMB
   SlicedBufferSizeInKB64/SlicedBufferSizeInKB
   FlushDataBufferSizeInMB16/FlushDataBufferSizeInMB
   FlushIndexBufferSizeInMB4/FlushIndexBufferSizeInMB
   ColumnIndexSizeInKB64/ColumnIndexSizeInKB

   MemtableThroughputInMB16/MemtableThroughputInMB
   BinaryMemtableThroughputInMB32/BinaryMemtableThroughputInMB
   MemtableOperationsInMillions0.01/MemtableOperationsInMillions
   MemtableObjectCountInMillions0.01/MemtableObjectCountInMillions
   MemtableFlushAfterMinutes60/MemtableFlushAfterMinutes
   ConcurrentReads4/ConcurrentReads
   ConcurrentWrites8/ConcurrentWrites
 /Storage


 [2]
  INFO 13:36:41,062 Super1 has reached its threshold; switching in a fresh
 Memtable at
 CommitLogContext(file='d:/cassandra/commitlog\CommitLog-1271849783703.log',
 position=5417524)
  INFO 13:36:41,062 Enqueuing flush of Memtable(Super1)@15385755
  INFO 13:36:41,062 Writing Memtable(Super1)@15385755
  INFO 13:36:42,062 Completed flushing
 d:\cassandra\data\Keyspace1\Super1-711-Data.db
  INFO 13:36:45,781 Super1 has reached its threshold; switching in a fresh
 Memtable at
 CommitLogContext(file='d:/cassandra/commitlog\CommitLog-1271849783703.log',
 position=6065637)
  INFO 13:36:45,781 Enqueuing flush of Memtable(Super1)@15578910
  INFO 13:36:45,796 Writing Memtable(Super1)@15578910
  INFO 13:36:46,109 Completed flushing
 d:\cassandra\data\Keyspace1\Super1-712-Data.db
  INFO 13:36:54,296 GC for ConcurrentMarkSweep: 7149 ms, 58337240
 reclaimed leaving 922392600 used; max is 1174208512
  INFO 13:36:54,593 Super1 has reached its threshold; switching in a fresh
 Memtable at
 CommitLogContext(file='d:/cassandra/commitlog\CommitLog-1271849783703.log',
 position=6722241)
  INFO 13:36:54,593 Enqueuing flush of Memtable(Super1)@24468872
  INFO 13:36:54,593 Writing Memtable(Super1)@24468872
  INFO 13:36:55,421 Completed

Re: Cassandra tuning for running test on a desktop

[Nicolas Labrot]

Thanks Mark.

Cassandra is maybe too much for my need ;)


On Wed, Apr 21, 2010 at 4:45 PM, Mark Greene green...@gmail.com wrote:

 Hit send to early

 That being said a lot of people running Cassandra in production are using
 4-6GB max heaps on 8GB machines, don't know if that helps but hopefully
 gives you some perspective.


 On Wed, Apr 21, 2010 at 10:39 AM, Mark Greene green...@gmail.com wrote:

 RAM doesn't necessarily need to be proportional but I would say the number
 of nodes does. You can't just throw a bazillion inserts at one node. This is
 the main benefit of Cassandra is that if you start hitting your capacity,
 you add more machines and distribute the keys across more machines.


 On Wed, Apr 21, 2010 at 9:07 AM, Nicolas Labrot nith...@gmail.comwrote:

 So does it means the RAM needed is proportionnal with the data handled ?

 Or Cassandra need a minimum amount or RAM when dataset is big?

 I must confess this OOM behaviour is strange.


 On Wed, Apr 21, 2010 at 2:54 PM, Mark Jones mjo...@imagehawk.comwrote:

  On my 4GB machine I’m giving it 3GB and having no trouble with 60+
 million 500 byte columns



 *From:* Nicolas Labrot [mailto:nith...@gmail.com]
 *Sent:* Wednesday, April 21, 2010 7:47 AM
 *To:* user@cassandra.apache.org
 *Subject:* Re: Cassandra tuning for running test on a desktop



 I have try 1400M, and Cassandra OOM too.

 Is there another solution ? My data isn't very big.

 It seems that is the merge of the db

  On Wed, Apr 21, 2010 at 2:14 PM, Mark Greene green...@gmail.com
 wrote:

 Trying increasing Xmx. 1G is probably not enough for the amount of
 inserts you are doing.



 On Wed, Apr 21, 2010 at 8:10 AM, Nicolas Labrot nith...@gmail.com
 wrote:

 Hello,

 For my first message I will first thanks Cassandra contributors for
 their great works.

 I have a parameter issue with Cassandra (I hope it's just a parameter
 issue). I'm using Cassandra 6.0.1 with Hector client on my desktop. It's a
 simple dual core with 4GB of RAM on WinXP. I have keep the default JVM
 option inside cassandra.bat (Xmx1G)

 I'm trying to insert 3 millions of SC with 6 Columns each inside 1 CF
 (named Super1). The insertion go to 1 millions of SC (without slowdown) and
 Cassandra crash because of an OOM. (I store an average of 100 bytes per SC
 with a max of 10kB).
 I have aggressively decreased all the memories parameters without any
 respect to the consistency (My config is here [1]), the cache is turn off
 but Cassandra still go to OOM. I have joined the last line of the Cassandra
 life [2].

 What can I do to fix my issue ?  Is there another solution than
 increasing the Xmx ?

 Thanks for your help,

 Nicolas





 [1]
   Keyspaces
 Keyspace Name=Keyspace1
   ColumnFamily Name=Super1
 ColumnType=Super
 CompareWith=BytesType
 CompareSubcolumnsWith=BytesType /

 ReplicaPlacementStrategyorg.apache.cassandra.locator.RackUnawareStrategy/ReplicaPlacementStrategy
   ReplicationFactor1/ReplicationFactor

 EndPointSnitchorg.apache.cassandra.locator.EndPointSnitch/EndPointSnitch
 /Keyspace
   /Keyspaces
   CommitLogRotationThresholdInMB32/CommitLogRotationThresholdInMB

   DiskAccessModeauto/DiskAccessMode
   RowWarningThresholdInMB64/RowWarningThresholdInMB
   SlicedBufferSizeInKB64/SlicedBufferSizeInKB
   FlushDataBufferSizeInMB16/FlushDataBufferSizeInMB
   FlushIndexBufferSizeInMB4/FlushIndexBufferSizeInMB
   ColumnIndexSizeInKB64/ColumnIndexSizeInKB

   MemtableThroughputInMB16/MemtableThroughputInMB
   BinaryMemtableThroughputInMB32/BinaryMemtableThroughputInMB
   MemtableOperationsInMillions0.01/MemtableOperationsInMillions
   MemtableObjectCountInMillions0.01/MemtableObjectCountInMillions
   MemtableFlushAfterMinutes60/MemtableFlushAfterMinutes
   ConcurrentReads4/ConcurrentReads
   ConcurrentWrites8/ConcurrentWrites
 /Storage


 [2]
  INFO 13:36:41,062 Super1 has reached its threshold; switching in a
 fresh Memtable at
 CommitLogContext(file='d:/cassandra/commitlog\CommitLog-1271849783703.log',
 position=5417524)
  INFO 13:36:41,062 Enqueuing flush of Memtable(Super1)@15385755
  INFO 13:36:41,062 Writing Memtable(Super1)@15385755
  INFO 13:36:42,062 Completed flushing
 d:\cassandra\data\Keyspace1\Super1-711-Data.db
  INFO 13:36:45,781 Super1 has reached its threshold; switching in a
 fresh Memtable at
 CommitLogContext(file='d:/cassandra/commitlog\CommitLog-1271849783703.log',
 position=6065637)
  INFO 13:36:45,781 Enqueuing flush of Memtable(Super1)@15578910
  INFO 13:36:45,796 Writing Memtable(Super1)@15578910
  INFO 13:36:46,109 Completed flushing
 d:\cassandra\data\Keyspace1\Super1-712-Data.db
  INFO 13:36:54,296 GC for ConcurrentMarkSweep: 7149 ms, 58337240
 reclaimed leaving 922392600 used; max is 1174208512
  INFO 13:36:54,593 Super1 has reached its threshold; switching in a
 fresh Memtable at
 CommitLogContext(file='d:/cassandra/commitlog\CommitLog-1271849783703.log',
 position=6722241)
  INFO 13:36:54,593

Re: Cassandra tuning for running test on a desktop

[Mark Greene]

Maybe, maybe not. Presumably if you are running a RDMS with any reasonable
amount of traffic now a days, it's sitting on a machine with 4-8G of memory
at least.

On Wed, Apr 21, 2010 at 10:48 AM, Nicolas Labrot nith...@gmail.com wrote:

 Thanks Mark.

 Cassandra is maybe too much for my need ;)



 On Wed, Apr 21, 2010 at 4:45 PM, Mark Greene green...@gmail.com wrote:

 Hit send to early

 That being said a lot of people running Cassandra in production are using
 4-6GB max heaps on 8GB machines, don't know if that helps but hopefully
 gives you some perspective.


 On Wed, Apr 21, 2010 at 10:39 AM, Mark Greene green...@gmail.com wrote:

 RAM doesn't necessarily need to be proportional but I would say the
 number of nodes does. You can't just throw a bazillion inserts at one node.
 This is the main benefit of Cassandra is that if you start hitting your
 capacity, you add more machines and distribute the keys across more
 machines.


 On Wed, Apr 21, 2010 at 9:07 AM, Nicolas Labrot nith...@gmail.comwrote:

 So does it means the RAM needed is proportionnal with the data handled ?

 Or Cassandra need a minimum amount or RAM when dataset is big?

 I must confess this OOM behaviour is strange.


 On Wed, Apr 21, 2010 at 2:54 PM, Mark Jones mjo...@imagehawk.comwrote:

  On my 4GB machine I’m giving it 3GB and having no trouble with 60+
 million 500 byte columns



 *From:* Nicolas Labrot [mailto:nith...@gmail.com]
 *Sent:* Wednesday, April 21, 2010 7:47 AM
 *To:* user@cassandra.apache.org
 *Subject:* Re: Cassandra tuning for running test on a desktop



 I have try 1400M, and Cassandra OOM too.

 Is there another solution ? My data isn't very big.

 It seems that is the merge of the db

  On Wed, Apr 21, 2010 at 2:14 PM, Mark Greene green...@gmail.com
 wrote:

 Trying increasing Xmx. 1G is probably not enough for the amount of
 inserts you are doing.



 On Wed, Apr 21, 2010 at 8:10 AM, Nicolas Labrot nith...@gmail.com
 wrote:

 Hello,

 For my first message I will first thanks Cassandra contributors for
 their great works.

 I have a parameter issue with Cassandra (I hope it's just a parameter
 issue). I'm using Cassandra 6.0.1 with Hector client on my desktop. It's a
 simple dual core with 4GB of RAM on WinXP. I have keep the default JVM
 option inside cassandra.bat (Xmx1G)

 I'm trying to insert 3 millions of SC with 6 Columns each inside 1 CF
 (named Super1). The insertion go to 1 millions of SC (without slowdown) 
 and
 Cassandra crash because of an OOM. (I store an average of 100 bytes per SC
 with a max of 10kB).
 I have aggressively decreased all the memories parameters without any
 respect to the consistency (My config is here [1]), the cache is turn off
 but Cassandra still go to OOM. I have joined the last line of the 
 Cassandra
 life [2].

 What can I do to fix my issue ?  Is there another solution than
 increasing the Xmx ?

 Thanks for your help,

 Nicolas





 [1]
   Keyspaces
 Keyspace Name=Keyspace1
   ColumnFamily Name=Super1
 ColumnType=Super
 CompareWith=BytesType
 CompareSubcolumnsWith=BytesType /

 ReplicaPlacementStrategyorg.apache.cassandra.locator.RackUnawareStrategy/ReplicaPlacementStrategy
   ReplicationFactor1/ReplicationFactor

 EndPointSnitchorg.apache.cassandra.locator.EndPointSnitch/EndPointSnitch
 /Keyspace
   /Keyspaces
   CommitLogRotationThresholdInMB32/CommitLogRotationThresholdInMB

   DiskAccessModeauto/DiskAccessMode
   RowWarningThresholdInMB64/RowWarningThresholdInMB
   SlicedBufferSizeInKB64/SlicedBufferSizeInKB
   FlushDataBufferSizeInMB16/FlushDataBufferSizeInMB
   FlushIndexBufferSizeInMB4/FlushIndexBufferSizeInMB
   ColumnIndexSizeInKB64/ColumnIndexSizeInKB

   MemtableThroughputInMB16/MemtableThroughputInMB
   BinaryMemtableThroughputInMB32/BinaryMemtableThroughputInMB
   MemtableOperationsInMillions0.01/MemtableOperationsInMillions
   MemtableObjectCountInMillions0.01/MemtableObjectCountInMillions
   MemtableFlushAfterMinutes60/MemtableFlushAfterMinutes
   ConcurrentReads4/ConcurrentReads
   ConcurrentWrites8/ConcurrentWrites
 /Storage


 [2]
  INFO 13:36:41,062 Super1 has reached its threshold; switching in a
 fresh Memtable at
 CommitLogContext(file='d:/cassandra/commitlog\CommitLog-1271849783703.log',
 position=5417524)
  INFO 13:36:41,062 Enqueuing flush of Memtable(Super1)@15385755
  INFO 13:36:41,062 Writing Memtable(Super1)@15385755
  INFO 13:36:42,062 Completed flushing
 d:\cassandra\data\Keyspace1\Super1-711-Data.db
  INFO 13:36:45,781 Super1 has reached its threshold; switching in a
 fresh Memtable at
 CommitLogContext(file='d:/cassandra/commitlog\CommitLog-1271849783703.log',
 position=6065637)
  INFO 13:36:45,781 Enqueuing flush of Memtable(Super1)@15578910
  INFO 13:36:45,796 Writing Memtable(Super1)@15578910
  INFO 13:36:46,109 Completed flushing
 d:\cassandra\data\Keyspace1\Super1-712-Data.db
  INFO 13:36:54,296 GC for ConcurrentMarkSweep: 7149 ms, 58337240
 reclaimed

Re: Cassandra tuning for running test on a desktop

[Stu Hood]

Nicolas,

Were all of those super column writes going to the same row? 
http://wiki.apache.org/cassandra/CassandraLimitations

Thanks,
Stu

-Original Message-
From: Nicolas Labrot nith...@gmail.com
Sent: Wednesday, April 21, 2010 11:54am
To: user@cassandra.apache.org
Subject: Re: Cassandra tuning for running test on a desktop

I donnot have a website ;)

I'm testing the viability of Cassandra to store XML documents and make fast
search queries. 4000 XML files (80MB of XML) create with my datamodel (one
SC per XML node) 100 SC which make Cassandra go OOM with Xmx 1GB. On the
contrary an xml DB like eXist handles 4000 XML doc without any problem with
an acceptable amount of memories.

What I like with Cassandra is his simplicity and his scalability. eXist is
not able to scale with data, the only viable solution his marklogic which
cost an harm and a feet... :)

I will install linux and buy some memories to continue my test.

Could a Cassandra developper give me the technical reason of this OOM ?





On Wed, Apr 21, 2010 at 5:13 PM, Mark Greene green...@gmail.com wrote:

 Maybe, maybe not. Presumably if you are running a RDMS with any reasonable
 amount of traffic now a days, it's sitting on a machine with 4-8G of memory
 at least.


 On Wed, Apr 21, 2010 at 10:48 AM, Nicolas Labrot nith...@gmail.comwrote:

 Thanks Mark.

 Cassandra is maybe too much for my need ;)



 On Wed, Apr 21, 2010 at 4:45 PM, Mark Greene green...@gmail.com wrote:

 Hit send to early

 That being said a lot of people running Cassandra in production are using
 4-6GB max heaps on 8GB machines, don't know if that helps but hopefully
 gives you some perspective.


 On Wed, Apr 21, 2010 at 10:39 AM, Mark Greene green...@gmail.comwrote:

 RAM doesn't necessarily need to be proportional but I would say the
 number of nodes does. You can't just throw a bazillion inserts at one node.
 This is the main benefit of Cassandra is that if you start hitting your
 capacity, you add more machines and distribute the keys across more
 machines.


 On Wed, Apr 21, 2010 at 9:07 AM, Nicolas Labrot nith...@gmail.comwrote:

 So does it means the RAM needed is proportionnal with the data handled
 ?

 Or Cassandra need a minimum amount or RAM when dataset is big?

 I must confess this OOM behaviour is strange.


 On Wed, Apr 21, 2010 at 2:54 PM, Mark Jones mjo...@imagehawk.comwrote:

  On my 4GB machine I’m giving it 3GB and having no trouble with 60+
 million 500 byte columns



 *From:* Nicolas Labrot [mailto:nith...@gmail.com]
 *Sent:* Wednesday, April 21, 2010 7:47 AM
 *To:* user@cassandra.apache.org
 *Subject:* Re: Cassandra tuning for running test on a desktop



 I have try 1400M, and Cassandra OOM too.

 Is there another solution ? My data isn't very big.

 It seems that is the merge of the db

  On Wed, Apr 21, 2010 at 2:14 PM, Mark Greene green...@gmail.com
 wrote:

 Trying increasing Xmx. 1G is probably not enough for the amount of
 inserts you are doing.



 On Wed, Apr 21, 2010 at 8:10 AM, Nicolas Labrot nith...@gmail.com
 wrote:

 Hello,

 For my first message I will first thanks Cassandra contributors for
 their great works.

 I have a parameter issue with Cassandra (I hope it's just a parameter
 issue). I'm using Cassandra 6.0.1 with Hector client on my desktop. It's 
 a
 simple dual core with 4GB of RAM on WinXP. I have keep the default JVM
 option inside cassandra.bat (Xmx1G)

 I'm trying to insert 3 millions of SC with 6 Columns each inside 1 CF
 (named Super1). The insertion go to 1 millions of SC (without slowdown) 
 and
 Cassandra crash because of an OOM. (I store an average of 100 bytes per 
 SC
 with a max of 10kB).
 I have aggressively decreased all the memories parameters without any
 respect to the consistency (My config is here [1]), the cache is turn off
 but Cassandra still go to OOM. I have joined the last line of the 
 Cassandra
 life [2].

 What can I do to fix my issue ?  Is there another solution than
 increasing the Xmx ?

 Thanks for your help,

 Nicolas





 [1]
   Keyspaces
 Keyspace Name=Keyspace1
   ColumnFamily Name=Super1
 ColumnType=Super
 CompareWith=BytesType
 CompareSubcolumnsWith=BytesType /

 ReplicaPlacementStrategyorg.apache.cassandra.locator.RackUnawareStrategy/ReplicaPlacementStrategy
   ReplicationFactor1/ReplicationFactor

 EndPointSnitchorg.apache.cassandra.locator.EndPointSnitch/EndPointSnitch
 /Keyspace
   /Keyspaces
   CommitLogRotationThresholdInMB32/CommitLogRotationThresholdInMB

   DiskAccessModeauto/DiskAccessMode
   RowWarningThresholdInMB64/RowWarningThresholdInMB
   SlicedBufferSizeInKB64/SlicedBufferSizeInKB
   FlushDataBufferSizeInMB16/FlushDataBufferSizeInMB
   FlushIndexBufferSizeInMB4/FlushIndexBufferSizeInMB
   ColumnIndexSizeInKB64/ColumnIndexSizeInKB

   MemtableThroughputInMB16/MemtableThroughputInMB
   BinaryMemtableThroughputInMB32/BinaryMemtableThroughputInMB