RE: Unable to start one Cassandra node: OutOfMemoryError

[Walsh, Stephen]

Glad to help :P


From: Mikhail Strebkov [mailto:streb...@gmail.com]
Sent: 10 December 2015 22:35
To: user@cassandra.apache.org
Subject: Re: Unable to start one Cassandra node: OutOfMemoryError

Steve, thanks a ton! Removing compactions_in_progress helped! Now the node is 
running again.

p.s. Sorry for referring to you by the last name in my last email, I got 
confused.

On Thu, Dec 10, 2015 at 2:09 AM, Walsh, Stephen 
> wrote:
8GB is the max recommended for heap size and that’s if you have 32GB or more 
available.

We use 6GB on our 16GB machines and its very stable

The out of memory could be coming from cassandra reloading 
compactions_in_progress into memory, you can check this from the log files if 
needs be.
You can safely delete this folder inside the data directory.

This can happen if you didn’t stop cassandra with a drain command and wait for 
the compactions to finish.
Last time we hit it – was due to testing HA when we forced killed an entire 
cluster.

Steve



From: Jeff Jirsa 
[mailto:jeff.ji...@crowdstrike.com]
Sent: 10 December 2015 02:49
To: user@cassandra.apache.org
Subject: Re: Unable to start one Cassandra node: OutOfMemoryError

8G is probably too small for a G1 heap. Raise your heap or try CMS instead.

71% of your heap is collections – may be a weird data model quirk, but try CMS 
first and see if that behaves better.



From: Mikhail Strebkov
Reply-To: "user@cassandra.apache.org"
Date: Wednesday, December 9, 2015 at 5:26 PM
To: "user@cassandra.apache.org"
Subject: Unable to start one Cassandra node: OutOfMemoryError

Hi everyone,

While upgrading our 5 machines cluster from DSE version 4.7.1 (Cassandra 2.1.8) 
to DSE version: 4.8.2 (Cassandra 2.1.11)  one of the nodes can't start with 
OutOfMemoryError.

We're using HotSpot 64-Bit Server VM/1.8.0_45 and G1 garbage collector with 8 
GiB heap.

Average node size is 300 GiB.

I looked at the heap dump with YourKit profiler 
(www.yourkit.com) and it was quite hard since it's so 
big, but can't get much out of it: http://i.imgur.com/fIRImma.png

As far as I understand the report, there are 1,332,812 instances of 
org.apache.cassandra.db.Row which retain 8 GiB. I don't understand why all of 
them are still strongly reachable?

Please help me to debug this. I don't know even where to start.
I feel very uncomfortable with 1 node running 4.8.2, 1 node down and 3 nodes 
running 4.7.1 at the same time.

Thanks,
Mikhail


This email (including any attachments) is proprietary to Aspect Software, Inc. 
and may contain information that is confidential. If you have received this 
message in error, please do not read, copy or forward this message. Please 
notify the sender immediately, delete it from your system and destroy any 
copies. You may not further disclose or distribute this email or its 
attachments.

This email (including any attachments) is proprietary to Aspect Software, Inc. 
and may contain information that is confidential. If you have received this 
message in error, please do not read, copy or forward this message. Please 
notify the sender immediately, delete it from your system and destroy any 
copies. You may not further disclose or distribute this email or its 
attachments.

can't make any permissions change in 2.2.4

[Kai Wang]

I used to able to add/drop users and modify permissions in 2.1.1. After
upgrading to 2.2.4, I can't modify any of those. "List all permissions"
returns me all the permissions I setup before the upgrade. But I can't add
new permission or add new users in cqlsh. "create user" and "grant" didn't
report any errors.

Re: C# CQL Driver - Each Session maintains its own HostConnectionPool and Host connections

[Carl Yeksigian]

Hi Soto,

You are going to have more luck using the C# driver list instead of the
cassandra-user list.

https://groups.google.com/a/lists.datastax.com/forum/#!forum/csharp-driver-user

-Carl

On Thu, Dec 17, 2015 at 3:01 PM, Sotirios Delimanolis 
wrote:

> Hey,
>
> I wanted to ask here before I opened an issue for the driver.
>
> I'm using version 2.7.3 of the driver.
>
> The PoolingOptions
> 
>  define
> core connections and max connections that limit the number of connections
> that should be opened to each host.
>
> However, each Session object you retrieve through Cluster.Connect maintains
> its own ConcurrentDictionary of HostConnectionPools. In our application,
> we have 3 keyspaces. We perform 3 Cluster.Connect calls. When we
> eventually start using the corresponding Session objects, we eventually
> end up with three times the number of actual connections to each Cassandra
> host.
>
> Is this intended? Shouldn't the Session objects share the same dictionary
> or HostConnectionPools?
>
> The Host objects are Cluster-scoped singletons. This has the effect that
> each HostConnectionPool object you create registers an additional
> OnHostDown and OnHostUp event with the corresponding Host object. Again,
> this doesn't seem right.
>
> Please clarify if this is as it should be.
>
> Thanks,
> Soto
>
>

C# CQL Driver - Each Session maintains its own HostConnectionPool and Host connections

[Sotirios Delimanolis]

Hey,
I wanted to ask here before I opened an issue for the driver.
I'm using version 2.7.3 of the driver.
The PoolingOptions define core connections and max connections that limit the 
number of connections that should be opened to each host.
However, each Session object you retrieve through Cluster.Connect maintains its 
own ConcurrentDictionary of HostConnectionPools. In our application, we have 3 
keyspaces. We perform 3 Cluster.Connect calls. When we eventually start using 
the corresponding Session objects, we eventually end up with three times the 
number of actual connections to each Cassandra host.
Is this intended? Shouldn't the Session objects share the same dictionary or 
HostConnectionPools?
The Host objects are Cluster-scoped singletons. This has the effect that each 
HostConnectionPool object you create registers an additional OnHostDown and 
OnHostUp event with the corresponding Host object. Again, this doesn't seem 
right.
Please clarify if this is as it should be.
Thanks,Soto

Re: can't make any permissions change in 2.2.4

[Robert Coli]

On Thu, Dec 17, 2015 at 9:10 AM, Kai Wang  wrote:

> I used to able to add/drop users and modify permissions in 2.1.1. After
> upgrading to 2.2.4, I can't modify any of those. "List all permissions"
> returns me all the permissions I setup before the upgrade. But I can't add
> new permission or add new users in cqlsh. "create user" and "grant" didn't
> report any errors.
>

I would :

1) report the issue to the cassandra bug tracker at issues.apache.org
2) reply to this mail letting the list know the URL of your bug

=Rob

Re: C# CQL Driver - Each Session maintains its own HostConnectionPool and Host connections

[Sotirios Delimanolis]

Cool, thanks. I'll repost there. 

On Thursday, December 17, 2015 12:20 PM, Carl Yeksigian 
 wrote:
 

 Hi Soto,

You are going to have more luck using the C# driver list instead of the 
cassandra-user list.

https://groups.google.com/a/lists.datastax.com/forum/#!forum/csharp-driver-user

-Carl

On Thu, Dec 17, 2015 at 3:01 PM, Sotirios Delimanolis  
wrote:

Hey,
I wanted to ask here before I opened an issue for the driver.
I'm using version 2.7.3 of the driver.
The PoolingOptions define core connections and max connections that limit the 
number of connections that should be opened to each host.
However, each Session object you retrieve through Cluster.Connect maintains its 
own ConcurrentDictionary of HostConnectionPools. In our application, we have 3 
keyspaces. We perform 3 Cluster.Connect calls. When we eventually start using 
the corresponding Session objects, we eventually end up with three times the 
number of actual connections to each Cassandra host.
Is this intended? Shouldn't the Session objects share the same dictionary or 
HostConnectionPools?
The Host objects are Cluster-scoped singletons. This has the effect that each 
HostConnectionPool object you create registers an additional OnHostDown and 
OnHostUp event with the corresponding Host object. Again, this doesn't seem 
right.
Please clarify if this is as it should be.
Thanks,Soto




  

Re: [Marketing Mail] can't make any permissions change in 2.2.4

[Reynald Bourtembourg]

Hi,

Maybe your problem comes from the new role based access control in 
Cassandra introduced in Cassandra 2.2.


http://www.datastax.com/dev/blog/role-based-access-control-in-cassandra

The /Upgrading /section of this blog post is specifying the following:

"/For systems already using the internal auth implementations, the 
process for converting existing data during a rolling upgrade is 
straightforward. As each node is restarted, it will attempt to convert 
any data in the legacy tables into the new schema. Until enough nodes to 
satisfy the replication strategy for the //|system_auth|//keyspace are 
upgraded and so have the new schema, this conversion will fail with the 
failure being reported in the system log. During the upgrade, 
Cassandra’s internal auth classes will continue to use the legacy 
tables, so clients experience no disruption. Issuing //DCL 
//statements during 
an upgrade is not supported. Once all nodes are upgraded, an operator 
with superuser privileges should drop the legacy tables, which will 
prompt Cassandra to switch over to the new tables without requiring any 
further intervention./


//

/As I mentioned, this is just a part of a wider reworking of the auth 
subsystem in Cassandra planned for inclusion in the 2.2 release. You can 
check out more detail and follow progress in //CASSANDRA-8394 
//./"


This does not seem to be explained in the Cassandra 2.2 Upgrading 
section of the NEWS.txt file:


https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-2.2.4

Hoping this helps,
Reynald

On 17/12/2015 18:10, Kai Wang wrote:
I used to able to add/drop users and modify permissions in 2.1.1. 
After upgrading to 2.2.4, I can't modify any of those. "List all 
permissions" returns me all the permissions I setup before the 
upgrade. But I can't add new permission or add new users in cqlsh. 
"create user" and "grant" didn't report any errors.

Re: Would data be lost by nodetool removenode force

[Shuo Chen]

The version is 2.0.7

Why we want to add a new node is that somebody creates an index on certain
table and the cluster becomes slow, so that guy tries to restart the node.
However  only 3 out of 4 nodes can restart and 1 node cannot restart . So
we intend to add a new node however it shows error that cannot find certain
cf-id and cannot be started.

So we decided to remove this newly added node by nodetool removenode. Here
is the thing... We have already started remove this newly added node. Only
a few data (80kb) has been migrated to this newly added node. The process
of removenode seems hanging

So how to resolve this situation, can we add another node? or using remove
force to this newly added node?

On Fri, Dec 18, 2015 at 9:09 AM, Robert Coli  wrote:

> On Thu, Dec 17, 2015 at 4:44 PM, Shuo Chen  wrote:
>
>>I have a 4 node cluster with status 3 UN and 1 DN. I am trying to add
>> a new node into the cluster but it is also dead. So the cluster is now 3 UN
>> and 2 DN. However I didnot run nodetool cleanup on any nodes. And just
>> several KBs of data is migrated to this newly added node.
>>
>> All the keyspaces are replicated for 3. I am trying to remove this newly
>> add node using nodetool removenode. But it hangs for 24 hours with remove
>> node status.
>>
>> RemovalStatus: Removing token (-9104065154580789913). Waiting for
>> replication confirmation from [/192.168.148.29,/192.168.148.24,/
>> 192.168.148.23].
>>
>> Is it safe to use removenode force to remove this newly added node? Will
>> any data be lost? Thanks!
>>
>
> In general it is not "safe" to use "removenode" to remove a node from a
> cluster, because removenode by definition reduces distinct replica count.
>
> That's why a departing node should always use "decommission," so it
> streams the data it has, including any data it might be the only node it
> has, to the new nodes responsible for the range.
>
> Your email seems to prompt questions It doesn't seem like your "real"
> question is about removenode at all?
>
> Why, for example, is your new node you are adding into the cluster "dead"?
>
> Are you trying to replace the original DN with your new node? If so, use
> "replace_address"?
>
> =Rob
> PS - also, what version of cassandra?
>



-- 
*陈硕* *Shuo Chen*
chenatu2...@gmail.com
chens...@whaty.com

Would data be lost by nodetool removenode force

[Shuo Chen]

Hi,
   I have a 4 node cluster with status 3 UN and 1 DN. I am trying to add a
new node into the cluster but it is also dead. So the cluster is now 3 UN
and 2 DN. However I didnot run nodetool cleanup on any nodes. And just
several KBs of data is migrated to this newly added node.

All the keyspaces are replicated for 3. I am trying to remove this newly
add node using nodetool removenode. But it hangs for 24 hours with remove
node status.

RemovalStatus: Removing token (-9104065154580789913). Waiting for
replication confirmation from [/192.168.148.29,/192.168.148.24,/
192.168.148.23].

Is it safe to use removenode force to remove this newly added node? Will
any data be lost? Thanks!
-- 
*陈硕* *Shuo Chen*
chenatu2...@gmail.com
chens...@whaty.com

Re: Would data be lost by nodetool removenode force

[Robert Coli]

On Thu, Dec 17, 2015 at 4:44 PM, Shuo Chen  wrote:

>I have a 4 node cluster with status 3 UN and 1 DN. I am trying to add a
> new node into the cluster but it is also dead. So the cluster is now 3 UN
> and 2 DN. However I didnot run nodetool cleanup on any nodes. And just
> several KBs of data is migrated to this newly added node.
>
> All the keyspaces are replicated for 3. I am trying to remove this newly
> add node using nodetool removenode. But it hangs for 24 hours with remove
> node status.
>
> RemovalStatus: Removing token (-9104065154580789913). Waiting for
> replication confirmation from [/192.168.148.29,/192.168.148.24,/
> 192.168.148.23].
>
> Is it safe to use removenode force to remove this newly added node? Will
> any data be lost? Thanks!
>

In general it is not "safe" to use "removenode" to remove a node from a
cluster, because removenode by definition reduces distinct replica count.

That's why a departing node should always use "decommission," so it streams
the data it has, including any data it might be the only node it has, to
the new nodes responsible for the range.

Your email seems to prompt questions It doesn't seem like your "real"
question is about removenode at all?

Why, for example, is your new node you are adding into the cluster "dead"?

Are you trying to replace the original DN with your new node? If so, use
"replace_address"?

=Rob
PS - also, what version of cassandra?