Re: Convert single node C* to cluster (rebalancing problem)

[Junaid Nasir]

sorry that was an old notetool status output (old cluster with a few
configuration changes). new configuration is

--  Address  Load   Tokens   Owns (effective)  Host ID
  Rack
UN  10.128.1.2   270.75 GiB  256  52.6%
e4427611-c247-42ee-9404-371e177f5f17  rack1
UN  10.128.1.10  388.52 KiB  256  47.4%
7059e622-7861-495b-a69f-5a68876364dd  rack1

10.128.1.2 is seed node, and no other C* node is running on the network.
data_file_directories on seed node is not in default location, it's on
another disk. if that helps

On Thu, Jun 1, 2017 at 4:49 AM, Akhil Mehra  wrote:

> Hi Junaid,
>
> I noticed in the log files that data is being streamed from/to 10.128.1.2
> but the address of your original node is 10.128.0.7.
>
> Are there any other Cassandra nodes on your local network the same cluster
> name.
>
> Regards,
> Akhil
>
> > 10.128.1.2
> > On 31/05/2017, at 10:15 PM, Junaid Nasir  wrote:
> >
> > after nodetool repair on new node following log appears in cassandra log
> >
> > INFO  [StreamConnectionEstablisher:1] 2017-05-31 10:07:23,496
> StreamSession.java:266 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac]
> Starting streaming to /10.128.1.2
> > INFO  [StreamConnectionEstablisher:1] 2017-05-31 10:07:23,501
> StreamCoordinator.java:264 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac,
> ID#0] Beginning stream session with /10.128.1.2
> > INFO  [STREAM-IN-/10.128.1.2:7000] 2017-05-31 10:07:23,545
> StreamResultFuture.java:173 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac
> ID#0] Prepare completed. Receiving 2 files(131.594KiB), sending 0
> files(0.000KiB)
> > INFO  [StreamReceiveTask:1] 2017-05-31 10:07:23,705
> StreamResultFuture.java:187 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac]
> Session with /10.128.1.2 is complete
> > INFO  [StreamReceiveTask:1] 2017-05-31 10:07:23,707
> StreamResultFuture.java:219 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac]
> All sessions completed
> >
> >
> >
> > On Wed, May 31, 2017 at 2:50 PM, Junaid Nasir  wrote:
> > Cassandra version is 3.10, and yes its not a production server. i have
> seen some warnings in logs saying token exist on both servers. other than
> that nothing. if you need any more settings/details please ask. thank you
> for your time
> >
> >
> > On Wed, May 31, 2017 at 2:04 PM, Oleksandr Shulgin <
> oleksandr.shul...@zalando.de> wrote:
> > On Wed, May 31, 2017 at 9:55 AM, Junaid Nasir  wrote:
> > Cassandra ensure that adding or removing nodes are very easy and that
> load is balanced between nodes when a change is made. but it's not working
> in my case.
> > I have a single node C* deployment (with 270 GB of data) and want to
> load balance the data on multiple nodes,
> >
> > I guess it's fair to assume this is not a production "cluster"?
> >
> > I followed this guide
> > `nodetool status` shows 2 nodes but load is not balanced between them
> > Datacenter: dc1
> > ===
> > Status=Up/Down
> > |/ State=Normal/Leaving/Joining/M
> > oving
> > --  Address  Load   Tokens   Owns (effective)  Host ID
> Rack
> > UN  10.128.0.7   270.75 GiB  256  48.6%
> 1a3f6faa-4376-45a8-9c20-11480a
> > e5664c  rack1
> > UN  10.128.0.14  414.36 KiB  256  51.4%
> 66a89fbf-08ba-4b5d-9f10-55d52a
> > 199b41  rack1
> > I can imagine you are referring to Load column here, which shows only
> 400 KB for the new node.  Did the newly added node actually bootstrap?  Are
> there any error/warning messages in the Cassandra log?
> >
> > I also ran 'nodetool repair' on new node but result is same. any
> pointers would be appreciated :)
> >
> > Hm, this is not expected.  Even if the node didn't bootstrap, repair
> should have streamed the data it is responsible for.
> >
> > conf file of new node
> > cluster_name: 'cluster1'
> >  - seeds: "10.128.0.7"
> >
> > num_tokens: 256
> > endpoint_snitch: GossipingPropertyFileSnitch
> >
> > I see that you've tried to add the new node as a normal (non-seed) node,
> which supposed to work and it should have bootstrapped itself.  Which
> version of Cassandra is it exactly?
> >
> > --
> > Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176
> 127-59-707
> >
> >
> >
>
>

Re: Restarting nodes and reported load

[Anthony Grasso]

Hi Daniel,

When you say that the nodes have to be restarted, are you just restarting
the Cassandra service or are you restarting the machine?
How are you reclaiming disk space at the moment? Does disk space free up
after the restart?

Regarding storage on nodes, keep in mind the more data stored on a node,
the longer some operations to maintain that data will take to complete. In
addition, the more data that is on each node, the long it will take to
stream data to other nodes. Whether it is replacing a down node or
inserting a new node, having a large amount of data on each node will mean
that it takes longer for a node to join the cluster if it is streaming the
data.

Kind regards,
Anthony

On 30 May 2017 at 02:43, Daniel Steuernol  wrote:

> The cluster is running with RF=3, right now each node is storing about 3-4
> TB of data. I'm using r4.2xlarge EC2 instances, these have 8 vCPU's, 61 GB
> of RAM, and the disks attached for the data drive are gp2 ssd ebs volumes
> with 10k iops. I guess this brings up the question of what's a good marker
> to decide on whether to increase disk space vs provisioning a new node?
>
>
>
> On May 29 2017, at 9:35 am, tommaso barbugli  wrote:
>
>> Hi Daniel,
>>
>> This is not normal. Possibly a capacity problem. Whats the RF, how much
>> data do you store per node and what kind of servers do you use (core count,
>> RAM, disk, ...)?
>>
>> Cheers,
>> Tommaso
>>
>> On Mon, May 29, 2017 at 6:22 PM, Daniel Steuernol 
>> wrote:
>>
>>
>> I am running a 6 node cluster, and I have noticed that the reported load
>> on each node rises throughout the week and grows way past the actual disk
>> space used and available on each node. Also eventually latency for
>> operations suffers and the nodes have to be restarted. A couple questions
>> on this, is this normal? Also does cassandra need to be restarted every few
>> days for best performance? Any insight on this behaviour would be helpful.
>>
>> Cheers,
>> Daniel
>> - To
>> unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For
>> additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>
>> - To
> unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional
> commands, e-mail: user-h...@cassandra.apache.org
>

Re: Convert single node C* to cluster (rebalancing problem)

[Akhil Mehra]

Hi Junaid,

I noticed in the log files that data is being streamed from/to 10.128.1.2 but 
the address of your original node is 10.128.0.7.

Are there any other Cassandra nodes on your local network the same cluster 
name. 

Regards,
Akhil

> 10.128.1.2
> On 31/05/2017, at 10:15 PM, Junaid Nasir  wrote:
> 
> after nodetool repair on new node following log appears in cassandra log
> 
> INFO  [StreamConnectionEstablisher:1] 2017-05-31 10:07:23,496 
> StreamSession.java:266 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac] 
> Starting streaming to /10.128.1.2
> INFO  [StreamConnectionEstablisher:1] 2017-05-31 10:07:23,501 
> StreamCoordinator.java:264 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac, 
> ID#0] Beginning stream session with /10.128.1.2
> INFO  [STREAM-IN-/10.128.1.2:7000] 2017-05-31 10:07:23,545 
> StreamResultFuture.java:173 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac 
> ID#0] Prepare completed. Receiving 2 files(131.594KiB), sending 0 
> files(0.000KiB)
> INFO  [StreamReceiveTask:1] 2017-05-31 10:07:23,705 
> StreamResultFuture.java:187 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac] 
> Session with /10.128.1.2 is complete
> INFO  [StreamReceiveTask:1] 2017-05-31 10:07:23,707 
> StreamResultFuture.java:219 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac] 
> All sessions completed
> 
> 
> 
> On Wed, May 31, 2017 at 2:50 PM, Junaid Nasir  wrote:
> Cassandra version is 3.10, and yes its not a production server. i have seen 
> some warnings in logs saying token exist on both servers. other than that 
> nothing. if you need any more settings/details please ask. thank you for your 
> time 
> 
> 
> On Wed, May 31, 2017 at 2:04 PM, Oleksandr Shulgin 
>  wrote:
> On Wed, May 31, 2017 at 9:55 AM, Junaid Nasir  wrote:
> Cassandra ensure that adding or removing nodes are very easy and that load is 
> balanced between nodes when a change is made. but it's not working in my case.
> I have a single node C* deployment (with 270 GB of data) and want to load 
> balance the data on multiple nodes,
> 
> I guess it's fair to assume this is not a production "cluster"?
>  
> I followed this guide 
> `nodetool status` shows 2 nodes but load is not balanced between them
> Datacenter: dc1
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/M
> oving
> --  Address  Load   Tokens   Owns (effective)  Host IDRack
> UN  10.128.0.7   270.75 GiB  256  48.6%
> 1a3f6faa-4376-45a8-9c20-11480a
> e5664c  rack1
> UN  10.128.0.14  414.36 KiB  256  51.4%
> 66a89fbf-08ba-4b5d-9f10-55d52a
> 199b41  rack1
> I can imagine you are referring to Load column here, which shows only 400 KB 
> for the new node.  Did the newly added node actually bootstrap?  Are there 
> any error/warning messages in the Cassandra log?
>  
> I also ran 'nodetool repair' on new node but result is same. any pointers 
> would be appreciated :)
> 
> Hm, this is not expected.  Even if the node didn't bootstrap, repair should 
> have streamed the data it is responsible for.
> 
> conf file of new node
> cluster_name: 'cluster1'
>  - seeds: "10.128.0.7"
> 
> num_tokens: 256
> endpoint_snitch: GossipingPropertyFileSnitch
> 
> I see that you've tried to add the new node as a normal (non-seed) node, 
> which supposed to work and it should have bootstrapped itself.  Which version 
> of Cassandra is it exactly?
> 
> -- 
> Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176 
> 127-59-707
> 
> 
> 


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: How to avoid flush if the data can fit into memtable

[Kevin O'Connor]

Great post Akhil! Thanks for explaining that.

On Mon, May 29, 2017 at 5:43 PM, Akhil Mehra  wrote:

> Hi Preetika,
>
> After thinking about your scenario I believe your small SSTable size might
> be due to data compression. By default, all tables enable SSTable
> compression.
>
> Let go through your scenario. Let's say you have allocated 4GB to your
> Cassandra node. Your *memtable_heap_space_in_mb* and
>
> *memtable_offheap_space_in_mb  *will roughly come to around 1GB. Since
> you have memtable_cleanup_threshold to .50 table cleanup will be
> triggered when total allocated memtable space exceeds 1/2GB. Note the
> cleanup threshold is .50 of 1GB and not a combination of heap and off heap
> space. This memtable allocation size is the total amount available for all
> tables on your node. This includes all system related keyspaces. The
> cleanup process will write the largest memtable to disk.
>
> For your case, I am assuming that you are on a *single node with only one
> table with insert activity*. I do not think the commit log will trigger a
> flush in this circumstance as by default the commit log has 8192 MB of
> space unless the commit log is placed on a very small disk.
>
> I am assuming your table on disk is smaller than 500MB because of
> compression. You can disable compression on your table and see if this
> helps get the desired size.
>
> I have written up a blog post explaining memtable flushing (
> http://abiasforaction.net/apache-cassandra-memtable-flush/)
>
> Let me know if you have any other question.
>
> I hope this helps.
>
> Regards,
> Akhil Mehra
>
>
> On Fri, May 26, 2017 at 6:58 AM, preetika tyagi 
> wrote:
>
>> I agree that for such a small data, Cassandra is obviously not needed.
>> However, this is purely an experimental setup by using which I'm trying to
>> understand how and exactly when memtable flush is triggered. As I mentioned
>> in my post, I read the documentation and tweaked the parameters accordingly
>> so that I never hit memtable flush but it is still doing that. As far the
>> the setup is concerned, I'm just using 1 node and running Cassandra using
>> "cassandra -R" option and then running some queries to insert some dummy
>> data.
>>
>> I use the schema from CASSANDRA_HOME/tools/cqlstress-insanity-example.yaml
>> and add "durable_writes=false" in the keyspace_definition.
>>
>> @Daemeon - The previous post lead to this post but since I was unaware of
>> memtable flush and I assumed memtable flush wasn't happening, the previous
>> post was about something else (throughput/latency etc.). This post is
>> explicitly about exactly when memtable is being dumped to the disk. Didn't
>> want to confuse two different goals that's why posted a new one.
>>
>> On Thu, May 25, 2017 at 10:38 AM, Avi Kivity  wrote:
>>
>>> It doesn't have to fit in memory. If your key distribution has strong
>>> temporal locality, then a larger memtable that can coalesce overwrites
>>> greatly reduces the disk I/O load for the memtable flush and subsequent
>>> compactions. Of course, I have no idea if the is what the OP had in mind.
>>>
>>>
>>> On 05/25/2017 07:14 PM, Jonathan Haddad wrote:
>>>
>>> Sorry for the confusion.  That was for the OP.  I wrote it quickly right
>>> after waking up.
>>>
>>> What I'm asking is why does the OP want to keep his data in the memtable
>>> exclusively?  If the goal is to "make reads fast", then just turn on row
>>> caching.
>>>
>>> If there's so little data that it fits in memory (300MB), and there
>>> aren't going to be any writes past the initial small dataset, why use
>>> Cassandra?  It sounds like the wrong tool for this job.  Sounds like
>>> something that could easily be stored in S3 and loaded in memory when the
>>> app is fired up.
>>>
>>> On Thu, May 25, 2017 at 8:06 AM Avi Kivity  wrote:
>>>
 Not sure whether you're asking me or the original poster, but the more
 times data gets overwritten in a memtable, the less it has to be compacted
 later on (and even without overwrites, larger memtables result in less
 compaction).

 On 05/25/2017 05:59 PM, Jonathan Haddad wrote:

 Why do you think keeping your data in the memtable is a what you need
 to do?
 On Thu, May 25, 2017 at 7:16 AM Avi Kivity  wrote:

> Then it doesn't have to (it still may, for other reasons).
>
> On 05/25/2017 05:11 PM, preetika tyagi wrote:
>
> What if the commit log is disabled?
>
> On May 25, 2017 4:31 AM, "Avi Kivity"  wrote:
>
>> Cassandra has to flush the memtable occasionally, or the commit log
>> grows without bounds.
>>
>> On 05/25/2017 03:42 AM, preetika tyagi wrote:
>>
>> Hi,
>>
>> I'm running Cassandra with a very small dataset so that the data can
>> exist on memtable only. Below are my configurations:
>>
>> In jvm.options:
>>
>> 

RE: Stable version apache cassandra 3.X /3.0.X

[Mark Furlong]

I need to reduce my disk footprint, how is 3.0.14 for stability? Also, where do 
I find upgrade instructions and version requirements?

Thanks
Mark
801-705-7115 office

From: Carlos Rolo [mailto:r...@pythian.com]
Sent: Wednesday, May 31, 2017 11:17 AM
To: Jonathan Haddad 
Cc: Junaid Nasir ; pabbireddy avinash 
; user@cassandra.apache.org
Subject: Re: Stable version apache cassandra 3.X /3.0.X

On sync in Jon.

Only go 3.0.x if you REALLY need something from there (ex: MV) even then, be 
carefull.

3.x wait for 3.11.x. 3.10 if you REALLY need something from there right now.

Latest 2.2.x or 2.1.x if you are just doing baseline Cassandra and need the 
stability.

Regards,

Carlos Juzarte Rolo
Cassandra Consultant / Datastax Certified Architect / Cassandra MVP

Pythian - Love your data

rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin: 
linkedin.com/in/carlosjuzarterolo

Mobile: +351 918 918 100
www.pythian.com

On Wed, May 31, 2017 at 5:48 PM, Jonathan Haddad 
> wrote:
I really wouldn't go by the tick tock blog post, considering tick tock is dead.

I'm still not wild about putting any 3.0 or 3.x into production.  3.0 removed 
off heap memtables and there have been enough bugs in the storage engine that 
I'm still wary.  My hope is to see 3.11.x get enough bug fixes to where most 
people just skip 3.0 altogether.  I'm not sure if we're there yet though.


On Wed, May 31, 2017 at 9:43 AM Junaid Nasir 
> wrote:
as mentioned here http://www.datastax.com/dev/blog/cassandra-2-2-3-0-and-beyond
Under normal conditions, we will NOT release 3.x.y stability releases for x > 
0.  That is, we will have a traditional 3.0.y stability series, but the 
odd-numbered bugfix-only releases will fill that role for the tick-tock series 
— recognizing that occasionally we will need to be flexible enough to release 
an emergency fix in the case of a critical bug or security vulnerability.
We do recognize that it will take some time for tick-tock releases to deliver 
production-level stability, which is why we will continue to deliver 2.2.y and 
3.0.y bugfix releases.  (But if we do demonstrate that tick-tock can deliver 
the stability we want, there will be no need for a 4.0.y bugfix series, only 
4.x tick-tock.)

On Wed, May 31, 2017 at 9:02 PM, pabbireddy avinash 
> wrote:
Hi,

We are planning to deploy a cassandra production cluster on 3.X /3.0.X . Please 
let us know if there is any stable version  in 3.X/3.0.X that we could deploy 
in production .

Regards,
Avinash.




--

Re: Stable version apache cassandra 3.X /3.0.X

[Carlos Rolo]

On sync in Jon.

Only go 3.0.x if you REALLY need something from there (ex: MV) even then,
be carefull.

3.x wait for 3.11.x. 3.10 if you REALLY need something from there right now.

Latest 2.2.x or 2.1.x if you are just doing baseline Cassandra and need the
stability.

Regards,

Carlos Juzarte Rolo
Cassandra Consultant / Datastax Certified Architect / Cassandra MVP

Pythian - Love your data

rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
*linkedin.com/in/carlosjuzarterolo
*
Mobile: +351 918 918 100
www.pythian.com

On Wed, May 31, 2017 at 5:48 PM, Jonathan Haddad  wrote:

> I really wouldn't go by the tick tock blog post, considering tick tock is
> dead.
>
> I'm still not wild about putting any 3.0 or 3.x into production.  3.0
> removed off heap memtables and there have been enough bugs in the storage
> engine that I'm still wary.  My hope is to see 3.11.x get enough bug fixes
> to where most people just skip 3.0 altogether.  I'm not sure if we're there
> yet though.
>
>
> On Wed, May 31, 2017 at 9:43 AM Junaid Nasir  wrote:
>
>> as mentioned here http://www.datastax.com/dev/blog/cassandra-2-2-3-0-
>> and-beyond
>>
>>> Under normal conditions, we will NOT release 3.x.y stability releases
>>> for x > 0.  That is, we will have a traditional 3.0.y stability series, but
>>> the odd-numbered bugfix-only releases will fill that role for the tick-tock
>>> series — recognizing that occasionally we will need to be flexible enough
>>> to release an emergency fix in the case of a critical bug or security
>>> vulnerability.
>>> We do recognize that it will take some time for tick-tock releases to
>>> deliver production-level stability, which is why we will continue to
>>> deliver 2.2.y and 3.0.y bugfix releases.  (But if we do demonstrate that
>>> tick-tock can deliver the stability we want, there will be no need for a
>>> 4.0.y bugfix series, only 4.x tick-tock.)
>>
>>
>> On Wed, May 31, 2017 at 9:02 PM, pabbireddy avinash <
>> pabbireddyavin...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> We are planning to deploy a cassandra production cluster on 3.X /3.0.X .
>>> Please let us know if there is any stable version  in 3.X/3.0.X that we
>>> could deploy in production .
>>>
>>> Regards,
>>> Avinash.
>>>
>>
>>

-- 


--

Re: Stable version apache cassandra 3.X /3.0.X

[Jonathan Haddad]

I really wouldn't go by the tick tock blog post, considering tick tock is
dead.

I'm still not wild about putting any 3.0 or 3.x into production.  3.0
removed off heap memtables and there have been enough bugs in the storage
engine that I'm still wary.  My hope is to see 3.11.x get enough bug fixes
to where most people just skip 3.0 altogether.  I'm not sure if we're there
yet though.


On Wed, May 31, 2017 at 9:43 AM Junaid Nasir  wrote:

> as mentioned here
> http://www.datastax.com/dev/blog/cassandra-2-2-3-0-and-beyond
>
>> Under normal conditions, we will NOT release 3.x.y stability releases for
>> x > 0.  That is, we will have a traditional 3.0.y stability series, but the
>> odd-numbered bugfix-only releases will fill that role for the tick-tock
>> series — recognizing that occasionally we will need to be flexible enough
>> to release an emergency fix in the case of a critical bug or security
>> vulnerability.
>> We do recognize that it will take some time for tick-tock releases to
>> deliver production-level stability, which is why we will continue to
>> deliver 2.2.y and 3.0.y bugfix releases.  (But if we do demonstrate that
>> tick-tock can deliver the stability we want, there will be no need for a
>> 4.0.y bugfix series, only 4.x tick-tock.)
>
>
> On Wed, May 31, 2017 at 9:02 PM, pabbireddy avinash <
> pabbireddyavin...@gmail.com> wrote:
>
>> Hi,
>>
>> We are planning to deploy a cassandra production cluster on 3.X /3.0.X .
>> Please let us know if there is any stable version  in 3.X/3.0.X that we
>> could deploy in production .
>>
>> Regards,
>> Avinash.
>>
>
>

Re: Stable version apache cassandra 3.X /3.0.X

[Junaid Nasir]

as mentioned here
http://www.datastax.com/dev/blog/cassandra-2-2-3-0-and-beyond

> Under normal conditions, we will NOT release 3.x.y stability releases for
> x > 0.  That is, we will have a traditional 3.0.y stability series, but the
> odd-numbered bugfix-only releases will fill that role for the tick-tock
> series — recognizing that occasionally we will need to be flexible enough
> to release an emergency fix in the case of a critical bug or security
> vulnerability.
> We do recognize that it will take some time for tick-tock releases to
> deliver production-level stability, which is why we will continue to
> deliver 2.2.y and 3.0.y bugfix releases.  (But if we do demonstrate that
> tick-tock can deliver the stability we want, there will be no need for a
> 4.0.y bugfix series, only 4.x tick-tock.)


On Wed, May 31, 2017 at 9:02 PM, pabbireddy avinash <
pabbireddyavin...@gmail.com> wrote:

> Hi,
>
> We are planning to deploy a cassandra production cluster on 3.X /3.0.X .
> Please let us know if there is any stable version  in 3.X/3.0.X that we
> could deploy in production .
>
> Regards,
> Avinash.
>

Stable version apache cassandra 3.X /3.0.X

[pabbireddy avinash]

Hi,

We are planning to deploy a cassandra production cluster on 3.X /3.0.X .
Please let us know if there is any stable version  in 3.X/3.0.X that we
could deploy in production .

Regards,
Avinash.

Re: Netty SSL memory leak

[Tom van der Woerdt]

Hi John,

That's the bug I filed the ticket for, yup. I recommend updating to a newer
Cassandra version (3.0.11 or newer), which fixes this issue (and many
others).

Tom


On Wed, May 31, 2017 at 12:39 AM, John Sanda  wrote:

> I have Cassandra 3.0.9 cluster that is hitting OutOfMemoryErrors with byte
> buffer allocation. The stack trace looks like:
>
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:694) ~[na:1.8.0_131]
> at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123)
> ~[na:1.8.0_131]
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
> ~[na:1.8.0_131]
> at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:434)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:179)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.buffer.PoolArena.allocate(PoolArena.java:168)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.buffer.PoolArena.allocate(PoolArena.java:98)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(
> PooledByteBufAllocator.java:250) ~[netty-all-4.0.23.Final.jar:
> 4.0.23.Final]
> at io.netty.buffer.AbstractByteBufAllocator.directBuffer(
> AbstractByteBufAllocator.java:155) ~[netty-all-4.0.23.Final.jar:
> 4.0.23.Final]
> at io.netty.buffer.AbstractByteBufAllocator.directBuffer(
> AbstractByteBufAllocator.java:146) ~[netty-all-4.0.23.Final.jar:
> 4.0.23.Final]
> at io.netty.buffer.AbstractByteBufAllocator.buffer(
> AbstractByteBufAllocator.java:83) ~[netty-all-4.0.23.Final.jar:
> 4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:1265)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.allocateOutNetBuf(SslHandler.java:1275)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:453)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:432)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(
> AbstractChannelHandlerContext.java:688) ~[netty-all-4.0.23.Final.jar:
> 4.0.23.Final]
>
> I do not yet have a heap dump. The two relevant tickets are
> CASSANDRA-13114 
>  and CASSANDRA-13126
> . The upstream
> Netty ticket is 3057 .
> Cassandra 3.0.11 upgraded Netty to the version with the fix. Is there
> anything I can check to confirm that this is in fact the issue I am hitting?
>
> Secondly, is there a way to monitor for this? The OOME does not cause the
> JVM to exit. Instead, the logs are getting filled up with OutOfMemoryErrors.
> nodetool status reports UN, and nodetool statusbinary reports running.
>
> --
>
> - John
>

Re: recovering from failed repair , cassandra 3.10

[Micha]

The error which keeps it from starting is below.
The files like
"mc_txn_anticompactionafterrepair_19a46410-459f-11e7-91c7-4f4e8666b5c8.log"
are on both disks of a node but are different .

Of course, just renaming (deleting) the two files (or making them equal)
makes cassandra start again. But I would like to know the right way to
handle this.
I start the repair again with increased log level.


thanks for answering,
 Michael








ERROR 08:26:20 Mismatched line in file
mc_txn_anticompactionafterrepair_19a46410-459f-11e7-91c7-4f4e8666b5c8.log:
got
'ADD:[/data/2/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0/mc-4348-big,0,8][1394849421]'
expected
'ADD:[/data/2/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0/mc-4349-big,0,8][910462411]',
giving up
ERROR 08:26:20 Failed to read records for transaction log
[mc_txn_anticompactionafterrepair_19a46410-459f-11e7-91c7-4f4e8666b5c8.log
in
/data/1/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0,
/data/2/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0]
ERROR 08:26:20 Unexpected disk state: failed to read transaction log
[mc_txn_anticompactionafterrepair_19a46410-459f-11e7-91c7-4f4e8666b5c8.log
in
/data/1/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0,
/data/2/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0]
Files and contents follow:
/data/1/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0/mc_txn_anticompactionafterrepair_19a46410-459f-11e7-91c7-4f4e8666b5c8.log

ADD:[/data/2/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0/mc-4349-big,0,8][910462411]

REMOVE:[/data/1/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0/mc-4241-big,1495845618000,8][2443235315]

REMOVE:[/data/1/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0/mc-4249-big,1495856254000,8][681858089]
COMMIT:[,0,0][2613697770]
/data/2/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0/mc_txn_anticompactionafterrepair_19a46410-459f-11e7-91c7-4f4e8666b5c8.log

ADD:[/data/2/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0/mc-4348-big,0,8][1394849421]
***Does not match

in first replica file

ADD:[/data/2/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0/mc-4349-big,0,8][910462411]

REMOVE:[/data/1/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0/mc-4241-big,1495845618000,8][2443235315]

REMOVE:[/data/1/cassandra/data/KEYSPACE/TABLE-8e40c6b0f4fa11e6a7912b3358087dc0/mc-4249-big,1495856254000,8][681858089]
COMMIT:[,0,0][2613697770]






On 31.05.2017 11:10, Oleksandr Shulgin wrote:
> On Wed, May 31, 2017 at 9:11 AM, Micha  > wrote:
> 
> Hi,
> 
> after failed repair on a three node cluster all nodes were down.
> 
> 
> To clarify, was it failed repair that brought the nodes down so that you
> had to start them back?  Do you see any error messages or stack trace in
> the logs?
>  
> 
> It cannot start, since it finds a mismatch in a
> mc_txn_anticompactionafterrepair log file:
> "got ADD "
> "expected "ADD:..."
> 
> 
> The two log files are different:
> one has "ADD, ADD; REMOVE, REMOVE, COMMIT"
> the other is missing an "ADD"
> 
> 
> I assume this is about commit log.  There doesn't seem to be a separate
> log file named "mc_txn_anticompactionafterrepair" in your Cassandra version.
> 
> Each of the nodes give this error.
> 
> sstableutil -c  also gives this error.
> 
> How to deal with this?
> 
> 
> I would try removing the faulty commit log file(s) and try to start the
> node again, until it works.  This might mean that you'll have to remove
> all commit logs, but it's better than being completely down, I assume.
> 
> -- 
> Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176
> 127-59-707
> 

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Convert single node C* to cluster (rebalancing problem)

[Junaid Nasir]

after nodetool repair on new node following log appears in cassandra log

INFO  [StreamConnectionEstablisher:1] 2017-05-31 10:07:23,496
> StreamSession.java:266 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac]
> Starting streaming to /10.128.1.2
> INFO  [StreamConnectionEstablisher:1] 2017-05-31 10:07:23,501
> StreamCoordinator.java:264 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac,
> ID#0] Beginning stream session with /10.128.1.2
> INFO  [STREAM-IN-/10.128.1.2:7000] 2017-05-31 10:07:23,545
> StreamResultFuture.java:173 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac
> ID#0] Prepare completed. Receiving 2 files(131.594KiB), sending 0
> files(0.000KiB)
> INFO  [StreamReceiveTask:1] 2017-05-31 10:07:23,705
> StreamResultFuture.java:187 - [Stream
> #f0eb9670-45e8-11e7-a17b-81e9a18c6eac] Session with /10.128.1.2 is
> complete
> INFO  [StreamReceiveTask:1] 2017-05-31 10:07:23,707
> StreamResultFuture.java:219 - [Stream
> #f0eb9670-45e8-11e7-a17b-81e9a18c6eac] All sessions completed




On Wed, May 31, 2017 at 2:50 PM, Junaid Nasir  wrote:

> Cassandra version is 3.10, and yes its not a production server. i have
> seen some warnings in logs saying token exist on both servers. other than
> that nothing. if you need any more settings/details please ask. thank you
> for your time
>
>
> On Wed, May 31, 2017 at 2:04 PM, Oleksandr Shulgin <
> oleksandr.shul...@zalando.de> wrote:
>
>> On Wed, May 31, 2017 at 9:55 AM, Junaid Nasir  wrote:
>>
>>> Cassandra ensure that adding or removing nodes are very easy and that
>>> load is balanced between nodes when a change is made. but it's not working
>>> in my case.
>>> I have a single node C* deployment (with 270 GB of data) and want to
>>> load balance the data on multiple nodes,
>>>
>>
>> I guess it's fair to assume this is not a production "cluster"?
>>
>>
>>> I followed this guide
>>> 
>>>
>>> `nodetool status` shows 2 nodes but load is not balanced between them
>>>
>>> Datacenter: dc1
>>> ===
>>> Status=Up/Down
>>> |/ State=Normal/Leaving/Joining/Moving
>>> --  Address  Load   Tokens   Owns (effective)  Host IDRack
>>> UN  10.128.0.7   270.75 GiB  256  48.6%
>>> 1a3f6faa-4376-45a8-9c20-11480ae5664c  rack1
>>> UN  10.128.0.14  414.36 KiB  256  51.4%
>>> 66a89fbf-08ba-4b5d-9f10-55d52a199b41  rack1
>>>
>>> I can imagine you are referring to Load column here, which shows only
>> 400 KB for the new node.  Did the newly added node actually bootstrap?  Are
>> there any error/warning messages in the Cassandra log?
>>
>>
>>> I also ran 'nodetool repair' on new node but result is same. any
>>> pointers would be appreciated :)
>>>
>>
>> Hm, this is not expected.  Even if the node didn't bootstrap, repair
>> should have streamed the data it is responsible for.
>>
>> conf file of new node
>>>
>>> cluster_name: 'cluster1'
>>>  - seeds: "10.128.0.7"
>>> num_tokens: 256
>>> endpoint_snitch: GossipingPropertyFileSnitch
>>>
>>> I see that you've tried to add the new node as a normal (non-seed) node,
>> which supposed to work and it should have bootstrapped itself.  Which
>> version of Cassandra is it exactly?
>>
>> --
>> Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176
>> 127-59-707 <+49%20176%2012759707>
>>
>>
>

Re: Convert single node C* to cluster (rebalancing problem)

[Junaid Nasir]

Cassandra version is 3.10, and yes its not a production server. i have seen
some warnings in logs saying token exist on both servers. other than that
nothing. if you need any more settings/details please ask. thank you for
your time


On Wed, May 31, 2017 at 2:04 PM, Oleksandr Shulgin <
oleksandr.shul...@zalando.de> wrote:

> On Wed, May 31, 2017 at 9:55 AM, Junaid Nasir  wrote:
>
>> Cassandra ensure that adding or removing nodes are very easy and that
>> load is balanced between nodes when a change is made. but it's not working
>> in my case.
>> I have a single node C* deployment (with 270 GB of data) and want to load
>> balance the data on multiple nodes,
>>
>
> I guess it's fair to assume this is not a production "cluster"?
>
>
>> I followed this guide
>> 
>>
>> `nodetool status` shows 2 nodes but load is not balanced between them
>>
>> Datacenter: dc1
>> ===
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address  Load   Tokens   Owns (effective)  Host IDRack
>> UN  10.128.0.7   270.75 GiB  256  48.6%
>> 1a3f6faa-4376-45a8-9c20-11480ae5664c  rack1
>> UN  10.128.0.14  414.36 KiB  256  51.4%
>> 66a89fbf-08ba-4b5d-9f10-55d52a199b41  rack1
>>
>> I can imagine you are referring to Load column here, which shows only 400
> KB for the new node.  Did the newly added node actually bootstrap?  Are
> there any error/warning messages in the Cassandra log?
>
>
>> I also ran 'nodetool repair' on new node but result is same. any pointers
>> would be appreciated :)
>>
>
> Hm, this is not expected.  Even if the node didn't bootstrap, repair
> should have streamed the data it is responsible for.
>
> conf file of new node
>>
>> cluster_name: 'cluster1'
>>  - seeds: "10.128.0.7"
>> num_tokens: 256
>> endpoint_snitch: GossipingPropertyFileSnitch
>>
>> I see that you've tried to add the new node as a normal (non-seed) node,
> which supposed to work and it should have bootstrapped itself.  Which
> version of Cassandra is it exactly?
>
> --
> Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176
> 127-59-707 <+49%20176%2012759707>
>
>

Re: recovering from failed repair , cassandra 3.10

[Oleksandr Shulgin]

On Wed, May 31, 2017 at 9:11 AM, Micha  wrote:

> Hi,
>
> after failed repair on a three node cluster all nodes were down.
>

To clarify, was it failed repair that brought the nodes down so that you
had to start them back?  Do you see any error messages or stack trace in
the logs?


> It cannot start, since it finds a mismatch in a
> mc_txn_anticompactionafterrepair log file:
> "got ADD "
> "expected "ADD:..."
>
>
> The two log files are different:
> one has "ADD, ADD; REMOVE, REMOVE, COMMIT"
> the other is missing an "ADD"
>

I assume this is about commit log.  There doesn't seem to be a separate log
file named "mc_txn_anticompactionafterrepair" in your Cassandra version.

Each of the nodes give this error.
>
> sstableutil -c  also gives this error.
>
> How to deal with this?
>

I would try removing the faulty commit log file(s) and try to start the
node again, until it works.  This might mean that you'll have to remove all
commit logs, but it's better than being completely down, I assume.

-- 
Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176
127-59-707

Re: Convert single node C* to cluster (rebalancing problem)

[Oleksandr Shulgin]

On Wed, May 31, 2017 at 9:55 AM, Junaid Nasir  wrote:

> Cassandra ensure that adding or removing nodes are very easy and that load
> is balanced between nodes when a change is made. but it's not working in my
> case.
> I have a single node C* deployment (with 270 GB of data) and want to load
> balance the data on multiple nodes,
>

I guess it's fair to assume this is not a production "cluster"?


> I followed this guide
> 
>
> `nodetool status` shows 2 nodes but load is not balanced between them
>
> Datacenter: dc1
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address  Load   Tokens   Owns (effective)  Host IDRack
> UN  10.128.0.7   270.75 GiB  256  48.6%
> 1a3f6faa-4376-45a8-9c20-11480ae5664c  rack1
> UN  10.128.0.14  414.36 KiB  256  51.4%
> 66a89fbf-08ba-4b5d-9f10-55d52a199b41  rack1
>
> I can imagine you are referring to Load column here, which shows only 400
KB for the new node.  Did the newly added node actually bootstrap?  Are
there any error/warning messages in the Cassandra log?


> I also ran 'nodetool repair' on new node but result is same. any pointers
> would be appreciated :)
>

Hm, this is not expected.  Even if the node didn't bootstrap, repair should
have streamed the data it is responsible for.

conf file of new node
>
> cluster_name: 'cluster1'
>  - seeds: "10.128.0.7"
> num_tokens: 256
> endpoint_snitch: GossipingPropertyFileSnitch
>
> I see that you've tried to add the new node as a normal (non-seed) node,
which supposed to work and it should have bootstrapped itself.  Which
version of Cassandra is it exactly?

-- 
Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176
127-59-707

Convert single node C* to cluster (rebalancing problem)

[Junaid Nasir]

Cassandra ensure that adding or removing nodes are very easy and that load
is balanced between nodes when a change is made. but it's not working in my
case.
I have a single node C* deployment (with 270 GB of data) and want to load
balance the data on multiple nodes, I followed this guide


`nodetool status` shows 2 nodes but load is not balanced between them

Datacenter: dc1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address  Load   Tokens   Owns (effective)  Host IDRack
UN  10.128.0.7   270.75 GiB  256  48.6%
1a3f6faa-4376-45a8-9c20-11480ae5664c  rack1
UN  10.128.0.14  414.36 KiB  256  51.4%
66a89fbf-08ba-4b5d-9f10-55d52a199b41  rack1

I also ran 'nodetool repair' on new node but result is same. any pointers
would be appreciated :)

conf file of new node

cluster_name: 'cluster1'
 - seeds: "10.128.0.7"
num_tokens: 256
endpoint_snitch: GossipingPropertyFileSnitch

Thanks,
Junaid

recovering from failed repair , cassandra 3.10

[Micha]

Hi,

after failed repair on a three node cluster all nodes were down.
It cannot start, since it finds a mismatch in a
mc_txn_anticompactionafterrepair log file:
"got ADD "
"expected "ADD:..."


The two log files are different:
one has "ADD, ADD; REMOVE, REMOVE, COMMIT"
the other is missing an "ADD"


Each of the nodes give this error.

sstableutil -c  also gives this error.

How to deal with this?


thanks,
 Michael




-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org