from:"Akhil Mehra"

Re: Begginer with cassandra Apache - Gossip

2015-03-15 Thread Akhil Mehra

Hi Jean,

I have written three article that might be of help to you:

http://abiasforaction.net/an-introduction-to-apache-cassandra/ 
http://abiasforaction.net/an-introduction-to-apache-cassandra/

http://abiasforaction.net/cassandra-architecture/ 
http://abiasforaction.net/cassandra-architecture/

http://abiasforaction.net/a-practical-introduction-to-cassandra-query-language/ 
http://abiasforaction.net/a-practical-introduction-to-cassandra-query-language/

These three should help you get started. Feedback will be highly appreciated. I 
am looking to know if these article helped you understand Cassandra 
fundamentals better.

I would also highly recommend the tutorials on the Datastax site:

http://www.datastax.com/resources/tutorials 
http://www.datastax.com/resources/tutorials

I hope this helps.

Cheers,
Akhil

 On 16/03/2015, at 8:13 am, jean paul researche...@gmail.com wrote:
 
 Hello All,
 
 Please, i'm newer with cassandra Apache. i'm sooo interested.  i know some 
 details about its architecture, some details also about gossip protocol.
 
 I'd like to test it, to more understand :) i ask if there is a howto tutorial 
 that can help me ? 
 
 Thanks a lot for help.
 Best Regards.

Re: Begginer with cassandra Apache - Gossip

2015-03-15 Thread Akhil Mehra

Hi Jean,

Check our the following urls:

https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/gms/

https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/gms/Gossiper.java
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/gms/Gossiper.java

https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/gms/Gossiper.java

Is this what your were looking for. I hope this helps.

Cheers,
Akhil

On 16/03/2015, at 10:43 am, jean paul researche...@gmail.com wrote:

Ok i will read all documents :) thanks a lot for help :)
Please, can i you send me some details (url of source code) about the
implementation of gossip protocol used in cassandra ? thanks a lot for help :)
i find this https://github.com/bpot/node-gossip
https://github.com/bpot/node-gossip

Bests.

2015-03-15 22:38 GMT+01:00 Akhil Mehra akhilme...@gmail.com
mailto:akhilme...@gmail.com:
Hi Jean,

I have written three article that might be of help to you:

http://abiasforaction.net/an-introduction-to-apache-cassandra/
http://abiasforaction.net/an-introduction-to-apache-cassandra/

http://abiasforaction.net/cassandra-architecture/
http://abiasforaction.net/cassandra-architecture/

http://abiasforaction.net/a-practical-introduction-to-cassandra-query-language/

These three should help you get started. Feedback will be highly appreciated.
I am looking to know if these article helped you understand Cassandra
fundamentals better.

I would also highly recommend the tutorials on the Datastax site:

http://www.datastax.com/resources/tutorials
http://www.datastax.com/resources/tutorials

I hope this helps.

Cheers,
Akhil

On 16/03/2015, at 8:13 am, jean paul researche...@gmail.com
mailto:researche...@gmail.com wrote:

Hello All,

Please, i'm newer with cassandra Apache. i'm sooo interested. i know some
details about its architecture, some details also about gossip protocol.

I'd like to test it, to more understand :) i ask if there is a howto
tutorial that can help me ?

Thanks a lot for help.
Best Regards.

Re: Begginer with cassandra Apache - Gossip

2015-03-16 Thread Akhil Mehra

Hi Jean,

I see you are getting a lot of help on other threads. Can you please elaborate
a bit more on the problem you are trying to solve. Would nodetool status help
(http://www.datastax.com/documentation/cassandra/2.0/cassandra/tools/toolsStatus.html

http://www.datastax.com/documentation/cassandra/2.0/cassandra/tools/toolsStatus.html)

Not sure why you trying to dig into the code?

Cheers,
Akhil

On 17/03/2015, at 12:00 am, jean paul researche...@gmail.com wrote:

Hello :) Thanks a lot for you help :)

Please, i have another question about gossip and failure detection.
i'd like to test step by step all components in cassandra. i'd like to test
first of all gossip protocol
https://wiki.apache.org/cassandra/ArchitectureGossip
https://wiki.apache.org/cassandra/ArchitectureGossip and the failure
detection

I have tens of nodes in my cluster. i'd like to get the states of these
nodes(alive, not etc ..) My problem is that I do not know how to start. Can
you guide me please a litte bit ?

Step 1: i have to download all codes in gms
https://github.com/apache/cassandra/tree/trunk/src/java/org/apache/cassandra/gms

https://github.com/apache/cassandra/tree/trunk/src/java/org/apache/cassandra/gms
? that's it ?

Thank you so much for help. i'd like to know how to start please :)
Best regards.

2015-03-15 23:44 GMT+01:00 Akhil Mehra akhilme...@gmail.com
mailto:akhilme...@gmail.com:
Hi Jean,

Check our the following urls:

https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/gms/

https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/gms/Gossiper.java
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/gms/Gossiper.java

https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/gms/Gossiper.java

Is this what your were looking for. I hope this helps.

Cheers,
Akhil

On 16/03/2015, at 10:43 am, jean paul researche...@gmail.com
mailto:researche...@gmail.com wrote:

Ok i will read all documents :) thanks a lot for help :)
Please, can i you send me some details (url of source code) about the
implementation of gossip protocol used in cassandra ? thanks a lot for help
:)
i find this https://github.com/bpot/node-gossip
https://github.com/bpot/node-gossip

Bests.

2015-03-15 22:38 GMT+01:00 Akhil Mehra akhilme...@gmail.com
mailto:akhilme...@gmail.com:
Hi Jean,

I have written three article that might be of help to you:

http://abiasforaction.net/an-introduction-to-apache-cassandra/
http://abiasforaction.net/an-introduction-to-apache-cassandra/

http://abiasforaction.net/cassandra-architecture/
http://abiasforaction.net/cassandra-architecture/

http://abiasforaction.net/a-practical-introduction-to-cassandra-query-language/

These three should help you get started. Feedback will be highly
appreciated. I am looking to know if these article helped you understand
Cassandra fundamentals better.

I would also highly recommend the tutorials on the Datastax site:

http://www.datastax.com/resources/tutorials
http://www.datastax.com/resources/tutorials

I hope this helps.

Cheers,
Akhil

On 16/03/2015, at 8:13 am, jean paul researche...@gmail.com
mailto:researche...@gmail.com wrote:

Hello All,

Please, i'm newer with cassandra Apache. i'm sooo interested. i know some
details about its architecture, some details also about gossip protocol.

I'd like to test it, to more understand :) i ask if there is a howto
tutorial that can help me ?

Thanks a lot for help.
Best Regards.

What are the best ways to learn Apache Cassandra

2015-12-19 Thread Akhil Mehra

What are some things you wish you knew when you started learning Apache
Cassandra.

What are some of the best resources you have come across to learn Apache
Cassandra. Books, blogs etc. I am looking for tips on key concepts,
principles that you wish you were exposed to when you started learning
Apache Cassandra.

What were the main pain points when trying to get to grips with Cassandra.

Essentially I am looking for all tips that will help shorten the learning
curve.

Thanks
Regards,
Akhil Mehra

Re: MUTATION messages were dropped in last 5000 ms for cross node timeout

2017-08-04 Thread Akhil Mehra

Glad I could be of help :)

Hopefully the partition size resize goes smoothly.

Regards,
Akhil

> On 4/08/2017, at 5:41 AM, ZAIDI, ASAD A <az1...@att.com> wrote:
> 
> Hi Akhil,
>  
> Thank you for your reply.
>  
> I kept testing different timeout numbers over last week and eventually 
> settled at setting *_request_timeout_in_ms parameters at 1.5minutes for 
> coordinator wait time. That is the number where I donot see any dropped 
> mutations. 
>  
> Also asked developers to tweak data model where we saw bunch of tables with 
> really large partition size , some are ranging  Partition-key size around 
> ~6.6GB.. we’re now working to reduce the partition size of the tables. I am 
> hoping corrected data model will help reduce coordinator wait time (get back 
> to default number!)  again.
>  
> Thank again/Asad
>  
> From: Akhil Mehra [mailto:akhilme...@gmail.com] 
> Sent: Friday, July 21, 2017 4:24 PM
> To: user@cassandra.apache.org
> Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node 
> timeout
>  
> Hi Asad,
>  
> The 5000 ms is not configurable 
> (https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/src/java/org/apache/cassandra/net/MessagingService.java#L423
>  
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_cassandra_blob_8b3a60b9a7dbefeecc06bace617279612ec7092d_src_java_org_apache_cassandra_net_MessagingService.java-23L423=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=dp_TvjXTbUtu3Iu43aZ83eHl1fgW6l4P4PSQglF855g=USbrEM6jaGFIRKSUhJBx3VAkSSrXzid0db6TDV1vrDs=>).
>  This just the time after which the number of dropped messages are reported. 
> Thus dropped messages are reported every 5000ms. 
>  
> If you are looking to tweak the number of ms after which a message is 
> considered dropped then you need to use the write_request_timeout_in_ms.  The 
> write_request_timeout_in_ms 
> (http://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configCassandra_yaml_r.html
>  
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.datastax.com_en_cassandra_2.1_cassandra_configuration_configCassandra-5Fyaml-5Fr.html=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=dp_TvjXTbUtu3Iu43aZ83eHl1fgW6l4P4PSQglF855g=ab1NW9WoAXIlxT2kWjsiYFVaVidEnC_MB770pwTtqLs=>)
>  can be used to increase the mutation timeout. By default it is set to 2000ms.
>  
> I hope that helps.
>  
> Regards,
> Akhil
>  
>  
> On 22/07/2017, at 2:46 AM, ZAIDI, ASAD A <az1...@att.com 
> <mailto:az1...@att.com>> wrote:
>  
> Hi Akhil,
>  
> Thank you for your reply. Previously, I did ‘tune’ various timeouts – 
> basically increased them a bit but none of those parameter listed in the link 
> matches with that “were dropped in last 5000 ms”.
> I was wondering from where that [5000ms] number is coming from when,  like I 
> mentioned before, none of any timeout parameter settings matches that #!
>  
> Load is intermittently high but again cpu queue length never goes beyond 
> medium depth. I wonder if there is some internal limit that I’m still not 
> aware of.
>  
> Thanks/Asad
>  
>  
> From: Akhil Mehra [mailto:akhilme...@gmail.com <mailto:akhilme...@gmail.com>] 
> Sent: Thursday, July 20, 2017 3:47 PM
> To: user@cassandra.apache.org <mailto:user@cassandra.apache.org>
> Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node 
> timeout
>  
> Hi Asad,
>  
> http://cassandra.apache.org/doc/latest/faq/index.html#why-message-dropped 
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__cassandra.apache.org_doc_latest_faq_index.html-23why-2Dmessage-2Ddropped=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=WcHuHKcjg2YCsAbw2NR_0-CiHr9JNxtCzYikia16mpo=0_0pQfoOZLuswpQ_lE-AU2bTMFLgRbR4k4Kh8vEOZSk=>
>  
> As mentioned in the link above this is a load shedding mechanism used by 
> Cassandra.
>  
> Is you cluster under heavy load?
>  
> Regards,
> Akhil
>  
>  
> On 21/07/2017, at 3:27 AM, ZAIDI, ASAD A <az1...@att.com 
> <mailto:az1...@att.com>> wrote:
>  
> Hello Folks –
>  
> I’m using apache-cassandra 2.2.8.
>  
> I see many messages like below in my system.log file. In Cassandra.yaml file 
> [ cross_node_timeout: true] is set and NTP server is also running correcting 
> clock drift on 16node cluster. I do not see pending or blocked HintedHandoff  
> in tpstats output though there are bunch of MUTATIONS dropped observed.
>  
> 
> INFO  [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 - 
> MUTATION messages were dropped in last 5000 ms: 822 for internal timeout and 
> 2152 for cross node timeout
> 
>  
> I’m seeking help here if you please let me know what I need to check in order 
> to address these cross node timeouts.
>  
> Thank you,
> Asad

Re: MemtablePostFlush pending

2017-08-13 Thread Akhil Mehra

HI Asad,

The post flush task frees up allocated commit log segments.

Apart for commit log segment allocation the post flush task "synchronises 
custom secondary indexes and provides ordering guarantees for futures on 
switchMemtable/flush
etc, which expect to be able to wait until the flush (and all prior flushes) 
requested have completed."

The post flush executor is a single threaded thread pool that cannot be tuned 
(https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L160-L167
 
)

Are you using secondary indexes? I there a high write throughput which is 
resulting in frequent meltable flushes?

Regards,
Akhil





> On 12/08/2017, at 7:05 AM, ZAIDI, ASAD A  wrote:
> 
> Hello Folks,
>  
> I’m using Cassandra 2.2 on 14 node cluster.
>  
> Now a days, I’m observing memtablepostflush pending number going high , this 
> happens intermittently. I’m looking if  Is there way to ‘tune’ 
> memtablepostflush stage?
>  
> Thanks/ASad

Re: MUTATION messages were dropped in last 5000 ms for cross node timeout

2017-07-21 Thread Akhil Mehra

Hi Asad,

The 5000 ms is not configurable 
(https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/src/java/org/apache/cassandra/net/MessagingService.java#L423
 
<https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/src/java/org/apache/cassandra/net/MessagingService.java#L423>).
 This just the time after which the number of dropped messages are reported. 
Thus dropped messages are reported every 5000ms. 

If you are looking to tweak the number of ms after which a message is 
considered dropped then you need to use the write_request_timeout_in_ms.  The 
write_request_timeout_in_ms 
(http://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configCassandra_yaml_r.html
 
<http://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configCassandra_yaml_r.html>)
 can be used to increase the mutation timeout. By default it is set to 2000ms.

I hope that helps.

Regards,
Akhil


> On 22/07/2017, at 2:46 AM, ZAIDI, ASAD A <az1...@att.com> wrote:
> 
> Hi Akhil,
>  
> Thank you for your reply. Previously, I did ‘tune’ various timeouts – 
> basically increased them a bit but none of those parameter listed in the link 
> matches with that “were dropped in last 5000 ms”.
> I was wondering from where that [5000ms] number is coming from when,  like I 
> mentioned before, none of any timeout parameter settings matches that #!
>  
> Load is intermittently high but again cpu queue length never goes beyond 
> medium depth. I wonder if there is some internal limit that I’m still not 
> aware of.
>  
> Thanks/Asad
>  
>  
> From: Akhil Mehra [mailto:akhilme...@gmail.com] 
> Sent: Thursday, July 20, 2017 3:47 PM
> To: user@cassandra.apache.org
> Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node 
> timeout
>  
> Hi Asad,
>  
> http://cassandra.apache.org/doc/latest/faq/index.html#why-message-dropped 
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__cassandra.apache.org_doc_latest_faq_index.html-23why-2Dmessage-2Ddropped=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=WcHuHKcjg2YCsAbw2NR_0-CiHr9JNxtCzYikia16mpo=0_0pQfoOZLuswpQ_lE-AU2bTMFLgRbR4k4Kh8vEOZSk=>
>  
> As mentioned in the link above this is a load shedding mechanism used by 
> Cassandra.
>  
> Is you cluster under heavy load?
>  
> Regards,
> Akhil
>  
>  
> On 21/07/2017, at 3:27 AM, ZAIDI, ASAD A <az1...@att.com 
> <mailto:az1...@att.com>> wrote:
>  
> Hello Folks –
>  
> I’m using apache-cassandra 2.2.8.
>  
> I see many messages like below in my system.log file. In Cassandra.yaml file 
> [ cross_node_timeout: true] is set and NTP server is also running correcting 
> clock drift on 16node cluster. I do not see pending or blocked HintedHandoff  
> in tpstats output though there are bunch of MUTATIONS dropped observed.
>  
> 
> INFO  [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 - 
> MUTATION messages were dropped in last 5000 ms: 822 for internal timeout and 
> 2152 for cross node timeout
> 
>  
> I’m seeking help here if you please let me know what I need to check in order 
> to address these cross node timeouts.
>  
> Thank you,
> Asad

Re: MUTATION messages were dropped in last 5000 ms for cross node timeout

2017-07-20 Thread Akhil Mehra

Hi Asad,

http://cassandra.apache.org/doc/latest/faq/index.html#why-message-dropped 


As mentioned in the link above this is a load shedding mechanism used by 
Cassandra.

Is you cluster under heavy load?

Regards,
Akhil


> On 21/07/2017, at 3:27 AM, ZAIDI, ASAD A  wrote:
> 
> Hello Folks –
>  
> I’m using apache-cassandra 2.2.8.
>  
> I see many messages like below in my system.log file. In Cassandra.yaml file 
> [ cross_node_timeout: true] is set and NTP server is also running correcting 
> clock drift on 16node cluster. I do not see pending or blocked HintedHandoff  
> in tpstats output though there are bunch of MUTATIONS dropped observed.
>  
> 
> INFO  [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 - 
> MUTATION messages were dropped in last 5000 ms: 822 for internal timeout and 
> 2152 for cross node timeout
> 
>  
> I’m seeking help here if you please let me know what I need to check in order 
> to address these cross node timeouts.
>  
> Thank you,
> Asad

Re: nodetool repair failure

2017-06-29 Thread Akhil Mehra

01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
> /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
> /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
> /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
> /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
> /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
> /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
> /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
> /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
> /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
> /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message to 
> /xx.xx.xx.91
> [2017-06-29 19:02:04,842] Some repair failed
> [2017-06-29 19:02:04,848] Repair command #1 finished in 1 minute 2 seconds
> error: Repair job has failed with the error message: [2017-06-29 
> 19:02:04,842] Some repair failed
> -- StackTrace --
> java.lang.RuntimeException: Repair job has failed with the error message: 
> [2017-06-29 19:02:04,842] Some repair failed
>   at org.apache.cassandra.tools. RepairRunner.progress( 
> RepairRunner.java:116)
>   at org.apache.cassandra.utils. progress.jmx. 
> JMXNotificationProgressListene r.handleNotification( 
> JMXNotificationProgressListene r.java:77)
>   at com.sun.jmx.remote.internal. ClientNotifForwarder$ NotifFetcher. 
> dispatchNotification( ClientNotifForwarder.java:583)
>   at com.sun.jmx.remote.internal. ClientNotifForwarder$ 
> NotifFetcher.doRun( ClientNotifForwarder.java:533)
>   at com.sun.jmx.remote.internal. ClientNotifForwarder$ NotifFetcher.run( 
> ClientNotifForwarder.java:452)
>   at com.sun.jmx.remote.internal. ClientNotifForwarder$ 
> LinearExecutor$1.run( ClientNotifForwarder.java:108)
> 
> 
> 
> FYI I am running repair from xx.xx.xx.91 node and its a 5 node cluster 
> xx.xx.xx.91-xx.xx.xx.95
> 
> On Wed, Jun 28, 2017 at 5:16 PM, Akhil Mehra <akhilme...@gmail.com 
> <mailto:akhilme...@gmail.com>> wrote:
> nodetool repair has a trace option 
> 
> nodetool repair -tr yourkeyspacename
> 
> see if that provides you with additional information.
> 
> Regards,
> Akhil 
> 
>> On 28/06/2017, at 2:25 AM, Balaji Venkatesan <venkatesan.bal...@gmail.com 
>> <mailto:venkatesan.bal...@gmail.com>> wrote:
>> 
>> 
>> We use Apache Cassandra 3.10-13 
>> 
>> On Jun 26, 2017 8:41 PM, "Michael Shuler" <mich...@pbandjelly.org 
>> <mailto:mich...@pbandjelly.org>> wrote:
>> What version of Cassandra?
>> 
>> --
>> Michael
>> 
>> On 06/26/2017 09:53 PM, Balaji Venkatesan wrote:
>> > Hi All,
>> >
>> > When I run nodetool repair on a keyspace I constantly get  "Some repair
>> > failed" error, there are no sufficient info to debug more. Any help?
>> >
>> > Here is the stacktrace
>> >
>> > == == ==
>> > [2017-06-27 02:44:34,275] Some repair failed
>> > [2017-06-27 02:44:34,279] Repair command #3 finished in 33 seconds
>> > error: Repair job has failed with the error message: [2017-06-27
>> > 02:44:34,275] Some repair failed
>> > -- StackTrace --
>> > java.lang.RuntimeException: Repair job has failed with the error
>> > message: [2017-06-27 02:44:34,275] Some repair failed
>> > at org.apache.cassandra.tools.Rep airRunner.progress(RepairRunne 
>> > r.java:116)
>> > at
>> > org.apache.cassandra.utils.pro <http://org.apache.cassandra.utils.pro/> 
>> > gress.jmx.JMXNotificationProgr essListener. handleNotification(JMXNotifica 
>> > tionProgressListener.java:77)
>> > at
>> > com.sun.jmx.remote.internal.Cl <http://com.sun.jmx.remote.internal.cl/> 
>> > ientNotifForwarder$NotifFetche r.dispatchNotification(ClientN 
>> > otifForwarder.java:583)
>> > at
>> > com.sun.jmx.remote.internal.Cl <http://com.sun.jmx.remote.internal.cl/> 
>> > ientNotifForwarder$NotifFetche r.doRun(ClientNotifForwarder. java:533)
>> > at
>> > com.sun.jmx.remote.internal.Cl <http://com.sun.jmx.remote.internal.cl/> 
>> > ientNotifForwarder$NotifFetche r.run(ClientNotifForwarder. java:452)
>> > at
>> > com.sun.jmx.remote.internal.Cl <http://com.sun.jmx.remote.internal.cl/> 
>> > ientNotifForwarder$LinearExecu tor$1.run(ClientNotifForwarder .java:108)
>> > == == ==
>> >
>> >
>> > --
>> > Thanks,
>> > Balaji Venkatesan.
>> 
>> 
>> -- -- -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apa che.org 
>> <mailto:user-unsubscr...@cassandra.apache.org>
>> For additional commands, e-mail: user-h...@cassandra.apache.org 
>> <mailto:user-h...@cassandra.apache.org>
>> 
>> 
> 
> 
> 
> 
> -- 
> Thanks,
> Balaji Venkatesan.
> 
> 
> 
> -- 
> Thanks,
> Balaji Venkatesan.

Re: timeoutexceptions with UDF causing cassandra forceful exits

2017-06-29 Thread Akhil Mehra

By default user_function_timeout_policy is set to die i.e. warn and kill the 
JVM. Please find below a source code snippet that outlines possible setting.

   /**
 * Defines what to do when a UDF ran longer than 
user_defined_function_fail_timeout.
 * Possible options are:
 * - 'die' - i.e. it is able to emit a warning to the client before the 
Cassandra Daemon will shut down.
 * - 'die_immediate' - shut down C* daemon immediately (effectively prevent 
the chance that the client will receive a warning).
 * - 'ignore' - just log - the most dangerous option.
 * (Only valid, if enable_user_defined_functions_threads==true)
 */
public UserFunctionTimeoutPolicy user_function_timeout_policy = 
UserFunctionTimeoutPolicy.die;

To answer your question. Yes it is normal for Cassandra to shut down due to a 
rogue UDF.

Warm Regards,
Akhil Mehra

> On 30/06/2017, at 11:17 AM, Gopal, Dhruva <dhruva.go...@aspect.com> wrote:
> 
> Hi –
>   Is it normal for cassand to be shutdown forcefully on timeout exceptions 
> when using UDFs? We are admittedly trying some load tests on our dev 
> environments which may be somewhat constrained, but didn’t expect to see 
> forceful shutdowns such as these when we ran our tests. We’re running 
> Cassandra 3.10. Dev environment is an MBP (Core i7/16GB RAM). Sample error 
> below – feedback will be much appreciated.
>  
> ERROR [NonPeriodicTasks:1] 2017-06-29 10:48:54,476 
> JVMStabilityInspector.java:142 - JVM state determined to be unstable.  
> Exiting forcefully due to:
> java.util.concurrent.TimeoutException: User defined function reporting.latest 
> : (map<text, frozen<tuple<bigint, text>>>, bigint, timestamp, boolean) -> 
> map<text, frozen<tuple<bigint, text>>> ran longer than 1500ms - will stop 
> Cassandra VM
> at 
> org.apache.cassandra.cql3.functions.UDFunction.async(UDFunction.java:483) 
> ~[apache-cassandra-3.10.jar:3.10]
> at 
> org.apache.cassandra.cql3.functions.UDFunction.executeAsync(UDFunction.java:398)
>  ~[apache-cassandra-3.10.jar:3.10]
> at 
> org.apache.cassandra.cql3.functions.UDFunction.execute(UDFunction.java:298) 
> ~[apache-cassandra-3.10.jar:3.10]
> at 
> org.apache.cassandra.cql3.selection.ScalarFunctionSelector.getOutput(ScalarFunctionSelector.java:61)
>  ~[apache-cassandra-3.10.jar:3.10]
> at 
> org.apache.cassandra.cql3.selection.Selection$SelectionWithProcessing$1.getOutputRow(Selection.java:592)
>  ~[apache-cassandra-3.10.jar:3.10]
> at 
> org.apache.cassandra.cql3.selection.Selection$ResultSetBuilder.getOutputRow(Selection.java:430)
>  ~[apache-cassandra-3.10.jar:3.10]
> at 
> org.apache.cassandra.cql3.selection.Selection$ResultSetBuilder.build(Selection.java:417)
>  ~[apache-cassandra-3.10.jar:3.10]
> at 
> org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:763)
>  ~[apache-cassandra-3.10.jar:3.10]
> at 
> org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:400)
>  ~[apache-cassandra-3.10.jar:3.10]
> at 
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:378)
>  ~[apache-cassandra-3.10.jar:3.10]
> at 
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:251)
>  ~[apache-cassandra-3.10.jar:3.10]
> at 
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:79)
>  ~[apache-cassandra-3.10.jar:3.10]
> at 
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:217)
>  ~[apache-cassandra-3.10.jar:3.10]
> at 
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:523)
>  ~[apache-cassandra-3.10.jar:3.10]
> at 
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:500)
>  ~[apache-cassandra-3.10.jar:3.10]
> at 
> org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:146)
>  ~[apache-cassandra-3.10.jar:3.10]
> at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517)
>  ~[apache-cassandra-3.10.jar:3.10]
> at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
>  ~[apache-cassandra-3.10.jar:3.10]
> at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>  ~[netty-all-4.0.39.Final.jar:4.0.39.Final]
> at 
> io.netty.channel.AbstractChannelHandlerCont

Re: nodetool repair failure

2017-06-28 Thread Akhil Mehra

nodetool repair has a trace option 

nodetool repair -tr yourkeyspacename

see if that provides you with additional information.

Regards,
Akhil 

> On 28/06/2017, at 2:25 AM, Balaji Venkatesan  
> wrote:
> 
> 
> We use Apache Cassandra 3.10-13 
> 
> On Jun 26, 2017 8:41 PM, "Michael Shuler"  > wrote:
> What version of Cassandra?
> 
> --
> Michael
> 
> On 06/26/2017 09:53 PM, Balaji Venkatesan wrote:
> > Hi All,
> >
> > When I run nodetool repair on a keyspace I constantly get  "Some repair
> > failed" error, there are no sufficient info to debug more. Any help?
> >
> > Here is the stacktrace
> >
> > ==
> > [2017-06-27 02:44:34,275] Some repair failed
> > [2017-06-27 02:44:34,279] Repair command #3 finished in 33 seconds
> > error: Repair job has failed with the error message: [2017-06-27
> > 02:44:34,275] Some repair failed
> > -- StackTrace --
> > java.lang.RuntimeException: Repair job has failed with the error
> > message: [2017-06-27 02:44:34,275] Some repair failed
> > at org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:116)
> > at
> > org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)
> > at
> > com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583)
> > at
> > com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533)
> > at
> > com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
> > at
> > com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)
> > ==
> >
> >
> > --
> > Thanks,
> > Balaji Venkatesan.
> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
> 
> For additional commands, e-mail: user-h...@cassandra.apache.org 
> 
> 
>

Re: C* 3 node issue -Urgent

2017-08-23 Thread Akhil Mehra

The cqlsh image say bad credentials. Just confirming that you have the correct 
username/password when logging on.

By turing on authentication I am assuming you mean using the 
PasswordAuthenticator instead of the AllowAllAuthenticator in the yaml.

Cheers,
Akhil

> On 23/08/2017, at 8:59 PM, Jonathan Baynes  
> wrote:
> 
> I will also  mention I am on:
>  
> C* 3.0.11
> Linux Oracle red hat 7.1
> Java 1.8.0.31
> Python 2.7
>  
> From: Jonathan Baynes 
> Sent: 23 August 2017 09:47
> To: 'user@cassandra.apache.org '
> Cc: Stewart Allman
> Subject: C* 3 node issue -Urgent
>  
> Hi Everyone.
>  
> I  need the communities help here.
>  
> I have attempted this morning to turn on JMX authentication for Nodetool. 
> I’ve gone into the Cassandra-env.sh file and updated the following:
>  
> LOCAL_JMX=No
> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
> JVM_OPTS="$JVM_OPTS 
> -Dcom.sun.management.jmxremote.password.file=/opt/cassandra/application/jmxremote.password"
>  
> Then I restarted each node, I have 3 of them  in a single DC, Single Rack.
> 115.63 (seed)
> 115.64
> 115.65 (seed)
>  
> Nodetool status shows this
>  
> 
>  
> In the Yaml I have Authentication turned ON.
> Cassandra has been working flawlessly up until the moment I restarted the 
> nodes.
> 
> as it stands right now.. 
>  
> 115.63 is up, but I cannot connect to CQLSH
> I get the following error 
> 
>  
> When I check the logs I on 115.64 and 115.65 I can see this
>  
> WARN  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:355 
> - CassandraRoleManager skipped default role setup: some nodes were not ready
> INFO  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:394 
> - Setup task failed with error, rescheduling
> Caused by: java.lang.RuntimeException: 
> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve 
> consistency level LOCAL_ONE
> at 
> org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:512)
>  ~[apache-cassandra-3.0.11.jar:3.0.11]
> at 
> org.apache.cassandra.auth.CassandraRoleManager.collectRoles(CassandraRoleManager.java:480)
>  ~[apache-cassandra-3.0.11.jar:3.0.11]
> at 
> org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:284)
>  ~[apache-cassandra-3.0.11.jar:3.0.11]
> at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:122) 
> ~[apache-cassandra-3.0.11.jar:3.0.11]
> at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:119) 
> ~[apache-cassandra-3.0.11.jar:3.0.11]
> at 
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
>  ~[guava-18.0.jar:na]
> at 
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) 
> ~[guava-18.0.jar:na]
> at 
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
>  ~[guava-18.0.jar:na]
> at 
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) 
> ~[guava-18.0.jar:na]
> ... 38 common frames omitted
> Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot 
> achieve consistency level LOCAL_ONE
>  
>  
> I have no idea what to do, so any help would be amazing right now..
>  
> Thanks
> J
>  
> Jonathan Baynes
> DBA
> Tradeweb Europe Limited
> Moor Place  •  1 Fore Street Avenue  •  London EC2Y 9DT
> P +44 (0)20 77760988  •  F +44 (0)20 7776 3201  •  M +44 (0) xx
> jonathan.bay...@tradeweb.com 
>  
>     follow us:   
> 
> 
> —
> A leading marketplace  for 
> electronic fixed income, derivatives and ETF trading
>  
> 
> 
> This e-mail may contain confidential and/or privileged information. If you 
> are not the intended recipient (or have received this e-mail in error) please 
> notify the sender immediately and destroy it. Any unauthorized copying, 
> disclosure or distribution of the material in this e-mail is strictly 
> forbidden. Tradeweb reserves the right to monitor all e-mail communications 
> through its networks. If you do not wish to receive marketing emails about 
> our products / services, please let us know by contacting us, either by email 
> at contac...@tradeweb.com  or by writing to us 
> at the registered office of Tradeweb in the UK, which is: Tradeweb Europe 
> Limited (company number 3912826), 1 Fore Street Avenue London EC2Y 9DT. To 
> see our privacy policy, visit our website @ www.tradeweb.com 
> .
>

Re: C* 3 node issue -Urgent

2017-08-23 Thread Akhil Mehra

I am assuming the following guide or similar was followed to add JMX 
authentication:

http://docs.datastax.com/en/cassandra/3.0/cassandra/configuration/secureJmxAuthentication.html
 
<http://docs.datastax.com/en/cassandra/3.0/cassandra/configuration/secureJmxAuthentication.html>



> On 23/08/2017, at 9:14 PM, Jonathan Baynes <jonathan.bay...@tradeweb.com> 
> wrote:
> 
> When trying to connect to CQLSH I get this
>  
> “Cannot achieve consistency level QUORUM”
>  
> 2 of my 3 nodes are down. So this error is correct. But how do I sign into 
> CQLSH to change this? Or better, how do I get the other 2 nodes back up?
>  
> From: Akhil Mehra [mailto:akhilme...@gmail.com <mailto:akhilme...@gmail.com>] 
> Sent: 23 August 2017 10:05
> To: user@cassandra.apache.org <mailto:user@cassandra.apache.org>
> Subject: Re: C* 3 node issue -Urgent
>  
> The cqlsh image say bad credentials. Just confirming that you have the 
> correct username/password when logging on.
> 
> By turing on authentication I am assuming you mean using the 
> PasswordAuthenticator instead of the AllowAllAuthenticator in the yaml.
> 
> Cheers,
> Akhil
> 
> On 23/08/2017, at 8:59 PM, Jonathan Baynes <jonathan.bay...@tradeweb.com 
> <mailto:jonathan.bay...@tradeweb.com>> wrote:
>  
> I will also  mention I am on:
>  
> C* 3.0.11
> Linux Oracle red hat 7.1
> Java 1.8.0.31
> Python 2.7
>  
> From: Jonathan Baynes 
> Sent: 23 August 2017 09:47
> To: 'user@cassandra.apache.org <mailto:user@cassandra.apache.org>'
> Cc: Stewart Allman
> Subject: C* 3 node issue -Urgent
>  
> Hi Everyone.
>  
> I  need the communities help here.
>  
> I have attempted this morning to turn on JMX authentication for Nodetool. 
> I’ve gone into the Cassandra-env.sh file and updated the following:
>  
> LOCAL_JMX=No
> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
> JVM_OPTS="$JVM_OPTS 
> -Dcom.sun.management.jmxremote.password.file=/opt/cassandra/application/jmxremote.password"
>  
> Then I restarted each node, I have 3 of them  in a single DC, Single Rack.
> 115.63 (seed)
> 115.64
> 115.65 (seed)
>  
> Nodetool status shows this
>  
> 
>  
> In the Yaml I have Authentication turned ON.
> Cassandra has been working flawlessly up until the moment I restarted the 
> nodes.
> 
> as it stands right now.. 
>  
> 115.63 is up, but I cannot connect to CQLSH
> I get the following error 
> 
>  
> When I check the logs I on 115.64 and 115.65 I can see this
>  
> WARN  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:355 
> - CassandraRoleManager skipped default role setup: some nodes were not ready
> INFO  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:394 
> - Setup task failed with error, rescheduling
> Caused by: java.lang.RuntimeException: 
> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve 
> consistency level LOCAL_ONE
> at 
> org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:512)
>  ~[apache-cassandra-3.0.11.jar:3.0.11]
> at 
> org.apache.cassandra.auth.CassandraRoleManager.collectRoles(CassandraRoleManager.java:480)
>  ~[apache-cassandra-3.0.11.jar:3.0.11]
> at 
> org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:284)
>  ~[apache-cassandra-3.0.11.jar:3.0.11]
> at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:122) 
> ~[apache-cassandra-3.0.11.jar:3.0.11]
> at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:119) 
> ~[apache-cassandra-3.0.11.jar:3.0.11]
> at 
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
>  ~[guava-18.0.jar:na]
> at 
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) 
> ~[guava-18.0.jar:na]
> at 
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
>  ~[guava-18.0.jar:na]
> at 
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) 
> ~[guava-18.0.jar:na]
> ... 38 common frames omitted
> Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot 
> achieve consistency level LOCAL_ONE
>  
>  
> I have no idea what to do, so any help would be amazing right now..
>  
> Thanks
> J
>  
> Jonathan Baynes
> DBA
> Tradeweb Europe Limited
> Moor Place  •  1 Fore Street Avenue  •  London EC2Y 9DT
> P +44 (0)20 77760988  •  F +44 (0)20 7776 3201  •  M +44 (0) xx
> jonathan.bay...@tradeweb.com <mailto:jonathan.bay...@tradeweb.com>
>  
>  <http://www.tradeweb.

Re: C* 3 node issue -Urgent

2017-08-23 Thread Akhil Mehra

You could try reverting your JMX authentication changes for the time being if 
getting you nodes up is a priority.

At least you will be able to isolate the problem i.e. is it the Password 
authenticator or the jmx changes causing the problem.

Cheers,
Akhil

PS Sorry for the silly questions just trying to eliminate the obvious.

> On 23/08/2017, at 9:14 PM, Jonathan Baynes <jonathan.bay...@tradeweb.com> 
> wrote:
> 
> When trying to connect to CQLSH I get this
>  
> “Cannot achieve consistency level QUORUM”
>  
> 2 of my 3 nodes are down. So this error is correct. But how do I sign into 
> CQLSH to change this? Or better, how do I get the other 2 nodes back up?
>  
> From: Akhil Mehra [mailto:akhilme...@gmail.com <mailto:akhilme...@gmail.com>] 
> Sent: 23 August 2017 10:05
> To: user@cassandra.apache.org <mailto:user@cassandra.apache.org>
> Subject: Re: C* 3 node issue -Urgent
>  
> The cqlsh image say bad credentials. Just confirming that you have the 
> correct username/password when logging on.
> 
> By turing on authentication I am assuming you mean using the 
> PasswordAuthenticator instead of the AllowAllAuthenticator in the yaml.
> 
> Cheers,
> Akhil
> 
> On 23/08/2017, at 8:59 PM, Jonathan Baynes <jonathan.bay...@tradeweb.com 
> <mailto:jonathan.bay...@tradeweb.com>> wrote:
>  
> I will also  mention I am on:
>  
> C* 3.0.11
> Linux Oracle red hat 7.1
> Java 1.8.0.31
> Python 2.7
>  
> From: Jonathan Baynes 
> Sent: 23 August 2017 09:47
> To: 'user@cassandra.apache.org <mailto:user@cassandra.apache.org>'
> Cc: Stewart Allman
> Subject: C* 3 node issue -Urgent
>  
> Hi Everyone.
>  
> I  need the communities help here.
>  
> I have attempted this morning to turn on JMX authentication for Nodetool. 
> I’ve gone into the Cassandra-env.sh file and updated the following:
>  
> LOCAL_JMX=No
> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
> JVM_OPTS="$JVM_OPTS 
> -Dcom.sun.management.jmxremote.password.file=/opt/cassandra/application/jmxremote.password"
>  
> Then I restarted each node, I have 3 of them  in a single DC, Single Rack.
> 115.63 (seed)
> 115.64
> 115.65 (seed)
>  
> Nodetool status shows this
>  
> 
>  
> In the Yaml I have Authentication turned ON.
> Cassandra has been working flawlessly up until the moment I restarted the 
> nodes.
> 
> as it stands right now.. 
>  
> 115.63 is up, but I cannot connect to CQLSH
> I get the following error 
> 
>  
> When I check the logs I on 115.64 and 115.65 I can see this
>  
> WARN  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:355 
> - CassandraRoleManager skipped default role setup: some nodes were not ready
> INFO  [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:394 
> - Setup task failed with error, rescheduling
> Caused by: java.lang.RuntimeException: 
> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve 
> consistency level LOCAL_ONE
> at 
> org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:512)
>  ~[apache-cassandra-3.0.11.jar:3.0.11]
> at 
> org.apache.cassandra.auth.CassandraRoleManager.collectRoles(CassandraRoleManager.java:480)
>  ~[apache-cassandra-3.0.11.jar:3.0.11]
> at 
> org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:284)
>  ~[apache-cassandra-3.0.11.jar:3.0.11]
> at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:122) 
> ~[apache-cassandra-3.0.11.jar:3.0.11]
> at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:119) 
> ~[apache-cassandra-3.0.11.jar:3.0.11]
> at 
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
>  ~[guava-18.0.jar:na]
> at 
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) 
> ~[guava-18.0.jar:na]
> at 
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
>  ~[guava-18.0.jar:na]
> at 
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) 
> ~[guava-18.0.jar:na]
> ... 38 common frames omitted
> Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot 
> achieve consistency level LOCAL_ONE
>  
>  
> I have no idea what to do, so any help would be amazing right now..
>  
> Thanks
> J
>  
> Jonathan Baynes
> DBA
> Tradeweb Europe Limited
> Moor Place  •  1 Fore Street Avenue  •  London EC2Y 9DT
> P +44 (0)20 77760988  •  F +44 (0)20 7776 3201  •  M +44 (0) xx
> jonathan.bay...@tradeweb.com <mailto:jonathan.bay...

Re: memtable_allocation_type on Cassandra 2.1.x

2017-05-22 Thread Akhil Mehra

I believe off-heap storage was reintroduced in 3.4 (
https://issues.apache.org/jira/browse/CASSANDRA-9472). It was removed from
3.0 due to the refactoring of the storage engine.  Check out
http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1 to get
an overview of the pros and cons of using off-heap storage.

Regards,
Akhil Mehra

On Tue, May 23, 2017 at 12:32 AM, varun saluja <saluj...@gmail.com> wrote:

> Hi Experts,
>
> I have some concerns regarding memtable parameters for my current version.
> 2.1.8.
>
> As per documentation , its mentioned to have Off-heap memtables in
> Cassandra 2.1 . And in releases 3.2.0 and 3.2.1, the only option that works
> is: heap-buffers.
> Can you Please suggest what value should be use for below paramteres in
> 2.1.x
> memtable_allocation_type :
>
> Regards,
> Varun Saluja
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: memtable_allocation_type on Cassandra 2.1.x

2017-05-24 Thread Akhil Mehra

Hi Varun,

Look at the recommendation for offheap_objects and memtable flush writers
and readers in the following guide
https://tobert.github.io/pages/als-cassandra-21-tuning-guide.html. In the
guide and cassandra.yaml the default is suggested as a good starting point.

If you want to use the default just omit the setting
of memtable_heap_space_in_mb instead of setting it to 0. Note in newer
versions of Cassandra setting the memtable_cleanup_threshold is deprecated.
The default value is said to be the only reasonable setting.

Regards,
Akhil


On Wed, May 24, 2017 at 1:41 PM, varun saluja <saluj...@gmail.com> wrote:

> Thanks Akhil for response.
>
> I have set memtable_allocation_type as Off-heap. But cassandra 2.1.x does
> not allow to set *memtable_heap_space_in_mb: 0.*
>
> It mentions , we need to assign some positive value to heap space. In such
> case, will memtable still use jvm heap space.
>
> Can anyone suggest below parameters.
>
> memtable_flush_writers:
> memtable_cleanup_threshold:
>
>
> PS : We have high write intensive workload . 5Node cluster (12 Core , 62GB
> RAM and flash disk per node)
>
>
> Regards,
> Varun Saluja
>
> On 23 May 2017 at 03:26, Akhil Mehra <akhilme...@gmail.com> wrote:
>
>> I believe off-heap storage was reintroduced in 3.4 (
>> https://issues.apache.org/jira/browse/CASSANDRA-9472). It was removed
>> from 3.0 due to the refactoring of the storage engine.  Check out
>> http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1 to
>> get an overview of the pros and cons of using off-heap storage.
>>
>> Regards,
>> Akhil Mehra
>>
>>
>> On Tue, May 23, 2017 at 12:32 AM, varun saluja <saluj...@gmail.com>
>> wrote:
>>
>>> Hi Experts,
>>>
>>> I have some concerns regarding memtable parameters for my current
>>> version. 2.1.8.
>>>
>>> As per documentation , its mentioned to have Off-heap memtables in
>>> Cassandra 2.1 . And in releases 3.2.0 and 3.2.1, the only option that works
>>> is: heap-buffers.
>>> Can you Please suggest what value should be use for below paramteres in
>>> 2.1.x
>>> memtable_allocation_type :
>>>
>>> Regards,
>>> Varun Saluja
>>>
>>>
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>>
>>>
>>
>

Re: [Cassandra] Ignoring interval time

2017-05-30 Thread Akhil Mehra

The debug output is from the failure detector in the gossip module. Code
can be found here
https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/src/java/org/apache/cassandra/gms/FailureDetector.java#L450-L474
.

The debug logging above is reporting around an acknowledgement latency
between  2 to 3 seconds for each gossip message sent by the node to the
respective IP addresses. This is above the expected 2 sec threshold as
stated in MAX_INTERVAL_IN_NANO.

As Varun pointed out your cluster is probably under too much load.

The pressure on your cluster might also be causing the read repairs you
reported in your earlier email.

Regards,
Akhil


On Wed, May 31, 2017 at 7:21 AM, Varun Gupta  wrote:

> Can you please check Cassandra Stats, if cluster is under too much load.
> This is the symptom, not the root cause.
>
> On Tue, May 30, 2017 at 2:33 AM, Abhishek Kumar Maheshwari <
> abhishek.maheshw...@timesinternet.in> wrote:
>
>> Hi All,
>>
>>
>>
>> Please let me know why this debug log is coming:
>>
>>
>>
>> DEBUG [GossipStage:1] 2017-05-30 15:01:31,496 FailureDetector.java:456 -
>> Ignoring interval time of 2000686406 for /XXX.XX.XXX.204
>>
>> DEBUG [GossipStage:1] 2017-05-30 15:01:34,497 FailureDetector.java:456 -
>> Ignoring interval time of 2349724693 <(234)%20972-4693> for
>> /XXX.XX.XXX.207
>>
>> DEBUG [GossipStage:1] 2017-05-30 15:01:34,497 FailureDetector.java:456 -
>> Ignoring interval time of 2000655389 for /XXX.XX.XXX.206
>>
>> DEBUG [GossipStage:1] 2017-05-30 15:01:34,497 FailureDetector.java:456 -
>> Ignoring interval time of 2000721304 for /XXX.XX.XXX.201
>>
>> DEBUG [GossipStage:1] 2017-05-30 15:01:34,497 FailureDetector.java:456 -
>> Ignoring interval time of 2000770809 for /XXX.XX.XXX.202
>>
>> DEBUG [GossipStage:1] 2017-05-30 15:01:34,497 FailureDetector.java:456 -
>> Ignoring interval time of 2000825217 for /XXX.XX.XXX.209
>>
>> DEBUG [GossipStage:1] 2017-05-30 15:01:35,449 FailureDetector.java:456 -
>> Ignoring interval time of 2953167747 for /XXX.XX.XXX.205
>>
>> DEBUG [GossipStage:1] 2017-05-30 15:01:37,497 FailureDetector.java:456 -
>> Ignoring interval time of 2047662469 <(204)%20766-2469> for
>> /XXX.XX.XXX.205
>>
>> DEBUG [GossipStage:1] 2017-05-30 15:01:37,497 FailureDetector.java:456 -
>> Ignoring interval time of 2000717144 for /XXX.XX.XXX.207
>>
>> DEBUG [GossipStage:1] 2017-05-30 15:01:37,497 FailureDetector.java:456 -
>> Ignoring interval time of 2000780785 for /XXX.XX.XXX.201
>>
>> DEBUG [GossipStage:1] 2017-05-30 15:01:38,497 FailureDetector.java:456 -
>> Ignoring interval time of 2000113606 for /XXX.XX.XXX.209
>>
>> DEBUG [GossipStage:1] 2017-05-30 15:01:39,121 FailureDetector.java:456 -
>> Ignoring interval time of 2334491585 for /XXX.XX.XXX.204
>>
>> DEBUG [GossipStage:1] 2017-05-30 15:01:39,497 FailureDetector.java:456 -
>> Ignoring interval time of 2000209788 for /XXX.XX.XXX.207
>>
>> DEBUG [GossipStage:1] 2017-05-30 15:01:39,497 FailureDetector.java:456 -
>> Ignoring interval time of 2000226568 for /XXX.XX.XXX.208
>>
>> DEBUG [GossipStage:1] 2017-05-30 15:01:42,178 FailureDetector.java:456 -
>> Ignoring interval time of 2390977968 for /XXX.XX.XXX.204
>>
>>
>>
>> *Thanks & Regards,*
>> *Abhishek Kumar Maheshwari*
>> *+91- 805591 <+91%208%2005591> (Mobile)*
>>
>> Times Internet Ltd. | A Times of India Group Company
>>
>> FC - 6, Sector 16A, Film City,  Noida,  U.P. 201301 | INDIA
>>
>> *P** Please do not print this email unless it is absolutely necessary.
>> Spread environmental awareness.*
>>
>>
>>
>
>

Re: How to avoid flush if the data can fit into memtable

2017-05-29 Thread Akhil Mehra

Hi Preetika,

After thinking about your scenario I believe your small SSTable size might
be due to data compression. By default, all tables enable SSTable
compression.

Let go through your scenario. Let's say you have allocated 4GB to your
Cassandra node. Your *memtable_heap_space_in_mb* and

*memtable_offheap_space_in_mb  *will roughly come to around 1GB. Since you
have memtable_cleanup_threshold to .50 table cleanup will be triggered when
total allocated memtable space exceeds 1/2GB. Note the cleanup threshold is
.50 of 1GB and not a combination of heap and off heap space. This memtable
allocation size is the total amount available for all tables on your node.
This includes all system related keyspaces. The cleanup process will write
the largest memtable to disk.

For your case, I am assuming that you are on a *single node with only one
table with insert activity*. I do not think the commit log will trigger a
flush in this circumstance as by default the commit log has 8192 MB of
space unless the commit log is placed on a very small disk.

I am assuming your table on disk is smaller than 500MB because of
compression. You can disable compression on your table and see if this
helps get the desired size.

I have written up a blog post explaining memtable flushing (
http://abiasforaction.net/apache-cassandra-memtable-flush/)

Let me know if you have any other question.

I hope this helps.

Regards,
Akhil Mehra

On Fri, May 26, 2017 at 6:58 AM, preetika tyagi <preetikaty...@gmail.com>
wrote:

> I agree that for such a small data, Cassandra is obviously not needed.
> However, this is purely an experimental setup by using which I'm trying to
> understand how and exactly when memtable flush is triggered. As I mentioned
> in my post, I read the documentation and tweaked the parameters accordingly
> so that I never hit memtable flush but it is still doing that. As far the
> the setup is concerned, I'm just using 1 node and running Cassandra using
> "cassandra -R" option and then running some queries to insert some dummy
> data.
>
> I use the schema from CASSANDRA_HOME/tools/cqlstress-insanity-example.yaml
> and add "durable_writes=false" in the keyspace_definition.
>
> @Daemeon - The previous post lead to this post but since I was unaware of
> memtable flush and I assumed memtable flush wasn't happening, the previous
> post was about something else (throughput/latency etc.). This post is
> explicitly about exactly when memtable is being dumped to the disk. Didn't
> want to confuse two different goals that's why posted a new one.
>
> On Thu, May 25, 2017 at 10:38 AM, Avi Kivity <a...@scylladb.com> wrote:
>
>> It doesn't have to fit in memory. If your key distribution has strong
>> temporal locality, then a larger memtable that can coalesce overwrites
>> greatly reduces the disk I/O load for the memtable flush and subsequent
>> compactions. Of course, I have no idea if the is what the OP had in mind.
>>
>>
>> On 05/25/2017 07:14 PM, Jonathan Haddad wrote:
>>
>> Sorry for the confusion.  That was for the OP.  I wrote it quickly right
>> after waking up.
>>
>> What I'm asking is why does the OP want to keep his data in the memtable
>> exclusively?  If the goal is to "make reads fast", then just turn on row
>> caching.
>>
>> If there's so little data that it fits in memory (300MB), and there
>> aren't going to be any writes past the initial small dataset, why use
>> Cassandra?  It sounds like the wrong tool for this job.  Sounds like
>> something that could easily be stored in S3 and loaded in memory when the
>> app is fired up.
>>
>> On Thu, May 25, 2017 at 8:06 AM Avi Kivity <a...@scylladb.com> wrote:
>>
>>> Not sure whether you're asking me or the original poster, but the more
>>> times data gets overwritten in a memtable, the less it has to be compacted
>>> later on (and even without overwrites, larger memtables result in less
>>> compaction).
>>>
>>> On 05/25/2017 05:59 PM, Jonathan Haddad wrote:
>>>
>>> Why do you think keeping your data in the memtable is a what you need to
>>> do?
>>> On Thu, May 25, 2017 at 7:16 AM Avi Kivity <a...@scylladb.com> wrote:
>>>
>>>> Then it doesn't have to (it still may, for other reasons).
>>>>
>>>> On 05/25/2017 05:11 PM, preetika tyagi wrote:
>>>>
>>>> What if the commit log is disabled?
>>>>
>>>> On May 25, 2017 4:31 AM, "Avi Kivity" <a...@scylladb.com> wrote:
>>>>
>>>>> Cassandra has to flush the memtable occasionally, or the commit log
>>>>> grows without bounds.
>>>>>
>&

Re: org.apache.cassandra.service.DigestMismatchException: Mismatch for key DecoratedKey

2017-05-30 Thread Akhil Mehra

This blog post 
(http://thelastpickle.com/blog/2011/05/15/Deletes-and-Tombstones.html 
) 
provides good explenation on the exception in your debug log.

Regards,
Akhil

> On 30/05/2017, at 9:29 PM, Abhishek Kumar Maheshwari 
>  wrote:
> 
> Hi All,
>  
> I am getting below exception in debug.log.
>  
> DEBUG [ReadRepairStage:636754] 2017-05-30 14:49:44,259 ReadCallback.java:234 
> - Digest mismatch:
> org.apache.cassandra.service.DigestMismatchException: Mismatch for key 
> DecoratedKey(4329955402556695061, 
> 000808440801579b425c4000) 
> (343b7ef24feb594118ecb4bf7680d07f vs d41d8cd98f00b204e9800998ecf8427e)
> at 
> org.apache.cassandra.service.DigestResolver.resolve(DigestResolver.java:85) 
> ~[apache-cassandra-3.0.9.jar:3.0.9]
> at 
> org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:225)
>  ~[apache-cassandra-3.0.9.jar:3.0.9]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_101]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
>  
>  
> Please let me know why it’s coming?
>  
> Thanks & Regards,
> Abhishek Kumar Maheshwari
> +91- 805591 (Mobile)
> Times Internet Ltd. | A Times of India Group Company
> FC - 6, Sector 16A, Film City,  Noida,  U.P. 201301 | INDIA
> P Please do not print this email unless it is absolutely necessary. Spread 
> environmental awareness.
>  
>

Re: Convert single node C* to cluster (rebalancing problem)

2017-05-31 Thread Akhil Mehra

Hi Junaid,

I noticed in the log files that data is being streamed from/to 10.128.1.2 but 
the address of your original node is 10.128.0.7.

Are there any other Cassandra nodes on your local network the same cluster 
name. 

Regards,
Akhil

> 10.128.1.2
> On 31/05/2017, at 10:15 PM, Junaid Nasir  wrote:
> 
> after nodetool repair on new node following log appears in cassandra log
> 
> INFO  [StreamConnectionEstablisher:1] 2017-05-31 10:07:23,496 
> StreamSession.java:266 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac] 
> Starting streaming to /10.128.1.2
> INFO  [StreamConnectionEstablisher:1] 2017-05-31 10:07:23,501 
> StreamCoordinator.java:264 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac, 
> ID#0] Beginning stream session with /10.128.1.2
> INFO  [STREAM-IN-/10.128.1.2:7000] 2017-05-31 10:07:23,545 
> StreamResultFuture.java:173 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac 
> ID#0] Prepare completed. Receiving 2 files(131.594KiB), sending 0 
> files(0.000KiB)
> INFO  [StreamReceiveTask:1] 2017-05-31 10:07:23,705 
> StreamResultFuture.java:187 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac] 
> Session with /10.128.1.2 is complete
> INFO  [StreamReceiveTask:1] 2017-05-31 10:07:23,707 
> StreamResultFuture.java:219 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac] 
> All sessions completed
> 
> 
> 
> On Wed, May 31, 2017 at 2:50 PM, Junaid Nasir  wrote:
> Cassandra version is 3.10, and yes its not a production server. i have seen 
> some warnings in logs saying token exist on both servers. other than that 
> nothing. if you need any more settings/details please ask. thank you for your 
> time 
> 
> 
> On Wed, May 31, 2017 at 2:04 PM, Oleksandr Shulgin 
>  wrote:
> On Wed, May 31, 2017 at 9:55 AM, Junaid Nasir  wrote:
> Cassandra ensure that adding or removing nodes are very easy and that load is 
> balanced between nodes when a change is made. but it's not working in my case.
> I have a single node C* deployment (with 270 GB of data) and want to load 
> balance the data on multiple nodes,
> 
> I guess it's fair to assume this is not a production "cluster"?
>  
> I followed this guide 
> `nodetool status` shows 2 nodes but load is not balanced between them
> Datacenter: dc1
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/M
> oving
> --  Address  Load   Tokens   Owns (effective)  Host IDRack
> UN  10.128.0.7   270.75 GiB  256  48.6%
> 1a3f6faa-4376-45a8-9c20-11480a
> e5664c  rack1
> UN  10.128.0.14  414.36 KiB  256  51.4%
> 66a89fbf-08ba-4b5d-9f10-55d52a
> 199b41  rack1
> I can imagine you are referring to Load column here, which shows only 400 KB 
> for the new node.  Did the newly added node actually bootstrap?  Are there 
> any error/warning messages in the Cassandra log?
>  
> I also ran 'nodetool repair' on new node but result is same. any pointers 
> would be appreciated :)
> 
> Hm, this is not expected.  Even if the node didn't bootstrap, repair should 
> have streamed the data it is responsible for.
> 
> conf file of new node
> cluster_name: 'cluster1'
>  - seeds: "10.128.0.7"
> 
> num_tokens: 256
> endpoint_snitch: GossipingPropertyFileSnitch
> 
> I see that you've tried to add the new node as a normal (non-seed) node, 
> which supposed to work and it should have bootstrapped itself.  Which version 
> of Cassandra is it exactly?
> 
> -- 
> Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176 
> 127-59-707
> 
> 
> 


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Convert single node C* to cluster (rebalancing problem)

2017-06-02 Thread Akhil Mehra

So now the data is evenly balanced in both nodes? 

Refer to the following documentation to get a better understanding of the 
roc_address and the broadcast_rpc_address 
https://www.instaclustr.com/demystifying-cassandras-broadcast_address/ 
. I am 
surprised that your node started up with rpc_broadcast_address set as this is 
an unsupported property. I am assuming you are using Cassandra version 3.10.


Regards,
Akhil

> On 2/06/2017, at 11:06 PM, Junaid Nasir  wrote:
> 
> I am able to get it working. I added a new node with following changes
> #rpc_address:0.0.0.0
> rpc_address: 10.128.1.11
> #rpc_broadcast_address:10.128.1.11
> rpc_address was set to 0.0.0.0, (I ran into a problem previously regarding 
> remote connection and made these changes 
> https://stackoverflow.com/questions/12236898/apache-cassandra-remote-access 
> )
>  
> 
> should it be happening?
> 
> On Thu, Jun 1, 2017 at 6:31 PM, Vladimir Yudovin  > wrote:
> Did you run "nodetool cleanup" on first node after second was bootstrapped? 
> It should clean rows not belonging to node after tokens changed.
> 
> Best regards, Vladimir Yudovin, 
> Winguzone  - Cloud Cassandra Hosting
> 
> 
>  On Wed, 31 May 2017 03:55:54 -0400 Junaid Nasir  > wrote 
> 
> Cassandra ensure that adding or removing nodes are very easy and that load is 
> balanced between nodes when a change is made. but it's not working in my case.
> I have a single node C* deployment (with 270 GB of data) and want to load 
> balance the data on multiple nodes, I followed this guide 
> 
>  
> `nodetool status` shows 2 nodes but load is not balanced between them
> Datacenter: dc1
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address  Load   Tokens   Owns (effective)  Host IDRack
> UN  10.128.0.7   270.75 GiB  256  48.6%
> 1a3f6faa-4376-45a8-9c20-11480ae5664c  rack1
> UN  10.128.0.14  414.36 KiB  256  51.4%
> 66a89fbf-08ba-4b5d-9f10-55d52a199b41  rack1
> I also ran 'nodetool repair' on new node but result is same. any pointers 
> would be appreciated :)
> 
> conf file of new node
> cluster_name: 'cluster1'
>  - seeds: "10.128.0.7"
> num_tokens: 256
> endpoint_snitch: GossipingPropertyFileSnitch
> Thanks,
> Junaid
> 
>

Re: Convert single node C* to cluster (rebalancing problem)

2017-06-01 Thread Akhil Mehra

, 3213269181965324853, 
3972662756438857798, -1808499161350392500, -5552429155141285488, 
2768019514490102470, -2381885168558935166, 7598271141891576988, 
-5968675860637356104, -2178161882622813874, -2782395662355757709, 
4662660828871465894, 6726990970215445064, 691223925843765893, 
-3732536320705038428, 2169053732177722520, -3467691490997179851, 
-3201672755574011994, -706634586120752453, -7297234535099792750, 
4195063085031070570, 1024797669903232596, -5102042883065498245, 
-4295412307398568491, -686079656172478689, -7652228004418103329, 
-5922734755429174917, 6562442130946482224, 3419893918407781185, 
-7840156446781283061, -3209525913297552052, -8254134338430272746, 
-7543559272928655856, -6413145215334169356, 8189387753488304279, 
133576451402117013, 6859840908124654784, -480832477584575919, 
-4466949465409090307, 2224334850433074431, -1077941859365184025, 
7694746877711316656, -1541425238506019058, -2694798376156556512, 
-752352477219592169, 1911593773128947549, 1053380063512932771, 
2369074212175473237, 1511764544820953277, -916955813829019462, 
-3389255868702958135, -4853221732440365560, 436528405098383237, 
1482375829041075317, -5200032867873743382, -2359116065936986284, 
2323558012249686328, -8714456372962073495, 8264655970455461946, 
-6377550880012897671, 1904431293553435268, 4281876827769946039, 
129146370702108712, -7734952979230886745, -8242033642929677698, 
2814257954427384288, -3054904268940900861, -450367636631970004, 
8109127225549270088, 8020800599127010921, -2679593348517582224, 
-1752859515012359396, 4479567094782500113, 6631436750259638425, 
-5598101192868431054, 2794504876633839794, -1756288804973602742, 
-534202438069841833, 7027493063976551471, 3982737257386418900, 
-930887821667260064, 273533908252754639, -1440739443874625754, 
-2507143240715618713, 2102474087594804091, -7697404848019572605, 
-8658346826860002424, -4596011304911479578, 6459167476271897881, 
6805481881299404326, -7080476038981754572, 5767237378480651194, 
-978605465701070447, -2322705636227730109, -681083377956364436, 
-5892502600903861513, -3301298109594166057, 8752787692307586763, 
-7189731757057511702, 8976826014517372173, -6975924228178934185, 
4829963829825594588, -2886015764818641366, -7753659165428942667, 
-5923615337534732994, 7275900434506691605, 4164818915231369384, 
5154263935677458810, 301044058442370317, -7764756071469383634, 
7327453618420416121, -2412303784295409238, 781599758351322350, 
6625371277047208299, -7611076121968222555, 6127681154442724891, 
7313542022544470677, 7222174446262831041, -9138442343184845349, 
-4110447459792797189, 2534623752979121051]
INFO  [main] 2017-06-01 07:20:32,544 StorageService.java:1435 - JOINING: Finish 
joining ring

Regards,
Akhil

> On 1/06/2017, at 5:28 PM, Junaid Nasir <jna...@an10.io> wrote:
> 
> sorry that was an old notetool status output (old cluster with a few 
> configuration changes). new configuration is
> 
> --  Address  Load   Tokens   Owns (effective)  Host ID
>Rack
> UN  10.128.1.2   270.75 GiB  256  52.6% 
> e4427611-c247-42ee-9404-371e177f5f17  rack1
> UN  10.128.1.10  388.52 KiB  256  47.4% 
> 7059e622-7861-495b-a69f-5a68876364dd  rack1
> 
> 10.128.1.2 is seed node, and no other C* node is running on the network.
> data_file_directories on seed node is not in default location, it's on 
> another disk. if that helps
> 
> On Thu, Jun 1, 2017 at 4:49 AM, Akhil Mehra <akhilme...@gmail.com 
> <mailto:akhilme...@gmail.com>> wrote:
> Hi Junaid,
> 
> I noticed in the log files that data is being streamed from/to 10.128.1.2 but 
> the address of your original node is 10.128.0.7.
> 
> Are there any other Cassandra nodes on your local network the same cluster 
> name.
> 
> Regards,
> Akhil
> 
> > 10.128.1.2
> > On 31/05/2017, at 10:15 PM, Junaid Nasir <jna...@an10.io 
> > <mailto:jna...@an10.io>> wrote:
> >
> > after nodetool repair on new node following log appears in cassandra log
> >
> > INFO  [StreamConnectionEstablisher:1] 2017-05-31 10:07:23,496 
> > StreamSession.java:266 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac] 
> > Starting streaming to /10.128.1.2 <http://10.128.1.2/>
> > INFO  [StreamConnectionEstablisher:1] 2017-05-31 10:07:23,501 
> > StreamCoordinator.java:264 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac, 
> > ID#0] Beginning stream session with /10.128.1.2 <http://10.128.1.2/>
> > INFO  [STREAM-IN-/10.128.1.2:7000 <http://10.128.1.2:7000/>] 2017-05-31 
> > 10:07:23,545 StreamResultFuture.java:173 - [Stream 
> > #f0eb9670-45e8-11e7-a17b-81e9a18c6eac ID#0] Prepare completed. Receiving 2 
> > files(131.594KiB), sending 0 files(0.000KiB)
> > INFO  [StreamReceiveTask:1] 2017-05-31 10:07:23,705 
> > StreamResultFuture.java:187 -

Re: How to avoid flush if the data can fit into memtable

2017-06-01 Thread Akhil Mehra

Kevin, Stefan thanks for the positive feedback and questions.

Stefan in the blog post I am writing generally based on Apache Cassandra 
defaults. The meltable cleanup threshold is 1/(1+ memtable_flush_writers). By 
default the meltable_flush_writers defaults to two. This comes to 33 percent of 
the allocated memory. I have updated the blog post adding in this missing 
detail :)

In the email I was trying to address the OP’s original question. I mentioned .5 
because the OP had set the memtable_cleanup_threshold to .50. This is 50% of 
the allocated memory. I was also mentioning that clean up triggered when either 
on or off heap memory reaches the clean up threshold. Please refer to 
https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/src/java/org/apache/cassandra/utils/memory/MemtableCleanerThread.java#L46-L49
 
<https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/src/java/org/apache/cassandra/utils/memory/MemtableCleanerThread.java#L46-L49>
 .

I hope that helps.

Regards,
Akhil



 
> On 2/06/2017, at 2:04 AM, Stefan Litsche <stefan.lits...@zalando.de> wrote:
> 
> Hello Akhil,
> 
> thanks for your great blog post.
> One thing I cannot bring together:
> In the answer mail you write:
> "Note the cleanup threshold is .50 of 1GB and not a combination of heap and 
> off heap space."
> In your blog post you write:
> "memtable_cleanup_threshold is the default value i.e. 33 percent of the total 
> memtable heap and off heap memory."
> 
> Could you clarify this?
> 
> Thanks
> Stefan
> 
> 
> 2017-05-30 2:43 GMT+02:00 Akhil Mehra <akhilme...@gmail.com>:
> Hi Preetika,
> 
> After thinking about your scenario I believe your small SSTable size might be 
> due to data compression. By default, all tables enable SSTable compression. 
> 
> Let go through your scenario. Let's say you have allocated 4GB to your 
> Cassandra node. Your memtable_heap_space_in_mb and 
> memtable_offheap_space_in_mb  will roughly come to around 1GB. Since you have 
> memtable_cleanup_threshold to .50 table cleanup will be triggered when total 
> allocated memtable space exceeds 1/2GB. Note the cleanup threshold is .50 of 
> 1GB and not a combination of heap and off heap space. This memtable 
> allocation size is the total amount available for all tables on your node. 
> This includes all system related keyspaces. The cleanup process will write 
> the largest memtable to disk.
> 
> For your case, I am assuming that you are on a single node with only one 
> table with insert activity. I do not think the commit log will trigger a 
> flush in this circumstance as by default the commit log has 8192 MB of space 
> unless the commit log is placed on a very small disk. 
> 
> I am assuming your table on disk is smaller than 500MB because of 
> compression. You can disable compression on your table and see if this helps 
> get the desired size.
> 
> I have written up a blog post explaining memtable flushing 
> (http://abiasforaction.net/apache-cassandra-memtable-flush/)
> 
> Let me know if you have any other question. 
> 
> I hope this helps.
> 
> Regards,
> Akhil Mehra 
> 
> 
> On Fri, May 26, 2017 at 6:58 AM, preetika tyagi <preetikaty...@gmail.com> 
> wrote:
> I agree that for such a small data, Cassandra is obviously not needed. 
> However, this is purely an experimental setup by using which I'm trying to 
> understand how and exactly when memtable flush is triggered. As I mentioned 
> in my post, I read the documentation and tweaked the parameters accordingly 
> so that I never hit memtable flush but it is still doing that. As far the the 
> setup is concerned, I'm just using 1 node and running Cassandra using 
> "cassandra -R" option and then running some queries to insert some dummy data.
> 
> I use the schema from CASSANDRA_HOME/tools/cqlstress-insanity-example.yaml 
> and add "durable_writes=false" in the keyspace_definition.
> 
> @Daemeon - The previous post lead to this post but since I was unaware of 
> memtable flush and I assumed memtable flush wasn't happening, the previous 
> post was about something else (throughput/latency etc.). This post is 
> explicitly about exactly when memtable is being dumped to the disk. Didn't 
> want to confuse two different goals that's why posted a new one.
> 
> On Thu, May 25, 2017 at 10:38 AM, Avi Kivity <a...@scylladb.com> wrote:
> It doesn't have to fit in memory. If your key distribution has strong 
> temporal locality, then a larger memtable that can coalesce overwrites 
> greatly reduces the disk I/O load for the memtable flush and subsequent 
> compactions. Of course, I have no idea if the is what the OP had in mind.
> 
> 
&

Re: Convert single node C* to cluster (rebalancing problem)

2017-06-12 Thread Akhil Mehra

auto_bootstrap is true by default. Ensure its set to true. On startup look
at your logs for your auto_bootstrap value.  Look at the node configuration
line in your log file.

Akhil

On Mon, Jun 12, 2017 at 6:18 PM, Junaid Nasir <jna...@an10.io> wrote:

> No, I didn't set it (left it at default value)
>
> On Fri, Jun 9, 2017 at 3:18 AM, ZAIDI, ASAD A <az1...@att.com> wrote:
>
>> Did you make sure auto_bootstrap property is indeed set to [true] when
>> you added the node?
>>
>>
>>
>> *From:* Junaid Nasir [mailto:jna...@an10.io]
>> *Sent:* Monday, June 05, 2017 6:29 AM
>> *To:* Akhil Mehra <akhilme...@gmail.com>
>> *Cc:* Vladimir Yudovin <vla...@winguzone.com>; user@cassandra.apache.org
>> *Subject:* Re: Convert single node C* to cluster (rebalancing problem)
>>
>>
>>
>> not evenly, i have setup a new cluster with subset of data (around 5gb).
>> using the configuration above I am getting these results
>>
>>
>>
>> Datacenter: datacenter1
>>
>> ===
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> --  Address  Load   Tokens   Owns (effective)  Host ID Rack
>>
>> UN  10.128.2.1   4.86 GiB   256  44.9% 
>> e4427611-c247-42ee-9404-371e177f5f17  rack1
>>
>> UN  10.128.2.10  725.03 MiB  256 55.1% 
>> 690d5620-99d3-4ae3-aebe-8f33af54a08b  rack1
>>
>> is there anything else I can tweak/check to make the distribution even?
>>
>>
>>
>> On Sat, Jun 3, 2017 at 3:30 AM, Akhil Mehra <akhilme...@gmail.com> wrote:
>>
>> So now the data is evenly balanced in both nodes?
>>
>>
>>
>> Refer to the following documentation to get a better understanding of the
>> roc_address and the broadcast_rpc_address https://
>> www.instaclustr.com/demystifying-cassandras-broadcast_address/
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.instaclustr.com_demystifying-2Dcassandras-2Dbroadcast-5Faddress_=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk=MaTA43pugg78xQNfaOQElhyvd8k7CjVqZPr3IWALdWI=>.
>> I am surprised that your node started up with rpc_broadcast_address set as
>> this is an unsupported property. I am assuming you are using Cassandra
>> version 3.10.
>>
>>
>>
>>
>>
>> Regards,
>>
>> Akhil
>>
>>
>>
>> On 2/06/2017, at 11:06 PM, Junaid Nasir <jna...@an10.io> wrote:
>>
>>
>>
>> I am able to get it working. I added a new node with following changes
>>
>> #rpc_address:0.0.0.0
>>
>> rpc_address: 10.128.1.11
>>
>> #rpc_broadcast_address:10.128.1.11
>>
>> rpc_address was set to 0.0.0.0, (I ran into a problem previously
>> regarding remote connection and made these changes
>> https://stackoverflow.com/questions/12236898/apache-cassandr
>> a-remote-access
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_12236898_apache-2Dcassandra-2Dremote-2Daccess=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk=oj8BCLiyBDqqVQNqfGohFeujtqlzAkd-uwS878d4mg4=>
>> )
>>
>>
>>
>> should it be happening?
>>
>>
>>
>> On Thu, Jun 1, 2017 at 6:31 PM, Vladimir Yudovin <vla...@winguzone.com>
>> wrote:
>>
>> Did you run "nodetool cleanup" on first node after second was
>> bootstrapped? It should clean rows not belonging to node after tokens
>> changed.
>>
>>
>>
>> Best regards, Vladimir Yudovin,
>>
>> *Winguzone
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__winguzone.com_-3Ffrom-3Dlist=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk=Q1M5YRAsw0iUQKOIulEmO72RhdENQCRhpqZSjgxxHos=>
>> - Cloud Cassandra Hosting*
>>
>>
>>
>>
>>
>>  On Wed, 31 May 2017 03:55:54 -0400 *Junaid Nasir <jna...@an10.io
>> <jna...@an10.io>>* wrote 
>>
>>
>>
>> Cassandra ensure that adding or removing nodes are very easy and that
>> load is balanced between nodes when a change is made. but it's not working
>> in my case.
>>
>> I have a single node C* deployment (with 270 GB of data) and want to load
>> balance the data on multiple nodes, I followed this guide
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.datastax.com_en_cassandra_2.1_cassandra_operations_ops-5Fadd-5Fnode-5Fto-5Fcluster-5Ft.h

Re: Data in multi disks is not evenly distributed

2017-06-13 Thread Akhil Mehra

Hi,

I came across the following method (
https://github.com/apache/cassandra/blob/afd68abe60742c6deb6357ba4605268dfb3d06ea/src/java/org/apache/cassandra/service/StorageService.java#L5006-L5021).
It seems data is evenly split across disks according to local token ranges.

It might be that data stored is not evenly spread across your partition key
and thus the imbalance in disk usage.

Regards,
Akhil



On Mon, Jun 12, 2017 at 4:59 PM, Erick Ramirez  wrote:

> That's the cause of the imbalance -- an excessively large sstable which
> suggests to me that at some point you performed a manual major compaction
> with nodetool compact.
>
> If the table is using STCS, there won't be other compaction partners in
> the near future so you split the sstable manually with sstablesplit
> (offline tool so requires C* on the node to be shutdown temporarily).
>
>
> On Mon, Jun 12, 2017 at 1:25 PM, Xihui He  wrote:
>
>> Hi Vladimir,
>>
>> The disks size are all the same, 1.8T as show in df, and only used by
>> cassandra.
>> It seems to me that maybe the compacted lagetst file is on data01 which
>> uses 1.1T.
>>
>> Thanks,
>> Xihui
>>
>> On 11 June 2017 at 17:26, Vladimir Yudovin  wrote:
>>
>>> Hi,
>>>
>>> Do your disks have the same size? AFAK Cassandra distributes data with
>>> proportion to disk size, i.e. keeps the same percent of busy space.
>>>
>>> Best regards, Vladimir Yudovin,
>>> *Winguzone  - Cloud Cassandra Hosting*
>>>
>>>
>>>  On Wed, 07 Jun 2017 06:15:48 -0400 *Xihui He >> >* wrote 
>>>
>>> Dear All,
>>>
>>> We are using multiple disks per node and find the data is not evenly
>>> distributed (data01 uses 1.1T, but data02 uses 353G). Is this expected? If
>>> data01 becomes full, would the node be still writable? We are using 2.2.6.
>>>
>>> Thanks,
>>> Xihui
>>>
>>> data_file_directories:
>>> - /data00/cassandra
>>> - /data01/cassandra
>>> - /data02/cassandra
>>> - /data03/cassandra
>>> - /data04/cassandra
>>>
>>> df
>>> /dev/sde1   1.8T  544G  1.2T  32% /data03
>>> /dev/sdc1   1.8T  1.1T  683G  61% /data01
>>> /dev/sdf1   1.8T  491G  1.3T  29% /data04
>>> /dev/sdd1   1.8T  353G  1.4T  21% /data02
>>> /dev/sdb1   1.8T  285G  1.5T  17% /data00
>>>
>>> root@n9-016-015:~# du -sh /data01/cassandra/album_media_feature/*
>>> 143M /data01/cassandra/album_media_feature/media_feature_blur-066
>>> e5700c41511e5beacf197ae340934
>>> 4.4G /data01/cassandra/album_media_feature/media_feature_c1-dbadf
>>> 930c41411e5974743d3a691d887
>>> 56K /data01/cassandra/album_media_feature/media_feature_duplicat
>>> e-09d4b380c41511e58501e9aa37be91a5
>>> 16K /data01/cassandra/album_media_feature/media_feature_emotion-
>>> b8570470054d11e69fb88f073bab8267
>>> 240M /data01/cassandra/album_media_feature/media_feature_exposure
>>> -f55449c0c41411e58f5c9b66773b60c3
>>> 649M /data01/cassandra/album_media_feature/media_feature_group-f8
>>> de0cc0c41411e5827b995f709095c8
>>> 22G /data01/cassandra/album_media_feature/media_feature_multi_cl
>>> ass-cf3bb72006c511e69fb88f073bab8267
>>> 44K /data01/cassandra/album_media_feature/media_feature_pool5-11
>>> 85b200c41511e5b7d8757e25e34d67
>>> 15G /data01/cassandra/album_media_feature/media_feature_poster-f
>>> cf45850c41411e597bb1507d1856305
>>> 8.0K /data01/cassandra/album_media_feature/media_feature_quality-
>>> 155d9500c41511e5974743d3a691d887
>>> 17G /data01/cassandra/album_media_feature/media_feature_quality_
>>> rc-51babf50dba811e59fb88f073bab8267
>>> 8.7G /data01/cassandra/album_media_feature/media_feature_scene-00
>>> 8a5050c41511e59ebcc3582d286c8d
>>> 8.0K /data01/cassandra/album_media_feature/media_region_features_
>>> v4-29a0cd10150611e6bd3e3f41faa2612a
>>> 971G /data01/cassandra/album_media_feature/media_region_features_
>>> v5-1b805470a3d711e68121757e9ac51b7b
>>>
>>> root@n9-016-015:~# du -sh /data02/cassandra/album_media_feature/*
>>> 1.6G /data02/cassandra/album_media_feature/media_feature_blur-066
>>> e5700c41511e5beacf197ae340934
>>> 44G /data02/cassandra/album_media_feature/media_feature_c1-dbadf
>>> 930c41411e5974743d3a691d887
>>> 64K /data02/cassandra/album_media_feature/media_feature_duplicat
>>> e-09d4b380c41511e58501e9aa37be91a5
>>> 75G /data02/cassandra/album_media_feature/media_feature_emotion-
>>> b8570470054d11e69fb88f073bab8267
>>> 2.0G /data02/cassandra/album_media_feature/media_feature_exposure
>>> -f55449c0c41411e58f5c9b66773b60c3
>>> 21G /data02/cassandra/album_media_feature/media_feature_group-f8
>>> de0cc0c41411e5827b995f709095c8
>>> 336M /data02/cassandra/album_media_feature/media_feature_multi_cl
>>> ass-cf3bb72006c511e69fb88f073bab8267
>>> 44K /data02/cassandra/album_media_feature/media_feature_pool5-11
>>> 85b200c41511e5b7d8757e25e34d67
>>> 2.0G /data02/cassandra/album_media_feature/media_feature_poster-f
>>> cf45850c41411e597bb1507d1856305
>>> 8.0K

Re: Convert single node C* to cluster (rebalancing problem)

2017-06-15 Thread Akhil Mehra

Hi,

I put together a blog explaining possible reasons for an unbalanced
Cassandra nodes.

http://abiasforaction.net/unbalanced-cassandra-cluster/

Let me know if you have any questions.

Cheers,
Akhil


On Thu, Jun 15, 2017 at 5:54 PM, Affan Syed <as...@an10.io> wrote:

> John,
>
> I am a co-worker with Junaid -- he is out sick, so just wanted to confirm
> that one of your shots in the dark is correct. This is a RF of 1x
>
> "CREATE KEYSPACE orion WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '1'}  AND durable_writes = true;"
>
> However, how does the RF affect the redistribution of key/data?
>
> Affan
>
> - Affan
>
> On Wed, Jun 14, 2017 at 1:16 AM, John Hughes <johnthug...@gmail.com>
> wrote:
>
>> OP, I was just looking at your original numbers and I have some questions:
>>
>> 270GB on one node and 414KB on the other, but something close to 50/50 on
>> "Owns(effective)".
>> What replication factor are your keyspaces set up with? 1x or 2x or ??
>>
>> I would say you are seeing 50/50 because the tokens are allocated
>> 50/50(others on the list please correct what are for me really just
>> assumptions), but I would hazard a guess that your replication factor
>> is still 1x, so it isn't moving anything around. Or your keyspace
>> rplication is incorrect and isn't being distributed(I have had issues with
>> the AWSMultiRegionSnitch and not getting the region correct[us-east vs
>> us-east-1). It doesn't throw an error, but it doesn't work very well either
>> =)
>>
>> Can you do a 'describe keyspace XXX' and show the first line(the CREATE
>> KEYSPACE line).
>>
>> Mind you, these are all just shots in the dark from here.
>>
>> Cheers,
>>
>>
>> On Tue, Jun 13, 2017 at 3:13 AM Junaid Nasir <jna...@an10.io> wrote:
>>
>>> Is the OP expecting a perfect 50%/50% split?
>>>
>>>
>>> best result I got was 240gb/30gb split, which I think is not properly
>>> balanced.
>>>
>>>
>>>> Also, what are your outputs when you call out specific keyspaces? Do
>>>> the numbers get more even?
>>>
>>>
>>> i don't know what you mean by *call out specific key spaces?* can you
>>> please explain that a bit.
>>>
>>>
>>> If your schema is not modelled correctly you can easily end up unevenly
>>>> distributed data.
>>>
>>>
>>> I think that is the problem. initial 270gb data might not by modeled
>>> correctly. I have run a lot of tests on 270gb data including downsizing it
>>> to 5gb, they all resulted in same uneven distribution. I also tested a
>>> dummy dataset of 2gb which was balanced evenly. coming from rdb, I didn't
>>> give much thought to data modeling. can anyone please point me to some
>>> resources regarding this problem.
>>>
>>> On Tue, Jun 13, 2017 at 3:24 AM, Akhil Mehra <akhilme...@gmail.com>
>>> wrote:
>>>
>>>> Great point John.
>>>>
>>>> The OP should also note that data distribution also depends on your
>>>> schema and incoming data profile.
>>>>
>>>> If your schema is not modelled correctly you can easily end up unevenly
>>>> distributed data.
>>>>
>>>> Cheers,
>>>> Akhil
>>>>
>>>> On Tue, Jun 13, 2017 at 3:36 AM, John Hughes <johnthug...@gmail.com>
>>>> wrote:
>>>>
>>>>> Is the OP expecting a perfect 50%/50% split? That, to my experience,
>>>>> is not going to happen, it is almost always shifted from a fraction of a
>>>>> percent to a couple percent.
>>>>>
>>>>> Datacenter: eu-west
>>>>> ===
>>>>> Status=Up/Down
>>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>> --  AddressLoad   Tokens   Owns (effective)  Host ID
>>>>> Rack
>>>>> UN  XX.XX.XX.XX22.71 GiB  256  47.6%
>>>>> 57dafdde-2f62-467c-a8ff-c91e712f89c9  1c
>>>>> UN  XX.XX.XX.XX  17.17 GiB  256  51.3%
>>>>> d2a65c51-087d-48de-ae1f-a41142eb148d  1b
>>>>> UN  XX.XX.XX.XX  26.15 GiB  256  52.4%
>>>>> acf5dd34-5b81-4e5b-b7be-85a7fccd8e1c  1c
>>>>> UN  XX.XX.XX.XX   16.64 GiB  256  50.2%
>>>>> 6c8842dd-a966-467c-a7bc-bd6269ce3e7e  1a
>>>>> UN  XX.XX.XX.XX  24.39 GiB  256  49

Re: Convert single node C* to cluster (rebalancing problem)

2017-06-16 Thread Akhil Mehra

Good point Varun.

Here is how I understand it but feel free to disagree.

According to the documentation the load column in the nodetool status
output is "The amount of file system data under the cassandra data
directory after excluding all content in the snapshots subdirectories.
Because all SSTable data files are included, any data that is not cleaned
up, such as TTL-expired cell or tombstoned data) is counted.".

nodetool cleanup has been specifically added to remove unwanted data after
adding a new node to the cluster.

I am assuming its too expensive to calculate the actual load amount
excluding unwanted data.

Thus it seems the tool is working according to specification.

Cheers,
Akhil




On Fri, Jun 16, 2017 at 4:39 PM, Varun Gupta <var...@uber.com> wrote:

>
> Akhil,
>
> As per the blog, nodetool status shows data size for node1 even for token
> ranges it does not own. Ain't this is bug in Cassandra?
>
> Yes, on disk data will be present but it should be reflected in nodetool
> status.
>
> On Thu, Jun 15, 2017 at 6:17 PM, Akhil Mehra <akhilme...@gmail.com> wrote:
>
>> Hi,
>>
>> I put together a blog explaining possible reasons for an unbalanced
>> Cassandra nodes.
>>
>> http://abiasforaction.net/unbalanced-cassandra-cluster/
>>
>> Let me know if you have any questions.
>>
>> Cheers,
>> Akhil
>>
>>
>> On Thu, Jun 15, 2017 at 5:54 PM, Affan Syed <as...@an10.io> wrote:
>>
>>> John,
>>>
>>> I am a co-worker with Junaid -- he is out sick, so just wanted to
>>> confirm that one of your shots in the dark is correct. This is a RF of 1x
>>>
>>> "CREATE KEYSPACE orion WITH replication = {'class': 'SimpleStrategy',
>>> 'replication_factor': '1'}  AND durable_writes = true;"
>>>
>>> However, how does the RF affect the redistribution of key/data?
>>>
>>> Affan
>>>
>>> - Affan
>>>
>>> On Wed, Jun 14, 2017 at 1:16 AM, John Hughes <johnthug...@gmail.com>
>>> wrote:
>>>
>>>> OP, I was just looking at your original numbers and I have some
>>>> questions:
>>>>
>>>> 270GB on one node and 414KB on the other, but something close to 50/50
>>>> on "Owns(effective)".
>>>> What replication factor are your keyspaces set up with? 1x or 2x or ??
>>>>
>>>> I would say you are seeing 50/50 because the tokens are allocated
>>>> 50/50(others on the list please correct what are for me really just
>>>> assumptions), but I would hazard a guess that your replication factor
>>>> is still 1x, so it isn't moving anything around. Or your keyspace
>>>> rplication is incorrect and isn't being distributed(I have had issues with
>>>> the AWSMultiRegionSnitch and not getting the region correct[us-east vs
>>>> us-east-1). It doesn't throw an error, but it doesn't work very well either
>>>> =)
>>>>
>>>> Can you do a 'describe keyspace XXX' and show the first line(the CREATE
>>>> KEYSPACE line).
>>>>
>>>> Mind you, these are all just shots in the dark from here.
>>>>
>>>> Cheers,
>>>>
>>>>
>>>> On Tue, Jun 13, 2017 at 3:13 AM Junaid Nasir <jna...@an10.io> wrote:
>>>>
>>>>> Is the OP expecting a perfect 50%/50% split?
>>>>>
>>>>>
>>>>> best result I got was 240gb/30gb split, which I think is not properly
>>>>> balanced.
>>>>>
>>>>>
>>>>>> Also, what are your outputs when you call out specific keyspaces? Do
>>>>>> the numbers get more even?
>>>>>
>>>>>
>>>>> i don't know what you mean by *call out specific key spaces?* can you
>>>>> please explain that a bit.
>>>>>
>>>>>
>>>>> If your schema is not modelled correctly you can easily end up
>>>>>> unevenly distributed data.
>>>>>
>>>>>
>>>>> I think that is the problem. initial 270gb data might not by modeled
>>>>> correctly. I have run a lot of tests on 270gb data including downsizing it
>>>>> to 5gb, they all resulted in same uneven distribution. I also tested a
>>>>> dummy dataset of 2gb which was balanced evenly. coming from rdb, I didn't
>>>>> give much thought to data modeling. can anyone please point me to some
>>>>> resources regarding this problem.
>>>>>

Re: Don't print Ping caused error logs

2017-06-19 Thread Akhil Mehra

Just in case you are not aware using a load balancer is an anti patter. Please 
refer to 
(http://docs.datastax.com/en/landing_page/doc/landing_page/planning/planningAntiPatterns.html#planningAntiPatterns__AntiPatLoadBal
 
)

You can turnoff logging for a particular class using the nodetool 
setlogginglevel 
(http://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsSetLogLev.html 
).

In your case you can try setting the log level for 
org.apache.cassandra.transport.Message to warn using the following command

nodetool setlogginglevel org.apache.cassandra.transport.Message WARN

Obviously this will suppress all info level logging in the message class. 

I hope that helps.

Cheers,
Akhil




> On 19/06/2017, at 9:13 PM, wxn...@zjqunshuo.com wrote:
> 
> Hi,
> Our cluster nodes are behind a SLB(Service Load Balancer) with a VIP and the 
> Cassandra client access the cluster by the VIP. 
> System.log print the below IOException every several seconds. I guess it's 
> the SLB service which Ping the port 9042 of the Cassandra node periodically 
> and caused the exceptions print.
> Any method to prevend the Ping caused exceptions been print?
> 
> INFO  [SharedPool-Worker-1] 2017-06-19 16:54:15,997 Message.java:605 - 
> Unexpected exception during request; channel = [id: 0x332c09b7, 
> /10.253.106.210:9042]
> java.io.IOException: Error while read(...): Connection reset by peer
>   at io.netty.channel.epoll.Native.readAddress(Native Method) 
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
>   at 
> io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.doReadBytes(EpollSocketChannel.java:675)
>  ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
>   at 
> io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:714)
>  ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
>   at 
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326) 
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
>   at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264) 
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
>   at 
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>  ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
>   at 
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>  ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
>   at java.lang.Thread.run(Thread.java:745) [na:1.7.0_85]
> 
> Cheer,
> -Simon

Re: Cleaning up related issue

2017-06-19 Thread Akhil Mehra

Is the node with the large volume a new node or an existing node. If it is an 
existing node is this the one where the node tool cleanup failed.

Cheers,
Akhil

> On 19/06/2017, at 6:40 PM, wxn...@zjqunshuo.com wrote:
> 
> Hi,
> After adding a new node, I started cleaning up task to remove the old data on 
> the other 4 nodes. All went well except one node. The cleanup takes hours and 
> the Cassandra daemon crashed in the third node. I checked the node and found 
> the crash was because of OOM. The Cassandra data volume has zero space left. 
> I removed the temporary files which I believe created during the cleaning up 
> process and started Cassanndra. 
> 
> The node joined the cluster successfully, but one thing I found. From the 
> "nodetool status" output, the node takes much data than other nodes. Nomally 
> the load should be 700GB. But actually it's 1000GB. Why it is larger? Please 
> see the output below. 
> 
> UN  10.253.44.149   705.98 GB  256  40.4% 
> 9180b7c9-fa0b-4bbe-bf62-64a599c01e58  rack1
> UN  10.253.106.218  691.07 GB  256  39.9% 
> e24d13e2-96cb-4e8c-9d94-22498ad67c85  rack1
> UN  10.253.42.113   623.73 GB  256  39.3% 
> 385ad28c-0f3f-415f-9e0a-7fe8bef97e17  rack1
> UN  10.253.41.165   779.38 GB  256  40.1% 
> 46f37f06-9c45-492d-bd25-6fef7f926e38  rack1
> UN  10.253.106.210  1022.7 GB  256  40.3% 
> a31b6088-0cb2-40b4-ac22-aec718dbd035  rack1
> 
> Cheers,
> -Simon

Re: Cleaning up related issue

2017-06-19 Thread Akhil Mehra

When you add a new node into the cluster data is streamed for all the old nodes 
into the new node added. The new node is now responsible for data previously 
stored in the old node.

The clean up process removes unwanted data after adding a new node to the 
cluster.

In your case clean up failed on this node. 

I think this node still has unwanted data that has not been cleaned up.

Cheers,
Akhil 




> On 19/06/2017, at 7:00 PM, wxn...@zjqunshuo.com wrote:
> 
> Thanks for the quick response. It's the existing node where the cleanup 
> failed. It has a larger volume than other nodes.
>  
> From: Akhil Mehra
> Date: 2017-06-19 14:56
> To: wxn002
> CC: user
> Subject: Re: Cleaning up related issue
> Is the node with the large volume a new node or an existing node. If it is an 
> existing node is this the one where the node tool cleanup failed.
> 
> Cheers,
> Akhil
> 
>> On 19/06/2017, at 6:40 PM, wxn...@zjqunshuo.com wrote:
>> 
>> Hi,
>> After adding a new node, I started cleaning up task to remove the old data 
>> on the other 4 nodes. All went well except one node. The cleanup takes hours 
>> and the Cassandra daemon crashed in the third node. I checked the node and 
>> found the crash was because of OOM. The Cassandra data volume has zero space 
>> left. I removed the temporary files which I believe created during the 
>> cleaning up process and started Cassanndra. 
>> 
>> The node joined the cluster successfully, but one thing I found. From the 
>> "nodetool status" output, the node takes much data than other nodes. Nomally 
>> the load should be 700GB. But actually it's 1000GB. Why it is larger? Please 
>> see the output below. 
>> 
>> UN  10.253.44.149   705.98 GB  256  40.4% 
>> 9180b7c9-fa0b-4bbe-bf62-64a599c01e58  rack1
>> UN  10.253.106.218  691.07 GB  256  39.9% 
>> e24d13e2-96cb-4e8c-9d94-22498ad67c85  rack1
>> UN  10.253.42.113   623.73 GB  256  39.3% 
>> 385ad28c-0f3f-415f-9e0a-7fe8bef97e17  rack1
>> UN  10.253.41.165   779.38 GB  256  40.1% 
>> 46f37f06-9c45-492d-bd25-6fef7f926e38  rack1
>> UN  10.253.106.210  1022.7 GB  256  40.3% 
>> a31b6088-0cb2-40b4-ac22-aec718dbd035  rack1
>> 
>> Cheers,
>> -Simon


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Cleaning up related issue

2017-06-19 Thread Akhil Mehra

The nodetool cleanup docs explain this increase in disk space usage.

"Running the nodetool cleanupcommand causes a temporary increase in disk space 
usage proportional to the size of your largest SSTable. Disk I/O occurs when 
running this command."

http://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsCleanup.html 
<http://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsCleanup.html>

Cheers,
Akhil


> On 19/06/2017, at 7:47 PM, wxn...@zjqunshuo.com wrote:
> 
> Akhil, I agree with you that the node still has unwanted data, but why it has 
> more data than before cleaning up?
> 
> More background:
> Before cleaning up, the node has 790GB data. After cleaning up, I assume it 
> should has less data. But in fact it has 1000GB data which is larger than I 
> expected.
> Cassandra daemon crashed and left the files with the name with "tmp-" prefix 
> in the data directory which indicate the cleaning up task was not complete.
> 
> Cheers,
> -Simon
>  
> From: Akhil Mehra <mailto:akhilme...@gmail.com>
> Date: 2017-06-19 15:17
> To: wxn...@zjqunshuo.com <mailto:wxn...@zjqunshuo.com>
> CC: user <mailto:user@cassandra.apache.org>
> Subject: Re: Cleaning up related issue
> When you add a new node into the cluster data is streamed for all the old 
> nodes into the new node added. The new node is now responsible for data 
> previously stored in the old node.
>  
> The clean up process removes unwanted data after adding a new node to the 
> cluster.
>  
> In your case clean up failed on this node.
>  
> I think this node still has unwanted data that has not been cleaned up.
>  
> Cheers,
> Akhil
>  
>  
>  
>  
> > On 19/06/2017, at 7:00 PM, wxn...@zjqunshuo.com wrote:
> >
> > Thanks for the quick response. It's the existing node where the cleanup 
> > failed. It has a larger volume than other nodes.
> >  
> > From: Akhil Mehra
> > Date: 2017-06-19 14:56
> > To: wxn002
> > CC: user
> > Subject: Re: Cleaning up related issue
> > Is the node with the large volume a new node or an existing node. If it is 
> > an existing node is this the one where the node tool cleanup failed.
> >
> > Cheers,
> > Akhil
> >
> >> On 19/06/2017, at 6:40 PM, wxn...@zjqunshuo.com wrote:
> >>
> >> Hi,
> >> After adding a new node, I started cleaning up task to remove the old data 
> >> on the other 4 nodes. All went well except one node. The cleanup takes 
> >> hours and the Cassandra daemon crashed in the third node. I checked the 
> >> node and found the crash was because of OOM. The Cassandra data volume has 
> >> zero space left. I removed the temporary files which I believe created 
> >> during the cleaning up process and started Cassanndra.
> >>
> >> The node joined the cluster successfully, but one thing I found. From the 
> >> "nodetool status" output, the node takes much data than other nodes. 
> >> Nomally the load should be 700GB. But actually it's 1000GB. Why it is 
> >> larger? Please see the output below.
> >>
> >> UN  10.253.44.149   705.98 GB  256  40.4% 
> >> 9180b7c9-fa0b-4bbe-bf62-64a599c01e58  rack1
> >> UN  10.253.106.218  691.07 GB  256  39.9% 
> >> e24d13e2-96cb-4e8c-9d94-22498ad67c85  rack1
> >> UN  10.253.42.113   623.73 GB  256  39.3% 
> >> 385ad28c-0f3f-415f-9e0a-7fe8bef97e17  rack1
> >> UN  10.253.41.165   779.38 GB  256  40.1% 
> >> 46f37f06-9c45-492d-bd25-6fef7f926e38  rack1
> >> UN  10.253.106.210  1022.7 GB  256  40.3% 
> >> a31b6088-0cb2-40b4-ac22-aec718dbd035  rack1
> >>
> >> Cheers,
> >> -Simon

Re: Convert single node C* to cluster (rebalancing problem)

2017-06-12 Thread Akhil Mehra

Great point John.

The OP should also note that data distribution also depends on your schema
and incoming data profile.

If your schema is not modelled correctly you can easily end up unevenly
distributed data.

Cheers,
Akhil

On Tue, Jun 13, 2017 at 3:36 AM, John Hughes <johnthug...@gmail.com> wrote:

> Is the OP expecting a perfect 50%/50% split? That, to my experience, is
> not going to happen, it is almost always shifted from a fraction of a
> percent to a couple percent.
>
> Datacenter: eu-west
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  AddressLoad   Tokens   Owns (effective)  Host ID
> Rack
> UN  XX.XX.XX.XX22.71 GiB  256  47.6%
> 57dafdde-2f62-467c-a8ff-c91e712f89c9  1c
> UN  XX.XX.XX.XX  17.17 GiB  256  51.3%
> d2a65c51-087d-48de-ae1f-a41142eb148d  1b
> UN  XX.XX.XX.XX  26.15 GiB  256  52.4%
> acf5dd34-5b81-4e5b-b7be-85a7fccd8e1c  1c
> UN  XX.XX.XX.XX   16.64 GiB  256  50.2%
> 6c8842dd-a966-467c-a7bc-bd6269ce3e7e  1a
> UN  XX.XX.XX.XX  24.39 GiB  256  49.8%
> fd92525d-edf2-4974-8bc5-a350a8831dfa  1a
> UN  XX.XX.XX.XX   23.8 GiB   256  48.7%
> bdc597c0-718c-4ef6-b3ef-7785110a9923  1b
>
> Though maybe part of what you are experiencing can be cleared up by
> repair/compaction/cleanup. Also, what are your outputs when you call out
> specific keyspaces? Do the numbers get more even?
>
> Cheers,
>
> On Mon, Jun 12, 2017 at 5:22 AM Akhil Mehra <akhilme...@gmail.com> wrote:
>
>> auto_bootstrap is true by default. Ensure its set to true. On startup
>> look at your logs for your auto_bootstrap value.  Look at the node
>> configuration line in your log file.
>>
>> Akhil
>>
>> On Mon, Jun 12, 2017 at 6:18 PM, Junaid Nasir <jna...@an10.io> wrote:
>>
>>> No, I didn't set it (left it at default value)
>>>
>>> On Fri, Jun 9, 2017 at 3:18 AM, ZAIDI, ASAD A <az1...@att.com> wrote:
>>>
>>>> Did you make sure auto_bootstrap property is indeed set to [true] when
>>>> you added the node?
>>>>
>>>>
>>>>
>>>> *From:* Junaid Nasir [mailto:jna...@an10.io]
>>>> *Sent:* Monday, June 05, 2017 6:29 AM
>>>> *To:* Akhil Mehra <akhilme...@gmail.com>
>>>> *Cc:* Vladimir Yudovin <vla...@winguzone.com>;
>>>> user@cassandra.apache.org
>>>> *Subject:* Re: Convert single node C* to cluster (rebalancing problem)
>>>>
>>>>
>>>>
>>>> not evenly, i have setup a new cluster with subset of data (around
>>>> 5gb). using the configuration above I am getting these results
>>>>
>>>>
>>>>
>>>> Datacenter: datacenter1
>>>>
>>>> ===
>>>>
>>>> Status=Up/Down
>>>>
>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>
>>>> --  Address  Load   Tokens   Owns (effective)  Host ID Rack
>>>>
>>>> UN  10.128.2.1   4.86 GiB   256  44.9% 
>>>> e4427611-c247-42ee-9404-371e177f5f17  rack1
>>>>
>>>> UN  10.128.2.10  725.03 MiB  256 55.1% 
>>>> 690d5620-99d3-4ae3-aebe-8f33af54a08b  rack1
>>>>
>>>> is there anything else I can tweak/check to make the distribution even?
>>>>
>>>>
>>>>
>>>> On Sat, Jun 3, 2017 at 3:30 AM, Akhil Mehra <akhilme...@gmail.com>
>>>> wrote:
>>>>
>>>> So now the data is evenly balanced in both nodes?
>>>>
>>>>
>>>>
>>>> Refer to the following documentation to get a better understanding of
>>>> the roc_address and the broadcast_rpc_address https://
>>>> www.instaclustr.com/demystifying-cassandras-broadcast_address/
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.instaclustr.com_demystifying-2Dcassandras-2Dbroadcast-5Faddress_=DwMFaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk=MaTA43pugg78xQNfaOQElhyvd8k7CjVqZPr3IWALdWI=>.
>>>> I am surprised that your node started up with rpc_broadcast_address
>>>> set as this is an unsupported property. I am assuming you are using
>>>> Cassandra version 3.10.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Akhil
>>>>
>>>>
>>>>
>>>> On 2/06/2017, at 11:0

Re: UnauthorizedException: user has no select permissions when quering Cassandra

2017-08-29 Thread Akhil Mehra

I could be that authorisation is added to a node without increasing the 
replication factor for the system_auth keyspace 
(https://docs.datastax.com/en/cassandra/3.0/cassandra/configuration/secureConfigInternalAuth.html
 
)

or for some reason the authorisation has not replicated to a particular node.

Cheers,
Akhil
http://abiasforaction.net 



> On 30/08/2017, at 9:09 AM, Chuck Reynolds  wrote:
> 
> I’m receiving the following error when quering a table that I know the user 
> has super user rights to.
>  
> It only happens about 10% of the time.
>  
> com.datastax.driver.core.exceptions.UnauthorizedException: User test has no 
> SELECT permission on  or any of its parents.

Re: Conection refuse

2017-08-28 Thread Akhil Mehra

Is this a local install? If so you can set your rpc_address to 127.0.0.1.  The 
rpc_address are the IP addresses that will be allowed to make connections to 
Cassandra. 

I suspect the problem is that you have set your RPC address to your IP address 
with is different from 127.0.0.1. cqlsh by default connects to localhost.

You can set the rpc_address to 0.0.0.0 for allowing anyone to connect. Possible 
option to eliminate Cassandra rejecting connection. Note this is a security 
hole for production deployments.

Cheers,
Warm Regards,
Akhil Mehra
http://abiasforaction.net

> On 29/08/2017, at 10:45 AM, Amir Shahinpour <a...@holisticlabs.net> wrote:
> 
> Hi,
> 
> I am getting an error connecting to cqlsh. I am getting the following error. 
> 
> Connection error: ('Unable to connect to any servers', {'127.0.0.1': 
> error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection 
> refused")})
> 
> I change the Cassandra.yaml file setting for rpc_address to my ip address and 
> listen_address to localhost. 
> 
> 
> listen_address: localhost
> rpc_address: my_IP
> 
> I also tried to change the cassandra-env.sh  to add my IP address but still 
> same error. 
> 
> JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=my_IP"
> Any suggestion?
> 
> 
>

Re: configure pooling options to avoid BusyPoolException

2017-08-23 Thread Akhil Mehra

Since queries are asynchronously executed, you will need some mechanism in your 
code to queue your request. Try setting your setMaxQueueSize to meet your need. 
By default its 256

 
http://docs.datastax.com/en/latest-java-driver-api/com/datastax/driver/core/PoolingOptions.html#setMaxQueueSize-int-

If setMaxQueueSize does not do the trick for you then you might need to look at 
throttling your queries.

Regards,
Akhil


> On 24/08/2017, at 6:40 AM, Avi Levi  wrote:
> 
> Hi ,
> I need to execute large amount (millions) of select queries. but I am getting 
> BusyPoolExcption how can I avoid that ? I tried to configure the pooling 
> options but couldn't see that it had any impact 
> Any advice ?
> Failed to execute query SELECT * FROM my_table WHERE id = 'some_uuid' AND x 
> >= 1503501104 AND y < 1503501224;
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) 
> tried for query failed (tried: localhost/0:0:0:0:0:0:0:1:9042 
> (com.datastax.driver.core.exceptions.BusyPoolException: 
> [localhost/0:0:0:0:0:0:0:1] Pool is busy (no available connection and the 
> queue has reached its max size 256)))

Re: Measuring Read Repairs

2017-08-25 Thread Akhil Mehra

You can run nodetool tpstats on your nodes and check the output for the 
ReadRepairStage.

Cheers,
Akhil

> On 26/08/2017, at 6:54 AM, Chuck Reynolds  wrote:
> 
> I’m running queries based on the token ranges to initiate read repairs across 
> datacenter.
>  
> Example query with CONSISTENCY set to ALL
> SELECT token(test_guid), test_guid FROM test_table WHERE 
> token(test_guid)>6546138161478345924 AND token(test_guid)<6571069709219758671;
>  
> Is there a way to tell if read repairs are happening based on this query?
>  
> I’m not seeing anything in the logs.
>  
>

Re: Conection refuse

2017-08-29 Thread Akhil Mehra


If the data is not important then stop all nodes. On each node empty your 
commitlog, data, hints and saved_cache directories.

Start one node. Wait for it to boot up successfully i.e. logs have no errors 
and  you can connect to it using cqlsh.

Start your second node and make sure it bootstraps and becomes part of the 
cluster.  Since you will have no data this should be quick and simple.

Regards,
Akhil

> On 30/08/2017, at 1:24 PM, Amir Shahinpour <a...@holisticlabs.net> wrote:
> 
> Akhil,
> 
> Commit log directory from yaml file is: /var/lib/cassandra/commitlog
> 
> So basically I removed it. Can I copy a new one from another node? or somehow 
> generate one?
> 
> Yes, the rm -rf was on the original and the only node. I stopped the C* and 
> ran the rm -rf /var/lib/cassandra/*. 
> At this point, the data that I lost is not very important, because it was a 
> dev environment that I am setting up. But I have to be able to make this node 
> running and talking to the new node. Neither CQLSH nor nodetool works at this 
> time. 
> 
> Best, 
> Amirali
> 
> 
> On Tue, Aug 29, 2017 at 2:49 PM, Akhil Mehra <akhilme...@gmail.com> wrote:
> What directory was the data and commit logs stored on the original working 
> node. You can look up your cassandra.yaml to figure this out. Its good to 
> confirm.
> 
> Was the rm -rf run on the original working node?
> 
> Cheers,
> Akhil
> 
>> On 30/08/2017, at 9:37 AM, Amir Shahinpour <a...@holisticlabs.net> wrote:
>> 
>> Yes both of the nodes are down.
>> 
>> On Aug 29, 2017 2:30 PM, "Akhil Mehra" <akhilme...@gmail.com> wrote:
>> Cassandra is doing a health check when it is starting up and failing due to 
>> being unable to ready files in the system key space. Here is the comment in 
>> the segment of the code that threw the exception.
>> 
>> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/SystemKeyspace.java#L804-L810
>> /**
>>  * One of three things will happen if you try to read the system 
>> keyspace:
>>  * 1. files are present and you can read them: great
>>  * 2. no files are there: great (new node is assumed)
>>  * 3. files are present but you can't read them: bad
>>  * @throws ConfigurationException
>>  */
>> 
>> Removing files for bootstrapping (adding a new node) a node sounds 
>> incorrect. Depending on your configuration the /var/lib/cassandar by default 
>> houses table data, commit logs, hints and cache. An rm -rf on it sounds 
>> ominous.
>> 
>> Are both your nodes down i.e. you cannot cqlsh in any of your nodes?
>> 
>> Regards,
>> Akhil
>> 
>> 
>> 
>> 
>>> On 30/08/2017, at 9:01 AM, Amir Shahinpour <a...@holisticlabs.net> wrote:
>>> 
>>> Hi Lucas,
>>> 
>>> Thanks for your response. So I checked the system.log and I found the 
>>> following error at the end which I think is causing the problem.  
>>> 
>>> Fatal exception during initialization
>>> org.apache.cassandra.exceptions.ConfigurationException: Found system 
>>> keyspace files, but they couldn't be loaded!
>>> 
>>> It could be due to removing some of the data. I ran the following command 
>>> to remove some data. sudo rm -rf /var/lib/cassandra/*
>>> 
>>> I am new to Cassandra and I think I made a mistake. So I had only one node 
>>> which was working fine with my tables that I had. I wanted to add a second 
>>> node and start using the real power of Cassandra. So I follow one of post 
>>> that I found, there were some changes in cassandra.yaml file and afterwards 
>>> I had to remove the files and that's why I run the remove command. So right 
>>> now neither of CQLSH and nodetool works. Please let me know if you need any 
>>> other information. 
>>> 
>>> Here is a screenshot of the system.log. Thanks a lot for your help. 
>>> 
>>> Best, 
>>> Amir
>>> 
>>> 
>>> 
>>> On Tue, Aug 29, 2017 at 7:17 AM, Lucas Benevides 
>>> <lu...@maurobenevides.com.br> wrote:
>>> Hello Amir,
>>> 
>>> You should see the log. If it was installed by the apt-get tool, it should 
>>> be in /var/log/cassandra/system.log. 
>>> It can occur when the schema of the node you are trying to connect is out 
>>> of date with the cluster. 
>>> How many nodes are there in you cluster? 
>>> What is the output of "nodetool describecluster"?
>>> 
>>> Best regards,
>>> Luca

Re: Conection refuse

2017-08-29 Thread Akhil Mehra

To install make sure:

Your cluster name on each node are the same.
You configure your seed nodes. Make one node the seed node and added seed 
configuration to all your yaml files.
Ensure that intra-node communication is on the same port. By default this is on 
port 7000
Do not startup all nodes at the same time. Only add nodes when others have 
already bootstrapped. This is essential

I have writing a blog post outline installing Cassandra using  docker 
(http://abiasforaction.net/apache-cassandra-cluster-docker/ 
<http://abiasforaction.net/apache-cassandra-cluster-docker/>). Good way to 
creating a test cluster quickly.

Docker cluster is good for dev and test purposes only. Would not use docker for.

Alternately https://github.com/pcmanus/ccm <https://github.com/pcmanus/ccm> is 
a popular tool for creating and destroying local clusters.

I hope that helps.

Cheers,
Akhil



> On 30/08/2017, at 1:59 PM, Amir Shahinpour <a...@holisticlabs.net> wrote:
> 
> Thanks Akhil,
> 
> I will do it. For setting up my second node, do you have any good source that 
> I can follow to make sure I am doing everything correct? I have been googling 
> around and quite frankly all the source that I found in the Google were kind 
> of different from each other and I guess that is why I was not able to 
> connect these two nodes together. So I am still not sure what steps should I 
> take to add a new node to a cluster. Thanks again. 
> 
> Thanks in Advance,
> Amir
> 
> On Tue, Aug 29, 2017 at 6:53 PM, Akhil Mehra <akhilme...@gmail.com 
> <mailto:akhilme...@gmail.com>> wrote:
> 
> If the data is not important then stop all nodes. On each node empty your 
> commitlog, data, hints and saved_cache directories.
> 
> Start one node. Wait for it to boot up successfully i.e. logs have no errors 
> and  you can connect to it using cqlsh.
> 
> Start your second node and make sure it bootstraps and becomes part of the 
> cluster.  Since you will have no data this should be quick and simple.
> 
> Regards,
> Akhil
> 
>> On 30/08/2017, at 1:24 PM, Amir Shahinpour <a...@holisticlabs.net 
>> <mailto:a...@holisticlabs.net>> wrote:
>> 
>> Akhil,
>> 
>> Commit log directory from yaml file is: /var/lib/cassandra/commitlog
>> 
>> So basically I removed it. Can I copy a new one from another node? or 
>> somehow generate one?
>> 
>> Yes, the rm -rf was on the original and the only node. I stopped the C* and 
>> ran the rm -rf /var/lib/cassandra/*. 
>> At this point, the data that I lost is not very important, because it was a 
>> dev environment that I am setting up. But I have to be able to make this 
>> node running and talking to the new node. Neither CQLSH nor nodetool works 
>> at this time. 
>> 
>> Best, 
>> Amirali
>> 
>> 
>> On Tue, Aug 29, 2017 at 2:49 PM, Akhil Mehra <akhilme...@gmail.com 
>> <mailto:akhilme...@gmail.com>> wrote:
>> What directory was the data and commit logs stored on the original working 
>> node. You can look up your cassandra.yaml to figure this out. Its good to 
>> confirm.
>> 
>> Was the rm -rf run on the original working node?
>> 
>> Cheers,
>> Akhil
>> 
>>> On 30/08/2017, at 9:37 AM, Amir Shahinpour <a...@holisticlabs.net 
>>> <mailto:a...@holisticlabs.net>> wrote:
>>> 
>>> Yes both of the nodes are down.
>>> 
>>> On Aug 29, 2017 2:30 PM, "Akhil Mehra" <akhilme...@gmail.com 
>>> <mailto:akhilme...@gmail.com>> wrote:
>>> Cassandra is doing a health check when it is starting up and failing due to 
>>> being unable to ready files in the system key space. Here is the comment in 
>>> the segment of the code that threw the exception.
>>> 
>>> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/SystemKeyspace.java#L804-L810
>>>  
>>> <https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/SystemKeyspace.java#L804-L810>
>>> /**
>>>  * One of three things will happen if you try to read the system 
>>> keyspace:
>>>  * 1. files are present and you can read them: great
>>>  * 2. no files are there: great (new node is assumed)
>>>  * 3. files are present but you can't read them: bad
>>>  * @throws ConfigurationException
>>>  */
>>> 
>>> Removing files for bootstrapping (adding a new node) a node sounds 
>>> incorrect. Depending on your configuration the /var/lib/cassandar by 
>>> default houses table data, commit logs, hints and cache. An rm -rf

Re: Conection refuse

2017-08-29 Thread Akhil Mehra

What directory was the data and commit logs stored on the original working 
node. You can look up your cassandra.yaml to figure this out. Its good to 
confirm.

Was the rm -rf run on the original working node?

Cheers,
Akhil

> On 30/08/2017, at 9:37 AM, Amir Shahinpour <a...@holisticlabs.net> wrote:
> 
> Yes both of the nodes are down.
> 
> On Aug 29, 2017 2:30 PM, "Akhil Mehra" <akhilme...@gmail.com 
> <mailto:akhilme...@gmail.com>> wrote:
> Cassandra is doing a health check when it is starting up and failing due to 
> being unable to ready files in the system key space. Here is the comment in 
> the segment of the code that threw the exception.
> 
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/SystemKeyspace.java#L804-L810
>  
> <https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/SystemKeyspace.java#L804-L810>
> /**
>  * One of three things will happen if you try to read the system keyspace:
>  * 1. files are present and you can read them: great
>  * 2. no files are there: great (new node is assumed)
>  * 3. files are present but you can't read them: bad
>  * @throws ConfigurationException
>  */
> 
> Removing files for bootstrapping (adding a new node) a node sounds incorrect. 
> Depending on your configuration the /var/lib/cassandar by default houses 
> table data, commit logs, hints and cache. An rm -rf on it sounds ominous.
> 
> Are both your nodes down i.e. you cannot cqlsh in any of your nodes?
> 
> Regards,
> Akhil
> 
> 
> 
> 
>> On 30/08/2017, at 9:01 AM, Amir Shahinpour <a...@holisticlabs.net 
>> <mailto:a...@holisticlabs.net>> wrote:
>> 
>> Hi Lucas,
>> 
>> Thanks for your response. So I checked the system.log and I found the 
>> following error at the end which I think is causing the problem.  
>> 
>> Fatal exception during initialization
>> org.apache.cassandra.exceptions.ConfigurationException: Found system 
>> keyspace files, but they couldn't be loaded!
>> 
>> It could be due to removing some of the data. I ran the following command to 
>> remove some data. sudo rm -rf /var/lib/cassandra/*
>> 
>> I am new to Cassandra and I think I made a mistake. So I had only one node 
>> which was working fine with my tables that I had. I wanted to add a second 
>> node and start using the real power of Cassandra. So I follow one of post 
>> that I found, there were some changes in cassandra.yaml file and afterwards 
>> I had to remove the files and that's why I run the remove command. So right 
>> now neither of CQLSH and nodetool works. Please let me know if you need any 
>> other information. 
>> 
>> Here is a screenshot of the system.log. Thanks a lot for your help. 
>> 
>> Best, 
>> Amir
>> 
>> 
>> 
>> On Tue, Aug 29, 2017 at 7:17 AM, Lucas Benevides 
>> <lu...@maurobenevides.com.br <mailto:lu...@maurobenevides.com.br>> wrote:
>> Hello Amir,
>> 
>> You should see the log. If it was installed by the apt-get tool, it should 
>> be in /var/log/cassandra/system.log. 
>> It can occur when the schema of the node you are trying to connect is out of 
>> date with the cluster. 
>> How many nodes are there in you cluster? 
>> What is the output of "nodetool describecluster"?
>> 
>> Best regards,
>> Lucas Benevides
>> 
>> 2017-08-28 19:45 GMT-03:00 Amir Shahinpour <a...@holisticlabs.net 
>> <mailto:a...@holisticlabs.net>>:
>> Hi,
>> 
>> I am getting an error connecting to cqlsh. I am getting the following error. 
>> 
>> Connection error: ('Unable to connect to any servers', {'127.0.0.1': 
>> error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: 
>> Connection refused")})
>> 
>> I change the Cassandra.yaml file setting for rpc_address to my ip address 
>> and listen_address to localhost. 
>> 
>> 
>> listen_address: localhost
>> rpc_address: my_IP
>> 
>> I also tried to change the cassandra-env.sh  to add my IP address but still 
>> same error. 
>> 
>> JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=my_IP"
>> Any suggestion?
>> 
>> 
>> 
>> 
>> 
>

Re: UnauthorizedException: user has no select permissions when quering Cassandra

2017-08-29 Thread Akhil Mehra

What consistency are you running the query with? Does the query timeout even 
with a consistency of one?


> On 30/08/2017, at 9:49 AM, Chuck Reynolds <creyno...@ancestry.com> wrote:
> 
>  
> We have that keyspace replicated to the same number of nodes in the ring.
>  
> Right now I can’t even run select * from system_auth.users without it timeing 
> out.
>  
> From: Akhil Mehra <akhilme...@gmail.com <mailto:akhilme...@gmail.com>>
> Reply-To: "user@cassandra.apache.org <mailto:user@cassandra.apache.org>" 
> <user@cassandra.apache.org <mailto:user@cassandra.apache.org>>
> Date: Tuesday, August 29, 2017 at 3:46 PM
> To: user <user@cassandra.apache.org <mailto:user@cassandra.apache.org>>
> Subject: Re: UnauthorizedException: user has no select permissions when 
> quering Cassandra
>  
> https://docs.datastax.com/en/cassandra/3.0/cassandra/configuration/secureConfigInternalAuth.html
>  
> <https://docs.datastax.com/en/cassandra/3.0/cassandra/configuration/secureConfigInternalAuth.html>

Re: Conection refuse

2017-08-29 Thread Akhil Mehra

Cassandra is doing a health check when it is starting up and failing due to 
being unable to ready files in the system key space. Here is the comment in the 
segment of the code that threw the exception.

https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/SystemKeyspace.java#L804-L810
/**
 * One of three things will happen if you try to read the system keyspace:
 * 1. files are present and you can read them: great
 * 2. no files are there: great (new node is assumed)
 * 3. files are present but you can't read them: bad
 * @throws ConfigurationException
 */

Removing files for bootstrapping (adding a new node) a node sounds incorrect. 
Depending on your configuration the /var/lib/cassandar by default houses table 
data, commit logs, hints and cache. An rm -rf on it sounds ominous.

Are both your nodes down i.e. you cannot cqlsh in any of your nodes?

Regards,
Akhil




> On 30/08/2017, at 9:01 AM, Amir Shahinpour  wrote:
> 
> Hi Lucas,
> 
> Thanks for your response. So I checked the system.log and I found the 
> following error at the end which I think is causing the problem.  
> 
> Fatal exception during initialization
> org.apache.cassandra.exceptions.ConfigurationException: Found system keyspace 
> files, but they couldn't be loaded!
> 
> It could be due to removing some of the data. I ran the following command to 
> remove some data. sudo rm -rf /var/lib/cassandra/*
> 
> I am new to Cassandra and I think I made a mistake. So I had only one node 
> which was working fine with my tables that I had. I wanted to add a second 
> node and start using the real power of Cassandra. So I follow one of post 
> that I found, there were some changes in cassandra.yaml file and afterwards I 
> had to remove the files and that's why I run the remove command. So right now 
> neither of CQLSH and nodetool works. Please let me know if you need any other 
> information. 
> 
> Here is a screenshot of the system.log. Thanks a lot for your help. 
> 
> Best, 
> Amir
> 
> 
> 
> On Tue, Aug 29, 2017 at 7:17 AM, Lucas Benevides  > wrote:
> Hello Amir,
> 
> You should see the log. If it was installed by the apt-get tool, it should be 
> in /var/log/cassandra/system.log. 
> It can occur when the schema of the node you are trying to connect is out of 
> date with the cluster. 
> How many nodes are there in you cluster? 
> What is the output of "nodetool describecluster"?
> 
> Best regards,
> Lucas Benevides
> 
> 2017-08-28 19:45 GMT-03:00 Amir Shahinpour  >:
> Hi,
> 
> I am getting an error connecting to cqlsh. I am getting the following error. 
> 
> Connection error: ('Unable to connect to any servers', {'127.0.0.1': 
> error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection 
> refused")})
> 
> I change the Cassandra.yaml file setting for rpc_address to my ip address and 
> listen_address to localhost. 
> 
> 
> listen_address: localhost
> rpc_address: my_IP
> 
> I also tried to change the cassandra-env.sh  to add my IP address but still 
> same error. 
> 
> JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=my_IP"
> Any suggestion?
> 
> 
> 
> 
>

41 matches

Mail list logo