Re: cqlsh problem

[joseph gao]

hi, Patric, [image: 内嵌图片 1],
netstat -lepunt looks like above

2016-05-27 23:16 GMT+08:00 Patrick McFadin :

> Can you do a netstat -lepunt and show the output? If Cassandra is running
> you aren't trying to connect to the ip/port it's bound to.
>
> Patrick
>
>
> On Monday, May 23, 2016, joseph gao  wrote:
>
>> I used to think it's firewall/network issues too. So I make ufw to be
>> inactive. I really don't what's the reason.
>>
>> 2016-05-09 19:01 GMT+08:00 kurt Greaves :
>>
>>> Don't be fooled, despite saying tcp6 and :::*, it still listens on IPv4.
>>> As far as I'm aware this happens on all 2.1 Cassandra nodes, and may just
>>> be an oddity of netstat. It would be unrelated to your connection timeout
>>> issues, that's most likely related to firewall/network issues.
>>>
>>> On 9 May 2016 at 09:59, joseph gao  wrote:
>>>
 It doesn't work ,still using ipv6 [image: 内嵌图片 1]

 And I already set [image: 内嵌图片 2]

 Now I'm using 4.1.1 using 9160 port instead of 5.x.x。

 Hopefully this could be resolved, Thanks!

 2016-03-30 22:13 GMT+08:00 Alain RODRIGUEZ :

> Hi Joseph,
>
> why cassandra using tcp6 for 9042 port like :
>> tcp6   0  0 0.0.0.0:9042:::*
>>  LISTEN
>>
>
> if I remember correctly, in 2.1 and higher, cqlsh uses native
> transport, port 9042  (instead of thrift port 9160) and your clients (if
> any) are also probably using native transport (port 9042). So yes, this
> could be an issue indeed.
>
> You should have something like:
>
> tcp0  0  1.2.3.4:9042   :::*
> LISTEN
>
> You are using IPv6 and no rpc address. Try setting it to the listen
> address and using IPv4.
>
> C*heers,
>
> ---
>
> Alain Rodriguez - al...@thelastpickle.com
>
> France
>
> The Last Pickle - Apache Cassandra Consulting
>
> http://www.thelastpickle.com
>
> 2016-03-30 6:09 GMT+02:00 joseph gao :
>
>> why cassandra using tcp6 for 9042 port like :
>> tcp6   0  0 0.0.0.0:9042:::*
>>  LISTEN
>> would this be the problem
>>
>> 2016-03-30 11:34 GMT+08:00 joseph gao :
>>
>>> still have not fixed it . cqlsh: error: no such option:
>>> --connect-timeout
>>> cqlsh version 5.0.1
>>>
>>>
>>>
>>> 2016-03-25 16:46 GMT+08:00 Alain RODRIGUEZ :
>>>
 Hi Joseph.

 As I can't reproduce here, I believe you are having network issue
 of some kind.

 MacBook-Pro:~ alain$ cqlsh --version
 cqlsh 5.0.1
 MacBook-Pro:~ alain$ echo 'DESCRIBE KEYSPACES;' | cqlsh
 --connect-timeout=5 --request-timeout=10
 system_traces  system
 MacBook-Pro:~ alain$

 It's been a few days, did you manage to fix it ?

 C*heers,
 ---
 Alain Rodriguez - al...@thelastpickle.com
 France

 The Last Pickle - Apache Cassandra Consulting
 http://www.thelastpickle.com

 2016-03-21 9:59 GMT+01:00 joseph gao :

> cqlsh version 5.0.1. nodetool tpstats looks good, log looks good.
> And I used specified port 9042. And it immediately returns fail (less 
> than
> 3 seconds). By the way where should I use '--connect-timeout', cqlsh 
> seems
> don't have such parameters.
>
> 2016-03-18 17:29 GMT+08:00 Alain RODRIGUEZ :
>
>> Is the node fully healthy or rejecting some requests ?
>>
>> What are the outputs for "grep -i "ERROR"
>> /var/log/cassandra/system.log" and "nodetool tpstats"?
>>
>> Any error? Any pending / blocked or dropped messages?
>>
>> Also did you try using distinct ports (9160 for thrift, 9042 for
>> native) - out of curiosity, not sure this will help.
>>
>> What is your version of cqlsh "cqlsh --version" ?
>>
>> doesn't work most times. But some time it just work fine
>>>
>>
>> Do you fill like this is due to a timeout (query being too big,
>> cluster being to busy)? Try setting this higher:
>>
>> --connect-timeout=CONNECT_TIMEOUT
>>
>> Specify the connection timeout in seconds
>> (default: 5 seconds).
>>
>>   --request-timeout=REQUEST_TIMEOUT
>>
>> Specify the default request timeout in
>> seconds (default: 10 seconds).
>>
>> C*heers,
>> ---
>> Alain Rodriguez - 

Spark Memory Error - Not enough space to cache broadcast

[Cassa L]

Hi,

I'm using spark 1.5.1 version. I am reading data from Kafka into Spark
and writing it into Cassandra after processing it. Spark job starts
fine and runs all good for some time until I start getting below
errors. Once these errors come, job start to lag behind and I see that
job has scheduling and processing delays in streaming  UI.

Worker memory is 6GB, executor-memory is 5GB, I also tried to tweak
memoryFraction parameters. Nothing works.


16/06/13 21:26:02 INFO MemoryStore: ensureFreeSpace(4044) called with
curMem=565394, maxMem=2778495713
16/06/13 21:26:02 INFO MemoryStore: Block broadcast_69652_piece0
stored as bytes in memory (estimated size 3.9 KB, free 2.6 GB)
16/06/13 21:26:02 INFO TorrentBroadcast: Reading broadcast variable
69652 took 2 ms
16/06/13 21:26:02 WARN MemoryStore: Failed to reserve initial memory
threshold of 1024.0 KB for computing block broadcast_69652 in memory.
16/06/13 21:26:02 WARN MemoryStore: Not enough space to cache
broadcast_69652 in memory! (computed 496.0 B so far)
16/06/13 21:26:02 INFO MemoryStore: Memory use = 556.1 KB (blocks) +
2.6 GB (scratch space shared across 0 tasks(s)) = 2.6 GB. Storage
limit = 2.6 GB.
16/06/13 21:26:02 WARN MemoryStore: Persisting block broadcast_69652
to disk instead.
16/06/13 21:26:02 INFO BlockManager: Found block rdd_100761_1 locally
16/06/13 21:26:02 INFO Executor: Finished task 0.0 in stage 71577.0
(TID 452316). 2043 bytes result sent to driver


Thanks,

L

How to print out the metrics information, like compaction and garbage collection?

[Jun Wu]

Hi there,
   I've deployed 6 node in Amazon EC2. I'm trying to monitor some metrics for 
each node and print them out when I write/read data into Cassandra. 
Specifically I want to print out the information about garbage collection and 
compaction.
   I do notice that there's metrics for compaction in o.a.c.metrics. 
CompactionMetrics.java. But I don't know how to get them, I've tried method 
below:  CompactionMetrics metrics = new CompactionMetrics();   
System.out.println(metrics.pendingtasks);
   But it doesn't work. 
*  Another/same question is about the garbage collection, any 
idea on where should I refer to and get the information about garbage 
collection, like garbage collection count, collection time, etc.
   Any hint will be appreciated.
Thanks!
Jun   

Re: Why there is no native shutdown command in cassandra

[Anshu Vajpayee]

https://issues.apache.org/jira/browse/CASSANDRA-12001 for your reference.

On Mon, Jun 13, 2016 at 11:55 PM, Jake Luciani  wrote:

> If that's true it's a bug then. can you open a ticket and include the
> logs? https://issues.apache.org/jira/browse/CASSANDRA
>
> On Mon, Jun 13, 2016 at 2:19 PM, Anshu Vajpayee 
> wrote:
>
>> I just tested. It doesn't flush memtables like nodetool drain/flush
>> command. Means it only does crash for the node, no graceful shutdown.
>>
>>
>>
>> On Mon, Jun 13, 2016 at 10:51 PM, Jake Luciani  wrote:
>>
>>> Yeah same as drain.  Just exits at the end.
>>>
>>> On Mon, Jun 13, 2016 at 1:11 PM, Anshu Vajpayee <
>>> anshu.vajpa...@gmail.com> wrote:
>>>
 Thanks for information.

 Does stopdaemon also flush memtables  and stop trift and CQL interface
 before shutting down the daemon ?  does node also announce  shutting down
 message  in ring  ?


 On Mon, Jun 13, 2016 at 10:14 PM, Jake Luciani 
 wrote:

> If you want to understand why, it's because C* was designed to be
> crash-only.
>
> https://www.usenix.org/conference/hotos-ix/crash-only-software
>
> Since this is great for the project but bad for operators experience
> we have later added this stopdaemon command.
>
> On Mon, Jun 13, 2016 at 12:37 PM, Anshu Vajpayee <
> anshu.vajpa...@gmail.com> wrote:
>
>> As per Documentation(pasted as below), It does not stop Daemon . I
>> tested also.I was looking for graceful shutdown  for Cassandra
>> Daemon.Description
>> 
>>
>> Flushes all memtables from the node to SSTables on disk. Cassandra
>> stops listening for connections from the client and other nodes. You need
>> to restart Cassandra after running nodetool drain. You typically use
>> this command before upgrading a node to a new version of Cassandra. To
>> simply flush memtables to disk, use nodetool flush.
>>
>> On Mon, Jun 13, 2016 at 10:00 PM, Jeff Jirsa <
>> jeff.ji...@crowdstrike.com> wrote:
>>
>>> `nodetool drain`
>>>
>>>
>>>
>>>
>>>
>>> *From: *Anshu Vajpayee 
>>> *Reply-To: *"user@cassandra.apache.org" 
>>> *Date: *Monday, June 13, 2016 at 9:28 AM
>>> *To: *"user@cassandra.apache.org" 
>>> *Subject: *Why there is no native shutdown command in cassandra
>>>
>>>
>>>
>>> Hi All
>>>
>>>
>>>
>>> Why we dont have native shutdown command in Cassandra ?
>>>
>>>
>>>
>>> Every software provides graceful shutdown command.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> ​Regards,
>>>
>>> Anshu​
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>> --
>> *Regards,*
>> *Anshu *
>>
>>
>>
>
>
> --
> http://twitter.com/tjake
>



 --
 *Regards,*
 *Anshu *



>>>
>>>
>>> --
>>> http://twitter.com/tjake
>>>
>>
>>
>>
>> --
>> *Regards,*
>> *Anshu *
>>
>>
>>
>
>
> --
> http://twitter.com/tjake
>



-- 
*Regards,*
*Anshu *

Re: Why there is no native shutdown command in cassandra

[Jake Luciani]

If that's true it's a bug then. can you open a ticket and include the logs?
https://issues.apache.org/jira/browse/CASSANDRA

On Mon, Jun 13, 2016 at 2:19 PM, Anshu Vajpayee 
wrote:

> I just tested. It doesn't flush memtables like nodetool drain/flush
> command. Means it only does crash for the node, no graceful shutdown.
>
>
>
> On Mon, Jun 13, 2016 at 10:51 PM, Jake Luciani  wrote:
>
>> Yeah same as drain.  Just exits at the end.
>>
>> On Mon, Jun 13, 2016 at 1:11 PM, Anshu Vajpayee > > wrote:
>>
>>> Thanks for information.
>>>
>>> Does stopdaemon also flush memtables  and stop trift and CQL interface
>>> before shutting down the daemon ?  does node also announce  shutting down
>>> message  in ring  ?
>>>
>>>
>>> On Mon, Jun 13, 2016 at 10:14 PM, Jake Luciani  wrote:
>>>
 If you want to understand why, it's because C* was designed to be
 crash-only.

 https://www.usenix.org/conference/hotos-ix/crash-only-software

 Since this is great for the project but bad for operators experience we
 have later added this stopdaemon command.

 On Mon, Jun 13, 2016 at 12:37 PM, Anshu Vajpayee <
 anshu.vajpa...@gmail.com> wrote:

> As per Documentation(pasted as below), It does not stop Daemon . I
> tested also.I was looking for graceful shutdown  for Cassandra Daemon.
> Description
> 
>
> Flushes all memtables from the node to SSTables on disk. Cassandra
> stops listening for connections from the client and other nodes. You need
> to restart Cassandra after running nodetool drain. You typically use
> this command before upgrading a node to a new version of Cassandra. To
> simply flush memtables to disk, use nodetool flush.
>
> On Mon, Jun 13, 2016 at 10:00 PM, Jeff Jirsa <
> jeff.ji...@crowdstrike.com> wrote:
>
>> `nodetool drain`
>>
>>
>>
>>
>>
>> *From: *Anshu Vajpayee 
>> *Reply-To: *"user@cassandra.apache.org" 
>> *Date: *Monday, June 13, 2016 at 9:28 AM
>> *To: *"user@cassandra.apache.org" 
>> *Subject: *Why there is no native shutdown command in cassandra
>>
>>
>>
>> Hi All
>>
>>
>>
>> Why we dont have native shutdown command in Cassandra ?
>>
>>
>>
>> Every software provides graceful shutdown command.
>>
>>
>>
>>
>>
>>
>>
>> ​Regards,
>>
>> Anshu​
>>
>>
>>
>>
>>
>
>
>
> --
> *Regards,*
> *Anshu *
>
>
>


 --
 http://twitter.com/tjake

>>>
>>>
>>>
>>> --
>>> *Regards,*
>>> *Anshu *
>>>
>>>
>>>
>>
>>
>> --
>> http://twitter.com/tjake
>>
>
>
>
> --
> *Regards,*
> *Anshu *
>
>
>


-- 
http://twitter.com/tjake

Re: Why there is no native shutdown command in cassandra

[Anshu Vajpayee]

I just tested. It doesn't flush memtables like nodetool drain/flush
command. Means it only does crash for the node, no graceful shutdown.



On Mon, Jun 13, 2016 at 10:51 PM, Jake Luciani  wrote:

> Yeah same as drain.  Just exits at the end.
>
> On Mon, Jun 13, 2016 at 1:11 PM, Anshu Vajpayee 
> wrote:
>
>> Thanks for information.
>>
>> Does stopdaemon also flush memtables  and stop trift and CQL interface
>> before shutting down the daemon ?  does node also announce  shutting down
>> message  in ring  ?
>>
>>
>> On Mon, Jun 13, 2016 at 10:14 PM, Jake Luciani  wrote:
>>
>>> If you want to understand why, it's because C* was designed to be
>>> crash-only.
>>>
>>> https://www.usenix.org/conference/hotos-ix/crash-only-software
>>>
>>> Since this is great for the project but bad for operators experience we
>>> have later added this stopdaemon command.
>>>
>>> On Mon, Jun 13, 2016 at 12:37 PM, Anshu Vajpayee <
>>> anshu.vajpa...@gmail.com> wrote:
>>>
 As per Documentation(pasted as below), It does not stop Daemon . I
 tested also.I was looking for graceful shutdown  for Cassandra Daemon.
 Description
 

 Flushes all memtables from the node to SSTables on disk. Cassandra
 stops listening for connections from the client and other nodes. You need
 to restart Cassandra after running nodetool drain. You typically use
 this command before upgrading a node to a new version of Cassandra. To
 simply flush memtables to disk, use nodetool flush.

 On Mon, Jun 13, 2016 at 10:00 PM, Jeff Jirsa <
 jeff.ji...@crowdstrike.com> wrote:

> `nodetool drain`
>
>
>
>
>
> *From: *Anshu Vajpayee 
> *Reply-To: *"user@cassandra.apache.org" 
> *Date: *Monday, June 13, 2016 at 9:28 AM
> *To: *"user@cassandra.apache.org" 
> *Subject: *Why there is no native shutdown command in cassandra
>
>
>
> Hi All
>
>
>
> Why we dont have native shutdown command in Cassandra ?
>
>
>
> Every software provides graceful shutdown command.
>
>
>
>
>
>
>
> ​Regards,
>
> Anshu​
>
>
>
>
>



 --
 *Regards,*
 *Anshu *



>>>
>>>
>>> --
>>> http://twitter.com/tjake
>>>
>>
>>
>>
>> --
>> *Regards,*
>> *Anshu *
>>
>>
>>
>
>
> --
> http://twitter.com/tjake
>



-- 
*Regards,*
*Anshu *

Re: java.lang.OutOfMemoryError: Java heap space

[Tobin Landricombe]

Hi Ben,

We're using the akka persistence layer which doesn't give me much scope for 
remodelling data.

So, on the assumption that the guys who wrote the persistence layer knew what 
they were doing, I followed your suggestion to increase RAM (still only to a 
miserly 8gig, which the startup script has decided means the JVM should be 
started with -Xms1968M, -Xmx1968M, -Xmn200M) and now the new nodes are coming 
up.

Thanks for your help,
Tobin

> On 12 Jun 2016, at 06:52, Ben Slater  wrote:
> 
> I should add - there is probably an option (c) of fiddling with a bunch of 
> tuning parameters to try to nurse things through with your current config but 
> I’m not sure that’s useful unless you really need to make the current set up 
> work for some reason.
> 
> On Sun, 12 Jun 2016 at 15:23 Ben Slater  wrote:
> Hi Tobin,
> 
> 4G RAM is a pretty small machine to be using to run Cassandra. As I 
> mentioned, 8G of heap is the normal recommendation for a production machine 
> which means you need at least 14-16G total (and can get performance benefit 
> from more).
> 
> I agree disk space doesn’t look to really be an issue here and I’m not sure 
> what impact degraded mode has but it doesn’t really sound good :-) (I think 
> it’s caused by Is swap disabled? : false - ie you have swap enabled which is 
> not recommended). 
> 
> In this case, I would expect that the relatively large partition(s) (175MB in 
> the warning) in conjunction with the low heap allocation is what is causing 
> C* to run out of heap. Heap exhaustion often manifests when C* has to compact 
> a large partition. When you add a new node the data that gets streamed across 
> has to be compacted which is why you’ll see it on the new node but node the 
> existing nodes (yet).
> 
> So, I’d say your options are either (a) get more memory and increase heap 
> space or (b) remodel your data with a partition key that does not create such 
> large partitions (generally, smaller is better if it meets your functional 
> needs and stay under 10MB to avoid having to tune specifically to meet the 
> needs of large partitions). And, there is a fair chance you need to do (b) 
> for a healthy cluster in the long run.
> 
> Cheers
> Ben
> 
> On Sat, 11 Jun 2016 at 20:52 Tobin Landricombe  wrote:
> Hi Ben,
> 
> I think the degraded mode is caused by one or both of these...
> • WARN  [main] 2016-06-10 14:23:01,690 StartupChecks.java:118 - 
> jemalloc shared library could not be preloaded to speed up memory allocations
> • WARN  [main] 2016-06-10 14:23:01,691 StartupChecks.java:150 - JMX 
> is not enabled to receive remote connections. Please see cassandra-env.sh for 
> more info.
> ...neither of which should cause the heap issue.
> 
> The disk space isn't low for our (test) usage and again shouldn't cause the 
> heap issue.
> 
> Which leaves the large partition. I couldn't find what is considered a large 
> partition. Is it possible that syncing the large partition is causing 
> problems? Why would it only affect the new node, not the running ones?
> 
> I looked at increasing the heap space but after reviewing the docs, the 
> current settings look correct for the machines.
> 
> All the nodes are running on VMs with 2 cores and 4gig RAM. Neither they nor 
> the hypervisor are showing much load.
> 
> Thanks for your help,
> Tobin
> 
> > On 10 Jun 2016, at 22:18, Ben Slater  wrote:
> >
> > The short-term fix is probably to try increasing heap space (in 
> > cassandra-env.sh). 8GB in the most standard but more may help in some 
> > circumstances.
> >
> > That said, your logs are pointing to a number of other issues which won’t 
> > be helping and probably need to be fixed for long-term stability:
> > - swap enabled ( Cassandra server running in degraded mode. Is swap 
> > disabled? : false,  Address space adequate? : true,  nofile limit adequate? 
> > : true, nproc limit adequate? : true)
> > - low disk space ( Only 36948 MB free across all data volumes. Consider 
> > adding more capacity to your cluster or removing obsolete snapshots)
> > - large partitions ( Writing large partition feed/messages:MANAGER:0 
> > (175811867 bytes))
> >
> > Cheers
> > Ben
> >
> > On Sat, 11 Jun 2016 at 01:11 Tobin Landricombe  wrote:
> > Hi,
> >
> > I've been googling various parts of this all day but none of the 
> > suggestions seem to fit.
> >
> > I have 2 nodes, one of which is a seed. I'm trying to add a third node but, 
> > after a few minutes in the UJ state, the node dies with the above error 
> > (http://pastebin.com/iRvYfuAu).
> >
> > Here are the warnings from the logs: http://pastebin.com/vYLvsHrv
> >
> > I've googled them but nothing seems appropriate.
> >
> > Debug log part 1: http://pastebin.com/b8ZSYtqV
> > Debug log part 2: http://pastebin.com/1Bbb7Vf8
> >
> > Thanks for any suggestions,
> > Tobin
> >
> > --
> > 
> > Ben 

Re: select query on entire primary key returning more than one row in result

[Siddharth Verma]

No, all rows were not the same.
Querying only on the partition key gives 20 rows.
In the erroneous result, while querying on partition key and clustering
key, we got 16 of those 20 rows.

And for "*tombstone_threshold"* there isn't any entry at column family
level.

Thanks,
Siddharth Verma

Re: Why there is no native shutdown command in cassandra

[Jake Luciani]

Yeah same as drain.  Just exits at the end.

On Mon, Jun 13, 2016 at 1:11 PM, Anshu Vajpayee 
wrote:

> Thanks for information.
>
> Does stopdaemon also flush memtables  and stop trift and CQL interface
> before shutting down the daemon ?  does node also announce  shutting down
> message  in ring  ?
>
>
> On Mon, Jun 13, 2016 at 10:14 PM, Jake Luciani  wrote:
>
>> If you want to understand why, it's because C* was designed to be
>> crash-only.
>>
>> https://www.usenix.org/conference/hotos-ix/crash-only-software
>>
>> Since this is great for the project but bad for operators experience we
>> have later added this stopdaemon command.
>>
>> On Mon, Jun 13, 2016 at 12:37 PM, Anshu Vajpayee <
>> anshu.vajpa...@gmail.com> wrote:
>>
>>> As per Documentation(pasted as below), It does not stop Daemon . I
>>> tested also.I was looking for graceful shutdown  for Cassandra Daemon.
>>> Description
>>> 
>>>
>>> Flushes all memtables from the node to SSTables on disk. Cassandra stops
>>> listening for connections from the client and other nodes. You need to
>>> restart Cassandra after running nodetool drain. You typically use this
>>> command before upgrading a node to a new version of Cassandra. To simply
>>> flush memtables to disk, use nodetool flush.
>>>
>>> On Mon, Jun 13, 2016 at 10:00 PM, Jeff Jirsa >> > wrote:
>>>
 `nodetool drain`





 *From: *Anshu Vajpayee 
 *Reply-To: *"user@cassandra.apache.org" 
 *Date: *Monday, June 13, 2016 at 9:28 AM
 *To: *"user@cassandra.apache.org" 
 *Subject: *Why there is no native shutdown command in cassandra



 Hi All



 Why we dont have native shutdown command in Cassandra ?



 Every software provides graceful shutdown command.







 ​Regards,

 Anshu​





>>>
>>>
>>>
>>> --
>>> *Regards,*
>>> *Anshu *
>>>
>>>
>>>
>>
>>
>> --
>> http://twitter.com/tjake
>>
>
>
>
> --
> *Regards,*
> *Anshu *
>
>
>


-- 
http://twitter.com/tjake

Re: Why there is no native shutdown command in cassandra

[Anshu Vajpayee]

Thanks for information.

Does stopdaemon also flush memtables  and stop trift and CQL interface
before shutting down the daemon ?  does node also announce  shutting down
message  in ring  ?


On Mon, Jun 13, 2016 at 10:14 PM, Jake Luciani  wrote:

> If you want to understand why, it's because C* was designed to be
> crash-only.
>
> https://www.usenix.org/conference/hotos-ix/crash-only-software
>
> Since this is great for the project but bad for operators experience we
> have later added this stopdaemon command.
>
> On Mon, Jun 13, 2016 at 12:37 PM, Anshu Vajpayee  > wrote:
>
>> As per Documentation(pasted as below), It does not stop Daemon . I tested
>> also.I was looking for graceful shutdown  for Cassandra Daemon.
>> Description
>> 
>>
>> Flushes all memtables from the node to SSTables on disk. Cassandra stops
>> listening for connections from the client and other nodes. You need to
>> restart Cassandra after running nodetool drain. You typically use this
>> command before upgrading a node to a new version of Cassandra. To simply
>> flush memtables to disk, use nodetool flush.
>>
>> On Mon, Jun 13, 2016 at 10:00 PM, Jeff Jirsa 
>> wrote:
>>
>>> `nodetool drain`
>>>
>>>
>>>
>>>
>>>
>>> *From: *Anshu Vajpayee 
>>> *Reply-To: *"user@cassandra.apache.org" 
>>> *Date: *Monday, June 13, 2016 at 9:28 AM
>>> *To: *"user@cassandra.apache.org" 
>>> *Subject: *Why there is no native shutdown command in cassandra
>>>
>>>
>>>
>>> Hi All
>>>
>>>
>>>
>>> Why we dont have native shutdown command in Cassandra ?
>>>
>>>
>>>
>>> Every software provides graceful shutdown command.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> ​Regards,
>>>
>>> Anshu​
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>> --
>> *Regards,*
>> *Anshu *
>>
>>
>>
>
>
> --
> http://twitter.com/tjake
>



-- 
*Regards,*
*Anshu *

Tick Tock version numbers

[Francisco Reyes]

I have a small, 3 machine, cluster with 3.2.1

Can I upgrade them to 3.6 from 3.2? Or is it advisable to upgrade to 
each intermediary version?


Based on what I have gather seems like it is matter of:
bring node down
install new version
bring up
run nodetool upgradesstables -a

Is that the correct process?

Re: Why there is no native shutdown command in cassandra

[Anshu Vajpayee]

Thanks, I am able to run the stopdaemon option in version 2.1.
but it doesn't print anything in log for log level INFO about shutdown.

In case of log level DEBUG , it prints shutdown information.

INFO  [RMI TCP Connection(4)-127.0.0.1] 2016-06-13 09:49:29,223
CassandraDaemon.java:410 - Cassandra shutting down...
INFO  [RMI TCP Connection(4)-127.0.0.1] 2016-06-13 09:49:29,223
ThriftServer.java:142 - Stop listening to thrift clients
INFO  [RMI TCP Connection(4)-127.0.0.1] 2016-06-13 09:49:29,235
Server.java:213 - Stop listening for CQL clients


On Mon, Jun 13, 2016 at 10:01 PM, DuyHai Doan  wrote:

> In Cassandra 3.x, I think there is a "nodetool stopdaemon" command
>
> On Mon, Jun 13, 2016 at 6:28 PM, Anshu Vajpayee 
> wrote:
>
>> Hi All
>>
>> Why we dont have native shutdown command in Cassandra ?
>>
>> Every software provides graceful shutdown command.
>>
>>
>>
>> ​Regards,
>> Anshu​
>>
>>
>>
>


-- 
*Regards,*
*Anshu *

Re: Installing Cassandra from Tarball

[Bhuvan Rawal]

Hi Steve,

Please find the responses in line:

WARN  15:41:58 Unable to lock JVM memory (ENOMEM). This can result in part
> of the JVM being swapped out, especially with mmapped I/O enabled. Increase
> RLIMIT_MEMLOCK or run Cassandra as root.
>
 You can edit -* /etc/security/limits.conf *and put these lines in there

* - memlock unlimited
* - nofile 10
* - nproc 32768
* - as unlimited

and reload the properties by command $ sudo sysctl -p
and then checking it :
$ ulimit -l
and for the cassandra process by :
$ cat /proc//limits
Source - Datastax Troubleshooting


WARN  15:41:58 jemalloc shared library could not be preloaded to speed up
> memory allocations
>
If you want to allocate off heap memory using jemalloc then uncomment this
line in *cassandra-env.sh* and provide appropriate jemalloc path
JVM_OPTS="$JVM_OPTS -Djava.library.path=/lib/"

WARN  15:41:58 JMX is not enabled to receive remote connections. Please see
> cassandra-env.sh for more info.
>
By default JMX is enabled only for local connections, if you want to debug
from a remote machine mark LOCAL_JMX=no  in *cassandra-env.sh*

WARN  15:41:58 Cassandra server running in degraded mode. Is swap disabled?
> : true,  Address space adequate? : true,  nofile limit adequate? : false,
> nproc limit adequate? : false
>
You need to disable swap in order to avoid this message, using swap space
can have serious performance implications. Make sure you disable fstab
entry as well for swap partition.

Thanks & Regards,
Bhuvan

Re: Why there is no native shutdown command in cassandra

[Jake Luciani]

If you want to understand why, it's because C* was designed to be
crash-only.

https://www.usenix.org/conference/hotos-ix/crash-only-software

Since this is great for the project but bad for operators experience we
have later added this stopdaemon command.

On Mon, Jun 13, 2016 at 12:37 PM, Anshu Vajpayee 
wrote:

> As per Documentation(pasted as below), It does not stop Daemon . I tested
> also.I was looking for graceful shutdown  for Cassandra Daemon.Description
>
> 
>
> Flushes all memtables from the node to SSTables on disk. Cassandra stops
> listening for connections from the client and other nodes. You need to
> restart Cassandra after running nodetool drain. You typically use this
> command before upgrading a node to a new version of Cassandra. To simply
> flush memtables to disk, use nodetool flush.
>
> On Mon, Jun 13, 2016 at 10:00 PM, Jeff Jirsa 
> wrote:
>
>> `nodetool drain`
>>
>>
>>
>>
>>
>> *From: *Anshu Vajpayee 
>> *Reply-To: *"user@cassandra.apache.org" 
>> *Date: *Monday, June 13, 2016 at 9:28 AM
>> *To: *"user@cassandra.apache.org" 
>> *Subject: *Why there is no native shutdown command in cassandra
>>
>>
>>
>> Hi All
>>
>>
>>
>> Why we dont have native shutdown command in Cassandra ?
>>
>>
>>
>> Every software provides graceful shutdown command.
>>
>>
>>
>>
>>
>>
>>
>> ​Regards,
>>
>> Anshu​
>>
>>
>>
>>
>>
>
>
>
> --
> *Regards,*
> *Anshu *
>
>
>


-- 
http://twitter.com/tjake

Re: select query on entire primary key returning more than one row in result

[Anshu Vajpayee]

were all rows same? If not what was different ?

What was droppable tombstone  compaction  ratio for that table/CF?

On Mon, Jun 13, 2016 at 6:11 PM, Siddharth Verma <
verma.siddha...@snapdeal.com> wrote:

> Running nodetool compact fixed the issue.
>
> Could someone help out as why it occurred.
>
>
>


-- 
*Regards,*
*Anshu *

Re: Why there is no native shutdown command in cassandra

[Anshu Vajpayee]

As per Documentation(pasted as below), It does not stop Daemon . I tested
also.I was looking for graceful shutdown  for Cassandra Daemon.Description


Flushes all memtables from the node to SSTables on disk. Cassandra stops
listening for connections from the client and other nodes. You need to
restart Cassandra after running nodetool drain. You typically use this
command before upgrading a node to a new version of Cassandra. To simply
flush memtables to disk, use nodetool flush.

On Mon, Jun 13, 2016 at 10:00 PM, Jeff Jirsa 
wrote:

> `nodetool drain`
>
>
>
>
>
> *From: *Anshu Vajpayee 
> *Reply-To: *"user@cassandra.apache.org" 
> *Date: *Monday, June 13, 2016 at 9:28 AM
> *To: *"user@cassandra.apache.org" 
> *Subject: *Why there is no native shutdown command in cassandra
>
>
>
> Hi All
>
>
>
> Why we dont have native shutdown command in Cassandra ?
>
>
>
> Every software provides graceful shutdown command.
>
>
>
>
>
>
>
> ​Regards,
>
> Anshu​
>
>
>
>
>



-- 
*Regards,*
*Anshu *

Re: Why there is no native shutdown command in cassandra

[DuyHai Doan]

In Cassandra 3.x, I think there is a "nodetool stopdaemon" command

On Mon, Jun 13, 2016 at 6:28 PM, Anshu Vajpayee 
wrote:

> Hi All
>
> Why we dont have native shutdown command in Cassandra ?
>
> Every software provides graceful shutdown command.
>
>
>
> ​Regards,
> Anshu​
>
>
>

Re: Why there is no native shutdown command in cassandra

[Jeff Jirsa]

`nodetool drain`

 

 

From: Anshu Vajpayee 
Reply-To: "user@cassandra.apache.org" 
Date: Monday, June 13, 2016 at 9:28 AM
To: "user@cassandra.apache.org" 
Subject: Why there is no native shutdown command in cassandra

 

Hi All 

 

Why we dont have native shutdown command in Cassandra ?

 

Every software provides graceful shutdown command.  

 

 

 

​Regards,

Anshu​

 

 



smime.p7s
Description: S/MIME cryptographic signature

Why there is no native shutdown command in cassandra

[Anshu Vajpayee]

Hi All

Why we dont have native shutdown command in Cassandra ?

Every software provides graceful shutdown command.



​Regards,
Anshu​

Installing Cassandra from Tarball

[Steve Anderson]

I am trying out Cassandra and have installed it on a single Amazon Linux AMI 
instance (.

I downloaded the Cassandra 3.6 tarball. However when I run Cassandra I notice 
there are the following warnings:

WARN  15:41:58 Small commitlog volume detected at /var/lib/cassandra/commitlog; 
setting commitlog_total_space_in_mb to 1983.  You can override this in 
cassandra.yaml
WARN  15:41:58 Only 5.812GiB free across all data volumes. Consider adding more 
capacity to your cluster or removing obsolete snapshots
…
WARN  15:41:58 Unable to lock JVM memory (ENOMEM). This can result in part of 
the JVM being swapped out, especially with mmapped I/O enabled. Increase 
RLIMIT_MEMLOCK or run Cassandra as root.
WARN  15:41:58 jemalloc shared library could not be preloaded to speed up 
memory allocations
WARN  15:41:58 JMX is not enabled to receive remote connections. Please see 
cassandra-env.sh for more info.
INFO  15:41:58 Initializing SIGAR library
WARN  15:41:58 Cassandra server running in degraded mode. Is swap disabled? : 
true,  Address space adequate? : true,  nofile limit adequate? : false, nproc 
limit adequate? : false


I assume the first two warnings are no big deal, but am unsure about the 
others. How do I confirm Cassandra is up and running OK?

Also, I notice when I shutdown Cassandra with ctrl-c, there is no message 
indicating Cassandra is shutting down.

Am wondering if I should just install Cassandra using a RPM distribution from 
DataStax. 

Steve
—
"Surely, those who believe, those who are Jewish, the Christians, and the 
converts; anyone who (1) believes in God, (2) believes in the Last Day, and (3) 
leads a righteous life, will receive their recompense from their Lord; they 
have nothing to fear, nor will they grieve." (Quran 2:62, 5:69) …learn more at 
www.masjidtucson.org

Re: select query on entire primary key returning more than one row in result

[Siddharth Verma]

Running nodetool compact fixed the issue.

Could someone help out as why it occurred.

select query on entire primary key returning more than one row in result

[Siddharth Verma]

Hi,
We are facing this issue on production,
We upgraded our cassandra from 3.0.3 to 3.5

When we ran a query with partition key and clustering column(entire primary
key specified), we get 16 rows in return.

We have 2DC's, each with RF 3 for our keyspace.

1. We connected with cqlsh, and setting consistency to local_one, and
tracing on, we saw that, we got correct result on 3, and erroneous results
on 3.
Correct result : only 1 row
Erroneous result : 16 rows

2. we executed the statement while specifying only the clustering column
with ALLOW FILTERING, then we got the only one record for that partition
key.

3. While upgrading, we dropped key_cache folder on some, not all.

What could be the causes and how to fix this issue?

We speculate that it might be due to cache.

Any help would be appreciated.

Thanks
Siddharth Verma

Streaming from 1 node only when adding a new DC

[Fabien Rousseau]

Hello,

We've tested adding a new DC from an existing DC having 3 nodes and RF=3
(ie all nodes have all data).
During the rebuild process, only one node of the first DC streamed data to
the 3 nodes of the second DC.

Our goal is to minimise the time it takes to rebuild a DC and would like to
be able to stream from all nodes.

Starting C* with debug logs, it appears that all nodes, when computing
their "streaming plan" returns the same node for all ranges.
This is probably because all nodes in DC2 have the same view of the ring.

I understand that when bootstrapping a new node, it's preferable to stream
from the node being replaced, but when rebuilding a new DC, it should
probably select sources "randomly" (rather than always selecting the same
source for a specific range).
What do you think ?

Best Regards,
Fabien