Re: Scale SASI index

[Jordan West]

On Mon, Sep 17, 2018 at 10:52 PM onmstester onmstester 
wrote:

> By adding new nodes to cluster, should i rebuild SASI indexes on all nodes
> ?
>

It isn’t necessary and will result in redundant work. On new nodes the
indexes will be built as data is streamed in. On existing nodes, compaction
or cleanup will take care of removing unneeded indexes when the data is
removed.


>
>

Re: jmxterm "#NullPointerException: No such PID "

[Yuki Morishita]

This is because Cassandra sets -XX:+PerfDisableSharedMem JVM option by default.
This prevents tools such as jps to list jvm processes.
See https://issues.apache.org/jira/browse/CASSANDRA-9242 for detail.

You can work around by doing what Riccardo said.
On Tue, Sep 18, 2018 at 9:41 PM Philip Ó Condúin
 wrote:
>
> Hi Riccardo,
>
> Yes that works for me:
>
> Welcome to JMX terminal. Type "help" for available commands.
> $> open localhost:7199
> #Connection to localhost:7199 is opened
> $>domains
> #following domains are available
> JMImplementation
> ch.qos.logback.classic
> com.sun.management
> java.lang
> java.nio
> java.util.logging
> org.apache.cassandra.db
> org.apache.cassandra.hints
> org.apache.cassandra.internal
> org.apache.cassandra.metrics
> org.apache.cassandra.net
> org.apache.cassandra.request
> org.apache.cassandra.service
> $>
>
> I can work with this :-)
>
> Not sure why the JVM is not listed when issuing the JVMS command, maybe its a 
> server setting, our production servers find the Cass JVM.  I've spent half 
> the day trying to figure it out so I think I'll just put it to bed now and 
> work on something else.
>
> Regards,
> Phil
>
> On Tue, 18 Sep 2018 at 13:34, Riccardo Ferrari  wrote:
>>
>> Hi Philip,
>>
>> I've used jmxterm myself without any problems particular problems. On my 
>> systems too, I don't get the cassandra daemon listed when issuing the `jvms` 
>> command but I never spent much time investigating it.
>> Assuming you have not changed anything relevant in the cassandra-env.sh you 
>> can connect using jmxterm by issuing 'open 127.0.0.1:7199'. Would that work 
>> for you?
>>
>> HTH,
>>
>>
>>
>> On Tue, Sep 18, 2018 at 2:00 PM, Philip Ó Condúin  
>> wrote:
>>>
>>> Further info:
>>>
>>> I would expect to see the following when I list the jvm's:
>>>
>>> Welcome to JMX terminal. Type "help" for available commands.
>>> $>jvms
>>> 25815(m) - org.apache.cassandra.service.CassandraDaemon
>>> 17628( ) - jmxterm-1.0-alpha-4-uber.jar
>>>
>>> But jmxtem is not picking up the JVM for Cassandra for some reason.
>>>
>>> Can someone point me in the right direction?  Is there settings in the 
>>> cassandra-env.sh file I need to amend to get jmxterm to find the cass jvm?
>>>
>>> Im not finding much about it on google.
>>>
>>> Thanks,
>>> Phil
>>>
>>>
>>> On Tue, 18 Sep 2018 at 12:09, Philip Ó Condúin  
>>> wrote:

 Hi All,

 I need a little advice.  I'm trying to access the JMX terminal using 
 jmxterm-1.0-alpha-4-uber.jar with a very simple default install of C* 
 3.11.3

 I keep getting the following:

 [cassandra@reaper-1 conf]$ java -jar jmxterm-1.0-alpha-4-uber.jar
 Welcome to JMX terminal. Type "help" for available commands.
 $>open 1666
 #NullPointerException: No such PID 1666
 $>

 C* is running with a PID of 1666.  I've tried setting JMX_LOCAL=no and 
 have even created a new VM to test it.

 Does anyone know what I might be doing wrong here?

 Kind Regards,
 Phil

>>>
>>>
>>> --
>>> Regards,
>>> Phil
>>
>>
>
>
> --
> Regards,
> Phil

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Mutation of bytes is too large for the maxiumum size of

[Soumya Jena]

The client should notice this on their side . If you want to see on the
server log one idea may be is to enable the debug mode .
You can set it specifically for org.apache.cassandra.transport
Something like  nodetool setlogginglevel org.apache.cassandra.transport
DEBUG
If you are lucky enough :)  (i.e. not too much chatter around the same
time) , you should see the query just before that WARN message appears in
log .
You can turn off the debugging once you get the info.
Good luck !!

On Mon, Sep 17, 2018 at 9:06 PM Saladi Naidu 
wrote:

> Any clues on this topic?
>
> Naidu Saladi
>
>
> On Thursday, September 6, 2018 9:41 AM, Saladi Naidu
>  wrote:
>
>
> We are receiving following error
>
> 9140-at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.0.10.jar:3.0.10]
>
> 9141-at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
>
> 9142:WARN  [SharedPool-Worker-1] 2018-09-06 14:29:46,071 
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread 
> Thread[SharedPool-Worker-1,5,main]: {}
>
> 9143-java.lang.IllegalArgumentException: Mutation of 16777251 bytes is too 
> large for the maximum size of 16777216
>
> 9144-at 
> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:256) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
>
>
> I found following link that explained the cause
>
> By design intent the maximum allowed segment size is 50% of the configured
> commit_log_segment_size_in_mb. This is so Cassandra avoids writing segments
> with large amounts of empty space.
> To elaborate; up to two 32MB segments will fit into 64MB, however 40MB
> will only fit once leaving a larger amount of unused space.
>
> "I would like to find what table/column family this write/mutation is
> causing this error so that I can reach out to right application team, log
> does not provide any details regarding the mutation at all, is there a way
> to find that out
>
> Mutation of bytes is too large for the maxiumum size of
> 
>
> Mutation of bytes is too large for the maxiumum size of
> Summary Apache Cassandra will discard mutations larger than a
> predetermined size. This note addresses why this h...
>
> 
>
>
>
>
> Naidu Saladi
>
>
>

Re: Communication between 2 Cassandra Clusters

[Ghazi Naceur]

Hello,

Thank you ! I appreciate your help.

On Tue, Sep 18, 2018, 17:35 Joseph Arriola  wrote:

> Hi, the nodes are still communicating because they have not yet done the
> logical separation, this is done by changing the snitch for each node in
> the cluster in the cassandra.yaml file of the node and then changing the
> name of the DC in cassandra-rackdc.properties. It is very important to do a
> total cleaning of the nodes.
>
> You could follow this links
>
>
> https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsSwitchSnitch.html
>
>
> https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsAddDCToCluster.html
>
> 2018-09-18 10:22 GMT-06:00 Ghazi Naceur :
>
>> Hello everyone,
>>
>> I am facing a problem with my 2 Cassandra Clusters. In fact, I created
>> the first one and it does contain 5 nodes. After that, I created the second
>> one as a duplicate of the first. I changed the seeds parameters and I was
>> assuming that the first cluster will not communicate with the second, but I
>> always see communication between both clusters. I didn't change, though,
>> the DC names and the cluster names (so they have the same DC name and the
>> same cluster name), and both clusters are in the same network.
>> I need to separate between the 2 clusters. Is there some configuration to
>> be applied in order to perform the isolation between clusters ?
>>
>> Thank you for your help.
>>
>
>

Re: Communication between 2 Cassandra Clusters

[Joseph Arriola]

Hi, the nodes are still communicating because they have not yet done the
logical separation, this is done by changing the snitch for each node in
the cluster in the cassandra.yaml file of the node and then changing the
name of the DC in cassandra-rackdc.properties. It is very important to do a
total cleaning of the nodes.

You could follow this links

https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsSwitchSnitch.html

https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsAddDCToCluster.html

2018-09-18 10:22 GMT-06:00 Ghazi Naceur :

> Hello everyone,
>
> I am facing a problem with my 2 Cassandra Clusters. In fact, I created the
> first one and it does contain 5 nodes. After that, I created the second one
> as a duplicate of the first. I changed the seeds parameters and I was
> assuming that the first cluster will not communicate with the second, but I
> always see communication between both clusters. I didn't change, though,
> the DC names and the cluster names (so they have the same DC name and the
> same cluster name), and both clusters are in the same network.
> I need to separate between the 2 clusters. Is there some configuration to
> be applied in order to perform the isolation between clusters ?
>
> Thank you for your help.
>

Communication between 2 Cassandra Clusters

[Ghazi Naceur]

Hello everyone,

I am facing a problem with my 2 Cassandra Clusters. In fact, I created the
first one and it does contain 5 nodes. After that, I created the second one
as a duplicate of the first. I changed the seeds parameters and I was
assuming that the first cluster will not communicate with the second, but I
always see communication between both clusters. I didn't change, though,
the DC names and the cluster names (so they have the same DC name and the
same cluster name), and both clusters are in the same network.
I need to separate between the 2 clusters. Is there some configuration to
be applied in order to perform the isolation between clusters ?

Thank you for your help.

RE: [EXTERNAL] Re: Adding datacenter and data verification

[Durity, Sean R]

You are correct that altering the keyspace replication settings does not 
actually move any data. It only affects new writes or reads. System_auth is one 
that needs to be repaired quickly OR, if your number of users/permissions is 
relatively small, you can just reinsert them after the alter to the table. The 
data will get written to all the proper, new nodes.


Sean Durity

From: Pradeep Chhetri 
Sent: Tuesday, September 18, 2018 1:55 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: Adding datacenter and data verification

Hi Eunsu,

By going through the documentation, I think you are right, you shouldn't use 
withUsedHostsPerRemoteDc because it will contact nodes in other datacenters.  
No i don't use withUsedHostsPerRemoteDc, but instead i use withLocalDc option.

On Tue, Sep 18, 2018 at 11:02 AM, Eunsu Kim 
mailto:eunsu.bil...@gmail.com>> wrote:
Yes, I altered the system_auth key space before adding the data center.

However, I suspect that the new data center did not get the system_auth data 
and therefore could not authenticate to the client. Because the new data center 
did not get the replica count by altering keyspace.

Do your clients have the 'withUsedHostsPerRemoteDc' option?



On 18 Sep 2018, at 1:17 PM, Pradeep Chhetri 
mailto:prad...@stashaway.com>> wrote:

Hello Eunsu,

I am also using PasswordAuthenticator in my cassandra cluster. I didn't come 
across this issue while doing the exercise on preprod.

Are you sure that you changed the configuration of system_auth keyspace before 
adding the new datacenter using this:

ALTER KEYSPACE system_auth WITH REPLICATION = {'class': 
'NetworkTopologyStrategy', 'datacenter1': '3'};

Regards,
Pradeep



On Tue, Sep 18, 2018 at 7:23 AM, Eunsu Kim 
mailto:eunsu.bil...@gmail.com>> wrote:

In my case, there were authentication issues when adding data centers.

I was using a PasswordAuthenticator.

As soon as the datacenter was added, the following authentication error log was 
recorded on the client log file.

com.datastax.driver.core.exceptions.AuthenticationException: Authentication 
error on host /xxx.xxx.xxx.xx:9042: Provided username apm and/or password are 
incorrect

I was using DCAwareRoundRobinPolicy, but I guess it's probably because of the 
withUsedHostsPerRemoteDc option.

I took several steps and the error log disappeared. It is probably ’nodetool 
rebuild' after altering the system_auth table.

However, the procedure was not clearly defined.



On 18 Sep 2018, at 2:40 AM, Pradeep Chhetri 
mailto:prad...@stashaway.com>> wrote:

Hello Alain,

Thank you very much for reviewing it. You answer on seed nodes cleared my 
doubts. I will update it as per your suggestion.

I have few followup questions on decommissioning of datacenter:

- Do i need to run nodetool repair -full on each of the nodes (old + new dc 
nodes) before starting the decommissioning process of old dc.
- We have around 15 apps using cassandra cluster. I want to make sure that all 
queries before starting the new datacenter are going with right consistency 
level i.e LOCAL_QUORUM instead of QUORUM. Is there a way i can log the 
consistency level of each query somehow in some log file.

Regards,
Pradeep

On Mon, Sep 17, 2018 at 9:26 PM, Alain RODRIGUEZ 
mailto:arodr...@gmail.com>> wrote:
Hello Pradeep,

It looks good to me and it's a cool runbook for you to follow and for others to 
reuse.

To make sure that cassandra nodes in one datacenter can see the nodes of the 
other datacenter, add the seed node of the new datacenter in any of the old 
datacenter’s nodes and restart that node.

Nodes seeing each other from the distinct rack is not related to seeds. It's 
indeed recommended to use seeds from all the datacenter (a couple or 3). I 
guess it's to increase availability on seeds node and/or maybe to make sure 
local seeds are available.

You can perfectly (and even have to) add your second datacenter nodes using 
seeds from the first data center. A bootstrapping node should never be in the 
list of seeds unless it's the first node of the cluster. Add nodes, then make 
them seeds.


Le lun. 17 sept. 2018 à 11:25, Pradeep Chhetri 
mailto:prad...@stashaway.com>> a écrit :
Hello everyone,

Can someone please help me in validating the steps i am following to migrate 
cassandra snitch.

Regards,
Pradeep

On Wed, Sep 12, 2018 at 1:38 PM, Pradeep Chhetri 
mailto:prad...@stashaway.com>> wrote:
Hello

I am running cassandra 3.11.3 5-node cluster on AWS with SimpleSnitch. I was 
testing the process to migrate to GPFS using AWS region as the datacenter name 
and AWS zone as the rack name in my preprod environment and was able to achieve 
it.

But before decommissioning the older datacenter, I want to verify that the data 
in newer dc is in consistence with the one in older dc. Is there any easy way 
to do that.

Do you suggest running a full repair before decommissioning the nodes of older 
datacenter ?

I am using the steps documented here: 

RE: [EXTERNAL] Re: cold vs hot data

[Durity, Sean R]

Wouldn’t you have the same problem with two similar tables with different 
primary keys (eg., UserByID and UserByName)? This is a very common pattern in 
Cassandra – inserting into multiple tables… That’s what batches are for – 
atomicity.
I don’t understand the additional concern here.



Sean Durity

From: DuyHai Doan 
Sent: Monday, September 17, 2018 4:23 PM
To: user 
Subject: Re: [EXTERNAL] Re: cold vs hot data

Sean

Without transactions à la SQL, how can you guarantee atomicity between both 
tables for upserts ? I mean, one write could succeed with hot table and fail 
for cold table

The only solution I see is using logged batch, with a huge overhead and perf 
hit on for the writes

On Mon, Sep 17, 2018 at 8:28 PM, Durity, Sean R 
mailto:sean_r_dur...@homedepot.com>> wrote:
An idea:

On initial insert, insert into 2 tables:
Hot with short TTL
Cold/archive with a longer (or no) TTL
Then your hot data is always in the same table, but being expired. And you can 
access the archive table only for the more rare circumstances. Then you could 
have the HOT table on a different volume of faster storage. If the hot/cold 
tables are in different keyspaces, then you could also have different 
replication (a HOT DC and an archive DC, for example)


Sean Durity


-Original Message-
From: Mateusz 
mailto:mateusz-li...@ant.gliwice.pl>>
Sent: Friday, September 14, 2018 2:40 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: cold vs hot data

On piątek, 14 września 2018 02:46:43 CEST Alaa Zubaidi (PDF) wrote:
> The data can grow to +100TB however the hot data will be in most cases
> less than 10TB but we still need to keep the rest of data accessible.
> Anyone has this problem?
> What is the best way to make the cluster more efficient?
> Is there a way to somehow automatically move the old data to different
> storage (rack, dc, etc)?
> Any ideas?

We solved it using lvmcache.

--
Mateusz
(...) mam brata - poważny, domator, liczykrupa, hipokryta, pobożniś,
krótko mówiąc - podpora społeczeństwa."
Nikos Kazantzakis - "Grek Zorba"




-
To unsubscribe, e-mail: 
user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: 
user-h...@cassandra.apache.org



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

-
To unsubscribe, e-mail: 
user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: 
user-h...@cassandra.apache.org




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Re: jmxterm "#NullPointerException: No such PID "

[Philip Ó Condúin]

Hi Riccardo,

Yes that works for me:

Welcome to JMX terminal. Type "help" for available commands.
$> open localhost:7199
#Connection to localhost:7199 is opened
$>domains
#following domains are available
JMImplementation
ch.qos.logback.classic
com.sun.management
java.lang
java.nio
java.util.logging
org.apache.cassandra.db
org.apache.cassandra.hints
org.apache.cassandra.internal
org.apache.cassandra.metrics
org.apache.cassandra.net
org.apache.cassandra.request
org.apache.cassandra.service
$>

I can work with this :-)

Not sure why the JVM is not listed when issuing the JVMS command, maybe its
a server setting, our production servers find the Cass JVM.  I've spent
half the day trying to figure it out so I think I'll just put it to bed now
and work on something else.

Regards,
Phil

On Tue, 18 Sep 2018 at 13:34, Riccardo Ferrari  wrote:

> Hi Philip,
>
> I've used jmxterm myself without any problems particular problems. On my
> systems too, I don't get the cassandra daemon listed when issuing the
> `jvms` command but I never spent much time investigating it.
> Assuming you have not changed anything relevant in the cassandra-env.sh
> you can connect using jmxterm by issuing 'open 127.0.0.1:7199'. Would
> that work for you?
>
> HTH,
>
>
>
> On Tue, Sep 18, 2018 at 2:00 PM, Philip Ó Condúin <
> philipocond...@gmail.com> wrote:
>
>> Further info:
>>
>> I would expect to see the following when I list the jvm's:
>>
>> Welcome to JMX terminal. Type "help" for available commands.
>> $>jvms
>> *25815(m) - org.apache.cassandra.service.CassandraDaemon*
>> 17628( ) - jmxterm-1.0-alpha-4-uber.jar
>>
>> But jmxtem is not picking up the JVM for Cassandra for some reason.
>>
>> Can someone point me in the right direction?  Is there settings in the
>> cassandra-env.sh file I need to amend to get jmxterm to find the cass jvm?
>>
>> Im not finding much about it on google.
>>
>> Thanks,
>> Phil
>>
>>
>> On Tue, 18 Sep 2018 at 12:09, Philip Ó Condúin 
>> wrote:
>>
>>> Hi All,
>>>
>>> I need a little advice.  I'm trying to access the JMX terminal using
>>> *jmxterm-1.0-alpha-4-uber.jar* with a very simple default install of C*
>>> 3.11.3
>>>
>>> I keep getting the following:
>>>
>>> [cassandra@reaper-1 conf]$ java -jar jmxterm-1.0-alpha-4-uber.jar
>>> Welcome to JMX terminal. Type "help" for available commands.
>>> $>open 1666
>>> *#NullPointerException: No such PID 1666*
>>> $>
>>>
>>> C* is running with a PID of 1666.  I've tried setting JMX_LOCAL=no and
>>> have even created a new VM to test it.
>>>
>>> Does anyone know what I might be doing wrong here?
>>>
>>> Kind Regards,
>>> Phil
>>>
>>>
>>
>> --
>> Regards,
>> Phil
>>
>
>

-- 
Regards,
Phil

Re: jmxterm "#NullPointerException: No such PID "

[Riccardo Ferrari]

Hi Philip,

I've used jmxterm myself without any problems particular problems. On my
systems too, I don't get the cassandra daemon listed when issuing the
`jvms` command but I never spent much time investigating it.
Assuming you have not changed anything relevant in the cassandra-env.sh you
can connect using jmxterm by issuing 'open 127.0.0.1:7199'. Would that work
for you?

HTH,



On Tue, Sep 18, 2018 at 2:00 PM, Philip Ó Condúin 
wrote:

> Further info:
>
> I would expect to see the following when I list the jvm's:
>
> Welcome to JMX terminal. Type "help" for available commands.
> $>jvms
> *25815(m) - org.apache.cassandra.service.CassandraDaemon*
> 17628( ) - jmxterm-1.0-alpha-4-uber.jar
>
> But jmxtem is not picking up the JVM for Cassandra for some reason.
>
> Can someone point me in the right direction?  Is there settings in the
> cassandra-env.sh file I need to amend to get jmxterm to find the cass jvm?
>
> Im not finding much about it on google.
>
> Thanks,
> Phil
>
>
> On Tue, 18 Sep 2018 at 12:09, Philip Ó Condúin 
> wrote:
>
>> Hi All,
>>
>> I need a little advice.  I'm trying to access the JMX terminal using
>> *jmxterm-1.0-alpha-4-uber.jar* with a very simple default install of C*
>> 3.11.3
>>
>> I keep getting the following:
>>
>> [cassandra@reaper-1 conf]$ java -jar jmxterm-1.0-alpha-4-uber.jar
>> Welcome to JMX terminal. Type "help" for available commands.
>> $>open 1666
>> *#NullPointerException: No such PID 1666*
>> $>
>>
>> C* is running with a PID of 1666.  I've tried setting JMX_LOCAL=no and
>> have even created a new VM to test it.
>>
>> Does anyone know what I might be doing wrong here?
>>
>> Kind Regards,
>> Phil
>>
>>
>
> --
> Regards,
> Phil
>

Re: jmxterm "#NullPointerException: No such PID "

[Philip Ó Condúin]

Further info:

I would expect to see the following when I list the jvm's:

Welcome to JMX terminal. Type "help" for available commands.
$>jvms
*25815(m) - org.apache.cassandra.service.CassandraDaemon*
17628( ) - jmxterm-1.0-alpha-4-uber.jar

But jmxtem is not picking up the JVM for Cassandra for some reason.

Can someone point me in the right direction?  Is there settings in the
cassandra-env.sh file I need to amend to get jmxterm to find the cass jvm?

Im not finding much about it on google.

Thanks,
Phil


On Tue, 18 Sep 2018 at 12:09, Philip Ó Condúin 
wrote:

> Hi All,
>
> I need a little advice.  I'm trying to access the JMX terminal using
> *jmxterm-1.0-alpha-4-uber.jar* with a very simple default install of C*
> 3.11.3
>
> I keep getting the following:
>
> [cassandra@reaper-1 conf]$ java -jar jmxterm-1.0-alpha-4-uber.jar
> Welcome to JMX terminal. Type "help" for available commands.
> $>open 1666
> *#NullPointerException: No such PID 1666*
> $>
>
> C* is running with a PID of 1666.  I've tried setting JMX_LOCAL=no and
> have even created a new VM to test it.
>
> Does anyone know what I might be doing wrong here?
>
> Kind Regards,
> Phil
>
>

-- 
Regards,
Phil

Re: What is the cons of changing LCS fanout option to 100 or even bigger?

[Marcus Eriksson]

problem would be that for every file you flush, you would recompact all of
L1 - files are flushed to L0, then compacted together with all overlapping
files in L1.

On Tue, Sep 18, 2018 at 4:53 AM 健 戴  wrote:

> Hi,
>
> I have one table having 2T data saved in c* each node.
> And if using LCS, the data will have 5 level:
>
>
>- L1: 160M * 10 = 1.6G
>- L2: 1.6G * 10 = 16G
>- L3: 16G * 10 = 160G
>- L4: 160G * 10 = 1.6T
>- L5: 1.6T * 10 = 16T
>
> When I looking into the source code, I found an option: fanout_size.
>
> The default value is 10. What about change this value to 100? Then the
> level will reduce to 3:
>
>- L1: 160M * 100 = 16G
>- L2: 16G * 100 = 1.6T
>- L3: 1.6T * 100 = 160T
>
> Or even could I set this to 1? And all files are in a same level.
> Should it be better then?
> What is the cons of the bigger value of this option?
>
> Thanks for your help.
>
>
> Jian
>

jmxterm "#NullPointerException: No such PID "

[Philip Ó Condúin]

Hi All,

I need a little advice.  I'm trying to access the JMX terminal using
*jmxterm-1.0-alpha-4-uber.jar* with a very simple default install of C*
3.11.3

I keep getting the following:

[cassandra@reaper-1 conf]$ java -jar jmxterm-1.0-alpha-4-uber.jar
Welcome to JMX terminal. Type "help" for available commands.
$>open 1666
*#NullPointerException: No such PID 1666*
$>

C* is running with a PID of 1666.  I've tried setting JMX_LOCAL=no and have
even created a new VM to test it.

Does anyone know what I might be doing wrong here?

Kind Regards,
Phil

Re: Major compaction ignoring one SSTable? (was Re: Fresh SSTable files (due to repair?) in a static table (was Re: Drop TTLd rows: upgradesstables -a or scrub?))

[Oleksandr Shulgin]

On Tue, Sep 18, 2018 at 10:38 AM Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com> wrote:

>
> any indications in Cassandra log about insufficient disk space during
> compactions?
>

Bingo!  The following was logged around the time compaction was started
(and I only looked around when it was finishing):

Not enough space for compaction, 284674.12MB estimated.  Reducing scope.

That still leaves a question why the estimate doesn't take into account the
tombstones which will be dropped in the process.  Because actually it takes
only slightly more than 100GB in the end, as seen on the other nodes.

Thanks, Thomas!
--
Alex

RE: Major compaction ignoring one SSTable? (was Re: Fresh SSTable files (due to repair?) in a static table (was Re: Drop TTLd rows: upgradesstables -a or scrub?))

[Steinmaurer, Thomas]

Alex,

any indications in Cassandra log about insufficient disk space during 
compactions?

Thomas

From: Oleksandr Shulgin 
Sent: Dienstag, 18. September 2018 10:01
To: User 
Subject: Major compaction ignoring one SSTable? (was Re: Fresh SSTable files 
(due to repair?) in a static table (was Re: Drop TTLd rows: upgradesstables -a 
or scrub?))

On Mon, Sep 17, 2018 at 4:29 PM Oleksandr Shulgin 
mailto:oleksandr.shul...@zalando.de>> wrote:

Thanks for your reply!  Indeed it could be coming from single-SSTable 
compaction, this I didn't think about.  By any chance looking into 
compaction_history table could be useful to trace it down?

Hello,

Yet another unexpected thing we are seeing is that after a major compaction 
completed on one of the nodes there are two SSTables instead of only one (time 
is UTC):

-rw-r--r-- 1 999 root 99G Sep 18 00:13 mc-583-big-Data.db -rw-r--r-- 1 999 root 
70G Mar 8 2018 mc-74-big-Data.db

The more recent one must be the result of major compaction on this table, but 
why the other one from March was not included?

The logs don't help to understand the reason, and from compaction history on 
this node the following record seems to be the only trace:

@ Row 1
---+--
 id| b6feb180-bad7-11e8-9f42-f1a67c22839a
 bytes_in  | 223804299627
 bytes_out | 105322622473
 columnfamily_name | XXX
 compacted_at  | 2018-09-18 00:13:48+
 keyspace_name | YYY
 rows_merged   | {1: 31321943, 2: 11722759, 3: 382232, 4: 23405, 5: 2250, 
6: 134}

This also doesn't tell us a lot.

This has happened only on one node out of 10 where the same command was used to 
start major compaction on this table.

Any ideas what could be the reason?

For now we have just started major compaction again to ensure these two last 
tables are compacted together, but we would really like to understand the 
reason for this behavior.

Regards,
--
Alex

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313

Re: Read timeouts when performing rolling restart

[Riccardo Ferrari]

Allright, coming back after some homeworks. Thank you Alain

About hardware:
m1.xlarge 4vcpu 15GB ram
- 4 spinning disks configured in raid0

#About compactors:
- I 've moved them back to 2 concurrent compactors. I can say in general I
don't see more than 80-ish pending compactions (during compaction times).
This is true when using watch nodetool tpstats too
- Throughput is 16MB/s

#About Hints
When a node boots up I clearly see a spike on pending compactions (still
around 80-ish). During boot when it starts receiving hints the system load
grows to an unsastainalbe level (30+ )in the logs I get the message
"[HINTS|MUTATION] messages were dropped in the last 5000ms..."
Now, after tuning compactors I still see some dropped messages (all of them
MUTATION or. HINTS). On some nodes is as low 0 on other nodes as high as
32k. In particular out of 6 nodes there is one doing 32k dropped messages
one doing 16k and one few hundreds, and somehow the are always the same
nodes.

#About GC
I have moved all my nodes on CMS: Xms and Xmx 8G and Xmn4G. You already
helped on the JVM tuning. Despite G1 was doing pretty good CMS turned out
to be more consistent. Under heavy load G1 can stop longer than CMS. GC
pauses as seen on couple of nodes are between 200 and 430ms.


# Couple of notes:
I see some nodes with higher system load (not data) than other in
particular one of the nodes despite has 170+GB of data its system load
barely and rarely reach 2.0.
I recently adopted Reaper (Thanks TLP!). Out of my 2 keyspaces one is
running (repairing) just fine the second one that is bigger/older is simply
not progressing. Maybe this can give a hint on where to look into...

Thanks!

On Fri, Sep 14, 2018 at 11:54 AM, Alain RODRIGUEZ 
wrote:

> Hello Riccardo.
>
> Going to VPC, use GPFS and NTS all sounds very reasonable to me. As you
> said, that's another story. Good luck with this. Yet the current topology
> should also work well and I am wondering why the query does not find any
> other replica available.
>
> About your problem at hand:
>
> It's unclear to me at this point if the nodes are becoming unresponsive.
> My main thought on your first email was that you were facing some issue
> were due to the topology or to the client configuration you were missing
> replicas, but I cannot see what's wrong (if not authentication indeed, but
> you don't use it).
> Then I am thinking it might be indeed due to many nodes getting extremely
> busy at the moment of the restart (of any of the nodes), because of this:
>
> After rising the compactors to 4 I still see some dropped messages for
>> HINT and MUTATIONS. This happens during startup. Reason is "for internal
>> timeout". Maybe too many compactors?
>
>
> Some tuning information/hints:
>
> * The number of *concurrent_compactor* should be between 1/4 and 1/2 of
> the total number of cores and generally no more than 8. It should ideally
> never be equal to the number of CPU cores as we want power available to
> process requests at any moment.
> * Another common bottleneck is the disk throughput. If compactions are
> running too fast, it can harm as well. I would fix the number of
> concurrent_compactors as mentioned above and act on the compaction
> throughput instead
> * If hints are a problem, or rather saying, to make sure they are involved
> in the issue you see, why not disabling hints completely on all nodes and
> try a restart? Anything that can be disabled is an optimization. You do not
> need hinted handoff if you run a repair later on (or if you operate with a
> strong consistency and do not perform deletes for example). You can give
> this a try: https://github.com/apache/cassandra/blob/cassandra-3.0.
> 6/conf/cassandra.yaml#L44-L46.
> * Not as brutal, you can try slowing down the hint transfer speed:
> https://github.com/apache/cassandra/blob/cassandra-3.0.6/conf/
> cassandra.yaml#L57-L67.
> * Check for GC that would be induced by the pressure put by hints
> delivery, compactions and the first load of the memory on machine start.
> Any GC activity that would be shown in the logs?
> * As you are using AWS, tuning the phi_convict_threshold to 10-12 could
> help as well not making the node down (if that's what happens).
> * Do you see any specific part of the hardware being the bottleneck or
> bing especially strongly used during a restart? Maybe use:
> 'dstat -D  -lvrn 10' (where  is like 'xvdq'). I believe this
> command shows Bytes, not bits, thus ‘50M' is 50 MB or 400 Mb.
> * What hardware are you using?
> * Could you also run a 'watch -n1 -d "nodetool tpstats"' during the nodes
> restart as well and see what threads are 'PENDING' during the restart. For
> instance, if the flush_writter is pending, the next write to this table has
> to wait for the data to be flushed. It can be multiple things, but having
> an interactive view of the pending requests might lead you to the root
> cause of the issue.
>
> C*heers,
> ---
> Alain Rodriguez - @arodream - 

Major compaction ignoring one SSTable? (was Re: Fresh SSTable files (due to repair?) in a static table (was Re: Drop TTLd rows: upgradesstables -a or scrub?))

[Oleksandr Shulgin]

On Mon, Sep 17, 2018 at 4:29 PM Oleksandr Shulgin <
oleksandr.shul...@zalando.de> wrote:

>
> Thanks for your reply!  Indeed it could be coming from single-SSTable
> compaction, this I didn't think about.  By any chance looking into
> compaction_history table could be useful to trace it down?
>

Hello,

Yet another unexpected thing we are seeing is that after a major compaction
completed on one of the nodes there are two SSTables instead of only one
(time is UTC):

-rw-r--r-- 1 999 root 99G Sep 18 00:13 mc-583-big-Data.db -rw-r--r-- 1 999
root 70G Mar 8 2018 mc-74-big-Data.db

The more recent one must be the result of major compaction on this table,
but why the other one from March was not included?

The logs don't help to understand the reason, and from compaction history
on this node the following record seems to be the only trace:

@ Row 1
---+--
 id| b6feb180-bad7-11e8-9f42-f1a67c22839a
 bytes_in  | 223804299627
 bytes_out | 105322622473
 columnfamily_name | XXX
 compacted_at  | 2018-09-18 00:13:48+
 keyspace_name | YYY
 rows_merged   | {1: 31321943, 2: 11722759, 3: 382232, 4: 23405, 5:
2250, 6: 134}

This also doesn't tell us a lot.

This has happened only on one node out of 10 where the same command was
used to start major compaction on this table.

Any ideas what could be the reason?

For now we have just started major compaction again to ensure these two
last tables are compacted together, but we would really like to understand
the reason for this behavior.

Regards,
--
Alex

Re: Fresh SSTable files (due to repair?) in a static table (was Re: Drop TTLd rows: upgradesstables -a or scrub?)

[Oleksandr Shulgin]

On Mon, Sep 17, 2018 at 4:41 PM Jeff Jirsa  wrote:

> Marcus’ idea of row lifting seems more likely, since you’re using STCS -
> it’s an optimization to “lift” expensive reads into a single sstable for
> future reads (if a read touches more than - I think - 4? sstables, we copy
> it back into the memtable so it’s flushed into a single sstable), so if you
> have STCS and you’re still doing reads, it could definitely be that.
>

A-ha, that's eye-opening: it could definitely be that.  Thanks for
explanation!

--
Alex

Re: Scale SASI index

[Hannu Kröger]

You shouldn’t need to. You just scale up and run ”nodetool cleanup” and that 
will take care of it. 

Hannu

> onmstester onmstester  kirjoitti 18.9.2018 kello 8.52:
> 
> By adding new nodes to cluster, should i rebuild SASI indexes on all nodes ?
> 
>