Re: Read Latency Doubles After Shrinking Cluster and Never Recovers

[Nicolas Guyomar]

Really wild guess : do you monitor I/O performance and are positive those
are the same over the past year ? (network becoming a little busier, hard
drive a bit slower and so on) ?
Wild guess 2 : a new 'monitoring' software (log shipping agent for
instance) added meanwhile on the box ?

On 11 June 2018 at 16:56, Jeff Jirsa  wrote:

> No
>
> --
> Jeff Jirsa
>
>
> On Jun 11, 2018, at 7:49 AM, Fd Habash  wrote:
>
> I will check for both.
>
>
>
> On a different subject, I have read some user testimonies that running
> ‘nodetool cleanup’ requires a C* process reboot at least around 2.2.8. Is
> this true?
>
>
>
>
>
> 
> Thank you
>
>
>
> *From: *Nitan Kainth 
> *Sent: *Monday, June 11, 2018 10:40 AM
> *To: *user@cassandra.apache.org
> *Subject: *Re: Read Latency Doubles After Shrinking Cluster and Never
> Recovers
>
>
>
> I think it would because it Cassandra will process more sstables to create
> response to read queries.
>
>
>
> Now after clean if the data volume is same and compaction has been
> running, I can’t think of any more diagnostic step. Let’s wait for other
> experts to comment.
>
>
>
> Can you also check sstable count for each table just to be sure that they
> are not extraordinarily high?
>
> Sent from my iPhone
>
>
> On Jun 11, 2018, at 10:21 AM, Fd Habash  wrote:
>
> Yes we did after adding the three nodes back and a full cluster repair as
> well.
>
>
>
> But even it we didn’t run cleanup, would it have impacted read latency the
> fact that some nodes still have sstables that they no longer need?
>
>
>
> Thanks
>
>
>
> 
> Thank you
>
>
>
> *From: *Nitan Kainth 
> *Sent: *Monday, June 11, 2018 10:18 AM
> *To: *user@cassandra.apache.org
> *Subject: *Re: Read Latency Doubles After Shrinking Cluster and Never
> Recovers
>
>
>
> Did you run cleanup too?
>
>
>
> On Mon, Jun 11, 2018 at 10:16 AM, Fred Habash  wrote:
>
> I have hit dead-ends every where I turned on this issue.
>
>
>
> We had a 15-node cluster  that was doing 35 ms all along for years. At
> some point, we made a decision to shrink it to 13. Read latency rose to
> near 70 ms. Shortly after, we decided this was not acceptable, so we added
> the three nodes back in. Read latency dropped to near 50 ms and it has been
> hovering around this value for over 6 months now.
>
>
>
> Repairs run regularly, load on cluster nodes is even,  application
> activity profile has not changed.
>
>
>
> Why are we unable to get back the same read latency now that the cluster
> is 15 nodes large same as it was before?
>
>
>
> --
>
>
>
> 
> Thank you
>
>
>
>
>
>
>
>
>

Re: apache-cassandra 2.2.8 rpm

[Nicolas Guyomar]

Hi,

I believe this rpm was built by Datastax right ?
https://rpm.datastax.com/community/noarch/ is what you seem to be looking
for
Otherwise newest rpm are here :
https://www.apache.org/dist/cassandra/redhat/22x/

On 5 June 2018 at 16:21, ukevg...@gmail.com  wrote:

> Hi everybody,
>
> I am not able to find an RPM package for apache-cassandra 2.2.8
>
> Is there anyone who can share a link I really couldn't find it.
>
> Thank you
>
> Ev
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: how to immediately delete tombstones

[Nicolas Guyomar]

Hi,

You need to manually force compaction if you do not care ending up with one
big sstable (nodetool compact)

On 31 May 2018 at 11:07, onmstester onmstester  wrote:

> Hi,
> I've deleted 50% of my data row by row now disk usage of cassandra data is
> more than 80%.
> The gc_grace of table was default (10 days), now i set that to 0, although
> many compactions finished but no space reclaimed so far.
> How could i force deletion of tombstones in sstables and reclaim the disk
> used by deleted rows?
> I'm using cassandra on a single node.
>
> Sent using Zoho Mail 
>
>
>

Re: Adding new nodes to cluster to speedup pending compactions

[Nicolas Guyomar]

Hi Mikhail,

Could you please provide :
- your cluster version/topology (number of nodes, cpu, ram available etc)
- what kind of underlying storage you are using
- cfstat using -H option cause I'm never sure I'm converting bytes=>GB

You are storing 1Tb per node, so long running compaction is not really a
surprise, you can play with concurrent compaction thread number, compaction
throughput to begin with


On 27 April 2018 at 16:59, Mikhail Tsaplin  wrote:

> Hi,
> I have a five nodes C* cluster suffering from a big number of pending
> compaction tasks: 1) 571; 2) 91; 3) 367; 4) 22; 5) 232
>
> Initially, it was holding one big table (table_a). With Spark, I read that
> table, extended its data and stored in a second table_b. After this
> copying/extending process the number of compaction tasks in the cluster has
> grown up. From nodetool cfstats (see output at the bottom): table_a has 20
> SSTables and table_b has 18219.
>
> As I understand table_b has a big SSTables number because data was
> transferred from one table to another within a short time and eventually
> this tables will be compacted. But now I have to read whole data from this
> table_b and send it to Elasticsearch. When Spark reads this table some
> Cassandra nodes are dying because of OOM.
>
> I think that when compaction will be completed - the Spark reading job
> will work fine.
>
> The question is how can I speed up compaction process, what if I will add
> another two nodes to cluster - will compaction finish faster? Or data will
> be copied to new nodes but compaction will continue on the original set of
> SSTables?
>
>
> *Nodetool cfstats output:
>
> Table: table_a
> SSTable count: 20
> Space used (live): 1064889308052
> Space used (total): 1064889308052
> Space used by snapshots (total): 0
> Off heap memory used (total): 1118106937
> SSTable Compression Ratio: 0.12564594959566894
> Number of keys (estimate): 56238959
> Memtable cell count: 76824
> Memtable data size: 115531402
> Memtable off heap memory used: 0
> Memtable switch count: 17
> Local read count: 0
> Local read latency: NaN ms
> Local write count: 77308
> Local write latency: 0.045 ms
> Pending flushes: 0
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 120230328
> Bloom filter off heap memory used: 120230168
> Index summary off heap memory used: 2837249
> Compression metadata off heap memory used: 995039520
> Compacted partition minimum bytes: 1110
> Compacted partition maximum bytes: 52066354
> Compacted partition mean bytes: 133152
> Average live cells per slice (last five minutes): NaN
> Maximum live cells per slice (last five minutes): 0
> Average tombstones per slice (last five minutes): NaN
> Maximum tombstones per slice (last five minutes): 0
>
>
> nodetool cfstats table_b
> Keyspace: dump_es
> Read Count: 0
> Read Latency: NaN ms.
> Write Count: 0
> Write Latency: NaN ms.
> Pending Flushes: 0
> Table: table_b
> SSTable count: 18219
> Space used (live): 1316641151665
> Space used (total): 1316641151665
> Space used by snapshots (total): 0
> Off heap memory used (total): 3863604976
> SSTable Compression Ratio: 0.20387645535477916
> Number of keys (estimate): 712032622
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 0
> Local read count: 0
> Local read latency: NaN ms
> Local write count: 0
> Local write latency: NaN ms
> Pending flushes: 0
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 2382971488
> Bloom filter off heap memory used: 2742320056
> Index summary off heap memory used: 371500752
> Compression metadata off heap memory used: 749784168
> Compacted partition minimum bytes: 771
> Compacted partition maximum bytes: 1629722
> Compacted partition mean bytes: 3555
> Average live cells per slice (last five minutes): 132.375
> Maximum live cells per slice (last five minutes): 149
> Average tombstones per slice (last five 

Re: 答复: 答复: Cassandra 3.7 - Problem with Repairs - all nodes failing

[Nicolas Guyomar]

Hi,

I believe there is no such official apache documentation, and since
Datastax removed the old guide to redirect to the apache one, you are left
with Google to look for guidance.

As a general advice  (from what I can recall) :
- you might want to run nodetool cleanup before upgrading to free so space
if you forget to do it after a node addition
- do not let your repair scheduler launch a repair while upgrading  (wait
for ongoing compaction to complete if possible)
- check the java version you are using, and update if needed
- upgrade one node at a time and wait for U/N state in nodetool status
- run upgradesstable on each node one at a time once every node is up and
running with the new version (make sure you have some free space left on
disk because that will rewrite existing sstable in-place)

If you can afford it, check the upgrade procedure on a test cluster as
always ;)
- Some default configuration might be changed between version (java GC for
instance maybe ? ) => compare your cassandra.yaml and jvm.options with the
new one !

Cross check what I wrote with what you can find on the internet !


On 20 April 2018 at 10:21, Xiangfei Ni <xiangfei...@cm-dt.com> wrote:

> Hi Nicolas,
>
> Thanks for your reply.
>
> The doc you sent to me only includes what have been upgrade between every
> version,but I want the doc of how to do the online upgrade,do you have this
> kind of doc?
>
>
>
> Best Regards,
>
>
>
> 倪项菲*/ **David Ni*
>
> 中移德电网络科技有限公司
>
> Virtue Intelligent Network Ltd, co.
>
> Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
>
> Mob: +86 13797007811|Tel: + 86 27 5024 2516
>
>
>
> *发件人:* Nicolas Guyomar <nicolas.guyo...@gmail.com>
> *发送时间:* 2018年4月20日 15:44
> *收件人:* user@cassandra.apache.org
> *主题:* Re: 答复: Cassandra 3.7 - Problem with Repairs - all nodes failing
>
>
>
> Hi,
>
>
>
> You can have a look to https://github.com/apache/
> cassandra/blob/trunk/NEWS.txt  which list every modification / advice for
> upgrading between each C* version
>
>
>
> On 20 April 2018 at 09:25, Xiangfei Ni <xiangfei...@cm-dt.com> wrote:
>
> By the way,is there official documentation for online upgrade cassandra
> from 3.9 to 3.11.2 which I can follow?
>
>
>
> Best Regards,
>
>
>
> 倪项菲*/ **David Ni*
>
> 中移德电网络科技有限公司
>
> Virtue Intelligent Network Ltd, co.
>
> Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
>
> Mob: +86 13797007811|Tel: + 86 27 5024 2516
>
>
>
> *发件人:* Anup Shirolkar <anup.shirol...@instaclustr.com>
> *发送时间:* 2018年4月20日 11:34
> *收件人:* user@cassandra.apache.org
> *主题:* Re: Cassandra 3.7 - Problem with Repairs - all nodes failing
>
>
>
> Contd.
>
>
>
> Upgrading from 3.7 to 3.11.1 will not involving any major changes.
>
> It can be achieved without any downtime and it should not impact on
> Cassandra clients.
>
> You can test the upgrade on a test cluster to be sure if you are
> considering to upgrade prod.
>
>
>
> Thanks,
>
> Anup
>
>
>
> On 20 April 2018 at 13:28, Anup Shirolkar <anup.shirol...@instaclustr.com>
> wrote:
>
> Hi Leena,
>
>
>
> The repairs are most likely failing because of some bug in Cassandra 3.7.
> I don't have a JIRA reference handy but there are quite some issues in this
> version.
>
>
>
> Considering your scenario, it is highly recommended that you should
> upgrade to 3.11.1.
>
> Although, you have mentioned that upgrading is not an option, I would like
> to tell you that
>
>
>
> On 19 April 2018 at 23:19, Leena Ghatpande <lghatpa...@hotmail.com> wrote:
>
> we have 8 node prod cluster running on cassandra 3.7. Our 2 largest tables
> have around 100M and 30M rows respectively while all others are relatively
> smaller.
>
> we have been running repairs on alternate days on 2 of our keyspaces.
> We run repair on each node in the cluster with the -pr option on every
> table within each keyspace individually. Repairs are run sequentially on
> each node
> These were working fine, but with no change on the systems, they have
> started failing since last month.
>
> The repairs have started failing for each table on every node with no
> specific error.
>
> I have tried running scrub on every table and then running repair , but
> still the repair fails for all tables.
>
> Our smallest table with only 100 rows also fails on repair.
>
> But if I run the repair with DC option (-dc localdatacenter) for local
> datacenters, then the repairs are successfully. Is this indication that the
> repairs are good?
> we would still want the repairs to work on individually tables as expected.
>
> Need help trying t

Re: 答复: Cassandra 3.7 - Problem with Repairs - all nodes failing

[Nicolas Guyomar]

Hi,

You can have a look to
https://github.com/apache/cassandra/blob/trunk/NEWS.txt  which list every
modification / advice for upgrading between each C* version

On 20 April 2018 at 09:25, Xiangfei Ni  wrote:

> By the way,is there official documentation for online upgrade cassandra
> from 3.9 to 3.11.2 which I can follow?
>
>
>
> Best Regards,
>
>
>
> 倪项菲*/ **David Ni*
>
> 中移德电网络科技有限公司
>
> Virtue Intelligent Network Ltd, co.
>
> Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
>
> Mob: +86 13797007811|Tel: + 86 27 5024 2516
>
>
>
> *发件人:* Anup Shirolkar 
> *发送时间:* 2018年4月20日 11:34
> *收件人:* user@cassandra.apache.org
> *主题:* Re: Cassandra 3.7 - Problem with Repairs - all nodes failing
>
>
>
> Contd.
>
>
>
> Upgrading from 3.7 to 3.11.1 will not involving any major changes.
>
> It can be achieved without any downtime and it should not impact on
> Cassandra clients.
>
> You can test the upgrade on a test cluster to be sure if you are
> considering to upgrade prod.
>
>
>
> Thanks,
>
> Anup
>
>
>
> On 20 April 2018 at 13:28, Anup Shirolkar 
> wrote:
>
> Hi Leena,
>
>
>
> The repairs are most likely failing because of some bug in Cassandra 3.7.
> I don't have a JIRA reference handy but there are quite some issues in this
> version.
>
>
>
> Considering your scenario, it is highly recommended that you should
> upgrade to 3.11.1.
>
> Although, you have mentioned that upgrading is not an option, I would like
> to tell you that
>
>
>
> On 19 April 2018 at 23:19, Leena Ghatpande  wrote:
>
> we have 8 node prod cluster running on cassandra 3.7. Our 2 largest tables
> have around 100M and 30M rows respectively while all others are relatively
> smaller.
>
> we have been running repairs on alternate days on 2 of our keyspaces.
> We run repair on each node in the cluster with the -pr option on every
> table within each keyspace individually. Repairs are run sequentially on
> each node
> These were working fine, but with no change on the systems, they have
> started failing since last month.
>
> The repairs have started failing for each table on every node with no
> specific error.
>
> I have tried running scrub on every table and then running repair , but
> still the repair fails for all tables.
>
> Our smallest table with only 100 rows also fails on repair.
>
> But if I run the repair with DC option (-dc localdatacenter) for local
> datacenters, then the repairs are successfully. Is this indication that the
> repairs are good?
> we would still want the repairs to work on individually tables as expected.
>
> Need help trying to get the repairs to work properly as we have a big
> migration planned for june .
>
> Upgrading cassandra is not an option right now.
>
>
> Here are some of the errors
> INFO  [AntiEntropyStage:1] 2018-04-18 20:36:51,461 RepairSession.java:181
> - [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d] Received merkle tree for
> clients from / IP
> ERROR [ValidationExecutor:213] 2018-04-18 20:36:51,461 Validator.java:261
> - Failed creating a merkle tree for [repair 
> #223c73c2-4372-11e8-8749-89fc1dde5b7d
> on secure/clients, [(1849652111528073119,1856811324137977760],
> (3733211856223440695,3737790228588239952], 
> (-2500456349659149537,-2498953852677197491],
> (1735271399836012489,1735412813423041471], 
> (1871725370007007817,1890457592856328448],
> (4316163881057906640,4323247409810431754], 
> (4286141602946572160,4308169130179803373],
> (5189663040558066167,5193871822490506231], 
> (7160723554094225326,7161133449395023060],
> (-4363807597425543488,-4361416517953194804], 
> (7008956720664744733,7022523551326267501],
> (-5742986989228874052,-5734436401879059890], 
> (1828335330499002859,1849652111528073119],
> (7072368932695202361,7144087505892848370], 
> (-5791935107311742541,-5781988493712029404],
> (7754917992280096132,7754953485457609099]]], /130.5.123.234 (see log for
> details)
> ERROR [ValidationExecutor:213] 2018-04-18 20:36:51,461
> CassandraDaemon.java:217 - Exception in thread
> Thread[ValidationExecutor:213,1,main]
> java.lang.NullPointerException: null
> INFO  [AntiEntropyStage:1] 2018-04-18 20:36:51,461 RepairSession.java:181
> - [repair #223c73c2-4372-11e8-8749-89fc1dde5b7d] Received merkle tree for
> clients from /IP
> ERROR [Repair#113:12] 2018-04-18 20:36:51,461 CassandraDaemon.java:217 -
> Exception in thread Thread[Repair#113:12,5,RMI Runtime]
> com.google.common.util.concurrent.UncheckedExecutionException:
> org.apache.cassandra.exceptions.RepairException: [repair
> #223c73c2-4372-11e8-8749-89fc1dde5b7d on secure/clients,
> [(1849652111528073119,1856811324137977760], 
> (3733211856223440695,3737790228588239952],
> (-2500456349659149537,-2498953852677197491], 
> (1735271399836012489,1735412813423041471],
> (1871725370007007817,1890457592856328448], 
> (4316163881057906640,4323247409810431754],
> (4286141602946572160,4308169130179803373], 
> 

Re: Mailing list server IPs

[Nicolas Guyomar]

Hi,

I receive similar messages from time to time, and I'm using Gmail ;)  I
believe I never missed a mail on the ML and that you can safely ignore this
message

On 13 April 2018 at 15:06, Jacques-Henri Berthemet <
jacques-henri.berthe...@genesys.com> wrote:

> Hi,
>
>
>
> I’m getting bounce messages from the ML from time to time, see attached
> example. Our IT told me that they need to whitelist all IPs used by
> Cassandra ML server. Is there a way to get those IPs?
>
>
>
> Sorry if it’s not really related to Cassandra itself but I didn’t find
> anything in http://untroubled.org/ezmlm/ezman/ezman5.html commands.
>
>
>
> Regards,
>
> --
>
> Jacques-Henri Berthemet
>
>
> -- Forwarded message --
> From: "user-h...@cassandra.apache.org" 
> To: Jacques-Henri Berthemet 
> Cc:
> Bcc:
> Date: Fri, 6 Apr 2018 20:47:22 +
> Subject: Warning from user@cassandra.apache.org
> Hi! This is the ezmlm program. I'm managing the
> user@cassandra.apache.org mailing list.
>
>
> Messages to you from the user mailing list seem to
> have been bouncing. I've attached a copy of the first bounce
> message I received.
>
> If this message bounces too, I will send you a probe. If the probe bounces,
> I will remove your address from the user mailing list,
> without further notice.
>
>
> I've kept a list of which messages from the user mailing list have
> bounced from your address.
>
> Copies of these messages may be in the archive.
> To retrieve a set of messages 123-145 (a maximum of 100 per request),
> send a short message to:
>
>
> To receive a subject and author list for the last 100 or so messages,
> send a short message to:
>
>
> Here are the message numbers:
>
>60535
>60536
>60548
>
> --- Enclosed is a copy of the bounce message I received.
>
> Return-Path: <>
> Received: (qmail 8848 invoked for bounce); 27 Mar 2018 14:22:11 -
> Date: 27 Mar 2018 14:22:11 -
> From: mailer-dae...@apache.org
> To: user-return-605...@cassandra.apache.org
> Subject: failure notice
>
>
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>

Re: Latest version and Features

[Nicolas Guyomar]

Everything is in the same document, you have a "New features" section plus
an "Upgrading" one

On 11 April 2018 at 17:24, Abdul Patel <abd786...@gmail.com> wrote:

> Nicolas,
> I do see all new features but instructions for upgrade are mentioned in
> next section ..not sure if i missed it ..can you share that section?
>
>
> On Wednesday, April 11, 2018, Abdul Patel <abd786...@gmail.com> wrote:
>
>> Thanks .this is perfect
>>
>> On Wednesday, April 11, 2018, Nicolas Guyomar <nicolas.guyo...@gmail.com>
>> wrote:
>>
>>> Sorry, I should have give you this link instead :
>>> https://github.com/apache/cassandra/blob/trunk/NEWS.txt
>>>
>>> You'll find everything you need IMHO
>>>
>>> On 11 April 2018 at 17:05, Abdul Patel <abd786...@gmail.com> wrote:
>>>
>>>> Thanks.
>>>>
>>>> Is the upgrade process staright forward do we have any documentation to
>>>> upgrade?
>>>>
>>>>
>>>> On Wednesday, April 11, 2018, Jonathan Haddad <j...@jonhaddad.com>
>>>> wrote:
>>>>
>>>>> Move to the latest 3.0, or if you're feeling a little more
>>>>> adventurous, 3.11.2.
>>>>>
>>>>> 4.0 discussion is happening now, nothing is decided.
>>>>>
>>>>> On Wed, Apr 11, 2018 at 7:35 AM Abdul Patel <abd786...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> Our company is planning for upgrading cassandra to maitain the audit
>>>>>> gudilines for patch cycle.
>>>>>> We are currently on 3.1.0, whats the latest stable version and what
>>>>>> are the new features?
>>>>>> Will it be better to wait for 4.0? Any news on what will be new
>>>>>> features in 4.0 ?
>>>>>>
>>>>>
>>>

Re: Latest version and Features

[Nicolas Guyomar]

Sorry, I should have give you this link instead :
https://github.com/apache/cassandra/blob/trunk/NEWS.txt

You'll find everything you need IMHO

On 11 April 2018 at 17:05, Abdul Patel  wrote:

> Thanks.
>
> Is the upgrade process staright forward do we have any documentation to
> upgrade?
>
>
> On Wednesday, April 11, 2018, Jonathan Haddad  wrote:
>
>> Move to the latest 3.0, or if you're feeling a little more adventurous,
>> 3.11.2.
>>
>> 4.0 discussion is happening now, nothing is decided.
>>
>> On Wed, Apr 11, 2018 at 7:35 AM Abdul Patel  wrote:
>>
>>> Hi All,
>>>
>>> Our company is planning for upgrading cassandra to maitain the audit
>>> gudilines for patch cycle.
>>> We are currently on 3.1.0, whats the latest stable version and what are
>>> the new features?
>>> Will it be better to wait for 4.0? Any news on what will be new features
>>> in 4.0 ?
>>>
>>

Re: Latest version and Features

[Nicolas Guyomar]

Hi,

New features can be found here :
https://github.com/apache/cassandra/blob/cassandra-3.11/CHANGES.txt


On 11 April 2018 at 16:51, Jonathan Haddad  wrote:

> Move to the latest 3.0, or if you're feeling a little more adventurous,
> 3.11.2.
>
> 4.0 discussion is happening now, nothing is decided.
>
> On Wed, Apr 11, 2018 at 7:35 AM Abdul Patel  wrote:
>
>> Hi All,
>>
>> Our company is planning for upgrading cassandra to maitain the audit
>> gudilines for patch cycle.
>> We are currently on 3.1.0, whats the latest stable version and what are
>> the new features?
>> Will it be better to wait for 4.0? Any news on what will be new features
>> in 4.0 ?
>>
>

Re: Upgrade to 3.11.2 disabled JMX

[Nicolas Guyomar]

Hi Lucas,

There are usually some logs in system.log at node startup regarding JMX
initialization, are those OK ?

On 5 April 2018 at 22:13, Lucas Benevides 
wrote:

> Dear community members,
>
> I have just upgraded my Cassandra from version 3.11.1 to 3.11.2. I kept my
> previous configuration files: cassandra.yaml and cassandra-env.sh. However,
> when I started the cassandra service, I couldn't connect via JMX (tried to
> to it with a java program, with JConsole and a prometheus client).
>
> When I run netstat -na it does not show port 7199 open.
> Tried to look at the logs but didn't see anything.
>
> Can you figure out why it happened and point any possible solution? Config
> files enable JMX with authtenticaion=false, but it doesn't work.
>
> Thanks in advance,
> Lucas Benevides
>

Re: Text or....

[Nicolas Guyomar]

Hi Shalom,

You might want to compress on application side before inserting in
Cassandra, using the algorithm on your choice, based on compression ratio
and speed that you found acceptable with your use case


On 4 April 2018 at 14:38, shalom sagges  wrote:

> Thanks DuyHai!
>
> I'm using the default table compression. Is there anything else I should
> look into?
> Regarding the table compression, I understand that for write heavy tables,
> it's best to keep the default and not compress it further. Have I
> understood correctly?
>
> On Wed, Apr 4, 2018 at 3:28 PM, DuyHai Doan  wrote:
>
>> Compress it and stores it as a blob.
>> Unless you ever need to index it but I guess even with SASI indexing a so
>> huge text block is not a good idea
>>
>> On Wed, Apr 4, 2018 at 2:25 PM, shalom sagges 
>> wrote:
>>
>>> Hi All,
>>>
>>> A certain application is writing ~55,000 characters for a single row.
>>> Most of these characters are entered to one column with "text" data type.
>>>
>>> This looks insanely large for one row.
>>> Would you suggest to change the data type from "text" to BLOB or any
>>> other option that might fit this scenario?
>>>
>>> Thanks!
>>>
>>
>>
>

Re: "READ messages were dropped ... for internal timeout" after big amount of writes

[Nicolas Guyomar]

Hi,

You also have 62 pending compactions at the same time, which is odd for
such a small dataset IHMO, are you triggering 'nodetool compact' with some
kind of cron you may have forgot after a test or something else ?
Do you have any monitoring in place ? If not, you could let some 'dstat
-tnrvl 10' for a while and look for inconsistency (huge I/O wait at some
point, blocked proc etc)




On 16 March 2018 at 07:33, Dmitry Simonov  wrote:

> Hello!
>
> We are experiencing problems with Cassandra 2.2.8.
> There is a cluster with 3 nodes.
> Problematic keyspace has RF=3 and contains 3 tables (current table sizes:
> 1Gb, 700Mb, 12Kb).
>
> Several times per day there are bursts of "READ messages were dropped ...
> for internal timeout" messages in logs (on every cassandra node). Duration:
> 5 - 15 minutes.
>
> During periods of drops there is always a queue of pending ReadStage tasks:
>
> Pool NameActive   Pending  Completed   Blocked  All 
> time blocked
> ReadStage3267 2976548410 0
>  0
> CompactionExecutor262 802136 0
>  0
>
> Others Active and Pending counters of tpstats are 0.
>
> During drops iostat says there is no read requests to disks, probably
> because all data fits in a disk cache:
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>   56,530,94   39,840,010,002,68
>
> Device: rrqm/s   wrqm/s r/s w/srMB/swMB/s avgrq-sz 
> avgqu-sz   await r_await w_await  svctm  %util
> sda   0,0011,000,00   26,00 0,00 9,09   715,92
>  0,78   30,310,00   30,31   2,46   6,40
> sdb   0,0011,000,00   33,00 0,0010,57   655,70
>  0,83   26,000,00   26,00   2,00   6,60
> sdc   0,00 1,000,00   30,50 0,0010,98   737,07
>  0,91   30,490,00   30,49   2,10   6,40
> sdd   0,0031,500,00   35,00 0,0011,17   653,50
>  0,98   28,170,00   28,17   1,83   6,40
> sde   0,0031,500,00   34,50 0,0010,82   642,10
>  0,67   19,540,00   19,54   1,39   4,80
> sdf   0,00 1,000,00   24,50 0,00 9,71   811,78
>  0,60   24,330,00   24,33   1,88   4,60
> sdg   0,00 1,000,00   23,00 0,00 8,93   795,15
>  0,51   22,260,00   22,26   1,91   4,40
> sdh   0,00 1,000,00   21,50 0,00 8,37   797,05
>  0,45   21,020,00   21,02   1,86   4,00
>
> Disks are SSDs.
>
> Before that drops "Local write count" for problematic table increases very
> fast (10k-30k/sec, while ordinary write rate is 10-30/sec) during 1 minute.
> After that drops start.
>
> Tried useding probabilistic tracing to determine which requests cause
> "write count" to increase, but see no "batch_mutate" queries at all, only
> reads!
>
> There are no GC warnings about long pauses
>
> Could you please help troubleshooting the issue?
>
> --
> Best Regards,
> Dmitry Simonov
>

Re: uneven data movement in one of the disk in Cassandra

[Nicolas Guyomar]

Hi,

This might be a compaction which is running, have you check that ?

On 9 March 2018 at 11:29, Yasir Saleem  wrote:

> Hi Team,
>
>   we are facing issue of uneven data movement in cassandra disk for
> specific which disk03 in our case, however all the disk are consuming
> around 60% of space but disk03 is taking 87% space. Here is configuration
> in yaml and current disk space:
>
> data_file_directories:
> - /data/disk01/cassandra/data_prod/data
> - /data/disk02/cassandra/data_prod/data
> - /data/disk03/cassandra/data_prod/data
> - /data/disk04/cassandra/data_prod/data
> - /data/disk05/cassandra/data_prod/data
>
> disk space:
>
> 734G  417G  280G  60% /data/disk02
> 734G  342G  355G  50% /data/disk05
> 734G  383G  314G  55% /data/disk04
> *734G  599G   98G  87% /data/disk03*
> 734G  499G  198G  60% /data/disk01
>
> Please note that we have tried to delete data several times but still
> space is continuously increasing in disk03. Please let me know if there is
> any workaround to resolve this issue.
>
> Regards,
>
> Yasir.
>
>

Re: cfhistograms InstanceNotFoundException EstimatePartitionSizeHistogram

[Nicolas Guyomar]

I got once this kind of problem and it was exactly what Chris explained.
Could you double check that on this remote host you do not have 2 versions
of cassandra and nodetool is pointing to the old one ?

On 6 March 2018 at 17:17, onmstester onmstester  wrote:

> One my PC i've the exactly same version of Cassandra and histograms
> command works perfectly so i'm sure that nothing is wrong with nodetool
> version
>
> Sent using Zoho Mail 
>
>
>  On Tue, 06 Mar 2018 17:38:04 +0330 *Chris Lohfink
> >* wrote 
>
> Make sure your using same version of nodetool as your version of
> Cassandra. That metric was renamed from EstimatedRowSize so if using a
> version of nodetool made for a more recent version you would get this error
> since EstimatePartitionSizeHistogram doesn’t exist on the older Cassandra
> host.
>
> Chris
>
> Sent from my iPhone
>
> On Mar 6, 2018, at 3:29 AM, onmstester onmstester 
> wrote:
>
> Running this command:
> nodetools cfhistograms keyspace1 table1
>
> throws this exception in production server:
> javax.management.InstanceNotFoundException: org.apache.cassandra.metrics:
> type=Table,keyspace=keyspace1,scope=table1,name=
> EstimatePartitionSizeHistogram
>
> But i have no problem in a test server with few data in it and same
> datamodel.
> I'm using Casssandra 3.
>
> Sent using Zoho Mail 
>
>
>
>

Re: failing GOSSIP on localhost flooding the debug log

[Nicolas Guyomar]

Whoops click "send" to fast

In SImpleSeedProvider :

String[] hosts = "10.1.20.10,10.1.21.10,10.1.22.10,".split(",", -1);
List seeds = new ArrayList(hosts.length);
for (String host : hosts)
{
System.out.println(InetAddress.getByName(host.trim()));
}

output :
/10.1.20.10
/10.1.21.10
/10.1.22.10
localhost/127.0.0.1

This might be consider a "bug" or a nice thing to fix by just ignoring
empty host don't you think ?

Have a nice day

On 2 March 2018 at 11:14, Nicolas Guyomar <nicolas.guyo...@gmail.com> wrote:

> Hi Marco,
>
> Could that be because your seed list has an extra comma in the end of the
> line, thus being interpreted by default as localhost by Cassandra ? And
> because you are listening on the node IP localhost is not reachable  (need
> to check to code to be sure)
> Here => seeds: '10.1.20.10,10.1.21.10,10.1.22.10,'
>
> Wild morning guess ;)
>
> On 2 March 2018 at 11:06, Marco Giovannini <usern...@gmail.com> wrote:
>
>> Hi,
>> ​
>> I'm running
>>  Cassandra
>>  a cluster of 3 nodes on AWS across 3 AZ (every instance has only one
>> interface).
>>
>> Cassandra version is 3.11.1.
>>
>> My debug log get flooded with
>>  messages like this one but the cluster work fine.
>>
>> D
>> EBUG [MessagingService-Outgoing-localhost/127.0.0.1-Gossip] 2018-02-28
>> 15:53:57,314 OutboundTcpConnection.java:545 - Unable to connect to
>> localhost/127.0.0.1
>>
>> Have you ever seen it this issue on your cluster?
>>
>> I attach my conf.
>>
>> I tried to play with the broadcast setting too as I found in the previous
>> discussion but without success.
>>
>> #broadcast_address: "10.1.20.10"
>>
>> https://lists.apache.org/thread.html/%3C561A65ED-7828-40AB-
>> 834d-de2dc8e57...@cisco.com%3E
>>
>> ​​
>> Regards,
>> Marco
>>
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>
>

Re: failing GOSSIP on localhost flooding the debug log

[Nicolas Guyomar]

Hi Marco,

Could that be because your seed list has an extra comma in the end of the
line, thus being interpreted by default as localhost by Cassandra ? And
because you are listening on the node IP localhost is not reachable  (need
to check to code to be sure)
Here => seeds: '10.1.20.10,10.1.21.10,10.1.22.10,'

Wild morning guess ;)

On 2 March 2018 at 11:06, Marco Giovannini  wrote:

> Hi,
> ​
> I'm running
>  Cassandra
>  a cluster of 3 nodes on AWS across 3 AZ (every instance has only one
> interface).
>
> Cassandra version is 3.11.1.
>
> My debug log get flooded with
>  messages like this one but the cluster work fine.
>
> D
> EBUG [MessagingService-Outgoing-localhost/127.0.0.1-Gossip] 2018-02-28
> 15:53:57,314 OutboundTcpConnection.java:545 - Unable to connect to
> localhost/127.0.0.1
>
> Have you ever seen it this issue on your cluster?
>
> I attach my conf.
>
> I tried to play with the broadcast setting too as I found in the previous
> discussion but without success.
>
> #broadcast_address: "10.1.20.10"
>
> https://lists.apache.org/thread.html/%3C561A65ED-7828-
> 40ab-834d-de2dc8e57...@cisco.com%3E
>
> ​​
> Regards,
> Marco
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>

Re: Current active queries and status/type

[Nicolas Guyomar]

Hi,

With
org.apache.cassandra.metrics:type=ClientRequest,scope=Read,name=Latency and
OneMinuteRate you can have such a metrics

As for the state of the request with regards to other node I do no think
you can have that IMHO with JMX  (this is available using TRACING per
request)


On 1 March 2018 at 15:50, D. Salvatore  wrote:

> Hello!
> There is any way to know how many queries a node is currently serving
> through JMX(or other tools)? And the state of the request so, for example,
> if the request is waiting for data from another node?
>
> Thanks
> Salvatore
>

Re: The home page of Cassandra is mobile friendly but the link to the third parties is not

[Nicolas Guyomar]

Works for me : https://issues.apache.org/jira/browse/CASSANDRA-14128

On 1 March 2018 at 10:36, Kenneth Brotman 
wrote:

> The link for 14128 doesn’t work and I can’t find it anywhere.
>
>
>
> Kenneth Brotman
>
>
>
> *From:* kurt greaves [mailto:k...@instaclustr.com]
> *Sent:* Wednesday, February 28, 2018 8:39 PM
> *To:* User
> *Subject:* Re: The home page of Cassandra is mobile friendly but the link
> to the third parties is not
>
>
>
> Already addressed in CASSANDRA-14128
> , however waiting
> on review/comments regarding what we actually do with this page.
>
>
>
> If you want to bring attention to JIRA's, user list is probably
> appropriate. I'd avoid spamming it too much though.
>
>
>
> On 26 February 2018 at 19:22, Kenneth Brotman <
> kenbrot...@yahoo.com.invalid > wrote:
>
> The home page of Cassandra is mobile friendly but the link to the third
> parties from that web page is not.  Any suggestions?
>
>
>
> I made a JIRA for it: https://issues.apache.org/
> jira/browse/CASSANDRA-14263
>
>
>
> Should posts about JIRA’s be on this list or the dev list?
>
>
>
> Kenneth Brotman
>
>
>
>
>
>
>

Cassandra Chunk Cache hints ?

[Nicolas Guyomar]

Hi everyone,

I'm trying to find information on the Chunk Cache, and the "only" thing I
found so far is the Jira :
https://issues.apache.org/jira/browse/CASSANDRA-5863 where this
functionality was added.

I'm wondering if this is something to be adjusted when it's full ? Are
there some rule of thumb for file_cache_size_in_mb ?

As an example, on a quite new cluster with barely 100Gb per node, this
cache is full:

Chunk Cache: entries 7680, size 480 MiB, capacity 480 MiB,
1059773 misses, 1964564 requests, 0.461 recent hit rate, 52360.062
microseconds miss latency

I'm wondering if I could benefit from reducing the number of "misses" by
increasing this cache ?

Is anyone tuning this cache already for some use case ?

Thank you

@Kenneth might be worth an entry in the documentation maybe ?

Re: Jon Haddad on Diagnosing Performance Problems in Production

[Nicolas Guyomar]

Is Jon blog post
https://academy.datastax.com/planet-cassandra/blog/cassandra-summit-recap-diagnosing-problems-in-production
was relocated somewhere ?

On 27 February 2018 at 16:34, Kenneth Brotman 
wrote:

> One presentation that I hope can get updated is Jon Haddad’s very thorough
> presentation on Diagnosing Performance Problems in Production.  I’ve seen
> another version somewhere where I believe he says something like “This
> should help you fix 99% of the problems you see.”  Seems right.
>
>
>
> I’m sure it will be well attended and well viewed for some time.  Here’s
> the version I found: https://www.youtube.com/watch?v=2JlUpgsEdN8
>
>
>
> If Jon did a new version I’d probably stop and watch it three times right
> now.
>
>
>
> If we started with that video inline on the Apache Cassandra web site in
> the troubleshooting section, that would help a lot of people because of the
> quality of the content and the density of the content.
>
>
>
> Kenneth Brotman
>

Re: Driver consistency issue

[Nicolas Guyomar]

Hi,

Adding the java-driver ML for further question, because this does look like
a bug

Are you able to reproduce it a clean environnement using the same C*
version and driver version ?


On 27 February 2018 at 10:05, Abhishek Kumar Maheshwari <
abhishek.maheshw...@timesinternet.in> wrote:

> Hi Alex,
>
> i have only One DC (with name DC1) and have only one keyspace. So i dont
> think so both of the scenario is possible. (yes in my case QUORUM is  
> equivalent
> of ALL)
>
> cqlsh> SELECT * FROM system_schema.keyspaces  where keyspace_name='adlog' ;
>
>  keyspace_name | durable_writes | replication
> ---++---
> 
>  adlog |   True | {'DC1': '2', 'class':
> 'org.apache.cassandra.locator.NetworkTopologyStrategy'}
>
>
> On Tue, Feb 27, 2018 at 2:27 PM, Oleksandr Shulgin <
> oleksandr.shul...@zalando.de> wrote:
>
>> On Tue, Feb 27, 2018 at 9:45 AM, Abhishek Kumar Maheshwari <
>> abhishek.maheshw...@timesinternet.in> wrote:
>>
>>>
>>> i have a KeySpace in Cassandra (cassandra version 3.0.9- total 12
>>> Servers )With below definition:
>>>
>>> {'DC1': '2', 'class': 'org.apache.cassandra.locator.
>>> NetworkTopologyStrategy'}
>>>
>>> Some time i am getting below exception
>>>
>>> [snip]
>>
>>> Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException:
>>> Cassandra timeout during write query at consistency QUORUM (3 replica were
>>> required but only 2 acknowledged the write)
>>> at com.datastax.driver.core.exceptions.WriteTimeoutException.co
>>> py(WriteTimeoutException.java:100)
>>> at com.datastax.driver.core.Responses$Error.asException(Respons
>>> es.java:134)
>>> at com.datastax.driver.core.RequestHandler$SpeculativeExecution
>>> .onSet(RequestHandler.java:525)
>>> at com.datastax.driver.core.Connection$Dispatcher.channelRead0(
>>> Connection.java:1077)
>>>
>>> why its waiting for acknowledged from 3rd server as replication factor
>>> is 2?
>>>
>>
>> I see two possibilities:
>>
>> 1) The data in this keyspace is replicated to another DC, so there is
>> also 'DC2': '2', for example, but you didn't show it.  In this case QUORUM
>> requires more than 2 nodes.
>> 2) The write was targeting a table in a different keyspace than you think.
>>
>> In any case QUORUM (or LOCAL_QUORUM) with RF=2 is equivalent of ALL.  Not
>> sure why would you use it in the first place.
>>
>> For consistency levels involving quorum you want to go with RF=3 in a
>> single DC.  For multi DC you should think if you want QUORUM or EACH_QUORUM
>> for your writes and figure out the RFs from that.
>>
>> Cheers,
>> --
>> Alex
>>
>>
>
>
> --
>
> *Thanks & Regards,*
> *Abhishek Kumar Maheshwari*
> *+91- 805591 <+91%208%2005591> (Mobile)*
>
> Times Internet Ltd. | A Times of India Group Company
>
> FC - 6, Sector 16A, Film City,  Noida,  U.P. 201301 | INDIA
>
> *P** Please do not print this email unless it is absolutely necessary.
> Spread environmental awareness.*
>

Re: Cassandra Hints monitoring

[Nicolas Guyomar]

I find DIFFERENCE to be working for dropped mutation, which IMHO works the
same way as Hint metrics
*select difference(sum("Dropped_Count")) FROM "cassandraDroppedMessage"
groupby host*  is valid when I check with nodetool  over a period of time

Not sure what is not working on your side

On 26 February 2018 at 11:02, Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:

> DIFFERENCE may not work here, if the Hints count is 10 for few hours, the
> difference is always is zero. which is not the correct value.
>
> On Mon, Feb 26, 2018 at 1:18 AM, Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
>> Thanks Alex,
>>
>> Let me try it.
>>
>>
>> On Monday, February 26, 2018, Oleksandr Shulgin <
>> oleksandr.shul...@zalando.de> wrote:
>>
>>> On Mon, Feb 26, 2018 at 10:02 AM, Jai Bheemsen Rao Dhanwada <
>>> jaibheem...@gmail.com> wrote:
>>>
 Thank you Alex,

 I tried "TotalHintsInProgress" already, and I don't see it sending the
 correct metrics. I used mean("TotalHintsInProgress") and I see 0
 always on grafana.
 Do you know what is the correct way to do rate or diff for hints using 
 "TotalHints"?
 I am currently using the below query

 SELECT mean("TotalHints_Count") FROM "cassandraStorage"
 WHERE $timeFilter GROUP BY time(10s)

>>>
>>> I've never really used InfluxDB, but it seems that DIFFERENCE is what
>>> you're looking for: https://docs.influxdata.c
>>> om/influxdb/v1.4/query_language/functions/#difference
>>>
>>> --
>>> Alex
>>>
>>>
>

Re: hardware sizing for insert only scenarios

[Nicolas Guyomar]

Hi,

Before looking for a sizing, have you try looking for application side
compression before inserting you data (this paper is really interresting
https://aaltodoc.aalto.fi/bitstream/handle/123456789/29099/master_Burman_
Michael_2017.pdf?sequence=1 ) ? For timeseries use case this is a major
storage cost saving !

IMHO active data are the one that might be involved in a compaction, so it
might be every data in your table if you are using STCS/LCS, but TWCS will
stop looking at old sstable once their time window is considered close ("An
SSTable from a bucket can never be compacted with an SSTable from another
bucket" => http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html




On 26 February 2018 at 09:58, onmstester onmstester 
wrote:

> Another Question on node density, in this scenario:
> 1. we should keep time series data of some years for a heavy write system
> in Cassandra (> 10K Ops in seconds)
> 2. the system is insert only and inserted data would never be updated
> 3. in partition key, we used number of months since 1970, so data for
> every month would be on separate partitions
> 4. because of rule 2, after the end of month previous partitions would
> never be accessed for write requests
> 5. more than 90% of read requests would concern current month partitions,
> so we merely access Old data, we should just keep them for that 10% of
> reports!
> 6. The overall read in comparison to writes are so small (like 0.0001 % of
> overall time)
>
> So, finally the question:
> Even in this scenario would the active data be the whole data (this month
> + all previous months)? or the one which would be accessed for most reads
> and writes (only the past two months)?
> Could i use more than 3TB  per node for this scenario?
> something like:
> CPU: 5 Core
> RAM: 32 GB
> Disk: 5 TB
>
> Sent using Zoho Mail 
>
>
>

Re: Is it possible / makes it sense to limit concurrent streaming during bootstrapping new nodes?

[Nicolas Guyomar]

Yes you are right, it limit how much data a node will send while streaming
data (repair, boostrap etc) total to other node, so that is does not affec
this node performance.

Boostraping is initiated by the boostraping node itself, which determine,
based on his token, which nodes to ask data from, then it compute its
"streaming plan" and init every session at the same time  (look
at org.apache.cassandra.streaming.StreamResultFuture#init )

But I can see in the code  that connection for session are done
sequentially in a FixedPoolSize executor limited by the node number of
processor, so, if I understand correctly the code, this might be the limit
you might be looking for.

You should have at the same time a limited ongoing streaming session
because of that limit, but I have to admit that because of all the async
method/callback in the code I might be wrong :(


On 20 February 2018 at 14:08, Jürgen Albersdorfer <jalbersdor...@gmail.com>
wrote:

> Hi Nicolas,
> I have seen that ' stream_throughput_outbound_megabits_per_sec', but
> afaik this limits what each node will provide at a maximum.
> What I'm more concerned of is the vast amount of connections to handle and
> the concurrent threads of which at least two get started for every single
> streaming connection.
> I'm a former Java Developer and I know that Threads are expensive even to
> just have them sitting around. JVM is doing fine handling some hundreds of
> threads, but what about some thousands?
>
> And about the cleanup - we are currently massively scaling out a startup
> database. I thought of doing cleanup after that ramp-up phase when scaling
> slows down to maybe a node every 3 to 7 days in average.
> So for now I want to add 32 Machines at one after one and then care about
> the cleanup afterwards one by one.
>
> regards,
> Jürgen
>
> 2018-02-20 13:56 GMT+01:00 Nicolas Guyomar <nicolas.guyo...@gmail.com>:
>
>> Hi Jurgen,
>>
>> stream_throughput_outbound_megabits_per_sec is the "given total
>> throughput in Mbps", so it does limit the "concurrent throughput" IMHO,
>> is it not what you are looking for?
>>
>> The only limits I can think of are :
>> - number of connection between every node and the one boostrapping
>> - number of pending compaction (especially if you have lots of
>> keyspace/table) that could lead to some JVM problem maybe ?
>>
>> Anyway, because while bootstrapping, a node is not accepting reads,
>> configuration like compactionthroughput, concurrentcompactor and
>> streamingthroughput can be set on the fly using nodetool, so you can
>> quickly ajust them
>>
>> Out of curiosity, do you run "nodetool cleanup" in parallel on every
>> nodes left after a boostrap, or do you spread the "cleanup load" ? I have
>> not seen yet one adding a node every day like this ;) have fun !
>>
>>
>>
>> On 20 February 2018 at 13:42, Jürgen Albersdorfer <
>> jalbersdor...@gmail.com> wrote:
>>
>>> Hi, I'm wondering if it is possible resp. would it make sense to limit
>>> concurrent streaming when joining a new node to cluster.
>>>
>>> I'm currently operating a 15-Node C* Cluster (V 3.11.1) and joining
>>> another Node every day.
>>> The 'nodetool netstats' shows it always streams data from all other
>>> nodes.
>>>
>>> How far will this scale? - What happens when I have hundrets or even
>>> thousends of Nodes?
>>>
>>> Has anyone experience with such a Situation?
>>>
>>> Thanks, and regards
>>> Jürgen
>>>
>>
>>
>

Re: Is it possible / makes it sense to limit concurrent streaming during bootstrapping new nodes?

[Nicolas Guyomar]

Hi Jurgen,

stream_throughput_outbound_megabits_per_sec is the "given total throughput
in Mbps", so it does limit the "concurrent throughput" IMHO, is it not what
you are looking for?

The only limits I can think of are :
- number of connection between every node and the one boostrapping
- number of pending compaction (especially if you have lots of
keyspace/table) that could lead to some JVM problem maybe ?

Anyway, because while bootstrapping, a node is not accepting reads,
configuration like compactionthroughput, concurrentcompactor and
streamingthroughput
can be set on the fly using nodetool, so you can quickly ajust them

Out of curiosity, do you run "nodetool cleanup" in parallel on every nodes
left after a boostrap, or do you spread the "cleanup load" ? I have not
seen yet one adding a node every day like this ;) have fun !



On 20 February 2018 at 13:42, Jürgen Albersdorfer 
wrote:

> Hi, I'm wondering if it is possible resp. would it make sense to limit
> concurrent streaming when joining a new node to cluster.
>
> I'm currently operating a 15-Node C* Cluster (V 3.11.1) and joining
> another Node every day.
> The 'nodetool netstats' shows it always streams data from all other nodes.
>
> How far will this scale? - What happens when I have hundrets or even
> thousends of Nodes?
>
> Has anyone experience with such a Situation?
>
> Thanks, and regards
> Jürgen
>

Re: Cassandra is not running

[Nicolas Guyomar]

Hi,

Such a quick failure might indicate that you are trying to start Cassandra
with the latest JDK which is not yet supported.

You should have a look at the /var/log/system.log

On 13 February 2018 at 16:03, Irtiza Ali  wrote:

> Hello everyone:
>
> Whenever I try to run the Cassandra it stop with status:
>
> result of [sudo service cassandra status] command:
>
> ● cassandra.service - LSB: distributed storage system for structured data
>Loaded: loaded (/etc/init.d/cassandra; bad; vendor preset: enabled)
>Active: active (exited) since 2018-02-13 19:57:51 PKT; 31s ago
>  Docs: man:systemd-sysv-generator(8)
>   Process: 28844 ExecStop=/etc/init.d/cassandra stop (code=exited,
> status=0/SUCCESS)
>   Process: 28929 ExecStart=/etc/init.d/cassandra start (code=exited,
> status=0/SUCCESS)
>
> Anyone knows why cassandra is not running properly?
>
> With Regards
> Irtiza Ali
>

Re: GDPR, Right to Be Forgotten, and Cassandra

[Nicolas Guyomar]

Hi,

Thank you Jon for this DeletingCS I'll have a look.
 Not being able to query/access a user data, or at least not being able to
identify a client from its data (@Stefan *crypto shredding and
pseudonymization strategies)* makes sense now

I did interpret the legal text a bit too far on the physical deletion part
!

Thank you all



On 9 February 2018 at 21:42, Stefan Podkowinski <s...@apache.org> wrote:

> Deleting data "without undue delay" in Cassandra can be implemented by
> using crypto shredding and pseudonymization strategies in your data model.
> All you have to do is to make sure that throwing away a person's data
> encryption key will make it impossible to restore personal data and
> impossible to resolve any pseudonyms associated with that person.
>
>
> On 09.02.18 17:10, Nicolas Guyomar wrote:
>
> Hi everyone,
>
> Because of GDPR we really face the need to support “Right to Be Forgotten”
> requests => https://gdpr-info.eu/art-17-gdpr/  stating that *"the
> controller shall have the obligation to erase personal data without undue
> delay"*
>
> Because I usually meet customers that do not have that much clients,
> modeling one partition per client is almost always possible, easing
> deletion by partition key.
>
> Then, appart from triggering a manual compaction on impacted tables using
> STCS, I do not see how I can be GDPR compliant.
>
> I'm kind of surprised not to find any thread on that matter on the ML, do
> you guys have any modeling strategy that would make it easier to get rid of
> data ?
>
> Thank you for any given advice
>
> Nicolas
>
>
>

GDPR, Right to Be Forgotten, and Cassandra

[Nicolas Guyomar]

Hi everyone,

Because of GDPR we really face the need to support “Right to Be Forgotten”
requests => https://gdpr-info.eu/art-17-gdpr/  stating that *"the
controller shall have the obligation to erase personal data without undue
delay"*

Because I usually meet customers that do not have that much clients,
modeling one partition per client is almost always possible, easing
deletion by partition key.

Then, appart from triggering a manual compaction on impacted tables using
STCS, I do not see how I can be GDPR compliant.

I'm kind of surprised not to find any thread on that matter on the ML, do
you guys have any modeling strategy that would make it easier to get rid of
data ?

Thank you for any given advice

Nicolas

Re: Hints folder missing in Cassandra

[Nicolas Guyomar]

Hi,

There are no piece of code in Cassandra that would remove this folder. You
should start looking elsewhere, like other people mentioned (chef, ansible
and so on), good luck


On 8 February 2018 at 22:54, test user  wrote:

> Does anyone have more inputs on the missing hints folder, rather why it
> gets deleted.
>
> Has anyone run into this scenario before?
>
> Regards,
> Cassandra User
>
> On Wed, Feb 7, 2018 at 9:21 PM, test user  wrote:
>
>> The problem is even though, I get that O_RDONLY WARN message, if I try to
>> navigate to the path where the hints folder is stored, the folder is not
>> present.
>> I cannot check the permissions on that folder, its already missing, got
>> deleted somehow.
>>
>> I believe everything runs as the root user.
>>
>> I do see a lot of sstable activity performed by MemTableFlushWriter
>> (ColumnFamilyStore), CompactionExecutor, PerDiskFlushWriter (MemTable)
>> before and after this WARN message.
>>
>> It is not a space issue, I checked that already.
>>
>>
>>
>> On Wed, Feb 7, 2018 at 3:49 PM, Nate McCall 
>> wrote:
>>
>>>
>>> The environment is built using established images for Cassandra 3.10.
 Unfortunately the debug log does not indicate any errors before I start
 seeing the WARN for missing hints folder. I understand that hints file will
 be deleted after replay is complete, but not sure of the root cause of why
 the hints folder is getting deleted.
 When I look at the nodetool status or nodetool ring - it indicates that
 all nodes are up and running in normal state, no node went down. Also, I do
 not see anything the debug logs indicating that a node went down. In such a
 scenario, I am not sure why would HintsWriterExecutor would get triggered.


>>> That error code (O_RDONLY) in the log message indicates that the hints
>>> folder has had its permission bits set to read only.
>>>
>>> We've had several issues with some of the tools doing this type of thing
>>> when they are run as the root user. Is this specific node one on which you
>>> use any of the tools like sstableloader or similar? If so, are you running
>>> them as root?
>>>
>>> Another thought - if it is on a different partition than the data
>>> directory, is there free space left on the underlying device holding:
>>> /var/lib/cassandra/hints?
>>>
>>>
>>> --
>>> -
>>> Nate McCall
>>> Wellington, NZ
>>> @zznate
>>>
>>> CTO
>>> Apache Cassandra Consulting
>>> http://www.thelastpickle.com
>>>
>>
>>
>

Re: Bootstrapping fails with < 128GB RAM ...

[Nicolas Guyomar]

Ok then, following up on the wild guess : because you have quite a lot of
concurrent compactors, maybe it is too much concurrent compactions for the
jvm to deal with (taking into account that your load average of 106 seems
really high IMHO)

55Gb of data is not that much, you can try to reduce those concurrent
compactor to make sure your box is not under too much stress (how many
compaction do you have in parallel during boostrap ? )

In the end, it does seem that you're gonna have to share some heap dump for
further investigation (sorry I'm not gonna help lot on this matter)

On 7 February 2018 at 14:43, Jürgen Albersdorfer <jalbersdor...@gmail.com>
wrote:

> Hi Nicolas,
>
> Do you know how many sstables is this new node suppose to receive ?
>
>
> If I can find out this via nodetool netstats, then this would be 619 as
> following:
>
> # nodetool netstats
> Bootstrap b95371e0-0c0a-11e8-932b-f775227bf21c
> /192.168.1.215 - Receiving 71 files, 7744612158 bytes total. Already
> received 0 files, 893897583 bytes total
> /192.168.1.214 - Receiving 58 files, 5693392001 bytes total. Already
> received 0 files, 1078372756 bytes total
> /192.168.1.206 - Receiving 52 files, 3389096409 bytes total. Already
> received 3 files, 508592758 bytes total
> /192.168.1.213 - Receiving 59 files, 6041633329 bytes total. Already
> received 0 files, 1038760653 bytes total
> /192.168.1.231 - Receiving 79 files, 7579181689 bytes total. Already
> received 4 files, 38387859 bytes total
> /192.168.1.208 - Receiving 51 files, 3272885123 bytes total. Already
> received 3 files, 362450903 bytes total
> /192.168.1.207 - Receiving 56 files, 3028344200 bytes total. Already
> received 3 files, 57790197 bytes total
> /192.168.1.232 - Receiving 79 files, 7268716317 bytes total. Already
> received 1 files, 1127174421 bytes total
> /192.168.1.209 - Receiving 114 files, 21381846105 bytes total.
> Already received 1 files, 961497222 bytes total
>
>
> does disabling compaction_throughput_mb_per_sec or increasing concurrent_
>> compactors has any effect ?
>
>
> I will give it a try:
>
> # nodetool getcompactionthroughput
> Current compaction throughput: 128 MB/s
>
> # nodetool setcompactionthroughput 0
>
> # nodetool getcompactionthroughput
> Current compaction throughput: 0 MB/s
>
> # nodetool getconcurrentcompactors
> Current concurrent compactors in the system is:
> 16
>
>
> Which memtable_allocation_type are you using ?
>
>
> # grep memtable_allocation_type /etc/cassandra/conf/cassandra.yaml
> memtable_allocation_type: heap_buffers
>
>
> thanks so far, regards
> Juergen
>
> 2018-02-07 14:29 GMT+01:00 Nicolas Guyomar <nicolas.guyo...@gmail.com>:
>
>> Hi Jurgen,
>>
>> It does feel like some OOM during boostrap from previous C* v2, but that
>> sould be fixed in your version.
>>
>> Do you know how many sstables is this new node suppose to receive ?
>>
>> Juste a wild guess : it may have something to do with compaction not
>> keeping up because every other nodes are streaming data to this new one
>> (resulting in long lived object in the heap), does disabling
>> compaction_throughput_mb_per_sec or increasing concurrent_compactors has
>> any effect ?
>>
>> Which memtable_allocation_type are you using ?
>>
>>
>> On 7 February 2018 at 12:38, Jürgen Albersdorfer <jalbersdor...@gmail.com
>> > wrote:
>>
>>> Hi, I always face an issue when bootstrapping a Node having less than
>>> 184GB RAM (156GB JVM HEAP) on our 10 Node C* 3.11.1 Cluster.
>>> During bootstrap, when I watch the cassandra.log I observe a growth in
>>> JVM Heap Old Gen which gets not significantly freed any more.
>>> I know that JVM collects on Old Gen only when really needed. I can see
>>> collections, but there is always a remainder which
>>> seems to grow forever without ever getting freed.
>>> After the Node successfully Joined the Cluster, I can remove the extra
>>> 128GB of RAM I have given it for bootstrapping without any further effect.
>>>
>>> It feels like Cassandra will not forget about every single byte streamed
>>> over the Network over time during bootstrapping, - which would be a memory
>>> leak and a major problem, too.
>>>
>>> Or is there something I'm doing wrong? - Any Ideas?
>>>
>>> Here my observations on a failing Bootstrap - The following Node has
>>> 72GB RAM installed, 64GB of it are configured for JVM Heap Space.
>>>
>>> cassandra.log (truncated):
>>> INFO  [Service Thread] 2018-02-07 11:12:49,604 GCInspector.java:284 - G1
>>

Re: Bootstrapping fails with < 128GB RAM ...

[Nicolas Guyomar]

Hi Jurgen,

It does feel like some OOM during boostrap from previous C* v2, but that
sould be fixed in your version.

Do you know how many sstables is this new node suppose to receive ?

Juste a wild guess : it may have something to do with compaction not
keeping up because every other nodes are streaming data to this new one
(resulting in long lived object in the heap), does disabling
compaction_throughput_mb_per_sec or increasing concurrent_compactors has
any effect ?

Which memtable_allocation_type are you using ?


On 7 February 2018 at 12:38, Jürgen Albersdorfer 
wrote:

> Hi, I always face an issue when bootstrapping a Node having less than
> 184GB RAM (156GB JVM HEAP) on our 10 Node C* 3.11.1 Cluster.
> During bootstrap, when I watch the cassandra.log I observe a growth in JVM
> Heap Old Gen which gets not significantly freed any more.
> I know that JVM collects on Old Gen only when really needed. I can see
> collections, but there is always a remainder which
> seems to grow forever without ever getting freed.
> After the Node successfully Joined the Cluster, I can remove the extra
> 128GB of RAM I have given it for bootstrapping without any further effect.
>
> It feels like Cassandra will not forget about every single byte streamed
> over the Network over time during bootstrapping, - which would be a memory
> leak and a major problem, too.
>
> Or is there something I'm doing wrong? - Any Ideas?
>
> Here my observations on a failing Bootstrap - The following Node has 72GB
> RAM installed, 64GB of it are configured for JVM Heap Space.
>
> cassandra.log (truncated):
> INFO  [Service Thread] 2018-02-07 11:12:49,604 GCInspector.java:284 - G1
> Young Generation GC in 984ms.  G1 Eden Space: 14763950080 -> 0; G1 Old Gen:
> 36960206856 -> 39661338640; G1 Survivor Space: 2785017856 -> 1476395008;
> INFO  [Service Thread] 2018-02-07 11:13:00,108 GCInspector.java:284 - G1
> Young Generation GC in 784ms.  G1 Eden Space: 18387828736 -> 0; G1 Old Gen:
> 39661338640 -> 41053847560; G1 Survivor Space: 1476395008 -> 1845493760;
> INFO  [Service Thread] 2018-02-07 11:13:08,639 GCInspector.java:284 - G1
> Young Generation GC in 718ms.  G1 Eden Space: 16743661568 -> 0; G1 Old Gen:
> 41053847560 -> 42832232472; G1 Survivor Space: 1845493760 -> 1375731712;
> INFO  [Service Thread] 2018-02-07 11:13:18,271 GCInspector.java:284 - G1
> Young Generation GC in 546ms.  G1 Eden Space: 15535702016 -> 0; G1 Old Gen:
> 42831004832 -> 44206736544; G1 Survivor Space: 1375731712 -> 1006632960;
> INFO  [Service Thread] 2018-02-07 11:13:35,364 GCInspector.java:284 - G1
> Young Generation GC in 638ms.  G1 Eden Space: 14025752576 -> 0; G1 Old Gen:
> 44206737048 -> 45213369488; G1 Survivor Space: 1778384896 -> 1610612736;
> INFO  [Service Thread] 2018-02-07 11:13:42,898 GCInspector.java:284 - G1
> Young Generation GC in 614ms.  G1 Eden Space: 13388218368 -> 0; G1 Old Gen:
> 45213369488 -> 46152893584; G1 Survivor Space: 1610612736 -> 1006632960;
> INFO  [Service Thread] 2018-02-07 11:13:58,291 GCInspector.java:284 - G1
> Young Generation GC in 400ms.  G1 Eden Space: 13119782912 -> 0; G1 Old Gen:
> 46136116376 -> 47171400848; G1 Survivor Space: 1275068416 -> 771751936;
> INFO  [Service Thread] 2018-02-07 11:14:23,071 GCInspector.java:284 - G1
> Young Generation GC in 303ms.  G1 Eden Space: 11676942336 -> 0; G1 Old Gen:
> 47710958232 -> 48239699096; G1 Survivor Space: 1207959552 -> 973078528;
> INFO  [Service Thread] 2018-02-07 11:14:46,157 GCInspector.java:284 - G1
> Young Generation GC in 305ms.  G1 Eden Space: 11005853696 -> 0; G1 Old Gen:
> 48903342232 -> 49289001104; G1 Survivor Space: 939524096 -> 973078528;
> INFO  [Service Thread] 2018-02-07 11:14:53,045 GCInspector.java:284 - G1
> Young Generation GC in 380ms.  G1 Eden Space: 10569646080 -> 0; G1 Old Gen:
> 49289001104 -> 49586732696; G1 Survivor Space: 973078528 -> 1308622848;
> INFO  [Service Thread] 2018-02-07 11:15:04,692 GCInspector.java:284 - G1
> Young Generation GC in 360ms.  G1 Eden Space: 9294577664 -> 0; G1 Old Gen:
> 50671712912 -> 51269944472; G1 Survivor Space: 905969664 -> 805306368;
> WARN  [Service Thread] 2018-02-07 11:15:07,317 GCInspector.java:282 - G1
> Young Generation GC in 1102ms.  G1 Eden Space: 2617245696 -> 0; G1 Old
> Gen: 51269944472 -> 47310521496; G1 Survivor Space: 805306368 ->
> 301989888;
> 
> INFO  [Service Thread] 2018-02-07 11:16:36,535 GCInspector.java:284 - G1
> Young Generation GC in 377ms.  G1 Eden Space: 7683964928 -> 0; G1 Old Gen:
> 51958433432 -> 52658554008; G1 Survivor Space: 1073741824 -> 1040187392;
> INFO  [Service Thread] 2018-02-07 11:16:41,756 GCInspector.java:284 - G1
> Young Generation GC in 340ms.  G1 Eden Space: 7046430720 -> 0; G1 Old Gen:
> 52624999576 -> 53299987616; G1 Survivor Space: 1040187392 -> 805306368;
> WARN  [Service Thread] 2018-02-07 11:16:44,087 GCInspector.java:282 - G1
> Young Generation GC in 1005ms.  G1 Eden Space: 2617245696 -> 0; G1 Old
> Gen: 53299987616 -> 49659331752; G1 

Re: SSH remote access,permissions issue

[Nicolas Guyomar]

Milenko, please reply to user@cassandra.apache.org not directly to me  ;)


You do not seem to have a cassandra process up and running, can you check
with your friend what he did please ? Maybe something is going on on this
cluster, do not try to start the node he might be working on it :)



On 7 February 2018 at 11:40, milenko markovic <milenko...@yahoo.co.uk>
wrote:

> qb3@qb3-Latitude-E6430-3:~$ cassandra -v
> 3.11.1
> qb3@qb3-Latitude-E6430-3:~$ ps -ef | grep cassandra
> qb3   7017  6859  0 11:40 pts/200:00:00 grep --color=auto cassandra
>
>
>
> On Wednesday, 7 February 2018, 11:34:28 CET, Nicolas Guyomar <
> nicolas.guyo...@gmail.com> wrote:
>
>
> I'm sorry Milenko but I do not understand what you are trying to do.
>
> Are you connected with ssh on a Cassandra node ?
> If so can you check that the process is running ?  (ps -ef | grep
> cassandra)  ?
>
> On 7 February 2018 at 11:31, milenko markovic <milenko...@yahoo.co.uk>
> wrote:
>
> Hi Nicolas
>
> Where is jvm.options file?I have only(after rm -rf
> /var/lib/cassandra/data/ system/*)
>
> /etc/cassandra$ ls
> cassandra.yaml  triggers
>
> netstat -an | grep 7199
>
> does not print anything
>
> Regards
> On Wednesday, 7 February 2018, 11:19:07 CET, Nicolas Guyomar <
> nicolas.guyo...@gmail.com> wrote:
>
>
> Hi Milenko,
>
> Can you check the JMX configuration in jvm.options file and make sure you
> can login without user/pwd ?
> Also, the node might be listening to a specific network interface, can you
> output 'netstat -an | grep 7199' for us ?  (assuming your JMX port config
> is 7199)
>
>
> On 7 February 2018 at 11:01, milenko markovic <milenko...@yahoo.co.uk.
> invalid <milenko...@yahoo.co.uk.invalid>> wrote:
>
> My friend has installed 4 nodes at his server in Norway.I wanted to add
> new nodes to cluster.I have logged in with ssh and password.
> ssh pi3@84.208.89.002 -p 22022
>
> Last login: Wed Feb  7 06:59:05 2018 from 94.189.152.722
>
>
> Later sudo su,and have repeated 100% exactly the same operations as he
> did.When I go for
>  nodetool status
> usage: nodetool [(-h  | --host )] [(-p  | --port )]
> [(-u  | --username )]
> [(-pw  | --password )]
> [(-pwf  | --password-file )]
> 
> []
>
> The most commonly used nodetool commands are:
> assassinate  Forcefully remove a dead node without
> re-replicating any data.  Use as a last resort if you cannot removenode
> bootstrapMonitor/manage node's bootstrap process
>
>
> It happens the same with any other nodetool command.
> less /etc/passwd
>
> qb3:x:1000:1000:qb3,,,:/home/ qb3:/bin/bash
> sshd:x:121:65534::/var/run/ sshd:/usr/sbin/nologin
> milenko:x:1001:1001:Milenko,,, :/home/milenko:/bin/bash
> ntp:x:122:129::/home/ntp:/bin/ false
> cassandra:x:123:130:Cassandra database,,,:/var/lib/ cassandra:/bin/false
>
> Why?
> User cassandra shell is not enabled?
> How to change Cassandra permissions?Is this related to remote accesss?He
> can add nodes without any problems.
>
>
>
>

Re: SSH remote access,permissions issue

[Nicolas Guyomar]

Hi Milenko,

Can you check the JMX configuration in jvm.options file and make sure you
can login without user/pwd ?
Also, the node might be listening to a specific network interface, can you
output 'netstat -an | grep 7199' for us ?  (assuming your JMX port config
is 7199)


On 7 February 2018 at 11:01, milenko markovic <
milenko...@yahoo.co.uk.invalid> wrote:

> My friend has installed 4 nodes at his server in Norway.I wanted to add
> new nodes to cluster.I have logged in with ssh and password.
> ssh pi3@84.208.89.002 -p 22022
>
> Last login: Wed Feb  7 06:59:05 2018 from 94.189.152.722
>
>
> Later sudo su,and have repeated 100% exactly the same operations as he
> did.When I go for
>  nodetool status
> usage: nodetool [(-h  | --host )] [(-p  | --port )]
> [(-u  | --username )]
> [(-pw  | --password )]
> [(-pwf  | --password-file )]
> 
> []
>
> The most commonly used nodetool commands are:
> assassinate  Forcefully remove a dead node without
> re-replicating any data.  Use as a last resort if you cannot removenode
> bootstrapMonitor/manage node's bootstrap process
>
>
> It happens the same with any other nodetool command.
> less /etc/passwd
>
> qb3:x:1000:1000:qb3,,,:/home/qb3:/bin/bash
> sshd:x:121:65534::/var/run/sshd:/usr/sbin/nologin
> milenko:x:1001:1001:Milenko,,,:/home/milenko:/bin/bash
> ntp:x:122:129::/home/ntp:/bin/false
> cassandra:x:123:130:Cassandra database,,,:/var/lib/cassandra:/bin/false
>
> Why?
> User cassandra shell is not enabled?
> How to change Cassandra permissions?Is this related to remote accesss?He
> can add nodes without any problems.
>
>

Re: Increased latency after setting row_cache_size_in_mb

[Nicolas Guyomar]

Your row hit rate is 0.971 which is already very high, IMHO there is
"nothing" left to do here if you can afford to store your entire dataset in
memory

Local read latency: 0.030 ms already seems good to me, what makes you think
that you can achieve more with the relative "small" box you are using ?

You have to keep an eye on other metrics which might be a limiting factor,
like cpu usage, JVM heap lifecycle and so on

For read heavy workflow it is sometimes advised to reduce chunk_length_in_kb
from the default 64kb to 4kb, see if it helps !

On 5 February 2018 at 13:09, mohsin k  wrote:

> Hey Rahul,
>
> Each partition has around 10 cluster keys. Based on nodetool, I can
> roughly estimate partition size to be less than 1KB.
>
> On Mon, Feb 5, 2018 at 5:37 PM, mohsin k 
> wrote:
>
>> Hey Nicolas,
>>
>> My goal is to reduce latency as much as possible. I did wait for warmup.
>> The test ran for more than 15mins, I am not sure why it shows 2mins though.
>>
>>
>>
>> On Mon, Feb 5, 2018 at 5:25 PM, Rahul Singh > > wrote:
>>
>>> What is the average size of your partitions / rows. 1GB may not be
>>> enough.
>>>
>>> Rahul
>>>
>>> On Feb 5, 2018, 6:52 AM -0500, mohsin k ,
>>> wrote:
>>>
>>> Hi,
>>>
>>> I have been looking into different configurations for tuning my
>>> cassandra servers. So, initially I loadtested server using cassandra-stress
>>> tool, with default configs and then tuning one by one config to measure
>>> impact of change. First config, I tried was setting "
>>> *row_cache_size_in_mb*" to 1000 (MB) in yaml, adding caching {'keys':
>>> 'ALL', *'rows_per_partition': 'ALL'*}. After changing these configs, I
>>> observed that latency has increased rather than decreasing. It would be
>>> really helpful if I get to understand why is this the case and what steps
>>> must be taken to decrease the latency.
>>>
>>> I am running a cluster with 4 nodes.
>>>
>>> Following is my schema:
>>>
>>> CREATE TABLE stresstest.user_to_segment (
>>> userid text,
>>> segmentid text,
>>> PRIMARY KEY (userid, segmentid)
>>> ) WITH CLUSTERING ORDER BY (segmentid DESC)
>>> AND bloom_filter_fp_chance = 0.1
>>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}
>>> AND comment = 'A table to hold blog segment user relation'
>>> AND compaction = {'class': 'org.apache.cassandra.db.compa
>>> ction.LeveledCompactionStrategy'}
>>> AND compression = {'chunk_length_in_kb': '64', 'class': '
>>> org.apache.cassandra.io.compress.LZ4Compressor'}
>>> AND crc_check_chance = 1.0
>>> AND dclocal_read_repair_chance = 0.1
>>> AND default_time_to_live = 0
>>> AND gc_grace_seconds = 864000
>>> AND max_index_interval = 2048
>>> AND memtable_flush_period_in_ms = 0
>>> AND min_index_interval = 128
>>> AND read_repair_chance = 0.0
>>> AND speculative_retry = '99PERCENTILE';
>>>
>>> Following are node specs:
>>> RAM: 4GB
>>> CPU: 4 Core
>>> HDD: 250BG
>>>
>>>
>>> Following is the output of 'nodetool info' after setting
>>> row_cache_size_in_mb:
>>>
>>> ID : d97dfbbf-1dc3-4d95-a1d9-c9a8d22a3d32
>>> Gossip active  : true
>>> Thrift active  : false
>>> Native Transport active: true
>>> Load   : 10.94 MiB
>>> Generation No  : 1517571163
>>> Uptime (seconds)   : 9169
>>> Heap Memory (MB)   : 136.01 / 3932.00
>>> Off Heap Memory (MB)   : 0.10
>>> Data Center: dc1
>>> Rack   : rack1
>>> Exceptions : 0
>>> Key Cache  : entries 125881, size 9.6 MiB, capacity 100 MiB,
>>> 107 hits, 126004 requests, 0.001 recent hit rate, 14400 save period in
>>> seconds
>>> Row Cache  : entries 125861, size 31.54 MiB, capacity 1000
>>> MiB, 4262684 hits, 4388545 requests, 0.971 recent hit rate, 0 save
>>> period in seconds
>>> Counter Cache  : entries 0, size 0 bytes, capacity 50 MiB, 0
>>> hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
>>> Chunk Cache: entries 273, size 17.06 MiB, capacity 480 MiB,
>>> 325 misses, 126623 requests, 0.997 recent hit rate, NaN microseconds miss
>>> latency
>>> Percent Repaired   : 100.0%
>>> Token  : (invoke with -T/--tokens to see all 256 tokens)
>>>
>>>
>>> Following is output of nodetool cfstats:
>>>
>>> Total number of tables: 37
>>> 
>>> Keyspace : stresstest
>>> Read Count: 4398162
>>> Read Latency: 0.02184742626579012 ms.
>>> Write Count: 0
>>> Write Latency: NaN ms.
>>> Pending Flushes: 0
>>> Table: user_to_segment
>>> SSTable count: 1
>>> SSTables in each level: [1, 0, 0, 0, 0, 0, 0, 0, 0]
>>> Space used (live): 11076103
>>> Space used (total): 11076103
>>> Space used by snapshots (total): 0
>>> Off heap memory used (total): 107981
>>> SSTable Compression Ratio: 0.5123353861375962
>>> Number of partitions (estimate): 125782
>>> Memtable cell count: 

Re: Increased latency after setting row_cache_size_in_mb

[Nicolas Guyomar]

Hi,

Could you explain a bit more what you are trying to achieve please ?
Performance tuning is by far the most complex problem we have to deal with,
and there are a lot of configuration changes that can be made on a C*
cluster

When you do performance tuning, do not forget that you need to warmup C*
JVM. Judging from the provided graph it seems to me that your test ran for
2min, which is really too short


On 5 February 2018 at 08:16, mohsin k  wrote:

> Hi,
>
> I have been looking into different configurations for tuning my cassandra
> servers. So, initially I loadtested server using cassandra-stress tool,
> with default configs and then tuning one by one config to measure impact of
> change. First config, I tried was setting "*row_cache_size_in_mb*" to
> 1000 (MB) in yaml, adding caching {'keys': 'ALL', *'rows_per_partition':
> 'ALL'*}. After changing these configs, I observed that latency has
> increased rather than decreasing. It would be really helpful if I get to
> understand why is this the case and what steps must be taken to decrease
> the latency.
>
> I am running a cluster with 4 nodes.
>
> Following is my schema:
>
> CREATE TABLE stresstest.user_to_segment (
> userid text,
> segmentid text,
> PRIMARY KEY (userid, segmentid)
> ) WITH CLUSTERING ORDER BY (segmentid DESC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}
> AND comment = 'A table to hold blog segment user relation'
> AND compaction = {'class': 'org.apache.cassandra.db.compa
> ction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_in_kb': '64', 'class': '
> org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
>
> Following are node specs:
> RAM: 4GB
> CPU: 4 Core
> HDD: 250BG
>
>
> Following is the output of 'nodetool info' after setting
> row_cache_size_in_mb:
>
> ID : d97dfbbf-1dc3-4d95-a1d9-c9a8d22a3d32
> Gossip active  : true
> Thrift active  : false
> Native Transport active: true
> Load   : 10.94 MiB
> Generation No  : 1517571163
> Uptime (seconds)   : 9169
> Heap Memory (MB)   : 136.01 / 3932.00
> Off Heap Memory (MB)   : 0.10
> Data Center: dc1
> Rack   : rack1
> Exceptions : 0
> Key Cache  : entries 125881, size 9.6 MiB, capacity 100 MiB,
> 107 hits, 126004 requests, 0.001 recent hit rate, 14400 save period in
> seconds
> Row Cache  : entries 125861, size 31.54 MiB, capacity 1000
> MiB, 4262684 hits, 4388545 requests, 0.971 recent hit rate, 0 save period
> in seconds
> Counter Cache  : entries 0, size 0 bytes, capacity 50 MiB, 0 hits,
> 0 requests, NaN recent hit rate, 7200 save period in seconds
> Chunk Cache: entries 273, size 17.06 MiB, capacity 480 MiB,
> 325 misses, 126623 requests, 0.997 recent hit rate, NaN microseconds miss
> latency
> Percent Repaired   : 100.0%
> Token  : (invoke with -T/--tokens to see all 256 tokens)
>
>
> Following is output of nodetool cfstats:
>
> Total number of tables: 37
> 
> Keyspace : stresstest
> Read Count: 4398162
> Read Latency: 0.02184742626579012 ms.
> Write Count: 0
> Write Latency: NaN ms.
> Pending Flushes: 0
> Table: user_to_segment
> SSTable count: 1
> SSTables in each level: [1, 0, 0, 0, 0, 0, 0, 0, 0]
> Space used (live): 11076103
> Space used (total): 11076103
> Space used by snapshots (total): 0
> Off heap memory used (total): 107981
> SSTable Compression Ratio: 0.5123353861375962
> Number of partitions (estimate): 125782
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 2
> Local read count: 4398162
> Local read latency: 0.030 ms
> Local write count: 0
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 0.0
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 79280
> Bloom filter off heap memory used: 79272
> Index summary off heap memory used: 26757
> Compression metadata off heap memory used: 1952
> Compacted partition minimum bytes: 43
> Compacted partition maximum bytes: 215
> Compacted partition mean bytes: 136
> Average live cells per slice (last five minutes): 5.719932432432432
> Maximum live cells per slice (last five minutes): 10
> Average tombstones per slice (last five minutes): 1.0
> Maximum tombstones per slice (last five minutes): 1
> Dropped Mutations: 0
>
>  Following are my results:
>  The blue graph is before setting row_cache_size_in_mb,
> orange is after
>
> Thanks,
> 

Re: How to start cluster after the yaml file has been changed?

[Nicolas Guyomar]

Hi,

You have an invalid yaml: file:/etc/cassandra/cassandra.yaml. I suppose
what you just changed is not yaml compatible, pay attention to space
betwenn semi-colon and value

You an use any tool like https://jsonformatter.org/yaml-formatter to help
fix this issue

On 5 February 2018 at 09:28, milenko markovic <
milenko...@yahoo.co.uk.invalid> wrote:

> I have changed yaml config file.Now I want to start my cassandra cluster.
> I have restarted node
>  cassandra.service - LSB: distributed storage system for structured data
>Loaded: loaded (/etc/init.d/cassandra; bad; vendor preset: enabled)
>Active: active (running) since ma. 2018-02-05 09:27:05 CET; 2s ago
>  Docs: man:systemd-sysv-generator(8)
>   Process: 4854 ExecStop=/etc/init.d/cassandra stop (code=exited,
> status=0/SUCCESS)
>   Process: 4863 ExecStart=/etc/init.d/cassandra start (code=exited,
> status=0/SUCCESS)
>CGroup: /system.slice/cassandra.service
>└─4944 java -Xloggc:/var/log/cassandra/gc.log -ea
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 
> -XX:+HeapDumpOnOutOfMemoryError
> -Xss256k -XX:StringTableSize=103 -XX:+AlwaysPreTouch -
>
> If I go for
>
> sudo -u milenko /usr/sbin/cassandra
>
> NFO  [main] 2018-02-05 09:27:31,681 YamlConfigurationLoader.java:89 -
> Configuration location: file:/etc/cassandra/cassandra.yaml
> Exception (org.apache.cassandra.exceptions.ConfigurationException)
> encountered during startup: Invalid yaml: file:/etc/cassandra/cassandra.
> yaml
>  Error: null; Can't construct a java object for tag:yaml.org
> ,2002:org.apache.cassandra.config.Config; exception=Cannot create
> property=seed_provider for JavaBean=org.apache.cassandra.
> config.Config@551bdc27; java.lang.reflect.InvocationTargetException;  in
> 'reader', line 10, column 1:
> cluster_name: 'Petter Cluster'
> ^
> Invalid yaml: file:/etc/cassandra/cassandra.yaml
>  Error: null; Can't construct a java object for tag:yaml.org
> ,2002:org.apache.cassandra.config.Config; exception=Cannot create
> property=seed_provider for JavaBean=org.apache.cassandra.
> config.Config@551bdc27; java.lang.reflect.InvocationTargetException;  in
> 'reader', line 10, column 1:
> cluster_name: 'Petter Cluster'
> ^
> ERROR [main] 2018-02-05 09:27:32,005 CassandraDaemon.java:706 - Exception
> encountered during startup: Invalid yaml: file:/etc/cassandra/cassandra.
> yaml
>  Error: null; Can't construct a java object for tag:yaml.org
> ,2002:org.apache.cassandra.config.Config; exception=Cannot create
> property=seed_provider for JavaBean=org.apache.cassandra.
> config.Config@551bdc27; java.lang.reflect.InvocationTargetException;  in
> 'reader', line 10, column 1:
> cluster_name: 'Petter Cluster'
>
> How to fix this?
>
>
>
>

Re: Decommissioned nodes and FailureDetector

[Nicolas Guyomar]

Hi,

Not sure if StorageService should be accessed, but you can check node
movement here :
'org.apache.cassandra.db:type=StorageService/LeavingNodes',
'org.apache.cassandra.db:type=StorageService/LiveNodes',
'org.apache.cassandra.db:type=StorageService/UnreachableNodes',
'org.apache.cassandra.db:type=StorageService/MovingNodes'

On 19 January 2018 at 09:42, Oleksandr Shulgin  wrote:

> Hello,
>
> Is there a better way to monitor for Cassandra nodes going Down than
> querying via JMX for a condition like FailureDetector.DownEndpointCount >
> 0?
>
> The problem for us is when any node is decommissioned, it affects the
> DownEndpointCount for another ~3 days (the famous 72 hours of gossip).
>
> Is there a similar metric to be observed which doesn't include nodes which
> are expected to be down?
>
> Regards,
> --
> Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176
> 127-59-707 <+49%20176%2012759707>
>
>

Re: Alter composite column

[Nicolas Guyomar]

Well it should be as easy as following this :
https://docs.datastax.com/en/cql/3.1/cql/cql_using/use_alter_add.html

But I'm worried that your initial requirement was to change the clustering
key, as Alexander stated, you need to create a new table and transfer your
data in it

On 18 January 2018 at 12:03, Joel Samuelsson <samuelsson.j...@gmail.com>
wrote:

> It was indeed created with C* 1.X
> Do you have any links or otherwise on how I would add the column4? I don't
> want to risk destroying my data.
>
> Best regards,
> Joel
>
> 2018-01-18 11:18 GMT+01:00 Nicolas Guyomar <nicolas.guyo...@gmail.com>:
>
>> Hi Joel,
>>
>> You cannot alter a table primary key.
>>
>> You can however alter your existing table to only add column4 using cqlsh
>> and cql, even if this table as created back with C* 1.X for instance
>>
>> On 18 January 2018 at 11:14, Joel Samuelsson <samuelsson.j...@gmail.com>
>> wrote:
>>
>>> So to rephrase that in CQL terms I have a table like this:
>>>
>>> CREATE TABLE events (
>>> key text,
>>> column1 int,
>>> column2 int,
>>> column3 text,
>>> value text,
>>> PRIMARY KEY(key, column1, column2, column3)
>>> ) WITH COMPACT STORAGE
>>>
>>> and I'd like to change it to:
>>> CREATE TABLE events (
>>> key text,
>>> column1 int,
>>> column2 int,
>>> column3 text,
>>> column4 text,
>>> value text,
>>> PRIMARY KEY(key, column1, column2, column3, column4)
>>> ) WITH COMPACT STORAGE
>>>
>>> Is this possible?
>>> Best regards,
>>> Joel
>>>
>>> 2018-01-12 16:53 GMT+01:00 Joel Samuelsson <samuelsson.j...@gmail.com>:
>>>
>>>> Hi,
>>>>
>>>> I have an older system (C* 2.1) using Thrift tables on which I want to
>>>> alter a column composite. Right now it looks like (int, int, string) but I
>>>> want it to be (int, int, string, string). Is it possible to do this on a
>>>> live cluster without deleting the old data? Can you point me to some
>>>> documentation about this? I can't seem to find it any more.
>>>>
>>>> Best regards,
>>>> Joel
>>>>
>>>
>>>
>>
>

Re: Alter composite column

[Nicolas Guyomar]

Hi Joel,

You cannot alter a table primary key.

You can however alter your existing table to only add column4 using cqlsh
and cql, even if this table as created back with C* 1.X for instance

On 18 January 2018 at 11:14, Joel Samuelsson 
wrote:

> So to rephrase that in CQL terms I have a table like this:
>
> CREATE TABLE events (
> key text,
> column1 int,
> column2 int,
> column3 text,
> value text,
> PRIMARY KEY(key, column1, column2, column3)
> ) WITH COMPACT STORAGE
>
> and I'd like to change it to:
> CREATE TABLE events (
> key text,
> column1 int,
> column2 int,
> column3 text,
> column4 text,
> value text,
> PRIMARY KEY(key, column1, column2, column3, column4)
> ) WITH COMPACT STORAGE
>
> Is this possible?
> Best regards,
> Joel
>
> 2018-01-12 16:53 GMT+01:00 Joel Samuelsson :
>
>> Hi,
>>
>> I have an older system (C* 2.1) using Thrift tables on which I want to
>> alter a column composite. Right now it looks like (int, int, string) but I
>> want it to be (int, int, string, string). Is it possible to do this on a
>> live cluster without deleting the old data? Can you point me to some
>> documentation about this? I can't seem to find it any more.
>>
>> Best regards,
>> Joel
>>
>
>

Re: Cassandra 3.11 fails to start with JDK8u162

[Nicolas Guyomar]

Thank you Thomas for starting this thread, I'm having exactly the same
issue on AWS EC2 RHEL-7.4_HVM-20180103-x86_64-2-Hourly2-GP2 (ami-dc13a4a1)
I was starting to bang my head on my desk !

So I'll try to downgrade back to 152 then !



On 18 January 2018 at 08:34, Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com> wrote:

> Hello,
>
>
>
> after switching from JDK8u152 to JDK8u162, Cassandra fails with the
> following stack trace upon startup.
>
>
>
> ERROR [main] 2018-01-18 07:33:18,804 CassandraDaemon.java:706 - Exception
> encountered during startup
>
> java.lang.AbstractMethodError: org.apache.cassandra.utils.
> JMXServerUtils$Exporter.exportObject(Ljava/rmi/Remote;ILjava/rmi/server/
> RMIClientSocketFactory;Ljava/rmi/server/RMIServerSocketFactory;Lsun/
> misc/ObjectInputFilter;)Ljava/rmi/Remote;
>
> at 
> javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:150)
> ~[na:1.8.0_162]
>
> at 
> javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:135)
> ~[na:1.8.0_162]
>
> at 
> javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:405)
> ~[na:1.8.0_162]
>
> at 
> org.apache.cassandra.utils.JMXServerUtils.createJMXServer(JMXServerUtils.java:104)
> ~[apache-cassandra-3.11.2-SNAPSHOT.jar:3.11.2-SNAPSHOT]
>
> at 
> org.apache.cassandra.service.CassandraDaemon.maybeInitJmx(CassandraDaemon.java:143)
> [apache-cassandra-3.11.2-SNAPSHOT.jar:3.11.2-SNAPSHOT]
>
> at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:188)
> [apache-cassandra-3.11.2-SNAPSHOT.jar:3.11.2-SNAPSHOT]
>
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:600)
> [apache-cassandra-3.11.2-SNAPSHOT.jar:3.11.2-SNAPSHOT]
>
> at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:689)
> [apache-cassandra-3.11.2-SNAPSHOT.jar:3.11.2-SNAPSHOT]
>
>
>
> Is this a known issue?
>
>
>
>
>
> Thanks,
>
> Thomas
>
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freist
> 
> ädterstra
> 
> ße 313
> 
>

Re: Cleanup blocking snapshots - Options?

[Nicolas Guyomar]

Hi,

It might really be a long shot, but I thought UserDefinedCompaction
triggered by JMX on a single sstable might remove data the node does not
own  (to answer your " *Any other way to re-write SSTables with data a node
owns after a cluster scale out" *question part)

I might be wrong though

On 15 January 2018 at 08:43, Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com> wrote:

> Hi Kurt,
>
>
>
> it was easily triggered with the mentioned combination (cleanup after
> extending the cluster) a few months ago, thus I guess it will be the same
> when I re-try. Due to the issue we simply omitted running cleanup then, but
> as disk space is becoming some sort of bottle-neck again, we need to
> re-evaluate this situation J
>
>
>
> Regards,
>
> Thomas
>
>
>
> *From:* kurt greaves [mailto:k...@instaclustr.com]
> *Sent:* Montag, 15. Jänner 2018 06:18
> *To:* User 
> *Subject:* Re: Cleanup blocking snapshots - Options?
>
>
>
> Disabling the snapshots is the best and only real option other than
> upgrading at the moment. Although apparently it was thought that there was
> only a small race condition in 2.1 that triggered this and it wasn't worth
> fixing. If you are triggering it easily maybe it is worth fixing in 2.1 as
> well. Does this happen consistently? Can you provide some more logs on the
> JIRA or better yet a way to reproduce?
>
>
>
> On 14 January 2018 at 16:12, Steinmaurer, Thomas <
> thomas.steinmau...@dynatrace.com> wrote:
>
> Hello,
>
>
>
> we are running 2.1.18 with vnodes in production and due to (
> https://issues.apache.org/jira/browse/CASSANDRA-11155) we can’t run
> cleanup e.g. after extending the cluster without blocking our hourly
> snapshots.
>
>
>
> What options do we have to get rid of partitions a node does not own
> anymore?
>
> · Using a version which has this issue fixed, although upgrading
> to 2.2+, due to various issues, is not an option at the moment
>
> · Temporarily disabling the hourly cron job before starting
> cleanup and re-enable after cleanup has finished
>
> · Any other way to re-write SSTables with data a node owns after
> a cluster scale out
>
>
>
> Thanks,
>
> Thomas
>
>
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freist
> 
> ädterstra
> 
> ße 313
> 
>
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
> 
>

Re: C* Logs to Kibana

[Nicolas Guyomar]

Hi,

I believe you can use Logstash to parse C* logs, using some grok pattern
like those :
https://gist.github.com/ibspoof/917a888adb08a819eab7163b97e018cb so that
you gain some nice insight of what your cluster is really doing !

It feel more "native" than to add some jar in C* lib in order to change
logging behavior, and it will be easier for you to post some log on this ML
if you keep the original format :)

On 10 January 2018 at 11:07, shalom sagges  wrote:

> Hi All,
>
> I want to push the Cassandra logs (version 3.x) to Kibana.
> Is there a way to configure the Cassandra logs to be in json format?
>
> If modifying the logs to json is not an option, I came across this blog
> post from about a year ago regarding that matter:
> https://medium.com/@alain.rastoul/pushing-cassandra-
> logs-into-elasticsearch-9be3b52af754
>
> Is that a good way of accomplishing that?
>
> Thanks!
>

Re: Error during select query - Found other issues with cluster too

[Nicolas Guyomar]

Hi,

Running manual compaction is usually not the right thing to do as you will
end with some huge sstables that won't be compacted for a while.
You should first try to find out why compactions were not happening on your
cluster, because 14k sstables (I assume you are talking about this
particular table, using STCS ? ) is a lot !

As Adama said, you really need to check ulimit for cassandra process
because it might be a reason why compaction are in error, but all those
errors should be logged in system.log

Glad you found the bad script, you already have a bucketed model
month+year, so it's up to you to decide if this is enough fine grained for
your use case, or if you need to refine it further using day/month/year
maybe so that you end up whith more partitions, but smaller and less
harmful for you JVM

It might be helpful for other reader of this ML that you describe your
cluster (C* version, number of nodes, memory per node stuff like that)

On 20 December 2017 at 12:19, Dipan Shah <dipan@hotmail.com> wrote:

> Hello Nicolas,
>
>
> Here's our data model:
>
>
>
>-
>
>CREATE TABLE hhahistory.history (
>-
>
>   tablename text,
>   -
>
>   columnname text,
>   -
>
>   tablekey bigint,
>   -
>
>   updateddate timestamp,
>   -
>
>   dateyearpart bigint,
>   -
>
>   historyid bigint,
>   -
>
>   appname text,
>   -
>
>   audittype text,
>   -
>
>   createddate timestamp,
>   -
>
>   dbsession uuid,
>   -
>
>   firstname text,
>   -
>
>   historybatch uuid,
>   -
>
>   historycassandraid uuid,
>   -
>
>   hostname text,
>   -
>
>   isvlm boolean,
>   -
>
>   lastname text,
>   -
>
>   loginname text,
>   -
>
>   newvalue text,
>   -
>
>   notes text,
>   -
>
>   oldvalue text,
>   -
>
>   reason text,
>   -
>
>   updatedby text,
>   -
>
>   updatedutcdate timestamp,
>   -
>
>   dbname text,
>   -
>
>   PRIMARY KEY (( tablename, columnname,dateyearpart ), tablekey,
>   updateddate, historyid));
>
>
> We are using this to store audit data of our primary SQL Server DB. Our
> primary key consists of the original table name, column name and the
> month+year combination.
>
>
> I just realized that a script had managed to sneak in more than 100
> million rows on the same day so that might me the reason for all this data
> going into the same partition. I'll see if I can do something about this.
>
>
> Thanks,
>
> Dipan Shah
>
>
> --
> *From:* Nicolas Guyomar <nicolas.guyo...@gmail.com>
> *Sent:* Wednesday, December 20, 2017 2:48 PM
> *To:* user@cassandra.apache.org
>
> *Subject:* Re: Error during select query - Found other issues with
> cluster too
>
> Hi Dipan,
>
> This seems like a really unbalanced modelisation, you have some very wide
> rows !
>
> Can you share your model and explain a bit what you are storing in this
> table ? Your partition key might not be appropriate
>
> On 20 December 2017 at 09:43, Dipan Shah <dipan@hotmail.com> wrote:
>
> Hello Kurt,
>
>
> I think I might have found the problem:
>
>
> Can you please look at the tablehistogram for a table and see if that
> seems to be the problem? I think the Max Partition Size and Cell Count are
> too high:
>
>
> *Percentile* *SSTables* *Write Latency (micros)* *Read Latency (micros)* 
> *Partition
> Size (bytes)* *Cell Count*
> 50.00% 0.00 0.00 0.00 29521 2299
> 75.00% 0.00 0.00 0.00 379022 29521
> 95.00% 0.00 0.00 0.00 5839588 454826
> 98.00% 0.00 0.00 0.00 30130992 2346799
> 99.00% 0.00 0.00 0.00 89970660 7007506
> Min 0.00 0.00 0.00 150 0
> Max 0.00 0.00 0.00 53142810146 1996099046
>
>
> Thanks,
>
> Dipan Shah
>
>
> --
> *From:* Dipan Shah <dipan@hotmail.com>
> *Sent:* Wednesday, December 20, 2017 12:04 PM
> *To:* User
> *Subject:* Re: Error during select query - Found other issues with
> cluster too
>
>
> Hello Kurt,
>
>
> We are using V 3.11.0 and I think this might a part of a bigger problem.
> I can see that nodes are failing in my cluster unexpectedly and also
> repair commands are failing.
>
>
> Repair command failure error:
>
>
> INFO  [Native-Transport-Requests-2] 2017-12-19 17:06:02,332
> Message.java:619 - Unexpected exception during request; channel = [id:
> 0xacc9a54a, L:/10.10.52.17:9042 ! R:/10.10.55.229:58712]
> io.netty.channel.

Re: Error during select query - Found other issues with cluster too

[Nicolas Guyomar]

Hi Dipan,

This seems like a really unbalanced modelisation, you have some very wide
rows !

Can you share your model and explain a bit what you are storing in this
table ? Your partition key might not be appropriate

On 20 December 2017 at 09:43, Dipan Shah  wrote:

> Hello Kurt,
>
>
> I think I might have found the problem:
>
>
> Can you please look at the tablehistogram for a table and see if that
> seems to be the problem? I think the Max Partition Size and Cell Count are
> too high:
>
>
> *Percentile* *SSTables* *Write Latency (micros)* *Read Latency (micros)* 
> *Partition
> Size (bytes)* *Cell Count*
> 50.00% 0.00 0.00 0.00 29521 2299
> 75.00% 0.00 0.00 0.00 379022 29521
> 95.00% 0.00 0.00 0.00 5839588 454826
> 98.00% 0.00 0.00 0.00 30130992 2346799
> 99.00% 0.00 0.00 0.00 89970660 7007506
> Min 0.00 0.00 0.00 150 0
> Max 0.00 0.00 0.00 53142810146 1996099046
>
>
> Thanks,
>
> Dipan Shah
>
>
> --
> *From:* Dipan Shah 
> *Sent:* Wednesday, December 20, 2017 12:04 PM
> *To:* User
> *Subject:* Re: Error during select query - Found other issues with
> cluster too
>
>
> Hello Kurt,
>
>
> We are using V 3.11.0 and I think this might a part of a bigger problem.
> I can see that nodes are failing in my cluster unexpectedly and also
> repair commands are failing.
>
>
> Repair command failure error:
>
>
> INFO  [Native-Transport-Requests-2] 2017-12-19 17:06:02,332
> Message.java:619 - Unexpected exception during request; channel = [id:
> 0xacc9a54a, L:/10.10.52.17:9042 ! R:/10.10.55.229:58712]
> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
> failed: Connection reset by peer
> at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source)
> ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> INFO  [Native-Transport-Requests-2] 2017-12-19 17:06:11,056
> Message.java:619 - Unexpected exception during request; channel = [id:
> 0xeebf628d, L:/10.10.52.17:9042 ! R:/10.10.55.229:58130]
> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
> failed: Connection reset by peer
>
> Node failure error:
>
>
> ERROR [STREAM-IN-/10.10.52.22:7000] 2017-12-20 01:17:17,691
> JVMStabilityInspector.java:142 - JVM state determined to be unstable.
> Exiting forcefully due to:
> java.io.FileNotFoundException: /home/install/cassandra-3.11.
> 0/data/data/hhahistory/history-065e0c90d9be11e7afbcdfeb48785ac5/mc-19095-big-Filter.db
> (Too many open files)
> at java.io.FileOutputStream.open0(Native Method) ~[na:1.8.0_131]
> at java.io.FileOutputStream.open(FileOutputStream.java:270)
> ~[na:1.8.0_131]
> at java.io.FileOutputStream.(FileOutputStream.java:213)
> ~[na:1.8.0_131]
> at java.io.FileOutputStream.(FileOutputStream.java:101)
> ~[na:1.8.0_131]
> at org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.
> flushBf(BigTableWriter.java:486) ~[apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.
> doPrepare(BigTableWriter.java:516) ~[apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.utils.concurrent.Transactional$
> AbstractTransactional.prepareToCommit(Transactional.java:173)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.io.sstable.format.big.BigTableWriter$
> TransactionalProxy.doPrepare(BigTableWriter.java:364)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.utils.concurrent.Transactional$
> AbstractTransactional.prepareToCommit(Transactional.java:173)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.utils.concurrent.Transactional$
> AbstractTransactional.finish(Transactional.java:184)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at 
> org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:264)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.finish(
> SimpleSSTableMultiWriter.java:59) ~[apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.io.sstable.format.RangeAwareSSTableWriter.finish(
> RangeAwareSSTableWriter.java:129) ~[apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.streaming.StreamReceiveTask.
> received(StreamReceiveTask.java:110) ~[apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.streaming.StreamSession.
> receive(StreamSession.java:656) ~[apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.streaming.StreamSession.
> messageReceived(StreamSession.java:523) ~[apache-cassandra-3.11.0.jar:
> 3.11.0]
> at org.apache.cassandra.streaming.ConnectionHandler$
> IncomingMessageHandler.run(ConnectionHandler.java:317)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
>
>
>
> Thanks,
>
> Dipan Shah
>
>
> --
> *From:* kurt greaves 
> *Sent:* Wednesday, December 20, 2017 2:23 AM
> *To:* User
> *Subject:* Re: Error during select query
>
> Can you send through the full stack 

Re: Data Node Density

[Nicolas Guyomar]

Hi Amit,

This is way too much data per node, official recommendation are to try to
stay below 2Tb per node, I have seen nodes up to 4Tb but then maintenance
gets really complicated (backup, boostrap, streaming for repair etc etc)

Nicolas

On 15 December 2017 at 15:01, Amit Agrawal 
wrote:

> Hi,
>
> We are trying to setup a 3 node cluster with 20 TB HD on each node.
> its a bare metal setup with 44 cores on each node.
>
> So in total 60 TB, 66 cores , 3 node cluster.
>
> The data velocity is very less, low access rates.
>
> has anyone tried with this configuration ?
>
> A bit urgent.
>
> Regards,
> -A
>
>
>

Re: Batch : Isolation and Atomicity for same partition on multiple table

[Nicolas Guyomar]

Hi Mickael,

Partition are related to the table they exist in, so in your case, you are
targeting 2 partitions in 2 different tables.
Therefore, IMHO, you will only get atomicity using your batch statement

On 11 December 2017 at 15:59, Mickael Delanoë  wrote:

> Hello,
>
> I have a question regarding batch isolation and atomicity with query using
> a same partition key.
>
> The Datastax documentation says about the batches :
> "Combines multiple DML statements to achieve atomicity and isolation when
> targeting a single partition or only atomicity when targeting multiple
> partitions. A batch applies all DMLs within a single partition before the
> data is available, ensuring atomicity and isolation.""
>
> But I try to find exactly what can be considered as a "single partition"
> and I cannot find a clear response yet. The examples and explanations
> always speak about partition with only one table used inside the batch. My
> concern is about partition when we use different table in a batch. So I
> would like some clarification.
>
> Here is my use case, I have 2 tables with the same partition-key which is
> "user_id" :
>
> CREATE TABLE tableA (
>user_id text,
>clustering text,
>value text,
>PRIMARY KEY (user_id, clustering));
>
> CREATE TABLE tableB (
>user_id text,
>clustering1 text,
>clustering2 text,
>value text,
>PRIMARY KEY (user_id, clustering1, clustering2));
>
> If I do a batch query like this :
>
> BEGIN BATCH
> INSERT INTO tableA (user_id, clustering, value) VALUES ('1234', 'c1',
> 'val1');
> INSERT INTO tableB (user_id, clustering1, clustering1, value) VALUES
> ('1234', 'cl1', 'cl2', 'avalue');
> APPLY BATCH;
>
> the DML statements uses the same partition-key, can we say they are
> targetting the same partition or, as the partition key are for different
> table, should we consider this is different partition? And so does this
> batch ensure atomicity and isolation (in the sense described in Datastax
> doc)? Or only atomicity?
>
> Thanks for you help,
> Mickaël Delanoë
>

Re: STCS leaving sstables behind

[Nicolas Guyomar]

Quick follow up : triggering a repair did the trick, sstables are starting
to get compacted.

Thank you

On 13 November 2017 at 15:53, Nicolas Guyomar <nicolas.guyo...@gmail.com>
wrote:

> Hi,
>
> I'm using default "nodetool repair" in 3.0.13 which I believe is full by
> default. I'm not using subrange repair
>
> Jeff you're right, "Nov 11 01:23 mc-6474-big-Data.db" is not yet marked as
> repaired, my repair routine is broken  (sorry Alexander I'm not using
> repear yet ;)  )
>
> I'm gonna fix my repair script and see if it unlock this situation
>
> Thank you
>
>
> On 13 November 2017 at 15:41, Alexander Dejanovski <a...@thelastpickle.com
> > wrote:
>
>> And actually, full repair with 3.0/3.x would have the same effect
>> (anticompation) unless you're using subrange repair.
>>
>> On Mon, Nov 13, 2017 at 3:28 PM Jeff Jirsa <jji...@gmail.com> wrote:
>>
>>> Running incremental repair puts sstables into a “repaired” set (and an
>>> unrepaired set), which results in something similar to what you’re
>>> describing.
>>>
>>> Were you running / did you run incremental repair ?
>>>
>>>
>>> --
>>> Jeff Jirsa
>>>
>>>
>>> On Nov 13, 2017, at 5:04 AM, Nicolas Guyomar <nicolas.guyo...@gmail.com>
>>> wrote:
>>>
>>> Hi everyone,
>>>
>>> I'm facing quite a strange behavior on STCS on 3.0.13, the strategy
>>> seems to have "forgotten" about old sstables, and started a completely new
>>> cycle from scratch, leaving the old sstables on disk untouched :
>>>
>>> Something happened on Nov 10 on every node, which resulted in all those
>>> sstables left behind :
>>>
>>> -rw-r--r--.  8 cassandra cassandra   15G Nov  9 22:22 mc-4828-big-Data.db
>>> -rw-r--r--.  8 cassandra cassandra  4.8G Nov 10 01:39 mc-4955-big-Data.db
>>> -rw-r--r--.  8 cassandra cassandra  2.4G Nov 10 01:45 mc-4957-big-Data.db
>>> -rw-r--r--.  8 cassandra cassandra  662M Nov 10 01:47 mc-4959-big-Data.db
>>> -rw-r--r--.  8 cassandra cassandra  2.8G Nov 10 03:46 mc-5099-big-Data.db
>>> -rw-r--r--.  8 cassandra cassandra  4.6G Nov 10 03:58 mc-5121-big-Data.db
>>> -rw-r--r--.  7 cassandra cassandra   53M Nov 10 08:45 mc-5447-big-Data.db
>>> -rw-r--r--.  7 cassandra cassandra  219M Nov 10 08:46 mc-5454-big-Data.db
>>> -rw-r--r--.  7 cassandra cassandra  650M Nov 10 08:46 mc-5452-big-Data.db
>>> -rw-r--r--.  7 cassandra cassandra  1.2G Nov 10 08:48 mc-5458-big-Data.db
>>> -rw-r--r--.  7 cassandra cassandra  1.5G Nov 10 08:50 mc-5465-big-Data.db
>>> -rw-r--r--.  7 cassandra cassandra  504M Nov 10 09:39 mc-5526-big-Data.db
>>> -rw-r--r--.  7 cassandra cassandra   57M Nov 10 09:40 mc-5527-big-Data.db
>>> -rw-r--r--.  7 cassandra cassandra  101M Nov 10 09:41 mc-5532-big-Data.db
>>> -rw-r--r--.  7 cassandra cassandra   86M Nov 10 09:41 mc-5533-big-Data.db
>>> -rw-r--r--.  7 cassandra cassandra  134M Nov 10 09:42 mc-5537-big-Data.db
>>> -rw-r--r--.  7 cassandra cassandra  3.9G Nov 10 09:54 mc-5538-big-Data.db
>>> *-rw-r--r--.  7 cassandra cassandra  1.3G Nov 10 09:57
>>> mc-5548-big-Data.db*
>>> -rw-r--r--.  6 cassandra cassandra   16G Nov 11 01:23 mc-6474-big-Data.db
>>> -rw-r--r--.  4 cassandra cassandra   17G Nov 12 06:44 mc-7898-big-Data.db
>>> -rw-r--r--.  3 cassandra cassandra  8.2G Nov 12 13:45 mc-8226-big-Data.db
>>> -rw-r--r--.  2 cassandra cassandra  6.8G Nov 12 22:38 mc-8581-big-Data.db
>>> -rw-r--r--.  2 cassandra cassandra  6.1G Nov 13 03:10 mc-8937-big-Data.db
>>> -rw-r--r--.  2 cassandra cassandra  3.1G Nov 13 04:12 mc-9019-big-Data.db
>>> -rw-r--r--.  2 cassandra cassandra  3.0G Nov 13 05:56 mc-9112-big-Data.db
>>> -rw-r--r--.  2 cassandra cassandra  1.2G Nov 13 06:14 mc-9138-big-Data.db
>>> -rw-r--r--.  2 cassandra cassandra  1.1G Nov 13 06:27 mc-9159-big-Data.db
>>> -rw-r--r--.  2 cassandra cassandra  1.2G Nov 13 06:46 mc-9182-big-Data.db
>>> -rw-r--r--.  1 cassandra cassandra  1.9G Nov 13 07:18 mc-9202-big-Data.db
>>> -rw-r--r--.  1 cassandra cassandra  353M Nov 13 07:22 mc-9207-big-Data.db
>>> -rw-r--r--.  1 cassandra cassandra  120M Nov 13 07:22 mc-9208-big-Data.db
>>> -rw-r--r--.  1 cassandra cassandra  100M Nov 13 07:23 mc-9209-big-Data.db
>>> -rw-r--r--.  1 cassandra cassandra   67M Nov 13 07:25 mc-9210-big-Data.db
>>> -rw-r--r--.  1 cassandra cassandra   51M Nov 13 07:25 mc-9211-big-Data.db
>>> -rw-r--r--.  1 cassandra cassandra   73M Nov 13 07:27 mc-9212-big-Data.db
>>>
>>>
&g

Re: STCS leaving sstables behind

[Nicolas Guyomar]

Hi,

I'm using default "nodetool repair" in 3.0.13 which I believe is full by
default. I'm not using subrange repair

Jeff you're right, "Nov 11 01:23 mc-6474-big-Data.db" is not yet marked as
repaired, my repair routine is broken  (sorry Alexander I'm not using
repear yet ;)  )

I'm gonna fix my repair script and see if it unlock this situation

Thank you


On 13 November 2017 at 15:41, Alexander Dejanovski <a...@thelastpickle.com>
wrote:

> And actually, full repair with 3.0/3.x would have the same effect
> (anticompation) unless you're using subrange repair.
>
> On Mon, Nov 13, 2017 at 3:28 PM Jeff Jirsa <jji...@gmail.com> wrote:
>
>> Running incremental repair puts sstables into a “repaired” set (and an
>> unrepaired set), which results in something similar to what you’re
>> describing.
>>
>> Were you running / did you run incremental repair ?
>>
>>
>> --
>> Jeff Jirsa
>>
>>
>> On Nov 13, 2017, at 5:04 AM, Nicolas Guyomar <nicolas.guyo...@gmail.com>
>> wrote:
>>
>> Hi everyone,
>>
>> I'm facing quite a strange behavior on STCS on 3.0.13, the strategy seems
>> to have "forgotten" about old sstables, and started a completely new cycle
>> from scratch, leaving the old sstables on disk untouched :
>>
>> Something happened on Nov 10 on every node, which resulted in all those
>> sstables left behind :
>>
>> -rw-r--r--.  8 cassandra cassandra   15G Nov  9 22:22 mc-4828-big-Data.db
>> -rw-r--r--.  8 cassandra cassandra  4.8G Nov 10 01:39 mc-4955-big-Data.db
>> -rw-r--r--.  8 cassandra cassandra  2.4G Nov 10 01:45 mc-4957-big-Data.db
>> -rw-r--r--.  8 cassandra cassandra  662M Nov 10 01:47 mc-4959-big-Data.db
>> -rw-r--r--.  8 cassandra cassandra  2.8G Nov 10 03:46 mc-5099-big-Data.db
>> -rw-r--r--.  8 cassandra cassandra  4.6G Nov 10 03:58 mc-5121-big-Data.db
>> -rw-r--r--.  7 cassandra cassandra   53M Nov 10 08:45 mc-5447-big-Data.db
>> -rw-r--r--.  7 cassandra cassandra  219M Nov 10 08:46 mc-5454-big-Data.db
>> -rw-r--r--.  7 cassandra cassandra  650M Nov 10 08:46 mc-5452-big-Data.db
>> -rw-r--r--.  7 cassandra cassandra  1.2G Nov 10 08:48 mc-5458-big-Data.db
>> -rw-r--r--.  7 cassandra cassandra  1.5G Nov 10 08:50 mc-5465-big-Data.db
>> -rw-r--r--.  7 cassandra cassandra  504M Nov 10 09:39 mc-5526-big-Data.db
>> -rw-r--r--.  7 cassandra cassandra   57M Nov 10 09:40 mc-5527-big-Data.db
>> -rw-r--r--.  7 cassandra cassandra  101M Nov 10 09:41 mc-5532-big-Data.db
>> -rw-r--r--.  7 cassandra cassandra   86M Nov 10 09:41 mc-5533-big-Data.db
>> -rw-r--r--.  7 cassandra cassandra  134M Nov 10 09:42 mc-5537-big-Data.db
>> -rw-r--r--.  7 cassandra cassandra  3.9G Nov 10 09:54 mc-5538-big-Data.db
>> *-rw-r--r--.  7 cassandra cassandra  1.3G Nov 10 09:57
>> mc-5548-big-Data.db*
>> -rw-r--r--.  6 cassandra cassandra   16G Nov 11 01:23 mc-6474-big-Data.db
>> -rw-r--r--.  4 cassandra cassandra   17G Nov 12 06:44 mc-7898-big-Data.db
>> -rw-r--r--.  3 cassandra cassandra  8.2G Nov 12 13:45 mc-8226-big-Data.db
>> -rw-r--r--.  2 cassandra cassandra  6.8G Nov 12 22:38 mc-8581-big-Data.db
>> -rw-r--r--.  2 cassandra cassandra  6.1G Nov 13 03:10 mc-8937-big-Data.db
>> -rw-r--r--.  2 cassandra cassandra  3.1G Nov 13 04:12 mc-9019-big-Data.db
>> -rw-r--r--.  2 cassandra cassandra  3.0G Nov 13 05:56 mc-9112-big-Data.db
>> -rw-r--r--.  2 cassandra cassandra  1.2G Nov 13 06:14 mc-9138-big-Data.db
>> -rw-r--r--.  2 cassandra cassandra  1.1G Nov 13 06:27 mc-9159-big-Data.db
>> -rw-r--r--.  2 cassandra cassandra  1.2G Nov 13 06:46 mc-9182-big-Data.db
>> -rw-r--r--.  1 cassandra cassandra  1.9G Nov 13 07:18 mc-9202-big-Data.db
>> -rw-r--r--.  1 cassandra cassandra  353M Nov 13 07:22 mc-9207-big-Data.db
>> -rw-r--r--.  1 cassandra cassandra  120M Nov 13 07:22 mc-9208-big-Data.db
>> -rw-r--r--.  1 cassandra cassandra  100M Nov 13 07:23 mc-9209-big-Data.db
>> -rw-r--r--.  1 cassandra cassandra   67M Nov 13 07:25 mc-9210-big-Data.db
>> -rw-r--r--.  1 cassandra cassandra   51M Nov 13 07:25 mc-9211-big-Data.db
>> -rw-r--r--.  1 cassandra cassandra   73M Nov 13 07:27 mc-9212-big-Data.db
>>
>>
>> TRACE logs for the Compaction Manager shows that sstables before Nov 10
>> are grouped in different buckets than the one after Nov 10.
>>
>> At first I thought off some coldness behavior that would filter those
>> "old" sstables, but looking at the code https://github.com/apache/
>> cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/compaction/
>> SizeTieredCompactionStrategy.java#L237 I don't see any coldness or time
>> pattern used to create bucket.
>>

STCS leaving sstables behind

[Nicolas Guyomar]

Hi everyone,

I'm facing quite a strange behavior on STCS on 3.0.13, the strategy seems
to have "forgotten" about old sstables, and started a completely new cycle
from scratch, leaving the old sstables on disk untouched :

Something happened on Nov 10 on every node, which resulted in all those
sstables left behind :

-rw-r--r--.  8 cassandra cassandra   15G Nov  9 22:22 mc-4828-big-Data.db
-rw-r--r--.  8 cassandra cassandra  4.8G Nov 10 01:39 mc-4955-big-Data.db
-rw-r--r--.  8 cassandra cassandra  2.4G Nov 10 01:45 mc-4957-big-Data.db
-rw-r--r--.  8 cassandra cassandra  662M Nov 10 01:47 mc-4959-big-Data.db
-rw-r--r--.  8 cassandra cassandra  2.8G Nov 10 03:46 mc-5099-big-Data.db
-rw-r--r--.  8 cassandra cassandra  4.6G Nov 10 03:58 mc-5121-big-Data.db
-rw-r--r--.  7 cassandra cassandra   53M Nov 10 08:45 mc-5447-big-Data.db
-rw-r--r--.  7 cassandra cassandra  219M Nov 10 08:46 mc-5454-big-Data.db
-rw-r--r--.  7 cassandra cassandra  650M Nov 10 08:46 mc-5452-big-Data.db
-rw-r--r--.  7 cassandra cassandra  1.2G Nov 10 08:48 mc-5458-big-Data.db
-rw-r--r--.  7 cassandra cassandra  1.5G Nov 10 08:50 mc-5465-big-Data.db
-rw-r--r--.  7 cassandra cassandra  504M Nov 10 09:39 mc-5526-big-Data.db
-rw-r--r--.  7 cassandra cassandra   57M Nov 10 09:40 mc-5527-big-Data.db
-rw-r--r--.  7 cassandra cassandra  101M Nov 10 09:41 mc-5532-big-Data.db
-rw-r--r--.  7 cassandra cassandra   86M Nov 10 09:41 mc-5533-big-Data.db
-rw-r--r--.  7 cassandra cassandra  134M Nov 10 09:42 mc-5537-big-Data.db
-rw-r--r--.  7 cassandra cassandra  3.9G Nov 10 09:54 mc-5538-big-Data.db
*-rw-r--r--.  7 cassandra cassandra  1.3G Nov 10 09:57 mc-5548-big-Data.db*
-rw-r--r--.  6 cassandra cassandra   16G Nov 11 01:23 mc-6474-big-Data.db
-rw-r--r--.  4 cassandra cassandra   17G Nov 12 06:44 mc-7898-big-Data.db
-rw-r--r--.  3 cassandra cassandra  8.2G Nov 12 13:45 mc-8226-big-Data.db
-rw-r--r--.  2 cassandra cassandra  6.8G Nov 12 22:38 mc-8581-big-Data.db
-rw-r--r--.  2 cassandra cassandra  6.1G Nov 13 03:10 mc-8937-big-Data.db
-rw-r--r--.  2 cassandra cassandra  3.1G Nov 13 04:12 mc-9019-big-Data.db
-rw-r--r--.  2 cassandra cassandra  3.0G Nov 13 05:56 mc-9112-big-Data.db
-rw-r--r--.  2 cassandra cassandra  1.2G Nov 13 06:14 mc-9138-big-Data.db
-rw-r--r--.  2 cassandra cassandra  1.1G Nov 13 06:27 mc-9159-big-Data.db
-rw-r--r--.  2 cassandra cassandra  1.2G Nov 13 06:46 mc-9182-big-Data.db
-rw-r--r--.  1 cassandra cassandra  1.9G Nov 13 07:18 mc-9202-big-Data.db
-rw-r--r--.  1 cassandra cassandra  353M Nov 13 07:22 mc-9207-big-Data.db
-rw-r--r--.  1 cassandra cassandra  120M Nov 13 07:22 mc-9208-big-Data.db
-rw-r--r--.  1 cassandra cassandra  100M Nov 13 07:23 mc-9209-big-Data.db
-rw-r--r--.  1 cassandra cassandra   67M Nov 13 07:25 mc-9210-big-Data.db
-rw-r--r--.  1 cassandra cassandra   51M Nov 13 07:25 mc-9211-big-Data.db
-rw-r--r--.  1 cassandra cassandra   73M Nov 13 07:27 mc-9212-big-Data.db


TRACE logs for the Compaction Manager shows that sstables before Nov 10 are
grouped in different buckets than the one after Nov 10.

At first I thought off some coldness behavior that would filter those "old"
sstables, but looking at the code
https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java#L237
I don't see any coldness or time pattern used to create bucket.

I tried restarting the node but the buckets are still grouping in 2
"groups" splitted around Nov 10

I may have missed sthg from the logs, but they are clear from error/warn at
that Nov 10 time

For what it's worth, restarting the node fixed nodetool status from
reporting a wrong Load (nearly 2TB per node instead à 300Gb) => we are
loading some data for a week now, it seems that this can happen sometimes

If anyone ever experienced that kind of behavior I'd be glad to know
whether it is OK or not, I'd like to avoid manually triggering JMX
UserDefinedCompaction ;)

Thank you

Re: poolingOptions not serializable?

[Nicolas Guyomar]

Hi Andrea,

Do you have the error using the builder ?

PoolingOptions poolingOptions = new PoolingOptions();
poolingOptions
.setMaxRequestsPerConnection(HostDistance.LOCAL, 32768)
.setMaxRequestsPerConnection(HostDistance.REMOTE, 1);


Builder builder = Cluster.builder();
builder.addContactPoint(CASSANDRA_ADDRESS);
builder.withPort(CASSANDRA_PORT);
builder.withPoolingOptions(poolingOptions);


sinkBuilderNormalStream
.setQuery("INSERT INTO keyspace_local.values_by_sensors_users"
+ " (user, sensor, timestamp, rdf_stream, observed_value, value)"
+ " VALUES (?, ?, ?, ?, ?, ?);")
.setClusterBuilder(builder)
.build();


On 4 November 2017 at 19:27, Andrea Giordano 
wrote:

> Hi,
> I’m using datastax driver to use Cassandra as sink for some data streams
> with Apache Flink:
> I have a problem executing my application raising an error about the full
> queue. I discovered that the default value is 256, probably too low for my
> load, so I have raised it using poolingOptions setting
> maxRequestsPerConnection as suggested here: http://docs.datastax.
> com/en/developer/java-driver/3.1/manual/pooling/.
>
> Unfortunately with the following code I obtain the following error when I
> launch it:
>
> The implementation of the ClusterBuilder is not serializable.
> The object probably contains or references non serializable fields.
>
>
> My code:
>
>
> PoolingOptions poolingOptions = new PoolingOptions();
> poolingOptions
>   .setMaxRequestsPerConnection(HostDistance.LOCAL, 32768)
>   .setMaxRequestsPerConnection(HostDistance.REMOTE, 1);
>
>
> ClusterBuilder cassandraBuilder = new ClusterBuilder() {
> private static final long serialVersionUID = 1L;
>
> @Override
> public Cluster buildCluster(Cluster.Builder builder) {
> return builder.addContactPoint(CASSANDRA_ADDRESS).withPort(CASSANDRA_PORT
> )..withPoolingOptions(poolingOptions).build();
> }
> };
>
>
> sinkBuilderNormalStream
> .setQuery("INSERT INTO keyspace_local.values_by_sensors_users"
> + " (user, sensor, timestamp, rdf_stream, observed_value, value)"
> + " VALUES (?, ?, ?, ?, ?, ?);")
> .setClusterBuilder(cassandraBuilder)
> .build();
>
>
> How can I deal with it?
>

Re: What is OneMinuteRate in Write Latency?

[Nicolas Guyomar]

Hi,

OneMinuteRate is the mean rate of write/s over a minute bucket of data
AFAIK.

You can find Latencies on every attributes whose name does not end with
"Rate"



On 2 November 2017 at 18:10, AI Rumman  wrote:

> Hi,
>
> I am trying to calculate the Read/second and Write/Second in my Cassandra
> 2.1 cluster. After searching and reading, I came to know about JMX bean
> "org.apache.cassandra.metrics:type=ClientRequest,scope=
> Write,name=Latency".
> Here I can see oneMinuteRate. I have started a brand new cluster and
> started collected these metrics from 0.
> When I started my first record, I can see
>
> Count = 1
>> OneMinuteRate = 0.01599111...
>
>
> Does it mean that my write/s is 0.0159911? Or does it mean that based on 1
> minute data, my write latency is 0.01599 where Write Latency refers to the
> response time for writing a record?
>
> Please help me understand the value.
>
> Thanks.
>
>
>
>
>

Re: can repair and bootstrap run simultaneously

[Nicolas Guyomar]

Hi,

I believe that as long as you use -local option in another DC this would be
safe, but repairing a DC while boostraping a new node in it does not seems
OK AFAIK.

On 24 October 2017 at 14:08, Peng Xiao <2535...@qq.com> wrote:

> Hi there,
>
> Can we add a new node (bootstrap) and run repair on another DC in the
> cluster or even run repair in the same DC?
>
> Thanks,
> Peng Xiao
>

Re: Not serving read requests while running nodetool repair

[Nicolas Guyomar]

Hi Thomas,

AFAIK  temporarily reading at LOCAL_QUORUM/QUORUM until nodetool repair is
finished is the way to go. You can still disable binary/thrift on the node
to "protect" it from acting as a coordinator, and complete its repair
quietly, but I'm not sure that would make such a huge difference in
recovery time.

If you disable gossip the node will be see as down, thus disabling repair
if I'm correct

If repair is taking too much time, or switching to Read Quorum not feasible
within your application, is it OK for you to shutdown the node, wipe its
data and launch a repair on it at an appropriate time windows ?



On 18 October 2017 at 08:04, Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com> wrote:

> Hello,
>
>
>
> due to performance/latency reasons, we are currently reading and writing
> time series data at consistency level  ONE/ANY.
>
>
>
> In case of a node being down and recovering after the default hinted
> handoff window of 3 hrs, we may potentially read stale data from the
> recovering node. Of course, from an operational POV, we will trigger a
> nodetool repair after the recovered node has start up, but to my
> understanding, this still may cause reading stale data from this particular
> node until nodetool repair is finished, which may take several hours. Is
> this correct?
>
>
>
> Is there a way (e.g. in newer releases 3.0/3.11) to tell a node not being
> part of read requests? Something like a counterpart of nodetool drain for
> writes?
>
>
>
> Other options:
>
>
>
> -  Disabling binary/thrift: Although the node won’t act as
> coordinator then for client requests, as it won’t be available from a
> client perspective, inter-node, the recovering node is being contacted by
> other nodes for serving requests, right?
>
> -  Disabling gossip: Basically the node is marked as DOWN, so
> treated like it is not available, thus not an option?
>
> -  Temporarily reading at LOCAL_QUORUM/QUORUM until nodetool
> repair is finished?
>
>
>
> Did I miss something?
>
>
>
> Thanks,
>
> Thomas
>
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freist
> 
> ädterstra
> 
> ße 313
> 
>

Re: Looking for advice and assistance upgrading from Cassandra 1.2.9

[Nicolas Guyomar]

Hi David,

Last time I did such an upgrade, I got stuck in 2.1.x because of OpsCenter
5.2 :
http://docs.datastax.com/en/landing_page/doc/landing_page/compatibility.html#compatibility__opsc-compatibility
.
This might have change since, but I do not think C* 2.2 will work with
OpsCenter 5.2

If you are in the process of updating your cluster toward the v3 (or even
better wait for v4 to be stable) in a near future, I'd recommend getting
rid of OpsCenter to begin with, and build a monitoring stack that would
enable you to keep a close eye on your cluster during this while migration
(or use a saas monitoring like datadog etc )

Furthermore it wasn't necessary for me to migrate from thrift towards CQL,
but that was kind of a mistake because I wasn't getting the full
performance improvement of v2 version thanks to CQL => if you can afford
it, and are using thrift right now => go for CQL ;)

Have fun I hope ;)

On 18 October 2017 at 00:26, kurt greaves  wrote:

> +1 what Jon said
>
> On 18 Oct. 2017 06:38, "Jon Haddad"  wrote:
>
>> I recommend going all the way to 2.2.
>>
>> On Oct 17, 2017, at 12:37 PM, Jeff Jirsa  wrote:
>>
>> You’ll go from 1.2 to 2.0 to 2.1 - should be basic steps:
>> - make sure you have all 1.2 sstables by running upgradesstable
>> - one node at a time, swap the 1.2 binaries for latest in 2.0
>> - once all nodes are 2.0, run upgradesstables on each node (one at a time)
>> - one node at a, swap the 2.0 binaries for latest 2.1
>> - once all are 2.1, upgradesstables again
>>
>> You’ll want to read NEWS.txt for the versions as you upgrade in case
>> there are any changes to config yaml or intermediate steps -
>> https://github.com/apache/cassandra/blob/cassandra-1.2/NEWS.txt for 1.2
>>
>> Not a bad idea to find a consultant that does this often because they may
>> know of some rough edges in the process - this isn’t an endorsement, but I
>> imagine the folks at Pythian do this a lot, and Instaclustr / TheLastPickle
>> both have teams of consultants to help you along as well.
>>
>>
>>
>> On Oct 17, 2017, at 11:20 AM, David Ellis  wrote:
>>
>> I have a large Cassandra 1.2.9 based enterprise application with a
>> considerable amount of data.
>>
>> I’d like to begin the process of upgrading my existing java application
>> and the database to Cassandra 2.1.
>>
>> Anyone have some suggestions where to begin? Any experts looking for some
>> consulting income :)
>>
>> David
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>
>>

Re: Drastic increase in disk usage after starting repair on 3.7

[Nicolas Guyomar]

Hi Paul,

This might be a long shot, but some repairs might fail to clear their
snapshot (not sure if its still the case with C* 3.7 however, I had the
problem on 2.X branche).
What does nodetool listsnapshot indicate ?

On 21 September 2017 at 05:49, kurt greaves  wrote:

> repair does overstream by design, so if that node is inconsistent you'd
> expect a bit of an increase. if you've got a backlog of compactions that's
> probably due to repair and likely the cause of the increase. if you're
> really worried you can rolling restart to stop the repair, otherwise maybe
> try increasing compaction throughput.
>

Re: old big tombstone data file occupy much disk space

[Nicolas Guyomar]

Wrong copy/paste !

Looking at the code, it should do nothing :

 // look up the sstables now that we're on the compaction executor, so we
don't try to re-compact
 // something that was already being compacted earlier.

On 4 September 2017 at 13:54, Nicolas Guyomar <nicolas.guyo...@gmail.com>
wrote:

> You'll get the WARN "Will not compact {}: it is not an active sstable"  :)
>
> On 4 September 2017 at 12:07, Shalom Sagges <shal...@liveperson.com>
> wrote:
>
>> By the way, does anyone know what happens if I run a user defined
>> compaction on an sstable that's already in compaction?
>>
>>
>>
>>
>>
>>
>> On Sun, Sep 3, 2017 at 2:55 PM, Shalom Sagges <shal...@liveperson.com>
>> wrote:
>>
>>> Try this blog by The Last Pickle:
>>>
>>> http://thelastpickle.com/blog/2016/10/18/user-defined-compaction.html
>>>
>>>
>>>
>>>
>>>
>>>
>>> Shalom Sagges
>>> DBA
>>> <http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
>>> <http://www.facebook.com/LivePersonInc> We Create Meaningful Connections
>>>
>>>
>>>
>>> On Sat, Sep 2, 2017 at 8:34 PM, Jeff Jirsa <jji...@gmail.com> wrote:
>>>
>>>> If you're on 3.0 (3.0.6 or 3.0.8 or newer I don't remember which), TWCS
>>>> was designed for ttl-only time series use cases
>>>>
>>>> Alternatively, if you have IO to spare, you may find LCS works as well
>>>> (it'll cause quite a bit more compaction, but a much higher chance to
>>>> compact away tombstones)
>>>>
>>>> There are also tombstone focused sub properties to more aggressively
>>>> compact sstables that have a lot of tombstones - check the docs for
>>>> "unchecked tombstone compaction" and "tombstone threshold" - enabling those
>>>> will enable more aggressive automatic single-sstable compactions
>>>>
>>>> --
>>>> Jeff Jirsa
>>>>
>>>>
>>>> On Sep 2, 2017, at 7:10 AM, qf zhou <zhouqf2...@gmail.com> wrote:
>>>>
>>>>
>>>> Yes, your are right. I am using STCS compaction strategy with some kind
>>>> of timeseries model. Too much disk space has been occupied.
>>>>
>>>> What should I  do to stop  the  disk full ?
>>>>
>>>>  I only want to keep 100 days data most recently,  so I set 
>>>> default_time_to_live
>>>> = 864(100 days ).
>>>>
>>>> I know I need to do something to stop the disk space cost, but I really
>>>> don’t know how to do it.
>>>>
>>>>
>>>> Here is the strategy of the big data table :
>>>>
>>>> AND compaction = {'class': 'org.apache.cassandra.db.compa
>>>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
>>>> 'min_threshold': '12', 'tombstone_threshold': '0.1',
>>>> 'unchecked_tombstone_compaction': 'true'}
>>>> AND compression = {'chunk_length_in_kb': '64', 'class': '
>>>> org.apache.cassandra.io.compress.LZ4Compressor'}
>>>> AND crc_check_chance = 1.0
>>>> AND dclocal_read_repair_chance = 0.1
>>>> AND default_time_to_live = 864
>>>> AND gc_grace_seconds = 432000
>>>>
>>>>
>>>>
>>>> 在 2017年9月2日，下午7:34，Nicolas Guyomar <nicolas.guyo...@gmail.com> 写道：
>>>>
>>>> your are using STCS compaction strategy with some kind of timeseries
>>>> model, and you are going to end up with yor disk full!
>>>>
>>>>
>>>>
>>>
>>
>> This message may contain confidential and/or privileged information.
>> If you are not the addressee or authorized to receive this on behalf of
>> the addressee you must not use, copy, disclose or take action based on this
>> message or any information herein.
>> If you have received this message in error, please advise the sender
>> immediately by reply email and delete this message. Thank you.
>>
>
>

Re: old big tombstone data file occupy much disk space

[Nicolas Guyomar]

You'll get the WARN "Will not compact {}: it is not an active sstable"  :)

On 4 September 2017 at 12:07, Shalom Sagges <shal...@liveperson.com> wrote:

> By the way, does anyone know what happens if I run a user defined
> compaction on an sstable that's already in compaction?
>
>
>
>
>
>
> On Sun, Sep 3, 2017 at 2:55 PM, Shalom Sagges <shal...@liveperson.com>
> wrote:
>
>> Try this blog by The Last Pickle:
>>
>> http://thelastpickle.com/blog/2016/10/18/user-defined-compaction.html
>>
>>
>>
>>
>>
>>
>> Shalom Sagges
>> DBA
>> <http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
>> <http://www.facebook.com/LivePersonInc> We Create Meaningful Connections
>>
>>
>>
>> On Sat, Sep 2, 2017 at 8:34 PM, Jeff Jirsa <jji...@gmail.com> wrote:
>>
>>> If you're on 3.0 (3.0.6 or 3.0.8 or newer I don't remember which), TWCS
>>> was designed for ttl-only time series use cases
>>>
>>> Alternatively, if you have IO to spare, you may find LCS works as well
>>> (it'll cause quite a bit more compaction, but a much higher chance to
>>> compact away tombstones)
>>>
>>> There are also tombstone focused sub properties to more aggressively
>>> compact sstables that have a lot of tombstones - check the docs for
>>> "unchecked tombstone compaction" and "tombstone threshold" - enabling those
>>> will enable more aggressive automatic single-sstable compactions
>>>
>>> --
>>> Jeff Jirsa
>>>
>>>
>>> On Sep 2, 2017, at 7:10 AM, qf zhou <zhouqf2...@gmail.com> wrote:
>>>
>>>
>>> Yes, your are right. I am using STCS compaction strategy with some kind
>>> of timeseries model. Too much disk space has been occupied.
>>>
>>> What should I  do to stop  the  disk full ?
>>>
>>>  I only want to keep 100 days data most recently,  so I set 
>>> default_time_to_live
>>> = 864(100 days ).
>>>
>>> I know I need to do something to stop the disk space cost, but I really
>>> don’t know how to do it.
>>>
>>>
>>> Here is the strategy of the big data table :
>>>
>>> AND compaction = {'class': 'org.apache.cassandra.db.compa
>>> ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
>>> 'min_threshold': '12', 'tombstone_threshold': '0.1',
>>> 'unchecked_tombstone_compaction': 'true'}
>>> AND compression = {'chunk_length_in_kb': '64', 'class': '
>>> org.apache.cassandra.io.compress.LZ4Compressor'}
>>> AND crc_check_chance = 1.0
>>> AND dclocal_read_repair_chance = 0.1
>>> AND default_time_to_live = 864
>>> AND gc_grace_seconds = 432000
>>>
>>>
>>>
>>> 在 2017年9月2日，下午7:34，Nicolas Guyomar <nicolas.guyo...@gmail.com> 写道：
>>>
>>> your are using STCS compaction strategy with some kind of timeseries
>>> model, and you are going to end up with yor disk full!
>>>
>>>
>>>
>>
>
> This message may contain confidential and/or privileged information.
> If you are not the addressee or authorized to receive this on behalf of
> the addressee you must not use, copy, disclose or take action based on this
> message or any information herein.
> If you have received this message in error, please advise the sender
> immediately by reply email and delete this message. Thank you.
>

Re: old big tombstone data file occupy much disk space

[Nicolas Guyomar]

Hi,

The nodetool command only shows what's going on on this particular node.
Validation Compaction means that Cassandra is computing Merkel Tree so that
this node can participate in an ongoing repair.

What kind of disk hardware do you have ? Node with To of data seems a lot
in regards to your first email "some few people know cassandra in my
company." !  Usually people tends to keep their node around 1Tb of data and
start adding new node in an horizontal scaling manner

The running compation 3fc33340-8e4e-11e7-9754-c981af5d39a9 Compaction
 gps  gpsfullwithstate 451.73 GiB 817.51 GiB bytes 55.26%   is really
going to end up creating a 817.51 GiB sstable, which seems huge.

At first sight I'd say that your are using STCS compaction strategy with
some kind of timeseries model, and you are going to end up with yor disk
full!



On 2 September 2017 at 03:46, qf zhou <zhouqf2...@gmail.com> wrote:

> After  I run  nodetool compactionstats -H,  it says that:
>
> pending tasks: 6
> - gps.gpsfullwithstate: 6
>
> id   compaction type keyspace table
>  completed  total  unit  progress
> 56ebd730-8ede-11e7-9754-c981af5d39a9 Validation  gps
>  gpsfullwithstate 478.67 GiB 4.59 TiB   bytes 10.19%
> 3fc33340-8e4e-11e7-9754-c981af5d39a9 Compaction  gps
>  gpsfullwithstate 451.73 GiB 817.51 GiB bytes 55.26%
> f9acc4b0-8edf-11e7-9754-c981af5d39a9 Validation  gps
>  gpsfullwithstate 472.36 GiB 5.32 TiB   bytes 8.67%
> 4af0b300-8f7a-11e7-9754-c981af5d39a9 Compaction  gps
>  gpsfullwithstate 3.76 GiB   75.37 GiB  bytes 5.00%
> f1282280-8edf-11e7-9754-c981af5d39a9 Validation  gps
>  gpsfullwithstate 474.95 GiB 4.59 TiB   bytes 10.11%
> 0ccb7b90-8ee0-11e7-9754-c981af5d39a9 Validation  gps
>  gpsfullwithstate 472.4 GiB  5.32 TiB   bytes 8.67%
>
> what does it mean? the difference between Validation and Compaction
>
>
> 在 2017年9月1日，下午8:36，Nicolas Guyomar <nicolas.guyo...@gmail.com> 写道：
>
> Hi,
>
> Well, the command you are using works for me on 3.0.9, I do not have any
> logs in INFO level when I force a compaction and everything works fine for
> me.
>
> Are you sure there is nothing happening behind the scene ? What dies
> 'nodetool compactionstats -H' says ?
>
> On 1 September 2017 at 12:05, qf zhou <zhouqf2...@gmail.com> wrote:
>
>> When I trigger the compaction with the full path,  I found nothing in the
>> system.log.  Nothing happens in the  terminal and it just stops there.
>>
>> #calling operation forceUserDefinedCompaction of mbean
>> org.apache.cassandra.db:type=CompactionManager
>>
>>
>>
>>
>> 在 2017年9月1日，下午5:06，qf zhou <zhouqf2...@gmail.com> 写道：
>>
>> I  found the  following log.  What does it mean ?
>>
>> INFO  [CompactionExecutor:11] 2017-09-01 16:55:47,909
>> NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot
>> allocate chunk of 1.000MiB
>> WARN  [RMI TCP Connection(1714)-127.0.0.1] 2017-09-01 17:02:42,516
>> CompactionManager.java:704 - Schema does not exist for file
>> mc-151276-big-Data.db. Skipping.
>>
>>
>> 在 2017年9月1日，下午4:54，Nicolas Guyomar <nicolas.guyo...@gmail.com> 写道：
>>
>> You should have a log coming from the CompactionManager (in cassandra
>> system.log) when you try the command, what does it says  ?
>>
>> On 1 September 2017 at 10:07, qf zhou <zhouqf2...@gmail.com> wrote:
>>
>>> When I run the command,  the following occurs and  it returns null.
>>>
>>> Is it normal ?
>>>
>>> echo "run -b org.apache.cassandra.db:type=CompactionManager
>>> forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar
>>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l
>>> localhost:7199
>>>
>>>
>>> Welcome to JMX terminal. Type "help" for available commands.
>>> $>run -b org.apache.cassandra.db:type=CompactionManager
>>> forceUserDefinedCompaction mc-100963-big-Data.db
>>> #calling operation forceUserDefinedCompaction of mbean
>>> org.apache.cassandra.db:type=CompactionManager
>>> #operation returns:
>>> null
>>>
>>>
>>>
>>>
>>> 在 2017年9月1日，下午3:49，Nicolas Guyomar <nicolas.guyo...@gmail.com> 写道：
>>>
>>> Hi,
>>>
>>> Last time I used forceUserDefinedCompaction, I got myself a headache
>>> because I was trying to use a full path like you're doing, but in fact it
>>> just need the sstable as parameter
>>>
>>> Can you just try :
>>>
>>> echo "run -b org.apache.cassandra.db:type=CompactionM

Re: old big tombstone data file occupy much disk space

[Nicolas Guyomar]

Hi,

Well, the command you are using works for me on 3.0.9, I do not have any
logs in INFO level when I force a compaction and everything works fine for
me.

Are you sure there is nothing happening behind the scene ? What dies
'nodetool compactionstats -H' says ?

On 1 September 2017 at 12:05, qf zhou <zhouqf2...@gmail.com> wrote:

> When I trigger the compaction with the full path,  I found nothing in the
> system.log.  Nothing happens in the  terminal and it just stops there.
>
> #calling operation forceUserDefinedCompaction of mbean
> org.apache.cassandra.db:type=CompactionManager
>
>
>
>
> 在 2017年9月1日，下午5:06，qf zhou <zhouqf2...@gmail.com> 写道：
>
> I  found the  following log.  What does it mean ?
>
> INFO  [CompactionExecutor:11] 2017-09-01 16:55:47,909 NoSpamLogger.java:91
> - Maximum memory usage reached (512.000MiB), cannot allocate chunk of
> 1.000MiB
> WARN  [RMI TCP Connection(1714)-127.0.0.1] 2017-09-01 17:02:42,516
> CompactionManager.java:704 - Schema does not exist for file
> mc-151276-big-Data.db. Skipping.
>
>
> 在 2017年9月1日，下午4:54，Nicolas Guyomar <nicolas.guyo...@gmail.com> 写道：
>
> You should have a log coming from the CompactionManager (in cassandra
> system.log) when you try the command, what does it says  ?
>
> On 1 September 2017 at 10:07, qf zhou <zhouqf2...@gmail.com> wrote:
>
>> When I run the command,  the following occurs and  it returns null.
>>
>> Is it normal ?
>>
>> echo "run -b org.apache.cassandra.db:type=CompactionManager
>> forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar
>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199
>>
>>
>> Welcome to JMX terminal. Type "help" for available commands.
>> $>run -b org.apache.cassandra.db:type=CompactionManager
>> forceUserDefinedCompaction mc-100963-big-Data.db
>> #calling operation forceUserDefinedCompaction of mbean
>> org.apache.cassandra.db:type=CompactionManager
>> #operation returns:
>> null
>>
>>
>>
>>
>> 在 2017年9月1日，下午3:49，Nicolas Guyomar <nicolas.guyo...@gmail.com> 写道：
>>
>> Hi,
>>
>> Last time I used forceUserDefinedCompaction, I got myself a headache
>> because I was trying to use a full path like you're doing, but in fact it
>> just need the sstable as parameter
>>
>> Can you just try :
>>
>> echo "run -b org.apache.cassandra.db:type=CompactionManager
>> forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar
>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199
>>
>>
>>
>> On 1 September 2017 at 08:29, qf zhou <zhouqf2...@gmail.com> wrote:
>>
>>>
>>> dataPath=/hdd3/cassandra/data/gps/gpsfullwithstate-073e51a0c
>>> db811e68dce511be6a305f6/mc-100963-big-Data.db
>>> echo "run -b org.apache.cassandra.db:type=CompactionManager
>>> forceUserDefinedCompaction $dataPath" | java -jar
>>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l
>>> localhost:7199
>>>
>>> In the above, I am using a jmx method. But it seems that the file size
>>> doesn’t change. My command is wrong ?
>>>
>>> > 在 2017年9月1日，下午2:17，Jeff Jirsa <jji...@gmail.com> 写道：
>>> >
>>> > User defined compaction to do a single sstable compaction on just that
>>> sstable
>>> >
>>> > It's a nodetool command in very recent versions, or a jmx method in
>>> older versions
>>> >
>>> >
>>> > --
>>> > Jeff Jirsa
>>> >
>>> >
>>> >> On Aug 31, 2017, at 11:04 PM, qf zhou <zhouqf2...@gmail.com> wrote:
>>> >>
>>> >> I am using  a cluster with  3 nodes and  the cassandra version is
>>> 3.0.9. I have used it about 6 months. Now each node has about 1.5T data in
>>> the disk.
>>> >> I found some sstables file are over 300G. Using the  sstablemetadata
>>> command,  I found it:  Estimated droppable tombstones: 0.9622972799707109.
>>> >> It is obvious that too much tombstone data exists.
>>> >> The default_time_to_live = 864(100 days) and   gc_grace_seconds =
>>> 432000(5 days).  Using nodetool  compactionstats, I found the some
>>> compaction processes exists.
>>> >> So I really  want to know how to clear tombstone data ?  otherwise
>>> the disk space will cost too much.
>>> >> I really need some help, because some few people know cassandra in my
>>> company.
>>> >> Than

Re: old big tombstone data file occupy much disk space

[Nicolas Guyomar]

Whoops sorry I mislead you with cassandra 2.1 behavior, you were right
giving your sstable full path , what kind of log do you have when you
trigger the compaction with the full path ?

On 1 September 2017 at 11:30, Nicolas Guyomar <nicolas.guyo...@gmail.com>
wrote:

> Well, not sure why you reached a memory usage limit, but according to the
> 3.0 branche's code : https://github.com/apache/
> cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/compaction/
> CompactionManager.java#L632 you just need to give the sstable filename,
> and Cassandra manage to find it based on cassandra version, sstable
> filename convention and so on
>
> Are you sure those sstables you are trying to get rid off are really in an
> active schema, and not some leftover from an old keyspace/table? This is
> what "schema does not exist" means to me.
>
> On 1 September 2017 at 11:06, qf zhou <zhouqf2...@gmail.com> wrote:
>
>> I  found the  following log.  What does it mean ?
>>
>> INFO  [CompactionExecutor:11] 2017-09-01 16:55:47,909
>> NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot
>> allocate chunk of 1.000MiB
>> WARN  [RMI TCP Connection(1714)-127.0.0.1] 2017-09-01 17:02:42,516
>> CompactionManager.java:704 - Schema does not exist for file
>> mc-151276-big-Data.db. Skipping.
>>
>>
>> 在 2017年9月1日，下午4:54，Nicolas Guyomar <nicolas.guyo...@gmail.com> 写道：
>>
>> You should have a log coming from the CompactionManager (in cassandra
>> system.log) when you try the command, what does it says  ?
>>
>> On 1 September 2017 at 10:07, qf zhou <zhouqf2...@gmail.com> wrote:
>>
>>> When I run the command,  the following occurs and  it returns null.
>>>
>>> Is it normal ?
>>>
>>> echo "run -b org.apache.cassandra.db:type=CompactionManager
>>> forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar
>>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l
>>> localhost:7199
>>>
>>>
>>> Welcome to JMX terminal. Type "help" for available commands.
>>> $>run -b org.apache.cassandra.db:type=CompactionManager
>>> forceUserDefinedCompaction mc-100963-big-Data.db
>>> #calling operation forceUserDefinedCompaction of mbean
>>> org.apache.cassandra.db:type=CompactionManager
>>> #operation returns:
>>> null
>>>
>>>
>>>
>>>
>>> 在 2017年9月1日，下午3:49，Nicolas Guyomar <nicolas.guyo...@gmail.com> 写道：
>>>
>>> Hi,
>>>
>>> Last time I used forceUserDefinedCompaction, I got myself a headache
>>> because I was trying to use a full path like you're doing, but in fact it
>>> just need the sstable as parameter
>>>
>>> Can you just try :
>>>
>>> echo "run -b org.apache.cassandra.db:type=CompactionManager
>>> forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar
>>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l
>>> localhost:7199
>>>
>>>
>>>
>>> On 1 September 2017 at 08:29, qf zhou <zhouqf2...@gmail.com> wrote:
>>>
>>>>
>>>> dataPath=/hdd3/cassandra/data/gps/gpsfullwithstate-073e51a0c
>>>> db811e68dce511be6a305f6/mc-100963-big-Data.db
>>>> echo "run -b org.apache.cassandra.db:type=CompactionManager
>>>> forceUserDefinedCompaction $dataPath" | java -jar
>>>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l
>>>> localhost:7199
>>>>
>>>> In the above, I am using a jmx method. But it seems that the file size
>>>> doesn’t change. My command is wrong ?
>>>>
>>>> > 在 2017年9月1日，下午2:17，Jeff Jirsa <jji...@gmail.com> 写道：
>>>> >
>>>> > User defined compaction to do a single sstable compaction on just
>>>> that sstable
>>>> >
>>>> > It's a nodetool command in very recent versions, or a jmx method in
>>>> older versions
>>>> >
>>>> >
>>>> > --
>>>> > Jeff Jirsa
>>>> >
>>>> >
>>>> >> On Aug 31, 2017, at 11:04 PM, qf zhou <zhouqf2...@gmail.com> wrote:
>>>> >>
>>>> >> I am using  a cluster with  3 nodes and  the cassandra version is
>>>> 3.0.9. I have used it about 6 months. Now each node has about 1.5T data in
>>>> the disk.
>>>> >> I found some sstables file are over 300G. Using the  sstablemetadata
>>>&

Re: old big tombstone data file occupy much disk space

[Nicolas Guyomar]

Well, not sure why you reached a memory usage limit, but according to the
3.0 branche's code :
https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L632
you just need to give the sstable filename, and Cassandra manage to find it
based on cassandra version, sstable filename convention and so on

Are you sure those sstables you are trying to get rid off are really in an
active schema, and not some leftover from an old keyspace/table? This is
what "schema does not exist" means to me.

On 1 September 2017 at 11:06, qf zhou <zhouqf2...@gmail.com> wrote:

> I  found the  following log.  What does it mean ?
>
> INFO  [CompactionExecutor:11] 2017-09-01 16:55:47,909 NoSpamLogger.java:91
> - Maximum memory usage reached (512.000MiB), cannot allocate chunk of
> 1.000MiB
> WARN  [RMI TCP Connection(1714)-127.0.0.1] 2017-09-01 17:02:42,516
> CompactionManager.java:704 - Schema does not exist for file
> mc-151276-big-Data.db. Skipping.
>
>
> 在 2017年9月1日，下午4:54，Nicolas Guyomar <nicolas.guyo...@gmail.com> 写道：
>
> You should have a log coming from the CompactionManager (in cassandra
> system.log) when you try the command, what does it says  ?
>
> On 1 September 2017 at 10:07, qf zhou <zhouqf2...@gmail.com> wrote:
>
>> When I run the command,  the following occurs and  it returns null.
>>
>> Is it normal ?
>>
>> echo "run -b org.apache.cassandra.db:type=CompactionManager
>> forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar
>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199
>>
>>
>> Welcome to JMX terminal. Type "help" for available commands.
>> $>run -b org.apache.cassandra.db:type=CompactionManager
>> forceUserDefinedCompaction mc-100963-big-Data.db
>> #calling operation forceUserDefinedCompaction of mbean
>> org.apache.cassandra.db:type=CompactionManager
>> #operation returns:
>> null
>>
>>
>>
>>
>> 在 2017年9月1日，下午3:49，Nicolas Guyomar <nicolas.guyo...@gmail.com> 写道：
>>
>> Hi,
>>
>> Last time I used forceUserDefinedCompaction, I got myself a headache
>> because I was trying to use a full path like you're doing, but in fact it
>> just need the sstable as parameter
>>
>> Can you just try :
>>
>> echo "run -b org.apache.cassandra.db:type=CompactionManager
>> forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar
>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199
>>
>>
>>
>> On 1 September 2017 at 08:29, qf zhou <zhouqf2...@gmail.com> wrote:
>>
>>>
>>> dataPath=/hdd3/cassandra/data/gps/gpsfullwithstate-073e51a0c
>>> db811e68dce511be6a305f6/mc-100963-big-Data.db
>>> echo "run -b org.apache.cassandra.db:type=CompactionManager
>>> forceUserDefinedCompaction $dataPath" | java -jar
>>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l
>>> localhost:7199
>>>
>>> In the above, I am using a jmx method. But it seems that the file size
>>> doesn’t change. My command is wrong ?
>>>
>>> > 在 2017年9月1日，下午2:17，Jeff Jirsa <jji...@gmail.com> 写道：
>>> >
>>> > User defined compaction to do a single sstable compaction on just that
>>> sstable
>>> >
>>> > It's a nodetool command in very recent versions, or a jmx method in
>>> older versions
>>> >
>>> >
>>> > --
>>> > Jeff Jirsa
>>> >
>>> >
>>> >> On Aug 31, 2017, at 11:04 PM, qf zhou <zhouqf2...@gmail.com> wrote:
>>> >>
>>> >> I am using  a cluster with  3 nodes and  the cassandra version is
>>> 3.0.9. I have used it about 6 months. Now each node has about 1.5T data in
>>> the disk.
>>> >> I found some sstables file are over 300G. Using the  sstablemetadata
>>> command,  I found it:  Estimated droppable tombstones: 0.9622972799707109.
>>> >> It is obvious that too much tombstone data exists.
>>> >> The default_time_to_live = 864(100 days) and   gc_grace_seconds =
>>> 432000(5 days).  Using nodetool  compactionstats, I found the some
>>> compaction processes exists.
>>> >> So I really  want to know how to clear tombstone data ?  otherwise
>>> the disk space will cost too much.
>>> >> I really need some help, because some few people know cassandra in my
>>> company.
>>> >> Thank you very much!
>>> >>
>>> >>
>>> >> -
>>> >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> >> For additional commands, e-mail: user-h...@cassandra.apache.org
>>> >>
>>> >
>>> > -
>>> > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> > For additional commands, e-mail: user-h...@cassandra.apache.org
>>> >
>>>
>>>
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>>
>>>
>>
>>
>
>

Re: old big tombstone data file occupy much disk space

[Nicolas Guyomar]

You should have a log coming from the CompactionManager (in cassandra
system.log) when you try the command, what does it says  ?

On 1 September 2017 at 10:07, qf zhou <zhouqf2...@gmail.com> wrote:

> When I run the command,  the following occurs and  it returns null.
>
> Is it normal ?
>
> echo "run -b org.apache.cassandra.db:type=CompactionManager
> forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar
> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199
>
>
> Welcome to JMX terminal. Type "help" for available commands.
> $>run -b org.apache.cassandra.db:type=CompactionManager
> forceUserDefinedCompaction mc-100963-big-Data.db
> #calling operation forceUserDefinedCompaction of mbean
> org.apache.cassandra.db:type=CompactionManager
> #operation returns:
> null
>
>
>
>
> 在 2017年9月1日，下午3:49，Nicolas Guyomar <nicolas.guyo...@gmail.com> 写道：
>
> Hi,
>
> Last time I used forceUserDefinedCompaction, I got myself a headache
> because I was trying to use a full path like you're doing, but in fact it
> just need the sstable as parameter
>
> Can you just try :
>
> echo "run -b org.apache.cassandra.db:type=CompactionManager
> forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar
> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199
>
>
>
> On 1 September 2017 at 08:29, qf zhou <zhouqf2...@gmail.com> wrote:
>
>>
>> dataPath=/hdd3/cassandra/data/gps/gpsfullwithstate-073e51a0c
>> db811e68dce511be6a305f6/mc-100963-big-Data.db
>> echo "run -b org.apache.cassandra.db:type=CompactionManager
>> forceUserDefinedCompaction $dataPath" | java -jar
>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199
>>
>> In the above, I am using a jmx method. But it seems that the file size
>> doesn’t change. My command is wrong ?
>>
>> > 在 2017年9月1日，下午2:17，Jeff Jirsa <jji...@gmail.com> 写道：
>> >
>> > User defined compaction to do a single sstable compaction on just that
>> sstable
>> >
>> > It's a nodetool command in very recent versions, or a jmx method in
>> older versions
>> >
>> >
>> > --
>> > Jeff Jirsa
>> >
>> >
>> >> On Aug 31, 2017, at 11:04 PM, qf zhou <zhouqf2...@gmail.com> wrote:
>> >>
>> >> I am using  a cluster with  3 nodes and  the cassandra version is
>> 3.0.9. I have used it about 6 months. Now each node has about 1.5T data in
>> the disk.
>> >> I found some sstables file are over 300G. Using the  sstablemetadata
>> command,  I found it:  Estimated droppable tombstones: 0.9622972799707109.
>> >> It is obvious that too much tombstone data exists.
>> >> The default_time_to_live = 864(100 days) and   gc_grace_seconds =
>> 432000(5 days).  Using nodetool  compactionstats, I found the some
>> compaction processes exists.
>> >> So I really  want to know how to clear tombstone data ?  otherwise the
>> disk space will cost too much.
>> >> I really need some help, because some few people know cassandra in my
>> company.
>> >> Thank you very much!
>> >>
>> >>
>> >> -
>> >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> >> For additional commands, e-mail: user-h...@cassandra.apache.org
>> >>
>> >
>> > -
>> > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> > For additional commands, e-mail: user-h...@cassandra.apache.org
>> >
>>
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>
>
>

Re: old big tombstone data file occupy much disk space

[Nicolas Guyomar]

Hi,

Last time I used forceUserDefinedCompaction, I got myself a headache
because I was trying to use a full path like you're doing, but in fact it
just need the sstable as parameter

Can you just try :

echo "run -b org.apache.cassandra.db:type=CompactionManager
forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar
/opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199



On 1 September 2017 at 08:29, qf zhou  wrote:

>
> dataPath=/hdd3/cassandra/data/gps/gpsfullwithstate-
> 073e51a0cdb811e68dce511be6a305f6/mc-100963-big-Data.db
> echo "run -b org.apache.cassandra.db:type=CompactionManager
> forceUserDefinedCompaction $dataPath" | java -jar 
> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar
>  -l localhost:7199
>
> In the above, I am using a jmx method. But it seems that the file size
> doesn’t change. My command is wrong ?
>
> > 在 2017年9月1日，下午2:17，Jeff Jirsa  写道：
> >
> > User defined compaction to do a single sstable compaction on just that
> sstable
> >
> > It's a nodetool command in very recent versions, or a jmx method in
> older versions
> >
> >
> > --
> > Jeff Jirsa
> >
> >
> >> On Aug 31, 2017, at 11:04 PM, qf zhou  wrote:
> >>
> >> I am using  a cluster with  3 nodes and  the cassandra version is
> 3.0.9. I have used it about 6 months. Now each node has about 1.5T data in
> the disk.
> >> I found some sstables file are over 300G. Using the  sstablemetadata
> command,  I found it:  Estimated droppable tombstones: 0.9622972799707109.
> >> It is obvious that too much tombstone data exists.
> >> The default_time_to_live = 864(100 days) and   gc_grace_seconds =
> 432000(5 days).  Using nodetool  compactionstats, I found the some
> compaction processes exists.
> >> So I really  want to know how to clear tombstone data ?  otherwise the
> disk space will cost too much.
> >> I really need some help, because some few people know cassandra in my
> company.
> >> Thank you very much!
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> >> For additional commands, e-mail: user-h...@cassandra.apache.org
> >>
> >
> > -
> > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: user-h...@cassandra.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: Working With Prepared Statements

[Nicolas Guyomar]

I would suggest to read this post by the last pickle:
http://thelastpickle.com/blog/2016/09/15/Null-bindings-on-prepared-statements-and-undesired-tombstone-creation.html
 and make sure you are not concerned by the mentioned behavior, because
some people still choose to use C* v2.X because of some bugs in v3 branch

Have fun !

On 29 August 2017 at 13:43, Shalom Sagges <shal...@liveperson.com> wrote:

> Sounds great then.
>
> Thanks a lot guys! 
>
>
> Shalom Sagges
> DBA
> <http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
> <http://www.facebook.com/LivePersonInc> We Create Meaningful Connections
>
>
>
> On Tue, Aug 29, 2017 at 2:41 PM, Nicolas Guyomar <
> nicolas.guyo...@gmail.com> wrote:
>
>> Hi Shalom,
>>
>> AFAIK, you are completely safe with prepared statement, there are no
>> caveats using them, and you will have better performance.
>>
>> Make sure to only prepare them once ;)
>>
>> On 29 August 2017 at 13:41, Matija Gobec <matija0...@gmail.com> wrote:
>>
>>> I don't see any disadvantages or warning signs. You will see a
>>> performance increase on moderate request rate frequency.
>>>
>>> On Tue, Aug 29, 2017 at 1:28 PM, Shalom Sagges <shal...@liveperson.com>
>>> wrote:
>>>
>>>> Hi Matija,
>>>>
>>>> I just wish to know if there are any disadvantages when using prepared
>>>> statement or any warning signs I should look for. Queries will run multiple
>>>> times so it fits the use case.
>>>>
>>>> Thanks!
>>>>
>>>>
>>>> Shalom Sagges
>>>> DBA
>>>> <http://www.linkedin.com/company/164748>
>>>> <http://twitter.com/liveperson> <http://www.facebook.com/LivePersonInc> We
>>>> Create Meaningful Connections
>>>>
>>>>
>>>>
>>>> On Tue, Aug 29, 2017 at 2:18 PM, Matija Gobec <matija0...@gmail.com>
>>>> wrote:
>>>>
>>>>> Do you have any concrete questions re prepared statements?
>>>>>
>>>>> They are faster to execute since the statement is already parsed and
>>>>> in C* and you just pass the parameters. No additional statement processing
>>>>> is needed.
>>>>>
>>>>> Matija
>>>>>
>>>>> On Tue, Aug 29, 2017 at 12:33 PM, Shalom Sagges <
>>>>> shal...@liveperson.com> wrote:
>>>>>
>>>>>> Insights, anyone?
>>>>>>
>>>>>>
>>>>>> Shalom Sagges
>>>>>> DBA
>>>>>> <http://www.linkedin.com/company/164748>
>>>>>> <http://twitter.com/liveperson>
>>>>>> <http://www.facebook.com/LivePersonInc> We Create Meaningful
>>>>>> Connections
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Aug 28, 2017 at 10:43 AM, Shalom Sagges <
>>>>>> shal...@liveperson.com> wrote:
>>>>>>
>>>>>>> Hi Everyone,
>>>>>>>
>>>>>>> I want to start working with Prepared Statements.
>>>>>>>
>>>>>>> I've read https://docs.datastax.com/en/developer/java-driver/3.1/
>>>>>>> manual/statements/prepared/ and just wanted to know if there are
>>>>>>> any other considerations I need to take into account when deciding to 
>>>>>>> use
>>>>>>> Prepared Statements.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>>
>>>>>>> Shalom Sagges
>>>>>>> DBA
>>>>>>> <http://www.linkedin.com/company/164748>
>>>>>>> <http://twitter.com/liveperson>
>>>>>>> <http://www.facebook.com/LivePersonInc> We Create Meaningful
>>>>>>> Connections
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> This message may contain confidential and/or privileged information.
>>>>>> If you are not the addressee or authorized to receive this on behalf
>>>>>> of the addressee you must not use, copy, disclose or take action based on
>>>>>> this message or any information herein.
>>>>>> If you have received this message in error, please advise the sender
>>>>>> immediately by reply email and delete this message. Thank you.
>>>>>>
>>>>>
>>>>>
>>>>
>>>> This message may contain confidential and/or privileged information.
>>>> If you are not the addressee or authorized to receive this on behalf of
>>>> the addressee you must not use, copy, disclose or take action based on this
>>>> message or any information herein.
>>>> If you have received this message in error, please advise the sender
>>>> immediately by reply email and delete this message. Thank you.
>>>>
>>>
>>>
>>
>
> This message may contain confidential and/or privileged information.
> If you are not the addressee or authorized to receive this on behalf of
> the addressee you must not use, copy, disclose or take action based on this
> message or any information herein.
> If you have received this message in error, please advise the sender
> immediately by reply email and delete this message. Thank you.
>

Re: Working With Prepared Statements

[Nicolas Guyomar]

Hi Shalom,

AFAIK, you are completely safe with prepared statement, there are no
caveats using them, and you will have better performance.

Make sure to only prepare them once ;)

On 29 August 2017 at 13:41, Matija Gobec  wrote:

> I don't see any disadvantages or warning signs. You will see a performance
> increase on moderate request rate frequency.
>
> On Tue, Aug 29, 2017 at 1:28 PM, Shalom Sagges 
> wrote:
>
>> Hi Matija,
>>
>> I just wish to know if there are any disadvantages when using prepared
>> statement or any warning signs I should look for. Queries will run multiple
>> times so it fits the use case.
>>
>> Thanks!
>>
>>
>> Shalom Sagges
>> DBA
>>  
>>  We Create Meaningful Connections
>>
>>
>>
>> On Tue, Aug 29, 2017 at 2:18 PM, Matija Gobec 
>> wrote:
>>
>>> Do you have any concrete questions re prepared statements?
>>>
>>> They are faster to execute since the statement is already parsed and in
>>> C* and you just pass the parameters. No additional statement processing is
>>> needed.
>>>
>>> Matija
>>>
>>> On Tue, Aug 29, 2017 at 12:33 PM, Shalom Sagges 
>>> wrote:
>>>
 Insights, anyone?


 Shalom Sagges
 DBA
 
   We
 Create Meaningful Connections



 On Mon, Aug 28, 2017 at 10:43 AM, Shalom Sagges  wrote:

> Hi Everyone,
>
> I want to start working with Prepared Statements.
>
> I've read https://docs.datastax.com/en/developer/java-driver/3.1/
> manual/statements/prepared/ and just wanted to know if there are any
> other considerations I need to take into account when deciding to use
> Prepared Statements.
>
> Thanks!
>
>
> Shalom Sagges
> DBA
> 
> 
>  We Create Meaningful
> Connections
>
>
>


 This message may contain confidential and/or privileged information.
 If you are not the addressee or authorized to receive this on behalf of
 the addressee you must not use, copy, disclose or take action based on this
 message or any information herein.
 If you have received this message in error, please advise the sender
 immediately by reply email and delete this message. Thank you.

>>>
>>>
>>
>> This message may contain confidential and/or privileged information.
>> If you are not the addressee or authorized to receive this on behalf of
>> the addressee you must not use, copy, disclose or take action based on this
>> message or any information herein.
>> If you have received this message in error, please advise the sender
>> immediately by reply email and delete this message. Thank you.
>>
>
>

Re: How to find dataSize at client side?

[Nicolas Guyomar]

Hi,

The list is opened :
https://groups.google.com/a/lists.datastax.com/forum/#!forum/java-driver-user,
feel free to subscribe.

Datastax is the main maintainer of the java driver, which is open source (
https://github.com/datastax/java-driver ) , which is not the same driver as
the DSE one : https://github.com/datastax/java-dse-driver



On 24 May 2017 at 10:53, techpyaasa . <techpya...@gmail.com> wrote:

> Hi Nicolas
>
> I think only DataStax Enterprise(paid) c* version can ask questions/get
> support from datastax :(
>
> On Tue, May 23, 2017 at 9:44 PM, techpyaasa . <techpya...@gmail.com>
> wrote:
>
>> Thanks for your reply..
>>
>> On Tue, May 23, 2017 at 7:40 PM, Nicolas Guyomar <
>> nicolas.guyo...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> If you were to know the batch size on client side to make sure it does
>>> not get above the 5kb limit, so that you can "limit the number of
>>> statements in a batch", I would suspect you do not need batch at all right
>>> ? See  https://inoio.de/blog/2016/01/13/cassandra-to-batch-or-not-
>>> to-batch/
>>>
>>> As for your question, you might get an answer on the java driver ML :
>>> java-driver-u...@lists.datastax.com
>>>
>>>
>>> On 23 May 2017 at 15:25, techpyaasa . <techpya...@gmail.com> wrote:
>>>
>>>>
>>>> * WARN [SharedPool-Worker-1] 2017-05-22 20:28:46,204
>>>> BatchStatement.java (line 253) Batch of prepared statements for
>>>> [site24x7.wm_rawstats_tb, site24x7.wm_rawstats] is of size 6122, exceeding
>>>> specified threshold of 5120 by 1002*
>>>> We are frequently getting this message in logs, so I wanted to restrict
>>>> inserts at client side by calculating *dataSize* of insert/batch
>>>> statements before sending it to c* servers.
>>>>
>>>> We are using datastax java drivers , how can I get dataSize here??
>>>>
>>>>
>>>> Any ideas??
>>>>
>>>> Thanks in advance
>>>> TechPyaasa
>>>>
>>>
>>>
>>
>

Re: How to find dataSize at client side?

[Nicolas Guyomar]

Hi,

If you were to know the batch size on client side to make sure it does not
get above the 5kb limit, so that you can "limit the number of statements in
a batch", I would suspect you do not need batch at all right ? See
https://inoio.de/blog/2016/01/13/cassandra-to-batch-or-not-to-batch/

As for your question, you might get an answer on the java driver ML :
java-driver-u...@lists.datastax.com


On 23 May 2017 at 15:25, techpyaasa .  wrote:

>
> * WARN [SharedPool-Worker-1] 2017-05-22 20:28:46,204 BatchStatement.java
> (line 253) Batch of prepared statements for [site24x7.wm_rawstats_tb,
> site24x7.wm_rawstats] is of size 6122, exceeding specified threshold of
> 5120 by 1002*
> We are frequently getting this message in logs, so I wanted to restrict
> inserts at client side by calculating *dataSize* of insert/batch
> statements before sending it to c* servers.
>
> We are using datastax java drivers , how can I get dataSize here??
>
>
> Any ideas??
>
> Thanks in advance
> TechPyaasa
>

Re: Is it safe to upgrade 2.2.6 to 3.0.13?

[Nicolas Guyomar]

Hi Xihui,

I was looking for this documentation also, but I believe datastax removed
it, and it is not available yet on the apache website

As far as I remember, intermediate version was needed if  C* Version <
2.1.7.

You should be safe starting from 2.2.6, but testing the upgrade on a
dedicated platform is always a good idea.

Nicolas

On 19 May 2017 at 09:02, Xihui He  wrote:

> Hi All,
>
> We are planning to upgrade our production cluster to 3.x, but I can't find
> the upgrade guide anymore.
> Can I upgrade to 3.0.13 from 2.2.6 directly? Is a interim version
> necessary?
>
> Thanks,
> Xihui
>

Re: TRUNCATE on a disk almost full - possible?

[Nicolas Guyomar]

Hi Kunal,

Timeout usually occured in the client (eg cqlsh), it does not mean that the
truncate operation is interrupted.

Have you checked that you have no old snapshot (automatic snaphost for
instance) that you could get rid off to get some space back ?

On 21 April 2017 at 11:27, benjamin roth  wrote:

> Truncate needs no space. It just creates a hard link of all affected
> SSTables under the corresponding -SNAPSHOT dir (at least with default
> settings) and then removes the SSTables.
> Also this operation should be rather fast as it is mostly a file-deletion
> process with some metadata updates.
>
> 2017-04-21 11:21 GMT+02:00 Kunal Gangakhedkar :
>
>> Hi all,
>>
>> We have a CF that's grown too large - it's not getting actively used in
>> the app right now.
>> The on-disk size of the . directory is ~407GB and I have only
>> ~40GB free left on the disk.
>>
>> I understand that if I trigger a TRUNCATE on this CF, cassandra will try
>> to take snapshot.
>> My question:
>> Is the ~40GB enough to safely truncate this table?
>>
>> I will manually remove the . directory once the truncate is
>> completed.
>>
>> Also, while browsing through earlier msgs regarding truncate, I noticed
>> that it's possible to get OperationTimedOut
>> 
>> exception. Does that stop the truncate operation?
>>
>> Is there any other safe way to clean up the CF?
>>
>> Thanks,
>> Kunal
>>
>
>

Re: Can we get username and timestamp in cqlsh_history?

[Nicolas Guyomar]

Hi Anuja,

What your are looking for is provided as part of DSE :
https://docs.datastax.com/en/datastax_enterprise/5.0/datastax_enterprise/sec/auditEnabling.html

On 1 April 2017 at 20:15, Vladimir Yudovin  wrote:

> Hi anuja,
>
> I don't thinks there is a way to do this without creating custom Cassandra
> build.
> There is mutations logs and somewhere on list was thread about parsing
> them, but I'm not sure it's what you need.
>
> Best regards, Vladimir Yudovin,
> *Winguzone  - Cloud Cassandra Hosting*
>
>
>  On Wed, 29 Mar 2017 07:37:15 -0400 *anuja jain  >* wrote 
>
> Hi,
> I have a cassandra cluster having a lot of keyspaces and users. I want to
> get the history of cql commands along with the username and the time at
> which the command is run.
> Also if we are running some commands from GUI tools like
> Devcenter,dbeaver, can we log those commands too? If yes, how?
>
> Thanks,
> Anuja
>
>
>

Re: Determining if data will be created on Cassandra Write Exceptions

[Nicolas Guyomar]

Hi,

If my understanding is correct, as long as you know that at least one node
acknowledge a write, it will be replicated at some point in the cluster. A
retry failure depends on what you consider a failure :)

If you absolutely need LOCAL_QUORUM to succeed, and for any reason
Cassandra can't at the time you try to use it, then I believe you have to
kind of rollback. But again, if the first write can't be done at
LOCAL_QUORUM, neither will your rollback => This must be what Edward called
" a square peg round hole discussion"

I personnaly believe that, with a well planned repair strategy, plus
read_repair and  hints mecanism, downgradingConsistencyRetryPolicy is quite
acceptable in term of consistency

Again, there are no magic rules when it comes to handle exception, because
it deeply depends on what you want to achieve.

Nicolas


On 16 February 2017 at 16:33, rouble <r.ou...@gmail.com> wrote:

> Thanks for the links. I get that all queries need to be idempotent, and we
> should use retries for data consistency.
>
> But, what happens when the retries fail? Then, the data *may* be there. To
> maintain consistency we need to rollback any created data, correct?
>
> tia,
> rouble
>
> On Wed, Feb 15, 2017 at 4:53 AM, Nicolas Guyomar <
> nicolas.guyo...@gmail.com> wrote:
>
>> Hi Rouble,
>>
>> I usually have to read javadoc in java driver to get my ideas straight
>> regarding exception handling.
>>
>> You can find informations reading : http://docs.datastax.com/en/
>> drivers/java/3.1/com/datastax/driver/core/policies/RetryPolicy.html  and
>> for instance http://docs.datastax.com/en/drivers/java/3.1/com/da
>> tastax/driver/core/policies/DowngradingConsistencyRetryPolicy.html  with
>> the onWriteTimeout method which differentiate several case of error.
>>
>> As Edward stated, you can know how many replica acknowleged the write in
>> Cassandra response.
>>
>> Keep in mind that retrying usually mean your write query is idempotent or
>> you don't care having duplicate entries
>>
>>
>> On 14 February 2017 at 21:49, Edward Capriolo <edlinuxg...@gmail.com>
>> wrote:
>>
>>>
>>>
>>> On Tue, Feb 14, 2017 at 2:30 PM, rouble <r.ou...@gmail.com> wrote:
>>>
>>>> Cassandra Gurus,
>>>>
>>>> I have a question open on stackoverlow on how to determine if data is
>>>> actually created when a Write exception is thrown:
>>>> http://stackoverflow.com/questions/42231140/determin
>>>> ing-if-data-will-be-created-on-cassandra-write-exceptions
>>>>
>>>> From my discussion on the question, it seems that on *any* Cassandra
>>>> write, *any* exception, means the data may or may not be written. With the
>>>> exception of InvalidQueryException.
>>>>
>>>> I find this unsettling. Maybe I need time to adjust from my RDBMS
>>>> background, but how is Cassandra supposed to be used for systems that need
>>>> user feedback? or is it?
>>>>
>>>> Let me use the simple example of account creation. User tries to create
>>>> an account, and we need to indicate one way or the other whether the
>>>> account was created. Lets say a WriteTimeoutException is thrown while
>>>> trying to add the user. User may or may not be written, what do we tell the
>>>> user? Should we just rollback the change and tell the user that it failed.
>>>> This seems like the only thing we can do deterministically (and I notice
>>>> someone doing just that here: http://stackoverflow.com/a/348
>>>> 60495/215120).
>>>>
>>>> How are people handling WriteTimeoutExceptions or
>>>> UnavailableExceptions? Rolling back in every case does not seem practical.
>>>>
>>>> tia,
>>>> Rouble
>>>>
>>>
>>> There is a difference between WriteTimeoutException and
>>> UnavailableException.
>>>
>>> UnavailableException indicates the write was never even attempted.
>>>
>>> WriteTimeoutException means the write was attempted. I believe you can
>>> interrogate the exception to determine if the operation was successful on
>>> any of the natural endpoints.
>>>
>>> The way to "cope" is idempotent writes and retries.  If that model does
>>> not fit it is a square peg round hole discussion.
>>>
>>
>>
>

Re: Determining if data will be created on Cassandra Write Exceptions

[Nicolas Guyomar]

Hi Rouble,

I usually have to read javadoc in java driver to get my ideas straight
regarding exception handling.

You can find informations reading :
http://docs.datastax.com/en/drivers/java/3.1/com/datastax/driver/core/policies/RetryPolicy.html
 and for instance
http://docs.datastax.com/en/drivers/java/3.1/com/datastax/driver/core/policies/DowngradingConsistencyRetryPolicy.html
 with the onWriteTimeout method which differentiate several case of error.

As Edward stated, you can know how many replica acknowleged the write in
Cassandra response.

Keep in mind that retrying usually mean your write query is idempotent or
you don't care having duplicate entries


On 14 February 2017 at 21:49, Edward Capriolo  wrote:

>
>
> On Tue, Feb 14, 2017 at 2:30 PM, rouble  wrote:
>
>> Cassandra Gurus,
>>
>> I have a question open on stackoverlow on how to determine if data is
>> actually created when a Write exception is thrown: http://stackoverflow.c
>> om/questions/42231140/determining-if-data-will-be-created-on
>> -cassandra-write-exceptions
>>
>> From my discussion on the question, it seems that on *any* Cassandra
>> write, *any* exception, means the data may or may not be written. With the
>> exception of InvalidQueryException.
>>
>> I find this unsettling. Maybe I need time to adjust from my RDBMS
>> background, but how is Cassandra supposed to be used for systems that need
>> user feedback? or is it?
>>
>> Let me use the simple example of account creation. User tries to create
>> an account, and we need to indicate one way or the other whether the
>> account was created. Lets say a WriteTimeoutException is thrown while
>> trying to add the user. User may or may not be written, what do we tell the
>> user? Should we just rollback the change and tell the user that it failed.
>> This seems like the only thing we can do deterministically (and I notice
>> someone doing just that here: http://stackoverflow.com/a/34860495/215120
>> ).
>>
>> How are people handling WriteTimeoutExceptions or UnavailableExceptions?
>> Rolling back in every case does not seem practical.
>>
>> tia,
>> Rouble
>>
>
> There is a difference between WriteTimeoutException and
> UnavailableException.
>
> UnavailableException indicates the write was never even attempted.
>
> WriteTimeoutException means the write was attempted. I believe you can
> interrogate the exception to determine if the operation was successful on
> any of the natural endpoints.
>
> The way to "cope" is idempotent writes and retries.  If that model does
> not fit it is a square peg round hole discussion.
>