Adding disk to operating C*

[Eunsu Kim]

Hello,

I use 5 nodes to create a cluster of Cassandra. (SSD 1TB)

I'm trying to mount an additional disk(SSD 1TB) on each node because each disk 
usage growth rate is higher than I expected. Then I will add the the directory 
to data_file_directories in cassanra.yaml

Can I get advice from who have experienced this situation?
If we go through the above steps one by one, will we be able to complete the 
upgrade without losing data?
The replication strategy is SimpleStrategy, RF 2.

Thank you in advance
-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: One time major deletion/purge vs periodic deletion

[Charulata Sharma (charshar)]

Well it’s not like that. We don’t just purge. There are business rules which 
will decide the records to be purged or archived and then purged, so cannot 
rely on TTL.

Thanks,
Charu

From: Jens Rantil 
Reply-To: "user@cassandra.apache.org" 
Date: Tuesday, March 6, 2018 at 12:34 AM
To: "user@cassandra.apache.org" 
Subject: Re: One time major deletion/purge vs periodic deletion

Sounds like you are using Cassandra as a queue. It's an antibiotic pattern. 
What I would do would be to rely on TTL for removal of data and use the TWCS 
compaction strategy to handle removal and you just focus on insertion.
On Tue, Mar 6, 2018, 07:39 Charulata Sharma (charshar) 
> wrote:
Hi,

  Wanted the community’s feedback on deciding the schedule of Archive and 
Purge job.
Is it better to Purge a large volume of data at regular intervals (like run A 
jobs once in 3 months ) or purge smaller amounts more frequently (run the job 
weekly??)

Some estimates on the number of deletes performed would be…upto 80-90K  rows 
purged in 3 months vs 10K deletes every week ??

Thanks,
Charu

--

Jens Rantil
Backend Developer @ Tink

Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
For urgent matters you can reach me at +46-708-84 18 32.

Re: Cassandra 2.1.18 - Concurrent nodetool repair resulting in > 30K SSTables for a single small (GBytes) CF

[kurt greaves]

>
>  What we did have was some sort of overlapping between our daily repair
> cronjob and the newly added node still in progress joining. Don’t know if
> this sort of combination might causing troubles.

I wouldn't be surprised if this caused problems. Probably want to avoid
that.

with waiting a few minutes after each finished execution and every time I
> see “… out of sync …” log messages in context of the repair, so it looks
> like, that each repair execution is detecting inconsistencies. Does this
> make sense?

Well it doesn't, but there have been issues in the past that caused exactly
this problem. I was under the impression they were all fixed by 2.1.18
though.

Additionally, we are writing at CL ANY, reading at ONE and repair chance
> for the 2 CFs in question is default 0.1

Have you considered writing at least at CL [LOCAL_]ONE? At the very least
it would rule out if there's a problem with hints.
​

Re: Cassandra/Spark failing to process large table

[Ben Slater]

Hi Faraz

Yes, it likely does mean there is inconsistency in the replicas. However,
you shouldn’t be too freaked out about it - Cassandra is design to allow
for this inconsistency to occur and the consistency levels allow you to
achieve consistent results despite replicas not being consistent. To keep
you replicas as consistent as possible (which is still a good thing), you
do need to regularly run repairs (once a week is the standard
recommendation for full repairs). Inconsistency can result from a whole
range of conditions from nodes being down the cluster being overloaded to
network issues.

Cheers
Ben

On Tue, 6 Mar 2018 at 22:18 Faraz Mateen  wrote:

> Thanks a lot for the response.
>
> Setting consistency to ALL/TWO started giving me consistent  count results
> on both cqlsh and spark. As expected, my query time has increased by 1.5x (
> Before, it was taking ~1.6 hours but with consistency level ALL, same query
> is taking ~2.4 hours to complete.)
>
> Does this mean my replicas are out of sync? When I first started pushing
> data to cassandra, I had a single node setup. Then I added two more nodes,
> changed replication factor to 2 and ran nodetool repair to distribute data
> to all the nodes. So, according to my understanding the nodes should have
> passively replicated data among themselves to remain in sync.
>
> Do I need to run repairs repeatedly to keep data in sync?
> How can I further debug why my replicas were not in sync before?
>
> Thanks,
> Faraz
>
> On Sun, Mar 4, 2018 at 9:46 AM, Ben Slater 
> wrote:
>
>> Both CQLSH and the Spark Cassandra query at consistent level ONE
>> (LOCAL_ONE for Spark connector) by default so if there is any inconsistency
>> in your replicas this can resulting in inconsistent query results.
>>
>> See http://cassandra.apache.org/doc/latest/tools/cqlsh.html and
>> https://github.com/datastax/spark-cassandra-connector/blob/master/doc/reference.md
>>  for
>> info on how to chance consistency. If you are unsure of how consistent the
>> on-disk replicas are (eg if you have been writing at CL One or haven’t run
>> repaires) that using consistency level all should give you the most
>> consistent results but requires all replicas to be available for the query
>> to succeed. If you are using QUORUM for your writes then querying at QUORUM
>> or LOCAL_QUORUM as appropriate should give you consistent results.
>>
>> Cheers
>> Ben
>>
>> On Sun, 4 Mar 2018 at 00:59 Kant Kodali  wrote:
>>
>>> The fact that cqlsh itself gives different results tells me that this
>>> has nothing to do with spark. Moreover, spark results are monotonically
>>> increasing which seem to be more consistent than cqlsh. so I believe
>>> spark can be taken out of the equation.
>>>
>>>  Now, while you are running these queries is there another process or
>>> thread that is writing also at the same time ? If yes then your results are
>>> fine but If it's not, you may want to try nodetool flush first and then run
>>> these iterations again?
>>>
>>> Thanks!
>>>
>>>
>>> On Fri, Mar 2, 2018 at 11:17 PM, Faraz Mateen  wrote:
>>>
 Hi everyone,

 I am trying to use spark to process a large cassandra table (~402
 million entries and 84 columns) but I am getting inconsistent results.
 Initially the requirement was to copy some columns from this table to
 another table. After copying the data, I noticed that some entries in the
 new table were missing. To verify that I took count of the large source
 table but I am getting different values each time. I tried the queries on a
 smaller table (~7 million records) and the results were fine.

 Initially, I attempted to take count using pyspark. Here is my pyspark
 script:

 spark = SparkSession.builder.appName("Datacopy App").getOrCreate()
 df = 
 spark.read.format("org.apache.spark.sql.cassandra").options(table=sourcetable,
  keyspace=sourcekeyspace).load().cache()
 df.createOrReplaceTempView("data")
 query = ("select count(1) from data " )
 vgDF = spark.sql(query)
 vgDF.show(10)

 Spark submit command is as follows:

 ~/spark-2.1.0-bin-hadoop2.7/bin/spark-submit --master 
 spark://10.128.0.18:7077 --packages 
 datastax:spark-cassandra-connector:2.0.1-s_2.11 --conf 
 spark.cassandra.connection.host="10.128.1.1,10.128.1.2,10.128.1.3" --conf 
 "spark.storage.memoryFraction=1" --conf spark.local.dir=/media/db/ 
 --executor-memory 10G --num-executors=6 --executor-cores=2 
 --total-executor-cores 18 pyspark_script.py

 The above spark submit process takes ~90 minutes to complete. I ran it
 three times and here are the counts I got:

 Spark iteration 1:  402273852
 Spark iteration 2:  402273884
 Spark iteration 3:  402274209

 Spark does not show any error or exception during the entire process. I
 ran the same query in cqlsh thrice 

Re: DCAwareRoundRobinPolicy question

[Justin Cameron]

Hi Nicu,

By default (with those lines commented), your application will only read
data from the local datacenter.

If you get different results when those lines are uncommented it means that
the data is not consistent between your datacenters. You'll need to run a
full repair to ensure that the data is consistent across both datacenters.

For your migration you should join the new DC, run a full repair, then
switch your applications over to the new DC (i.e. change withLocalDc()).
You can then safely decommission the nodes in your original DC.

I would recommend doing the repair one node at a time (sub-range repair
would be even better), to allow you to better control the process and limit
the potential performance impact on your applications.

Justin

On Tue, 6 Mar 2018 at 23:18 Marasoiu, Nicu 
wrote:

> Hi,
> We normally use a single data center, but when switching from one to
> another we will temporarily have two. The applications works with the local
> DC part of the cluster.
> Normally the configuration of the driver would contain (the commented
> lines absent):
>
> .withLoadBalancingPolicy(DCAwareRoundRobinPolicy.builder()
> .withLocalDc(localDataCenter)
> //  .withUsedHostsPerRemoteDc(3)
> //  .allowRemoteDCsForLocalConsistencyLevel()
> .build())
>
> However, the queries returned empty. Uncommenting the lines, the queries
> returned data. Multiple runs on multiple apps in multiple locations/dcs,
> gave data with lines commented/absent.
>
> All these apps connect to a cassandra cluster within a single data center
> currently. We are not clear why the empty results happened. Do you have any
> idea what might have caused this?
> Having the commented lines uncommented is dangerous in the context of how
> the data center migration is planned, since the queries would be able to
> reach the remote nodes which are not having all the data initially.
>
> Please advise,
> Thanks,
> Nicu Marasoiu
> Geschäftsanschrift/Business address: METRO SYSTEMS GmbH, Metro-Straße 12,
> 40235 Düsseldorf, Germany
> 
> Aufsichtsrat/Supervisory Board: Heiko Hutmacher (Vorsitzender/ Chairman)
> Geschäftsführung/Management Board: Dr. Dirk Toepfer (Vorsitzender/CEO),
> Wim van Herwijnen
> Sitz Düsseldorf, Amtsgericht Düsseldorf, HRB 18232/Registered Office
> Düsseldorf, Commercial Register of the Düsseldorf Local Court, HRB 18232
>
> Betreffend Mails von *@metrosystems.net
> Die in dieser E-Mail enthaltenen Nachrichten und Anhänge sind
> ausschließlich für den bezeichneten Adressaten bestimmt. Sie können
> rechtlich geschützte, vertrauliche Informationen enthalten. Falls Sie nicht
> der bezeichnete Empfänger oder zum Empfang dieser E-Mail nicht berechtigt
> sind, ist die Verwendung, Vervielfältigung oder Weitergabe der Nachrichten
> und Anhänge untersagt. Falls Sie diese E-Mail irrtümlich erhalten haben,
> informieren Sie bitte unverzüglich den Absender und vernichten Sie die
> E-Mail.
>
> Regarding mails from *@metrosystems.net
> This e-mail message and any attachment are intended exclusively for the
> named addressee. They may contain confidential information which may also
> be protected by professional secrecy. Unless you are the named addressee
> (or authorised to receive for the addressee) you may not copy or use this
> message or any attachment or disclose the contents to anyone else. If this
> e-mail was sent to you by mistake please notify the sender immediately and
> delete this e-mail.
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
> --


*Justin Cameron*Senior Software Engineer





This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).

This email and any attachments may contain confidential and legally
privileged information.  If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.

Re: Seed nodes of DC2 creating own versions of system keyspaces

[Jeff Jirsa]

On Tue, Mar 6, 2018 at 9:50 AM, Oleksandr Shulgin <
oleksandr.shul...@zalando.de> wrote:

> On 6 Mar 2018 16:55, "Jeff Jirsa"  wrote:
>
> On Mar 6, 2018, at 12:32 AM, Oleksandr Shulgin <
> oleksandr.shul...@zalando.de> wrote:
>
> On 5 Mar 2018 16:13, "Jeff Jirsa"  wrote:
>
> On Mar 5, 2018, at 6:40 AM, Oleksandr Shulgin <
> oleksandr.shul...@zalando.de> wrote:
>
> We were deploying a second DC today with 3 seed nodes (30 nodes in total)
> and we have noticed that all seed nodes reported the following:
>
> INFO  10:20:50 Create new Keyspace: KeyspaceMetadata{name=system_traces,
> params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{
> class=org.apache.cassandra.locator.SimpleStrategy,
> replication_factor=2}}, ...
>
> followed by similar lines for system_distributed and system_auth.  Is this
> to be expected?
>
> They’re written with timestamp=0 to ensure they’re created at least once,
> but if you’ve ever issued an ALTER to the table or keyspace, your modified
> version will win through normal schema reconciliation process.
>
>
> OK.  Any specific reason why non-bootstrapping nodes don't wait for schema
> propagation before joining the ring?
>
>
>
> They do in 3.0 and newer, the built in keyspaces still get auto created
> before that happens
>
>
> We are seeing this on 3.0.15, but if it's no longer the case with newer
> versions, then fine.
>
>
>
Sorry, I wasnt as precise as I should have been:

In 3.0 and newer, a bootstrapping node will wait until it has schema before
it bootstraps. HOWEVER, we make the ssystem_auth/system_distributed, etc
keyspaces as a node starts up, before it requests the schema from the rest
of the cluster.

You will see some schema exchanges go through the cluster as new 3.0 nodes
come online, but it's a no-op schema change.

Re: system.size_estimates - safe to remove sstables?

[Chris Lohfink]

While its off you can delete the files in the directory yeah

Chris

> On Mar 6, 2018, at 2:35 AM, Kunal Gangakhedkar  
> wrote:
> 
> Hi Chris,
> 
> I checked for snapshots and backups - none found.
> Also, we're not using opscenter, hadoop or spark or any such tool.
> 
> So, do you think we can just remove the cf and restart the service?
> 
> Thanks,
> Kunal
> 
> On 5 March 2018 at 21:52, Chris Lohfink  > wrote:
> Any chance space used by snapshots? What files exist there that are taking up 
> space?
> 
> > On Mar 5, 2018, at 1:02 AM, Kunal Gangakhedkar  > > wrote:
> >
> > Hi all,
> >
> > I have a 2-node cluster running cassandra 2.1.18.
> > One of the nodes has run out of disk space and died - almost all of it 
> > shows up as occupied by size_estimates CF.
> > Out of 296GiB, 288GiB shows up as consumed by size_estimates in 'du -sh' 
> > output.
> >
> > This is while the other node is chugging along - shows only 25MiB consumed 
> > by size_estimates (du -sh output).
> >
> > Any idea why this descripancy?
> > Is it safe to remove the size_estimates sstables from the affected node and 
> > restart the service?
> >
> > Thanks,
> > Kunal
> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
> 
> For additional commands, e-mail: user-h...@cassandra.apache.org 
> 
> 
> 

Re: Cassandra at Instagram with Dikang Gu interview by Jeff Carpenter

[Goutham reddy]

It’s an interesting conversation. For more details about the pluggable
storage engine here is the link.

Blog:
https://thenewstack.io/instagram-supercharges-cassandra-pluggable-rocksdb-storage-engine/

JIRA:
https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-13475


On Tue, Mar 6, 2018 at 9:01 AM Kenneth Brotman 
wrote:

> Just released on DataStax Distributed Data Show, DiKang Gu of Instagram
> interviewed by author Jeff Carpenter.
>
> Found it really interesting:  Shadow clustering, migrating from 2.2 to
> 3.0, using the Rocks DB as a pluggable storage engine for Cassandra
>
>
> https://academy.datastax.com/content/distributed-data-show-episode-37-cassandra-instagram-dikang-gu
>
>
>
> Kenneth Brotman
>
-- 
Regards
Goutham Reddy

Re: Seed nodes of DC2 creating own versions of system keyspaces

[Oleksandr Shulgin]

On 6 Mar 2018 16:55, "Jeff Jirsa"  wrote:

On Mar 6, 2018, at 12:32 AM, Oleksandr Shulgin 
wrote:

On 5 Mar 2018 16:13, "Jeff Jirsa"  wrote:

On Mar 5, 2018, at 6:40 AM, Oleksandr Shulgin 
wrote:

We were deploying a second DC today with 3 seed nodes (30 nodes in total)
and we have noticed that all seed nodes reported the following:

INFO  10:20:50 Create new Keyspace: KeyspaceMetadata{name=system_traces,
params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{
class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=2}},
...

followed by similar lines for system_distributed and system_auth.  Is this
to be expected?

They’re written with timestamp=0 to ensure they’re created at least once,
but if you’ve ever issued an ALTER to the table or keyspace, your modified
version will win through normal schema reconciliation process.


OK.  Any specific reason why non-bootstrapping nodes don't wait for schema
propagation before joining the ring?



They do in 3.0 and newer, the built in keyspaces still get auto created
before that happens


We are seeing this on 3.0.15, but if it's no longer the case with newer
versions, then fine.

Thanks,
--
Alex

Cassandra at Instagram with Dikang Gu interview by Jeff Carpenter

[Kenneth Brotman]

Just released on DataStax Distributed Data Show, DiKang Gu of Instagram
interviewed by author Jeff Carpenter.

Found it really interesting:  Shadow clustering, migrating from 2.2 to 3.0,
using the Rocks DB as a pluggable storage engine for Cassandra

https://academy.datastax.com/content/distributed-data-show-episode-37-cassan
dra-instagram-dikang-gu

 

Kenneth Brotman

Re: Rocksandra blog post

[Dikang Gu]

Thanks everyone!

@Kyrylo, In Rocksandra world, the storage engine is built on top of
RocksDB, which is another LSM tree based storage engine. So the immutable
sstables and compactions are managed by RocksDB instances. RocksDB supports
different compaction strategies, similar to STCS and LCS. The compactions
are managed by C++, so it will not generate any pressure to JVM.

@Romain, Thanks, and Rocksandra is not limited to only support key/value
data model. We support most of the data types and table data model in
Cassandra.

Dikang.

On Tue, Mar 6, 2018 at 1:48 AM, Romain Hardouin  wrote:

> Rocksandra is very interesting for key/value data model. Let's hope it
> will land in C* upstream in the near future thanks to pluggable storage.
> Thanks Dikang!
>
>
>
> Le mardi 6 mars 2018 à 10:06:16 UTC+1, Kyrylo Lebediev <
> kyrylo_lebed...@epam.com> a écrit :
>
>
> Thanks for sharing, Dikang!
>
> Impressive results.
>
>
> As you plugged in different storage engine, it's interesting how you're
> dealing with compactions in Rocksandra?
>
> Is there still the concept of immutable SSTables + compaction strategies
> or it was changed somehow?
>
>
> Best,
>
> Kyrill
>
> --
> *From:* Dikang Gu 
> *Sent:* Monday, March 5, 2018 8:26 PM
> *To:* d...@cassandra.apache.org; cassandra
> *Subject:* Rocksandra blog post
>
> As some of you already know, Instagram Cassandra team is working on the
> project to use RocksDB as Cassandra's storage engine.
>
> Today, we just published a blog post about the work we have done, and more
> excitingly, we published the benchmark metrics in AWS environment.
>
> Check it out here:
> https://engineering.instagram.com/open-sourcing-a-10x-
> reduction-in-apache-cassandra-tail-latency-d64f86b43589
>
> Thanks
> Dikang
>
>


-- 
Dikang

Re: cfhistograms InstanceNotFoundException EstimatePartitionSizeHistogram

[Nicolas Guyomar]

I got once this kind of problem and it was exactly what Chris explained.
Could you double check that on this remote host you do not have 2 versions
of cassandra and nodetool is pointing to the old one ?

On 6 March 2018 at 17:17, onmstester onmstester  wrote:

> One my PC i've the exactly same version of Cassandra and histograms
> command works perfectly so i'm sure that nothing is wrong with nodetool
> version
>
> Sent using Zoho Mail 
>
>
>  On Tue, 06 Mar 2018 17:38:04 +0330 *Chris Lohfink
> >* wrote 
>
> Make sure your using same version of nodetool as your version of
> Cassandra. That metric was renamed from EstimatedRowSize so if using a
> version of nodetool made for a more recent version you would get this error
> since EstimatePartitionSizeHistogram doesn’t exist on the older Cassandra
> host.
>
> Chris
>
> Sent from my iPhone
>
> On Mar 6, 2018, at 3:29 AM, onmstester onmstester 
> wrote:
>
> Running this command:
> nodetools cfhistograms keyspace1 table1
>
> throws this exception in production server:
> javax.management.InstanceNotFoundException: org.apache.cassandra.metrics:
> type=Table,keyspace=keyspace1,scope=table1,name=
> EstimatePartitionSizeHistogram
>
> But i have no problem in a test server with few data in it and same
> datamodel.
> I'm using Casssandra 3.
>
> Sent using Zoho Mail 
>
>
>
>

Re: cfhistograms InstanceNotFoundException EstimatePartitionSizeHistogram

[onmstester onmstester]

One my PC i've the exactly same version of Cassandra and histograms command 
works perfectly so i'm sure that nothing is wrong with nodetool version


Sent using Zoho Mail






 On Tue, 06 Mar 2018 17:38:04 +0330 Chris Lohfink 
clohf...@apple.com wrote 




Make sure your using same version of nodetool as your version of Cassandra. 
That metric was renamed from EstimatedRowSize so if using a version of nodetool 
made for a more recent version you would get this error since 
EstimatePartitionSizeHistogram doesn’t exist on the older Cassandra host.



Chris



Sent from my iPhone



On Mar 6, 2018, at 3:29 AM, onmstester onmstester onmstes...@zoho.com 
wrote:






Running this command:

nodetools cfhistograms keyspace1 table1



throws this exception in production server:

javax.management.InstanceNotFoundException: 
org.apache.cassandra.metrics:type=Table,keyspace=keyspace1,scope=table1,name=EstimatePartitionSizeHistogram



But i have no problem in a test server with few data in it and same datamodel.

I'm using Casssandra 3.

Sent using Zoho Mail

Re: Seed nodes of DC2 creating own versions of system keyspaces

[Jeff Jirsa]

-- 
Jeff Jirsa


> On Mar 6, 2018, at 12:32 AM, Oleksandr Shulgin  
> wrote:
> 
> On 5 Mar 2018 16:13, "Jeff Jirsa"  wrote:
>> On Mar 5, 2018, at 6:40 AM, Oleksandr Shulgin  
>> wrote:
>> We were deploying a second DC today with 3 seed nodes (30 nodes in total) 
>> and we have noticed that all seed nodes reported the following:
>> 
>> INFO  10:20:50 Create new Keyspace: KeyspaceMetadata{name=system_traces, 
>> params=KeyspaceParams{durable_writes=true, 
>> replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy,
>>  replication_factor=2}}, ...
>> 
>> followed by similar lines for system_distributed and system_auth.  Is this 
>> to be expected?
> They’re written with timestamp=0 to ensure they’re created at least once, but 
> if you’ve ever issued an ALTER to the table or keyspace, your modified 
> version will win through normal schema reconciliation process.
> 
> OK.  Any specific reason why non-bootstrapping nodes don't wait for schema 
> propagation before joining the ring?
> 


They do in 3.0 and newer, the built in keyspaces still get auto created before 
that happens

Re: data types storage saving

[Carl Mueller]

If you're willing to do the data type conversion in insert and retrieval,
the you could use blobs as a sort of "adaptive length int" AFAIK

On Tue, Mar 6, 2018 at 6:02 AM, onmstester onmstester 
wrote:

> I'm using int data type for one of my columns but for 99.99...% its data
> never would be > 65K, Should i change it to smallint (It would save some
> Gigabytes disks in a few months) or Cassandra Compression would take care
> of it in storage?
> What about blob data type ? Isn't  better to use it in such cases? could i
> alter column type from smallInt to int in future if needed?
>
> Sent using Zoho Mail 
>
>
>

Re: [External] Re: Whch version is the best version to run now?

[Anuj Wadehra]

We evaluated both 3.0.x and 3.11.x. +1 for 3.11.2 as we faced major performance 
issues with 3.0.x. We have NOT evaluated new features on 3.11.x.
Anuj

Sent from Yahoo Mail on Android 
 
  On Tue, 6 Mar 2018 at 19:35, Alain RODRIGUEZ wrote:   
Hello Tom,

It's good to hear this kind of feedbacks,
Thanks for sharing.

3.11.x seems to get more love from the community wrt patches. This is why I'd 
recommend 3.11.x for new projects.


I also agree with this analysis.

Stay away from any of the 2.x series, they're going EOL soonish and the newer 
versions are very stable.


+1 here as well. Maybe add that 3.11.x, that is described as 'very stable' 
above, aims at stabilizing Cassandra after the tick-tock releases and is a 'bug 
fix' series and brings features developed during this period, even though it is 
needed to be careful with of some the new features, even in latest 3.11.x 
versions.
I did not work that much with it yet, but I think I would pick 3.11.2 as well 
for a new cluster at the moment.
C*heers,
---Alain Rodriguez - @arodream - 
alain@thelastpickle.comFrance / Spain
The Last Pickle - Apache Cassandra Consultinghttp://www.thelastpickle.com

2018-03-05 12:39 GMT+00:00 Tom van der Woerdt :

We run on the order of a thousand Cassandra nodes in production. Most of that 
is 3.0.16, but new clusters are defaulting to 3.11.2 and some older clusters 
have been upgraded to it as well.
All of the bugs I encountered in 3.11.x were also seen in 3.0.x, but 3.11.x 
seems to get more love from the community wrt patches. This is why I'd 
recommend 3.11.x for new projects.
Stay away from any of the 2.x series, they're going EOL soonish and the newer 
versions are very stable.

Tom van der WoerdtSite Reliability Engineer

Booking.com B.V.
Vijzelstraat 66-80 Amsterdam 1017HL NetherlandsThe world's #1 accommodation 
site 
43 languages, 198+ offices worldwide, 120,000+ global destinations, 1,550,000+ 
room nights booked every day 
No booking fees, best price always guaranteed 
Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)

On Sat, Mar 3, 2018 at 12:25 AM, Jeff Jirsa  wrote:

I’d personally be willing to run 3.0.16
3.11.2 or 3 whatever should also be similar, but I haven’t personally tested it 
at any meaningful scale

-- Jeff Jirsa

On Mar 2, 2018, at 2:37 PM, Kenneth Brotman  
wrote:



Seems like a lot of people are running old versions of Cassandra.  What is the 
best version, most reliable stable version to use now?

 

Kenneth Brotman





  

Re: Rocksandra blog post

[Carl Mueller]

Basically they are avoiding gc, right? Not necessarily improving on the
theoreticals of sstables and LSM trees.

Why didn't they use/try scylla? I'd be interested to see that benchmark.

On Tue, Mar 6, 2018 at 3:48 AM, Romain Hardouin  wrote:

> Rocksandra is very interesting for key/value data model. Let's hope it
> will land in C* upstream in the near future thanks to pluggable storage.
> Thanks Dikang!
>
>
>
> Le mardi 6 mars 2018 à 10:06:16 UTC+1, Kyrylo Lebediev <
> kyrylo_lebed...@epam.com> a écrit :
>
>
> Thanks for sharing, Dikang!
>
> Impressive results.
>
>
> As you plugged in different storage engine, it's interesting how you're
> dealing with compactions in Rocksandra?
>
> Is there still the concept of immutable SSTables + compaction strategies
> or it was changed somehow?
>
>
> Best,
>
> Kyrill
>
> --
> *From:* Dikang Gu 
> *Sent:* Monday, March 5, 2018 8:26 PM
> *To:* d...@cassandra.apache.org; cassandra
> *Subject:* Rocksandra blog post
>
> As some of you already know, Instagram Cassandra team is working on the
> project to use RocksDB as Cassandra's storage engine.
>
> Today, we just published a blog post about the work we have done, and more
> excitingly, we published the benchmark metrics in AWS environment.
>
> Check it out here:
> https://engineering.instagram.com/open-sourcing-a-10x-
> reduction-in-apache-cassandra-tail-latency-d64f86b43589
>
> Thanks
> Dikang
>
>

Re: [External] Re: Whch version is the best version to run now?

[Carlos Rolo]

Hello,

Our 5 cents. Either 3.0.16 or 3.11.x
We are really happy with the way 3.11.1/2 is behaving.
We still have a lot of really well behaving Clusters in 2.1/2.2 latest.


Regards,

Carlos Juzarte Rolo
Cassandra Consultant / Datastax Certified Architect / Cassandra MVP

Pythian - Love your data

rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
*linkedin.com/in/carlosjuzarterolo
*
Mobile: +351 918 918 100
www.pythian.com

On Tue, Mar 6, 2018 at 2:05 PM, Alain RODRIGUEZ  wrote:

> Hello Tom,
>
> It's good to hear this kind of feedbacks,
>
> Thanks for sharing.
>
> 3.11.x seems to get more love from the community wrt patches. This is why
>> I'd recommend 3.11.x for new projects.
>>
>
> I also agree with this analysis.
>
> Stay away from any of the 2.x series, they're going EOL soonish and the
>> newer versions are very stable.
>>
>
> +1 here as well. Maybe add that 3.11.x, that is described as 'very stable'
> above, aims at stabilizing Cassandra after the tick-tock releases and is a
> 'bug fix' series and brings features developed during this period, even
> though it is needed to be careful with of some the new features, even in
> latest 3.11.x versions.
>
> I did not work that much with it yet, but I think I would pick 3.11.2 as
> well for a new cluster at the moment.
>
> C*heers,
>
> ---
> Alain Rodriguez - @arodream - al...@thelastpickle.com
> France / Spain
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
>
> 2018-03-05 12:39 GMT+00:00 Tom van der Woerdt <
> tom.vanderwoe...@booking.com>:
>
>> We run on the order of a thousand Cassandra nodes in production. Most of
>> that is 3.0.16, but new clusters are defaulting to 3.11.2 and some older
>> clusters have been upgraded to it as well.
>>
>> All of the bugs I encountered in 3.11.x were also seen in 3.0.x, but
>> 3.11.x seems to get more love from the community wrt patches. This is why
>> I'd recommend 3.11.x for new projects.
>>
>> Stay away from any of the 2.x series, they're going EOL soonish and the
>> newer versions are very stable.
>>
>> Tom van der Woerdt
>> Site Reliability Engineer
>>
>> Booking.com B.V.
>> Vijzelstraat 66
>> -80
>> Amsterdam 1017HL Netherlands
>> [image: Booking.com] 
>> The world's #1 accommodation site
>> 43 languages, 198+ offices worldwide, 120,000+ global destinations,
>> 1,550,000+ room nights booked every day
>> No booking fees, best price always guaranteed
>> Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)
>>
>> On Sat, Mar 3, 2018 at 12:25 AM, Jeff Jirsa  wrote:
>>
>>> I’d personally be willing to run 3.0.16
>>>
>>> 3.11.2 or 3 whatever should also be similar, but I haven’t personally
>>> tested it at any meaningful scale
>>>
>>>
>>> --
>>> Jeff Jirsa
>>>
>>>
>>> On Mar 2, 2018, at 2:37 PM, Kenneth Brotman <
>>> kenbrot...@yahoo.com.INVALID> wrote:
>>>
>>> Seems like a lot of people are running old versions of Cassandra.  What
>>> is the best version, most reliable stable version to use now?
>>>
>>>
>>>
>>> Kenneth Brotman
>>>
>>>
>>
>

-- 


--

Re: cfhistograms InstanceNotFoundException EstimatePartitionSizeHistogram

[Chris Lohfink]

Make sure your using same version of nodetool as your version of Cassandra. 
That metric was renamed from EstimatedRowSize so if using a version of nodetool 
made for a more recent version you would get this error since 
EstimatePartitionSizeHistogram doesn’t exist on the older Cassandra host.

Chris

Sent from my iPhone

> On Mar 6, 2018, at 3:29 AM, onmstester onmstester  wrote:
> 
> Running this command:
> nodetools cfhistograms keyspace1 table1
> 
> throws this exception in production server:
> javax.management.InstanceNotFoundException: 
> org.apache.cassandra.metrics:type=Table,keyspace=keyspace1,scope=table1,name=EstimatePartitionSizeHistogram
> 
> But i have no problem in a test server with few data in it and same datamodel.
> I'm using Casssandra 3.
> Sent using Zoho Mail
> 
> 
> 

Re: [External] Re: Whch version is the best version to run now?

[Alain RODRIGUEZ]

Hello Tom,

It's good to hear this kind of feedbacks,

Thanks for sharing.

3.11.x seems to get more love from the community wrt patches. This is why
> I'd recommend 3.11.x for new projects.
>

I also agree with this analysis.

Stay away from any of the 2.x series, they're going EOL soonish and the
> newer versions are very stable.
>

+1 here as well. Maybe add that 3.11.x, that is described as 'very stable'
above, aims at stabilizing Cassandra after the tick-tock releases and is a
'bug fix' series and brings features developed during this period, even
though it is needed to be careful with of some the new features, even in
latest 3.11.x versions.

I did not work that much with it yet, but I think I would pick 3.11.2 as
well for a new cluster at the moment.

C*heers,

---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com


2018-03-05 12:39 GMT+00:00 Tom van der Woerdt 
:

> We run on the order of a thousand Cassandra nodes in production. Most of
> that is 3.0.16, but new clusters are defaulting to 3.11.2 and some older
> clusters have been upgraded to it as well.
>
> All of the bugs I encountered in 3.11.x were also seen in 3.0.x, but
> 3.11.x seems to get more love from the community wrt patches. This is why
> I'd recommend 3.11.x for new projects.
>
> Stay away from any of the 2.x series, they're going EOL soonish and the
> newer versions are very stable.
>
> Tom van der Woerdt
> Site Reliability Engineer
>
> Booking.com B.V.
> Vijzelstraat 66
> -80
> Amsterdam 1017HL Netherlands
> [image: Booking.com] 
> The world's #1 accommodation site
> 43 languages, 198+ offices worldwide, 120,000+ global destinations,
> 1,550,000+ room nights booked every day
> No booking fees, best price always guaranteed
> Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)
>
> On Sat, Mar 3, 2018 at 12:25 AM, Jeff Jirsa  wrote:
>
>> I’d personally be willing to run 3.0.16
>>
>> 3.11.2 or 3 whatever should also be similar, but I haven’t personally
>> tested it at any meaningful scale
>>
>>
>> --
>> Jeff Jirsa
>>
>>
>> On Mar 2, 2018, at 2:37 PM, Kenneth Brotman 
>> wrote:
>>
>> Seems like a lot of people are running old versions of Cassandra.  What
>> is the best version, most reliable stable version to use now?
>>
>>
>>
>> Kenneth Brotman
>>
>>
>

Re: Please give input or approval of JIRA 14128 so we can continue document cleanup

[Alain RODRIGUEZ]

Hello Kenneth,

I believe this belongs to the dev list.

Please be mindful about this, I think it matters for you as you will get
faster answers and for us as we will be reading what we expect to find in
each place, and save time.

We all know here, as Cassandra users, how much it is important to partition
things correctly to improve scalability and performances, right? :)

Thanks for getting involved,

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2018-03-05 3:51 GMT+00:00 Kenneth Brotman :

> Two months ago Kurt Greaves cleaned up the home page of the website which
> currently has broken links and other issues.  We need to get that JIRA
> rapped up.  Further improvements, scores of them are coming.  Get ready!
> Please take time soon to review the patch he submitted.
>
> https://issues.apache.org/jira/browse/CASSANDRA-14128
>
>
>
> Kenneth Brotman
>
>
>
>
>
>
>

DCAwareRoundRobinPolicy question

[Marasoiu, Nicu]

Hi,
We normally use a single data center, but when switching from one to another we 
will temporarily have two. The applications works with the local DC part of the 
cluster.
Normally the configuration of the driver would contain (the commented lines 
absent):

.withLoadBalancingPolicy(DCAwareRoundRobinPolicy.builder()
.withLocalDc(localDataCenter)
//  .withUsedHostsPerRemoteDc(3)
//  .allowRemoteDCsForLocalConsistencyLevel()
.build())

However, the queries returned empty. Uncommenting the lines, the queries 
returned data. Multiple runs on multiple apps in multiple locations/dcs, gave 
data with lines commented/absent.

All these apps connect to a cassandra cluster within a single data center 
currently. We are not clear why the empty results happened. Do you have any 
idea what might have caused this?
Having the commented lines uncommented is dangerous in the context of how the 
data center migration is planned, since the queries would be able to reach the 
remote nodes which are not having all the data initially.

Please advise,
Thanks,
Nicu Marasoiu
Geschäftsanschrift/Business address: METRO SYSTEMS GmbH, Metro-Straße 12, 40235 
Düsseldorf, Germany
Aufsichtsrat/Supervisory Board: Heiko Hutmacher (Vorsitzender/ Chairman)
Geschäftsführung/Management Board: Dr. Dirk Toepfer (Vorsitzender/CEO), Wim van 
Herwijnen
Sitz Düsseldorf, Amtsgericht Düsseldorf, HRB 18232/Registered Office 
Düsseldorf, Commercial Register of the Düsseldorf Local Court, HRB 18232

Betreffend Mails von *@metrosystems.net
Die in dieser E-Mail enthaltenen Nachrichten und Anhänge sind ausschließlich 
für den bezeichneten Adressaten bestimmt. Sie können rechtlich geschützte, 
vertrauliche Informationen enthalten. Falls Sie nicht der bezeichnete Empfänger 
oder zum Empfang dieser E-Mail nicht berechtigt sind, ist die Verwendung, 
Vervielfältigung oder Weitergabe der Nachrichten und Anhänge untersagt. Falls 
Sie diese E-Mail irrtümlich erhalten haben, informieren Sie bitte unverzüglich 
den Absender und vernichten Sie die E-Mail.

Regarding mails from *@metrosystems.net
This e-mail message and any attachment are intended exclusively for the named 
addressee. They may contain confidential information which may also be 
protected by professional secrecy. Unless you are the named addressee (or 
authorised to receive for the addressee) you may not copy or use this message 
or any attachment or disclose the contents to anyone else. If this e-mail was 
sent to you by mistake please notify the sender immediately and delete this 
e-mail.


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

data types storage saving

[onmstester onmstester]

I'm using int data type for one of my columns but for 99.99...% its data never 
would be  65K, Should i change it to smallint (It would save some Gigabytes 
disks in a few months) or Cassandra Compression would take care of it in 
storage? 

What about blob data type ? Isn't  better to use it in such cases? could i 
alter column type from smallInt to int in future if needed?


Sent using Zoho Mail

Re: Cassandra/Spark failing to process large table

[Faraz Mateen]

Thanks a lot for the response.

Setting consistency to ALL/TWO started giving me consistent  count results
on both cqlsh and spark. As expected, my query time has increased by 1.5x (
Before, it was taking ~1.6 hours but with consistency level ALL, same query
is taking ~2.4 hours to complete.)

Does this mean my replicas are out of sync? When I first started pushing
data to cassandra, I had a single node setup. Then I added two more nodes,
changed replication factor to 2 and ran nodetool repair to distribute data
to all the nodes. So, according to my understanding the nodes should have
passively replicated data among themselves to remain in sync.

Do I need to run repairs repeatedly to keep data in sync?
How can I further debug why my replicas were not in sync before?

Thanks,
Faraz

On Sun, Mar 4, 2018 at 9:46 AM, Ben Slater 
wrote:

> Both CQLSH and the Spark Cassandra query at consistent level ONE
> (LOCAL_ONE for Spark connector) by default so if there is any inconsistency
> in your replicas this can resulting in inconsistent query results.
>
> See http://cassandra.apache.org/doc/latest/tools/cqlsh.html and
> https://github.com/datastax/spark-cassandra-connector/blob/master/doc/
> reference.md for info on how to chance consistency. If you are unsure of
> how consistent the on-disk replicas are (eg if you have been writing at CL
> One or haven’t run repaires) that using consistency level all should give
> you the most consistent results but requires all replicas to be available
> for the query to succeed. If you are using QUORUM for your writes then
> querying at QUORUM or LOCAL_QUORUM as appropriate should give you
> consistent results.
>
> Cheers
> Ben
>
> On Sun, 4 Mar 2018 at 00:59 Kant Kodali  wrote:
>
>> The fact that cqlsh itself gives different results tells me that this has
>> nothing to do with spark. Moreover, spark results are monotonically
>> increasing which seem to be more consistent than cqlsh. so I believe
>> spark can be taken out of the equation.
>>
>>  Now, while you are running these queries is there another process or
>> thread that is writing also at the same time ? If yes then your results are
>> fine but If it's not, you may want to try nodetool flush first and then run
>> these iterations again?
>>
>> Thanks!
>>
>>
>> On Fri, Mar 2, 2018 at 11:17 PM, Faraz Mateen  wrote:
>>
>>> Hi everyone,
>>>
>>> I am trying to use spark to process a large cassandra table (~402
>>> million entries and 84 columns) but I am getting inconsistent results.
>>> Initially the requirement was to copy some columns from this table to
>>> another table. After copying the data, I noticed that some entries in the
>>> new table were missing. To verify that I took count of the large source
>>> table but I am getting different values each time. I tried the queries on a
>>> smaller table (~7 million records) and the results were fine.
>>>
>>> Initially, I attempted to take count using pyspark. Here is my pyspark
>>> script:
>>>
>>> spark = SparkSession.builder.appName("Datacopy App").getOrCreate()
>>> df = 
>>> spark.read.format("org.apache.spark.sql.cassandra").options(table=sourcetable,
>>>  keyspace=sourcekeyspace).load().cache()
>>> df.createOrReplaceTempView("data")
>>> query = ("select count(1) from data " )
>>> vgDF = spark.sql(query)
>>> vgDF.show(10)
>>>
>>> Spark submit command is as follows:
>>>
>>> ~/spark-2.1.0-bin-hadoop2.7/bin/spark-submit --master 
>>> spark://10.128.0.18:7077 --packages 
>>> datastax:spark-cassandra-connector:2.0.1-s_2.11 --conf 
>>> spark.cassandra.connection.host="10.128.1.1,10.128.1.2,10.128.1.3" --conf 
>>> "spark.storage.memoryFraction=1" --conf spark.local.dir=/media/db/ 
>>> --executor-memory 10G --num-executors=6 --executor-cores=2 
>>> --total-executor-cores 18 pyspark_script.py
>>>
>>> The above spark submit process takes ~90 minutes to complete. I ran it
>>> three times and here are the counts I got:
>>>
>>> Spark iteration 1:  402273852
>>> Spark iteration 2:  402273884
>>> Spark iteration 3:  402274209
>>>
>>> Spark does not show any error or exception during the entire process. I
>>> ran the same query in cqlsh thrice and got different results again:
>>>
>>> Cqlsh iteration 1:   402273598
>>> Cqlsh iteration 2:   402273499
>>> Cqlsh iteration 3:   402273515
>>>
>>> I am unable to find out why I am getting different outcomes from the
>>> same query. Cassandra system logs (*/var/log/cassandra/system.log*) has
>>> shown the following error message just once:
>>>
>>> ERROR [SSTableBatchOpen:3] 2018-02-27 09:48:23,592 CassandraDaemon.java:226 
>>> - Exception in thread Thread[SSTableBatchOpen:3,5,main]
>>> java.lang.AssertionError: Stats component is missing for sstable 
>>> /media/db/datakeyspace/sensordata1-acfa7880acba11e782fd9bf3ae460699/mc-58617-big
>>> at 
>>> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:460)
>>>  ~[apache-cassandra-3.9.jar:3.9]
>>> 

Re: ReplayException after shutdown and restart

[André Schütz]

Another information about this error.

Before creating the keyspace and the tables, we removed all former
commitlog entries and data from the local data folder. "Testimport" is
the keyspace for the tables.

/cassandra/data/commitlog/*
/cassandra/data/data/testimport/*

After the filling of th tables, there were correctly two tables under
the keyspace:

/cassandra/data/data/testimport/table1-hash1
/cassandra/data/data/testimport/table2-hash2

After the shutdown and the restart, we had 4 entries:

/cassandra/data/data/testimport/table1-hash1
/cassandra/data/data/testimport/table1-hash3
/cassandra/data/data/testimport/table2-hash2
/cassandra/data/data/testimport/table2-hash4

This maybe helps to give us a hint what went wrong?

Thanks,
Andre



On Tue, 6 Mar 2018 11:37:17 +0100
André Schütz  wrote:

> Hi,
> 
> we created a keyspace that contained two tables. These tables are
> filled with data. For a shutdown of the machine, we performed "nodetool
> flush" (and also in a second try "nodetool drain"). After that, we
> stopped the Cassandra service with "SigTerm"
> 
> We changed nothing after the system stop and restarted the node. The
> following error occured during restart.
> 
> [code]
> ERROR 10:14:33 Exiting due to error while processing commit log during 
> initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: 
> Unexpected error deserializing mutation; saved to 
> /tmp/mutation8563274912263154073dat.  This may be caused by replaying a 
> mutation against a table with the same name but incompatible schema.  
> Exception follows: org.apache.cassandra.serializers.MarshalException: 
> Expected 4 byte long for date (0)
> [/code]
> 
> Before the shutdown, the data was correctly and without errors available 
> within the database.
> 
> Any ideas about that error?
> 
> Thank you in advance.
> Andre
> 
> -- 
> Andre Schütz
> COO / Founder - Wegtam GmbH
> an...@wegtam.com | P: +49 (0) 381-80 699 041 | M: +49 (0) 176-218 02 604
> www.wegtam.com | www.tensei-data.com | www.wegtam.net
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 


-- 
Andre Schütz
COO / Founder - Wegtam GmbH
an...@wegtam.com | P: +49 (0) 381-80 699 041 | M: +49 (0) 176-218 02 604
www.wegtam.com | www.tensei-data.com | www.wegtam.net

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

ReplayException after shutdown and restart

[André Schütz]

Hi,

we created a keyspace that contained two tables. These tables are
filled with data. For a shutdown of the machine, we performed "nodetool
flush" (and also in a second try "nodetool drain"). After that, we
stopped the Cassandra service with "SigTerm"

We changed nothing after the system stop and restarted the node. The
following error occured during restart.

[code]
ERROR 10:14:33 Exiting due to error while processing commit log during 
initialization.
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: 
Unexpected error deserializing mutation; saved to 
/tmp/mutation8563274912263154073dat.  This may be caused by replaying a 
mutation against a table with the same name but incompatible schema.  Exception 
follows: org.apache.cassandra.serializers.MarshalException: Expected 4 byte 
long for date (0)
[/code]

Before the shutdown, the data was correctly and without errors available within 
the database.

Any ideas about that error?

Thank you in advance.
Andre

-- 
Andre Schütz
COO / Founder - Wegtam GmbH
an...@wegtam.com | P: +49 (0) 381-80 699 041 | M: +49 (0) 176-218 02 604
www.wegtam.com | www.tensei-data.com | www.wegtam.net

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Rocksandra blog post

[Romain Hardouin]

 Rocksandra is very interesting for key/value data model. Let's hope it will 
land in C* upstream in the near future thanks to pluggable storage.Thanks 
Dikang!


Le mardi 6 mars 2018 à 10:06:16 UTC+1, Kyrylo Lebediev 
 a écrit :  
 
 #yiv7016643451 #yiv7016643451 -- P 
{margin-top:0;margin-bottom:0;}#yiv7016643451 
Thanks for sharing, Dikang! 


Impressive results. 





As you plugged in different storage engine, it's interesting how you're dealing 
with compactions in Rocksandra? 

Is there still the concept of immutable SSTables + compaction strategies or it 
was changed somehow?




Best, 


Kyrill


From: Dikang Gu 
Sent: Monday, March 5, 2018 8:26 PM
To: d...@cassandra.apache.org; cassandra
Subject: Rocksandra blog post As some of you already know, Instagram Cassandra 
team is working on the project to use RocksDB as Cassandra's storage engine. 
Today, we just published a blog post about the work we have done, and more 
excitingly, we published the benchmark metrics in AWS environment.
Check it out here: 
https://engineering.instagram.com/open-sourcing-a-10x-reduction-in-apache-cassandra-tail-latency-d64f86b43589

ThanksDikang

  

cfhistograms InstanceNotFoundException EstimatePartitionSizeHistogram

[onmstester onmstester]

Running this command:

nodetools cfhistograms keyspace1 table1



throws this exception in production server:

javax.management.InstanceNotFoundException: 
org.apache.cassandra.metrics:type=Table,keyspace=keyspace1,scope=table1,name=EstimatePartitionSizeHistogram


But i have no problem in a test server with few data in it and same datamodel.

I'm using Casssandra 3.
Sent using Zoho Mail

Re: Rocksandra blog post

[Kyrylo Lebediev]

Thanks for sharing, Dikang!

Impressive results.


As you plugged in different storage engine, it's interesting how you're dealing 
with compactions in Rocksandra?

Is there still the concept of immutable SSTables + compaction strategies or it 
was changed somehow?


Best,

Kyrill


From: Dikang Gu 
Sent: Monday, March 5, 2018 8:26 PM
To: d...@cassandra.apache.org; cassandra
Subject: Rocksandra blog post

As some of you already know, Instagram Cassandra team is working on the project 
to use RocksDB as Cassandra's storage engine.

Today, we just published a blog post about the work we have done, and more 
excitingly, we published the benchmark metrics in AWS environment.

Check it out here:
https://engineering.instagram.com/open-sourcing-a-10x-reduction-in-apache-cassandra-tail-latency-d64f86b43589

Thanks
Dikang

RE: Cassandra 2.1.18 - Concurrent nodetool repair resulting in > 30K SSTables for a single small (GBytes) CF

[Steinmaurer, Thomas]

Hi Kurt,

our provisioning layer allows extending a cluster only one-by-one, thus we 
didn’t add multiple nodes at the same time.

What we did have was some sort of overlapping between our daily repair cronjob 
and the newly added node still in progress joining. Don’t know if this sort of 
combination might causing troubles.

I did some further testing and run on the same node the following repair call.

nodetool repair -pr ks cf1 cf2

with waiting a few minutes after each finished execution and every time I see 
“… out of sync …” log messages in context of the repair, so it looks like, that 
each repair execution is detecting inconsistencies. Does this make sense?

As said: We are using vnodes (256), RF=3. Additionally, we are writing at CL 
ANY, reading at ONE and repair chance for the 2 CFs in question is default 0.1

Currently testing a few consecutives executions without -pr on the same node.

Thanks,
Thomas

From: kurt greaves [mailto:k...@instaclustr.com]
Sent: Montag, 05. März 2018 01:10
To: User 
Subject: Re: Cassandra 2.1.18 - Concurrent nodetool repair resulting in > 30K 
SSTables for a single small (GBytes) CF

Repairs with vnodes is likely to cause a lot of small SSTables if you have 
inconsistencies (at least 1 per vnode). Did you have any issues when adding 
nodes, or did you add multiple nodes at a time? Anything that could have lead 
to a bit of inconsistency could have been the cause.

I'd probably avoid running the repairs across all the nodes simultaneously and 
instead spread them out over a week. That likely made it worse. Also worth 
noting that in versions 3.0+ you won't be able to run nodetool repair in such a 
way because anti-compaction will be triggered which will fail if multiple 
anti-compactions are attempted simultaneously (if you run multiple repairs 
simultaneously).

Have a look at orchestrating your repairs with TLP's fork of 
cassandra-reaper.
​
The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313

Re: Current active queries and status/type

[Jens Rantil]

You can do sampling of tracing on a table to avoid some of the overhead.

On Fri, Mar 2, 2018, 00:23 D. Salvatore  wrote:

> Hi Nicolas,
> Thank you very much for the response.
> I am looking into something with a smaller time frame than a minute.
> Tracing is a good way to get these information but it introduces a huge
> overhead in the system that I'd like to avoid it.
>
> Thanks
> Salvatore
>
> 2018-03-01 15:08 GMT+00:00 Nicolas Guyomar :
>
>> Hi,
>>
>> With
>> org.apache.cassandra.metrics:type=ClientRequest,scope=Read,name=Latency and
>> OneMinuteRate you can have such a metrics
>>
>> As for the state of the request with regards to other node I do no think
>> you can have that IMHO with JMX  (this is available using TRACING per
>> request)
>>
>>
>> On 1 March 2018 at 15:50, D. Salvatore  wrote:
>>
>>> Hello!
>>> There is any way to know how many queries a node is currently serving
>>> through JMX(or other tools)? And the state of the request so, for example,
>>> if the request is waiting for data from another node?
>>>
>>> Thanks
>>> Salvatore
>>>
>>
>>
> --

Jens Rantil
Backend Developer @ Tink

Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
For urgent matters you can reach me at +46-708-84 18 32.

Re: system.size_estimates - safe to remove sstables?

[Kunal Gangakhedkar]

Hi Chris,

I checked for snapshots and backups - none found.
Also, we're not using opscenter, hadoop or spark or any such tool.

So, do you think we can just remove the cf and restart the service?

Thanks,
Kunal

On 5 March 2018 at 21:52, Chris Lohfink  wrote:

> Any chance space used by snapshots? What files exist there that are taking
> up space?
>
> > On Mar 5, 2018, at 1:02 AM, Kunal Gangakhedkar 
> wrote:
> >
> > Hi all,
> >
> > I have a 2-node cluster running cassandra 2.1.18.
> > One of the nodes has run out of disk space and died - almost all of it
> shows up as occupied by size_estimates CF.
> > Out of 296GiB, 288GiB shows up as consumed by size_estimates in 'du -sh'
> output.
> >
> > This is while the other node is chugging along - shows only 25MiB
> consumed by size_estimates (du -sh output).
> >
> > Any idea why this descripancy?
> > Is it safe to remove the size_estimates sstables from the affected node
> and restart the service?
> >
> > Thanks,
> > Kunal
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: One time major deletion/purge vs periodic deletion

[Jens Rantil]

Sounds like you are using Cassandra as a queue. It's an antibiotic pattern.
What I would do would be to rely on TTL for removal of data and use the
TWCS compaction strategy to handle removal and you just focus on insertion.

On Tue, Mar 6, 2018, 07:39 Charulata Sharma (charshar) 
wrote:

> Hi,
>
>
>
>   Wanted the community’s feedback on deciding the schedule of Archive
> and Purge job.
>
> Is it better to Purge a large volume of data at regular intervals (like
> run A jobs once in 3 months ) or purge smaller amounts more frequently
> (run the job weekly??)
>
>
>
> Some estimates on the number of deletes performed would be…upto 80-90K
>  rows purged in 3 months vs 10K deletes every week ??
>
>
>
> Thanks,
>
> Charu
>
>
>
-- 

Jens Rantil
Backend Developer @ Tink

Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
For urgent matters you can reach me at +46-708-84 18 32.

Re: Seed nodes of DC2 creating own versions of system keyspaces

[Oleksandr Shulgin]

On 5 Mar 2018 16:13, "Jeff Jirsa"  wrote:

On Mar 5, 2018, at 6:40 AM, Oleksandr Shulgin 
wrote:

We were deploying a second DC today with 3 seed nodes (30 nodes in total)
and we have noticed that all seed nodes reported the following:

INFO  10:20:50 Create new Keyspace: KeyspaceMetadata{name=system_traces,
params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{
class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=2}},
...

followed by similar lines for system_distributed and system_auth.  Is this
to be expected?

They’re written with timestamp=0 to ensure they’re created at least once,
but if you’ve ever issued an ALTER to the table or keyspace, your modified
version will win through normal schema reconciliation process.


OK.  Any specific reason why non-bootstrapping nodes don't wait for schema
propagation before joining the ring?

--
Alex