Re: Tool to manage cassandra

[Oskar Kjellin]

Do you mean Priam nitan?


> On 16 Jun 2017, at 16:48, Nitan Kainth  wrote:
> 
> you can try atlas from netflix, it is open source.
> 
>> On Jun 16, 2017, at 8:52 AM, Surbhi Gupta  wrote:
>> 
>> If u are using dse then u can use opscenter 
>> 
>> On Fri, Jun 16, 2017 at 6:01 AM Ram Bhatia  wrote:
>>> Hi
>>> 
>>> 
>>> 
>>> 
>>>  
>>> 
>>> 
>>> 
>>> 
>>> May I know, if there a tool similar to Oracle Enterprise Manager for 
>>> managing Cassandra ?
>>> 
>>> 
>>> 
>>> 
>>>  
>>> 
>>> 
>>> 
>>> 
>>> Thank you in advance for your help,
>>> 
>>> 
>>> 
>>> 
>>> Ram Bhatia
>>> 
>>> 
>>> 
>>> 
>>> -
>>> 
>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> 
>>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>> 
>>> 
>>> 
> 

Re: Reg:- Cassandra Data modelling for Search

[Oskar Kjellin]

Agree, I meant as Jonathan said to use C* for primary key and as a primary
storage and ES as an indexed version of what you have in cassandra.

2017-06-12 19:19 GMT+02:00 DuyHai Doan :

> Sorry, I misread some reply I had the impression that people recommend ES
> as primary datastore
>
> On Mon, Jun 12, 2017 at 7:12 PM, Jonathan Haddad 
> wrote:
>
>> Nobody is promoting ES as a primary datastore in this thread.  Every
>> mention of it is to accompany C*.
>>
>>
>>
>> On Mon, Jun 12, 2017 at 10:03 AM DuyHai Doan 
>> wrote:
>>
>>> For all those promoting ES as a PRIMARY datastore, please read this
>>> before:
>>>
>>> https://discuss.elastic.co/t/elasticsearch-as-a-primary-data
>>> base/85733/13
>>>
>>> There are a lot of warning before recommending ES as a datastore.
>>>
>>> The answer from Pilato, ES official evangelist:
>>>
>>>
>>>- You absolutely care about your data and you want to be able to
>>>reindex in all cases. You need for that a datastore. A datastore can be a
>>>filesystem where you store JSON, HDFS, and/or a database you prefer and 
>>> you
>>>are confident with. About how to inject data in it, you may want to read:
>>>http://david.pilato.fr/blog/2015/05/09/advanced-search
>>>-for-your-legacy-application/7
>>>
>>> 
>>>.
>>>
>>>
>>>
>>>
>>> On Mon, Jun 12, 2017 at 5:08 PM, Michael Mior 
>>> wrote:
>>>
 For queries 1-5 this seems like a potentially good use case for
 materialized views. Create one table with the videos stored by ID and the
 materialized views for each of the queries.

 --
 Michael Mior
 mm...@apache.org


 2017-06-11 22:40 GMT-04:00 @Nandan@ :

> Hi,
>
> Currently, I am working on data modeling for Video Company in which we
> have different types of users as well as different user functionality.
> But currently, my concern is about Search video module based on
> different fields.
>
> Query patterns are as below:-
> 1) Select video by actor.
> 2) select video by producer.
> 3) select video by music.
> 4) select video by actor and producer.
> 5) select video by actor and music.
>
> Note: - In short, We want to establish an advanced search module by
> which we can search by anyway and get the desired results.
>
> During a search , we need partial search also such that if any user
> can search "Harry" title, then we are able to give them result as all
> videos whose
>  title contains "Harry" at any location.
>
> As per my ideas, I have to create separate tables such as
> video_by_actor, video_by_producer etc.. and implement solr query on all
> tables. Otherwise,
> is there any others way by which we can implement this search module
> effectively.
>
> Please suggest.
>
> Best regards,
>


>>>
>

Re: Reg:- Cassandra Data modelling for Search

[Oskar Kjellin]

I haven't run solr with Cassandra myself. I just meant to run elasticsearch as 
a completely separate service and write there as well. 

> On 12 Jun 2017, at 09:45, @Nandan@ <nandanpriyadarshi...@gmail.com> wrote:
> 
> Do you mean to use Elastic Search with Cassandra?
> Even I am thinking to use Apache Solr With Cassandra. 
> In that case I have to create distributed tables such as:-
> 1) video_by_title, video_by_actor, video_by_year  etc..
> 2) After creating Tables , will have to configure solr core on all tables. 
> 
> Is it like this ?
> 
> 
> 
>  
> 
>> On Mon, Jun 12, 2017 at 3:19 PM, Oskar Kjellin <oskar.kjel...@gmail.com> 
>> wrote:
>> Why not elasticsearch for this use case? It will make your life much simpler
>> 
>> > On 12 Jun 2017, at 04:40, @Nandan@ <nandanpriyadarshi...@gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > Currently, I am working on data modeling for Video Company in which we 
>> > have different types of users as well as different user functionality.
>> > But currently, my concern is about Search video module based on different 
>> > fields.
>> >
>> > Query patterns are as below:-
>> > 1) Select video by actor.
>> > 2) select video by producer.
>> > 3) select video by music.
>> > 4) select video by actor and producer.
>> > 5) select video by actor and music.
>> >
>> > Note: - In short, We want to establish an advanced search module by which 
>> > we can search by anyway and get the desired results.
>> >
>> > During a search , we need partial search also such that if any user can 
>> > search "Harry" title, then we are able to give them result as all videos 
>> > whose
>> >  title contains "Harry" at any location.
>> >
>> > As per my ideas, I have to create separate tables such as video_by_actor, 
>> > video_by_producer etc.. and implement solr query on all tables. Otherwise,
>> > is there any others way by which we can implement this search module 
>> > effectively.
>> >
>> > Please suggest.
>> >
>> > Best regards,
> 

Re: Reg:- Cassandra Data modelling for Search

[Oskar Kjellin]

Why not elasticsearch for this use case? It will make your life much simpler 

> On 12 Jun 2017, at 04:40, @Nandan@  wrote:
> 
> Hi, 
> 
> Currently, I am working on data modeling for Video Company in which we have 
> different types of users as well as different user functionality. 
> But currently, my concern is about Search video module based on different 
> fields. 
> 
> Query patterns are as below:-
> 1) Select video by actor.
> 2) select video by producer.
> 3) select video by music.
> 4) select video by actor and producer. 
> 5) select video by actor and music. 
> 
> Note: - In short, We want to establish an advanced search module by which we 
> can search by anyway and get the desired results. 
> 
> During a search , we need partial search also such that if any user can 
> search "Harry" title, then we are able to give them result as all videos whose
>  title contains "Harry" at any location. 
> 
> As per my ideas, I have to create separate tables such as video_by_actor, 
> video_by_producer etc.. and implement solr query on all tables. Otherwise,
> is there any others way by which we can implement this search module 
> effectively. 
> 
> Please suggest.
> 
> Best regards,

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Using Cassandra for my usecase

[Oskar Kjellin]

You could put the tenant as a column that is part of the clustering key. That 
avoids large partitions. 

On 12 Jun 2017, at 07:14, Erick Ramirez  wrote:

>> Given my use case is cassandra the best suited one or is there any other 
>> database which suits my requirement better?
> 
> Probably not the right forum for that question. It's like walking into a Ford 
> dealership and asking if the Mustang is the best car for you. 
> 
> In any case, you would choose Cassandra because you require:
> - high availability
> - very fast reads
> - no single-point-of-failure
> - no downtime
> - you have a scale problem
> - etc
> 
>> What would be best way to implement multi-tenancy?
> 
> The "best" way is what works for your use case based on testing you've done. 
> As you already are aware in the example you provided, adding a column as the 
> tenant indicator could lead to large partitions so you need to be careful 
> about how you model your data.
> 
> Some implementations completely side-step this by distributing tenants across 
> keyspaces but that may not suit your needs.
> 
>> Given that I need to query by multiple dimensions would denormalized tables 
>> work better or should I be using materialized views?
> 
> With denormalised tables, your application needs to implement the logic for 
> batching the updates together.
> 
> With materialised views, that complexity is managed for you by C* but you 
> need to be aware of the performance impact associated with it. For example 
> with RF=3 on the base table, MV adds another RF=3 for an additional table so 
> RF=3+3. A second MV increases RF=3+3+3 and so on.
> 
>> Anything else that I need to consider based on your experiences with 
>> cassandra?
> 
> 
> Multi-tenancy can be difficult particularly for complex use cases. Test, test 
> and test. And make sure you always correctly size your cluster with enough 
> nodes.
> 
> You need to limit the number of tables to about 200 at the most (regardless 
> of the number of keyspaces). Having too many tables puts pressure on the heap 
> of each node.
> 
> Good luck!
> 
>> On Sun, Jun 11, 2017 at 2:07 AM, Govindarajan Srinivasaraghavan 
>>  wrote:
>> Hi All,
>> 
>> Just to give a background I'm working on a project where I need to store 
>> fast incoming time series data and have rest api's to query and serve the 
>> data to users when needed. The data as such is a single JSON which is 1kb in 
>> size and the data has to be purged after a specific time period (say few 
>> weeks or months). The incoming rate would be approximately 100k messages per 
>> second and the biggest challenge is the data should be query-able by 
>> multiple dimensions with sorting, paging and data dump options. 
>> 
>> I started looking into database options and felt like cassandra might be a 
>> good choice for my use case since the requirement needs faster writes. In 
>> order to query by multiple dimensions I had to insert the same record into 
>> multiple denormalized tables (around 8 tables). Now I need to implement 
>> multitenancy and having an extra column in the partition key to query by 
>> tenant will not work since there will be some tenants with huge amounts of 
>> data compared to the rest. My other option is to have the tenant identifier 
>> appended to the table names so that I can perform per teannt queries easily. 
>> 
>> Here are my questions for which I need some help.
>> - Given my use case is cassandra the best suited one or is there any other 
>> database which suits my requirement better?
>> - What would be best way to implement multi-tenancy?
>> - Given that I need to query by multiple dimensions would denormalized 
>> tables work better or should I be using materialized views?
>> - Anything else that I need to consider based on your experiences with 
>> cassandra?
>> 
>> Thanks
> 

Re: Apache Cassandra - Configuration Management

[Oskar Kjellin]

If you're on AWS you can use Netflix Priam 


> On 17 May 2017, at 18:40, Abhishek Gupta  wrote:
> 
> Hi Zaidi,
> 
> We use Chef for the configuration management of our 14 node cluster.
> 
> You can have a look at Chef or maybe some other config management tools too 
> like Ansible and Puppet.
> 
> 
> 
> Thanks,
> Abhishek
> 
>> On May 17, 2017 10:08 PM, "DuyHai Doan"  wrote:
>> For configuration management there are tons of tools out there:
>> 
>> - ansible
>> - chef
>> - puppet
>> - saltstack
>> 
>> I surely forgot a few others 
>> 
>> 
>>> On Wed, May 17, 2017 at 6:33 PM, ZAIDI, ASAD A  wrote:
>>> Good Morning Folks –
>>> 
>>>  
>>> 
>>> I’m running 14 nodes Cassandra cluster in two data centers , each node is 
>>> has roughly 1.5TB. we’re anticipating more load therefore we’ll be 
>>> expanding cluster with additional nodes.
>>> 
>>> At this time, I’m kind of struggling to keep consistent cassandra.yaml file 
>>> on each server – at this time, I’m maintaining yaml file manually. The only 
>>> tool I’ve is splunk  which  is only to ‘monitor‘ threads.
>>> 
>>>  
>>> 
>>> Would you guy please suggest  open source tool that can help maintain the 
>>> cluster. I’ll really appreciate your reply – Thanks/Asad
>>> 
>> 

Re: Downside to running multiple nodetool repairs at the same time?

[Oskar Kjellin]

Yes I really think you should consider this Eugene. Triggering on all replicas 
at the same time will increase your latencies noticeably. 


> On 21 Apr 2017, at 22:25, Jon Haddad <jonathan.had...@gmail.com> wrote:
> 
> We (The Last Pickle) forked reaper a while ago and added support for 3.0.
> 
> https://github.com/thelastpickle/cassandra-reaper
> 
> We set up a mailing list here for Reaper specific questions: 
> https://groups.google.com/forum/#!forum/tlp-apache-cassandra-reaper-users
> 
> Jon
> 
>> On Apr 21, 2017, at 1:11 PM, eugene miretsky <eugene.miret...@gmail.com> 
>> wrote:
>> 
>> The Spotify repo (https://github.com/spotify/cassandra-reaper) seems to not 
>> be maintained anymore. I'm not sure if they even support Cassandra 3.0 
>> (https://github.com/spotify/cassandra-reaper/issues/140). 
>> 
>> Regardless, in Cassandra 3.0 repairs are
>> 1) Incremental, which means that the same SSTables will not be repaired 
>> twice. 
>> 2) Parallel, which means that when you call repair, all nodes repair at the 
>> same time. 
>> 
>> I suppose that in the worst case, calling repair from X nodes could trigger 
>> X repair processes (that will each trigger a Markel tree building on each 
>> node). But I would assume that Cassandra prevents this by making sure that 
>> there is only one repair process running per node. 
>> 
>> 
>> 
>>> On Fri, Apr 21, 2017 at 2:43 AM, Oskar Kjellin <oskar.kjel...@gmail.com> 
>>> wrote:
>>> It will create more overhead on your cluster. Consider using something like 
>>> reaper to manage.
>>> 
>>> > On 21 Apr 2017, at 00:57, eugene miretsky <eugene.miret...@gmail.com> 
>>> > wrote:
>>> >
>>> > In Cassandra 3.0 the default nodetool repair behaviour is incremental and 
>>> > parallel.
>>> > Is there a downside to triggering repair from multiple nodes at the same 
>>> > time?
>>> >
>>> > Basically, instead of scheduling a cron job on one node to run repair, I 
>>> > want to schedule the job on every node (this way, I don't have to worry 
>>> > about repair if the one node goes down). Alternatively, I could build a 
>>> > smarter solution for HA repair jobs, but that seems like an overkill.
>> 
> 

Will query on PK read entire partition?

[Oskar Kjellin]

If I have a table like this:

PRIMARY KEY ((userid),deviceid)

And I query
SELECT * FROM devices where userid= ? and deviceid = ?

Will cassandra read the entire partition for the userid? So if I lots of
tombstones for userid, will they get scanned?

I guess this depends on how the bloomfilter is working. Does it contain
partitioning key or primary key?

We're using 2.0.17 if it matters.

/Oskar

Re: Downside to running multiple nodetool repairs at the same time?

[Oskar Kjellin]

It will create more overhead on your cluster. Consider using something like 
reaper to manage. 

> On 21 Apr 2017, at 00:57, eugene miretsky  wrote:
> 
> In Cassandra 3.0 the default nodetool repair behaviour is incremental and 
> parallel. 
> Is there a downside to triggering repair from multiple nodes at the same 
> time? 
> 
> Basically, instead of scheduling a cron job on one node to run repair, I want 
> to schedule the job on every node (this way, I don't have to worry about 
> repair if the one node goes down). Alternatively, I could build a smarter 
> solution for HA repair jobs, but that seems like an overkill.

Re: Node always dieing

[Oskar Kjellin]

You changed to 6 nodes because you were running out of disk? But you still 
replicate 100% to all so you don't gain anything 



> On 10 Apr 2017, at 13:48, Cogumelos Maravilha  
> wrote:
> 
> No.
> 
> nodetool status, nodetool describecluster also nodetool ring shows a correct 
> cluster.
> 
> Not all nodes needs to be a seed, but can be.
> 
> I had also ALTER KEYSPACE system_auth WITH REPLICATION = {   'class' : 
> 'SimpleStrategy', 'replication_factor' : 6 } AND durable_writes = false;
> 
> And the first command on the new node was  nodetool repair system_auth
> 
> 
>> On 04/10/2017 12:37 PM, Chris Mawata wrote:
>> Notice
>> .SimpleSeedProvider{seeds=10.100.100.19, 10.100.100.85, 10.100.100.185, 
>> 10.100.100.161, 10.100.100.52, 10.100.1000.213};
>> 
>> Why do you have all six of your nodes as seeds? is it possible that the last 
>> one you added used itself as the seed and is isolated?
>> 
>> On Thu, Apr 6, 2017 at 6:48 AM, Cogumelos Maravilha 
>>  wrote:
>>> Yes C* is running as cassandra:
>>> 
>>> cassand+  2267 1 99 10:18 ?00:02:56 java 
>>> -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:Threa...
>>> 
>>> INFO  [main] 2017-04-06 10:35:42,956 Config.java:474 - Node 
>>> configuration:[allocate_tokens_for_keyspace=null; 
>>> authenticator=PasswordAuthenticator; authorizer=CassandraAuthorizer; 
>>> auto_bootstrap=true; auto_snapshot=true; back_pressure_enabled=false; 
>>> back_pressure_strategy=org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9,
>>>  factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50; 
>>> batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; 
>>> broadcast_address=null; broadcast_rpc_address=null; 
>>> buffer_pool_use_heap_if_exhausted=true; 
>>> cas_contention_timeout_in_ms=600; cdc_enabled=false; 
>>> cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; 
>>> cdc_total_space_in_mb=0; client_encryption_options=; 
>>> cluster_name=company; column_index_cache_size_in_kb=2; 
>>> column_index_size_in_kb=64; commit_failure_policy=ignore; 
>>> commitlog_compression=null; commitlog_directory=/mnt/cassandra/commitlog; 
>>> commitlog_max_compression_buffers_in_pool=3; 
>>> commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; 
>>> commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN; 
>>> commitlog_sync_period_in_ms=1; commitlog_total_space_in_mb=null;
>>>  compaction_large_partition_warning_threshold_mb=100; 
>>> compaction_throughput_mb_per_sec=16; concurrent_compactors=null; 
>>> concurrent_counter_writes=32; concurrent_materialized_view_writes=32; 
>>> concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; 
>>> counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; 
>>> counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=600; 
>>> credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; 
>>> credentials_validity_in_ms=2000; cross_node_timeout=false; 
>>> data_file_directories=[Ljava.lang.String;@223f3642; disk_access_mode=auto; 
>>> disk_failure_policy=ignore; disk_optimization_estimate_percentile=0.95; 
>>> disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; 
>>> dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; 
>>> dynamic_snitch_reset_interval_in_ms=60; 
>>> dynamic_snitch_update_interval_in_ms=100; 
>>> enable_scripted_user_defined_functions=false; 
>>> enable_user_defined_functions=false; 
>>> enable_user_defined_functions_threads=true; encryption_options=null; 
>>> endpoint_snitch=SimpleSnitch; file_cache_size_in_mb=null; 
>>> gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; 
>>> hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true; 
>>> hinted_handoff_throttle_in_kb=1024; hints_compression=null; 
>>> hints_directory=/mnt/cassandra/hints; hints_flush_period_in_ms=1; 
>>> incremental_backups=false; index_interval=null; 
>>> index_summary_capacity_in_mb=null; 
>>> index_summary_resize_interval_in_minutes=60; initial_token=null; 
>>> inter_dc_stream_throughput_outbound_megabits_per_sec=200; 
>>> inter_dc_tcp_nodelay=false; internode_authenticator=null; 
>>> internode_compression=dc; internode_recv_buff_size_in_bytes=0; 
>>> internode_send_buff_size_in_bytes=0; key_cache_keys_to_save=2147483647; 
>>> key_cache_save_period=14400; key_cache_size_in_mb=null; 
>>> listen_address=10.100.100.213; listen_interface=null; 
>>> listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; 
>>> max_hint_window_in_ms=1080; max_hints_delivery_threads=2; 
>>> max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; 
>>> max_streaming_retries=3; max_value_size_in_mb=256; 
>>> memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; 
>>> memtable_flush_writers=0; memtable_heap_space_in_mb=null; 
>>> 

Re: Cassandra data model right definition

[Oskar Kjellin]

It's not that easy as I recall this email thread 
https://groups.google.com/forum/m/#!topic/nosql-databases/ZLdgwCT_PNU

/Oskar 

> On 30 Sep 2016, at 18:40, Carlos Alonso  wrote:
> 
> Cassandra is a Wide Column Store http://db-engines.com/en/system/Cassandra
> 
> Carlos Alonso | Software Engineer | @calonso
> 
>> On 30 September 2016 at 18:24, Mehdi Bada  
>> wrote:
>> Hi all, 
>> 
>> I have a theoritical question: 
>> - Is Apache Cassandra really a column store?
>> Column store mean storing the data as column rather than as a rows. 
>> 
>> In fact C* store the data as row, and data is partionned with row key.
>> 
>> Finally, for me, Cassandra is a row oriented schema less DBMS Is it true 
>> for you also???
>> 
>> Many thanks in advance for your reply
>> 
>> Best Regards 
>> Mehdi Bada
>> 
>> 
>> Mehdi Bada | Consultant
>> Phone: +41 32 422 96 00 | Mobile: +41 79 928 75 48 | Fax: +41 32 422 96 15
>> dbi services, Rue de la Jeunesse 2, CH-2800 Delémont
>> mehdi.b...@dbi-services.com
>> www.dbi-services.com
>> 
>> 
>> 
>> 
>> ⇒ dbi services is recruiting Oracle & SQL Server experts ! – Join the team
> 

Re: How to obtain partition size

[Oskar Kjellin]

How about this tool?

https://github.com/instaclustr/cassandra-sstable-tools

> On 13 Mar 2017, at 17:56, Artur R  wrote:
> 
> Hello!
> 
> I can't find where C* stores information about partitions size (if stores it 
> at all).
> So, the questions;
> 
>  1. How to obtain the size (in rows or in bytes - doesn't matter) of some 
> particular partition?
> I know that there is system.size_estimates table with mean_partition_size, 
> but it's only mean size among all partitions.
> 
>  2. How to obtain the size of entire table?
> Again, does "mean_partition_size * partitions_count" (fields from 
> system.size_estimates table) == real size of the table?
> 
>  3. Is it possible to obtain size of rows by some clustering key within some 
> partition?
> 
> 
> Maybe one can obtain these information using Java driver or from C* system 
> tables?
> 
> 
> 

Re: Resources for fire drills

[Oskar Kjellin]

Throttle your compaction so low that it practically stops and then try so save 
the nodes to simulate not keeping up with compaction 

Sent from my iPhone

> On 1 Mar 2017, at 14:35, Stefan Podkowinski  wrote:
> 
> I've just created a page for this topic that we can use to collect some
> content:
> https://github.com/spodkowinski/cassandra-collab/blob/docs_firedrill/doc/source/operating/failure_scenarios.rst
> 
> I've invited both of you Malte and Benjamin as collaborators in github,
> so you can either push changes or use the github editor for changes.
> 
> Let me know if that would work for you.
> 
>> On 01.03.2017 13:56, benjamin roth wrote:
>> @Doc:
>> http://cassandra.apache.org/doc/latest/ is built from the git repo. So
>> you can add documentation in doc/source and submit a patch.
>> I personally think that is not the very best place or way to build a
>> knowledge DB but thats what we have.
>> 
>> 
>> 2017-03-01 13:39 GMT+01:00 Malte Pickhan > >:
>> 
>>Hi,
>> 
>>really cool that this discussion gets attention.
>> 
>>You are right my question was quite open.
>> 
>>For me it would already be helpful to compile a list like Ben
>>started with scenarios that can happen to a cluster
>>and what actions/strategies you have to take to resolve the incident
>>without loosing data and having a healthy cluster.
>> 
>>Ideally we would add some kind of rating of hard the scenario is to
>>be resolved so that teams can go through a kind of learning curve.
>> 
>>For the beginning I think it would already be sufficient to document
>>the steps how you can get a cluster into the situation which has
>>been described
>>in the scenario.
>> 
>>Hope it’s a bit clearer now what I mean.
>> 
>>Is there some kind of community space where we could start a
>>document for this purpose?
>> 
>>Best,
>> 
>>Malte
>> 
>>> On 1 Mar 2017, at 13:33, Stefan Podkowinski >> wrote:
>>> 
>>> I've been thinking about this for a while, but haven't found a
>>practical
>>> solution yet, although the term "fire drill" leaves a lot of room for
>>> interpretation. The most basic requirements I'd have for these kind of
>>> trainings would start with automated cluster provisioning for each
>>> scenario (either for teams or individuals) and provisioning of
>>test data
>>> for the cluster, with optionally some kind of load generator
>>constantly
>>> running in the background. I started to work on some Ansible scripts
>>> that would do that on AWS a couple of months ago, but it turned out to
>>> be a lot of work with all the details you have to take care of. So I'd
>>> be happy to hear about any existing resources on that as well!
>>> 
>>> 
 On 01.03.2017 10:59, Malte Pickhan wrote:
 Hi Cassandra users,
 
 I am looking for some resources/guides for firedrill scenarios
>>with apache cassandra.
 
 Do you know anything like that?
 
 Best,
 
 Malte
 
>> 
>> 

Re: Is periodic manual repair necessary?

[Oskar Kjellin]

Are you running multi dc?

Skickat från min iPad

> 27 feb. 2017 kl. 16:08 skrev Thakrar, Jayesh :
> 
> Suppose I have an application, where there are no deletes, only 5-10% of rows 
> being occasionally updated (and that too only once) and a lot of reads.
>  
> Furthermore, I have replication = 3 and both read and write are configured 
> for local_quorum.
>  
> Occasionally, servers do go into maintenance.
>  
> I understand when the maintenance is longer than the period for 
> hinted_handoffs to be preserved, they are lost and servers may have stale 
> data.
> But I do expect it to be rectified on reads. If the stale data is not read 
> again, I don’t care for it to be corrected as then the data will be 
> automatically purged because of TTL.
>  
> In such a situation, do I need to have a periodic (weekly?) manual/batch 
> read_repair process?
>  
> Thanks,
> Jayesh Thakrar

Cassandra blob vs base64 text

[Oskar Kjellin]

We currently have some cases where we store base64 as a text field instead
of a blob (running version 2.0.17).
I would like to move these to blob but wondering what benefits and
optimizations there are? The possible ones I can think of is (but there's
probably more):

* blob is stored as off heap ByteBuffers?
* blob won't be decompressed server side?

Are there any other reasons to switch to blobs? Or are we not going to see
any difference?

Thanks!

Re: inconsistent results

[Oskar Kjellin]

Repair might help. But you will end up in this situation again unless you 
read/write using quorum (may be local)

Sent from my iPhone

> On 14 Feb 2017, at 22:37, Josh England <j...@tgsmc.com> wrote:
> 
> All client interactions are from python (python-driver 3.7.1) using default 
> consistency (LOCAL_ONE I think).  Should I try repairing all nodes to make 
> sure all data is consistent?
> 
> -JE
> 
> 
>> On Tue, Feb 14, 2017 at 1:32 PM, Oskar Kjellin <oskar.kjel...@gmail.com> 
>> wrote:
>> What consistency levels are you using for reads/writes?
>> 
>> Sent from my iPhone
>> 
>> > On 14 Feb 2017, at 22:27, Josh England <j...@tgsmc.com> wrote:
>> >
>> > I'm running Cassandra 3.9 on CentOS 6.7 in a 6-node cluster.  I've got a 
>> > situation where the same query sometimes returns 2 records (correct), and 
>> > sometimes only returns 1 record (incorrect).  I've ruled out the 
>> > application and the indexing since this is reproducible directly from a 
>> > cqlsh shell with a simple select statement.  What is the best way to debug 
>> > what is happening here?
>> >
>> > -JE
>> >
> 

Re: inconsistent results

[Oskar Kjellin]

What consistency levels are you using for reads/writes?

Sent from my iPhone

> On 14 Feb 2017, at 22:27, Josh England  wrote:
> 
> I'm running Cassandra 3.9 on CentOS 6.7 in a 6-node cluster.  I've got a 
> situation where the same query sometimes returns 2 records (correct), and 
> sometimes only returns 1 record (incorrect).  I've ruled out the application 
> and the indexing since this is reproducible directly from a cqlsh shell with 
> a simple select statement.  What is the best way to debug what is happening 
> here?
> 
> -JE
> 

Re: Incremental repair for the first time

[Oskar Kjellin]

There is no harm in running it tho. If it's not needed it will simply 
terminate. Better to be safe

Sent from my iPhone

> On 9 Jan 2017, at 08:13, Amit Singh F  wrote:
> 
> Hi ,
>  
> Generally Upgradesstables are only recommended when you plan to move with 
> Major version like  from 2.0 to 2.1  or from 2.1 to 2.2 etc. Since you are 
> doing minor version upgrade no need to run upgradesstables utility.
>  
> Link by Datastax might be helpful to you :
>  
> https://support.datastax.com/hc/en-us/articles/208040036-Nodetool-upgradesstables-FAQ
>  
> From: Kathiresan S [mailto:kathiresanselva...@gmail.com] 
> Sent: Wednesday, January 04, 2017 12:22 AM
> To: user@cassandra.apache.org
> Subject: Re: Incremental repair for the first time
>  
> Thank you!
>  
> We are planning to upgrade to 3.0.10 for this issue.
>  
> From the NEWS txt file 
> (https://github.com/apache/cassandra/blob/trunk/NEWS.txt), it looks like 
> there is no need for sstableupgrade when we upgrade from 3.0.4 to 3.0.10 
> (i.e. Just installing 3.0.10 Cassandra would suffice and it will work with 
> the sstables created by 3.0.4 ?)
>  
> Could you please confirm (if i'm reading the upgrade instructions correctly)?
>  
> Thanks,
> Kathir
>  
> On Tue, Dec 20, 2016 at 5:28 PM, kurt Greaves  wrote:
> No workarounds, your best/only option is to upgrade (plus you get the benefit 
> of loads of other bug fixes).
>  
> On 16 December 2016 at 21:58, Kathiresan S  
> wrote:
> Thank you!
>  
> Is any work around available for this version? 
>  
> Thanks,
> Kathir
> 
> 
> On Friday, December 16, 2016, Jake Luciani  wrote:
> This was fixed post 3.0.4 please upgrade to latest 3.0 release
>  
> On Fri, Dec 16, 2016 at 4:49 PM, Kathiresan S  
> wrote:
> Hi,
>  
> We have a brand new Cassandra cluster (version 3.0.4) and we set up nodetool 
> repair scheduled for every day (without any options for repair). As per 
> documentation, incremental repair is the default in this case. 
> Should we do a full repair for the very first time on each node once and then 
> leave it to do incremental repair afterwards?
>  
> Problem we are facing:
>  
> On a random node, the repair process throws validation failed error, pointing 
> to some other node
>  
> For Eg. Node A, where the repair is run (without any option), throws below 
> error
>  
> Validation failed in /Node B
>  
> In Node B when we check the logs, below exception is seen at the same exact 
> time...
>  
> java.lang.RuntimeException: Cannot start multiple repair sessions over the 
> same sstables
> at 
> org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1087)
>  ~[apache-cassandra-3.0.4.jar:3.0.4]
> at 
> org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:80)
>  ~[apache-cassandra-3.0.4.jar:3.0.4]
> at 
> org.apache.cassandra.db.compaction.CompactionManager$10.call(CompactionManager.java:700)
>  ~[apache-cassandra-3.0.4.jar:3.0.4]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> ~[na:1.8.0_73]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  ~[na:1.8.0_73]
>  
> Can you please help on how this can be fixed?
>  
> Thanks,
> Kathir
> 
> 
> 
> --
> http://twitter.com/tjake
>  
>  

Re: Read efficiency question

[Oskar Kjellin]

Yes sorry I missed the double parenthesis in the first case. 

I may be a bit off here, but I don't think the coordinator pinpoints the row 
but just the node it needs to go to. 
It's more a case of creating smaller partitions, which makes for more even load 
among the cluster and the node will not have to read a whole lot of data into 
memory to just GC later on. 

If you think of Cassandra as a hash map (which it kind of is). You like the key 
to be as unique as possible to not have to go to a bucket and filter there, or 
create hot spots. 

Sent from my iPhone

> On 27 Dec 2016, at 17:12, Voytek Jarnot <voytek.jar...@gmail.com> wrote:
> 
> Thank you Oskar.  I think you may be missing the double parentheses in the 
> first example - difference is between partition key of (key1, key2, key3) and 
> (key1, key2).  With that in mind, I believe your answer would be that the 
> first example is more efficient?
> 
> Is this essentially a case of the coordinator node being able to exactly 
> pinpoint a row (first example) vs the coordinator node pinpointing the 
> partition and letting the partition-owning node refine down to the right row 
> using the clustering key (key3 in the second example)?
> 
>> On Tue, Dec 27, 2016 at 10:06 AM, Oskar Kjellin <oskar.kjel...@gmail.com> 
>> wrote:
>> The second one will be the most efficient.
>> How much depends on how unique key1 is.
>> 
>> In the first case everything for the same key1 will be on the same 
>> partition.  If it's not unique at all that will be very bad.
>> In the second case the combo of key1 and key2 will decide what partition.
>> 
>> If you don't ever have to find all key2 for a given key1 I don't see any 
>> reason to do case 1
>> 
>> 
>> > On 27 Dec 2016, at 16:42, Voytek Jarnot <voytek.jar...@gmail.com> wrote:
>> >
>> > Wondering if there's a difference when querying by primary key between the 
>> > two definitions below:
>> >
>> > primary key ((key1, key2, key3))
>> > primary key ((key1, key2), key3)
>> >
>> > In terms of read speed/efficiency... I don't have much of a reason 
>> > otherwise to prefer one setup over the other, so would prefer the most 
>> > efficient for querying.
>> >
>> > Thanks.
> 

Re: Read efficiency question

[Oskar Kjellin]

The second one will be the most efficient. 
How much depends on how unique key1 is. 

In the first case everything for the same key1 will be on the same partition.  
If it's not unique at all that will be very bad. 
In the second case the combo of key1 and key2 will decide what partition. 

If you don't ever have to find all key2 for a given key1 I don't see any reason 
to do case 1


> On 27 Dec 2016, at 16:42, Voytek Jarnot  wrote:
> 
> Wondering if there's a difference when querying by primary key between the 
> two definitions below:
> 
> primary key ((key1, key2, key3))
> primary key ((key1, key2), key3)
> 
> In terms of read speed/efficiency... I don't have much of a reason otherwise 
> to prefer one setup over the other, so would prefer the most efficient for 
> querying.
> 
> Thanks.

Re: Storing videos in cassandra

[Oskar Kjellin]

The actual video is not stored in Cassandra. You need to use a proper origin 
like s3. 

Although you can probably store it in Cassandra, it's not a good idea. 

Sent from my iPhone

> On 14 nov. 2016, at 18:02, raghavendra vutti  
> wrote:
> 
> Hi,
> 
>  Just wanted to know How does hulu or netflix store videos in cassandra.
> 
> Do they just use references to the video files in the form of URL's and store 
> in the DB??
> 
> could someone please me on this.
> 
> 
> Thanks,
> Raghavendra.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 

Re: Consistency when adding data to collections concurrently?

[Oskar Kjellin]

Could you not send both labels in one request? Race conditions should still be 
handled as Vladimir suggests. But in this specific case the client could send 
both as 1 request thus simplifying the solution. 

/Oskar

> On 12 nov. 2016, at 12:05, Vladimir Yudovin  wrote:
> 
> Hi Ali,
> 
> >What can I do so I end up with [1, 2] instead of either [1] or [2] after 
> >both requests have been processed?
> Use UPDATE, not INSERT. Thus new labels will be added to list, without 
> overwriting old ones. Also consider usage of SET instead of LIST to avoid 
> duplicates.  
> 
> Best regards, Vladimir Yudovin, 
> Winguzone - Hosted Cloud Cassandra
> Launch your cluster in minutes.
> 
> 
>  On Sat, 12 Nov 2016 05:34:24 -0500Ali Akhtar  
> wrote 
> 
> I have a table where each record contains a list of labels.
> 
> I have an endpoint which responds to new labels being added to a record by 
> the user.
> 
> Consider the following scenario:
> 
> - Record X, labels = []
> - User selects 2 labels, clicks a button, and 2 http requests are generated.
> - The server receives request for Label 1 and Label 2 at the same time.
> - Both requests see the labels as empty, add 1 label to the collection, and 
> send it.
> - Record state as label 1 request sees it: [1], as label 2 sees it: [2]
> 
> How will the above conflict be resolved? What can I do so I end up with [1, 
> 2] instead of either [1] or [2] after both requests have been processed?
> 

Re: How to confirm TWCS is fully in-place

[Oskar Kjellin]

Hi,

You could manually trigger it with nodetool compact. 

/Oskar 

> On 8 nov. 2016, at 21:47, Lahiru Gamathige  wrote:
> 
> Hi Users,
> 
> I am thinking of migrating our timeseries tables to use TWCS. I am using JMX 
> to set the new compaction and one node at a time and I am not sure how to 
> confirm that after the flush all the compaction is done in each node. I tried 
> this in a small cluster but after setting the compaction I didn't see any 
> compaction triggering  and ran nodetool flush and still didn't see a 
> compaction triggering.
> 
> Now I am about to do the same thing in our staging cluster, so curious how do 
> I confirm compaction ran in each node before I change the table schema 
> because I am worried it will start the compaction in all the nodes at the 
> same time.
> 
> Lahiru

Re: Cassandra on Cloud platforms experience

[Oskar Kjellin]

It is good enough if you do not care about automation.
If you care about automation (and you really should), GossipPropertyFileSnitch
is not good enough as it has to be manually updated. Same with availability
set, fault domain isn't available for the instance as metadata as it is on
AWS.

You still have a global limit for the storage account IOPS. So even tho you
might get thruput with a few servers, adding more means copying the VHD to
multiple storage accounts, making automation once again cumbersome.

2016-11-04 11:29 GMT+01:00 Vladimir Yudovin <vla...@winguzone.com>:

> Hi,
>
> >1. No native snitch
> It's not great problem. GossipPropertyFileSnitch is good enough.
>
> >2. No concept of availability zones.
> Azure does have such concept - Availability Set.
> <https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-windows-infrastructure-availability-sets-guidelines/>
> It provides three fault domain (availability zone in Amazon terms) and 20
> updates domains.
>
> >4. Even running SSDs will give you poor performance.
> It depends on disk size. 1T SSD provides 5000 IOPS.
>
>
> So in short:
> Amazon - provides data at rest encryption, flexible EBS storage (or local
> disks), availability zones.
> Azure - provides data at rest encryption, less flexible storage (or local
> disks), availability zones.
> SoftLayer - no data encryption, but they have unique feature -
> connectivity between different data centers (they call this VLAN spanning)
> without need in VPN or other tunneling. They don't have explicit AV zones,
> but you can put nodes in different DC in the same region (some locations)
> with relative low latency 1-1.5 ms. or purchase another VLAN in different
> pod for $25 per month in the same DC.
>
> We provide Cassandra cluster on all provider in many worldwide locations.
>
> Best regards, Vladimir Yudovin,
>
> *Winguzone <https://winguzone.com?from=list> - Hosted Cloud
> CassandraLaunch your cluster in minutes.*
>
>
>  On Fri, 04 Nov 2016 05:10:06 -0400*Oskar Kjellin
> <oskar.kjel...@gmail.com <oskar.kjel...@gmail.com>>* wrote 
>
> So I've run Cassandra on both Aws and azure. I would strongly suggest that
> if you have the option, run as far away from azure as you can.
>
> Here's a list of issues I have running Cassandra on azure:
> 1. No native snitch
> 2. No concept of availability zones. This makes it impossible for
> Cassandra to put replicas in different AZs. This will hurt your uptime and
> might incur loss of data. (They have something called a fault domain tho)
> 3. The disks have iops that land in the floppy disk range
> 4. Even running SSDs will give you poor performance.
> 5. Beware of the global storage account limit. This makes scaling out hurt
> performance if you put them on the same storage account. Which if your
> using images is your only choice.
>
> Sent from my iPhone
>
> > On 4 nov. 2016, at 00:22, cass savy <casss...@gmail.com> wrote:
> >
> > I would like to hear from the community on their experiences or lesson
> learnt on hosting Cassandra in cloud platforms like
> >
> > 1. Google Cloud Platform
> > 2. AWS
> > 3. Azure
> >
> > 1. Which cloud hosting is better and Why?
> > 2. What differences of C* over vendor provided NoSQL DB like (Bigtable,
> Dynamo,Azure Document DB)
> > 3. AWS is more mature in his offerings and Azure is getting there or its
> there already based on what I have been investigating so far?
> >
> > 4. What is drive to pick one vs another -Is it cost, infrastructure,
> hardware SKU, availability, scalability, performance,ease of deployment and
> maintenance,..etc?
> >
> > Please let me know your thoughts and suggestions if somebody has done a
> deep dive into these 3 cloud platforms for C*.
> >
> >
> > We use datastax cassandra and exploring new usecases in AWS and also
> evaluating or POC it in Azure/GCP
> >
>
>
>

Re: Cassandra on Cloud platforms experience

[Oskar Kjellin]

Also, if you're running it on azure you should have a look at Netflix Priam. 
That really helped us automate stuff. 

Dynamo db works well if you don't want to spend hours in running your cluster. 
If I were to have the option to run on AWS I would use Dynamo when possible. 

Sent from my iPhone

> On 4 nov. 2016, at 10:10, Oskar Kjellin <oskar.kjel...@gmail.com> wrote:
> 
> So I've run Cassandra on both Aws and azure. I would strongly suggest that if 
> you have the option, run as far away from azure as you can. 
> 
> Here's a list of issues I have running Cassandra on azure:
> 1. No native snitch 
> 2. No concept of availability zones. This makes it impossible for Cassandra 
> to put replicas in different AZs. This will hurt your uptime and might incur 
> loss of data. (They have something called a fault domain tho)
> 3. The disks have iops that land in the floppy disk range 
> 4. Even running SSDs will give you poor performance.
> 5. Beware of the global storage account limit. This makes scaling out hurt 
> performance if you put them on the same storage account. Which if your using 
> images is your only choice. 
> 
> Sent from my iPhone
> 
>> On 4 nov. 2016, at 00:22, cass savy <casss...@gmail.com> wrote:
>> 
>> I would like to hear from the community on their experiences or lesson 
>> learnt on hosting Cassandra in cloud platforms like
>> 
>> 1. Google Cloud Platform
>> 2. AWS
>> 3. Azure
>> 
>> 1.  Which cloud hosting is better and Why?
>> 2.  What differences of C* over vendor provided NoSQL DB like (Bigtable, 
>> Dynamo,Azure Document DB)
>> 3. AWS is more mature in his offerings and Azure is getting there or its 
>> there already based on what I have been investigating so far?
>> 
>> 4. What is drive to pick one vs another -Is it cost, infrastructure, 
>> hardware SKU, availability, scalability, performance,ease of deployment and 
>> maintenance,..etc?
>> 
>> Please let me know your thoughts and suggestions if somebody has done a deep 
>> dive into these 3 cloud platforms for C*.
>> 
>> 
>> We use datastax cassandra and exploring new usecases in AWS and also 
>> evaluating  or POC it in Azure/GCP
>> 

Re: Cassandra on Cloud platforms experience

[Oskar Kjellin]

So I've run Cassandra on both Aws and azure. I would strongly suggest that if 
you have the option, run as far away from azure as you can. 

Here's a list of issues I have running Cassandra on azure:
1. No native snitch 
2. No concept of availability zones. This makes it impossible for Cassandra to 
put replicas in different AZs. This will hurt your uptime and might incur loss 
of data. (They have something called a fault domain tho)
3. The disks have iops that land in the floppy disk range 
4. Even running SSDs will give you poor performance.
5. Beware of the global storage account limit. This makes scaling out hurt 
performance if you put them on the same storage account. Which if your using 
images is your only choice. 

Sent from my iPhone

> On 4 nov. 2016, at 00:22, cass savy  wrote:
> 
> I would like to hear from the community on their experiences or lesson learnt 
> on hosting Cassandra in cloud platforms like
> 
> 1. Google Cloud Platform
> 2. AWS
> 3. Azure
> 
> 1.  Which cloud hosting is better and Why?
> 2.  What differences of C* over vendor provided NoSQL DB like (Bigtable, 
> Dynamo,Azure Document DB)
> 3. AWS is more mature in his offerings and Azure is getting there or its 
> there already based on what I have been investigating so far?
> 
> 4. What is drive to pick one vs another -Is it cost, infrastructure, hardware 
> SKU, availability, scalability, performance,ease of deployment and 
> maintenance,..etc?
> 
> Please let me know your thoughts and suggestions if somebody has done a deep 
> dive into these 3 cloud platforms for C*.
> 
> 
> We use datastax cassandra and exploring new usecases in AWS and also 
> evaluating  or POC it in Azure/GCP
> 

Re: duplicate values for primary key

[Oskar Kjellin]

Hi,

Ran into the same issue when going to 3.5. Completely killed our cluster. Only 
way was to restore a backup.

/Oskar

> On 2 aug. 2016, at 07:54, Julien Anguenot  wrote:
> 
> Hey Jesse, 
> 
> You might wanna check and comment against that issue:
> 
>   https://issues.apache.org/jira/browse/CASSANDRA-11887
> 
>   J. 
> 
> --
> Julien Anguenot (@anguenot)
> 
>> On Aug 2, 2016, at 3:16 AM, Jesse Hodges  wrote:
>> 
>> Hi, I've got a bit of a conundrum. Recently I upgraded from 2.2.3 to 3.7.0 
>> (ddc distribution). Following the upgrade (though this condition may have 
>> existed prior to upgrade)..
>> 
>> I have a table with a simple partition key and multiple clustering keys, and 
>> I have duplicates of many of the primary keys!
>> 
>> I've tried various repair options and lots of searching, but nothing's 
>> really helping. I'm unsure of how to troubleshoot further or potentially 
>> consolidate these keys. This seems like a bug, but hopefully it's something 
>> simple I missed. I'm also willing to troubleshoot further as needed, but I 
>> could use a few getting started pointers.
>> 
>> Example output: primary key is 
>> (partition_id,alarm_id,tenant_id,account_id,source,metric)
>> 
>> 
>> cqlsh:alarms> select * from alarms.last_seen_state  where partition_id=10 
>> and alarm_id='59893';
>> 
>>  partition_id | alarm_id | tenant_id| account_id 
>> | source | metric | last_seen   | value
>> --+--+--++++-+---
>>10 |59893 | f50f8413-57bb-4eb5-a37c-7482a63ea9a5 |  10303 
>> | PORTAL |CPU | 2016-08-01 15:27:37.00+ | 1
>>10 |59893 | f50f8413-57bb-4eb5-a37c-7482a63ea9a5 |  10303 
>> | PORTAL |CPU | 2016-08-01 15:07:15.00+ | 1
>> 
>> Thanks, Jesse
> 

Re: Upgrade from 3.0.6 to 3.7.

[Oskar Kjellin]

Hi,

If I were you I would stick on 3.0.x. I haven't experienced 3.x to be 
production ready. We went to prod with 3.5 and I wish we hadn't. 

/Oskar


> On 23 juni 2016, at 10:56, Bienek, Marcin  wrote:
> 
> Hi, 
> 
> We are short before going in prod with our cassandra cluster. Now I wonder if 
> this maybe  (while still not fully in prod) a good moment to switch from the 
> 3.0.x to the new tick-tock versions.
> On planet cassandra the tick-tock article mentions: 
> 
> “…We do recognize that it will take some time for tick-tock releases to 
> deliver production-level stability, which is why we will continue to deliver 
> 2.2.y and 3.0.y bugfix releases.  (But if we do demonstrate that tick-tock 
> can deliver the stability we want, there will be no need for a 4.0.y bugfix 
> series, only 4.x tick-tock.)…" 
> 
> But the article is from June 2015 so I guess it might not be that valid 
> anymore. What are your experiences with the newer versions ?  
> Is there anything to keep in mind before the upgrade from 3.0.x ?  
> 
> BR, 
> Bieniu 

Re: Cluster not working after upgrade from 2.1.12 to 3.5.0

[Oskar Kjellin]

Did you see similar issues when querying using a driver? Because we get no 
results in the driver what so ever 

Sent from my iPhone

> On 21 juni 2016, at 18:50, Julien Anguenot <jul...@anguenot.org> wrote:
> 
> See my comments on the issue: I had to truncate and reinsert data in
> these corrupted tables.
> 
> AFAIK, there is no evidence that UDTs are responsible of this bad behavior.
> 
>> On Tue, Jun 21, 2016 at 11:45 AM, Oskar Kjellin <oskar.kjel...@gmail.com> 
>> wrote:
>> Yea I saw that one. We're not using UDT in the affected tables tho.
>> 
>> Did you resolve it?
>> 
>> Sent from my iPhone
>> 
>>> On 21 juni 2016, at 18:27, Julien Anguenot <jul...@anguenot.org> wrote:
>>> 
>>> I have experienced similar duplicate primary keys behavior with couple
>>> of tables after upgrading from 2.2.x to 3.0.x.
>>> 
>>> See comments on the Jira issue I opened at the time over there:
>>> https://issues.apache.org/jira/browse/CASSANDRA-11887
>>> 
>>> 
>>>> On Tue, Jun 21, 2016 at 10:47 AM, Oskar Kjellin <oskar.kjel...@gmail.com> 
>>>> wrote:
>>>> Hi,
>>>> 
>>>> We've done this upgrade in both dev and stage before and we did not see
>>>> similar issues.
>>>> After upgrading production today we have a lot issues tho.
>>>> 
>>>> The main issue is that the Datastax client quite often does not get the 
>>>> data
>>>> (even though it's the same query). I see similar flakyness by simply 
>>>> running
>>>> cqlsh, although it does return it returns broken data.
>>>> 
>>>> We are running a 3 node cluster with RF 3.
>>>> 
>>>> I have this table
>>>> 
>>>> CREATE TABLE keyspace.table (
>>>> 
>>>> a text,
>>>> 
>>>>   b text,
>>>> 
>>>>   c text,
>>>> 
>>>>   d list,
>>>> 
>>>>   e text,
>>>> 
>>>>   f timestamp,
>>>> 
>>>>   g list,
>>>> 
>>>>   h timestamp,
>>>> 
>>>>   PRIMARY KEY (a, b, c)
>>>> 
>>>> )
>>>> 
>>>> 
>>>> Every other time I query (not exactly every other time, but random) I get:
>>>> 
>>>> 
>>>> SELECT * from table where a = 'xxx' and b = 'xxx'
>>>> 
>>>> a | b | c | d | e | f
>>>> | g| h
>>>> 
>>>> -+--+---+--++-+---+-
>>>> 
>>>> xxx |  xxx | ccc | null |   null | 2089-11-30
>>>> 23:00:00.00+ | ['fff'] | 2014-12-31 23:00:00.00+
>>>> 
>>>> xxx |  xxx |   ddd | null |
>>>> null | 2099-01-01 00:00:00.00+ | ['fff'] | 2016-06-17
>>>> 13:29:36.00+
>>>> 
>>>> 
>>>> Which is the expected output.
>>>> 
>>>> 
>>>> But I also get:
>>>> 
>>>> a | b | c | d | e | f
>>>> | g| h
>>>> 
>>>> -+--+---+--++-+---+-
>>>> 
>>>> xxx |  xxx | ccc | null |   null |
>>>> null |  null |null
>>>> 
>>>> xxx |  xxx | ccc | null |   null | 2089-11-30
>>>> 23:00:00.00+ | ['fff'] |null
>>>> 
>>>> xxx |  xxx | ccc | null |   null |
>>>> null |  null | 2014-12-31 23:00:00.00+
>>>> 
>>>> xxx |  xxx |   ddd | null |
>>>> null |null |  null |
>>>> null
>>>> 
>>>> xxx |  xxx |   ddd | null |
>>>> null | 2099-01-01 00:00:00.00+ | ['fff'] |
>>>> null
>>>> 
>>>> xxx | 

Re: Cluster not working after upgrade from 2.1.12 to 3.5.0

[Oskar Kjellin]

Hmm, no way we can do that in prod :/

Sent from my iPhone

> On 21 juni 2016, at 18:50, Julien Anguenot <jul...@anguenot.org> wrote:
> 
> See my comments on the issue: I had to truncate and reinsert data in
> these corrupted tables.
> 
> AFAIK, there is no evidence that UDTs are responsible of this bad behavior.
> 
>> On Tue, Jun 21, 2016 at 11:45 AM, Oskar Kjellin <oskar.kjel...@gmail.com> 
>> wrote:
>> Yea I saw that one. We're not using UDT in the affected tables tho.
>> 
>> Did you resolve it?
>> 
>> Sent from my iPhone
>> 
>>> On 21 juni 2016, at 18:27, Julien Anguenot <jul...@anguenot.org> wrote:
>>> 
>>> I have experienced similar duplicate primary keys behavior with couple
>>> of tables after upgrading from 2.2.x to 3.0.x.
>>> 
>>> See comments on the Jira issue I opened at the time over there:
>>> https://issues.apache.org/jira/browse/CASSANDRA-11887
>>> 
>>> 
>>>> On Tue, Jun 21, 2016 at 10:47 AM, Oskar Kjellin <oskar.kjel...@gmail.com> 
>>>> wrote:
>>>> Hi,
>>>> 
>>>> We've done this upgrade in both dev and stage before and we did not see
>>>> similar issues.
>>>> After upgrading production today we have a lot issues tho.
>>>> 
>>>> The main issue is that the Datastax client quite often does not get the 
>>>> data
>>>> (even though it's the same query). I see similar flakyness by simply 
>>>> running
>>>> cqlsh, although it does return it returns broken data.
>>>> 
>>>> We are running a 3 node cluster with RF 3.
>>>> 
>>>> I have this table
>>>> 
>>>> CREATE TABLE keyspace.table (
>>>> 
>>>> a text,
>>>> 
>>>>   b text,
>>>> 
>>>>   c text,
>>>> 
>>>>   d list,
>>>> 
>>>>   e text,
>>>> 
>>>>   f timestamp,
>>>> 
>>>>   g list,
>>>> 
>>>>   h timestamp,
>>>> 
>>>>   PRIMARY KEY (a, b, c)
>>>> 
>>>> )
>>>> 
>>>> 
>>>> Every other time I query (not exactly every other time, but random) I get:
>>>> 
>>>> 
>>>> SELECT * from table where a = 'xxx' and b = 'xxx'
>>>> 
>>>> a | b | c | d | e | f
>>>> | g| h
>>>> 
>>>> -+--+---+--++-+---+-
>>>> 
>>>> xxx |  xxx | ccc | null |   null | 2089-11-30
>>>> 23:00:00.00+ | ['fff'] | 2014-12-31 23:00:00.00+
>>>> 
>>>> xxx |  xxx |   ddd | null |
>>>> null | 2099-01-01 00:00:00.00+ | ['fff'] | 2016-06-17
>>>> 13:29:36.00+
>>>> 
>>>> 
>>>> Which is the expected output.
>>>> 
>>>> 
>>>> But I also get:
>>>> 
>>>> a | b | c | d | e | f
>>>> | g| h
>>>> 
>>>> -+--+---+--++-+---+-
>>>> 
>>>> xxx |  xxx | ccc | null |   null |
>>>> null |  null |null
>>>> 
>>>> xxx |  xxx | ccc | null |   null | 2089-11-30
>>>> 23:00:00.00+ | ['fff'] |null
>>>> 
>>>> xxx |  xxx | ccc | null |   null |
>>>> null |  null | 2014-12-31 23:00:00.00+
>>>> 
>>>> xxx |  xxx |   ddd | null |
>>>> null |null |  null |
>>>> null
>>>> 
>>>> xxx |  xxx |   ddd | null |
>>>> null | 2099-01-01 00:00:00.00+ | ['fff'] |
>>>> null
>>>> 
>>>> xxx |  xxx |   ddd | null

Re: Cluster not working after upgrade from 2.1.12 to 3.5.0

[Oskar Kjellin]

Yea I saw that one. We're not using UDT in the affected tables tho. 

Did you resolve it?

Sent from my iPhone

> On 21 juni 2016, at 18:27, Julien Anguenot <jul...@anguenot.org> wrote:
> 
> I have experienced similar duplicate primary keys behavior with couple
> of tables after upgrading from 2.2.x to 3.0.x.
> 
> See comments on the Jira issue I opened at the time over there:
> https://issues.apache.org/jira/browse/CASSANDRA-11887
> 
> 
>> On Tue, Jun 21, 2016 at 10:47 AM, Oskar Kjellin <oskar.kjel...@gmail.com> 
>> wrote:
>> Hi,
>> 
>> We've done this upgrade in both dev and stage before and we did not see
>> similar issues.
>> After upgrading production today we have a lot issues tho.
>> 
>> The main issue is that the Datastax client quite often does not get the data
>> (even though it's the same query). I see similar flakyness by simply running
>> cqlsh, although it does return it returns broken data.
>> 
>> We are running a 3 node cluster with RF 3.
>> 
>> I have this table
>> 
>> CREATE TABLE keyspace.table (
>> 
>>  a text,
>> 
>>b text,
>> 
>>c text,
>> 
>>d list,
>> 
>>e text,
>> 
>>f timestamp,
>> 
>>g list,
>> 
>>h timestamp,
>> 
>>PRIMARY KEY (a, b, c)
>> 
>> )
>> 
>> 
>> Every other time I query (not exactly every other time, but random) I get:
>> 
>> 
>> SELECT * from table where a = 'xxx' and b = 'xxx'
>> 
>> a | b | c | d | e | f
>> | g| h
>> 
>> -+--+---+--++-+---+-
>> 
>> xxx |  xxx | ccc | null |   null | 2089-11-30
>> 23:00:00.00+ | ['fff'] | 2014-12-31 23:00:00.00+
>> 
>> xxx |  xxx |   ddd | null |
>> null | 2099-01-01 00:00:00.00+ | ['fff'] | 2016-06-17
>> 13:29:36.00+
>> 
>> 
>> Which is the expected output.
>> 
>> 
>> But I also get:
>> 
>> a | b | c | d | e | f
>> | g| h
>> 
>> -+--+---+--++-+---+-
>> 
>> xxx |  xxx | ccc | null |   null |
>> null |  null |null
>> 
>> xxx |  xxx | ccc | null |   null | 2089-11-30
>> 23:00:00.00+ | ['fff'] |null
>> 
>> xxx |  xxx | ccc | null |   null |
>> null |  null | 2014-12-31 23:00:00.00+
>> 
>> xxx |  xxx |   ddd | null |
>> null |null |  null |
>> null
>> 
>> xxx |  xxx |   ddd | null |
>> null | 2099-01-01 00:00:00.00+ | ['fff'] |
>> null
>> 
>> xxx |  xxx |   ddd | null |
>> null |null |  null | 2016-06-17
>> 13:29:36.00+
>> 
>> 
>> Notice that the same PK is returned 3 times. With different parts of the
>> data. I believe this is what's currently killing our production environment.
>> 
>> 
>> I'm running upgradesstables as of this moment, but it's not finished yet. I
>> started a repair before but nothing happened. The upgradesstables finished
>> now on 2 out of 3 nodes, but production is still down :/
>> 
>> 
>> We also see these in the logs, over and over again:
>> 
>> DEBUG [ReadRepairStage:4] 2016-06-21 15:44:01,119 ReadCallback.java:235 -
>> Digest mismatch:
>> 
>> org.apache.cassandra.service.DigestMismatchException: Mismatch for key
>> DecoratedKey(-1566729966326640413, 336b35356c49537731797a4a5f64627a797236)
>> (b3dcfcbeed6676eae7ff88cc1bd251fb vs 6e7e9225871374d68a7cdb54ae70726d)
>> 
>> at
>> org.apache.cassandra.service.DigestResolver.resolve(DigestResolver.java:85)
>> ~[apache-cassandra-3.5.0.jar:3.5.0]
>> 
>> at
>> org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:226)
>> ~[apache-cassandra-3.5.0.jar:3.5.0]
>> 
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> [na:1.8.0_72]
>> 
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [na:1.8.0_72]
>> 
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
>> 
>> 
>> Any help is much appreciated
> 
> 
> 
> -- 
> Julien Anguenot (@anguenot)

Cluster not working after upgrade from 2.1.12 to 3.5.0

[Oskar Kjellin]

Hi,

We've done this upgrade in both dev and stage before and we did not see
similar issues.
After upgrading production today we have a lot issues tho.

The main issue is that the Datastax client quite often does not get the
data (even though it's the same query). I see similar flakyness by simply
running cqlsh, although it does return it returns broken data.

We are running a 3 node cluster with RF 3.

I have this table

CREATE TABLE keyspace.table (

  a text,

b text,

c text,

d list,

e text,

f timestamp,

g list,

h timestamp,

PRIMARY KEY (a, b, c)

)


Every other time I query (not exactly every other time, but random) I get:


SELECT * from table where a = 'xxx' and b = 'xxx'

 a | b | c | d | e | f
| g| h

-+--+---+--++-+---+-

 xxx |  xxx | ccc | null |   null | 2089-11-30
23:00:00.00+ | ['fff'] | 2014-12-31 23:00:00.00+

 xxx |  xxx |   ddd | null |
null | 2099-01-01 00:00:00.00+ | ['fff'] | 2016-06-17
13:29:36.00+


Which is the expected output.


But I also get:

 a | b | c | d | e | f
| g| h

-+--+---+--++-+---+-

 xxx |  xxx | ccc | null |   null |
null |  null |null

 xxx |  xxx | ccc | null |   null | 2089-11-30
23:00:00.00+ | ['fff'] |null

 xxx |  xxx | ccc | null |   null |
null |  null | 2014-12-31 23:00:00.00+

 xxx |  xxx |   ddd | null |
null |null |  null |
null

 xxx |  xxx |   ddd | null |
null | 2099-01-01 00:00:00.00+ | ['fff'] |
  null

 xxx |  xxx |   ddd | null |
null |null |  null | 2016-06-17
13:29:36.00+


Notice that the same PK is returned 3 times. With different parts of the
data. I believe this is what's currently killing our production environment.


I'm running upgradesstables as of this moment, but it's not finished yet. I
started a repair before but nothing happened. The upgradesstables finished
now on 2 out of 3 nodes, but production is still down :/


We also see these in the logs, over and over again:

DEBUG [ReadRepairStage:4] 2016-06-21 15:44:01,119 ReadCallback.java:235 -
Digest mismatch:

org.apache.cassandra.service.DigestMismatchException: Mismatch for key
DecoratedKey(-1566729966326640413, 336b35356c49537731797a4a5f64627a797236)
(b3dcfcbeed6676eae7ff88cc1bd251fb vs 6e7e9225871374d68a7cdb54ae70726d)

at
org.apache.cassandra.service.DigestResolver.resolve(DigestResolver.java:85)
~[apache-cassandra-3.5.0.jar:3.5.0]

at
org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:226)
~[apache-cassandra-3.5.0.jar:3.5.0]

at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_72]

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_72]

at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]


Any help is much appreciated