Re: Client-side timeouts after dropping table

[Jonathan Haddad]

3.7 falls under the Tick Tock release cycle, which is almost completely
untested in production by experienced operators.  In the cases where it has
been tested, there have been numerous bugs found which I (and I think most
people on this list) consider to be show stoppers.  Additionally, the Tick
Tock release cycle puts the operator in the uncomfortable position of
having to decide between upgrading to a new version with new features
(probably new bugs) or back porting bug fixes from future versions
themselves.There will never be a 3.7.1 release which fixes bugs in 3.7
without adding new features.

https://github.com/apache/cassandra/blob/trunk/NEWS.txt

For new projects I recommend starting with the recently released 3.0.9.

Assuming the project changes it's policy on releases (all signs point to
yes), then by the time 4.0 rolls out a lot of the features which have been
released in the 3.x series will have matured a bit, so it's very possible
4.0 will stabilize faster than the usual 6 months it takes for a major
release.

All that said, there's nothing wrong with doing compatibility & smoke tests
against the latest 3.x release as well as 3.0 and reporting bugs back to
the Apache Cassandra JIRA, I'm sure it would be greatly appreciated.

https://issues.apache.org/jira/secure/Dashboard.jspa

Jon


On Tue, Sep 20, 2016 at 8:10 PM Jesse Hodges  wrote:

> Can you elaborate on why not 3.7?
>
> On Tue, Sep 20, 2016 at 7:41 PM, Jonathan Haddad 
> wrote:
>
>> If you haven't yet deployed to prod I strongly recommend *not* using 3.7.
>>
>>
>> What network storage are you using?  Outside of a handful of highly
>> experienced experts using EBS in very specific ways, it usually ends in
>> failure.
>>
>> On Tue, Sep 20, 2016 at 3:30 PM John Sanda  wrote:
>>
>>> I am deploying multiple Java web apps that connect to a Cassandra 3.7
>>> instance. Each app creates its own schema at start up. One of the schema
>>> changes involves dropping a table. I am seeing frequent client-side
>>> timeouts reported by the DataStax driver after the DROP TABLE statement is
>>> executed. I don't see this behavior in all environments. I do see it
>>> consistently in a QA environment in which Cassandra is running in docker
>>> with network storage, so writes are pretty slow from the get go. In my logs
>>> I see a lot of tables getting flushed, which I guess are all of the dirty
>>> column families in the respective commit log segment. Then I seen a whole
>>> bunch of flushes getting queued up. Can I reach a point in which too many
>>> table flushes get queued such that writes would be blocked?
>>>
>>>
>>> --
>>>
>>> - John
>>>
>>
>

Re: Client-side timeouts after dropping table

[Jesse Hodges]

Can you elaborate on why not 3.7?

On Tue, Sep 20, 2016 at 7:41 PM, Jonathan Haddad  wrote:

> If you haven't yet deployed to prod I strongly recommend *not* using 3.7.
>
> What network storage are you using?  Outside of a handful of highly
> experienced experts using EBS in very specific ways, it usually ends in
> failure.
>
> On Tue, Sep 20, 2016 at 3:30 PM John Sanda  wrote:
>
>> I am deploying multiple Java web apps that connect to a Cassandra 3.7
>> instance. Each app creates its own schema at start up. One of the schema
>> changes involves dropping a table. I am seeing frequent client-side
>> timeouts reported by the DataStax driver after the DROP TABLE statement is
>> executed. I don't see this behavior in all environments. I do see it
>> consistently in a QA environment in which Cassandra is running in docker
>> with network storage, so writes are pretty slow from the get go. In my logs
>> I see a lot of tables getting flushed, which I guess are all of the dirty
>> column families in the respective commit log segment. Then I seen a whole
>> bunch of flushes getting queued up. Can I reach a point in which too many
>> table flushes get queued such that writes would be blocked?
>>
>>
>> --
>>
>> - John
>>
>

Re: Client-side timeouts after dropping table

[John Sanda]

Thanks Nate. We do not have monitoring set up yet, but I should be able to
get the deployment updated with a metrics reporter. I'll update the thread
with my findings.

On Tue, Sep 20, 2016 at 10:30 PM, Nate McCall 
wrote:

> If you can get to them in the test env. you want to look in
> o.a.c.metrics.CommitLog for:
> - TotalCommitlogSize: if this hovers near commitlog_size_in_mb and never
> goes down, you are thrashing on segment allocation
> - WaitingOnCommit: this is the time spent waiting on calls to sync and
> will start to climb real fast if you cant sync within sync_interval
> - WaitingOnSegmentAllocation: how long it took to allocate a new commitlog
> segment, if it is all over the place it is IO bound
>
> Try turning all the commit log settings way down for low-IO test
> infrastructure like this. Maybe total commit log size of like 32mb with 4mb
> segments (or even lower depending on test data volume) so they basically
> flush constantly and don't try to hold any tables open. Also lower
> concurrent_writes substantially while you are at it to add some write
> throttling.
>
> On Wed, Sep 21, 2016 at 2:14 PM, John Sanda  wrote:
>
>> I have seen in various threads on the list that 3.0.x is probably best
>> for prod. Just wondering though if there is anything in particular in 3.7
>> to be weary of.
>>
>> I need to check with one of our QA engineers to get specifics on the
>> storage. Here is what I do know. We have a blade center running lots of
>> virtual machines for various testing. Some of those vm's are running
>> Cassandra and the Java web apps I previously mentioned via docker
>> containers. The storage is shared. Beyond that I don't have any more
>> specific details at the moment. I can also tell you that the storage can be
>> quite slow.
>>
>> I have come across different threads that talk to one degree or another
>> about the flush queue getting full. I have been looking at the code in
>> ColumnFamilyStore.java. Is perDiskFlushExecutors the thread pool I should
>> be interested in? It uses an unbounded queue, so I am not really sure what
>> it means for it to get full. Is there anything I can check or look for to
>> see if writes are getting blocked?
>>
>> On Tue, Sep 20, 2016 at 8:41 PM, Jonathan Haddad 
>> wrote:
>>
>>> If you haven't yet deployed to prod I strongly recommend *not* using
>>> 3.7.
>>>
>>> What network storage are you using?  Outside of a handful of highly
>>> experienced experts using EBS in very specific ways, it usually ends in
>>> failure.
>>>
>>> On Tue, Sep 20, 2016 at 3:30 PM John Sanda  wrote:
>>>
 I am deploying multiple Java web apps that connect to a Cassandra 3.7
 instance. Each app creates its own schema at start up. One of the schema
 changes involves dropping a table. I am seeing frequent client-side
 timeouts reported by the DataStax driver after the DROP TABLE statement is
 executed. I don't see this behavior in all environments. I do see it
 consistently in a QA environment in which Cassandra is running in docker
 with network storage, so writes are pretty slow from the get go. In my logs
 I see a lot of tables getting flushed, which I guess are all of the dirty
 column families in the respective commit log segment. Then I seen a whole
 bunch of flushes getting queued up. Can I reach a point in which too many
 table flushes get queued such that writes would be blocked?


 --

 - John

>>>
>>
>>
>> --
>>
>> - John
>>
>
>
>
> --
> -
> Nate McCall
> Wellington, NZ
> @zznate
>
> CTO
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>



-- 

- John

Re: Client-side timeouts after dropping table

[Nate McCall]

If you can get to them in the test env. you want to look in
o.a.c.metrics.CommitLog for:
- TotalCommitlogSize: if this hovers near commitlog_size_in_mb and never
goes down, you are thrashing on segment allocation
- WaitingOnCommit: this is the time spent waiting on calls to sync and will
start to climb real fast if you cant sync within sync_interval
- WaitingOnSegmentAllocation: how long it took to allocate a new commitlog
segment, if it is all over the place it is IO bound

Try turning all the commit log settings way down for low-IO test
infrastructure like this. Maybe total commit log size of like 32mb with 4mb
segments (or even lower depending on test data volume) so they basically
flush constantly and don't try to hold any tables open. Also lower
concurrent_writes substantially while you are at it to add some write
throttling.

On Wed, Sep 21, 2016 at 2:14 PM, John Sanda  wrote:

> I have seen in various threads on the list that 3.0.x is probably best for
> prod. Just wondering though if there is anything in particular in 3.7 to be
> weary of.
>
> I need to check with one of our QA engineers to get specifics on the
> storage. Here is what I do know. We have a blade center running lots of
> virtual machines for various testing. Some of those vm's are running
> Cassandra and the Java web apps I previously mentioned via docker
> containers. The storage is shared. Beyond that I don't have any more
> specific details at the moment. I can also tell you that the storage can be
> quite slow.
>
> I have come across different threads that talk to one degree or another
> about the flush queue getting full. I have been looking at the code in
> ColumnFamilyStore.java. Is perDiskFlushExecutors the thread pool I should
> be interested in? It uses an unbounded queue, so I am not really sure what
> it means for it to get full. Is there anything I can check or look for to
> see if writes are getting blocked?
>
> On Tue, Sep 20, 2016 at 8:41 PM, Jonathan Haddad 
> wrote:
>
>> If you haven't yet deployed to prod I strongly recommend *not* using 3.7.
>>
>>
>> What network storage are you using?  Outside of a handful of highly
>> experienced experts using EBS in very specific ways, it usually ends in
>> failure.
>>
>> On Tue, Sep 20, 2016 at 3:30 PM John Sanda  wrote:
>>
>>> I am deploying multiple Java web apps that connect to a Cassandra 3.7
>>> instance. Each app creates its own schema at start up. One of the schema
>>> changes involves dropping a table. I am seeing frequent client-side
>>> timeouts reported by the DataStax driver after the DROP TABLE statement is
>>> executed. I don't see this behavior in all environments. I do see it
>>> consistently in a QA environment in which Cassandra is running in docker
>>> with network storage, so writes are pretty slow from the get go. In my logs
>>> I see a lot of tables getting flushed, which I guess are all of the dirty
>>> column families in the respective commit log segment. Then I seen a whole
>>> bunch of flushes getting queued up. Can I reach a point in which too many
>>> table flushes get queued such that writes would be blocked?
>>>
>>>
>>> --
>>>
>>> - John
>>>
>>
>
>
> --
>
> - John
>



-- 
-
Nate McCall
Wellington, NZ
@zznate

CTO
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: Client-side timeouts after dropping table

[John Sanda]

I have seen in various threads on the list that 3.0.x is probably best for
prod. Just wondering though if there is anything in particular in 3.7 to be
weary of.

I need to check with one of our QA engineers to get specifics on the
storage. Here is what I do know. We have a blade center running lots of
virtual machines for various testing. Some of those vm's are running
Cassandra and the Java web apps I previously mentioned via docker
containers. The storage is shared. Beyond that I don't have any more
specific details at the moment. I can also tell you that the storage can be
quite slow.

I have come across different threads that talk to one degree or another
about the flush queue getting full. I have been looking at the code in
ColumnFamilyStore.java. Is perDiskFlushExecutors the thread pool I should
be interested in? It uses an unbounded queue, so I am not really sure what
it means for it to get full. Is there anything I can check or look for to
see if writes are getting blocked?

On Tue, Sep 20, 2016 at 8:41 PM, Jonathan Haddad  wrote:

> If you haven't yet deployed to prod I strongly recommend *not* using 3.7.
>
> What network storage are you using?  Outside of a handful of highly
> experienced experts using EBS in very specific ways, it usually ends in
> failure.
>
> On Tue, Sep 20, 2016 at 3:30 PM John Sanda  wrote:
>
>> I am deploying multiple Java web apps that connect to a Cassandra 3.7
>> instance. Each app creates its own schema at start up. One of the schema
>> changes involves dropping a table. I am seeing frequent client-side
>> timeouts reported by the DataStax driver after the DROP TABLE statement is
>> executed. I don't see this behavior in all environments. I do see it
>> consistently in a QA environment in which Cassandra is running in docker
>> with network storage, so writes are pretty slow from the get go. In my logs
>> I see a lot of tables getting flushed, which I guess are all of the dirty
>> column families in the respective commit log segment. Then I seen a whole
>> bunch of flushes getting queued up. Can I reach a point in which too many
>> table flushes get queued such that writes would be blocked?
>>
>>
>> --
>>
>> - John
>>
>


-- 

- John

Re: Client-side timeouts after dropping table

[Jonathan Haddad]

If you haven't yet deployed to prod I strongly recommend *not* using 3.7.

What network storage are you using?  Outside of a handful of highly
experienced experts using EBS in very specific ways, it usually ends in
failure.

On Tue, Sep 20, 2016 at 3:30 PM John Sanda  wrote:

> I am deploying multiple Java web apps that connect to a Cassandra 3.7
> instance. Each app creates its own schema at start up. One of the schema
> changes involves dropping a table. I am seeing frequent client-side
> timeouts reported by the DataStax driver after the DROP TABLE statement is
> executed. I don't see this behavior in all environments. I do see it
> consistently in a QA environment in which Cassandra is running in docker
> with network storage, so writes are pretty slow from the get go. In my logs
> I see a lot of tables getting flushed, which I guess are all of the dirty
> column families in the respective commit log segment. Then I seen a whole
> bunch of flushes getting queued up. Can I reach a point in which too many
> table flushes get queued such that writes would be blocked?
>
>
> --
>
> - John
>

Client-side timeouts after dropping table

[John Sanda]

I am deploying multiple Java web apps that connect to a Cassandra 3.7
instance. Each app creates its own schema at start up. One of the schema
changes involves dropping a table. I am seeing frequent client-side
timeouts reported by the DataStax driver after the DROP TABLE statement is
executed. I don't see this behavior in all environments. I do see it
consistently in a QA environment in which Cassandra is running in docker
with network storage, so writes are pretty slow from the get go. In my logs
I see a lot of tables getting flushed, which I guess are all of the dirty
column families in the respective commit log segment. Then I seen a whole
bunch of flushes getting queued up. Can I reach a point in which too many
table flushes get queued such that writes would be blocked?

-- 

- John

Using keyspaces for virtual clusters

[Dorian Hoxha]

Hi,

I need to separate clients data into multiple clusters and because I don't
like having multiple cql clients/connections on my app-code, I'm thinking
of creating many keyspaces and storing them into many virtual datacenters
(the servers will be in 1 logical datacenter, but separated by keyspaces).

Does that make sense (so growing up to 200 dcs of 3 servers each in best
case scenario)?

Does the cql-engine make a new connection (like "use keyspace") when
specifying "keyspace.table" on the query ?

Are the keyspaces+tables of dc1 stored in a cassandra node of dc2 ?(since
there is overhead with each keyspace + table which would probably break
this design)
Or is it just a simple map dcx--->ip1,ip2,ip3 ?

Thank you!

Re: Re : Generic keystore when enabling SSL

[sai krishnam raju potturi]

thanks Robert; we followed the instructions mentioned in
http://thelastpickle.com/blog/2015/09/30/hardening-cassandra
-step-by-step-part-1-server-to-server.html. It worked great.

 Due to the security policies in our company, we were asked to
use 3rd party signed certs. Since we'll require to manage 100's of
individual certs, we wanted to know if there is a work around with a
generic keystore and truststore.

thanks
Sai

Re: Re : Generic keystore when enabling SSL

[Andrew Tolbert]

Hi Sai,

I would recommend following the approach described in this article via The
Last Pickle: http://thelastpickle.com/blog/2015/09/30/hardening-cassandra
-step-by-step-part-1-server-to-server.html

It does a really good job of laying out a strategy for internode encryption
by rolling your own CA and trusting it instead of individual certificates
for each node:

Now this is where it all comes together. Since all of our instance-specific
> keys have now been signed by the CA, we can share this trust store instance
> across the cluster as it effectively just says “I’m going to trust all
> connections whose client certificates were signed by this CA.”
>

Thanks,
Andy

On Tue, Sep 20, 2016 at 12:20 PM, sai krishnam raju potturi <
pskraj...@gmail.com> wrote:

> hi;
>   has anybody enabled SSL using a generic keystore for node-to-node
> encryption. We're using 3rd party signed certificates, and want to avoid
> the hassle of managing 100's of certificates.
>
> thanks
> Sai
>



-- 

[image: DataStaxLogo copy3.png] 

Andrew Tolbert

Software Engineer in Test | (612)-222-6271 | andrew.tolb...@datastax.com


 

 

Re : Generic keystore when enabling SSL

[sai krishnam raju potturi]

hi;
  has anybody enabled SSL using a generic keystore for node-to-node
encryption. We're using 3rd party signed certificates, and want to avoid
the hassle of managing 100's of certificates.

thanks
Sai

RE: Question about replica and replication factor

[Jun Wu]

Great explanation!
For the single partition read, it makes sense to read data from only one 
replica. 
Thank you so much Ben!
Jun

From: ben.sla...@instaclustr.com
Date: Tue, 20 Sep 2016 05:30:43 +
Subject: Re: Question about replica and replication factor
To: wuxiaomi...@hotmail.com
CC: user@cassandra.apache.org

“replica” here means “a node that has a copy of the data for a given 
partition”. The scenario being discussed hear is CL > 1. In this case, rather 
than using up network and processing capacity sending all the data from all the 
nodes required to meet the consistency level, Cassandra gets the full data from 
one replica and  checksums from the others. Only if the checksums don’t match 
the full data does Cassandra need to get full data from all the relevant 
replicas.
I think the other point here is, conceptually, you should think of the 
coordinator as splitting up any query that hits multiple partitions into a set 
of queries, one per partition (there might be some optimisations that make this 
not quite physically correct but conceptually it’s about right). Discussion 
such as the one you quote above tend to be considering a single partition read 
(which is the most common kind of read in most uses of Cassandra).
CheersBen
On Tue, 20 Sep 2016 at 15:18 Jun Wu  wrote:


Yes, I think for my case, at least two nodes need to be contacted to get the 
full set of data.
But another thing comes up about dynamic snitch. It's the wrapped snitch and 
enabled by default and it'll choose the fastest/closest node to read data from. 
Another post is about 
this.http://www.datastax.com/dev/blog/dynamic-snitching-in-cassandra-past-present-and-future
 
The thing is why it's still emphasis only one replica to read data from. Below 
is from the post:
To begin, let’s first answer the most obvious question: what is dynamic 
snitching? To understand this, we’ll first recall what a snitch does. A 
snitch’s function is to determine which datacenters and racks are both written 
to and read from. So, why would that be ‘dynamic?’ This comes into play on the 
read side only (there’s nothing to be done for writes since we send them all 
and then block to until the consistency level is achieved.) When doing reads 
however, Cassandra only asks one node for the actual data, and, depending on 
consistency level and read repair chance, it asks the remaining replicas for 
checksums only. This means that it has a choice of however many replicas exist 
to ask for the actual data, and this is where the dynamic snitch goes to 
work.Since only one replica is sending the full data we need, we need to chose 
the best possible replica to ask, since if all we get back is checksums we have 
nothing useful to return to the user. The dynamic snitch handles this task by 
monitoring the performance of reads from the various replicas and choosing the 
best one based on this history.
Sent from my iPadOn Sep 20, 2016, at 00:03, Ben Slater 
 wrote:

If your read operation requires data from multiple partitions and the 
partitions are spread across multiple nodes then the coordinator has the job of 
contacting the multiple nodes to get the data and return to the client. So, in 
your scenario, if you did a select * from table (with no where clause) the 
coordinator would need to contact and execute a read on at least one other node 
to satisfy the query.
CheersBen
On Tue, 20 Sep 2016 at 14:50 Jun Wu  wrote:



Hi Ben,
Thanks for the quick response. 
It's clear about the example for single row/partition. However, normally 
data are not single row. Then for this case, I'm still confused. 
http://docs.datastax.com/en/cassandra/2.1/cassandra/dml/architectureClientRequestsRead_c.html
The link above gives an example of 10 nodes cluster with RF = 3. But the 
figure and the words in the post shows that the coordinator only contact/read 
data from one replica, and operate read repair for the left replicas. 
Also, how could read accross all nodes in the cluster? 
Thanks!
Jun


From: ben.sla...@instaclustr.com
Date: Tue, 20 Sep 2016 04:18:59 +
Subject: Re: Question about replica and replication factor
To: user@cassandra.apache.org

Each individual read (where a read is a single row or single partition) will 
read from one node (ignoring read repairs) as each partition will be contained 
entirely on a single node. To read the full set of data,  reads would hit at 
least two nodes (in practice, reads would likely end up being distributed 
across all the nodes in your cluster).
CheersBen
On Tue, 20 Sep 2016 at 14:09 Jun Wu  wrote:



Hi there,
I have a question about the replica and replication factor. 
For example, I have a cluster of 6 nodes in the same data center. 
Replication factor RF is set to 3  and the consistency level is default 1. 
According to this calculator http://www.ecyrd.com/cassandracalculator/, every 

Re: How to query '%' character using LIKE operator in Cassandra 3.7?

[Mikhail Krupitskiy]

Hi!

Have you had a chance to try your patch or solve the issue in an other way? 

Thanks,
Mikhail
> On 15 Sep 2016, at 16:02, DuyHai Doan  wrote:
> 
> Ok so I've found the source of the issue, it's pretty well hidden because it 
> is NOT in the SASI source code directly.
> 
> Here is the method where C* determines what kind of LIKE expression you're 
> using (LIKE_PREFIX , LIKE CONTAINS or LIKE_MATCHES)
> 
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/cql3/restrictions/SingleColumnRestriction.java#L733-L778
>  
> 
> 
> As you can see, it's pretty simple, maybe too simple. Indeed, they forget to 
> remove escape character BEFORE doing the matching so if your search is LIKE 
> '%%esc%', the detected expression is LIKE_CONTAINS.
> 
> A possible fix would be:
> 
> 1) convert the bytebuffer into plain String (UTF8 or ASCII, depending on the 
> column data type)
> 2) remove the escape character e.g. before parsing OR use some advanced regex 
> to exclude the %% from parsing e.g
> 
> Step 2) is dead easy but step 1) is harder because I don't know if converting 
> the bytebuffer into String at this stage of the CQL parser is expensive or 
> not (in term of computation)
> 
> Let me try a patch  
> 
> 
> 
> On Wed, Sep 14, 2016 at 9:42 AM, DuyHai Doan  > wrote:
> Ok you're right, I get your point
> 
> LIKE '%%esc%' --> startWith('%esc')
> 
> LIKE 'escape%%' -->  = 'escape%'
> 
> What I strongly suspect is that in the source code of SASI, we parse the % 
> xxx % expression BEFORE applying escape. That will explain the observed 
> behavior. E.g:
> 
> LIKE '%%esc%'  parsed as %xxx% where xxx = %esc
> 
> LIKE 'escape%%' parsed as xxx% where xxx =escape%
> 
> Let me check in the source code and try to reproduce the issue
> 
> 
> 
> On Tue, Sep 13, 2016 at 7:24 PM, Mikhail Krupitskiy 
> > 
> wrote:
> Looks like we have different understanding of what results are expected.
> I based my understanding on 
> http://docs.datastax.com/en/cql/3.3/cql/cql_using/useSASIIndex.html 
> 
> According to the doc ‘esc’ is a pattern for exact match and I guess that 
> there is no semantical difference between two LIKE patterns (both of patterns 
> should be treated as ‘exact match'): ‘%%esc’ and ‘esc’.
> 
>> SELECT * FROM escape WHERE val LIKE '%%esc%'; --> Give all results 
>> containing '%esc' so %escapeme is a possible match and also escape%esc
> Why ‘containing’? I expect that it should be ’starting’..
>> 
>> SELECT * FROM escape WHERE val LIKE 'escape%%' --> Give all results starting 
>> with 'escape%' so escape%me is a valid result and also escape%esc
> Why ’starting’? I expect that it should be ‘exact matching’.
> 
> Also I expect that “ LIKE ‘%s%sc%’ ” will return ‘escape%esc’ but it returns 
> nothing (CASSANDRA-12573).
> 
> What I’m missing?
> 
> Thanks,
> Mikhail
> 
>> On 13 Sep 2016, at 19:31, DuyHai Doan > > wrote:
>> 
>> CREATE CUSTOM INDEX ON test.escape(val) USING 'org.apache.cassandra.index.sa 
>> si.SASIIndex' WITH OPTIONS = {'mode': 
>> 'CONTAINS', 'analyzer_class': 'org.apache.cassandra.index.sa 
>> si.analyzer.NonTokenizingAnalyzer', 
>> 'case_sensitive': 'false'};
>> 
>> I don't see any problem in the results you got
>> 
>> SELECT * FROM escape WHERE val LIKE '%%esc%'; --> Give all results 
>> containing '%esc' so %escapeme is a possible match and also escape%esc
> Why ‘containing’? I expect that it should be ’starting’..
>> 
>> SELECT * FROM escape WHERE val LIKE 'escape%%' --> Give all results starting 
>> with 'escape%' so escape%me is a valid result and also escape%esc
> Why ’starting’? I expect that it should be ‘exact matching’.
> 
>> 
>> On Tue, Sep 13, 2016 at 5:58 PM, Mikhail Krupitskiy 
>> > 
>> wrote:
>> Thanks for the reply.
>> Could you please provide what index definition did you use?
>> With the index from my script I get the following results:
>> 
>> cqlsh:test> select * from escape;
>> 
>>  id | val
>> +---
>>   1 | %escapeme
>>   2 | escape%me
>>   3 | escape%esc
>> 
>> Contains search
>> 
>> cqlsh:test> SELECT * FROM escape WHERE val LIKE '%%esc%';
>> 
>>  id | val
>> +---
>>   1 | %escapeme
>>   3 | escape%esc
>> (2 rows)
>> 
>> 
>> Prefix search
>> 
>> cqlsh:test> SELECT * FROM escape WHERE val LIKE 'escape%%';
>> 
>>  id | val
>> +---
>>   2 | escape%me
>>   3 | escape%esc
>> 
>> Thanks,
>> Mikhail 
>> 
>>> On 13 Sep 2016, at 18:16, DuyHai Doan >> 

Re: Nodetool repair

[Li, Guangxing]

Hi,

I am using version 2.0.9. I have been looking into the logs to see if a
repair is finished. Each time a repair is started on a node, I am seeing
log line like "INFO [Thread-112920] 2016-09-16 19:00:43,805
StorageService.java (line 2646) Starting repair command #41, repairing 2048
ranges for keyspace groupmanager" in system.log. So I know that I am
expecting to see 2048 log lines like "INFO [AntiEntropySessions:109]
2016-09-16 19:27:20,662 RepairSession.java (line 282) [repair
#8b910950-7c43-11e6-88f3-f147ea74230b] session completed successfully".
Once I see 2048 such log lines, I know this repair has completed. But this
is not dependable since sometimes I am seeing less than 2048 but I know
there is no repair going on since I do not see any trace of repair in
system.log for a long time. So it seems to me that there is a clear way to
tell that a repair has started but there is no clear way to tell a repair
has ended. The only thing you can do is to watch the log and if you do not
see repair activity for a long time, the repair is done somehow. I am
wondering why the implementor did not put something in the log (e.g. ...
Repair command #41 has ended...) to clearly state that the repair has
completed.

Thanks.

George.

On Tue, Sep 20, 2016 at 2:54 AM, Jens Rantil  wrote:

> On Mon, Sep 19, 2016 at 3:07 PM Alain RODRIGUEZ 
> wrote:
>
> ...
>
>> - The size of your data
>> - The number of vnodes
>> - The compaction throughput
>> - The streaming throughput
>> - The hardware available
>> - The load of the cluster
>> - ...
>>
>
> I've also heard that the number of clustering keys per partition key could
> have an impact. Might be worth investigating.
>
> Cheers,
> Jens
> --
>
> Jens Rantil
> Backend Developer @ Tink
>
> Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
> For urgent matters you can reach me at +46-708-84 18 32.
>

Re: https://issues.apache.org/jira/browse/CASSANDRA-10961 fix

[Paulo Motta]

Hello Zhiyan,

Replying to the mailing list since this could help others. I'm not sure
what that could be, it's generally related to some kind of corruption,
perhaps CASSANDRA-10791. Although the message is similar to #10971, that is
restricted to streaming so it's a different issue here. Was this a one-off
problem or is it happening frequently? Did you upgrade recently? Did you
try running scrub in the affected node/sstables?

Thanks,

Paulo

2016-09-20 4:50 GMT-03:00 Zhiyan Shao :

> Hi Paul,
>
> I got a similar crash as #10961 but couldn't figure out where is the
> issue. Could you help take a look? The crash happened during select query
> with pagination. Here is the stack:
>
>
> ERROR [SharedPool-Worker-3] 2016-09-13 02:10:54,349 ErrorMessage.java:336
> - Unexpected exception during request java.lang.IllegalArgumentException:
> Not enough bytes. Offset: 2. Length: 837. Buffer size: 16 at
> org.apache.cassandra.db.composites.AbstractCType.checkRemaining(AbstractCType.java:362)
> ~[apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.db.compos
> ites.AbstractCompoundCellNameType.fromByteBuffer(AbstractCompoundCellNameType.java:98)
> ~[apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.service.p
> ager.SliceQueryPager.(SliceQueryPager.java:65)
> ~[apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.service.p
> ager.QueryPagers.pager(QueryPagers.java:91) 
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.service.pager.QueryPagers.pager(QueryPagers.java:100)
> ~[apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.service.p
> ager.QueryPagers.pager(QueryPagers.java:126)
> ~[apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.cql3.stat
> ements.SelectStatement.execute(SelectStatement.java:178)
> ~[apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.cql3.stat
> ements.SelectStatement.execute(SelectStatement.java:76)
> ~[apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.cql3.Quer
> yProcessor.processStatement(QueryProcessor.java:226)
> ~[apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.cql3.Quer
> yProcessor.processPrepared(QueryProcessor.java:466)
> ~[apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.cql3.Quer
> yProcessor.processPrepared(QueryProcessor.java:443)
> ~[apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.transport
> .messages.ExecuteMessage.execute(ExecuteMessage.java:142)
> ~[apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.transport
> .Message$Dispatcher.channelRead0(Message.java:507)
> [apache-cassandra-2.2.5.jar:2.2.5] at org.apache.cassandra.transport
> .Message$Dispatcher.channelRead0(Message.java:401)
> [apache-cassandra-2.2.5.jar:2.2.5] at io.netty.channel.SimpleChannel
> InboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.23.Final.jar:4.0.23.Final] at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannel
> Read(AbstractChannelHandlerContext.java:333)
> [netty-all-4.0.23.Final.jar:4.0.23.Final] at
> io.netty.channel.AbstractChannelHandlerContext.access$700(Ab
> stractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.channel.AbstractChannelHandlerContext$8.run(Abstrac
> tChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_74] at org.apache.cassandra.concurren
> t.AbstractLocalAwareExecutorService$FutureTask.run(AbstractL
> ocalAwareExecutorService.java:164) [apache-cassandra-2.2.5.jar:2.2.5] at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-2.2.5.jar:2.2.5] at java.lang.Thread.run(Thread.java:745)
> [na:1.8.0_74]
>
> Thanks,
> Zhiyan
>
>

Re: High load on few nodes in a DC.

[Pranay akula]

I was a able to find the hotspots causing the load,but the size of these
partitions are in KB and no tombstones and no.of sstables is only 2 what
else i need to debug to find the reason for high load for some nodes.
  we are also using unlogged batches is that can be the reason ?? how to
find which node is serving as a coordinator for un logged batches?? we are
using token-aware policy.

thanks



On Mon, Sep 19, 2016 at 12:29 PM, Pranay akula 
wrote:

> I was able to see most used partitions but the nodes with less load are
> serving more read and write requests for that particular partitions when
> compared to nodes with high load, how can i find if these nodes are serving
> as co-coordinators for those read and write requests ?? how can i find the
> token range for these particular partitions and which node is the primary
> for these partition ??
>
>
> Thanks
>
> On Mon, Sep 19, 2016 at 11:04 AM, Pranay akula  > wrote:
>
>> Hai Jeff,
>>
>> Thank, we are using RF 3 and cassandra version 2.1.8.
>>
>> Thanks
>> Pranay.
>>
>> On Mon, Sep 19, 2016 at 10:55 AM, Jeff Jirsa 
>> wrote:
>>
>>> Is your replication_factor 2? Or is it 3?  What version are you using?
>>>
>>>
>>>
>>> The most likely answer is some individual partition that’s either being
>>> written/read more than others, or is somehow impacting the cluster (wide
>>> rows are a natural candidate).
>>>
>>>
>>>
>>> You don’t mention your version, but most modern versions of Cassandra
>>> ship with ‘nodetool toppartitions’, which will help you identify frequently
>>> written/read partitions – perhaps you can use that to identify a hotspot
>>> due to some external behavior (some partition being read thousands of
>>> times, over and over could certainly drive up load).
>>>
>>>
>>>
>>> -  Jeff
>>>
>>>
>>>
>>> *From: *Pranay akula 
>>> *Reply-To: *"user@cassandra.apache.org" 
>>> *Date: *Monday, September 19, 2016 at 7:53 AM
>>> *To: *"user@cassandra.apache.org" 
>>> *Subject: *High load on few nodes in a DC.
>>>
>>>
>>>
>>> when our cluster was under load  i am seeing  1 or 2 nodes are on more
>>> load consistently when compared to others in dc i am not seeing any GC
>>> pauses or wide partitions  is this can be those nodes are continuously
>>> serving as coordinators ?? how can  i find what is the reason for high load
>>> on those two nodes ?? We are using Vnode.
>>>
>>>
>>>
>>>
>>>
>>> Thanks
>>>
>>> Pranay.
>>>
>>
>>
>

unsubscribe

[Jerry Poole]

IMPORTANT NOTICE: This e-mail, including any and all attachments, is intended 
for the addressee or its representative only. It is confidential and may be 
under legal privilege. Any form of unauthorized use, publication, reproduction, 
copying or disclosure of the content of this e-mail is not permitted. If you 
are not the intended recipient of this e-mail and its contents, please notify 
the sender immediately by reply e-mail and delete this e-mail and all its 
attachments subsequently.

[RELEASE] Apache Cassandra 3.0.9 released

[Jake Luciani]

The Cassandra team is pleased to announce the release of Apache Cassandra
version 3.0.9.

Apache Cassandra is a fully distributed database. It is the right choice
when you need scalability and high availability without compromising
performance.

 http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our download
section:

 http://cassandra.apache.org/download/

This version is a bug fix release[1] on the 3.0 series. As always, please
pay
attention to the release notes[2] and Let us know[3] if you were to
encounter
any problem.

Enjoy!

[1]: https://goo.gl/YfvFn8 (CHANGES.txt)
[2]: https://goo.gl/k9leqx (NEWS.txt)
[3]: https://issues.apache.org/jira/browse/CASSANDRA

Re: Export/Importing keyspace from a different sized cluster

[Romain Hardouin]

Also for testing purposes, you can send only one replica set to the Test DC. 
For instance with a RF=3 and 3 C* racks, you can just rsync/sstableload one 
rack. It will be faster and OK for tests.
Best,
Romain 

Le Mardi 20 septembre 2016 3h28, Michael Laws  a 
écrit :
 

 I put together a shell 
wrapper around nodetool/sstableloader that I’ve been running for the past few 
years – https://github.com/AppliedInfrastructure/cassandra-snapshot-toolsAlways 
seemed to work well for these kinds of scenarios…  Never really had to think 
about where SSTables were on the filesystem, etc. Mike From: Justin Sanciangco 
[mailto:jsancian...@blizzard.com] 
Sent: Monday, September 19, 2016 6:20 PM
To: user@cassandra.apache.org
Subject: RE: Export/Importing keyspace from a different sized cluster I am 
running cqlsh 5.0.1 | Cassandra 2.1.11.969 | DSE 4.8.3 | CQL spec 3.2.1 |  
Doing the below command seemed to worksstableloader -d  
 Thanks for the help!  From: Jeff Jirsa 
[mailto:jeff.ji...@crowdstrike.com] 
Sent: Monday, September 19, 2016 5:49 PM
To: user@cassandra.apache.org
Subject: Re: Export/Importing keyspace from a different sized cluster Something 
like that, depending on your version (which you didn’t specify). Note, though, 
that sstableloader is notoriously picky about the path to sstables. In 
particular, it really really really wants a directory structure that matches 
the directory structure on disk, and wants you to be at the equivalent of the 
parent/data_files_directory (so if you dump your sstables at 
/path/to/data/keyspace/table/, you’d want to run sstableloader from 
/path/to/data/ and provide keyspace/table/ as the location).   From: Justin 
Sanciangco 
Reply-To: "user@cassandra.apache.org" 
Date: Monday, September 19, 2016 at 5:44 PM
To: "user@cassandra.apache.org" 
Subject: RE: Export/Importing keyspace from a different sized cluster So if I 
rsync the the sstables say from source node 1 and source node 2 to target node 
1. Would I just run the command like this? From target hostsstableloader -d 
  From: Jeff Jirsa 
[mailto:jeff.ji...@crowdstrike.com] 
Sent: Monday, September 19, 2016 4:45 PM
To: user@cassandra.apache.org
Subject: Re: Export/Importing keyspace from a different sized cluster You can 
ship the sstables to the destination (or any other server with Cassandra binary 
tools installed) via ssh/rsync and run sstableloader on the destination cluster 
as well.  From: Justin Sanciangco 
Reply-To: "user@cassandra.apache.org" 
Date: Monday, September 19, 2016 at 2:49 PM
To: "user@cassandra.apache.org" 
Subject: Export/Importing keyspace from a different sized cluster Hello, 
Assuming I can’t get ports opened from source to target cluster to run 
sstableloader, what methods can I use to load a single keyspace from one 
cluster to another cluster of different size?  Appreciate the help… 
Thanks,Justin 

   

Re: Optimal value for concurrent_reads for a single NVMe Disk

[Romain Hardouin]

Hi,
You should make a benchmark with cassandra-stress to find the sweet spot. With 
NVMe I guess you can start with a high value, 128?
Please let us know the results of your findings, it's interesting to know if we 
can go crazy with such pieces of hardware :-)
Best,
Romain 

Le Mardi 20 septembre 2016 12h11, Thomas Julian  a 
écrit :
 

 Hello,

We are using Cassandra 2.1.13 with each node having a NVMe disk with the 
configuration of Total Capacity - 1.2TB, Alloted Capacity -  880GB. We would 
like to increase the default value of 32 for the param concurrent_reads. But 
the document says 

"(Default: 32)note For workloads with more data than can fit in memory, the 
bottleneck is reads fetching data from disk. Setting to (16 × number_of_drives) 
allows operations to queue low enough in the stack so that the OS and drives 
can reorder them. The default setting applies to both logical volume managed 
(LVM) and RAID drives."

https://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configCassandra_yaml_r.html#reference_ds_qfg_n1r_1k__concurrent_reads

According to this hardware specification, what could be the optimal value that 
can be set for concurrent_reads?

Best Regards,
Julian.







   

Re: How to alter the default value for concurrent_compactors

[Romain Hardouin]

Hi,
You can read and write the value of the following MBean via 
JMX:org.apache.cassandra.db:type=CompactionManager  - CoreCompactorThreads
 - MaximumCompactorThreads

If you modify CoreCompactorThreads it will be effective immediatly, I mean 
assuming you have some pending compactions, you will see N lines with nodetool 
compactionstats where N=.
Best,
Romain
 

Le Mardi 20 septembre 2016 13h50, Thomas Julian  a 
écrit :
 

 Hello,

We have commented out "concurrent_compactors" in our Cassandra 2.1.13 
installation. 
We would like to review this setting, as some issues indicate that the default 
configuration may affect read/write performance. 

https://issues.apache.org/jira/browse/CASSANDRA-8787
https://issues.apache.org/jira/browse/CASSANDRA-7139

Where can we see the value set for concurrent_compactors in our setup? Is it 
possible to update this configuration without a restart?

Best Regards,
Julian.




   

How to alter the default value for concurrent_compactors

[Thomas Julian]

Hello,



We have commented out "concurrent_compactors" in our Cassandra 2.1.13 
installation. 

We would like to review this setting, as some issues indicate that the default 
configuration may affect read/write performance. 



https://issues.apache.org/jira/browse/CASSANDRA-8787

https://issues.apache.org/jira/browse/CASSANDRA-7139



Where can we see the value set for concurrent_compactors in our setup? Is it 
possible to update this configuration without a restart?



Best Regards,

Julian.

Optimal value for concurrent_reads for a single NVMe Disk

[Thomas Julian]

Hello,



We are using Cassandra 2.1.13 with each node having a NVMe disk with the 
configuration of Total Capacity - 1.2TB, Alloted Capacity -  880GB. We would 
like to increase the default value of 32 for the param concurrent_reads. But 
the document says 



"(Default: 32)note For workloads with more data than can fit in memory, the 
bottleneck is reads fetching data from disk. Setting to (16 × number_of_drives) 
allows operations to queue low enough in the stack so that the OS and drives 
can reorder them. The default setting applies to both logical volume managed 
(LVM) and RAID drives."



https://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configCassandra_yaml_r.html#reference_ds_qfg_n1r_1k__concurrent_reads



According to this hardware specification, what could be the optimal value that 
can be set for concurrent_reads?



Best Regards,

Julian.

Re: cqlsh problem

[George Sigletos]

This appears in the system log:

Caused by: java.lang.RuntimeException:
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out -
received only 2 responses.
at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276)
~[apache-cassandra-2.1.14.jar:2.1.14]
at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524)
~[guava-16.0.jar:na]
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317)
~[guava-16.0.jar:na]
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280)
~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195)
~[guava-16.0.jar:na]


On Tue, Sep 20, 2016 at 11:12 AM, George Sigletos 
wrote:

> I am also getting the same error:
> cqlsh  -u cassandra -p cassandra
>
> Connection error: ('Unable to connect to any servers', {'':
> OperationTimedOut('errors=Timed out creating connection (5 seconds),
> last_host=None',)})
>
> But it is not consistent. Sometimes I manage to connect. It is random.
> Using 2.1.14
>
> On Tue, Jun 14, 2016 at 4:29 AM, joseph gao 
> wrote:
>
>> hi, Patric, [image: 内嵌图片 1],
>> netstat -lepunt looks like above
>>
>> 2016-05-27 23:16 GMT+08:00 Patrick McFadin :
>>
>>> Can you do a netstat -lepunt and show the output? If Cassandra is
>>> running you aren't trying to connect to the ip/port it's bound to.
>>>
>>> Patrick
>>>
>>>
>>> On Monday, May 23, 2016, joseph gao  wrote:
>>>
 I used to think it's firewall/network issues too. So I make ufw to be
 inactive. I really don't what's the reason.

 2016-05-09 19:01 GMT+08:00 kurt Greaves :

> Don't be fooled, despite saying tcp6 and :::*, it still listens on
> IPv4. As far as I'm aware this happens on all 2.1 Cassandra nodes, and may
> just be an oddity of netstat. It would be unrelated to your connection
> timeout issues, that's most likely related to firewall/network issues.
>
> On 9 May 2016 at 09:59, joseph gao  wrote:
>
>> It doesn't work ,still using ipv6 [image: 内嵌图片 1]
>>
>> And I already set [image: 内嵌图片 2]
>>
>> Now I'm using 4.1.1 using 9160 port instead of 5.x.x。
>>
>> Hopefully this could be resolved, Thanks!
>>
>> 2016-03-30 22:13 GMT+08:00 Alain RODRIGUEZ :
>>
>>> Hi Joseph,
>>>
>>> why cassandra using tcp6 for 9042 port like :
 tcp6   0  0 0.0.0.0:9042:::*
  LISTEN

>>>
>>> if I remember correctly, in 2.1 and higher, cqlsh uses native
>>> transport, port 9042  (instead of thrift port 9160) and your clients (if
>>> any) are also probably using native transport (port 9042). So yes, this
>>> could be an issue indeed.
>>>
>>> You should have something like:
>>>
>>> tcp0  0  1.2.3.4:9042   :::*
>>> LISTEN
>>>
>>> You are using IPv6 and no rpc address. Try setting it to the listen
>>> address and using IPv4.
>>>
>>> C*heers,
>>>
>>> ---
>>>
>>> Alain Rodriguez - al...@thelastpickle.com
>>>
>>> France
>>>
>>> The Last Pickle - Apache Cassandra Consulting
>>>
>>> http://www.thelastpickle.com
>>>
>>> 2016-03-30 6:09 GMT+02:00 joseph gao :
>>>
 why cassandra using tcp6 for 9042 port like :
 tcp6   0  0 0.0.0.0:9042:::*
  LISTEN
 would this be the problem

 2016-03-30 11:34 GMT+08:00 joseph gao :

> still have not fixed it . cqlsh: error: no such option:
> --connect-timeout
> cqlsh version 5.0.1
>
>
>
> 2016-03-25 16:46 GMT+08:00 Alain RODRIGUEZ :
>
>> Hi Joseph.
>>
>> As I can't reproduce here, I believe you are having network issue
>> of some kind.
>>
>> MacBook-Pro:~ alain$ cqlsh --version
>> cqlsh 5.0.1
>> MacBook-Pro:~ alain$ echo 'DESCRIBE KEYSPACES;' | cqlsh
>> --connect-timeout=5 --request-timeout=10
>> system_traces  system
>> MacBook-Pro:~ alain$

Re: large system hint partition

[Romain Hardouin]

Hi,
> More recent (I think 2.2) don't have this problem since they write hints to 
>the file system as per the commit log
Flat files hints were implemented starting from 3.0  
https://issues.apache.org/jira/browse/CASSANDRA-6230
Best,
Romain

Re: cqlsh problem

[George Sigletos]

I am also getting the same error:
cqlsh  -u cassandra -p cassandra

Connection error: ('Unable to connect to any servers', {'':
OperationTimedOut('errors=Timed out creating connection (5 seconds),
last_host=None',)})

But it is not consistent. Sometimes I manage to connect. It is random.
Using 2.1.14

On Tue, Jun 14, 2016 at 4:29 AM, joseph gao  wrote:

> hi, Patric, [image: 内嵌图片 1],
> netstat -lepunt looks like above
>
> 2016-05-27 23:16 GMT+08:00 Patrick McFadin :
>
>> Can you do a netstat -lepunt and show the output? If Cassandra is running
>> you aren't trying to connect to the ip/port it's bound to.
>>
>> Patrick
>>
>>
>> On Monday, May 23, 2016, joseph gao  wrote:
>>
>>> I used to think it's firewall/network issues too. So I make ufw to be
>>> inactive. I really don't what's the reason.
>>>
>>> 2016-05-09 19:01 GMT+08:00 kurt Greaves :
>>>
 Don't be fooled, despite saying tcp6 and :::*, it still listens on
 IPv4. As far as I'm aware this happens on all 2.1 Cassandra nodes, and may
 just be an oddity of netstat. It would be unrelated to your connection
 timeout issues, that's most likely related to firewall/network issues.

 On 9 May 2016 at 09:59, joseph gao  wrote:

> It doesn't work ,still using ipv6 [image: 内嵌图片 1]
>
> And I already set [image: 内嵌图片 2]
>
> Now I'm using 4.1.1 using 9160 port instead of 5.x.x。
>
> Hopefully this could be resolved, Thanks!
>
> 2016-03-30 22:13 GMT+08:00 Alain RODRIGUEZ :
>
>> Hi Joseph,
>>
>> why cassandra using tcp6 for 9042 port like :
>>> tcp6   0  0 0.0.0.0:9042:::*
>>>  LISTEN
>>>
>>
>> if I remember correctly, in 2.1 and higher, cqlsh uses native
>> transport, port 9042  (instead of thrift port 9160) and your clients (if
>> any) are also probably using native transport (port 9042). So yes, this
>> could be an issue indeed.
>>
>> You should have something like:
>>
>> tcp0  0  1.2.3.4:9042   :::*
>> LISTEN
>>
>> You are using IPv6 and no rpc address. Try setting it to the listen
>> address and using IPv4.
>>
>> C*heers,
>>
>> ---
>>
>> Alain Rodriguez - al...@thelastpickle.com
>>
>> France
>>
>> The Last Pickle - Apache Cassandra Consulting
>>
>> http://www.thelastpickle.com
>>
>> 2016-03-30 6:09 GMT+02:00 joseph gao :
>>
>>> why cassandra using tcp6 for 9042 port like :
>>> tcp6   0  0 0.0.0.0:9042:::*
>>>  LISTEN
>>> would this be the problem
>>>
>>> 2016-03-30 11:34 GMT+08:00 joseph gao :
>>>
 still have not fixed it . cqlsh: error: no such option:
 --connect-timeout
 cqlsh version 5.0.1



 2016-03-25 16:46 GMT+08:00 Alain RODRIGUEZ :

> Hi Joseph.
>
> As I can't reproduce here, I believe you are having network issue
> of some kind.
>
> MacBook-Pro:~ alain$ cqlsh --version
> cqlsh 5.0.1
> MacBook-Pro:~ alain$ echo 'DESCRIBE KEYSPACES;' | cqlsh
> --connect-timeout=5 --request-timeout=10
> system_traces  system
> MacBook-Pro:~ alain$
>
> It's been a few days, did you manage to fix it ?
>
> C*heers,
> ---
> Alain Rodriguez - al...@thelastpickle.com
> France
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> 2016-03-21 9:59 GMT+01:00 joseph gao :
>
>> cqlsh version 5.0.1. nodetool tpstats looks good, log looks
>> good. And I used specified port 9042. And it immediately returns 
>> fail (less
>> than 3 seconds). By the way where should I use '--connect-timeout', 
>> cqlsh
>> seems don't have such parameters.
>>
>> 2016-03-18 17:29 GMT+08:00 Alain RODRIGUEZ :
>>
>>> Is the node fully healthy or rejecting some requests ?
>>>
>>> What are the outputs for "grep -i "ERROR"
>>> /var/log/cassandra/system.log" and "nodetool tpstats"?
>>>
>>> Any error? Any pending / blocked or dropped messages?
>>>
>>> Also did you try using distinct ports (9160 for thrift, 9042 for
>>> native) - out of curiosity, not sure this will help.
>>>
>>> What is your version of cqlsh "cqlsh --version" ?
>>>
>>> doesn't work most times. But some time it just work fine

>>>
>>> Do you fill like this is due to a timeout (query being too big,
>>> 

Re: Nodetool repair

[Jens Rantil]

On Mon, Sep 19, 2016 at 3:07 PM Alain RODRIGUEZ  wrote:

...

> - The size of your data
> - The number of vnodes
> - The compaction throughput
> - The streaming throughput
> - The hardware available
> - The load of the cluster
> - ...
>

I've also heard that the number of clustering keys per partition key could
have an impact. Might be worth investigating.

Cheers,
Jens
-- 

Jens Rantil
Backend Developer @ Tink

Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
For urgent matters you can reach me at +46-708-84 18 32.