Re: TRUNCATE throws OperationTimedOut randomly

2016-09-29 Thread Romain Hardouin 
Hi,
@Edward > In older versions you can not control when this call will 
timeout,truncate_request_timeout_in_ms is available for many years, starting 
from 1.2. Maybe you have another setting parameter in mind?
@GeorgeTry to put cassandra logs in debug
Best,
Romain
 

Le Mercredi 28 septembre 2016 20h31, George Sigletos 
 a écrit :
 

 Even when I set a lower request-timeout in order to trigger a timeout, still 
no WARN or ERROR in the logs

On Wed, Sep 28, 2016 at 8:22 PM, George Sigletos  wrote:

Hi Joaquin,

Unfortunately neither WARN nor ERROR found in the system logs across the 
cluster when executing truncate. Sometimes it executes immediately, other times 
it takes 25 seconds, given that I have connected with --request-timeout=30 
seconds. 

The nodes are a bit busy compacting. On a freshly restarted cluster, truncate 
seems to work without problems.

Some warnings that I see around that time but not exactly when executing 
truncate are:
WARN  [CompactionExecutor:2] 2016-09-28 20:03:29,646 SSTableWriter.java:241 - 
Compacting large partition system/hints:6f2c3b31-4975- 470b-8f91-e706be89a83a 
(133819308 bytes

Kind regards,
George

On Wed, Sep 28, 2016 at 7:54 PM, Joaquin Casares  
wrote:

Hi George,
Try grepping for WARN and ERROR on the system.logs across all nodes when you 
run the command. Could you post any of the recent stacktraces that you see?
Cheers,
Joaquin Casares
ConsultantAustin, TX
Apache Cassandra Consultinghttp://www.thelastpickle.com
On Wed, Sep 28, 2016 at 12:43 PM, George Sigletos  
wrote:

Thanks a lot for your reply.

I understand that truncate is an expensive operation. But throwing a timeout 
while truncating a table that is already empty?

A workaround is to set a high --request-timeout when connecting. Even 20 
seconds is not always enough

Kind regards,
George


On Wed, Sep 28, 2016 at 6:59 PM, Edward Capriolo  wrote:

Truncate does a few things (based on version) 
  truncate takes snapshots  truncate causes a flush
  in very old versions truncate causes a schema migration.

In newer versions like cassandra 3.4 you have this knob.

# How long the coordinator should wait for truncates to complete# (This can be 
much longer, because unless auto_snapshot is disabled# we need to flush first 
so we can snapshot before removing the data.)truncate_request_timeout_in_ms : 
6

In older versions you can not control when this call will timeout, it is fairly 
normal that it does!

On Wed, Sep 28, 2016 at 12:50 PM, George Sigletos  
wrote:

Hello,

I keep executing a TRUNCATE command on an empty table and it throws 
OperationTimedOut randomly:

cassandra@cqlsh> truncate test.mytable;
OperationTimedOut: errors={}, last_host=cassiebeta-01
cassandra@cqlsh> truncate test.mytable;
OperationTimedOut: errors={}, last_host=cassiebeta-01

Having a 3 node cluster running 2.1.14. No connectivity problems. Has anybody 
come across the same error?

Thanks,
George

Re: TRUNCATE throws OperationTimedOut randomly

2016-09-28 Thread George Sigletos 
Even when I set a lower request-timeout in order to trigger a timeout,
still no WARN or ERROR in the logs

On Wed, Sep 28, 2016 at 8:22 PM, George Sigletos 
wrote:

> Hi Joaquin,
>
> Unfortunately neither WARN nor ERROR found in the system logs across the
> cluster when executing truncate. Sometimes it executes immediately, other
> times it takes 25 seconds, given that I have connected with
> --request-timeout=30 seconds.
>
> The nodes are a bit busy compacting. On a freshly restarted cluster,
> truncate seems to work without problems.
>
> Some warnings that I see around that time but not exactly when executing
> truncate are:
> WARN  [CompactionExecutor:2] 2016-09-28 20:03:29,646
> SSTableWriter.java:241 - Compacting large partition
> system/hints:6f2c3b31-4975-470b-8f91-e706be89a83a (133819308 bytes
>
> Kind regards,
> George
>
> On Wed, Sep 28, 2016 at 7:54 PM, Joaquin Casares <
> joaq...@thelastpickle.com> wrote:
>
>> Hi George,
>>
>> Try grepping for WARN and ERROR on the system.logs across all nodes when
>> you run the command. Could you post any of the recent stacktraces that you
>> see?
>>
>> Cheers,
>>
>> Joaquin Casares
>> Consultant
>> Austin, TX
>>
>> Apache Cassandra Consulting
>> http://www.thelastpickle.com
>>
>> On Wed, Sep 28, 2016 at 12:43 PM, George Sigletos > > wrote:
>>
>>> Thanks a lot for your reply.
>>>
>>> I understand that truncate is an expensive operation. But throwing a
>>> timeout while truncating a table that is already empty?
>>>
>>> A workaround is to set a high --request-timeout when connecting. Even 20
>>> seconds is not always enough
>>>
>>> Kind regards,
>>> George
>>>
>>>
>>> On Wed, Sep 28, 2016 at 6:59 PM, Edward Capriolo 
>>> wrote:
>>>
 Truncate does a few things (based on version)
   truncate takes snapshots
   truncate causes a flush
   in very old versions truncate causes a schema migration.

 In newer versions like cassandra 3.4 you have this knob.

 # How long the coordinator should wait for truncates to complete
 # (This can be much longer, because unless auto_snapshot is disabled
 # we need to flush first so we can snapshot before removing the data.)
 truncate_request_timeout_in_ms: 6


 In older versions you can not control when this call will timeout, it
 is fairly normal that it does!


 On Wed, Sep 28, 2016 at 12:50 PM, George Sigletos <
 sigle...@textkernel.nl> wrote:

> Hello,
>
> I keep executing a TRUNCATE command on an empty table and it throws
> OperationTimedOut randomly:
>
> cassandra@cqlsh> truncate test.mytable;
> OperationTimedOut: errors={}, last_host=cassiebeta-01
> cassandra@cqlsh> truncate test.mytable;
> OperationTimedOut: errors={}, last_host=cassiebeta-01
>
> Having a 3 node cluster running 2.1.14. No connectivity problems. Has
> anybody come across the same error?
>
> Thanks,
> George
>
>

>>>
>>
>

Re: TRUNCATE throws OperationTimedOut randomly

2016-09-28 Thread George Sigletos 
Hi Joaquin,

Unfortunately neither WARN nor ERROR found in the system logs across the
cluster when executing truncate. Sometimes it executes immediately, other
times it takes 25 seconds, given that I have connected with
--request-timeout=30 seconds.

The nodes are a bit busy compacting. On a freshly restarted cluster,
truncate seems to work without problems.

Some warnings that I see around that time but not exactly when executing
truncate are:
WARN  [CompactionExecutor:2] 2016-09-28 20:03:29,646 SSTableWriter.java:241
- Compacting large partition
system/hints:6f2c3b31-4975-470b-8f91-e706be89a83a (133819308 bytes

Kind regards,
George

On Wed, Sep 28, 2016 at 7:54 PM, Joaquin Casares 
wrote:

> Hi George,
>
> Try grepping for WARN and ERROR on the system.logs across all nodes when
> you run the command. Could you post any of the recent stacktraces that you
> see?
>
> Cheers,
>
> Joaquin Casares
> Consultant
> Austin, TX
>
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> On Wed, Sep 28, 2016 at 12:43 PM, George Sigletos 
> wrote:
>
>> Thanks a lot for your reply.
>>
>> I understand that truncate is an expensive operation. But throwing a
>> timeout while truncating a table that is already empty?
>>
>> A workaround is to set a high --request-timeout when connecting. Even 20
>> seconds is not always enough
>>
>> Kind regards,
>> George
>>
>>
>> On Wed, Sep 28, 2016 at 6:59 PM, Edward Capriolo 
>> wrote:
>>
>>> Truncate does a few things (based on version)
>>>   truncate takes snapshots
>>>   truncate causes a flush
>>>   in very old versions truncate causes a schema migration.
>>>
>>> In newer versions like cassandra 3.4 you have this knob.
>>>
>>> # How long the coordinator should wait for truncates to complete
>>> # (This can be much longer, because unless auto_snapshot is disabled
>>> # we need to flush first so we can snapshot before removing the data.)
>>> truncate_request_timeout_in_ms: 6
>>>
>>>
>>> In older versions you can not control when this call will timeout, it is
>>> fairly normal that it does!
>>>
>>>
>>> On Wed, Sep 28, 2016 at 12:50 PM, George Sigletos <
>>> sigle...@textkernel.nl> wrote:
>>>
 Hello,

 I keep executing a TRUNCATE command on an empty table and it throws
 OperationTimedOut randomly:

 cassandra@cqlsh> truncate test.mytable;
 OperationTimedOut: errors={}, last_host=cassiebeta-01
 cassandra@cqlsh> truncate test.mytable;
 OperationTimedOut: errors={}, last_host=cassiebeta-01

 Having a 3 node cluster running 2.1.14. No connectivity problems. Has
 anybody come across the same error?

 Thanks,
 George


>>>
>>
>

Re: TRUNCATE throws OperationTimedOut randomly

2016-09-28 Thread Joaquin Casares 
Hi George,

Try grepping for WARN and ERROR on the system.logs across all nodes when
you run the command. Could you post any of the recent stacktraces that you
see?

Cheers,

Joaquin Casares
Consultant
Austin, TX

Apache Cassandra Consulting
http://www.thelastpickle.com

On Wed, Sep 28, 2016 at 12:43 PM, George Sigletos 
wrote:

> Thanks a lot for your reply.
>
> I understand that truncate is an expensive operation. But throwing a
> timeout while truncating a table that is already empty?
>
> A workaround is to set a high --request-timeout when connecting. Even 20
> seconds is not always enough
>
> Kind regards,
> George
>
>
> On Wed, Sep 28, 2016 at 6:59 PM, Edward Capriolo 
> wrote:
>
>> Truncate does a few things (based on version)
>>   truncate takes snapshots
>>   truncate causes a flush
>>   in very old versions truncate causes a schema migration.
>>
>> In newer versions like cassandra 3.4 you have this knob.
>>
>> # How long the coordinator should wait for truncates to complete
>> # (This can be much longer, because unless auto_snapshot is disabled
>> # we need to flush first so we can snapshot before removing the data.)
>> truncate_request_timeout_in_ms: 6
>>
>>
>> In older versions you can not control when this call will timeout, it is
>> fairly normal that it does!
>>
>>
>> On Wed, Sep 28, 2016 at 12:50 PM, George Sigletos > > wrote:
>>
>>> Hello,
>>>
>>> I keep executing a TRUNCATE command on an empty table and it throws
>>> OperationTimedOut randomly:
>>>
>>> cassandra@cqlsh> truncate test.mytable;
>>> OperationTimedOut: errors={}, last_host=cassiebeta-01
>>> cassandra@cqlsh> truncate test.mytable;
>>> OperationTimedOut: errors={}, last_host=cassiebeta-01
>>>
>>> Having a 3 node cluster running 2.1.14. No connectivity problems. Has
>>> anybody come across the same error?
>>>
>>> Thanks,
>>> George
>>>
>>>
>>
>

Re: TRUNCATE throws OperationTimedOut randomly

2016-09-28 Thread George Sigletos 
Thanks a lot for your reply.

I understand that truncate is an expensive operation. But throwing a
timeout while truncating a table that is already empty?

A workaround is to set a high --request-timeout when connecting. Even 20
seconds is not always enough

Kind regards,
George


On Wed, Sep 28, 2016 at 6:59 PM, Edward Capriolo 
wrote:

> Truncate does a few things (based on version)
>   truncate takes snapshots
>   truncate causes a flush
>   in very old versions truncate causes a schema migration.
>
> In newer versions like cassandra 3.4 you have this knob.
>
> # How long the coordinator should wait for truncates to complete
> # (This can be much longer, because unless auto_snapshot is disabled
> # we need to flush first so we can snapshot before removing the data.)
> truncate_request_timeout_in_ms: 6
>
>
> In older versions you can not control when this call will timeout, it is
> fairly normal that it does!
>
>
> On Wed, Sep 28, 2016 at 12:50 PM, George Sigletos 
> wrote:
>
>> Hello,
>>
>> I keep executing a TRUNCATE command on an empty table and it throws
>> OperationTimedOut randomly:
>>
>> cassandra@cqlsh> truncate test.mytable;
>> OperationTimedOut: errors={}, last_host=cassiebeta-01
>> cassandra@cqlsh> truncate test.mytable;
>> OperationTimedOut: errors={}, last_host=cassiebeta-01
>>
>> Having a 3 node cluster running 2.1.14. No connectivity problems. Has
>> anybody come across the same error?
>>
>> Thanks,
>> George
>>
>>
>

Re: TRUNCATE throws OperationTimedOut randomly

2016-09-28 Thread Edward Capriolo 
Truncate does a few things (based on version)
  truncate takes snapshots
  truncate causes a flush
  in very old versions truncate causes a schema migration.

In newer versions like cassandra 3.4 you have this knob.

# How long the coordinator should wait for truncates to complete
# (This can be much longer, because unless auto_snapshot is disabled
# we need to flush first so we can snapshot before removing the data.)
truncate_request_timeout_in_ms: 6


In older versions you can not control when this call will timeout, it is
fairly normal that it does!


On Wed, Sep 28, 2016 at 12:50 PM, George Sigletos 
wrote:

> Hello,
>
> I keep executing a TRUNCATE command on an empty table and it throws
> OperationTimedOut randomly:
>
> cassandra@cqlsh> truncate test.mytable;
> OperationTimedOut: errors={}, last_host=cassiebeta-01
> cassandra@cqlsh> truncate test.mytable;
> OperationTimedOut: errors={}, last_host=cassiebeta-01
>
> Having a 3 node cluster running 2.1.14. No connectivity problems. Has
> anybody come across the same error?
>
> Thanks,
> George
>
>

TRUNCATE throws OperationTimedOut randomly

2016-09-28 Thread George Sigletos 
Hello,

I keep executing a TRUNCATE command on an empty table and it throws
OperationTimedOut randomly:

cassandra@cqlsh> truncate test.mytable;
OperationTimedOut: errors={}, last_host=cassiebeta-01
cassandra@cqlsh> truncate test.mytable;
OperationTimedOut: errors={}, last_host=cassiebeta-01

Having a 3 node cluster running 2.1.14. No connectivity problems. Has
anybody come across the same error?

Thanks,
George