Re: WriteTimeoutException with LWT after few milliseconds

[Jeff Jirsa]

On 2017-04-19 08:10 (-0700), Sylvain Lebresne  wrote: 
> This is https://issues.apache.org/jira/browse/CASSANDRA-9328 and I'd rather
> not repeat myself here so I'll let you read the details. Let's maybe not
> open another JIRA ticket though since we have this one.
> 

Thanks for the context, Sylvain.

Re: WriteTimeoutException with LWT after few milliseconds

[Sylvain Lebresne]

This is https://issues.apache.org/jira/browse/CASSANDRA-9328 and I'd rather
not repeat myself here so I'll let you read the details. Let's maybe not
open another JIRA ticket though since we have this one.

On Wed, Apr 19, 2017 at 4:29 PM, benjamin roth  wrote:

> Thanks, Jeff!
>
> As soon as I have some spare time I will try to reproduce and open a Jira
> for it.
>
> 2017-04-19 16:27 GMT+02:00 Jeff Jirsa :
>
>>
>>
>> On 2017-04-13 05:13 (-0700), benjamin roth  wrote:
>> > I found out that if the WTEs occur, there was already another process
>> > inserting the same primary key because I found duplicates in some places
>> > that perfectly match the WTE logs.
>> >
>> > Does anybody know, why this throws a WTE instead of returning
>> [applied]' =
>> > false ?
>> > This is quite confusing!
>> >
>>
>> Certainly seems wrong. May want to open a JIRA, especially if it's
>> reproducible. Should mention what version and client you're using.
>>
>>
>

Re: WriteTimeoutException with LWT after few milliseconds

[benjamin roth]

Thanks, Jeff!

As soon as I have some spare time I will try to reproduce and open a Jira
for it.

2017-04-19 16:27 GMT+02:00 Jeff Jirsa :

>
>
> On 2017-04-13 05:13 (-0700), benjamin roth  wrote:
> > I found out that if the WTEs occur, there was already another process
> > inserting the same primary key because I found duplicates in some places
> > that perfectly match the WTE logs.
> >
> > Does anybody know, why this throws a WTE instead of returning [applied]'
> =
> > false ?
> > This is quite confusing!
> >
>
> Certainly seems wrong. May want to open a JIRA, especially if it's
> reproducible. Should mention what version and client you're using.
>
>

Re: WriteTimeoutException with LWT after few milliseconds

[Jeff Jirsa]

On 2017-04-13 05:13 (-0700), benjamin roth  wrote: 
> I found out that if the WTEs occur, there was already another process
> inserting the same primary key because I found duplicates in some places
> that perfectly match the WTE logs.
> 
> Does anybody know, why this throws a WTE instead of returning [applied]' =
> false ?
> This is quite confusing!
> 

Certainly seems wrong. May want to open a JIRA, especially if it's 
reproducible. Should mention what version and client you're using.

Re: WriteTimeoutException with LWT after few milliseconds

[benjamin roth]

I found out that if the WTEs occur, there was already another process
inserting the same primary key because I found duplicates in some places
that perfectly match the WTE logs.

Does anybody know, why this throws a WTE instead of returning [applied]' =
false ?
This is quite confusing!

2017-04-12 17:41 GMT+02:00 Carlos Rolo :

> You can try to use TRACING to debug the situation, but for a LWT to fail
> so fast, the most probable cause is what you stated: "It is possible that
> there are concurrent inserts on the same PK - actually thats the reason why
> I use LWTs." AKA, someone inserted first.
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
> *linkedin.com/in/carlosjuzarterolo
> *
> Mobile: +351 918 918 100 <+351%20918%20918%20100>
> www.pythian.com
>
> On Wed, Apr 12, 2017 at 3:51 PM, Roland Otta 
> wrote:
>
>> sorry .. ignore my comment ...
>>
>> i missed your comment that the record is in the table ...
>>
>> On Wed, 2017-04-12 at 16:48 +0200, Roland Otta wrote:
>>
>> Hi Benjamin,
>>
>> its unlikely that i can assist you .. but nevertheless ... i give it a
>> try ;-)
>>
>> whats your consistency level for the insert?
>> what if one ore more nodes are marked down and proper consistency cant be
>> achieved?
>> of course the error message does not indicate that problem (as it says
>> its a timeout)... but in that case you would get an instant error for
>> inserts. wouldn't you?
>>
>> br,
>> roland
>>
>>
>>
>> On Wed, 2017-04-12 at 15:09 +0200, benjamin roth wrote:
>>
>> Hi folks,
>>
>> Can someone explain why that occurs?
>>
>> Write timeout after 0.006s
>> Query: 'INSERT INTO log_moment_import ("source", "reference", "user_id",
>> "moment_id", "date", "finished") VALUES (3, '1305821272790495', 65675537,
>> 0, '2017-04-12 13:00:51', NULL) IF NOT EXISTS
>> Primary key and parition key is source + reference
>> Message: Operation timed out - received only 1 responses.
>>
>> This appears every now and then in the log. When I check the for the
>> record in the table, it is there.
>> I could explain that, if the WTE occured after the configured write
>> timeout but it happens withing a few milliseconds.
>> Is this caused by lock contention? It is possible that there are
>> concurrent inserts on the same PK - actually thats the reason why I use
>> LWTs.
>>
>> Thanks!
>>
>>
>
> --
>
>
>
>

Re: WriteTimeoutException with LWT after few milliseconds

[benjamin roth]

Hi Roland,

LWTs set consistency level implicitly to SERIAL which requires at least
QUORUM.
No, no node is/was down. If that happens the query will fail with "Could
not achieve consistency level QUORUM ..."

2017-04-12 16:48 GMT+02:00 Roland Otta :

> Hi Benjamin,
>
> its unlikely that i can assist you .. but nevertheless ... i give it a try
> ;-)
>
> whats your consistency level for the insert?
> what if one ore more nodes are marked down and proper consistency cant be
> achieved?
> of course the error message does not indicate that problem (as it says its
> a timeout)... but in that case you would get an instant error for inserts.
> wouldn't you?
>
> br,
> roland
>
>
>
> On Wed, 2017-04-12 at 15:09 +0200, benjamin roth wrote:
>
> Hi folks,
>
> Can someone explain why that occurs?
>
> Write timeout after 0.006s
> Query: 'INSERT INTO log_moment_import ("source", "reference", "user_id",
> "moment_id", "date", "finished") VALUES (3, '1305821272790495', 65675537,
> 0, '2017-04-12 13:00:51', NULL) IF NOT EXISTS
> Primary key and parition key is source + reference
> Message: Operation timed out - received only 1 responses.
>
> This appears every now and then in the log. When I check the for the
> record in the table, it is there.
> I could explain that, if the WTE occured after the configured write
> timeout but it happens withing a few milliseconds.
> Is this caused by lock contention? It is possible that there are
> concurrent inserts on the same PK - actually thats the reason why I use
> LWTs.
>
> Thanks!
>
>

Re: WriteTimeoutException with LWT after few milliseconds

[Carlos Rolo]

You can try to use TRACING to debug the situation, but for a LWT to fail so
fast, the most probable cause is what you stated: "It is possible that
there are concurrent inserts on the same PK - actually thats the reason why
I use LWTs." AKA, someone inserted first.

Regards,

Carlos Juzarte Rolo
Cassandra Consultant / Datastax Certified Architect / Cassandra MVP

Pythian - Love your data

rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
*linkedin.com/in/carlosjuzarterolo
*
Mobile: +351 918 918 100
www.pythian.com

On Wed, Apr 12, 2017 at 3:51 PM, Roland Otta 
wrote:

> sorry .. ignore my comment ...
>
> i missed your comment that the record is in the table ...
>
> On Wed, 2017-04-12 at 16:48 +0200, Roland Otta wrote:
>
> Hi Benjamin,
>
> its unlikely that i can assist you .. but nevertheless ... i give it a try
> ;-)
>
> whats your consistency level for the insert?
> what if one ore more nodes are marked down and proper consistency cant be
> achieved?
> of course the error message does not indicate that problem (as it says its
> a timeout)... but in that case you would get an instant error for inserts.
> wouldn't you?
>
> br,
> roland
>
>
>
> On Wed, 2017-04-12 at 15:09 +0200, benjamin roth wrote:
>
> Hi folks,
>
> Can someone explain why that occurs?
>
> Write timeout after 0.006s
> Query: 'INSERT INTO log_moment_import ("source", "reference", "user_id",
> "moment_id", "date", "finished") VALUES (3, '1305821272790495', 65675537,
> 0, '2017-04-12 13:00:51', NULL) IF NOT EXISTS
> Primary key and parition key is source + reference
> Message: Operation timed out - received only 1 responses.
>
> This appears every now and then in the log. When I check the for the
> record in the table, it is there.
> I could explain that, if the WTE occured after the configured write
> timeout but it happens withing a few milliseconds.
> Is this caused by lock contention? It is possible that there are
> concurrent inserts on the same PK - actually thats the reason why I use
> LWTs.
>
> Thanks!
>
>

-- 


--

Re: WriteTimeoutException with LWT after few milliseconds

[Roland Otta]

sorry .. ignore my comment ...

i missed your comment that the record is in the table ...

On Wed, 2017-04-12 at 16:48 +0200, Roland Otta wrote:
Hi Benjamin,

its unlikely that i can assist you .. but nevertheless ... i give it a try ;-)

whats your consistency level for the insert?
what if one ore more nodes are marked down and proper consistency cant be 
achieved?
of course the error message does not indicate that problem (as it says its a 
timeout)... but in that case you would get an instant error for inserts. 
wouldn't you?

br,
roland



On Wed, 2017-04-12 at 15:09 +0200, benjamin roth wrote:
Hi folks,

Can someone explain why that occurs?

Write timeout after 0.006s
Query: 'INSERT INTO log_moment_import ("source", "reference", "user_id", 
"moment_id", "date", "finished") VALUES (3, '1305821272790495', 65675537, 0, 
'2017-04-12 13:00:51', NULL) IF NOT EXISTS
Primary key and parition key is source + reference
Message: Operation timed out - received only 1 responses.

This appears every now and then in the log. When I check the for the record in 
the table, it is there.
I could explain that, if the WTE occured after the configured write timeout but 
it happens withing a few milliseconds.
Is this caused by lock contention? It is possible that there are concurrent 
inserts on the same PK - actually thats the reason why I use LWTs.

Thanks!

Re: WriteTimeoutException with LWT after few milliseconds

[Roland Otta]

Hi Benjamin,

its unlikely that i can assist you .. but nevertheless ... i give it a try ;-)

whats your consistency level for the insert?
what if one ore more nodes are marked down and proper consistency cant be 
achieved?
of course the error message does not indicate that problem (as it says its a 
timeout)... but in that case you would get an instant error for inserts. 
wouldn't you?

br,
roland



On Wed, 2017-04-12 at 15:09 +0200, benjamin roth wrote:
Hi folks,

Can someone explain why that occurs?

Write timeout after 0.006s
Query: 'INSERT INTO log_moment_import ("source", "reference", "user_id", 
"moment_id", "date", "finished") VALUES (3, '1305821272790495', 65675537, 0, 
'2017-04-12 13:00:51', NULL) IF NOT EXISTS
Primary key and parition key is source + reference
Message: Operation timed out - received only 1 responses.

This appears every now and then in the log. When I check the for the record in 
the table, it is there.
I could explain that, if the WTE occured after the configured write timeout but 
it happens withing a few milliseconds.
Is this caused by lock contention? It is possible that there are concurrent 
inserts on the same PK - actually thats the reason why I use LWTs.

Thanks!

WriteTimeoutException with LWT after few milliseconds

[benjamin roth]

Hi folks,

Can someone explain why that occurs?

Write timeout after 0.006s
Query: 'INSERT INTO log_moment_import ("source", "reference", "user_id",
"moment_id", "date", "finished") VALUES (3, '1305821272790495', 65675537,
0, '2017-04-12 13:00:51', NULL) IF NOT EXISTS
Primary key and parition key is source + reference
Message: Operation timed out - received only 1 responses.

This appears every now and then in the log. When I check the for the record
in the table, it is there.
I could explain that, if the WTE occured after the configured write timeout
but it happens withing a few milliseconds.
Is this caused by lock contention? It is possible that there are concurrent
inserts on the same PK - actually thats the reason why I use LWTs.

Thanks!