[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[
https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001924#comment-15001924
]
Stefan Podkowinski commented on CASSANDRA-9131:
---
Since CASSANDRA-9649 {{state.getTimestamp()}} is now monotonically increasing
for the VM instead of only for each connection. Operating this way should be
pretty safe, as people should be well aware of the importance to keep clocks in
sync across the cluster. Now in that situation I don't really feel comfortable
with moving timestamp generation to the clients, as now potentially much more
servers are affected and need to be monitored for clock skew. The chances will
be much higher to have a single system introduce incorrect timestamp values
that will corrupt your data.
> Defining correct behavior during leap second insertion
> --
>
> Key: CASSANDRA-9131
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9131
> Project: Cassandra
> Issue Type: Bug
> Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue
> Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Jim Witschey
>Priority: Minor
> Fix For: 3.x
>
>
> On Linux platforms, the insertion of a leap second breaks the monotonicity of
> timestamps. This can make values appear to have been inserted into Cassandra
> in a different order than they were. I want to know what behavior is expected
> and desirable for inserts over this discontinuity.
> From a timestamp perspective, an inserted leap second looks like a repeat of
> the previous second:
> {code}
> $ while true ; do echo "`date +%s%N` `date -u`" ; sleep .5 ; done
> 1435708798171327029 Tue Jun 30 23:59:58 UTC 2015
> 1435708798679392477 Tue Jun 30 23:59:58 UTC 2015
> 1435708799187550335 Tue Jun 30 23:59:59 UTC 2015
> 1435708799695670453 Tue Jun 30 23:59:59 UTC 2015
> 1435708799203902068 Tue Jun 30 23:59:59 UTC 2015
> 1435708799712168566 Tue Jun 30 23:59:59 UTC 2015
> 1435708800220473932 Wed Jul 1 00:00:00 UTC 2015
> 1435708800728908190 Wed Jul 1 00:00:00 UTC 2015
> 1435708801237611983 Wed Jul 1 00:00:01 UTC 2015
> 1435708801746251996 Wed Jul 1 00:00:01 UTC 2015
> {code}
> Note that 23:59:59 repeats itself, and that the timestamps increase during
> the first time through, then step back down to the beginning of the second
> and increase again.
> As a result, the timestamps on values inserted during these seconds will be
> out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and
> synced them to shortly before the leap second would be inserted. During the
> insertion of the leap second, I ran a test with logic something like:
> {code}
> simple_insert = session.prepare(
> 'INSERT INTO test (foo, bar) VALUES (?, ?);')
> for i in itertools.count():
> # stop after midnight
> now = datetime.utcnow()
> last_midnight = now.replace(hour=0, minute=0,
> second=0, microsecond=0)
> seconds_since_midnight = (now - last_midnight).total_seconds()
> if 5 <= seconds_since_midnight <= 15:
> break
> session.execute(simple_insert, [i, i])
> result = session.execute("SELECT bar, WRITETIME(bar) FROM test;")
> {code}
> EDIT: This behavior occurs with server-generated timestamps; in this
> particular test, I set {{use_client_timestamp}} to {{False}}.
> Under normal circumstances, the values and writetimes would increase
> together, but when inserted over the leap second, they don't. These {{value,
> writetime}} pairs are sorted by writetime:
> {code}
> (582, 1435708799285000)
> (579, 1435708799339000)
> (583, 1435708799593000)
> (580, 1435708799643000)
> (584, 1435708799897000)
> (581, 1435708799958000)
> {code}
> The values were inserted in increasing order, but their writetimes are in a
> different order because of the repeated second. During the first instance of
> 23:59:59, the values 579, 580, and 581 were inserted at the beginning,
> middle, and end of the second. During the leap second, which is also
> 23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and
> end of the second. However, since the two seconds are the same second, they
> appear interleaved with respect to timestamps, as shown above.
> So, should I consider this behavior correct? If not, how should Cassandra
> correctly handle the discontinuity introduced by the insertion of a leap
> second?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[
https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527114#comment-14527114
]
Ryan McGuire commented on CASSANDRA-9131:
-
Assigned to [~slebresne] to provide protocol spec.
Defining correct behavior during leap second insertion
--
Key: CASSANDRA-9131
URL: https://issues.apache.org/jira/browse/CASSANDRA-9131
Project: Cassandra
Issue Type: Bug
Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue
Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Reporter: Jim Witschey
Assignee: Sylvain Lebresne
On Linux platforms, the insertion of a leap second breaks the monotonicity of
timestamps. This can make values appear to have been inserted into Cassandra
in a different order than they were. I want to know what behavior is expected
and desirable for inserts over this discontinuity.
From a timestamp perspective, an inserted leap second looks like a repeat of
the previous second:
{code}
$ while true ; do echo `date +%s%N` `date -u` ; sleep .5 ; done
1435708798171327029 Tue Jun 30 23:59:58 UTC 2015
1435708798679392477 Tue Jun 30 23:59:58 UTC 2015
1435708799187550335 Tue Jun 30 23:59:59 UTC 2015
1435708799695670453 Tue Jun 30 23:59:59 UTC 2015
1435708799203902068 Tue Jun 30 23:59:59 UTC 2015
1435708799712168566 Tue Jun 30 23:59:59 UTC 2015
1435708800220473932 Wed Jul 1 00:00:00 UTC 2015
1435708800728908190 Wed Jul 1 00:00:00 UTC 2015
1435708801237611983 Wed Jul 1 00:00:01 UTC 2015
1435708801746251996 Wed Jul 1 00:00:01 UTC 2015
{code}
Note that 23:59:59 repeats itself, and that the timestamps increase during
the first time through, then step back down to the beginning of the second
and increase again.
As a result, the timestamps on values inserted during these seconds will be
out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and
synced them to shortly before the leap second would be inserted. During the
insertion of the leap second, I ran a test with logic something like:
{code}
simple_insert = session.prepare(
'INSERT INTO test (foo, bar) VALUES (?, ?);')
for i in itertools.count():
# stop after midnight
now = datetime.utcnow()
last_midnight = now.replace(hour=0, minute=0,
second=0, microsecond=0)
seconds_since_midnight = (now - last_midnight).total_seconds()
if 5 = seconds_since_midnight = 15:
break
session.execute(simple_insert, [i, i])
result = session.execute(SELECT bar, WRITETIME(bar) FROM test;)
{code}
EDIT: This behavior occurs with server-generated timestamps; in this
particular test, I set {{use_client_timestamp}} to {{False}}.
Under normal circumstances, the values and writetimes would increase
together, but when inserted over the leap second, they don't. These {{value,
writetime}} pairs are sorted by writetime:
{code}
(582, 1435708799285000)
(579, 1435708799339000)
(583, 1435708799593000)
(580, 1435708799643000)
(584, 1435708799897000)
(581, 1435708799958000)
{code}
The values were inserted in increasing order, but their writetimes are in a
different order because of the repeated second. During the first instance of
23:59:59, the values 579, 580, and 581 were inserted at the beginning,
middle, and end of the second. During the leap second, which is also
23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and
end of the second. However, since the two seconds are the same second, they
appear interleaved with respect to timestamps, as shown above.
So, should I consider this behavior correct? If not, how should Cassandra
correctly handle the discontinuity introduced by the insertion of a leap
second?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[
https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499963#comment-14499963
]
Benedict commented on CASSANDRA-9131:
-
I can't recall entirely, but I don't think that was quite the way the
conversation resolved. It's not exactly expected behaviour on either side,
but obviously there's nothing we can do about clients that are subjected to
this bug, and we intend to deprecate server-side timestamps (if perhaps never
eliminate them entirely). So the question is
# if we should fix server side timestamps by making the clock universally
monotonically increasing (as opposed to only per-client connection) which would
at least somewhat mitigate this problem (but not eliminate it entirely, for any
sequence of inserts hitting multiple servers)
# if we should update the client protocol spec/docs to make clear that this
problem exists, and that clients are expected to work around it
## if we should offer some sample code, and work with the Java Driver team to
ensure this problem doesn't affect it
Defining correct behavior during leap second insertion
--
Key: CASSANDRA-9131
URL: https://issues.apache.org/jira/browse/CASSANDRA-9131
Project: Cassandra
Issue Type: Bug
Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue
Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Reporter: Jim Witschey
Assignee: Jim Witschey
On Linux platforms, the insertion of a leap second breaks the monotonicity of
timestamps. This can make values appear to have been inserted into Cassandra
in a different order than they were. I want to know what behavior is expected
and desirable for inserts over this discontinuity.
From a timestamp perspective, an inserted leap second looks like a repeat of
the previous second:
{code}
$ while true ; do echo `date +%s%N` `date -u` ; sleep .5 ; done
1435708798171327029 Tue Jun 30 23:59:58 UTC 2015
1435708798679392477 Tue Jun 30 23:59:58 UTC 2015
1435708799187550335 Tue Jun 30 23:59:59 UTC 2015
1435708799695670453 Tue Jun 30 23:59:59 UTC 2015
1435708799203902068 Tue Jun 30 23:59:59 UTC 2015
1435708799712168566 Tue Jun 30 23:59:59 UTC 2015
1435708800220473932 Wed Jul 1 00:00:00 UTC 2015
1435708800728908190 Wed Jul 1 00:00:00 UTC 2015
1435708801237611983 Wed Jul 1 00:00:01 UTC 2015
1435708801746251996 Wed Jul 1 00:00:01 UTC 2015
{code}
Note that 23:59:59 repeats itself, and that the timestamps increase during
the first time through, then step back down to the beginning of the second
and increase again.
As a result, the timestamps on values inserted during these seconds will be
out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and
synced them to shortly before the leap second would be inserted. During the
insertion of the leap second, I ran a test with logic something like:
{code}
simple_insert = session.prepare(
'INSERT INTO test (foo, bar) VALUES (?, ?);')
for i in itertools.count():
# stop after midnight
now = datetime.utcnow()
last_midnight = now.replace(hour=0, minute=0,
second=0, microsecond=0)
seconds_since_midnight = (now - last_midnight).total_seconds()
if 5 = seconds_since_midnight = 15:
break
session.execute(simple_insert, [i, i])
result = session.execute(SELECT bar, WRITETIME(bar) FROM test;)
{code}
EDIT: This behavior occurs with server-generated timestamps; in this
particular test, I set {{use_client_timestamp}} to {{False}}.
Under normal circumstances, the values and writetimes would increase
together, but when inserted over the leap second, they don't. These {{value,
writetime}} pairs are sorted by writetime:
{code}
(582, 1435708799285000)
(579, 1435708799339000)
(583, 1435708799593000)
(580, 1435708799643000)
(584, 1435708799897000)
(581, 1435708799958000)
{code}
The values were inserted in increasing order, but their writetimes are in a
different order because of the repeated second. During the first instance of
23:59:59, the values 579, 580, and 581 were inserted at the beginning,
middle, and end of the second. During the leap second, which is also
23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and
end of the second. However, since the two seconds are the same second, they
appear interleaved with respect to timestamps, as shown above.
So, should I consider this behavior correct? If not, how should Cassandra
correctly handle the discontinuity introduced by the insertion of a leap
second?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[
https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1453#comment-1453
]
Robert Stupp commented on CASSANDRA-9131:
-
Hm. I'm not convinced of client side timestamps. IMO maintaining timestamps on
the clients (either these are ”fat clients” or other servers) just moves the
problem to an area that might not be sensitive for correct system wall clock
(e.g. NTP). I've seen operations handling Win and Linux environments completely
separate - but both worlds with a constant time drift of several minutes (not
funny). I'm not completely against client provided timestamps - but would
prefer to make that an optional feature (i.e. move {{TIMESTAMP xx}} to the
protocol).
TL;DR just want to throw in an idea: We could encapsulate
{{System.currentTimeMillis()}} - if we detect that the clock went backwards, we
slow down ”our system clock” and vice versa if the clock moves forwards.
Defining correct behavior during leap second insertion
--
Key: CASSANDRA-9131
URL: https://issues.apache.org/jira/browse/CASSANDRA-9131
Project: Cassandra
Issue Type: Bug
Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue
Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Reporter: Jim Witschey
Assignee: Jim Witschey
On Linux platforms, the insertion of a leap second breaks the monotonicity of
timestamps. This can make values appear to have been inserted into Cassandra
in a different order than they were. I want to know what behavior is expected
and desirable for inserts over this discontinuity.
From a timestamp perspective, an inserted leap second looks like a repeat of
the previous second:
{code}
$ while true ; do echo `date +%s%N` `date -u` ; sleep .5 ; done
1435708798171327029 Tue Jun 30 23:59:58 UTC 2015
1435708798679392477 Tue Jun 30 23:59:58 UTC 2015
1435708799187550335 Tue Jun 30 23:59:59 UTC 2015
1435708799695670453 Tue Jun 30 23:59:59 UTC 2015
1435708799203902068 Tue Jun 30 23:59:59 UTC 2015
1435708799712168566 Tue Jun 30 23:59:59 UTC 2015
1435708800220473932 Wed Jul 1 00:00:00 UTC 2015
1435708800728908190 Wed Jul 1 00:00:00 UTC 2015
1435708801237611983 Wed Jul 1 00:00:01 UTC 2015
1435708801746251996 Wed Jul 1 00:00:01 UTC 2015
{code}
Note that 23:59:59 repeats itself, and that the timestamps increase during
the first time through, then step back down to the beginning of the second
and increase again.
As a result, the timestamps on values inserted during these seconds will be
out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and
synced them to shortly before the leap second would be inserted. During the
insertion of the leap second, I ran a test with logic something like:
{code}
simple_insert = session.prepare(
'INSERT INTO test (foo, bar) VALUES (?, ?);')
for i in itertools.count():
# stop after midnight
now = datetime.utcnow()
last_midnight = now.replace(hour=0, minute=0,
second=0, microsecond=0)
seconds_since_midnight = (now - last_midnight).total_seconds()
if 5 = seconds_since_midnight = 15:
break
session.execute(simple_insert, [i, i])
result = session.execute(SELECT bar, WRITETIME(bar) FROM test;)
{code}
EDIT: This behavior occurs with server-generated timestamps; in this
particular test, I set {{use_client_timestamp}} to {{False}}.
Under normal circumstances, the values and writetimes would increase
together, but when inserted over the leap second, they don't. These {{value,
writetime}} pairs are sorted by writetime:
{code}
(582, 1435708799285000)
(579, 1435708799339000)
(583, 1435708799593000)
(580, 1435708799643000)
(584, 1435708799897000)
(581, 1435708799958000)
{code}
The values were inserted in increasing order, but their writetimes are in a
different order because of the repeated second. During the first instance of
23:59:59, the values 579, 580, and 581 were inserted at the beginning,
middle, and end of the second. During the leap second, which is also
23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and
end of the second. However, since the two seconds are the same second, they
appear interleaved with respect to timestamps, as shown above.
So, should I consider this behavior correct? If not, how should Cassandra
correctly handle the discontinuity introduced by the insertion of a leap
second?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[
https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500011#comment-14500011
]
Benedict commented on CASSANDRA-9131:
-
The problem with server-side timestamps is unsafe retry by clients in the event
of a failure. CASSANDRA-6106 is a reference point, in that this provided both a
wrapper around microsecond resolution as well as a staggered application of
shifts in system clock time (also ensuring it never went backwards).
Defining correct behavior during leap second insertion
--
Key: CASSANDRA-9131
URL: https://issues.apache.org/jira/browse/CASSANDRA-9131
Project: Cassandra
Issue Type: Bug
Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue
Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Reporter: Jim Witschey
Assignee: Jim Witschey
On Linux platforms, the insertion of a leap second breaks the monotonicity of
timestamps. This can make values appear to have been inserted into Cassandra
in a different order than they were. I want to know what behavior is expected
and desirable for inserts over this discontinuity.
From a timestamp perspective, an inserted leap second looks like a repeat of
the previous second:
{code}
$ while true ; do echo `date +%s%N` `date -u` ; sleep .5 ; done
1435708798171327029 Tue Jun 30 23:59:58 UTC 2015
1435708798679392477 Tue Jun 30 23:59:58 UTC 2015
1435708799187550335 Tue Jun 30 23:59:59 UTC 2015
1435708799695670453 Tue Jun 30 23:59:59 UTC 2015
1435708799203902068 Tue Jun 30 23:59:59 UTC 2015
1435708799712168566 Tue Jun 30 23:59:59 UTC 2015
1435708800220473932 Wed Jul 1 00:00:00 UTC 2015
1435708800728908190 Wed Jul 1 00:00:00 UTC 2015
1435708801237611983 Wed Jul 1 00:00:01 UTC 2015
1435708801746251996 Wed Jul 1 00:00:01 UTC 2015
{code}
Note that 23:59:59 repeats itself, and that the timestamps increase during
the first time through, then step back down to the beginning of the second
and increase again.
As a result, the timestamps on values inserted during these seconds will be
out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and
synced them to shortly before the leap second would be inserted. During the
insertion of the leap second, I ran a test with logic something like:
{code}
simple_insert = session.prepare(
'INSERT INTO test (foo, bar) VALUES (?, ?);')
for i in itertools.count():
# stop after midnight
now = datetime.utcnow()
last_midnight = now.replace(hour=0, minute=0,
second=0, microsecond=0)
seconds_since_midnight = (now - last_midnight).total_seconds()
if 5 = seconds_since_midnight = 15:
break
session.execute(simple_insert, [i, i])
result = session.execute(SELECT bar, WRITETIME(bar) FROM test;)
{code}
EDIT: This behavior occurs with server-generated timestamps; in this
particular test, I set {{use_client_timestamp}} to {{False}}.
Under normal circumstances, the values and writetimes would increase
together, but when inserted over the leap second, they don't. These {{value,
writetime}} pairs are sorted by writetime:
{code}
(582, 1435708799285000)
(579, 1435708799339000)
(583, 1435708799593000)
(580, 1435708799643000)
(584, 1435708799897000)
(581, 1435708799958000)
{code}
The values were inserted in increasing order, but their writetimes are in a
different order because of the repeated second. During the first instance of
23:59:59, the values 579, 580, and 581 were inserted at the beginning,
middle, and end of the second. During the leap second, which is also
23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and
end of the second. However, since the two seconds are the same second, they
appear interleaved with respect to timestamps, as shown above.
So, should I consider this behavior correct? If not, how should Cassandra
correctly handle the discontinuity introduced by the insertion of a leap
second?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[
https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500259#comment-14500259
]
Sylvain Lebresne commented on CASSANDRA-9131:
-
bq. if we should offer some sample code
Yes, I suspect it would be appreciated if we were to provide some kind of
example/reference algorithm for this (maybe just in form of some pseudo-code),
probably in the protocol spec documentation.
Defining correct behavior during leap second insertion
--
Key: CASSANDRA-9131
URL: https://issues.apache.org/jira/browse/CASSANDRA-9131
Project: Cassandra
Issue Type: Bug
Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue
Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Reporter: Jim Witschey
Assignee: Jim Witschey
On Linux platforms, the insertion of a leap second breaks the monotonicity of
timestamps. This can make values appear to have been inserted into Cassandra
in a different order than they were. I want to know what behavior is expected
and desirable for inserts over this discontinuity.
From a timestamp perspective, an inserted leap second looks like a repeat of
the previous second:
{code}
$ while true ; do echo `date +%s%N` `date -u` ; sleep .5 ; done
1435708798171327029 Tue Jun 30 23:59:58 UTC 2015
1435708798679392477 Tue Jun 30 23:59:58 UTC 2015
1435708799187550335 Tue Jun 30 23:59:59 UTC 2015
1435708799695670453 Tue Jun 30 23:59:59 UTC 2015
1435708799203902068 Tue Jun 30 23:59:59 UTC 2015
1435708799712168566 Tue Jun 30 23:59:59 UTC 2015
1435708800220473932 Wed Jul 1 00:00:00 UTC 2015
1435708800728908190 Wed Jul 1 00:00:00 UTC 2015
1435708801237611983 Wed Jul 1 00:00:01 UTC 2015
1435708801746251996 Wed Jul 1 00:00:01 UTC 2015
{code}
Note that 23:59:59 repeats itself, and that the timestamps increase during
the first time through, then step back down to the beginning of the second
and increase again.
As a result, the timestamps on values inserted during these seconds will be
out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and
synced them to shortly before the leap second would be inserted. During the
insertion of the leap second, I ran a test with logic something like:
{code}
simple_insert = session.prepare(
'INSERT INTO test (foo, bar) VALUES (?, ?);')
for i in itertools.count():
# stop after midnight
now = datetime.utcnow()
last_midnight = now.replace(hour=0, minute=0,
second=0, microsecond=0)
seconds_since_midnight = (now - last_midnight).total_seconds()
if 5 = seconds_since_midnight = 15:
break
session.execute(simple_insert, [i, i])
result = session.execute(SELECT bar, WRITETIME(bar) FROM test;)
{code}
EDIT: This behavior occurs with server-generated timestamps; in this
particular test, I set {{use_client_timestamp}} to {{False}}.
Under normal circumstances, the values and writetimes would increase
together, but when inserted over the leap second, they don't. These {{value,
writetime}} pairs are sorted by writetime:
{code}
(582, 1435708799285000)
(579, 1435708799339000)
(583, 1435708799593000)
(580, 1435708799643000)
(584, 1435708799897000)
(581, 1435708799958000)
{code}
The values were inserted in increasing order, but their writetimes are in a
different order because of the repeated second. During the first instance of
23:59:59, the values 579, 580, and 581 were inserted at the beginning,
middle, and end of the second. During the leap second, which is also
23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and
end of the second. However, since the two seconds are the same second, they
appear interleaved with respect to timestamps, as shown above.
So, should I consider this behavior correct? If not, how should Cassandra
correctly handle the discontinuity introduced by the insertion of a leap
second?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[
https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500058#comment-14500058
]
Robert Stupp commented on CASSANDRA-9131:
-
bq. problem with server-side timestamps is unsafe retry by clients in the event
of a failure.
got it
Defining correct behavior during leap second insertion
--
Key: CASSANDRA-9131
URL: https://issues.apache.org/jira/browse/CASSANDRA-9131
Project: Cassandra
Issue Type: Bug
Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue
Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Reporter: Jim Witschey
Assignee: Jim Witschey
On Linux platforms, the insertion of a leap second breaks the monotonicity of
timestamps. This can make values appear to have been inserted into Cassandra
in a different order than they were. I want to know what behavior is expected
and desirable for inserts over this discontinuity.
From a timestamp perspective, an inserted leap second looks like a repeat of
the previous second:
{code}
$ while true ; do echo `date +%s%N` `date -u` ; sleep .5 ; done
1435708798171327029 Tue Jun 30 23:59:58 UTC 2015
1435708798679392477 Tue Jun 30 23:59:58 UTC 2015
1435708799187550335 Tue Jun 30 23:59:59 UTC 2015
1435708799695670453 Tue Jun 30 23:59:59 UTC 2015
1435708799203902068 Tue Jun 30 23:59:59 UTC 2015
1435708799712168566 Tue Jun 30 23:59:59 UTC 2015
1435708800220473932 Wed Jul 1 00:00:00 UTC 2015
1435708800728908190 Wed Jul 1 00:00:00 UTC 2015
1435708801237611983 Wed Jul 1 00:00:01 UTC 2015
1435708801746251996 Wed Jul 1 00:00:01 UTC 2015
{code}
Note that 23:59:59 repeats itself, and that the timestamps increase during
the first time through, then step back down to the beginning of the second
and increase again.
As a result, the timestamps on values inserted during these seconds will be
out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and
synced them to shortly before the leap second would be inserted. During the
insertion of the leap second, I ran a test with logic something like:
{code}
simple_insert = session.prepare(
'INSERT INTO test (foo, bar) VALUES (?, ?);')
for i in itertools.count():
# stop after midnight
now = datetime.utcnow()
last_midnight = now.replace(hour=0, minute=0,
second=0, microsecond=0)
seconds_since_midnight = (now - last_midnight).total_seconds()
if 5 = seconds_since_midnight = 15:
break
session.execute(simple_insert, [i, i])
result = session.execute(SELECT bar, WRITETIME(bar) FROM test;)
{code}
EDIT: This behavior occurs with server-generated timestamps; in this
particular test, I set {{use_client_timestamp}} to {{False}}.
Under normal circumstances, the values and writetimes would increase
together, but when inserted over the leap second, they don't. These {{value,
writetime}} pairs are sorted by writetime:
{code}
(582, 1435708799285000)
(579, 1435708799339000)
(583, 1435708799593000)
(580, 1435708799643000)
(584, 1435708799897000)
(581, 1435708799958000)
{code}
The values were inserted in increasing order, but their writetimes are in a
different order because of the repeated second. During the first instance of
23:59:59, the values 579, 580, and 581 were inserted at the beginning,
middle, and end of the second. During the leap second, which is also
23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and
end of the second. However, since the two seconds are the same second, they
appear interleaved with respect to timestamps, as shown above.
So, should I consider this behavior correct? If not, how should Cassandra
correctly handle the discontinuity introduced by the insertion of a leap
second?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[
https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500186#comment-14500186
]
Benedict commented on CASSANDRA-9131:
-
[~andrew.tolbert]: if you feel like taking a look at CASSANDRA-6106, this could
be a useful approach for the java-driver (and be adapted to other drivers).
Whether or not it uses the microsecond time is kind of irrelevant to the point
at hand (although potentially also helpful in itself), but the approach to
staggering the time corrections is very applicable. This would prevent the
only have fewer than 1000 inserts in a leap second problem, because the 1s
shift backwards in time would be spread over the proceeding minute, with each
second taking around 20ms longer to elapse than they otherwise would. Either my
or Sylvain's method of updating the clock time would suffice, and be a
tremendous improvement to behaviour here in the drivers.
Defining correct behavior during leap second insertion
--
Key: CASSANDRA-9131
URL: https://issues.apache.org/jira/browse/CASSANDRA-9131
Project: Cassandra
Issue Type: Bug
Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue
Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Reporter: Jim Witschey
Assignee: Jim Witschey
On Linux platforms, the insertion of a leap second breaks the monotonicity of
timestamps. This can make values appear to have been inserted into Cassandra
in a different order than they were. I want to know what behavior is expected
and desirable for inserts over this discontinuity.
From a timestamp perspective, an inserted leap second looks like a repeat of
the previous second:
{code}
$ while true ; do echo `date +%s%N` `date -u` ; sleep .5 ; done
1435708798171327029 Tue Jun 30 23:59:58 UTC 2015
1435708798679392477 Tue Jun 30 23:59:58 UTC 2015
1435708799187550335 Tue Jun 30 23:59:59 UTC 2015
1435708799695670453 Tue Jun 30 23:59:59 UTC 2015
1435708799203902068 Tue Jun 30 23:59:59 UTC 2015
1435708799712168566 Tue Jun 30 23:59:59 UTC 2015
1435708800220473932 Wed Jul 1 00:00:00 UTC 2015
1435708800728908190 Wed Jul 1 00:00:00 UTC 2015
1435708801237611983 Wed Jul 1 00:00:01 UTC 2015
1435708801746251996 Wed Jul 1 00:00:01 UTC 2015
{code}
Note that 23:59:59 repeats itself, and that the timestamps increase during
the first time through, then step back down to the beginning of the second
and increase again.
As a result, the timestamps on values inserted during these seconds will be
out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and
synced them to shortly before the leap second would be inserted. During the
insertion of the leap second, I ran a test with logic something like:
{code}
simple_insert = session.prepare(
'INSERT INTO test (foo, bar) VALUES (?, ?);')
for i in itertools.count():
# stop after midnight
now = datetime.utcnow()
last_midnight = now.replace(hour=0, minute=0,
second=0, microsecond=0)
seconds_since_midnight = (now - last_midnight).total_seconds()
if 5 = seconds_since_midnight = 15:
break
session.execute(simple_insert, [i, i])
result = session.execute(SELECT bar, WRITETIME(bar) FROM test;)
{code}
EDIT: This behavior occurs with server-generated timestamps; in this
particular test, I set {{use_client_timestamp}} to {{False}}.
Under normal circumstances, the values and writetimes would increase
together, but when inserted over the leap second, they don't. These {{value,
writetime}} pairs are sorted by writetime:
{code}
(582, 1435708799285000)
(579, 1435708799339000)
(583, 1435708799593000)
(580, 1435708799643000)
(584, 1435708799897000)
(581, 1435708799958000)
{code}
The values were inserted in increasing order, but their writetimes are in a
different order because of the repeated second. During the first instance of
23:59:59, the values 579, 580, and 581 were inserted at the beginning,
middle, and end of the second. During the leap second, which is also
23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and
end of the second. However, since the two seconds are the same second, they
appear interleaved with respect to timestamps, as shown above.
So, should I consider this behavior correct? If not, how should Cassandra
correctly handle the discontinuity introduced by the insertion of a leap
second?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[
https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500443#comment-14500443
]
Andy Tolbert commented on CASSANDRA-9131:
-
[~benedict], thanks. I agree if we could have a client timestamp
implementation in the drivers that is monotonically increasing while avoiding
the possibility of having operations with the same timestamp would be ideal. I
agree with [~slebresne] that a reference algorithm would be very helpful so all
drivers implement this in a specific way. I'll bring up this topic to the team.
I've seen it brought up in a number of issues with differing opinions, is the
current perspective that going forward client-provided timestamps will be
preferred over server timestamps? Depending on the perspective I can see it
being more important for clients to properly implement this. The current
behavior in all the drivers to my knowledge is that using client timestamps has
to be explicit and only the python and java drivers have a way to enable it for
all queries.
Defining correct behavior during leap second insertion
--
Key: CASSANDRA-9131
URL: https://issues.apache.org/jira/browse/CASSANDRA-9131
Project: Cassandra
Issue Type: Bug
Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue
Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Reporter: Jim Witschey
Assignee: Jim Witschey
On Linux platforms, the insertion of a leap second breaks the monotonicity of
timestamps. This can make values appear to have been inserted into Cassandra
in a different order than they were. I want to know what behavior is expected
and desirable for inserts over this discontinuity.
From a timestamp perspective, an inserted leap second looks like a repeat of
the previous second:
{code}
$ while true ; do echo `date +%s%N` `date -u` ; sleep .5 ; done
1435708798171327029 Tue Jun 30 23:59:58 UTC 2015
1435708798679392477 Tue Jun 30 23:59:58 UTC 2015
1435708799187550335 Tue Jun 30 23:59:59 UTC 2015
1435708799695670453 Tue Jun 30 23:59:59 UTC 2015
1435708799203902068 Tue Jun 30 23:59:59 UTC 2015
1435708799712168566 Tue Jun 30 23:59:59 UTC 2015
1435708800220473932 Wed Jul 1 00:00:00 UTC 2015
1435708800728908190 Wed Jul 1 00:00:00 UTC 2015
1435708801237611983 Wed Jul 1 00:00:01 UTC 2015
1435708801746251996 Wed Jul 1 00:00:01 UTC 2015
{code}
Note that 23:59:59 repeats itself, and that the timestamps increase during
the first time through, then step back down to the beginning of the second
and increase again.
As a result, the timestamps on values inserted during these seconds will be
out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and
synced them to shortly before the leap second would be inserted. During the
insertion of the leap second, I ran a test with logic something like:
{code}
simple_insert = session.prepare(
'INSERT INTO test (foo, bar) VALUES (?, ?);')
for i in itertools.count():
# stop after midnight
now = datetime.utcnow()
last_midnight = now.replace(hour=0, minute=0,
second=0, microsecond=0)
seconds_since_midnight = (now - last_midnight).total_seconds()
if 5 = seconds_since_midnight = 15:
break
session.execute(simple_insert, [i, i])
result = session.execute(SELECT bar, WRITETIME(bar) FROM test;)
{code}
EDIT: This behavior occurs with server-generated timestamps; in this
particular test, I set {{use_client_timestamp}} to {{False}}.
Under normal circumstances, the values and writetimes would increase
together, but when inserted over the leap second, they don't. These {{value,
writetime}} pairs are sorted by writetime:
{code}
(582, 1435708799285000)
(579, 1435708799339000)
(583, 1435708799593000)
(580, 1435708799643000)
(584, 1435708799897000)
(581, 1435708799958000)
{code}
The values were inserted in increasing order, but their writetimes are in a
different order because of the repeated second. During the first instance of
23:59:59, the values 579, 580, and 581 were inserted at the beginning,
middle, and end of the second. During the leap second, which is also
23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and
end of the second. However, since the two seconds are the same second, they
appear interleaved with respect to timestamps, as shown above.
So, should I consider this behavior correct? If not, how should Cassandra
correctly handle the discontinuity introduced by the insertion of a leap
second?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[
https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500127#comment-14500127
]
Andy Tolbert commented on CASSANDRA-9131:
-
{quote}
if we should update the client protocol spec/docs to make clear that this
problem exists, and that clients are expected to work around it
if we should offer some sample code, and work with the Java Driver team to
ensure this problem doesn't affect it
{quote}
This is something I'm actively looking at on the drivers side for each driver.
As of now only the python-driver and java-driver have a mechanism to enable
automatically set client timestamps. The other drivers that support 2.1 have a
mechanism of the client specifying the timestamp (they could also use 'USING
TIMESTAMP' i suppose), so it will be up to user / time implementation. Most of
the drivers will have an active means of setting client timestamp through API
by June 30th 2015.
* python-driver: [Session
use_client_timestamp|http://datastax.github.io/python-driver/api/cassandra/cluster.html?highlight=timestamp#cassandra.cluster.Session.use_client_timestamp]
not monotonic, uses time.time().
* java-driver:
[TimestampGenerator|http://www.datastax.com/drivers/java/2.1/com/datastax/driver/core/TimestampGenerator.html].
All client implementations are monotonic, but client can provide their own.
If 999 entries for same millisecond, will reloop over the same millisecond,
so its not completely monotonic. Possible problem if 1000 entries during
leap second
([code|https://github.com/datastax/java-driver/blob/2.1/driver-core/src/main/java/com/datastax/driver/core/AbstractTimestampGenerator.java])
Defining correct behavior during leap second insertion
--
Key: CASSANDRA-9131
URL: https://issues.apache.org/jira/browse/CASSANDRA-9131
Project: Cassandra
Issue Type: Bug
Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue
Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Reporter: Jim Witschey
Assignee: Jim Witschey
On Linux platforms, the insertion of a leap second breaks the monotonicity of
timestamps. This can make values appear to have been inserted into Cassandra
in a different order than they were. I want to know what behavior is expected
and desirable for inserts over this discontinuity.
From a timestamp perspective, an inserted leap second looks like a repeat of
the previous second:
{code}
$ while true ; do echo `date +%s%N` `date -u` ; sleep .5 ; done
1435708798171327029 Tue Jun 30 23:59:58 UTC 2015
1435708798679392477 Tue Jun 30 23:59:58 UTC 2015
1435708799187550335 Tue Jun 30 23:59:59 UTC 2015
1435708799695670453 Tue Jun 30 23:59:59 UTC 2015
1435708799203902068 Tue Jun 30 23:59:59 UTC 2015
1435708799712168566 Tue Jun 30 23:59:59 UTC 2015
1435708800220473932 Wed Jul 1 00:00:00 UTC 2015
1435708800728908190 Wed Jul 1 00:00:00 UTC 2015
1435708801237611983 Wed Jul 1 00:00:01 UTC 2015
1435708801746251996 Wed Jul 1 00:00:01 UTC 2015
{code}
Note that 23:59:59 repeats itself, and that the timestamps increase during
the first time through, then step back down to the beginning of the second
and increase again.
As a result, the timestamps on values inserted during these seconds will be
out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and
synced them to shortly before the leap second would be inserted. During the
insertion of the leap second, I ran a test with logic something like:
{code}
simple_insert = session.prepare(
'INSERT INTO test (foo, bar) VALUES (?, ?);')
for i in itertools.count():
# stop after midnight
now = datetime.utcnow()
last_midnight = now.replace(hour=0, minute=0,
second=0, microsecond=0)
seconds_since_midnight = (now - last_midnight).total_seconds()
if 5 = seconds_since_midnight = 15:
break
session.execute(simple_insert, [i, i])
result = session.execute(SELECT bar, WRITETIME(bar) FROM test;)
{code}
EDIT: This behavior occurs with server-generated timestamps; in this
particular test, I set {{use_client_timestamp}} to {{False}}.
Under normal circumstances, the values and writetimes would increase
together, but when inserted over the leap second, they don't. These {{value,
writetime}} pairs are sorted by writetime:
{code}
(582, 1435708799285000)
(579, 1435708799339000)
(583, 1435708799593000)
(580, 1435708799643000)
(584, 1435708799897000)
(581, 1435708799958000)
{code}
The values were inserted in increasing order, but their writetimes are in a
different order because of the repeated second. During the first instance of
23:59:59, the values 579, 580, and 581 were inserted at the beginning,
middle, and end of the second. During the
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[
https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484991#comment-14484991
]
Benedict commented on CASSANDRA-9131:
-
Huh. I guess I misunderstood Unix time until now. The problem we're going to
have here is that server-side timestamps are being deprecated, and this problem
is being pushed onto clients so that they can also perform safe retry of
non-idempotent writes (like counters). CASSANDRA-6106 was deprecated and would
have resolved this particular problem by ensuring the clock used by Cassandra
was monotonically increasing. I'm not sure I would want a solution of just
making the clock monotonically increasing, as we would get some potentially
problematic bunching during steps backwards/forwards (6106 attempts to spread
any movement in time across a period of a few seconds). I'm not sure we have
any other good solution to this, on either the client or server side..
Defining correct behavior during leap second insertion
--
Key: CASSANDRA-9131
URL: https://issues.apache.org/jira/browse/CASSANDRA-9131
Project: Cassandra
Issue Type: Bug
Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue
Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Reporter: Jim Witschey
On Linux platforms, the insertion of a leap second breaks the monotonicity of
timestamps. This can make values appear to have been inserted into Cassandra
in a different order than they were. I want to know what behavior is expected
and desirable for inserts over this discontinuity.
From a timestamp perspective, an inserted leap second looks like a repeat of
the previous second:
{code}
$ while true ; do echo `date +%s%N` `date -u` ; sleep .5 ; done
1435708798171327029 Tue Jun 30 23:59:58 UTC 2015
1435708798679392477 Tue Jun 30 23:59:58 UTC 2015
1435708799187550335 Tue Jun 30 23:59:59 UTC 2015
1435708799695670453 Tue Jun 30 23:59:59 UTC 2015
1435708799203902068 Tue Jun 30 23:59:59 UTC 2015
1435708799712168566 Tue Jun 30 23:59:59 UTC 2015
1435708800220473932 Wed Jul 1 00:00:00 UTC 2015
1435708800728908190 Wed Jul 1 00:00:00 UTC 2015
1435708801237611983 Wed Jul 1 00:00:01 UTC 2015
1435708801746251996 Wed Jul 1 00:00:01 UTC 2015
{code}
Note that 23:59:59 repeats itself, and that the timestamps increase during
the first time through, then step back down to the beginning of the second
and increase again.
As a result, the timestamps on values inserted during these seconds will be
out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and
synced them to shortly before the leap second would be inserted. During the
insertion of the leap second, I ran a test with logic something like:
{code}
simple_insert = session.prepare(
'INSERT INTO test (foo, bar) VALUES (?, ?);')
for i in itertools.count():
# stop after midnight
now = datetime.utcnow()
last_midnight = now.replace(hour=0, minute=0,
second=0, microsecond=0)
seconds_since_midnight = (now - last_midnight).total_seconds()
if 5 = seconds_since_midnight = 15:
break
session.execute(simple_insert, [i, i])
result = session.execute(SELECT bar, WRITETIME(bar) FROM test;)
{code}
Under normal circumstances, the values and writetimes would increase
together, but when inserted over the leap second, they don't. These {{value,
writetime}} pairs are sorted by writetime:
{code}
(582, 1435708799285000)
(579, 1435708799339000)
(583, 1435708799593000)
(580, 1435708799643000)
(584, 1435708799897000)
(581, 1435708799958000)
{code}
The values were inserted in increasing order, but their writetimes are in a
different order because of the repeated second. During the first instance of
23:59:59, the values 579, 580, and 581 were inserted at the beginning,
middle, and end of the second. During the leap second, which is also
23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and
end of the second. However, since the two seconds are the same second, they
appear interleaved with respect to timestamps, as shown above.
So, should I consider this behavior correct? If not, how should Cassandra
correctly handle the discontinuity introduced by the insertion of a leap
second?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[
https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485398#comment-14485398
]
T Jake Luciani commented on CASSANDRA-9131:
---
System.currentTimeMillis() uses CLOCK_REALTIME and will include leap seconds.
Defining correct behavior during leap second insertion
--
Key: CASSANDRA-9131
URL: https://issues.apache.org/jira/browse/CASSANDRA-9131
Project: Cassandra
Issue Type: Bug
Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue
Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Reporter: Jim Witschey
On Linux platforms, the insertion of a leap second breaks the monotonicity of
timestamps. This can make values appear to have been inserted into Cassandra
in a different order than they were. I want to know what behavior is expected
and desirable for inserts over this discontinuity.
From a timestamp perspective, an inserted leap second looks like a repeat of
the previous second:
{code}
$ while true ; do echo `date +%s%N` `date -u` ; sleep .5 ; done
1435708798171327029 Tue Jun 30 23:59:58 UTC 2015
1435708798679392477 Tue Jun 30 23:59:58 UTC 2015
1435708799187550335 Tue Jun 30 23:59:59 UTC 2015
1435708799695670453 Tue Jun 30 23:59:59 UTC 2015
1435708799203902068 Tue Jun 30 23:59:59 UTC 2015
1435708799712168566 Tue Jun 30 23:59:59 UTC 2015
1435708800220473932 Wed Jul 1 00:00:00 UTC 2015
1435708800728908190 Wed Jul 1 00:00:00 UTC 2015
1435708801237611983 Wed Jul 1 00:00:01 UTC 2015
1435708801746251996 Wed Jul 1 00:00:01 UTC 2015
{code}
Note that 23:59:59 repeats itself, and that the timestamps increase during
the first time through, then step back down to the beginning of the second
and increase again.
As a result, the timestamps on values inserted during these seconds will be
out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and
synced them to shortly before the leap second would be inserted. During the
insertion of the leap second, I ran a test with logic something like:
{code}
simple_insert = session.prepare(
'INSERT INTO test (foo, bar) VALUES (?, ?);')
for i in itertools.count():
# stop after midnight
now = datetime.utcnow()
last_midnight = now.replace(hour=0, minute=0,
second=0, microsecond=0)
seconds_since_midnight = (now - last_midnight).total_seconds()
if 5 = seconds_since_midnight = 15:
break
session.execute(simple_insert, [i, i])
result = session.execute(SELECT bar, WRITETIME(bar) FROM test;)
{code}
EDIT: This behavior occurs with server-generated timestamps; in this
particular test, I set {{use_client_timestamp}} to {{False}}.
Under normal circumstances, the values and writetimes would increase
together, but when inserted over the leap second, they don't. These {{value,
writetime}} pairs are sorted by writetime:
{code}
(582, 1435708799285000)
(579, 1435708799339000)
(583, 1435708799593000)
(580, 1435708799643000)
(584, 1435708799897000)
(581, 1435708799958000)
{code}
The values were inserted in increasing order, but their writetimes are in a
different order because of the repeated second. During the first instance of
23:59:59, the values 579, 580, and 581 were inserted at the beginning,
middle, and end of the second. During the leap second, which is also
23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and
end of the second. However, since the two seconds are the same second, they
appear interleaved with respect to timestamps, as shown above.
So, should I consider this behavior correct? If not, how should Cassandra
correctly handle the discontinuity introduced by the insertion of a leap
second?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[
https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485308#comment-14485308
]
Jonathan Ellis commented on CASSANDRA-9131:
---
This is a good example of why I'm not sure pushing timestamps to the client and
saying here, you figure it out is such a great idea. /cc [~slebresne]
Defining correct behavior during leap second insertion
--
Key: CASSANDRA-9131
URL: https://issues.apache.org/jira/browse/CASSANDRA-9131
Project: Cassandra
Issue Type: Bug
Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue
Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Reporter: Jim Witschey
On Linux platforms, the insertion of a leap second breaks the monotonicity of
timestamps. This can make values appear to have been inserted into Cassandra
in a different order than they were. I want to know what behavior is expected
and desirable for inserts over this discontinuity.
From a timestamp perspective, an inserted leap second looks like a repeat of
the previous second:
{code}
$ while true ; do echo `date +%s%N` `date -u` ; sleep .5 ; done
1435708798171327029 Tue Jun 30 23:59:58 UTC 2015
1435708798679392477 Tue Jun 30 23:59:58 UTC 2015
1435708799187550335 Tue Jun 30 23:59:59 UTC 2015
1435708799695670453 Tue Jun 30 23:59:59 UTC 2015
1435708799203902068 Tue Jun 30 23:59:59 UTC 2015
1435708799712168566 Tue Jun 30 23:59:59 UTC 2015
1435708800220473932 Wed Jul 1 00:00:00 UTC 2015
1435708800728908190 Wed Jul 1 00:00:00 UTC 2015
1435708801237611983 Wed Jul 1 00:00:01 UTC 2015
1435708801746251996 Wed Jul 1 00:00:01 UTC 2015
{code}
Note that 23:59:59 repeats itself, and that the timestamps increase during
the first time through, then step back down to the beginning of the second
and increase again.
As a result, the timestamps on values inserted during these seconds will be
out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and
synced them to shortly before the leap second would be inserted. During the
insertion of the leap second, I ran a test with logic something like:
{code}
simple_insert = session.prepare(
'INSERT INTO test (foo, bar) VALUES (?, ?);')
for i in itertools.count():
# stop after midnight
now = datetime.utcnow()
last_midnight = now.replace(hour=0, minute=0,
second=0, microsecond=0)
seconds_since_midnight = (now - last_midnight).total_seconds()
if 5 = seconds_since_midnight = 15:
break
session.execute(simple_insert, [i, i])
result = session.execute(SELECT bar, WRITETIME(bar) FROM test;)
{code}
Under normal circumstances, the values and writetimes would increase
together, but when inserted over the leap second, they don't. These {{value,
writetime}} pairs are sorted by writetime:
{code}
(582, 1435708799285000)
(579, 1435708799339000)
(583, 1435708799593000)
(580, 1435708799643000)
(584, 1435708799897000)
(581, 1435708799958000)
{code}
The values were inserted in increasing order, but their writetimes are in a
different order because of the repeated second. During the first instance of
23:59:59, the values 579, 580, and 581 were inserted at the beginning,
middle, and end of the second. During the leap second, which is also
23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and
end of the second. However, since the two seconds are the same second, they
appear interleaved with respect to timestamps, as shown above.
So, should I consider this behavior correct? If not, how should Cassandra
correctly handle the discontinuity introduced by the insertion of a leap
second?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[
https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485331#comment-14485331
]
Jim Witschey commented on CASSANDRA-9131:
-
Client-side timestamps don't have anything to do with this particular test; the
behavior described happens with {{use_client_timestamp = False}}. I'll revise
the ticket to specify that.
Defining correct behavior during leap second insertion
--
Key: CASSANDRA-9131
URL: https://issues.apache.org/jira/browse/CASSANDRA-9131
Project: Cassandra
Issue Type: Bug
Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue
Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Reporter: Jim Witschey
On Linux platforms, the insertion of a leap second breaks the monotonicity of
timestamps. This can make values appear to have been inserted into Cassandra
in a different order than they were. I want to know what behavior is expected
and desirable for inserts over this discontinuity.
From a timestamp perspective, an inserted leap second looks like a repeat of
the previous second:
{code}
$ while true ; do echo `date +%s%N` `date -u` ; sleep .5 ; done
1435708798171327029 Tue Jun 30 23:59:58 UTC 2015
1435708798679392477 Tue Jun 30 23:59:58 UTC 2015
1435708799187550335 Tue Jun 30 23:59:59 UTC 2015
1435708799695670453 Tue Jun 30 23:59:59 UTC 2015
1435708799203902068 Tue Jun 30 23:59:59 UTC 2015
1435708799712168566 Tue Jun 30 23:59:59 UTC 2015
1435708800220473932 Wed Jul 1 00:00:00 UTC 2015
1435708800728908190 Wed Jul 1 00:00:00 UTC 2015
1435708801237611983 Wed Jul 1 00:00:01 UTC 2015
1435708801746251996 Wed Jul 1 00:00:01 UTC 2015
{code}
Note that 23:59:59 repeats itself, and that the timestamps increase during
the first time through, then step back down to the beginning of the second
and increase again.
As a result, the timestamps on values inserted during these seconds will be
out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and
synced them to shortly before the leap second would be inserted. During the
insertion of the leap second, I ran a test with logic something like:
{code}
simple_insert = session.prepare(
'INSERT INTO test (foo, bar) VALUES (?, ?);')
for i in itertools.count():
# stop after midnight
now = datetime.utcnow()
last_midnight = now.replace(hour=0, minute=0,
second=0, microsecond=0)
seconds_since_midnight = (now - last_midnight).total_seconds()
if 5 = seconds_since_midnight = 15:
break
session.execute(simple_insert, [i, i])
result = session.execute(SELECT bar, WRITETIME(bar) FROM test;)
{code}
Under normal circumstances, the values and writetimes would increase
together, but when inserted over the leap second, they don't. These {{value,
writetime}} pairs are sorted by writetime:
{code}
(582, 1435708799285000)
(579, 1435708799339000)
(583, 1435708799593000)
(580, 1435708799643000)
(584, 1435708799897000)
(581, 1435708799958000)
{code}
The values were inserted in increasing order, but their writetimes are in a
different order because of the repeated second. During the first instance of
23:59:59, the values 579, 580, and 581 were inserted at the beginning,
middle, and end of the second. During the leap second, which is also
23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and
end of the second. However, since the two seconds are the same second, they
appear interleaved with respect to timestamps, as shown above.
So, should I consider this behavior correct? If not, how should Cassandra
correctly handle the discontinuity introduced by the insertion of a leap
second?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[
https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485842#comment-14485842
]
Jim Witschey commented on CASSANDRA-9131:
-
To sum up some conversation on #cassandra-dev: This is expected behavior for
server-generated timestamps. Client-generated timestamps will help, but won't
solve the problem completely. That's expected behavior as well.
Is that right? If so, feel free to close.
Defining correct behavior during leap second insertion
--
Key: CASSANDRA-9131
URL: https://issues.apache.org/jira/browse/CASSANDRA-9131
Project: Cassandra
Issue Type: Bug
Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue
Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Reporter: Jim Witschey
On Linux platforms, the insertion of a leap second breaks the monotonicity of
timestamps. This can make values appear to have been inserted into Cassandra
in a different order than they were. I want to know what behavior is expected
and desirable for inserts over this discontinuity.
From a timestamp perspective, an inserted leap second looks like a repeat of
the previous second:
{code}
$ while true ; do echo `date +%s%N` `date -u` ; sleep .5 ; done
1435708798171327029 Tue Jun 30 23:59:58 UTC 2015
1435708798679392477 Tue Jun 30 23:59:58 UTC 2015
1435708799187550335 Tue Jun 30 23:59:59 UTC 2015
1435708799695670453 Tue Jun 30 23:59:59 UTC 2015
1435708799203902068 Tue Jun 30 23:59:59 UTC 2015
1435708799712168566 Tue Jun 30 23:59:59 UTC 2015
1435708800220473932 Wed Jul 1 00:00:00 UTC 2015
1435708800728908190 Wed Jul 1 00:00:00 UTC 2015
1435708801237611983 Wed Jul 1 00:00:01 UTC 2015
1435708801746251996 Wed Jul 1 00:00:01 UTC 2015
{code}
Note that 23:59:59 repeats itself, and that the timestamps increase during
the first time through, then step back down to the beginning of the second
and increase again.
As a result, the timestamps on values inserted during these seconds will be
out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and
synced them to shortly before the leap second would be inserted. During the
insertion of the leap second, I ran a test with logic something like:
{code}
simple_insert = session.prepare(
'INSERT INTO test (foo, bar) VALUES (?, ?);')
for i in itertools.count():
# stop after midnight
now = datetime.utcnow()
last_midnight = now.replace(hour=0, minute=0,
second=0, microsecond=0)
seconds_since_midnight = (now - last_midnight).total_seconds()
if 5 = seconds_since_midnight = 15:
break
session.execute(simple_insert, [i, i])
result = session.execute(SELECT bar, WRITETIME(bar) FROM test;)
{code}
EDIT: This behavior occurs with server-generated timestamps; in this
particular test, I set {{use_client_timestamp}} to {{False}}.
Under normal circumstances, the values and writetimes would increase
together, but when inserted over the leap second, they don't. These {{value,
writetime}} pairs are sorted by writetime:
{code}
(582, 1435708799285000)
(579, 1435708799339000)
(583, 1435708799593000)
(580, 1435708799643000)
(584, 1435708799897000)
(581, 1435708799958000)
{code}
The values were inserted in increasing order, but their writetimes are in a
different order because of the repeated second. During the first instance of
23:59:59, the values 579, 580, and 581 were inserted at the beginning,
middle, and end of the second. During the leap second, which is also
23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and
end of the second. However, since the two seconds are the same second, they
appear interleaved with respect to timestamps, as shown above.
So, should I consider this behavior correct? If not, how should Cassandra
correctly handle the discontinuity introduced by the insertion of a leap
second?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9131) Defining correct behavior during leap second insertion
[
https://issues.apache.org/jira/browse/CASSANDRA-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484583#comment-14484583
]
Philip Thompson commented on CASSANDRA-9131:
I think this discussion is worth bringing up at #cassandra-dev as well.
Defining correct behavior during leap second insertion
--
Key: CASSANDRA-9131
URL: https://issues.apache.org/jira/browse/CASSANDRA-9131
Project: Cassandra
Issue Type: Bug
Environment: Linux ip-172-31-0-5 3.2.0-57-virtual #87-Ubuntu SMP Tue
Nov 12 21:53:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Reporter: Jim Witschey
On Linux platforms, the insertion of a leap second breaks the monotonicity of
timestamps. This can make values appear to have been inserted into Cassandra
in a different order than they were. I want to know what behavior is expected
and desirable for inserts over this discontinuity.
From a timestamp perspective, an inserted leap second looks like a repeat of
the previous second:
{code}
$ while true ; do echo `date +%s%N` `date -u` ; sleep .5 ; done
1435708798171327029 Tue Jun 30 23:59:58 UTC 2015
1435708798679392477 Tue Jun 30 23:59:58 UTC 2015
1435708799187550335 Tue Jun 30 23:59:59 UTC 2015
1435708799695670453 Tue Jun 30 23:59:59 UTC 2015
1435708799203902068 Tue Jun 30 23:59:59 UTC 2015
1435708799712168566 Tue Jun 30 23:59:59 UTC 2015
1435708800220473932 Wed Jul 1 00:00:00 UTC 2015
1435708800728908190 Wed Jul 1 00:00:00 UTC 2015
1435708801237611983 Wed Jul 1 00:00:01 UTC 2015
1435708801746251996 Wed Jul 1 00:00:01 UTC 2015
{code}
Note that 23:59:59 repeats itself, and that the timestamps increase during
the first time through, then step back down to the beginning of the second
and increase again.
As a result, the timestamps on values inserted during these seconds will be
out of order. I set up a 4-node cluster running under Ubuntu 12.04.3 and
synced them to shortly before the leap second would be inserted. During the
insertion of the leap second, I ran a test with logic something like:
{code}
simple_insert = session.prepare(
'INSERT INTO test (foo, bar) VALUES (?, ?);')
for i in itertools.count():
# stop after midnight
now = datetime.utcnow()
last_midnight = now.replace(hour=0, minute=0,
second=0, microsecond=0)
seconds_since_midnight = (now - last_midnight).total_seconds()
if 5 = seconds_since_midnight = 15:
break
session.execute(simple_insert, [i, i])
result = session.execute(SELECT bar, WRITETIME(bar) FROM test;)
{code}
Under normal circumstances, the values and writetimes would increase
together, but when inserted over the leap second, they don't. These {{value,
writetime}} pairs are sorted by writetime:
{code}
(582, 1435708799285000)
(579, 1435708799339000)
(583, 1435708799593000)
(580, 1435708799643000)
(584, 1435708799897000)
(581, 1435708799958000)
{code}
The values were inserted in increasing order, but their writetimes are in a
different order because of the repeated second. During the first instance of
23:59:59, the values 579, 580, and 581 were inserted at the beginning,
middle, and end of the second. During the leap second, which is also
23:59:59, 582, 583, and 584 were inserted, also at the beginning, middle, and
end of the second. However, since the two seconds are the same second, they
appear interleaved with respect to timestamps, as shown above.
So, should I consider this behavior correct? If not, how should Cassandra
correctly handle the discontinuity introduced by the insertion of a leap
second?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
