date:20180201

[jira] [Commented] (CASSANDRA-14212) Back port CASSANDRA-13080 to 3.11.2 (Use new token allocation for non bootstrap case as well)

2018-02-01 Thread mck (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349912#comment-16349912
 ] 

mck commented on CASSANDRA-14212:
-

|| branch || testall || dtest ||
| 
[cassandra-3.11_13080|https://github.com/thelastpickle/cassandra/tree/mck/cassandra-3.11_13080]
   | 
[testall|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.11_13080]
 | 
[dtest|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/266]
 |

> Back port CASSANDRA-13080 to 3.11.2 (Use new token allocation for non 
> bootstrap case as well)
> -
>
> Key: CASSANDRA-14212
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14212
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: mck
>Assignee: mck
>Priority: Major
>
> Backport CASSANDRA-13080 to 3.11.x
>  
> The patch applies without conflict to the {{cassandra-3.11}} and equally 
> concerns to users of Cassandra-3.11.1
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14092) Max ttl of 20 years will overflow localDeletionTime

2018-02-01 Thread Paulo Motta (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349833#comment-16349833
 ] 

Paulo Motta commented on CASSANDRA-14092:
-

{quote}To clarify, CAP and CAP_NOWARN will cap the expiry but we'll have NO 
INTENTION of ever fixing it in an upgrade?
{quote}
We're not promising to fix the correct TTL in an upgrade, but we could do it in 
some cases, but I prefer to leave this decision for later.
{quote}Or would they have to do a scrub to convert anything that got capped to 
its actual TTL?
{quote}
The scrub is just to fix SSTables of affected systems that overflowed from 
2018-01-19T03:14:06+00:00 until the upgrade and were backed up. As I said 
before, we're not doing any promise yet regarding honoring the actual TTL when 
it's capped. But if we were to implement this would probably be done during 
upgradesstables and not scrub.
{quote}I think it's worth pointing out that REJECT is a ticking time bomb.
{quote}
I agree but I don't feel strongly about the default, because the policies will 
be clearly specified in big letters in the NEWS.txt which is the document which 
everyone should read before upgrading, so if you don't want applications to 
break in your organization just change your policy to CAP.

Do you mind proof-reading the NEWS.txt and check if something is not clear/can 
be improved? Thanks!

> Max ttl of 20 years will overflow localDeletionTime
> ---
>
> Key: CASSANDRA-14092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14092
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Paulo Motta
>Assignee: Paulo Motta
>Priority: Blocker
> Fix For: 2.1.20, 2.2.12, 3.0.16, 3.11.2
>
>
> CASSANDRA-4771 added a max value of 20 years for ttl to protect against [year 
> 2038 overflow bug|https://en.wikipedia.org/wiki/Year_2038_problem] for 
> {{localDeletionTime}}.
> It turns out that next year the {{localDeletionTime}} will start overflowing 
> with the maximum ttl of 20 years ({{System.currentTimeMillis() + ttl(20 
> years) > Integer.MAX_VALUE}}), so we should remove this limitation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14092) Max ttl of 20 years will overflow localDeletionTime

2018-02-01 Thread Paulo Motta (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-14092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349824#comment-16349824
]

Paulo Motta commented on CASSANDRA-14092:
-

Thanks for the quick turnaround [~beobal]! See follow-up below:
{quote}The wording of the NEWS.txt entry is good, I do wonder if we should
maybe place it right at the top of the file rather than just in the 3.0.16
section for extra emphasis. Any thoughts on that?
{quote}
Good idea, I did this and also updated the text to contemplate the possibility
of data loss before this patch and how to fix it with scrub:
{noformat}
MAXIMUM TTL EXPIRATION DATE NOTICE
---

The maximum expiration timestamp that can be represented by the storage engine
is 2038-01-19T03:14:06+00:00,
which means that inserts with TTL that expire after this date are not currently
supported.

Prior to 3.0.16 in the 3.0.X series and 3.11.2 in the 3.11 series, there was no
protection against INSERTS
with TTL expiring after the maximum supported date, causing the expiration time
field to overflow and the
records to expire immediately. Expired records due to overflow may have been
removed permanently after a
compaction. The 2.1.X and 2.2.X series are not subject to data loss due to this
issue if assertions are enabled,
since an AssertionError is thrown during INSERT when the expiration time field
overflows on these versions.

In practice this issue will affect only users that use very large TTLs, close
to the maximum allowed value of
63072 seconds (20 years), starting from 2018-01-19T03:14:06+00:00. As time
progresses, the maximum supported
TTL will be gradually reduced as the the maximum expiration date approaches.
For instance, a user on an affected
version on 2028-01-19T03:14:06 with a TTL of 10 years will be affected by this
bug, so we urge users of very
large TTLs to upgrade to a version where this issue is addressed as soon as
possible.

Potentially affected users should inspect their SSTables and search for
negative min local deletion times to
detect this issue. SSTables in this state must be backed up immediately, as
they are subject to data loss
during auto-compactions, and may be recovered by running the sstablescrub tool
from versions 3.0.16+ and/or 3.11.2+.

The Cassandra project plans to fix this limitation in newer versions, but while
the fix is not available, operators
can decide which policy to apply when dealing with inserts with TTL exceeding
the maximum supported expiration date:
- REJECT: this is the default policy and will reject any requests with
expiration date timestamp after 2038-01-19T03:14:06+00:00.
- CAP: any insert with TTL expiring after 2038-01-19T03:14:06+00:00 will
expire on 2038-01-19T03:14:06+00:00 and the client will receive a warning.
- CAP_NOWARN: same as previous, except that the client warning will not be
emitted.

These policies may be specified via the
-Dcassandra.expiration_date_overflow_policy=POLICY startup option which can be
set in the jvm.options file.

See CASSANDRA-14092 for more details about this issue.
{noformat}
Please let me know what do you think of the updated text. We should also
probably publish this text (or a subset of it) during the release announcement
e-mail.

While writing the text above, I figured that there is also a remote possibility
of data loss in 2.1/2.2 if assertions are disabled, but didn't backport the
scrub recovery since it was not a straightforward backport and I didn't think
it was worth the effort right now. We can always do that later if necessary,
the most important thing right now is to ship the policies. To reflect this I
updated the 4th paragraph on 2.1 and 2.2 to:
{noformat}
2.1.X / 2.2.X users in the conditions above should not be subject to data loss
unless assertions are disabled, in which
case the suspect SSTables must be backed up immediately and manually recovered,
as they are subject to data loss
during auto-compaction.
{noformat}
{quote}I also have one piece of feedback on the policies; I don't see any
benefit in being able to turn off logging of capped expirations (especially
since we're using NoSpamLogger) but I do I think the client warning is useful.
{quote}
I agree and updated the patch with this suggestion, but at the same time I
think advanced operators may want to control the periodicity of the logging, so
I created a property
{{cassandra.expiration_overflow_warning_interval_minutes=5}} to control this.
{quote}I also noticed that the logging of a parse error/invalid value for the
policy sysprop is at DEBUG in the current patches, but it might be sensible to
draw a bit more attention to that if it happens.
{quote}
Agreed, changed the logging to WARN.

I finished the cleanup of the patch and already provided a version for all
branches. The 2.1 and 2.2 versions are pretty much the same, as well as the
3.0/3.11/trunk,

58 matches

Mail list logo