[jira] [Resolved] (CASSANDRA-13941) Refactor GcCompactionTest to avoid boxing
[
https://issues.apache.org/jira/browse/CASSANDRA-13941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa resolved CASSANDRA-13941.
Resolution: Fixed
Fix Version/s: (was: 4.x)
4.0
Committed to 4.0 as {{2ecadc88e4407ebd4b3325f42e0296e1e1dc8944}}
> Refactor GcCompactionTest to avoid boxing
> -
>
> Key: CASSANDRA-13941
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13941
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Jirsa
>Assignee: Jeff Jirsa
>Priority: Trivial
> Fix For: 4.0
>
>
> From GH PR #116: https://github.com/apache/cassandra/pull/116
> {quote}
> Hello,
> I am a graduate student at Oregon State University and as a part of my
> research and subject CS562 Applied Software Engineering project (Study and
> Refactoring of Functional Interface in Java), I have done refactoring of the
> Function to ToIntFunction. My project deals with the boxing and
> unboxing of Wrapper class and the primitive data-types. I want to know that
> whether the open source community is ready to accept these micro-optimizing
> refactorings or not.
> Thank you,
> Harsh Thakor
> {quote}
> Trivial patch, but it's right and low risk.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
cassandra git commit: Refactor GcCompactionTest to avoid boxing
Repository: cassandra
Updated Branches:
refs/heads/trunk d46c1f3ea -> 2ecadc88e
Refactor GcCompactionTest to avoid boxing
Closes #116
Patch by Hashrajsinh Thakor; Reviewed by Jeff Jirsa for CASSANDRA-13941
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2ecadc88
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2ecadc88
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2ecadc88
Branch: refs/heads/trunk
Commit: 2ecadc88e4407ebd4b3325f42e0296e1e1dc8944
Parents: d46c1f3
Author: Harshrajsinh Thakor
Authored: Sun Jun 4 20:34:38 2017 -0700
Committer: Jeff Jirsa
Committed: Fri Oct 6 23:27:57 2017 -0700
--
CHANGES.txt | 1 +
test/unit/org/apache/cassandra/cql3/GcCompactionTest.java | 10 +-
2 files changed, 6 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/2ecadc88/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index eeed794..b8d22cb 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
4.0
+ * Refactor GcCompactionTest to avoid boxing (CASSANDRA-13941)
* Checksum sstable metadata (CASSANDRA-13321)
* Expose recent histograms in JmxHistograms (CASSANDRA-13642)
* Fix buffer length comparison when decompressing in netty-based streaming
(CASSANDRA-13899)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/2ecadc88/test/unit/org/apache/cassandra/cql3/GcCompactionTest.java
--
diff --git a/test/unit/org/apache/cassandra/cql3/GcCompactionTest.java
b/test/unit/org/apache/cassandra/cql3/GcCompactionTest.java
index 5a78d31..a09d79e 100644
--- a/test/unit/org/apache/cassandra/cql3/GcCompactionTest.java
+++ b/test/unit/org/apache/cassandra/cql3/GcCompactionTest.java
@@ -24,7 +24,7 @@ import static org.junit.Assert.assertTrue;
import java.util.Collections;
import java.util.HashSet;
import java.util.Set;
-import java.util.function.Function;
+import java.util.function.ToIntFunction;
import com.google.common.collect.Iterables;
import org.junit.Test;
@@ -367,7 +367,7 @@ public class GcCompactionTest extends CQLTester
return ccd.cellsCount();
}
-int count(SSTableReader reader, Function predicate,
Function partitionPredicate)
+int count(SSTableReader reader, ToIntFunction predicate,
ToIntFunction partitionPredicate)
{
int instances = 0;
try (ISSTableScanner partitions = reader.getScanner())
@@ -376,15 +376,15 @@ public class GcCompactionTest extends CQLTester
{
try (UnfilteredRowIterator iter = partitions.next())
{
-instances += partitionPredicate.apply(iter);
+instances += partitionPredicate.applyAsInt(iter);
while (iter.hasNext())
{
Unfiltered atom = iter.next();
-instances += predicate.apply(atom);
+instances += predicate.applyAsInt(atom);
}
}
}
}
return instances;
}
-}
\ No newline at end of file
+}
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-13941) Refactor GcCompactionTest to avoid boxing
Jeff Jirsa created CASSANDRA-13941:
--
Summary: Refactor GcCompactionTest to avoid boxing
Key: CASSANDRA-13941
URL: https://issues.apache.org/jira/browse/CASSANDRA-13941
Project: Cassandra
Issue Type: Bug
Reporter: Jeff Jirsa
Assignee: Jeff Jirsa
Priority: Trivial
Fix For: 4.x
>From GH PR #116: https://github.com/apache/cassandra/pull/116
{quote}
Hello,
I am a graduate student at Oregon State University and as a part of my research
and subject CS562 Applied Software Engineering project (Study and Refactoring
of Functional Interface in Java), I have done refactoring of the
Function to ToIntFunction. My project deals with the boxing and
unboxing of Wrapper class and the primitive data-types. I want to know that
whether the open source community is ready to accept these micro-optimizing
refactorings or not.
Thank you,
Harsh Thakor
{quote}
Trivial patch, but it's right and low risk.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13930) Avoid grabbing the read lock when checking if compaction strategy should do defragmentation
[ https://issues.apache.org/jira/browse/CASSANDRA-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-13930: --- Reviewer: Jeff Jirsa > Avoid grabbing the read lock when checking if compaction strategy should do > defragmentation > --- > > Key: CASSANDRA-13930 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13930 > Project: Cassandra > Issue Type: Bug >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson > Fix For: 3.11.x, 4.x > > > We grab the read lock when checking whether the compaction strategy benefits > from defragmentation, avoid that. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
[
https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16195167#comment-16195167
]
Benjamin Lerer commented on CASSANDRA-13006:
Just realized that I forgot to switch from {{jmap}} to {{jcmd}}, for generating
heap histograms, as suggested by [~nibin.gv]. I will update my patches in the
coming days.
> Disable automatic heap dumps on OOM error
> -
>
> Key: CASSANDRA-13006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13006
> Project: Cassandra
> Issue Type: Bug
> Components: Configuration
>Reporter: anmols
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 3.0.15
>
> Attachments: 13006-3.0.9.txt
>
>
> With CASSANDRA-9861, a change was added to enable collecting heap dumps by
> default if the process encountered an OOM error. These heap dumps are stored
> in the Apache Cassandra home directory unless configured otherwise (see
> [Cassandra Support
> Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps]
> for this feature).
>
> The creation and storage of heap dumps aides debugging and investigative
> workflows, but is not be desirable for a production environment where these
> heap dumps may occupy a large amount of disk space and require manual
> intervention for cleanups.
>
> Managing heap dumps on out of memory errors and configuring the paths for
> these heap dumps are available as JVM options in JVM. The current behavior
> conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError.
>
> A patch can be proposed here that would make the heap dump on OOM error honor
> the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate
> heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM
> option.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-10786) Include hash of result set metadata in prepared statement id
[
https://issues.apache.org/jira/browse/CASSANDRA-10786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16195162#comment-16195162
]
Alex Petrov commented on CASSANDRA-10786:
-
I've updated the
[branch|https://github.com/ifesdjeen/cassandra/tree/10786-trunk], where I've
commented out the {{build.xml}} sections with {{java-driver}} dependency (in a
way similar that was done right before 3.0 release) and pulled in the latest
snapshot java driver from [the
branch|https://github.com/datastax/java-driver/pull/794] and the python driver
from [the
branch|https://github.com/datastax/python-driver/tree/808_prepared-metadata-id].
I'd like to merge it, but before I do wanted to make sure we're all on board
with that. I'll create an additional ticket that would be a 4.0 blocker to pull
in the latest release of both drivers and restore {{build.xml}} entries with
the corresponding versions to make sure this is not getting missed.
[~snazy] [~iamaleksey] [~jjirsa] it'd be great to get your feedback.
> Include hash of result set metadata in prepared statement id
>
>
> Key: CASSANDRA-10786
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10786
> Project: Cassandra
> Issue Type: Sub-task
> Components: CQL
>Reporter: Olivier Michallat
>Assignee: Alex Petrov
>Priority: Minor
> Labels: client-impacting, doc-impacting, protocolv5
> Fix For: 4.x
>
>
> *_Initial description:_*
> This is a follow-up to CASSANDRA-7910, which was about invalidating a
> prepared statement when the table is altered, to force clients to update
> their local copy of the metadata.
> There's still an issue if multiple clients are connected to the same host.
> The first client to execute the query after the cache was invalidated will
> receive an UNPREPARED response, re-prepare, and update its local metadata.
> But other clients might miss it entirely (the MD5 hasn't changed), and they
> will keep using their old metadata. For example:
> # {{SELECT * ...}} statement is prepared in Cassandra with md5 abc123,
> clientA and clientB both have a cache of the metadata (columns b and c)
> locally
> # column a gets added to the table, C* invalidates its cache entry
> # clientA sends an EXECUTE request for md5 abc123, gets UNPREPARED response,
> re-prepares on the fly and updates its local metadata to (a, b, c)
> # prepared statement is now in C*’s cache again, with the same md5 abc123
> # clientB sends an EXECUTE request for id abc123. Because the cache has been
> populated again, the query succeeds. But clientB still has not updated its
> metadata, it’s still (b,c)
> One solution that was suggested is to include a hash of the result set
> metadata in the md5. This way the md5 would change at step 3, and any client
> using the old md5 would get an UNPREPARED, regardless of whether another
> client already reprepared.
> -
> *_Resolution (2017/02/13):_*
> The following changes were made to native protocol v5:
> - the PREPARED response includes {{result_metadata_id}}, a hash of the result
> set metadata.
> - every EXECUTE message must provide {{result_metadata_id}} in addition to
> the prepared statement id. If it doesn't match the current one on the server,
> it means the client is operating on a stale schema.
> - to notify the client, the server returns a ROWS response with a new
> {{Metadata_changed}} flag, the new {{result_metadata_id}} and the updated
> result metadata (this overrides the {{No_metadata}} flag, even if the client
> had requested it)
> - the client updates its copy of the result metadata before it decodes the
> results.
> So the scenario above would now look like:
> # {{SELECT * ...}} statement is prepared in Cassandra with md5 abc123, and
> result set (b, c) that hashes to cde456
> # column a gets added to the table, C* does not invalidate its cache entry,
> but only updates the result set to (a, b, c) which hashes to fff789
> # client sends an EXECUTE request for (statementId=abc123, resultId=cde456)
> and skip_metadata flag
> # cde456!=fff789, so C* responds with ROWS(..., no_metadata=false,
> metadata_changed=true, new_metadata_id=fff789,col specs for (a,b,c))
> # client updates its column specifications, and will send the next execute
> queries with (statementId=abc123, resultId=fff789)
> This works the same with multiple clients.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
[ https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Lerer updated CASSANDRA-13006: --- Status: Patch Available (was: In Progress) > Disable automatic heap dumps on OOM error > - > > Key: CASSANDRA-13006 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13006 > Project: Cassandra > Issue Type: Bug > Components: Configuration >Reporter: anmols >Assignee: Benjamin Lerer >Priority: Minor > Fix For: 3.0.15 > > Attachments: 13006-3.0.9.txt > > > With CASSANDRA-9861, a change was added to enable collecting heap dumps by > default if the process encountered an OOM error. These heap dumps are stored > in the Apache Cassandra home directory unless configured otherwise (see > [Cassandra Support > Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps] > for this feature). > > The creation and storage of heap dumps aides debugging and investigative > workflows, but is not be desirable for a production environment where these > heap dumps may occupy a large amount of disk space and require manual > intervention for cleanups. > > Managing heap dumps on out of memory errors and configuring the paths for > these heap dumps are available as JVM options in JVM. The current behavior > conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError. > > A patch can be proposed here that would make the heap dump on OOM error honor > the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate > heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM > option. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
[
https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16195149#comment-16195149
]
Benjamin Lerer commented on CASSANDRA-13006:
[~urandom], [~brandon.williams], [~tjake]
Sorry, for the delay.
I pushed some patches for
[2.2|https://github.com/apache/cassandra/compare/cassandra-2.2...blerer:13006-2.2],
[3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...blerer:13006-3.0],
[3.11|https://github.com/apache/cassandra/compare/cassandra-3.11...blerer:13006-3.11]
and
[trunk|https://github.com/apache/cassandra/compare/trunk...blerer:13006-trunk].
The branches differs only at the level of the configuration files
({{cassandra-env.sh}} and {{cassandra-env.ps1}}).
The patches let the JVM handle the {{OutOfMemoryErrors}} throught the JVM
{{OnOutOfMemoryError}}, {{ExitOnOutOfMemoryError}} or
{{CrashOnOutOfMemoryError}} options.
As the {{ExitOnOutOfMemoryError}} and {{CrashOnOutOfMemoryError}} options are
only supported since Oracle JDK 7 update 101 and since JDK 8 update 92,
Cassandra uses by default the {{OnOutOfMemoryError}} option.
A startup check will emit a warning if none of the options is used. This check
is there to ensure that {{OOM}} errors are properly handled and that C* cannot
continue to run in an unstable state that could cause data corruption.
The patch add no check for the {{HeapDumpOnOutOfMemoryError}} option as in some
cases administrators prefer to disable them.
The {{cassandra-env.sh}} has a new variable {{JVM_ON_OUT_OF_MEMORY_ERROR_OPT}}
which should be use to specify the {{OnOutOfMemoryError}} option. As bash
commands split words on white spaces without taking quotes into account,
specifying the {{OnOutOfMemoryError}} as part of the {{JVM_OPTS}} variable
cannot work for an option value such as: {{kill -9 %p}}.
Before generating an heap dump, C* use to log an Heap histogram using {{jmap}}.
If the heap size was large, reading the heap dump could take a few hours and
the heap histogram can help to debug the problem much faster. The patches keep
the posibility to print an heap histogram on OOM error but disables it by
default. To enable it the {{cassandra.printHeapHistogramOnOutOfMemoryError}}
system property must be set to {{true}}.
As generating the histogram for only the live objects (using {{jmap
histo:live}}) would trigger a garbage collection before generating the
histogram, I prefered to stick with {{jmap histo}} to minimize the risks.
The previous implementation was suffering of 2 problems:
* If several OOM errors were thrown in a short time span, each of them would
trigger an heap histogram and an heap dump (see this
[comment|https://issues.apache.org/jira/browse/CASSANDRA-13006?focusedCommentId=16118421&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16118421])
* If an exception was thrown while C* was trying to generate the heap dump, C*
would not be shutdown and would continue running in an unstable state (see
CASSANDRA-13886)
The patches fix those problems for the case were an heap histogram need to be
logged. In the case were the histogram is not requested those problems do not
exist anymore.
CI looks good for the unit tests.
The changes to the {{cassandra}} startup script break the DTests but before
changing the {{DTests}} framework I would prefer having a first review of the
patches.
[~JoshuaMcKenzie] could you review the patches?
> Disable automatic heap dumps on OOM error
> -
>
> Key: CASSANDRA-13006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13006
> Project: Cassandra
> Issue Type: Bug
> Components: Configuration
>Reporter: anmols
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 3.0.15
>
> Attachments: 13006-3.0.9.txt
>
>
> With CASSANDRA-9861, a change was added to enable collecting heap dumps by
> default if the process encountered an OOM error. These heap dumps are stored
> in the Apache Cassandra home directory unless configured otherwise (see
> [Cassandra Support
> Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps]
> for this feature).
>
> The creation and storage of heap dumps aides debugging and investigative
> workflows, but is not be desirable for a production environment where these
> heap dumps may occupy a large amount of disk space and require manual
> intervention for cleanups.
>
> Managing heap dumps on out of memory errors and configuring the paths for
> these heap dumps are available as JVM options in JVM. The current behavior
> conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError.
>
> A patch can be proposed here that would make the heap dump on OOM error honor
> the HeapDumpOnOutO
[jira] [Commented] (CASSANDRA-13850) Modifying "cassandra-env.sh"
[ https://issues.apache.org/jira/browse/CASSANDRA-13850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16194878#comment-16194878 ] Chris Lohfink commented on CASSANDRA-13850: --- Ah I see it. PR lgtm. > Modifying "cassandra-env.sh" > > > Key: CASSANDRA-13850 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13850 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Amitkumar Ghatwal > Fix For: 4.x > > > Hi All, > Added support for arch in "cassandra-env.sh " with PR : > https://github.com/apache/cassandra/pull/149 > Regards, > Amit -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
cassandra git commit: Ninja fix a few possible metrics.rst doc bugs
Repository: cassandra Updated Branches: refs/heads/trunk 9663c0093 -> d46c1f3ea Ninja fix a few possible metrics.rst doc bugs Closes #159 Patch by Joaquin Casares; Reviewed by Chris Lohfink Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d46c1f3e Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d46c1f3e Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d46c1f3e Branch: refs/heads/trunk Commit: d46c1f3eaa0fe9b6662eb4063f9118ad57b3029e Parents: 9663c00 Author: Joaquin Casares Authored: Wed Oct 4 15:42:05 2017 -0500 Committer: Jeff Jirsa Committed: Fri Oct 6 09:40:43 2017 -0700 -- doc/source/operating/metrics.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/d46c1f3e/doc/source/operating/metrics.rst -- diff --git a/doc/source/operating/metrics.rst b/doc/source/operating/metrics.rst index 10afc1b..6559b53 100644 --- a/doc/source/operating/metrics.rst +++ b/doc/source/operating/metrics.rst @@ -185,7 +185,7 @@ Reported name format: ``org.apache.cassandra.metrics.ThreadPools...`` **JMX MBean** -``org.apache.cassandra.metrics:type=ThreadPools scope= type= name=`` +``org.apache.cassandra.metrics:type=ThreadPools path= scope= name=`` = == === Name Type Description @@ -405,10 +405,10 @@ Dropped writes are stored and retried by ``Hinted Handoff`` Reported name format: **Metric Name** -``org.apache.cassandra.metrics.DroppedMessages..`` +``org.apache.cassandra.metrics.DroppedMessage..`` **JMX MBean** -``org.apache.cassandra.metrics:type=DroppedMetrics scope= name=`` +``org.apache.cassandra.metrics:type=DroppedMessage scope= name=`` == == === Name Type Description - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13939) Mishandling of cells for removed/dropped columns when reading legacy files
[
https://issues.apache.org/jira/browse/CASSANDRA-13939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sylvain Lebresne updated CASSANDRA-13939:
-
Resolution: Fixed
Fix Version/s: (was: 3.11.x)
(was: 3.0.x)
3.11.2
3.0.16
Status: Resolved (was: Ready to Commit)
Committed thanks (there were a few failures on the CI builds, but that really
doesn't seem related (the code only updates code that is involved in reading
old sstables, and none of those tests was doing that)).
> Mishandling of cells for removed/dropped columns when reading legacy files
> --
>
> Key: CASSANDRA-13939
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13939
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
>Reporter: Sylvain Lebresne
>Assignee: Sylvain Lebresne
> Fix For: 3.0.16, 3.11.2
>
>
> The tl;dr is that there is a bug in reading legacy files that can manifests
> itself with a trace looking like this:
> {noformat}
> Exception (java.lang.IllegalStateException) encountered during startup: One
> row required, 2 found
> java.lang.IllegalStateException: One row required, 2 found
> at
> org.apache.cassandra.cql3.UntypedResultSet$FromResultSet.one(UntypedResultSet.java:84)
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.readTableTimestamp(LegacySchemaMigrator.java:254)
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.readTable(LegacySchemaMigrator.java:244)
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.lambda$readTables$7(LegacySchemaMigrator.java:238)
> at
> org.apache.cassandra.schema.LegacySchemaMigrator$$Lambda$126/591203139.accept(Unknown
> Source)
> at java.util.ArrayList.forEach(ArrayList.java:1249)
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.readTables(LegacySchemaMigrator.java:238)
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.readKeyspace(LegacySchemaMigrator.java:187)
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.lambda$readSchema$4(LegacySchemaMigrator.java:178)
> at
> org.apache.cassandra.schema.LegacySchemaMigrator$$Lambda$123/1612073393.accept(Unknown
> Source)
> at java.util.ArrayList.forEach(ArrayList.java:1249)
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.readSchema(LegacySchemaMigrator.java:178)
> {noformat}
> The reason this can happen has to do with the handling of legacy files.
> Legacy files are cell based while the 3.0 storage engine is primarily row
> based, so we group those cells into rows early in the deserialization process
> (in {{UnfilteredDeserializer.OldFormatDeserializer}}), but in doing so, we
> can only consider a row finished when we've either reach the end of the
> partition/file, or when we've read a cell that doesn't belong to that row.
> That second case means that when the deserializer returns a given row, the
> underlying file pointer may actually not positioned at the end of that row,
> but rather it may be past the first cell of the next row (which the
> deserializer remembers for future use). Long story short, when we try to
> detect if we're logically past our current index block in
> {{AbstractIterator.IndexState#isPastCurrentBlock(}}), we can't simply rely on
> the file pointer, which again may be a bit more advanced that we logically
> are, and that's the reason for the "correction" in that method. That
> correction is really just the amount of bytes remembered but not yet used in
> the deserializer.
> That "correction" is sometimes wrong however and that's due to the fact that
> in {{LegacyLayout#readLegacyAtom}}, if we get a cell for an dropped or
> removed cell, we ignore that cell (which, in itself, is fine). Problem is
> that this skipping is done within the {{LegacyLayout#readLegacyAtom}} method
> but without {{UnfilteredDeserializer.OldFormatDeserializer}} knowing about
> it. As such, the size of the skipped cell ends up being accounted in the
> "correction" bytes for the next cell we read. Lo and behold, if such cell for
> a removed/dropped column is both the last cell of a CQL row and just before
> an index boundary (pretty unlikely in general btw, but definitively
> possible), then the deserializer will count its size with the first cell of
> the next row, which happens to also be the first cell of the next index
> block. And when the code then tries to figure out if it crossed an index
> boundary, it will over-correct. That is, the {{indexState.updateBlock()}}
> call at the start of {{SSTableIterator.ForwardIndexedReader#computeNext}}
> will not work correctly. This can then make the code return a row that is
> after the requested slice end (and should thus not
[04/12] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11
* cassandra-3.0:
Fix handling of cells for removed column when reading legacy sstables
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/15276844
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/15276844
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/15276844
Branch: refs/heads/cassandra-3.11
Commit: 1527684462f903be6481324e401dc018da7e0e39
Parents: 710657d 5378ba2
Author: Sylvain Lebresne
Authored: Fri Oct 6 16:16:25 2017 +0200
Committer: Sylvain Lebresne
Committed: Fri Oct 6 16:16:25 2017 +0200
--
.../org/apache/cassandra/db/LegacyLayout.java | 70 +---
.../cassandra/db/UnfilteredDeserializer.java| 29 +---
2 files changed, 67 insertions(+), 32 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/15276844/src/java/org/apache/cassandra/db/LegacyLayout.java
--
diff --cc src/java/org/apache/cassandra/db/LegacyLayout.java
index 77aa59a,3ba96a6..9679a32
--- a/src/java/org/apache/cassandra/db/LegacyLayout.java
+++ b/src/java/org/apache/cassandra/db/LegacyLayout.java
@@@ -1021,29 -1037,36 +1038,36 @@@ public abstract class LegacyLayou
};
}
- public static LegacyAtom readLegacyAtom(CFMetaData metadata,
DataInputPlus in, boolean readAllAsDynamic) throws IOException
+ public static LegacyAtom readLegacyAtom(CFMetaData metadata,
DataInputPlus in, boolean readAllAsDynamic)
+ throws IOException, UnknownColumnException
{
- while (true)
- {
- ByteBuffer cellname = ByteBufferUtil.readWithShortLength(in);
- if (!cellname.hasRemaining())
- return null; // END_OF_ROW
+ ByteBuffer cellname = ByteBufferUtil.readWithShortLength(in);
+ if (!cellname.hasRemaining())
+ return null; // END_OF_ROW
- try
- {
- int b = in.readUnsignedByte();
- return (b & RANGE_TOMBSTONE_MASK) != 0
- ? readLegacyRangeTombstoneBody(metadata, in, cellname)
- : readLegacyCellBody(metadata, in, cellname, b,
SerializationHelper.Flag.LOCAL, readAllAsDynamic);
- }
- catch (UnknownColumnException e)
- {
- // We can get there if we read a cell for a dropped column,
and ff that is the case,
- // then simply ignore the cell is fine. But also not that we
ignore if it's the
- // system keyspace because for those table we actually remove
columns without registering
- // them in the dropped columns
- assert
metadata.ksName.equals(SchemaConstants.SYSTEM_KEYSPACE_NAME) ||
metadata.getDroppedColumnDefinition(e.columnName) != null : e.getMessage();
- }
+ try
+ {
+ int b = in.readUnsignedByte();
+ return (b & RANGE_TOMBSTONE_MASK) != 0
+? readLegacyRangeTombstoneBody(metadata, in, cellname)
+: readLegacyCellBody(metadata, in, cellname, b,
SerializationHelper.Flag.LOCAL, readAllAsDynamic);
+ }
+ catch (UnknownColumnException e)
+ {
+ // We legitimately can get here in 2 cases:
+ // 1) for system tables, because we've unceremoniously removed
columns (without registering them as dropped)
+ // 2) for dropped columns.
+ // In any other case, there is a mismatch between the schema and
the data, and we complain loudly in
+ // that case. Note that if we are in a legit case of an unknown
column, we want to simply skip that cell,
+ // but we don't do this here and re-throw the exception because
the calling code sometimes has to know
+ // about this happening. This does mean code calling this method
should handle this case properly.
-if (!metadata.ksName.equals(SystemKeyspace.NAME) &&
metadata.getDroppedColumnDefinition(e.columnName) == null)
++if (!metadata.ksName.equals(SchemaConstants.SYSTEM_KEYSPACE_NAME)
&& metadata.getDroppedColumnDefinition(e.columnName) == null)
+ throw new IllegalStateException(String.format("Got cell for
unknown column %s in sstable of %s.%s: " +
+ "This suggest a
problem with the schema which doesn't list " +
+ "this column.
Even if that column was dropped, it should have " +
+ "been listed as
such", metadata.ksName, metadata.cfName,
UTF8Type.instance.compose(e.columnName)), e);
+
[06/12] cassandra git commit: Merge commit '1527684462f903be6481324e401dc018da7e0e39' into trunk
Merge commit '1527684462f903be6481324e401dc018da7e0e39' into trunk * commit '1527684462f903be6481324e401dc018da7e0e39': Fix handling of cells for removed column when reading legacy sstables Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9815c466 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9815c466 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9815c466 Branch: refs/heads/trunk Commit: 9815c46679ad14b038f0314a9b9611e099750724 Parents: cfee3e9 1527684 Author: Sylvain Lebresne Authored: Fri Oct 6 16:16:53 2017 +0200 Committer: Sylvain Lebresne Committed: Fri Oct 6 16:16:53 2017 +0200 -- -- - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[11/12] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 * cassandra-3.0: Ninja-fix changelog Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3d09901b Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3d09901b Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3d09901b Branch: refs/heads/trunk Commit: 3d09901b4a0e67ce162283f85c34b1fe3c55dce3 Parents: 1527684 8a424ce Author: Sylvain Lebresne Authored: Fri Oct 6 16:18:36 2017 +0200 Committer: Sylvain Lebresne Committed: Fri Oct 6 16:18:36 2017 +0200 -- CHANGES.txt | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/3d09901b/CHANGES.txt -- diff --cc CHANGES.txt index e231bf2,6977be7..879397b --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,22 -1,8 +1,23 @@@ -3.0.16 +3.11.2 +Merged from 3.0: + * Mishandling of cells for removed/dropped columns when reading legacy files (CASSANDRA-13939) * Deserialise sstable metadata in nodetool verify (CASSANDRA-13922) -3.0.15 + +3.11.1 + * Fix the computation of cdc_total_space_in_mb for exabyte filesystems (CASSANDRA-13808) + * AbstractTokenTreeBuilder#serializedSize returns wrong value when there is a single leaf and overflow collisions (CASSANDRA-13869) + * Add a compaction option to TWCS to ignore sstables overlapping checks (CASSANDRA-13418) + * BTree.Builder memory leak (CASSANDRA-13754) + * Revert CASSANDRA-10368 of supporting non-pk column filtering due to correctness (CASSANDRA-13798) + * Add a skip read validation flag to cassandra-stress (CASSANDRA-13772) + * Fix cassandra-stress hang issues when an error during cluster connection happens (CASSANDRA-12938) + * Better bootstrap failure message when blocked by (potential) range movement (CASSANDRA-13744) + * "ignore" option is ignored in sstableloader (CASSANDRA-13721) + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652) + * Duplicate the buffer before passing it to analyser in SASI operation (CASSANDRA-13512) + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641) +Merged from 3.0: * Improve TRUNCATE performance (CASSANDRA-13909) * Implement short read protection on partition boundaries (CASSANDRA-13595) * Fix ISE thrown by UPI.Serializer.hasNext() for some SELECT queries (CASSANDRA-13911) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[12/12] cassandra git commit: Merge branch 'cassandra-3.11' into trunk
Merge branch 'cassandra-3.11' into trunk * cassandra-3.11: Ninja-fix changelog Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9663c009 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9663c009 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9663c009 Branch: refs/heads/trunk Commit: 9663c0093840a352faa08e3ffda6160745df75b8 Parents: 9815c46 3d09901 Author: Sylvain Lebresne Authored: Fri Oct 6 16:18:46 2017 +0200 Committer: Sylvain Lebresne Committed: Fri Oct 6 16:18:46 2017 +0200 -- CHANGES.txt | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/9663c009/CHANGES.txt -- diff --cc CHANGES.txt index 90c6b9c,879397b..eeed794 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,155 -1,6 +1,156 @@@ +4.0 + * Checksum sstable metadata (CASSANDRA-13321) + * Expose recent histograms in JmxHistograms (CASSANDRA-13642) + * Fix buffer length comparison when decompressing in netty-based streaming (CASSANDRA-13899) + * Properly close StreamCompressionInputStream to release any ByteBuf (CASSANDRA-13906) + * Add SERIAL and LOCAL_SERIAL support for cassandra-stress (CASSANDRA-13925) + * LCS needlessly checks for L0 STCS candidates multiple times (CASSANDRA-12961) + * Correctly close netty channels when a stream session ends (CASSANDRA-13905) + * Update lz4 to 1.4.0 (CASSANDRA-13741) + * Optimize Paxos prepare and propose stage for local requests (CASSANDRA-13862) + * Throttle base partitions during MV repair streaming to prevent OOM (CASSANDRA-13299) + * Use compaction threshold for STCS in L0 (CASSANDRA-13861) + * Fix problem with min_compress_ratio: 1 and disallow ratio < 1 (CASSANDRA-13703) + * Add extra information to SASI timeout exception (CASSANDRA-13677) + * Add incremental repair support for --hosts, --force, and subrange repair (CASSANDRA-13818) + * Rework CompactionStrategyManager.getScanners synchronization (CASSANDRA-13786) + * Add additional unit tests for batch behavior, TTLs, Timestamps (CASSANDRA-13846) + * Add keyspace and table name in schema validation exception (CASSANDRA-13845) + * Emit metrics whenever we hit tombstone failures and warn thresholds (CASSANDRA-13771) + * Make netty EventLoopGroups daemon threads (CASSANDRA-13837) + * Race condition when closing stream sessions (CASSANDRA-13852) + * NettyFactoryTest is failing in trunk on macOS (CASSANDRA-13831) + * Allow changing log levels via nodetool for related classes (CASSANDRA-12696) + * Add stress profile yaml with LWT (CASSANDRA-7960) + * Reduce memory copies and object creations when acting on ByteBufs (CASSANDRA-13789) + * Simplify mx4j configuration (Cassandra-13578) + * Fix trigger example on 4.0 (CASSANDRA-13796) + * Force minumum timeout value (CASSANDRA-9375) + * Use netty for streaming (CASSANDRA-12229) + * Use netty for internode messaging (CASSANDRA-8457) + * Add bytes repaired/unrepaired to nodetool tablestats (CASSANDRA-13774) + * Don't delete incremental repair sessions if they still have sstables (CASSANDRA-13758) + * Fix pending repair manager index out of bounds check (CASSANDRA-13769) + * Don't use RangeFetchMapCalculator when RF=1 (CASSANDRA-13576) + * Don't optimise trivial ranges in RangeFetchMapCalculator (CASSANDRA-13664) + * Use an ExecutorService for repair commands instead of new Thread(..).start() (CASSANDRA-13594) + * Fix race / ref leak in anticompaction (CASSANDRA-13688) + * Expose tasks queue length via JMX (CASSANDRA-12758) + * Fix race / ref leak in PendingRepairManager (CASSANDRA-13751) + * Enable ppc64le runtime as unsupported architecture (CASSANDRA-13615) + * Improve sstablemetadata output (CASSANDRA-11483) + * Support for migrating legacy users to roles has been dropped (CASSANDRA-13371) + * Introduce error metrics for repair (CASSANDRA-13387) + * Refactoring to primitive functional interfaces in AuthCache (CASSANDRA-13732) + * Update metrics to 3.1.5 (CASSANDRA-13648) + * batch_size_warn_threshold_in_kb can now be set at runtime (CASSANDRA-13699) + * Avoid always rebuilding secondary indexes at startup (CASSANDRA-13725) + * Upgrade JMH from 1.13 to 1.19 (CASSANDRA-13727) + * Upgrade SLF4J from 1.7.7 to 1.7.25 (CASSANDRA-12996) + * Default for start_native_transport now true if not set in config (CASSANDRA-13656) + * Don't add localhost to the graph when calculating where to stream from (CASSANDRA-13583) + * Make CDC availability more deterministic via hard-linking (CASSANDRA-12148) + * Allow skipping equality-restricted clustering columns in ORDER BY clause (CASSANDRA-10271) + * Use common nowInSec for validation compactions (CASSANDRA-13671) + * Improve handling of IR prepare failures (CAS
[07/12] cassandra git commit: Ninja-fix changelog
Ninja-fix changelog Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8a424cef Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8a424cef Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8a424cef Branch: refs/heads/cassandra-3.0 Commit: 8a424cef32270e75b3a4565793b2d436e6df181f Parents: 5378ba2 Author: Sylvain Lebresne Authored: Fri Oct 6 16:18:07 2017 +0200 Committer: Sylvain Lebresne Committed: Fri Oct 6 16:18:07 2017 +0200 -- CHANGES.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/8a424cef/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index df05f7f..6977be7 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,6 +1,6 @@ 3.0.16 - * Deserialise sstable metadata in nodetool verify (CASSANDRA-13922) - + * Mishandling of cells for removed/dropped columns when reading legacy files (CASSANDRA-13939) + * Deserialise sstable metadata in nodetool verify (CASSANDRA-13922) 3.0.15 * Improve TRUNCATE performance (CASSANDRA-13909) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[10/12] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 * cassandra-3.0: Ninja-fix changelog Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3d09901b Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3d09901b Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3d09901b Branch: refs/heads/cassandra-3.11 Commit: 3d09901b4a0e67ce162283f85c34b1fe3c55dce3 Parents: 1527684 8a424ce Author: Sylvain Lebresne Authored: Fri Oct 6 16:18:36 2017 +0200 Committer: Sylvain Lebresne Committed: Fri Oct 6 16:18:36 2017 +0200 -- CHANGES.txt | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/3d09901b/CHANGES.txt -- diff --cc CHANGES.txt index e231bf2,6977be7..879397b --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,22 -1,8 +1,23 @@@ -3.0.16 +3.11.2 +Merged from 3.0: + * Mishandling of cells for removed/dropped columns when reading legacy files (CASSANDRA-13939) * Deserialise sstable metadata in nodetool verify (CASSANDRA-13922) -3.0.15 + +3.11.1 + * Fix the computation of cdc_total_space_in_mb for exabyte filesystems (CASSANDRA-13808) + * AbstractTokenTreeBuilder#serializedSize returns wrong value when there is a single leaf and overflow collisions (CASSANDRA-13869) + * Add a compaction option to TWCS to ignore sstables overlapping checks (CASSANDRA-13418) + * BTree.Builder memory leak (CASSANDRA-13754) + * Revert CASSANDRA-10368 of supporting non-pk column filtering due to correctness (CASSANDRA-13798) + * Add a skip read validation flag to cassandra-stress (CASSANDRA-13772) + * Fix cassandra-stress hang issues when an error during cluster connection happens (CASSANDRA-12938) + * Better bootstrap failure message when blocked by (potential) range movement (CASSANDRA-13744) + * "ignore" option is ignored in sstableloader (CASSANDRA-13721) + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652) + * Duplicate the buffer before passing it to analyser in SASI operation (CASSANDRA-13512) + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641) +Merged from 3.0: * Improve TRUNCATE performance (CASSANDRA-13909) * Implement short read protection on partition boundaries (CASSANDRA-13595) * Fix ISE thrown by UPI.Serializer.hasNext() for some SELECT queries (CASSANDRA-13911) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[05/12] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11
* cassandra-3.0:
Fix handling of cells for removed column when reading legacy sstables
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/15276844
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/15276844
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/15276844
Branch: refs/heads/trunk
Commit: 1527684462f903be6481324e401dc018da7e0e39
Parents: 710657d 5378ba2
Author: Sylvain Lebresne
Authored: Fri Oct 6 16:16:25 2017 +0200
Committer: Sylvain Lebresne
Committed: Fri Oct 6 16:16:25 2017 +0200
--
.../org/apache/cassandra/db/LegacyLayout.java | 70 +---
.../cassandra/db/UnfilteredDeserializer.java| 29 +---
2 files changed, 67 insertions(+), 32 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/15276844/src/java/org/apache/cassandra/db/LegacyLayout.java
--
diff --cc src/java/org/apache/cassandra/db/LegacyLayout.java
index 77aa59a,3ba96a6..9679a32
--- a/src/java/org/apache/cassandra/db/LegacyLayout.java
+++ b/src/java/org/apache/cassandra/db/LegacyLayout.java
@@@ -1021,29 -1037,36 +1038,36 @@@ public abstract class LegacyLayou
};
}
- public static LegacyAtom readLegacyAtom(CFMetaData metadata,
DataInputPlus in, boolean readAllAsDynamic) throws IOException
+ public static LegacyAtom readLegacyAtom(CFMetaData metadata,
DataInputPlus in, boolean readAllAsDynamic)
+ throws IOException, UnknownColumnException
{
- while (true)
- {
- ByteBuffer cellname = ByteBufferUtil.readWithShortLength(in);
- if (!cellname.hasRemaining())
- return null; // END_OF_ROW
+ ByteBuffer cellname = ByteBufferUtil.readWithShortLength(in);
+ if (!cellname.hasRemaining())
+ return null; // END_OF_ROW
- try
- {
- int b = in.readUnsignedByte();
- return (b & RANGE_TOMBSTONE_MASK) != 0
- ? readLegacyRangeTombstoneBody(metadata, in, cellname)
- : readLegacyCellBody(metadata, in, cellname, b,
SerializationHelper.Flag.LOCAL, readAllAsDynamic);
- }
- catch (UnknownColumnException e)
- {
- // We can get there if we read a cell for a dropped column,
and ff that is the case,
- // then simply ignore the cell is fine. But also not that we
ignore if it's the
- // system keyspace because for those table we actually remove
columns without registering
- // them in the dropped columns
- assert
metadata.ksName.equals(SchemaConstants.SYSTEM_KEYSPACE_NAME) ||
metadata.getDroppedColumnDefinition(e.columnName) != null : e.getMessage();
- }
+ try
+ {
+ int b = in.readUnsignedByte();
+ return (b & RANGE_TOMBSTONE_MASK) != 0
+? readLegacyRangeTombstoneBody(metadata, in, cellname)
+: readLegacyCellBody(metadata, in, cellname, b,
SerializationHelper.Flag.LOCAL, readAllAsDynamic);
+ }
+ catch (UnknownColumnException e)
+ {
+ // We legitimately can get here in 2 cases:
+ // 1) for system tables, because we've unceremoniously removed
columns (without registering them as dropped)
+ // 2) for dropped columns.
+ // In any other case, there is a mismatch between the schema and
the data, and we complain loudly in
+ // that case. Note that if we are in a legit case of an unknown
column, we want to simply skip that cell,
+ // but we don't do this here and re-throw the exception because
the calling code sometimes has to know
+ // about this happening. This does mean code calling this method
should handle this case properly.
-if (!metadata.ksName.equals(SystemKeyspace.NAME) &&
metadata.getDroppedColumnDefinition(e.columnName) == null)
++if (!metadata.ksName.equals(SchemaConstants.SYSTEM_KEYSPACE_NAME)
&& metadata.getDroppedColumnDefinition(e.columnName) == null)
+ throw new IllegalStateException(String.format("Got cell for
unknown column %s in sstable of %s.%s: " +
+ "This suggest a
problem with the schema which doesn't list " +
+ "this column.
Even if that column was dropped, it should have " +
+ "been listed as
such", metadata.ksName, metadata.cfName,
UTF8Type.instance.compose(e.columnName)), e);
+
+
[01/12] cassandra git commit: Fix handling of cells for removed column when reading legacy sstables
Repository: cassandra
Updated Branches:
refs/heads/cassandra-3.0 c07441251 -> 8a424cef3
refs/heads/cassandra-3.11 710657d82 -> 3d09901b4
refs/heads/trunk cfee3e93b -> 9663c0093
Fix handling of cells for removed column when reading legacy sstables
Patch by Sylvain Lebresne; reviewed by Robert Stupp for CASSANDRA-13939
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5378ba27
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5378ba27
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5378ba27
Branch: refs/heads/cassandra-3.0
Commit: 5378ba27ff66b4f39e34d613f99b0f2eddf7990d
Parents: c074412
Author: Sylvain Lebresne
Authored: Thu Oct 5 10:30:56 2017 +0200
Committer: Sylvain Lebresne
Committed: Fri Oct 6 16:12:56 2017 +0200
--
.../org/apache/cassandra/db/LegacyLayout.java | 70 +---
.../cassandra/db/UnfilteredDeserializer.java| 29 +---
2 files changed, 67 insertions(+), 32 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/5378ba27/src/java/org/apache/cassandra/db/LegacyLayout.java
--
diff --git a/src/java/org/apache/cassandra/db/LegacyLayout.java
b/src/java/org/apache/cassandra/db/LegacyLayout.java
index 40b9fd3..3ba96a6 100644
--- a/src/java/org/apache/cassandra/db/LegacyLayout.java
+++ b/src/java/org/apache/cassandra/db/LegacyLayout.java
@@ -649,7 +649,7 @@ public abstract class LegacyLayout
boolean foundOne = false;
LegacyAtom atom;
-while ((atom = readLegacyAtom(metadata, in, false)) != null)
+while ((atom = readLegacyAtomSkippingUnknownColumn(metadata,in)) !=
null)
{
if (atom.isCell())
{
@@ -672,6 +672,23 @@ public abstract class LegacyLayout
return foundOne ? builder.build() : Rows.EMPTY_STATIC_ROW;
}
+private static LegacyAtom readLegacyAtomSkippingUnknownColumn(CFMetaData
metadata, DataInputPlus in)
+throws IOException
+{
+while (true)
+{
+try
+{
+return readLegacyAtom(metadata, in, false);
+}
+catch (UnknownColumnException e)
+{
+// Simply skip, as the method name implies.
+}
+}
+
+}
+
private static Row getNextRow(CellGrouper grouper, PeekingIterator cells)
{
if (!cells.hasNext())
@@ -1020,29 +1037,36 @@ public abstract class LegacyLayout
};
}
-public static LegacyAtom readLegacyAtom(CFMetaData metadata, DataInputPlus
in, boolean readAllAsDynamic) throws IOException
+public static LegacyAtom readLegacyAtom(CFMetaData metadata, DataInputPlus
in, boolean readAllAsDynamic)
+throws IOException, UnknownColumnException
{
-while (true)
-{
-ByteBuffer cellname = ByteBufferUtil.readWithShortLength(in);
-if (!cellname.hasRemaining())
-return null; // END_OF_ROW
-
-try
-{
-int b = in.readUnsignedByte();
-return (b & RANGE_TOMBSTONE_MASK) != 0
-? readLegacyRangeTombstoneBody(metadata, in, cellname)
-: readLegacyCellBody(metadata, in, cellname, b,
SerializationHelper.Flag.LOCAL, readAllAsDynamic);
-}
-catch (UnknownColumnException e)
-{
-// We can get there if we read a cell for a dropped column,
and ff that is the case,
-// then simply ignore the cell is fine. But also not that we
ignore if it's the
-// system keyspace because for those table we actually remove
columns without registering
-// them in the dropped columns
-assert metadata.ksName.equals(SystemKeyspace.NAME) ||
metadata.getDroppedColumnDefinition(e.columnName) != null : e.getMessage();
-}
+ByteBuffer cellname = ByteBufferUtil.readWithShortLength(in);
+if (!cellname.hasRemaining())
+return null; // END_OF_ROW
+
+try
+{
+int b = in.readUnsignedByte();
+return (b & RANGE_TOMBSTONE_MASK) != 0
+ ? readLegacyRangeTombstoneBody(metadata, in, cellname)
+ : readLegacyCellBody(metadata, in, cellname, b,
SerializationHelper.Flag.LOCAL, readAllAsDynamic);
+}
+catch (UnknownColumnException e)
+{
+// We legitimately can get here in 2 cases:
+// 1) for system tables, because we've unceremoniously removed
columns (without registering them as dropped)
+// 2) for dropped columns.
+// In any other case, there is a mismatch between the schema and
the data, and we complain l
[03/12] cassandra git commit: Fix handling of cells for removed column when reading legacy sstables
Fix handling of cells for removed column when reading legacy sstables
Patch by Sylvain Lebresne; reviewed by Robert Stupp for CASSANDRA-13939
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5378ba27
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5378ba27
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5378ba27
Branch: refs/heads/trunk
Commit: 5378ba27ff66b4f39e34d613f99b0f2eddf7990d
Parents: c074412
Author: Sylvain Lebresne
Authored: Thu Oct 5 10:30:56 2017 +0200
Committer: Sylvain Lebresne
Committed: Fri Oct 6 16:12:56 2017 +0200
--
.../org/apache/cassandra/db/LegacyLayout.java | 70 +---
.../cassandra/db/UnfilteredDeserializer.java| 29 +---
2 files changed, 67 insertions(+), 32 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/5378ba27/src/java/org/apache/cassandra/db/LegacyLayout.java
--
diff --git a/src/java/org/apache/cassandra/db/LegacyLayout.java
b/src/java/org/apache/cassandra/db/LegacyLayout.java
index 40b9fd3..3ba96a6 100644
--- a/src/java/org/apache/cassandra/db/LegacyLayout.java
+++ b/src/java/org/apache/cassandra/db/LegacyLayout.java
@@ -649,7 +649,7 @@ public abstract class LegacyLayout
boolean foundOne = false;
LegacyAtom atom;
-while ((atom = readLegacyAtom(metadata, in, false)) != null)
+while ((atom = readLegacyAtomSkippingUnknownColumn(metadata,in)) !=
null)
{
if (atom.isCell())
{
@@ -672,6 +672,23 @@ public abstract class LegacyLayout
return foundOne ? builder.build() : Rows.EMPTY_STATIC_ROW;
}
+private static LegacyAtom readLegacyAtomSkippingUnknownColumn(CFMetaData
metadata, DataInputPlus in)
+throws IOException
+{
+while (true)
+{
+try
+{
+return readLegacyAtom(metadata, in, false);
+}
+catch (UnknownColumnException e)
+{
+// Simply skip, as the method name implies.
+}
+}
+
+}
+
private static Row getNextRow(CellGrouper grouper, PeekingIterator cells)
{
if (!cells.hasNext())
@@ -1020,29 +1037,36 @@ public abstract class LegacyLayout
};
}
-public static LegacyAtom readLegacyAtom(CFMetaData metadata, DataInputPlus
in, boolean readAllAsDynamic) throws IOException
+public static LegacyAtom readLegacyAtom(CFMetaData metadata, DataInputPlus
in, boolean readAllAsDynamic)
+throws IOException, UnknownColumnException
{
-while (true)
-{
-ByteBuffer cellname = ByteBufferUtil.readWithShortLength(in);
-if (!cellname.hasRemaining())
-return null; // END_OF_ROW
-
-try
-{
-int b = in.readUnsignedByte();
-return (b & RANGE_TOMBSTONE_MASK) != 0
-? readLegacyRangeTombstoneBody(metadata, in, cellname)
-: readLegacyCellBody(metadata, in, cellname, b,
SerializationHelper.Flag.LOCAL, readAllAsDynamic);
-}
-catch (UnknownColumnException e)
-{
-// We can get there if we read a cell for a dropped column,
and ff that is the case,
-// then simply ignore the cell is fine. But also not that we
ignore if it's the
-// system keyspace because for those table we actually remove
columns without registering
-// them in the dropped columns
-assert metadata.ksName.equals(SystemKeyspace.NAME) ||
metadata.getDroppedColumnDefinition(e.columnName) != null : e.getMessage();
-}
+ByteBuffer cellname = ByteBufferUtil.readWithShortLength(in);
+if (!cellname.hasRemaining())
+return null; // END_OF_ROW
+
+try
+{
+int b = in.readUnsignedByte();
+return (b & RANGE_TOMBSTONE_MASK) != 0
+ ? readLegacyRangeTombstoneBody(metadata, in, cellname)
+ : readLegacyCellBody(metadata, in, cellname, b,
SerializationHelper.Flag.LOCAL, readAllAsDynamic);
+}
+catch (UnknownColumnException e)
+{
+// We legitimately can get here in 2 cases:
+// 1) for system tables, because we've unceremoniously removed
columns (without registering them as dropped)
+// 2) for dropped columns.
+// In any other case, there is a mismatch between the schema and
the data, and we complain loudly in
+// that case. Note that if we are in a legit case of an unknown
column, we want to simply skip that cell,
+// but we don't do this here and re-throw the excep
[02/12] cassandra git commit: Fix handling of cells for removed column when reading legacy sstables
Fix handling of cells for removed column when reading legacy sstables
Patch by Sylvain Lebresne; reviewed by Robert Stupp for CASSANDRA-13939
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5378ba27
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5378ba27
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5378ba27
Branch: refs/heads/cassandra-3.11
Commit: 5378ba27ff66b4f39e34d613f99b0f2eddf7990d
Parents: c074412
Author: Sylvain Lebresne
Authored: Thu Oct 5 10:30:56 2017 +0200
Committer: Sylvain Lebresne
Committed: Fri Oct 6 16:12:56 2017 +0200
--
.../org/apache/cassandra/db/LegacyLayout.java | 70 +---
.../cassandra/db/UnfilteredDeserializer.java| 29 +---
2 files changed, 67 insertions(+), 32 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/5378ba27/src/java/org/apache/cassandra/db/LegacyLayout.java
--
diff --git a/src/java/org/apache/cassandra/db/LegacyLayout.java
b/src/java/org/apache/cassandra/db/LegacyLayout.java
index 40b9fd3..3ba96a6 100644
--- a/src/java/org/apache/cassandra/db/LegacyLayout.java
+++ b/src/java/org/apache/cassandra/db/LegacyLayout.java
@@ -649,7 +649,7 @@ public abstract class LegacyLayout
boolean foundOne = false;
LegacyAtom atom;
-while ((atom = readLegacyAtom(metadata, in, false)) != null)
+while ((atom = readLegacyAtomSkippingUnknownColumn(metadata,in)) !=
null)
{
if (atom.isCell())
{
@@ -672,6 +672,23 @@ public abstract class LegacyLayout
return foundOne ? builder.build() : Rows.EMPTY_STATIC_ROW;
}
+private static LegacyAtom readLegacyAtomSkippingUnknownColumn(CFMetaData
metadata, DataInputPlus in)
+throws IOException
+{
+while (true)
+{
+try
+{
+return readLegacyAtom(metadata, in, false);
+}
+catch (UnknownColumnException e)
+{
+// Simply skip, as the method name implies.
+}
+}
+
+}
+
private static Row getNextRow(CellGrouper grouper, PeekingIterator cells)
{
if (!cells.hasNext())
@@ -1020,29 +1037,36 @@ public abstract class LegacyLayout
};
}
-public static LegacyAtom readLegacyAtom(CFMetaData metadata, DataInputPlus
in, boolean readAllAsDynamic) throws IOException
+public static LegacyAtom readLegacyAtom(CFMetaData metadata, DataInputPlus
in, boolean readAllAsDynamic)
+throws IOException, UnknownColumnException
{
-while (true)
-{
-ByteBuffer cellname = ByteBufferUtil.readWithShortLength(in);
-if (!cellname.hasRemaining())
-return null; // END_OF_ROW
-
-try
-{
-int b = in.readUnsignedByte();
-return (b & RANGE_TOMBSTONE_MASK) != 0
-? readLegacyRangeTombstoneBody(metadata, in, cellname)
-: readLegacyCellBody(metadata, in, cellname, b,
SerializationHelper.Flag.LOCAL, readAllAsDynamic);
-}
-catch (UnknownColumnException e)
-{
-// We can get there if we read a cell for a dropped column,
and ff that is the case,
-// then simply ignore the cell is fine. But also not that we
ignore if it's the
-// system keyspace because for those table we actually remove
columns without registering
-// them in the dropped columns
-assert metadata.ksName.equals(SystemKeyspace.NAME) ||
metadata.getDroppedColumnDefinition(e.columnName) != null : e.getMessage();
-}
+ByteBuffer cellname = ByteBufferUtil.readWithShortLength(in);
+if (!cellname.hasRemaining())
+return null; // END_OF_ROW
+
+try
+{
+int b = in.readUnsignedByte();
+return (b & RANGE_TOMBSTONE_MASK) != 0
+ ? readLegacyRangeTombstoneBody(metadata, in, cellname)
+ : readLegacyCellBody(metadata, in, cellname, b,
SerializationHelper.Flag.LOCAL, readAllAsDynamic);
+}
+catch (UnknownColumnException e)
+{
+// We legitimately can get here in 2 cases:
+// 1) for system tables, because we've unceremoniously removed
columns (without registering them as dropped)
+// 2) for dropped columns.
+// In any other case, there is a mismatch between the schema and
the data, and we complain loudly in
+// that case. Note that if we are in a legit case of an unknown
column, we want to simply skip that cell,
+// but we don't do this here and re-throw
[09/12] cassandra git commit: Ninja-fix changelog
Ninja-fix changelog Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8a424cef Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8a424cef Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8a424cef Branch: refs/heads/cassandra-3.11 Commit: 8a424cef32270e75b3a4565793b2d436e6df181f Parents: 5378ba2 Author: Sylvain Lebresne Authored: Fri Oct 6 16:18:07 2017 +0200 Committer: Sylvain Lebresne Committed: Fri Oct 6 16:18:07 2017 +0200 -- CHANGES.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/8a424cef/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index df05f7f..6977be7 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,6 +1,6 @@ 3.0.16 - * Deserialise sstable metadata in nodetool verify (CASSANDRA-13922) - + * Mishandling of cells for removed/dropped columns when reading legacy files (CASSANDRA-13939) + * Deserialise sstable metadata in nodetool verify (CASSANDRA-13922) 3.0.15 * Improve TRUNCATE performance (CASSANDRA-13909) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[08/12] cassandra git commit: Ninja-fix changelog
Ninja-fix changelog Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8a424cef Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8a424cef Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8a424cef Branch: refs/heads/trunk Commit: 8a424cef32270e75b3a4565793b2d436e6df181f Parents: 5378ba2 Author: Sylvain Lebresne Authored: Fri Oct 6 16:18:07 2017 +0200 Committer: Sylvain Lebresne Committed: Fri Oct 6 16:18:07 2017 +0200 -- CHANGES.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/8a424cef/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index df05f7f..6977be7 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,6 +1,6 @@ 3.0.16 - * Deserialise sstable metadata in nodetool verify (CASSANDRA-13922) - + * Mishandling of cells for removed/dropped columns when reading legacy files (CASSANDRA-13939) + * Deserialise sstable metadata in nodetool verify (CASSANDRA-13922) 3.0.15 * Improve TRUNCATE performance (CASSANDRA-13909) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13813) Don't let user drop (or generally break) tables in system_distributed
[
https://issues.apache.org/jira/browse/CASSANDRA-13813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16194621#comment-16194621
]
Sylvain Lebresne commented on CASSANDRA-13813:
--
bq. I don't believe it's reasonable to keep bugs in the codebase just because
they mask out other bugs.
Just to make sure my position is clear, I'm not suggesting we leave this ticket
unsolved. What I'm suggesting is that it would be possible to forbid truly
harmful changes (droping tables and adding/removing/renaming columns; that part
is the real bug to me) while still allowing the altering of table parameters
(which actually don't seem that problematic to me on principle). Sure, it
requires adding some special casing in {{AlterTableStatement}}, but it's not
very complex either.
bq. when their cluster are unable to pick up our altered defaults, because of
our bumped hardcoded timestamps being smaller than the new ones set via ALTER
TABLE
I'm not sure I understand the problem. If user have manually and knowingly
updated some table params, my guess is that they expect (even rely on) future
changes to defaults to not override their changes. Isn't the whole point why
we've picked 0 for our hardcoded timestamp in fact?
But if there is genunine reason I'm missing that make it dangerous for user to
alter those parameters, that's certainly important to understand. I'm pretty
sure some user _have_ done it (at least to work around CASSANDRA-12701,
possibly for other reasons), and if their change is a time-bomb, we kind of owe
them to disarm that bomb.
But anyway, to sum it up, the patch lgtm from a technical level, and I'm
definitively on board with forbidding clearly harmful changes. The only
specific part I'm unsure is the "altering table parameters" part. In light of
my current understanding, it doesn't look obviously harmful (I can even see it
being desirable if I'm being honest), so I'm uncomfortable removing the ability
in a minor release.
Happy to gather a few other opinions though and if the concensus is that the
patch is fine the way it currently is, no problem, not trying to veto this in
any way.
> Don't let user drop (or generally break) tables in system_distributed
> -
>
> Key: CASSANDRA-13813
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13813
> Project: Cassandra
> Issue Type: Bug
> Components: Distributed Metadata
>Reporter: Sylvain Lebresne
>Assignee: Aleksey Yeschenko
> Fix For: 3.0.x, 3.11.x
>
>
> There is not currently no particular restrictions on schema modifications to
> tables of the {{system_distributed}} keyspace. This does mean you can drop
> those tables, or even alter them in wrong ways like dropping or renaming
> columns. All of which is guaranteed to break stuffs (that is, repair if you
> mess up with on of it's table, or MVs if you mess up with
> {{view_build_status}}).
> I'm pretty sure this was never intended and is an oversight of the condition
> on {{ALTERABLE_SYSTEM_KEYSPACES}} in
> [ClientState|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/ClientState.java#L397].
> That condition is such that any keyspace not listed in
> {{ALTERABLE_SYSTEM_KEYSPACES}} (which happens to be the case for
> {{system_distributed}}) has no specific restrictions whatsoever, while given
> the naming it's fair to assume the intention that exactly the opposite.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13910) Consider deprecating (then removing) read_repair_chance/dclocal_read_repair_chance
[
https://issues.apache.org/jira/browse/CASSANDRA-13910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16194604#comment-16194604
]
Carlos Rolo commented on CASSANDRA-13910:
-
More times that I disable these setting than I left them unchanged. Good with
deprecating!
> Consider deprecating (then removing)
> read_repair_chance/dclocal_read_repair_chance
> --
>
> Key: CASSANDRA-13910
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13910
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Sylvain Lebresne
>Priority: Minor
> Labels: CommunityFeedbackRequested
>
> First, let me clarify so this is not misunderstood that I'm not *at all*
> suggesting to remove the read-repair mechanism of detecting and repairing
> inconsistencies between read responses: that mechanism is imo fine and
> useful. But the {{read_repair_chance}} and {{dclocal_read_repair_chance}}
> have never been about _enabling_ that mechanism, they are about querying all
> replicas (even when this is not required by the consistency level) for the
> sole purpose of maybe read-repairing some of the replica that wouldn't have
> been queried otherwise. Which btw, bring me to reason 1 for considering their
> removal: their naming/behavior is super confusing. Over the years, I've seen
> countless users (and not only newbies) misunderstanding what those options
> do, and as a consequence misunderstand when read-repair itself was happening.
> But my 2nd reason for suggesting this is that I suspect
> {{read_repair_chance}}/{{dclocal_read_repair_chance}} are, especially
> nowadays, more harmful than anything else when enabled. When those option
> kick in, what you trade-off is additional resources consumption (all nodes
> have to execute the read) for a _fairly remote chance_ of having some
> inconsistencies repaired on _some_ replica _a bit faster_ than they would
> otherwise be. To justify that last part, let's recall that:
> # most inconsistencies are actually fixed by hints in practice; and in the
> case where a node stay dead for a long time so that hints ends up timing-out,
> you really should repair the node when it comes back (if not simply
> re-bootstrapping it). Read-repair probably don't fix _that_ much stuff in
> the first place.
> # again, read-repair do happen without those options kicking in. If you do
> reads at {{QUORUM}}, inconsistencies will eventually get read-repaired all
> the same. Just a tiny bit less quickly.
> # I suspect almost everyone use a low "chance" for those options at best
> (because the extra resources consumption is real), so at the end of the day,
> it's up to chance how much faster this fixes inconsistencies.
> Overall, I'm having a hard time imagining real cases where that trade-off
> really make sense. Don't get me wrong, those options had their places a long
> time ago when hints weren't working all that well, but I think they bring
> more confusion than benefits now.
> And I think it's sane to reconsider stuffs every once in a while, and to
> clean up anything that may not make all that much sense anymore, which I
> think is the case here.
> Tl;dr, I feel the benefits brought by those options are very slim at best and
> well overshadowed by the confusion they bring, and not worth maintaining the
> code that supports them (which, to be fair, isn't huge, but getting rid of
> {{ReadCallback.AsyncRepairRunner}} wouldn't hurt for instance).
> Lastly, if the consensus here ends up being that they can have their use in
> weird case and that we fill supporting those cases is worth confusing
> everyone else and maintaining that code, I would still suggest disabling them
> totally by default.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13321) Add a checksum component for the sstable metadata (-Statistics.db) file
[ https://issues.apache.org/jira/browse/CASSANDRA-13321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-13321: Resolution: Fixed Fix Version/s: (was: 4.x) 4.0 Status: Resolved (was: Ready to Commit) tests look "good" - nothing should have been caused by this, committed, thanks for the review! > Add a checksum component for the sstable metadata (-Statistics.db) file > --- > > Key: CASSANDRA-13321 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13321 > Project: Cassandra > Issue Type: Improvement >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson > Fix For: 4.0 > > > Since we keep important information in the sstable metadata file now, we > should add a checksum component for it. One danger being if a bit gets > flipped in repairedAt we could consider the sstable repaired when it is not. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
cassandra git commit: Checksum sstable metadata
Repository: cassandra Updated Branches: refs/heads/trunk c9c792d54 -> cfee3e93b Checksum sstable metadata Patch by marcuse; reviewed by Jason Brown for CASSANDRA-13321 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/cfee3e93 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/cfee3e93 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/cfee3e93 Branch: refs/heads/trunk Commit: cfee3e93bec7e527391da93f5b810b0ef4fb0fc8 Parents: c9c792d Author: Marcus Eriksson Authored: Wed Apr 5 09:20:03 2017 +0200 Committer: Marcus Eriksson Committed: Fri Oct 6 15:25:28 2017 +0200 -- CHANGES.txt | 1 + .../cassandra/io/sstable/format/Version.java| 2 + .../io/sstable/format/big/BigFormat.java| 9 ++- .../io/sstable/metadata/MetadataSerializer.java | 66 +-- .../legacy_na_clust/na-1-big-CompressionInfo.db | Bin 87 -> 87 bytes .../legacy_na_clust/na-1-big-Data.db| Bin 5359 -> 5256 bytes .../legacy_na_clust/na-1-big-Digest.crc32 | 2 +- .../legacy_na_clust/na-1-big-Index.db | Bin 157553 -> 157553 bytes .../legacy_na_clust/na-1-big-Statistics.db | Bin 7070 -> 7103 bytes .../legacy_na_clust/na-1-big-TOC.txt| 8 +-- .../na-1-big-CompressionInfo.db | Bin 87 -> 87 bytes .../legacy_na_clust_compact/na-1-big-Data.db| Bin 5426 -> 5286 bytes .../na-1-big-Digest.crc32 | 2 +- .../legacy_na_clust_compact/na-1-big-Index.db | Bin 157553 -> 157553 bytes .../na-1-big-Statistics.db | Bin 7070 -> 7103 bytes .../legacy_na_clust_compact/na-1-big-TOC.txt| 8 +-- .../na-1-big-CompressionInfo.db | Bin 79 -> 79 bytes .../legacy_na_clust_counter/na-1-big-Data.db| Bin 6015 -> 5797 bytes .../na-1-big-Digest.crc32 | 2 +- .../legacy_na_clust_counter/na-1-big-Index.db | Bin 157553 -> 157553 bytes .../na-1-big-Statistics.db | Bin 7079 -> 7112 bytes .../legacy_na_clust_counter/na-1-big-TOC.txt| 8 +-- .../na-1-big-CompressionInfo.db | Bin 79 -> 79 bytes .../na-1-big-Data.db| Bin 6014 -> 5817 bytes .../na-1-big-Digest.crc32 | 2 +- .../na-1-big-Index.db | Bin 157553 -> 157553 bytes .../na-1-big-Statistics.db | Bin 7079 -> 7112 bytes .../na-1-big-TOC.txt| 8 +-- .../na-1-big-CompressionInfo.db | Bin 47 -> 47 bytes .../legacy_na_simple/na-1-big-Data.db | Bin 92 -> 88 bytes .../legacy_na_simple/na-1-big-Digest.crc32 | 2 +- .../legacy_na_simple/na-1-big-Statistics.db | Bin 4623 -> 4656 bytes .../legacy_na_simple/na-1-big-TOC.txt | 8 +-- .../na-1-big-CompressionInfo.db | Bin 47 -> 47 bytes .../legacy_na_simple_compact/na-1-big-Data.db | Bin 95 -> 91 bytes .../na-1-big-Digest.crc32 | 2 +- .../na-1-big-Statistics.db | Bin 4664 -> 4697 bytes .../legacy_na_simple_compact/na-1-big-TOC.txt | 8 +-- .../na-1-big-CompressionInfo.db | Bin 47 -> 47 bytes .../legacy_na_simple_counter/na-1-big-Data.db | Bin 145 -> 139 bytes .../na-1-big-Digest.crc32 | 2 +- .../na-1-big-Statistics.db | Bin 4632 -> 4665 bytes .../legacy_na_simple_counter/na-1-big-TOC.txt | 8 +-- .../na-1-big-CompressionInfo.db | Bin 47 -> 47 bytes .../na-1-big-Data.db| Bin 146 -> 142 bytes .../na-1-big-Digest.crc32 | 2 +- .../na-1-big-Statistics.db | Bin 4673 -> 4706 bytes .../na-1-big-TOC.txt| 8 +-- .../cassandra/io/sstable/LegacySSTableTest.java | 23 +++ 49 files changed, 134 insertions(+), 47 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/cfee3e93/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index e006db3..90c6b9c 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 4.0 + * Checksum sstable metadata (CASSANDRA-13321) * Expose recent histograms in JmxHistograms (CASSANDRA-13642) * Fix buffer length comparison when decompressing in netty-based streaming (CASSANDRA-13899) * Properly close StreamCompressionInputStream to release any ByteBuf (CASSANDRA-13906) http://git-wip-us.apache.org/repos/asf/cassandra/blob/cfee3e93/src/java/org/apache/cassandra/io/sstable/format/Version.java -- diff --git a/src/java/org/apache/cassandra/io/sstable/format/Version
[jira] [Commented] (CASSANDRA-13813) Don't let user drop (or generally break) tables in system_distributed
[
https://issues.apache.org/jira/browse/CASSANDRA-13813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16194557#comment-16194557
]
Aleksey Yeschenko commented on CASSANDRA-13813:
---
[~slebresne] I don't believe it's reasonable to keep bugs in the codebase just
because they mask out other bugs. Leaving this ability is dangerous, as it
breaks the assumptions we make about replicated system keyspaces, and the way
we alter them. Those who make these changes manually - because we weren't
careful enough and haven't shielded them from being able to - might have to pay
the price later, when their cluster are unable to pick up our altered defaults,
because of our bumped hardcoded timestamps being smaller than the new ones set
via {{ALTER TABLE}}.
FWIW, those who are really keen to work around this issue, might as well alter
{{system_schema.tables}} manually, and get the desired result. With just as
much risk.
Either way, again, I don't believe that keeping the bug in just because it
masks the bug in CASSANDRA-12701 is reasonable. So if you are only slightly
concerned and don't mind really strongly, I'd prefer to go ahead and fix this
issue here and now, before more people shoot themselves in the feet.
> Don't let user drop (or generally break) tables in system_distributed
> -
>
> Key: CASSANDRA-13813
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13813
> Project: Cassandra
> Issue Type: Bug
> Components: Distributed Metadata
>Reporter: Sylvain Lebresne
>Assignee: Aleksey Yeschenko
> Fix For: 3.0.x, 3.11.x
>
>
> There is not currently no particular restrictions on schema modifications to
> tables of the {{system_distributed}} keyspace. This does mean you can drop
> those tables, or even alter them in wrong ways like dropping or renaming
> columns. All of which is guaranteed to break stuffs (that is, repair if you
> mess up with on of it's table, or MVs if you mess up with
> {{view_build_status}}).
> I'm pretty sure this was never intended and is an oversight of the condition
> on {{ALTERABLE_SYSTEM_KEYSPACES}} in
> [ClientState|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/ClientState.java#L397].
> That condition is such that any keyspace not listed in
> {{ALTERABLE_SYSTEM_KEYSPACES}} (which happens to be the case for
> {{system_distributed}}) has no specific restrictions whatsoever, while given
> the naming it's fair to assume the intention that exactly the opposite.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13939) Mishandling of cells for removed/dropped columns when reading legacy files
[
https://issues.apache.org/jira/browse/CASSANDRA-13939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Stupp updated CASSANDRA-13939:
-
Status: Ready to Commit (was: Patch Available)
> Mishandling of cells for removed/dropped columns when reading legacy files
> --
>
> Key: CASSANDRA-13939
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13939
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
>Reporter: Sylvain Lebresne
>Assignee: Sylvain Lebresne
> Fix For: 3.0.x, 3.11.x
>
>
> The tl;dr is that there is a bug in reading legacy files that can manifests
> itself with a trace looking like this:
> {noformat}
> Exception (java.lang.IllegalStateException) encountered during startup: One
> row required, 2 found
> java.lang.IllegalStateException: One row required, 2 found
> at
> org.apache.cassandra.cql3.UntypedResultSet$FromResultSet.one(UntypedResultSet.java:84)
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.readTableTimestamp(LegacySchemaMigrator.java:254)
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.readTable(LegacySchemaMigrator.java:244)
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.lambda$readTables$7(LegacySchemaMigrator.java:238)
> at
> org.apache.cassandra.schema.LegacySchemaMigrator$$Lambda$126/591203139.accept(Unknown
> Source)
> at java.util.ArrayList.forEach(ArrayList.java:1249)
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.readTables(LegacySchemaMigrator.java:238)
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.readKeyspace(LegacySchemaMigrator.java:187)
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.lambda$readSchema$4(LegacySchemaMigrator.java:178)
> at
> org.apache.cassandra.schema.LegacySchemaMigrator$$Lambda$123/1612073393.accept(Unknown
> Source)
> at java.util.ArrayList.forEach(ArrayList.java:1249)
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.readSchema(LegacySchemaMigrator.java:178)
> {noformat}
> The reason this can happen has to do with the handling of legacy files.
> Legacy files are cell based while the 3.0 storage engine is primarily row
> based, so we group those cells into rows early in the deserialization process
> (in {{UnfilteredDeserializer.OldFormatDeserializer}}), but in doing so, we
> can only consider a row finished when we've either reach the end of the
> partition/file, or when we've read a cell that doesn't belong to that row.
> That second case means that when the deserializer returns a given row, the
> underlying file pointer may actually not positioned at the end of that row,
> but rather it may be past the first cell of the next row (which the
> deserializer remembers for future use). Long story short, when we try to
> detect if we're logically past our current index block in
> {{AbstractIterator.IndexState#isPastCurrentBlock(}}), we can't simply rely on
> the file pointer, which again may be a bit more advanced that we logically
> are, and that's the reason for the "correction" in that method. That
> correction is really just the amount of bytes remembered but not yet used in
> the deserializer.
> That "correction" is sometimes wrong however and that's due to the fact that
> in {{LegacyLayout#readLegacyAtom}}, if we get a cell for an dropped or
> removed cell, we ignore that cell (which, in itself, is fine). Problem is
> that this skipping is done within the {{LegacyLayout#readLegacyAtom}} method
> but without {{UnfilteredDeserializer.OldFormatDeserializer}} knowing about
> it. As such, the size of the skipped cell ends up being accounted in the
> "correction" bytes for the next cell we read. Lo and behold, if such cell for
> a removed/dropped column is both the last cell of a CQL row and just before
> an index boundary (pretty unlikely in general btw, but definitively
> possible), then the deserializer will count its size with the first cell of
> the next row, which happens to also be the first cell of the next index
> block. And when the code then tries to figure out if it crossed an index
> boundary, it will over-correct. That is, the {{indexState.updateBlock()}}
> call at the start of {{SSTableIterator.ForwardIndexedReader#computeNext}}
> will not work correctly. This can then make the code return a row that is
> after the requested slice end (and should thus not be returned) because it
> doesn't compare that row to said requested end due to thinking it's not on
> the last index block to read, even though it genuinely is.
> Anyway, the whole explanation is a tad complex, but the fix isn't: we need to
> move the skipping of cells for removed/dropped column a level up so the
> deserializer knows about it and don't silently coun
[jira] [Commented] (CASSANDRA-13939) Mishandling of cells for removed/dropped columns when reading legacy files
[
https://issues.apache.org/jira/browse/CASSANDRA-13939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16194540#comment-16194540
]
Robert Stupp commented on CASSANDRA-13939:
--
+1
> Mishandling of cells for removed/dropped columns when reading legacy files
> --
>
> Key: CASSANDRA-13939
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13939
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
>Reporter: Sylvain Lebresne
>Assignee: Sylvain Lebresne
> Fix For: 3.0.x, 3.11.x
>
>
> The tl;dr is that there is a bug in reading legacy files that can manifests
> itself with a trace looking like this:
> {noformat}
> Exception (java.lang.IllegalStateException) encountered during startup: One
> row required, 2 found
> java.lang.IllegalStateException: One row required, 2 found
> at
> org.apache.cassandra.cql3.UntypedResultSet$FromResultSet.one(UntypedResultSet.java:84)
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.readTableTimestamp(LegacySchemaMigrator.java:254)
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.readTable(LegacySchemaMigrator.java:244)
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.lambda$readTables$7(LegacySchemaMigrator.java:238)
> at
> org.apache.cassandra.schema.LegacySchemaMigrator$$Lambda$126/591203139.accept(Unknown
> Source)
> at java.util.ArrayList.forEach(ArrayList.java:1249)
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.readTables(LegacySchemaMigrator.java:238)
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.readKeyspace(LegacySchemaMigrator.java:187)
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.lambda$readSchema$4(LegacySchemaMigrator.java:178)
> at
> org.apache.cassandra.schema.LegacySchemaMigrator$$Lambda$123/1612073393.accept(Unknown
> Source)
> at java.util.ArrayList.forEach(ArrayList.java:1249)
> at
> org.apache.cassandra.schema.LegacySchemaMigrator.readSchema(LegacySchemaMigrator.java:178)
> {noformat}
> The reason this can happen has to do with the handling of legacy files.
> Legacy files are cell based while the 3.0 storage engine is primarily row
> based, so we group those cells into rows early in the deserialization process
> (in {{UnfilteredDeserializer.OldFormatDeserializer}}), but in doing so, we
> can only consider a row finished when we've either reach the end of the
> partition/file, or when we've read a cell that doesn't belong to that row.
> That second case means that when the deserializer returns a given row, the
> underlying file pointer may actually not positioned at the end of that row,
> but rather it may be past the first cell of the next row (which the
> deserializer remembers for future use). Long story short, when we try to
> detect if we're logically past our current index block in
> {{AbstractIterator.IndexState#isPastCurrentBlock(}}), we can't simply rely on
> the file pointer, which again may be a bit more advanced that we logically
> are, and that's the reason for the "correction" in that method. That
> correction is really just the amount of bytes remembered but not yet used in
> the deserializer.
> That "correction" is sometimes wrong however and that's due to the fact that
> in {{LegacyLayout#readLegacyAtom}}, if we get a cell for an dropped or
> removed cell, we ignore that cell (which, in itself, is fine). Problem is
> that this skipping is done within the {{LegacyLayout#readLegacyAtom}} method
> but without {{UnfilteredDeserializer.OldFormatDeserializer}} knowing about
> it. As such, the size of the skipped cell ends up being accounted in the
> "correction" bytes for the next cell we read. Lo and behold, if such cell for
> a removed/dropped column is both the last cell of a CQL row and just before
> an index boundary (pretty unlikely in general btw, but definitively
> possible), then the deserializer will count its size with the first cell of
> the next row, which happens to also be the first cell of the next index
> block. And when the code then tries to figure out if it crossed an index
> boundary, it will over-correct. That is, the {{indexState.updateBlock()}}
> call at the start of {{SSTableIterator.ForwardIndexedReader#computeNext}}
> will not work correctly. This can then make the code return a row that is
> after the requested slice end (and should thus not be returned) because it
> doesn't compare that row to said requested end due to thinking it's not on
> the last index block to read, even though it genuinely is.
> Anyway, the whole explanation is a tad complex, but the fix isn't: we need to
> move the skipping of cells for removed/dropped column a level up so the
> deserializer knows about it and don't sile
[jira] [Comment Edited] (CASSANDRA-13850) Modifying "cassandra-env.sh"
[ https://issues.apache.org/jira/browse/CASSANDRA-13850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16194483#comment-16194483 ] Amitkumar Ghatwal edited comment on CASSANDRA-13850 at 10/6/17 11:47 AM: - [~cnlwsu] Thanks for your comments . So Xss will work similar to Xmx, Xmn, Xms etc. if user defines it in jvm.opts then the values in cassandra-env.sh won't override it all. Please see comment here - "https://github.com/apache/cassandra/blob/trunk/conf/cassandra-env.sh#L137"; it states clearly that - *Check what parameters were defined on jvm.options file to avoid conflicts * .So we should be ideally safe that this PR wouldn't hamper much.Please do let me know your views as well on this ? was (Author: amitkumar_ghatwal): [~cnlwsu] Thanks for your comments . So Xss will work similar to Xmx, Xmn, Xms etc. if user defines it in jvm.opts then the values in cassandra-env.sh won't override it all. Please see comment here - "https://github.com/ghatwala/cassandra/blob/bb426ca48c0debef18f85a21fef97bcd84afeb2d/conf/cassandra-env.sh#L137"; it states clearly that - *Check what parameters were defined on jvm.options file to avoid conflicts * .So we should be ideally safe that this PR wouldn't hamper much.Please do let me know your views as well on this ? > Modifying "cassandra-env.sh" > > > Key: CASSANDRA-13850 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13850 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Amitkumar Ghatwal > Fix For: 4.x > > > Hi All, > Added support for arch in "cassandra-env.sh " with PR : > https://github.com/apache/cassandra/pull/149 > Regards, > Amit -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13850) Modifying "cassandra-env.sh"
[ https://issues.apache.org/jira/browse/CASSANDRA-13850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16194483#comment-16194483 ] Amitkumar Ghatwal commented on CASSANDRA-13850: --- [~cnlwsu] Thanks for your comments . So Xss will work similar to Xmx, Xmn, Xms etc. if user defines it in jvm.opts then the values in cassandra-env.sh won't override it all. Please see comment here - "https://github.com/ghatwala/cassandra/blob/bb426ca48c0debef18f85a21fef97bcd84afeb2d/conf/cassandra-env.sh#L137"; it states clearly that - *Check what parameters were defined on jvm.options file to avoid conflicts * .So we should be ideally safe that this PR wouldn't hamper much.Please do let me know your views as well on this ? > Modifying "cassandra-env.sh" > > > Key: CASSANDRA-13850 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13850 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Amitkumar Ghatwal > Fix For: 4.x > > > Hi All, > Added support for arch in "cassandra-env.sh " with PR : > https://github.com/apache/cassandra/pull/149 > Regards, > Amit -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Resolved] (CASSANDRA-13937) Cassandra node's startup time increased after increase count of big tables
[
https://issues.apache.org/jira/browse/CASSANDRA-13937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson resolved CASSANDRA-13937.
-
Resolution: Duplicate
Dupe with CASSANDRA-13215
> Cassandra node's startup time increased after increase count of big tables
> ---
>
> Key: CASSANDRA-13937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13937
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: RHEL 7.3
> JDK HotSpot 1.8.0_121-b13
> cassandra-3.11 cluster with 43 nodes in 9 datacenters
> 8vCPU, 100 GB RAM
>Reporter: Andrey Lataev
> Attachments: cassandra.zip, debug.zip
>
>
> In startup time Cassandra spends a long time on read some big Columnfamilies.
> For example, in debug.log:
> {code:java}
> grep SSTableReader.java:506 /var/log/cassandra/debug.log
> <...>
> DEBUG [SSTableBatchOpen:3] 2017-10-04 22:40:05,297 SSTableReader.java:506 -
> Opening
> /egov/data/cassandra/datafiles1/p00smevauditbody/messagelogbody20171003-b341cc709c7511e7b1cfed1e90eb03dc/mc-45242-big
> (19.280MiB)
> DEBUG [SSTableBatchOpen:5] 2017-10-04 22:42:14,188 SSTableReader.java:506 -
> Opening
> /egov/data/cassandra/datafiles1/p00smevauditbody/messagelogbody20171004-f82225509d3e11e7b1cfed1e90eb03dc/mc-49002-big
> (10.607MiB)
> <...>
> DEBUG [SSTableBatchOpen:4] 2017-10-04 22:42:19,792 SSTableReader.java:506 -
> Opening
> /egov/data/cassandra/datafiles1/p00smevauditbody/messagelogbody20171004-f82225509d3e11e7b1cfed1e90eb03dc/mc-47907-big
> (128.172MiB)
> DEBUG [SSTableBatchOpen:1] 2017-10-04 22:44:23,560 SSTableReader.java:506 -
> Opening
> /egov/data/cassandra/datafiles1/pk4smevauditbody/messagelogbody20170324-f918bfa0107b11e7adfc2d0b45a372ac/mc-4-big
> (96.310MiB)
> <..>
> {code}
> SSTableReader.java:506 spent ~ 2 min per every big table in p00smevauditbody
> keyspace.
> I was planned too keep similar tables for the full month...
> So it seems like Cassandra will need more then 1h on startup...
> Does it available to speed up SSTableBatchOpen ?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13813) Don't let user drop (or generally break) tables in system_distributed
[
https://issues.apache.org/jira/browse/CASSANDRA-13813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16194441#comment-16194441
]
Sylvain Lebresne commented on CASSANDRA-13813:
--
So the thing is that due to how the existing code is work, we currently allow
anything on the tables of the {{system_distributed}} tables. And while I doubt
there is any debate that we shouldn't let use drop those tables, or even
drop/rename columns of those tables, some user may have been relying on
altering these table _properties_. CASSANDRA-12701 particularly comes to mind
here: it's kind of a bug that the repair tables in there don't have TTLs and so
user may have legitimately work around it by adding the table TTL themselves.
The attached patch will remove the ability for this work-around (to be extra
precise, user that have already changed the properties would obviously be fine,
it's new user wanting to do it that wouldn't be able to), and this feels
uncomfortable to do in minor releases at least.
I guess, if we make sure to commit CASSANDRA-12701 before this, we at least
remove one imo legitimate reason to modify those tables params and that would
be better. Even without that though, modifying some system tables params isn't
an entirely crazy idea (we can debate whether that's truly useful, but it's not
crazy) and this remove that ability. Don't get me wrong btw, I know that we
currently only allow this for {{system_distributed}} tables and so it's
inconsistent with other system keyspace in the first place, and I'm not
definitively saying we should allow modifying params on all system tables. I'm
just raising the point that the patch removes something that could be
considered as useful by some, and doing so in a minor concerns me a little bit.
> Don't let user drop (or generally break) tables in system_distributed
> -
>
> Key: CASSANDRA-13813
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13813
> Project: Cassandra
> Issue Type: Bug
> Components: Distributed Metadata
>Reporter: Sylvain Lebresne
>Assignee: Aleksey Yeschenko
> Fix For: 3.0.x, 3.11.x
>
>
> There is not currently no particular restrictions on schema modifications to
> tables of the {{system_distributed}} keyspace. This does mean you can drop
> those tables, or even alter them in wrong ways like dropping or renaming
> columns. All of which is guaranteed to break stuffs (that is, repair if you
> mess up with on of it's table, or MVs if you mess up with
> {{view_build_status}}).
> I'm pretty sure this was never intended and is an oversight of the condition
> on {{ALTERABLE_SYSTEM_KEYSPACES}} in
> [ClientState|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/ClientState.java#L397].
> That condition is such that any keyspace not listed in
> {{ALTERABLE_SYSTEM_KEYSPACES}} (which happens to be the case for
> {{system_distributed}}) has no specific restrictions whatsoever, while given
> the naming it's fair to assume the intention that exactly the opposite.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
