[jira] [Created] (CASSANDRA-14904) SSTableloader doesn't understand listening for CQL connections on multiple ports
Kurt Greaves created CASSANDRA-14904: Summary: SSTableloader doesn't understand listening for CQL connections on multiple ports Key: CASSANDRA-14904 URL: https://issues.apache.org/jira/browse/CASSANDRA-14904 Project: Cassandra Issue Type: Bug Reporter: Kurt Greaves Assignee: Ian Cleasby Fix For: 4.0, 3.11.x sstableloader only searches the yaml for native_transport_port, so if native_transport_port_ssl is set and encryption is enabled sstableloader will fail to connect as it will use the non-SSL port for the connection. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14903) Nodetool cfstats prints index name twice
Kurt Greaves created CASSANDRA-14903:
Summary: Nodetool cfstats prints index name twice
Key: CASSANDRA-14903
URL: https://issues.apache.org/jira/browse/CASSANDRA-14903
Project: Cassandra
Issue Type: Bug
Components: Tools
Reporter: Kurt Greaves
Assignee: Ian Cleasby
Fix For: 4.0
{code:java}
CREATE TABLE test.test (
id int PRIMARY KEY,
data text
);
CREATE INDEX test_data_idx ON test.test (data);
ccm node1 nodetool cfstats test
Total number of tables: 40
Keyspace : test
Read Count: 0
Read Latency: NaN ms
Write Count: 0
Write Latency: NaN ms
Pending Flushes: 0
Table (index): test.test_data_idxtest.test_data_idx
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14421) Reenable upgrade tests
[
https://issues.apache.org/jira/browse/CASSANDRA-14421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16675896#comment-16675896
]
Kurt Greaves commented on CASSANDRA-14421:
--
[~aweisberg] I've done some work on this already in CASSANDRA-14443. The branch
I have there refactors how tests are selected and fixes both upgrade tests and
resource intensive tests. I've still got some minor fixes to make and it
probably needs a rebase, but it's probably a better starting point. Would
probably make sense to get it merged first. I can spend some time working on it
in the next week if you're interested.
> Reenable upgrade tests
> --
>
> Key: CASSANDRA-14421
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14421
> Project: Cassandra
> Issue Type: Task
> Components: Testing
>Reporter: Sam Tunnicliffe
>Assignee: Ariel Weisberg
>Priority: Major
> Fix For: 4.0
>
>
> Since dtests were switched to pytest & python3 in CASSANDRA-13134, the
> upgrade tests have been non-functional and are deselected by default (though
> even if you ran with the {{--execute-upgrade-tests}} they wouldn't work).
> They're further broken by CASSANDRA-14420, as {{upgrade_manifest}} relies on
> {{CASSANDRA_VERSION_FROM_BUILD}}. We need to get them, or something
> equivalent, up and running.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14780) Avoid creating empty compaction tasks after truncate
[
https://issues.apache.org/jira/browse/CASSANDRA-14780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Greaves updated CASSANDRA-14780:
-
Issue Type: Bug (was: Improvement)
> Avoid creating empty compaction tasks after truncate
>
>
> Key: CASSANDRA-14780
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14780
> Project: Cassandra
> Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Minor
> Fix For: 4.0
>
>
> There is a chance we create empty {{RepairFinishedCompactionTask}} after a
> table is truncated
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14780) Avoid creating empty compaction tasks after truncate
[
https://issues.apache.org/jira/browse/CASSANDRA-14780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Greaves updated CASSANDRA-14780:
-
Fix Version/s: (was: 4.x)
4.0
> Avoid creating empty compaction tasks after truncate
>
>
> Key: CASSANDRA-14780
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14780
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Minor
> Fix For: 4.0
>
>
> There is a chance we create empty {{RepairFinishedCompactionTask}} after a
> table is truncated
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14829) Make stop-server.bat wait for Cassandra to terminate
[ https://issues.apache.org/jira/browse/CASSANDRA-14829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Greaves updated CASSANDRA-14829: - Assignee: Georg Dietrich Status: Patch Available (was: Open) > Make stop-server.bat wait for Cassandra to terminate > > > Key: CASSANDRA-14829 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14829 > Project: Cassandra > Issue Type: Improvement > Components: Packaging > Environment: Windows 10 >Reporter: Georg Dietrich >Assignee: Georg Dietrich >Priority: Minor > Labels: easyfix, windows > Fix For: 3.11.x, 4.x, 4.0.x > > > While administering a single node Cassandra on Windows, I noticed that the > stop-server.bat script returns before the cassandra process has actually > terminated. For use cases like creating a script "shut down & create backup > of data directory without having to worry about open files, then restart", it > would be good to make stop-server.bat wait for Cassandra to terminate. > All that is needed for that is to change in > apache-cassandra-3.11.3\bin\stop-server.bat "start /B powershell /file ..." > to "start /WAIT /B powershell /file ..." (additional /WAIT parameter). > Does this sound reasonable? > Here is the pull request: https://github.com/apache/cassandra/pull/287 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Resolved] (CASSANDRA-14854) I am keep getting the error with major compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-14854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Greaves resolved CASSANDRA-14854. -- Resolution: Not A Problem Fix Version/s: (was: 3.0.x) > I am keep getting the error with major compaction > -- > > Key: CASSANDRA-14854 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14854 > Project: Cassandra > Issue Type: Bug >Reporter: udkantheti >Priority: Major > > Cannot perform a full major compaction as repaired and unrepaired sstables > cannot be compacted together. These two set of sstables will be compacted > separately. > > Can you please suggest what needs to be done to avoid this ? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14854) I am keep getting the error with major compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-14854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1705#comment-1705 ] Kurt Greaves commented on CASSANDRA-14854: -- That's not an error and is intentional. It's just informational. For the record, this is an issue tracker for bugs in Apache Cassandra. If you need help using Cassandra, I suggest you email the user mailing list or try the irc channel http://cassandra.apache.org/community/ > I am keep getting the error with major compaction > -- > > Key: CASSANDRA-14854 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14854 > Project: Cassandra > Issue Type: Bug >Reporter: udkantheti >Priority: Major > > Cannot perform a full major compaction as repaired and unrepaired sstables > cannot be compacted together. These two set of sstables will be compacted > separately. > > Can you please suggest what needs to be done to avoid this ? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14826) cassandra spinning forever on 1 thread while initializing keyspace
[
https://issues.apache.org/jira/browse/CASSANDRA-14826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652988#comment-16652988
]
Kurt Greaves commented on CASSANDRA-14826:
--
Cassandra doesn't do a great job of supporting such dense nodes. You probably
want to add more nodes...
> cassandra spinning forever on 1 thread while initializing keyspace
> --
>
> Key: CASSANDRA-14826
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14826
> Project: Cassandra
> Issue Type: Bug
>Reporter: ingard mevåg
>Assignee: Marcus Eriksson
>Priority: Major
> Attachments: Screen Shot 2018-10-16 at 14.51.27.png
>
>
> When starting cassandra 3.0.17 it takes a long time to initialize a keyspace.
> top shows 1 thread spinning at 100% cpu. Thread dump shows:
> {code:java}
> "main" - Thread t@1
> java.lang.Thread.State: RUNNABLE
> at java.util.TimSort.mergeHi(TimSort.java:850)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeCollapse(TimSort.java:441)
> at java.util.TimSort.sort(TimSort.java:245)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1454)
> at java.util.Collections.sort(Collections.java:175)
> at
> org.apache.cassandra.db.compaction.LeveledManifest.canAddSSTable(LeveledManifest.java:243)
> at
> org.apache.cassandra.db.compaction.LeveledManifest.add(LeveledManifest.java:146)
> - locked (a
> org.apache.cassandra.db.compaction.LeveledManifest)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.addSSTable(LeveledCompactionStrategy.java:298)
> at
> org.apache.cassandra.db.compaction.CompactionStrategyManager.startup(CompactionStrategyManager.java:135)
> at
> org.apache.cassandra.db.compaction.CompactionStrategyManager.reload(CompactionStrategyManager.java:187)
> - locked <742bfce7> (a
> org.apache.cassandra.db.compaction.CompactionStrategyManager)
> at
> org.apache.cassandra.db.compaction.CompactionStrategyManager.(CompactionStrategyManager.java:75)
> at
> org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:408)
> at
> org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:363)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:579)
> - locked <4e4f20d2> (a java.lang.Class)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:556)
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:368)
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:305)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:129)
> - locked <5318346c> (a java.lang.Class)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:106)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:262)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:569)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:697)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14798) Improve wording around partitioner selection
[
https://issues.apache.org/jira/browse/CASSANDRA-14798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641318#comment-16641318
]
Kurt Greaves commented on CASSANDRA-14798:
--
Make:
{quote}If you are upgrading, you should set this to the same partitioner that
you are currently using.
{quote}
into
{quote}If you are adding nodes or upgrading, you should set this to the same
partitioner that you are currently using.
{quote}
The patch needs to be just your changes. Make sure you have the most recent
copy of the repo and just generate a patch off your changes. You don't even
have to commit them, just make the change locally and then generate a patch
file off the diff:
{{git diff conf/cassandra.yaml > 14798-trunk.patch}}
> Improve wording around partitioner selection
>
>
> Key: CASSANDRA-14798
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14798
> Project: Cassandra
> Issue Type: Improvement
> Components: Documentation and Website
>Reporter: Aaron Ploetz
>Assignee: Aaron Ploetz
>Priority: Trivial
> Fix For: 4.0
>
> Attachments: 14798-trunk.txt
>
>
> Given some recent community interactions on Stack Overflow, Nate McCall asked
> me provide some stronger wording on partitioner selection. Specifically, in
> further discouraging people from using the other partitioners (namely, the
> ByteOrderedPartitioner).
> Right now, this is the language that I'm leaning toward:
> {{# The partitioner is responsible for distributing groups of rows (by}}
> {{# partition key) across nodes in the cluster. The partitioner can NOT be}}
> {{# changed without reloading all data. If you are upgrading, you should set
> this}}
> {{# to the same partitioner that you are currently using.}}
> {{#}}
> {{# The default partitioner is the Murmur3Partitioner. Older partitioners}}
> {{# such as the RandomPartitioner, ByteOrderedPartitioner, and}}
> {{# OrderPreservingPartitioner have been included for backward compatibility
> only.}}
> {{# For new clusters, you should NOT change this value.}}
> {{#}}
> {{partitioner: org.apache.cassandra.dht.Murmur3Partitioner }}
> I'm open to suggested improvements.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-10789) Allow DBAs to kill individual client sessions from certain IP(s) and temporarily block subsequent connections without bouncing JVM
[ https://issues.apache.org/jira/browse/CASSANDRA-10789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Greaves updated CASSANDRA-10789: - Status: Patch Available (was: Awaiting Feedback) > Allow DBAs to kill individual client sessions from certain IP(s) and > temporarily block subsequent connections without bouncing JVM > -- > > Key: CASSANDRA-10789 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10789 > Project: Cassandra > Issue Type: Improvement > Components: Coordination >Reporter: Wei Deng >Assignee: Damien Stevenson >Priority: Major > Labels: 4.0-feature-freeze-review-requested > Fix For: 4.x > > Attachments: 10789-trunk-dtest.txt, 10789-trunk.txt > > > In production, there could be hundreds of clients connected to a Cassandra > cluster (maybe even from different applications), and if they use DataStax > Java Driver, each client will establish at least one TCP connection to a > Cassandra server (see > https://datastax.github.io/java-driver/2.1.9/features/pooling/). This is all > normal and at any given time, you can indeed see hundreds of ESTABLISHED > connections to port 9042 on a C* server (from netstat -na). The problem is > that sometimes when a C* cluster is under heavy load, when the DBA identifies > some client session that sends abusive amount of traffic to the C* server and > would like to stop it, they would like a lightweight approach rather than > shutting down the JVM or rolling restart the whole cluster to kill all > hundreds of connections in order to kill a single client session. If the DBA > had root privilege, they would have been able to do something at the OS > network level to achieve the same goal but oftentimes enterprise DBA role is > separate from OS sysadmin role, so the DBAs usually don't have that privilege. > This is especially helpful when you have a multi-tenant C* cluster and you > want to have the impact for handling such client to be minimal to the other > applications. This feature (killing individual session) seems to be a common > feature in other databases (regardless of whether the client has some > reconnect logic or not). It could be implemented as a JMX MBean method and > exposed through nodetool to the DBAs. > Note due to CQL driver's automated reconnection, simply killing the currently > connected client session will not work well, so the JMX parameter should be > an IP address or a list of IP addresses, so that the Cassandra server can > terminate existing connection with that IP, and block future connection > attempts from that IP for the remaining time until the JVM is restarted. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-10789) Allow DBAs to kill individual client sessions from certain IP(s) and temporarily block subsequent connections without bouncing JVM
[ https://issues.apache.org/jira/browse/CASSANDRA-10789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Greaves updated CASSANDRA-10789: - Status: Awaiting Feedback (was: In Progress) > Allow DBAs to kill individual client sessions from certain IP(s) and > temporarily block subsequent connections without bouncing JVM > -- > > Key: CASSANDRA-10789 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10789 > Project: Cassandra > Issue Type: Improvement > Components: Coordination >Reporter: Wei Deng >Assignee: Damien Stevenson >Priority: Major > Labels: 4.0-feature-freeze-review-requested > Fix For: 4.x > > Attachments: 10789-trunk-dtest.txt, 10789-trunk.txt > > > In production, there could be hundreds of clients connected to a Cassandra > cluster (maybe even from different applications), and if they use DataStax > Java Driver, each client will establish at least one TCP connection to a > Cassandra server (see > https://datastax.github.io/java-driver/2.1.9/features/pooling/). This is all > normal and at any given time, you can indeed see hundreds of ESTABLISHED > connections to port 9042 on a C* server (from netstat -na). The problem is > that sometimes when a C* cluster is under heavy load, when the DBA identifies > some client session that sends abusive amount of traffic to the C* server and > would like to stop it, they would like a lightweight approach rather than > shutting down the JVM or rolling restart the whole cluster to kill all > hundreds of connections in order to kill a single client session. If the DBA > had root privilege, they would have been able to do something at the OS > network level to achieve the same goal but oftentimes enterprise DBA role is > separate from OS sysadmin role, so the DBAs usually don't have that privilege. > This is especially helpful when you have a multi-tenant C* cluster and you > want to have the impact for handling such client to be minimal to the other > applications. This feature (killing individual session) seems to be a common > feature in other databases (regardless of whether the client has some > reconnect logic or not). It could be implemented as a JMX MBean method and > exposed through nodetool to the DBAs. > Note due to CQL driver's automated reconnection, simply killing the currently > connected client session will not work well, so the JMX parameter should be > an IP address or a list of IP addresses, so that the Cassandra server can > terminate existing connection with that IP, and block future connection > attempts from that IP for the remaining time until the JVM is restarted. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-14781) Log message when mutation passed to CommitLog#add(Mutation) is too large is not descriptive enough
[ https://issues.apache.org/jira/browse/CASSANDRA-14781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Greaves reassigned CASSANDRA-14781: Assignee: Tom Petracca > Log message when mutation passed to CommitLog#add(Mutation) is too large is > not descriptive enough > -- > > Key: CASSANDRA-14781 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14781 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths >Reporter: Jordan West >Assignee: Tom Petracca >Priority: Major > Fix For: 3.0.x, 3.11.x, 4.0.x > > Attachments: CASSANDRA-14781.patch, CASSANDRA-14781_3.0.patch > > > When hitting > [https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/commitlog/CommitLog.java#L256-L257], > the log message produced does not help the operator track down what data is > being written. At a minimum the keyspace and cfIds involved would be useful > (and are available) – more detail might not be reasonable to include. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14781) Log message when mutation passed to CommitLog#add(Mutation) is too large is not descriptive enough
[ https://issues.apache.org/jira/browse/CASSANDRA-14781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Greaves updated CASSANDRA-14781: - Status: Patch Available (was: Open) > Log message when mutation passed to CommitLog#add(Mutation) is too large is > not descriptive enough > -- > > Key: CASSANDRA-14781 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14781 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths >Reporter: Jordan West >Assignee: Tom Petracca >Priority: Major > Fix For: 3.0.x, 3.11.x, 4.0.x > > Attachments: CASSANDRA-14781.patch, CASSANDRA-14781_3.0.patch > > > When hitting > [https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/commitlog/CommitLog.java#L256-L257], > the log message produced does not help the operator track down what data is > being written. At a minimum the keyspace and cfIds involved would be useful > (and are available) – more detail might not be reasonable to include. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Resolved] (CASSANDRA-14784) Evaluate defaults in cassandra.yaml for 4.0
[ https://issues.apache.org/jira/browse/CASSANDRA-14784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Greaves resolved CASSANDRA-14784. -- Resolution: Duplicate Fix Version/s: (was: 4.0) > Evaluate defaults in cassandra.yaml for 4.0 > --- > > Key: CASSANDRA-14784 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14784 > Project: Cassandra > Issue Type: Improvement > Components: Configuration >Reporter: Dinesh Joshi >Priority: Major > Labels: 4.0 > > Some settings default to values that are known to cause issues for example > num_tokens has a default value of 256. Anybody who has operated Cassandra > knows to lower it. Setting this as default will cause users issues and has > proven problematic. > Let’s evaluate the defaults in cassandra.yaml and update them for Cassandra > 4.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14784) Evaluate defaults in cassandra.yaml for 4.0
[ https://issues.apache.org/jira/browse/CASSANDRA-14784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630190#comment-16630190 ] Kurt Greaves commented on CASSANDRA-14784: -- Resolving as duplicate of CASSANDRA-13701 > Evaluate defaults in cassandra.yaml for 4.0 > --- > > Key: CASSANDRA-14784 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14784 > Project: Cassandra > Issue Type: Improvement > Components: Configuration >Reporter: Dinesh Joshi >Priority: Major > Labels: 4.0 > > Some settings default to values that are known to cause issues for example > num_tokens has a default value of 256. Anybody who has operated Cassandra > knows to lower it. Setting this as default will cause users issues and has > proven problematic. > Let’s evaluate the defaults in cassandra.yaml and update them for Cassandra > 4.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14791) [utest] tests unable to write system tmp directory
[
https://issues.apache.org/jira/browse/CASSANDRA-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630175#comment-16630175
]
Kurt Greaves commented on CASSANDRA-14791:
--
All the hosts are meant to be set up in the same way... However INFRA use
puppet, which I have absolutely no faith in. The slaves are all completely
managed by INFRA so it's probably a case for an INFRA ticket :/
> [utest] tests unable to write system tmp directory
> --
>
> Key: CASSANDRA-14791
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14791
> Project: Cassandra
> Issue Type: Task
> Components: Testing
>Reporter: Jay Zhuang
>Priority: Minor
>
> Some tests are failing from time to time because it cannot write to directory
> {{/tmp/}}:
> https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-trunk-test/lastCompletedBuild/testReport/org.apache.cassandra.streaming.compression/CompressedInputStreamTest/testException/
> {noformat}
> java.lang.RuntimeException: java.nio.file.AccessDeniedException:
> /tmp/na-1-big-Data.db
> at
> org.apache.cassandra.io.util.SequentialWriter.openChannel(SequentialWriter.java:119)
> at
> org.apache.cassandra.io.util.SequentialWriter.(SequentialWriter.java:152)
> at
> org.apache.cassandra.io.util.SequentialWriter.(SequentialWriter.java:141)
> at
> org.apache.cassandra.io.compress.CompressedSequentialWriter.(CompressedSequentialWriter.java:82)
> at
> org.apache.cassandra.streaming.compression.CompressedInputStreamTest.testCompressedReadWith(CompressedInputStreamTest.java:119)
> at
> org.apache.cassandra.streaming.compression.CompressedInputStreamTest.testException(CompressedInputStreamTest.java:78)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.nio.file.AccessDeniedException: /tmp/na-1-big-Data.db
> at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
> at java.nio.channels.FileChannel.open(FileChannel.java:287)
> at java.nio.channels.FileChannel.open(FileChannel.java:335)
> at
> org.apache.cassandra.io.util.SequentialWriter.openChannel(SequentialWriter.java:100)
> {noformat}
> I guess it's because some Jenkins slaves don't have proper permission set.
> For slave {{cassandra16}}, the tests are fine:
> https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-trunk-test/723/testReport/junit/org.apache.cassandra.streaming.compression/CompressedInputStreamTest/testException/history/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14793) Improve system table handling when losing a disk when using JBOD
[ https://issues.apache.org/jira/browse/CASSANDRA-14793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630172#comment-16630172 ] Kurt Greaves commented on CASSANDRA-14793: -- IDK that pinning the system tables really fixes the issue as we'll still have the problem of losing one disk means a whole node replacement. Theoretically it should be possible to rebuild the (important) system tables in the case they get corrupted. Most things currently should be able to be retrieved from gossip/other nodes. How about if there's something we currently can't retrieve we just store it in the peers table on all the nodes. I can't imagine the amount of info we'd have to store would be very big. Then we can just provide a startup flag to indicate that a node should try and rebuild its system tables. > Improve system table handling when losing a disk when using JBOD > > > Key: CASSANDRA-14793 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14793 > Project: Cassandra > Issue Type: Bug >Reporter: Marcus Eriksson >Priority: Major > Fix For: 4.0 > > > We should improve the way we handle disk failures when losing a disk in a > JBOD setup > One way could be to pin the system tables to a special data directory. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12294) LDAP Authentication
[ https://issues.apache.org/jira/browse/CASSANDRA-12294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621439#comment-16621439 ] Kurt Greaves commented on CASSANDRA-12294: -- There's been a lot of interest in getting our LDAP authenticator in tree, so I'm just going to mention it here and assign this to me. TBH it still needs some work, and ATM it doesn't support 4.0 (but that's irrelevant), but going to dump it here for reference. Not going to be doing any work on this for the duration of the feature freeze, and it'll be targeted for 4.1, but just leaving this here so people/users can express opinions if they see fit. Repo for reference: https://github.com/instaclustr/cassandra-ldap > LDAP Authentication > --- > > Key: CASSANDRA-12294 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12294 > Project: Cassandra > Issue Type: New Feature > Components: Distributed Metadata >Reporter: Daniel Kleviansky >Assignee: Kurt Greaves >Priority: Minor > Labels: security > Fix For: 4.x > > > Addition of an LDAP authentication plugin, in tree, along side the default > authenticator, so that Cassandra can leverage existing LDAP-speaking servers > to manage user logins. > DSE offers this: [Enabling LDAP authentication | > https://docs.datastax.com/en/datastax_enterprise/4.6/datastax_enterprise/sec/secLdapEnabling.html], > but does not exist in vanilla C* as far as I can tell. > Ideally would like to introduce this as part of the 2.2.x branch, as this is > what is currently running in client production environment, and where it is > needed at the moment. > Would aim for support of at least Microsoft Active Directory running on > Windows Server 2012. > Work in progress: https://github.com/lqid/cassandra — Branch 12294-22 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-12294) LDAP Authentication
[ https://issues.apache.org/jira/browse/CASSANDRA-12294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Greaves reassigned CASSANDRA-12294: Assignee: Kurt Greaves (was: Daniel Kleviansky) > LDAP Authentication > --- > > Key: CASSANDRA-12294 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12294 > Project: Cassandra > Issue Type: New Feature > Components: Distributed Metadata >Reporter: Daniel Kleviansky >Assignee: Kurt Greaves >Priority: Minor > Labels: security > Fix For: 4.x > > > Addition of an LDAP authentication plugin, in tree, along side the default > authenticator, so that Cassandra can leverage existing LDAP-speaking servers > to manage user logins. > DSE offers this: [Enabling LDAP authentication | > https://docs.datastax.com/en/datastax_enterprise/4.6/datastax_enterprise/sec/secLdapEnabling.html], > but does not exist in vanilla C* as far as I can tell. > Ideally would like to introduce this as part of the 2.2.x branch, as this is > what is currently running in client production environment, and where it is > needed at the moment. > Would aim for support of at least Microsoft Active Directory running on > Windows Server 2012. > Work in progress: https://github.com/lqid/cassandra — Branch 12294-22 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14463) Prevent the generation of new tokens when using replace_address flag
[ https://issues.apache.org/jira/browse/CASSANDRA-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Greaves updated CASSANDRA-14463: - Status: Patch Available (was: Open) > Prevent the generation of new tokens when using replace_address flag > > > Key: CASSANDRA-14463 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14463 > Project: Cassandra > Issue Type: Bug >Reporter: Vincent White >Assignee: Vincent White >Priority: Minor > > This is a follow up to/replacement of CASSANDRA-14073. > The behaviour that I want to avoid is someone trying to replace a node with > the replace_address flag and mistakenly having that node listed in its own > seed list which causes the node to generate a new set of random tokens before > joining the ring. > Currently anytime an unbootstrapped node is listed in its own seed list and > initial_token isn't set in the yaml, Cassandra will generate a new set of > random tokens and join the ring regardless of whether it was replacing a > previous node or not. > We could simply check for this configuration and refuse to start but I it's > probably better (particularly for 3.0.X) if it's handled in the same manner > as skipping streaming with the allow_unsafe_replace flag that was introduced > in 3.X . This would still allow 3.0.X users the ability to re-bootstrap nodes > without needing to re-stream all the data to the node again, which can be > useful. > We currently handle replacing without streaming different;y between 3.0.X and > 3.X. In 3.X we have the allow_unsafe_replace JVM flag to allow the use of > auto_bootstrap: false in combination with the replace_address option. But in > 3.0.X to perform the replacement of a node with the same IP address without > streaming I believe you need to: > * Set replace_address (because the address is already in gossip) > * Include the node in its own seed list (to skip bootstrapping/streaming) > * Set the initial_token to the token/s owned by the previous node (to > prevent it generating new tokens. > I believe if 3.0.X simply refused to start when a node has itself in its seed > list and replace_address set this will completely block this operation. > Example patches to fix this edge case using allow_unsafe_replace: > > ||Branch|| > |[3.0.x|https://github.com/apache/cassandra/compare/trunk...vincewhite:30-no_clobber]| > |[3.x|https://github.com/apache/cassandra/compare/trunk...vincewhite:311-no_clobber]| -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14525) streaming failure during bootstrap makes new node into inconsistent state
[
https://issues.apache.org/jira/browse/CASSANDRA-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620099#comment-16620099
]
Kurt Greaves commented on CASSANDRA-14525:
--
Looks good to me. Probably need another run of the dtests though :/
> streaming failure during bootstrap makes new node into inconsistent state
> -
>
> Key: CASSANDRA-14525
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14525
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Major
> Fix For: 4.0, 2.2.x, 3.0.x
>
>
> If bootstrap fails for newly joining node (most common reason is due to
> streaming failure) then Cassandra state remains in {{joining}} state which is
> fine but Cassandra also enables Native transport which makes overall state
> inconsistent. This further creates NullPointer exception if auth is enabled
> on the new node, please find reproducible steps here:
> For example if bootstrap fails due to streaming errors like
> {quote}java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1256)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:894)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:660)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:573)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:330)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:567)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:695)
> [apache-cassandra-3.0.16.jar:3.0.16]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:187)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:440)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:540)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:307)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
> {quote}
> then variable [StorageService.java::dataAvailable
> |https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageService.java#L892]
> will be {{false}}. Since {{dataAvailable}} is {{false}} hence it will not
> call [StorageService.java::finishJoiningRing
> |https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageService.java#L933]
> and as a result
> [StorageService.java::doAuthSetup|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageService.java#L999]
> will not be
[jira] [Assigned] (CASSANDRA-14463) Prevent the generation of new tokens when using replace_address flag
[ https://issues.apache.org/jira/browse/CASSANDRA-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Greaves reassigned CASSANDRA-14463: Assignee: Vincent White > Prevent the generation of new tokens when using replace_address flag > > > Key: CASSANDRA-14463 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14463 > Project: Cassandra > Issue Type: Bug >Reporter: Vincent White >Assignee: Vincent White >Priority: Minor > > This is a follow up to/replacement of CASSANDRA-14073. > The behaviour that I want to avoid is someone trying to replace a node with > the replace_address flag and mistakenly having that node listed in its own > seed list which causes the node to generate a new set of random tokens before > joining the ring. > Currently anytime an unbootstrapped node is listed in its own seed list and > initial_token isn't set in the yaml, Cassandra will generate a new set of > random tokens and join the ring regardless of whether it was replacing a > previous node or not. > We could simply check for this configuration and refuse to start but I it's > probably better (particularly for 3.0.X) if it's handled in the same manner > as skipping streaming with the allow_unsafe_replace flag that was introduced > in 3.X . This would still allow 3.0.X users the ability to re-bootstrap nodes > without needing to re-stream all the data to the node again, which can be > useful. > We currently handle replacing without streaming different;y between 3.0.X and > 3.X. In 3.X we have the allow_unsafe_replace JVM flag to allow the use of > auto_bootstrap: false in combination with the replace_address option. But in > 3.0.X to perform the replacement of a node with the same IP address without > streaming I believe you need to: > * Set replace_address (because the address is already in gossip) > * Include the node in its own seed list (to skip bootstrapping/streaming) > * Set the initial_token to the token/s owned by the previous node (to > prevent it generating new tokens. > I believe if 3.0.X simply refused to start when a node has itself in its seed > list and replace_address set this will completely block this operation. > Example patches to fix this edge case using allow_unsafe_replace: > > ||Branch|| > |[3.0.x|https://github.com/apache/cassandra/compare/trunk...vincewhite:30-no_clobber]| > |[3.x|https://github.com/apache/cassandra/compare/trunk...vincewhite:311-no_clobber]| -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-14721) sstabledump displays incorrect value for "position" key
[
https://issues.apache.org/jira/browse/CASSANDRA-14721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Greaves reassigned CASSANDRA-14721:
Assignee: Cameron Zemek
> sstabledump displays incorrect value for "position" key
> ---
>
> Key: CASSANDRA-14721
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14721
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
>Reporter: Damien Stevenson
>Assignee: Cameron Zemek
>Priority: Minor
> Attachments: cassandra-dump.patch
>
>
> When partitions with multiple rows are displayed using sstabledump, the
> "position" value the first row of each partition is incorrect.
> For example:
> {code:java}
> sstabledump mc-1-big-Data.db
> [
> {
> "partition" : {
> "key" : [ "1", "24" ],
> "position" : 0
> },
> "rows" : [
> {
> "type" : "row",
> "position" : 66,
> "clustering" : [ "2013-12-10 00:00:00.000Z" ],
> "liveness_info" : { "tstamp" : "2018-09-12T05:01:09.290086Z" },
> "cells" : [
> { "name" : "centigrade", "value" : 8 },
> { "name" : "chanceofrain", "value" : 0.1 },
> { "name" : "feelslike", "value" : 8 },
> { "name" : "humidity", "value" : 0.76 },
> { "name" : "wind", "value" : 10.0 }
> ]
> },
> {
> "type" : "row",
> "position" : 66,
> "clustering" : [ "2013-12-11 00:00:00.000Z" ],
> "liveness_info" : { "tstamp" : "2018-09-12T05:01:09.295369Z" },
> "cells" : [
> { "name" : "centigrade", "value" : 4 },
> { "name" : "chanceofrain", "value" : 0.3 },
> { "name" : "feelslike", "value" : 4 },
> { "name" : "humidity", "value" : 0.9 },
> { "name" : "wind", "value" : 12.0 }
> ]
> },
> {
> "type" : "row",
> "position" : 105,
> "clustering" : [ "2013-12-12 00:00:00.000Z" ],
> "liveness_info" : { "tstamp" : "2018-09-12T05:01:09.300841Z" },
> "cells" : [
> { "name" : "centigrade", "value" : 3 },
> { "name" : "chanceofrain", "value" : 0.2 },
> { "name" : "feelslike", "value" : 3 },
> { "name" : "humidity", "value" : 0.68 },
> { "name" : "wind", "value" : 6.0 }
> ]
> }
> ]
> }
> ]
> {code}
> The expected output is:
> {code:java}
> [
> {
> "partition" : {
> "key" : [ "1", "24" ],
> "position" : 0
> },
> "rows" : [
> {
> "type" : "row",
> "position" : 28,
> "clustering" : [ "2013-12-10 00:00:00.000Z" ],
> "liveness_info" : { "tstamp" : "2018-09-12T05:01:09.290086Z" },
> "cells" : [
> { "name" : "centigrade", "value" : 8 },
> { "name" : "chanceofrain", "value" : 0.1 },
> { "name" : "feelslike", "value" : 8 },
> { "name" : "humidity", "value" : 0.76 },
> { "name" : "wind", "value" : 10.0 }
> ]
> },
> {
> "type" : "row",
> "position" : 66,
> "clustering" : [ "2013-12-11 00:00:00.000Z" ],
> "liveness_info" : { "tstamp" : "2018-09-12T05:01:09.295369Z" },
> "cells" : [
> { "name" : "centigrade", "value" : 4 },
> { "name" : "chanceofrain", "value" : 0.3 },
> { "name" : "feelslike", "value" : 4 },
> { "name" : "humidity", "value" : 0.9 },
> { "name" : "wind", "value" : 12.0 }
> ]
> },
> {
> "type" : "row",
> "position" : 105,
> "clustering" : [ "2013-12-12 00:00:00.000Z" ],
> "liveness_info" : { "tstamp" : "2018-09-12T05:01:09.300841Z" },
> "cells" : [
> { "name" : "centigrade", "value" : 3 },
> { "name" : "chanceofrain", "value" : 0.2 },
> { "name" : "feelslike", "value" : 3 },
> { "name" : "humidity", "value" : 0.68 },
> { "name" : "wind", "value" : 6.0 }
> ]
> }
> ]
> }
> ]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14721) sstabledump displays incorrect value for "position" key
[
https://issues.apache.org/jira/browse/CASSANDRA-14721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Greaves updated CASSANDRA-14721:
-
Status: Patch Available (was: Open)
> sstabledump displays incorrect value for "position" key
> ---
>
> Key: CASSANDRA-14721
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14721
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
>Reporter: Damien Stevenson
>Assignee: Cameron Zemek
>Priority: Minor
> Attachments: cassandra-dump.patch
>
>
> When partitions with multiple rows are displayed using sstabledump, the
> "position" value the first row of each partition is incorrect.
> For example:
> {code:java}
> sstabledump mc-1-big-Data.db
> [
> {
> "partition" : {
> "key" : [ "1", "24" ],
> "position" : 0
> },
> "rows" : [
> {
> "type" : "row",
> "position" : 66,
> "clustering" : [ "2013-12-10 00:00:00.000Z" ],
> "liveness_info" : { "tstamp" : "2018-09-12T05:01:09.290086Z" },
> "cells" : [
> { "name" : "centigrade", "value" : 8 },
> { "name" : "chanceofrain", "value" : 0.1 },
> { "name" : "feelslike", "value" : 8 },
> { "name" : "humidity", "value" : 0.76 },
> { "name" : "wind", "value" : 10.0 }
> ]
> },
> {
> "type" : "row",
> "position" : 66,
> "clustering" : [ "2013-12-11 00:00:00.000Z" ],
> "liveness_info" : { "tstamp" : "2018-09-12T05:01:09.295369Z" },
> "cells" : [
> { "name" : "centigrade", "value" : 4 },
> { "name" : "chanceofrain", "value" : 0.3 },
> { "name" : "feelslike", "value" : 4 },
> { "name" : "humidity", "value" : 0.9 },
> { "name" : "wind", "value" : 12.0 }
> ]
> },
> {
> "type" : "row",
> "position" : 105,
> "clustering" : [ "2013-12-12 00:00:00.000Z" ],
> "liveness_info" : { "tstamp" : "2018-09-12T05:01:09.300841Z" },
> "cells" : [
> { "name" : "centigrade", "value" : 3 },
> { "name" : "chanceofrain", "value" : 0.2 },
> { "name" : "feelslike", "value" : 3 },
> { "name" : "humidity", "value" : 0.68 },
> { "name" : "wind", "value" : 6.0 }
> ]
> }
> ]
> }
> ]
> {code}
> The expected output is:
> {code:java}
> [
> {
> "partition" : {
> "key" : [ "1", "24" ],
> "position" : 0
> },
> "rows" : [
> {
> "type" : "row",
> "position" : 28,
> "clustering" : [ "2013-12-10 00:00:00.000Z" ],
> "liveness_info" : { "tstamp" : "2018-09-12T05:01:09.290086Z" },
> "cells" : [
> { "name" : "centigrade", "value" : 8 },
> { "name" : "chanceofrain", "value" : 0.1 },
> { "name" : "feelslike", "value" : 8 },
> { "name" : "humidity", "value" : 0.76 },
> { "name" : "wind", "value" : 10.0 }
> ]
> },
> {
> "type" : "row",
> "position" : 66,
> "clustering" : [ "2013-12-11 00:00:00.000Z" ],
> "liveness_info" : { "tstamp" : "2018-09-12T05:01:09.295369Z" },
> "cells" : [
> { "name" : "centigrade", "value" : 4 },
> { "name" : "chanceofrain", "value" : 0.3 },
> { "name" : "feelslike", "value" : 4 },
> { "name" : "humidity", "value" : 0.9 },
> { "name" : "wind", "value" : 12.0 }
> ]
> },
> {
> "type" : "row",
> "position" : 105,
> "clustering" : [ "2013-12-12 00:00:00.000Z" ],
> "liveness_info" : { "tstamp" : "2018-09-12T05:01:09.300841Z" },
> "cells" : [
> { "name" : "centigrade", "value" : 3 },
> { "name" : "chanceofrain", "value" : 0.2 },
> { "name" : "feelslike", "value" : 3 },
> { "name" : "humidity", "value" : 0.68 },
> { "name" : "wind", "value" : 6.0 }
> ]
> }
> ]
> }
> ]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14285) Comma at the end of the end of the seed list is interpretated as localhost
[
https://issues.apache.org/jira/browse/CASSANDRA-14285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605136#comment-16605136
]
Kurt Greaves commented on CASSANDRA-14285:
--
IDK how many people were doing it, or if we do it elsewhere in ccm/our testing,
but this also means you can't have an empty seed list - which used to resolve
to localhost. Honestly, that's probably less surprising, especially considering
it's not documented at the moment, but thought I'd mention it here because it
surprised me.
FTR, if you're using {{ccm add}}, you now _have_ to specify {{-s}} for at least
one node. If people think this is annoying or find other issues we can easily
add to this patch to handle that case.
> Comma at the end of the end of the seed list is interpretated as localhost
> --
>
> Key: CASSANDRA-14285
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14285
> Project: Cassandra
> Issue Type: Bug
> Components: Configuration
>Reporter: Marco
>Assignee: Nicolas Guyomar
>Priority: Minor
> Fix For: 4.0
>
>
> Seeds: '10.1.20.10,10.1.21.10,10.1.22.10,' cause a flood of the debug log
> with messages like this one.
> DEBUG [MessagingService-Outgoing-localhost/127.0.0.1-Gossip] 2018-02-28
> 15:53:57,314 OutboundTcpConnection.java:545 - Unable to connect to
> localhost/[127.0.0.1|http://127.0.0.1/]
> This code provide by Nicolas Guyomar provide the reason of the issue.
> In SImpleSeedProvider :
>
> String[] hosts = "10.1.20.10,10.1.21.10,10.1.22.10,".split(",", -1);
> List seeds = new ArrayList(hosts.length);
> for (String host : hosts)
> {
> System.out.println(InetAddress.getByName(host.trim()));
> }
>
> output :
> /[10.1.20.10|http://10.1.20.10/]
> /[10.1.21.10|http://10.1.21.10/]
> /[10.1.22.10|http://10.1.22.10/]
> localhost/[127.0.0.1|http://127.0.0.1/]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14687) Revert parameterization of AuthCacheMBean interface
[
https://issues.apache.org/jira/browse/CASSANDRA-14687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602801#comment-16602801
]
Kurt Greaves commented on CASSANDRA-14687:
--
+1. My bad, should have checked that.
> Revert parameterization of AuthCacheMBean interface
> ---
>
> Key: CASSANDRA-14687
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14687
> Project: Cassandra
> Issue Type: Bug
> Components: Auth
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
>Priority: Major
> Fix For: 4.0
>
>
> In CASSANDRA-14662, a type parameter {{}} and {{invalidate}} method
> were added to {{AuthCacheMBean}} with the intention that this would
> automatically expose via JMX the {{invalidate}} method from {{AuthCache}}
> itself. Actually, this is not the case, as type erasure obscures the actual
> type of the parameter and so JMX clients like jconsole and jmc disable access
> to the method.
> Only {{PasswordAuthenticator.CredentialsCache}} provided method like this
> previously, via {{CredentialsCacheMBean extends AuthCacheMBean}}, so the most
> straightforward fix is to just revert the change to {{AuthCacheMBean}}.
> Future/alternative cache implementations can continue to specify their own
> MBean interfaces to add specialised methods as before.
> I should've caught this before committing the CASSANDRA-14662 as it broke a
> couple of dtests in AuthTest, mea culpa.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14525) streaming failure during bootstrap makes new node into inconsistent state
[
https://issues.apache.org/jira/browse/CASSANDRA-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594895#comment-16594895
]
Kurt Greaves commented on CASSANDRA-14525:
--
Thanks so much. I have been meaning to get back to this but since we missed the
boat on 3.11.3 I figured this could wait until Sept 1st. If you want you can go
ahead and fix the tests, otherwise if not I'll have a look next week :)
> streaming failure during bootstrap makes new node into inconsistent state
> -
>
> Key: CASSANDRA-14525
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14525
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Major
> Fix For: 4.0, 2.2.x, 3.0.x
>
>
> If bootstrap fails for newly joining node (most common reason is due to
> streaming failure) then Cassandra state remains in {{joining}} state which is
> fine but Cassandra also enables Native transport which makes overall state
> inconsistent. This further creates NullPointer exception if auth is enabled
> on the new node, please find reproducible steps here:
> For example if bootstrap fails due to streaming errors like
> {quote}java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1256)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:894)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:660)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:573)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:330)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:567)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:695)
> [apache-cassandra-3.0.16.jar:3.0.16]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:187)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:440)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:540)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:307)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
> {quote}
> then variable [StorageService.java::dataAvailable
> |https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageService.java#L892]
> will be {{false}}. Since {{dataAvailable}} is {{false}} hence it will not
> call [StorageService.java::finishJoiningRing
> |https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageService.java#L933]
> and as a result
>
[jira] [Commented] (CASSANDRA-14662) Refactor AuthCache
[
https://issues.apache.org/jira/browse/CASSANDRA-14662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594889#comment-16594889
]
Kurt Greaves commented on CASSANDRA-14662:
--
Sorry I was being totally dumb and at some point along the way I'd convinced
myself they were all using the same functions. Herp derp waste everybody's
time. I can't say I'm totally convinced about the indirection and the clarity
provided by it but it's not important.
I've made all the changes as you specified above (I think). I left in changes
to accessibility of certain functions/variables in AuthCache to make it easier
to extend if people need to (e.g they need/want to set more cache policies)
which I think are fine, and left in added comments.
|[trunk|https://github.com/apache/cassandra/compare/trunk...kgreav:authcache2]|
|[utests|https://circleci.com/gh/kgreav/cassandra/208]|
> Refactor AuthCache
> --
>
> Key: CASSANDRA-14662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14662
> Project: Cassandra
> Issue Type: Improvement
> Components: Auth
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Major
> Labels: security
> Fix For: 4.x
>
>
> When building an LDAP IAuthenticator plugin I ran into a few issues when
> trying to reuse the AuthCache similar to how PasswordAuthenticator implements
> it. Most of the problems stemmed from the underlying cache being inaccessible
> and not being able to override {{initCache}} properly.
> Anyway, I've had a stab at refactoring AuthCache with the following
> improvements:
> # Make it possible to extend and override all necessary methods (initCache,
> init, validate)
> # Makes it possible to specify a {{CacheLoader}} rather than just a
> {{Function}}, allowing you to have a get/load that throws exceptions.
> # Use AuthCache on its own rather than extending it for each use case
> ({{invalidate(K)}} moved to be part of MBean)
> # Provided a builder that uses sane defaults so we don't have unnecessary
> repeated code everywhere
> The refactor made all the extensions of AuthCache unnecessary, so I've
> simplified those cases to use AuthCache and removed any classes extending
> AuthCache. I also removed some noop compatibility classes that were marked to
> be removed in 4.0.
> Also added some tests in AuthCacheTest.
> |[trunk|https://github.com/apache/cassandra/compare/trunk...kgreav:authcache]|
> |[utests|https://circleci.com/gh/kgreav/cassandra/206]|
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14662) Refactor AuthCache
[
https://issues.apache.org/jira/browse/CASSANDRA-14662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591490#comment-16591490
]
Kurt Greaves commented on CASSANDRA-14662:
--
Thanks for the quick review.
bq. So, I'm good with the changes to the MBeans but I'm not super keen on any
of the other bits. Can you verify that the problems you had with the custom
IAuthenticator can't be solved by modifying the load function as described?
Yep thanks, that's a worthwhile solution. Will remove the unnecessary
CacheLoader options.
bq. With the specialized impls the boilerplate is hidden behind the
constructor, so the caller doesn't need to know about it.
I don't really agree here. There's a specialised implementation for each usage
of the cache, so I don't think we really make anything any clearer. All we've
done is add a layer of indirection. If the implementations were used more than
once I think that could make sense, but none of them are, and we don't even use
their {{get()}} calls more than once.
{quote}The builder pattern is most useful when constructors have a mix of
mandatory and optional arguments. AuthCache itself doesn't have any optional
arguments, but some of the values are domain specific and so the
implementations can supply them.
Making things like the validity & refresh values optional and defaulting them
to an arbitrary set of config props is not an improvement.
{quote}
Why is having defaults not an improvement? They are literally the same for
every usage/implementation of the AuthCache in the codebase which means they
aren't arbitrary for all the existing use cases. Seems like a case for sane
defaults to me?
I mean, I'm not super tied to the builder, but I really don't see the point in
having the exact same code in four different places so that we can claim that
the caller benefits by not having to "know" about it, especially because we
haven't found reason to expose these implementations anyway so it's always just
an implementation detail of the (single) class that uses each one.
> Refactor AuthCache
> --
>
> Key: CASSANDRA-14662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14662
> Project: Cassandra
> Issue Type: Improvement
> Components: Auth
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Major
> Labels: security
> Fix For: 4.x
>
>
> When building an LDAP IAuthenticator plugin I ran into a few issues when
> trying to reuse the AuthCache similar to how PasswordAuthenticator implements
> it. Most of the problems stemmed from the underlying cache being inaccessible
> and not being able to override {{initCache}} properly.
> Anyway, I've had a stab at refactoring AuthCache with the following
> improvements:
> # Make it possible to extend and override all necessary methods (initCache,
> init, validate)
> # Makes it possible to specify a {{CacheLoader}} rather than just a
> {{Function}}, allowing you to have a get/load that throws exceptions.
> # Use AuthCache on its own rather than extending it for each use case
> ({{invalidate(K)}} moved to be part of MBean)
> # Provided a builder that uses sane defaults so we don't have unnecessary
> repeated code everywhere
> The refactor made all the extensions of AuthCache unnecessary, so I've
> simplified those cases to use AuthCache and removed any classes extending
> AuthCache. I also removed some noop compatibility classes that were marked to
> be removed in 4.0.
> Also added some tests in AuthCacheTest.
> |[trunk|https://github.com/apache/cassandra/compare/trunk...kgreav:authcache]|
> |[utests|https://circleci.com/gh/kgreav/cassandra/206]|
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction
[ https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589503#comment-16589503 ] Kurt Greaves commented on CASSANDRA-10540: -- We've been asking to help for months now, if that's all it needed we could of had them written by now. [This|https://github.com/apache/cassandra/compare/trunk...krummas:marcuse/10540] is your latest branch correct? I'll start looking at writing tests, although there's minimal hope of having anything ready by the first. > RangeAwareCompaction > > > Key: CASSANDRA-10540 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10540 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Major > Labels: 4.0-feature-freeze-review-requested, compaction, lcs, > vnodes > Fix For: 4.x > > > Broken out from CASSANDRA-6696, we should split sstables based on ranges > during compaction. > Requirements; > * dont create tiny sstables - keep them bunched together until a single vnode > is big enough (configurable how big that is) > * make it possible to run existing compaction strategies on the per-range > sstables > We should probably add a global compaction strategy parameter that states > whether this should be enabled or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-10726) Read repair inserts should not be blocking
[
https://issues.apache.org/jira/browse/CASSANDRA-10726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589495#comment-16589495
]
Kurt Greaves commented on CASSANDRA-10726:
--
Great, thanks Blake. I just needed a quick fix to get my tests running :).
> Read repair inserts should not be blocking
> --
>
> Key: CASSANDRA-10726
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10726
> Project: Cassandra
> Issue Type: Improvement
> Components: Coordination
>Reporter: Richard Low
>Assignee: Blake Eggleston
>Priority: Major
> Fix For: 4.0
>
>
> Today, if there’s a digest mismatch in a foreground read repair, the insert
> to update out of date replicas is blocking. This means, if it fails, the read
> fails with a timeout. If a node is dropping writes (maybe it is overloaded or
> the mutation stage is backed up for some other reason), all reads to a
> replica set could fail. Further, replicas dropping writes get more out of
> sync so will require more read repair.
> The comment on the code for why the writes are blocking is:
> {code}
> // wait for the repair writes to be acknowledged, to minimize impact on any
> replica that's
> // behind on writes in case the out-of-sync row is read multiple times in
> quick succession
> {code}
> but the bad side effect is that reads timeout. Either the writes should not
> be blocking or we should return success for the read even if the write times
> out.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14662) Refactor AuthCache
[
https://issues.apache.org/jira/browse/CASSANDRA-14662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Greaves updated CASSANDRA-14662:
-
Status: Patch Available (was: Open)
> Refactor AuthCache
> --
>
> Key: CASSANDRA-14662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14662
> Project: Cassandra
> Issue Type: Improvement
> Components: Auth
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Major
>
> When building an LDAP IAuthenticator plugin I ran into a few issues when
> trying to reuse the AuthCache similar to how PasswordAuthenticator implements
> it. Most of the problems stemmed from the underlying cache being inaccessible
> and not being able to override {{initCache}} properly.
> Anyway, I've had a stab at refactoring AuthCache with the following
> improvements:
> # Make it possible to extend and override all necessary methods (initCache,
> init, validate)
> # Makes it possible to specify a {{CacheLoader}} rather than just a
> {{Function}}, allowing you to have a get/load that throws exceptions.
> # Use AuthCache on its own rather than extending it for each use case
> ({{invalidate(K)}} moved to be part of MBean)
> # Provided a builder that uses sane defaults so we don't have unnecessary
> repeated code everywhere
> The refactor made all the extensions of AuthCache unnecessary, so I've
> simplified those cases to use AuthCache and removed any classes extending
> AuthCache. I also removed some noop compatibility classes that were marked to
> be removed in 4.0.
> Also added some tests in AuthCacheTest.
> |[trunk|https://github.com/apache/cassandra/compare/trunk...kgreav:authcache]|
> |[utests|https://circleci.com/gh/kgreav/cassandra/206]|
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14662) Refactor AuthCache
Kurt Greaves created CASSANDRA-14662:
Summary: Refactor AuthCache
Key: CASSANDRA-14662
URL: https://issues.apache.org/jira/browse/CASSANDRA-14662
Project: Cassandra
Issue Type: Improvement
Components: Auth
Reporter: Kurt Greaves
Assignee: Kurt Greaves
When building an LDAP IAuthenticator plugin I ran into a few issues when trying
to reuse the AuthCache similar to how PasswordAuthenticator implements it. Most
of the problems stemmed from the underlying cache being inaccessible and not
being able to override {{initCache}} properly.
Anyway, I've had a stab at refactoring AuthCache with the following
improvements:
# Make it possible to extend and override all necessary methods (initCache,
init, validate)
# Makes it possible to specify a {{CacheLoader}} rather than just a
{{Function}}, allowing you to have a get/load that throws exceptions.
# Use AuthCache on its own rather than extending it for each use case
({{invalidate(K)}} moved to be part of MBean)
# Provided a builder that uses sane defaults so we don't have unnecessary
repeated code everywhere
The refactor made all the extensions of AuthCache unnecessary, so I've
simplified those cases to use AuthCache and removed any classes extending
AuthCache. I also removed some noop compatibility classes that were marked to
be removed in 4.0.
Also added some tests in AuthCacheTest.
|[trunk|https://github.com/apache/cassandra/compare/trunk...kgreav:authcache]|
|[utests|https://circleci.com/gh/kgreav/cassandra/206]|
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14662) Refactor AuthCache
[
https://issues.apache.org/jira/browse/CASSANDRA-14662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Greaves updated CASSANDRA-14662:
-
Fix Version/s: 4.x
> Refactor AuthCache
> --
>
> Key: CASSANDRA-14662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14662
> Project: Cassandra
> Issue Type: Improvement
> Components: Auth
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Major
> Fix For: 4.x
>
>
> When building an LDAP IAuthenticator plugin I ran into a few issues when
> trying to reuse the AuthCache similar to how PasswordAuthenticator implements
> it. Most of the problems stemmed from the underlying cache being inaccessible
> and not being able to override {{initCache}} properly.
> Anyway, I've had a stab at refactoring AuthCache with the following
> improvements:
> # Make it possible to extend and override all necessary methods (initCache,
> init, validate)
> # Makes it possible to specify a {{CacheLoader}} rather than just a
> {{Function}}, allowing you to have a get/load that throws exceptions.
> # Use AuthCache on its own rather than extending it for each use case
> ({{invalidate(K)}} moved to be part of MBean)
> # Provided a builder that uses sane defaults so we don't have unnecessary
> repeated code everywhere
> The refactor made all the extensions of AuthCache unnecessary, so I've
> simplified those cases to use AuthCache and removed any classes extending
> AuthCache. I also removed some noop compatibility classes that were marked to
> be removed in 4.0.
> Also added some tests in AuthCacheTest.
> |[trunk|https://github.com/apache/cassandra/compare/trunk...kgreav:authcache]|
> |[utests|https://circleci.com/gh/kgreav/cassandra/206]|
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction
[ https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588733#comment-16588733 ] Kurt Greaves commented on CASSANDRA-10540: -- [~krummas] Any hope of getting this in by 4.0? I'd be committed to testing/verifying/benchmarking post freeze if we can get it in. Code-wise seems fine to me, and our testing so far seems to show that it's viable. Any reason we can't get this committed and fix/revert during freeze if issues come up? > RangeAwareCompaction > > > Key: CASSANDRA-10540 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10540 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Major > Labels: 4.0-feature-freeze-review-requested, compaction, lcs, > vnodes > Fix For: 4.x > > > Broken out from CASSANDRA-6696, we should split sstables based on ranges > during compaction. > Requirements; > * dont create tiny sstables - keep them bunched together until a single vnode > is big enough (configurable how big that is) > * make it possible to run existing compaction strategies on the per-range > sstables > We should probably add a global compaction strategy parameter that states > whether this should be enabled or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-10726) Read repair inserts should not be blocking
[
https://issues.apache.org/jira/browse/CASSANDRA-10726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588731#comment-16588731
]
Kurt Greaves commented on CASSANDRA-10726:
--
[~bdeggleston], looks like this [broke
eclipse-warnings|https://circleci.com/gh/kgreav/cassandra/204] because we don't
close the SingleRangeResponse in {{sendNextRequests}}. I fixed it with [this
change|https://github.com/apache/cassandra/commit/75f9bcb40bd98f4c0c364f9d26d46be88a1f107f].
> Read repair inserts should not be blocking
> --
>
> Key: CASSANDRA-10726
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10726
> Project: Cassandra
> Issue Type: Improvement
> Components: Coordination
>Reporter: Richard Low
>Assignee: Blake Eggleston
>Priority: Major
> Fix For: 4.0
>
>
> Today, if there’s a digest mismatch in a foreground read repair, the insert
> to update out of date replicas is blocking. This means, if it fails, the read
> fails with a timeout. If a node is dropping writes (maybe it is overloaded or
> the mutation stage is backed up for some other reason), all reads to a
> replica set could fail. Further, replicas dropping writes get more out of
> sync so will require more read repair.
> The comment on the code for why the writes are blocking is:
> {code}
> // wait for the repair writes to be acknowledged, to minimize impact on any
> replica that's
> // behind on writes in case the out-of-sync row is read multiple times in
> quick succession
> {code}
> but the bad side effect is that reads timeout. Either the writes should not
> be blocking or we should return success for the read even if the write times
> out.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12464) Investigate the potential improvement of parallelism on higher level compactions in LCS
[ https://issues.apache.org/jira/browse/CASSANDRA-12464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578084#comment-16578084 ] Kurt Greaves commented on CASSANDRA-12464: -- All yours [~psivaraju] > Investigate the potential improvement of parallelism on higher level > compactions in LCS > --- > > Key: CASSANDRA-12464 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12464 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Wei Deng >Assignee: Pramod K Sivaraju >Priority: Major > Labels: lcs, lhf, performance > > According to LevelDB's design doc > [here|https://github.com/google/leveldb/blob/master/doc/impl.md#compactions], > "A compaction merges the contents of the picked files to produce a sequence > of level-(L+1) files", it will "switch to producing a new level-(L+1) file > after the current output file has reached the target file size" (in our case > 160MB), it will also "switch to a new output file when the key range of the > current output file has grown enough to overlap more than ten level-(L+2) > files". This is to ensure "that a later compaction of a level-(L+1) file will > not pick up too much data from level-(L+2)." > Our current code in LeveledCompactionStrategy doesn't implement this last > rule, but we might be able to quickly implement it and see how much a > compaction throughput improvement it can deliver. Potentially we can create a > scenario where a number of large L0 SSTables are present (e.g. 200GB after > switching from STCS) and let it to create thousands of L1 SSTables overflow, > and see how fast LCS can digest this much data from L1 and properly > upper-level them to completion. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-12464) Investigate the potential improvement of parallelism on higher level compactions in LCS
[ https://issues.apache.org/jira/browse/CASSANDRA-12464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Greaves reassigned CASSANDRA-12464: Assignee: Pramod K Sivaraju (was: Pramod) > Investigate the potential improvement of parallelism on higher level > compactions in LCS > --- > > Key: CASSANDRA-12464 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12464 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Wei Deng >Assignee: Pramod K Sivaraju >Priority: Major > Labels: lcs, lhf, performance > > According to LevelDB's design doc > [here|https://github.com/google/leveldb/blob/master/doc/impl.md#compactions], > "A compaction merges the contents of the picked files to produce a sequence > of level-(L+1) files", it will "switch to producing a new level-(L+1) file > after the current output file has reached the target file size" (in our case > 160MB), it will also "switch to a new output file when the key range of the > current output file has grown enough to overlap more than ten level-(L+2) > files". This is to ensure "that a later compaction of a level-(L+1) file will > not pick up too much data from level-(L+2)." > Our current code in LeveledCompactionStrategy doesn't implement this last > rule, but we might be able to quickly implement it and see how much a > compaction throughput improvement it can deliver. Potentially we can create a > scenario where a number of large L0 SSTables are present (e.g. 200GB after > switching from STCS) and let it to create thousands of L1 SSTables overflow, > and see how fast LCS can digest this much data from L1 and properly > upper-level them to completion. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-12464) Investigate the potential improvement of parallelism on higher level compactions in LCS
[ https://issues.apache.org/jira/browse/CASSANDRA-12464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Greaves reassigned CASSANDRA-12464: Assignee: Pramod > Investigate the potential improvement of parallelism on higher level > compactions in LCS > --- > > Key: CASSANDRA-12464 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12464 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Wei Deng >Assignee: Pramod >Priority: Major > Labels: lcs, lhf, performance > > According to LevelDB's design doc > [here|https://github.com/google/leveldb/blob/master/doc/impl.md#compactions], > "A compaction merges the contents of the picked files to produce a sequence > of level-(L+1) files", it will "switch to producing a new level-(L+1) file > after the current output file has reached the target file size" (in our case > 160MB), it will also "switch to a new output file when the key range of the > current output file has grown enough to overlap more than ten level-(L+2) > files". This is to ensure "that a later compaction of a level-(L+1) file will > not pick up too much data from level-(L+2)." > Our current code in LeveledCompactionStrategy doesn't implement this last > rule, but we might be able to quickly implement it and see how much a > compaction throughput improvement it can deliver. Potentially we can create a > scenario where a number of large L0 SSTables are present (e.g. 200GB after > switching from STCS) and let it to create thousands of L1 SSTables overflow, > and see how fast LCS can digest this much data from L1 and properly > upper-level them to completion. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13701) Lower default num_tokens
[ https://issues.apache.org/jira/browse/CASSANDRA-13701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552732#comment-16552732 ] Kurt Greaves commented on CASSANDRA-13701: -- Possibly, but it's doubling up on meanings for a single property. I can see a case where your default is not necessarily the same RF as your biggest keyspace (which is what you want to set the allocation algorithm to), and it's not exactly clear how it would work with multi-dc and different RF's per DC. I think Datastax did it right by moving to specifying the RF including multi-dc in the yaml. > Lower default num_tokens > > > Key: CASSANDRA-13701 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13701 > Project: Cassandra > Issue Type: Improvement >Reporter: Chris Lohfink >Priority: Minor > > For reasons highlighted in CASSANDRA-7032, the high number of vnodes is not > necessary. It is very expensive for operations processes and scanning. Its > come up a lot and its pretty standard and known now to always reduce the > num_tokens within the community. We should just lower the defaults. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14557) Consider adding default and required keyspace replication options
[
https://issues.apache.org/jira/browse/CASSANDRA-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552725#comment-16552725
]
Kurt Greaves commented on CASSANDRA-14557:
--
How does this work when adding a new DC? Will it be the default RF in each DC?
> Consider adding default and required keyspace replication options
> -
>
> Key: CASSANDRA-14557
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14557
> Project: Cassandra
> Issue Type: Improvement
> Components: Configuration
>Reporter: Sumanth Pasupuleti
>Assignee: Sumanth Pasupuleti
>Priority: Minor
> Labels: 4.0-feature-freeze-review-requested
> Fix For: 4.0
>
> Attachments: 14557-trunk.txt
>
>
> Ending up with a keyspace of RF=1 is unfortunately pretty easy in C* right
> now - the system_auth table for example is created with RF=1 (to take into
> account single node setups afaict from CASSANDRA-5112), and a user can
> further create a keyspace with RF=1 posing availability and streaming risks
> (e.g. rebuild).
> I propose we add two configuration options in cassandra.yaml:
> # {{default_keyspace_rf}} (default: 1) - If replication factors are not
> specified, use this number.
> # {{required_minimum_keyspace_rf}} (default: unset) - Prevent users from
> creating a keyspace with an RF less than what is configured
> These settings could further be re-used to:
> * Provide defaults for new keyspaces created with SimpleStrategy or
> NetworkTopologyStrategy (CASSANDRA-14303)
> * Make the automatic token [allocation
> algorithm|https://issues.apache.org/jira/browse/CASSANDRA-13701?focusedCommentId=16095662=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16095662]
> interface more intuitive allowing easy use of the new token allocation
> algorithm.
> At the end of the day, if someone really wants to allow RF=1, they simply
> don’t set the setting. For backwards compatibility the default remains 1 and
> C* would create with RF=1, and would default to current behavior of allowing
> any RF on keyspaces.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14325) Java executable check succeeds despite no java on PATH
[ https://issues.apache.org/jira/browse/CASSANDRA-14325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Greaves updated CASSANDRA-14325: - Assignee: Angelo Polo Status: Patch Available (was: Open) > Java executable check succeeds despite no java on PATH > -- > > Key: CASSANDRA-14325 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14325 > Project: Cassandra > Issue Type: Bug > Components: Lifecycle >Reporter: Angelo Polo >Assignee: Angelo Polo >Priority: Minor > Attachments: bin_cassandra.patch > > > The check -z $JAVA on line 102 of bin/cassandra currently always succeeds if > JAVA_HOME is not set since in this case JAVA gets set directly to 'java'. The > error message "_Unable to find java executable. Check JAVA_HOME and PATH > environment variables._" will never be echoed on a PATH misconfiguration. If > java isn't on the PATH the failure will instead occur on line 95 of > cassandra-env.sh at the java version check. > It would be better to check consistently for the java executable in one place > in bin/cassandra. Also we don't want users to mistakenly think they have a > java version problem when they in fact have a PATH problem. > See proposed patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14328) Invalid metadata has been detected for role
[
https://issues.apache.org/jira/browse/CASSANDRA-14328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552714#comment-16552714
]
Kurt Greaves commented on CASSANDRA-14328:
--
[~prnvjndl] If this issue is still occurring can you send the output of:
{code}
select * from system_auth.roles where role = 'utorjwcnruzzlzafxffgyqmlvkxiqcgb'
{code}
You can remove any credentials if you see fit.
> Invalid metadata has been detected for role
> ---
>
> Key: CASSANDRA-14328
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14328
> Project: Cassandra
> Issue Type: Bug
> Components: CQL
>Reporter: Pranav Jindal
>Priority: Major
>
> Cassandra Version : 3.10
> One node was replaced and was successfully up and working but CQL-SH fails
> with error.
>
> CQL-SH error:
>
> {code:java}
> Connection error: ('Unable to connect to any servers', {'10.180.0.150':
> AuthenticationFailed('Failed to authenticate to 10.180.0.150: Error from
> server: code= [Server error] message="java.lang.RuntimeException: Invalid
> metadata has been detected for role utorjwcnruzzlzafxffgyqmlvkxiqcgb"',)})
> {code}
>
> Cassandra server ERROR:
> {code:java}
> WARN [Native-Transport-Requests-1] 2018-03-20 13:37:17,894
> CassandraRoleManager.java:96 - An invalid value has been detected in the
> roles table for role utorjwcnruzzlzafxffgyqmlvkxiqcgb. If you are unable to
> login, you may need to disable authentication and confirm that values in that
> table are accurate
> ERROR [Native-Transport-Requests-1] 2018-03-20 13:37:17,895 Message.java:623
> - Unexpected exception during request; channel = [id: 0xdfc3604f,
> L:/10.180.0.150:9042 - R:/10.180.0.150:51668]
> java.lang.RuntimeException: Invalid metadata has been detected for role
> utorjwcnruzzlzafxffgyqmlvkxiqcgb
> at
> org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:99)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:82)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:528)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:503)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.auth.CassandraRoleManager.canLogin(CassandraRoleManager.java:310)
> ~[apache-cassandra-3.10.jar:3.10]
> at org.apache.cassandra.service.ClientState.login(ClientState.java:271)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:80)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517)
> [apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
> [apache-cassandra-3.10.jar:3.10]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
> [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357)
> [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_121]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> [apache-cassandra-3.10.jar:3.10]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
> [apache-cassandra-3.10.jar:3.10]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.cassandra.cql3.UntypedResultSet$Row.getBoolean(UntypedResultSet.java:273)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:88)
> ~[apache-cassandra-3.10.jar:3.10]
> ... 16 common frames omitted
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Reopened] (CASSANDRA-14401) Attempted serializing to buffer exceeded maximum of 65535 bytes
[
https://issues.apache.org/jira/browse/CASSANDRA-14401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Greaves reopened CASSANDRA-14401:
--
Was going to close this but realised it's possibly still actually a bug in
Cassandra.
[~arokhin] do you have an example schema and query string that causes this?
Also what plugin you are using if that applies?
> Attempted serializing to buffer exceeded maximum of 65535 bytes
>
>
> Key: CASSANDRA-14401
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14401
> Project: Cassandra
> Issue Type: Bug
>Reporter: Artem Rokhin
>Priority: Major
>
> Cassandra version: 3.11.2
> 3 nodes cluster
> The following exception appears on all 3 nodes and after awhile cluster
> becomes unreposnsive
>
> {code}
> java.lang.AssertionError: Attempted serializing to buffer exceeded maximum of
> 65535 bytes: 67661
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithShortLength(ByteBufferUtil.java:309)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.db.filter.RowFilter$Expression$Serializer.serialize(RowFilter.java:547)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.db.filter.RowFilter$Serializer.serialize(RowFilter.java:1143)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.db.ReadCommand$Serializer.serialize(ReadCommand.java:726)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.db.ReadCommand$Serializer.serialize(ReadCommand.java:683)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.io.ForwardingVersionedSerializer.serialize(ForwardingVersionedSerializer.java:45)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:385)
> [apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:337)
> [apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:263)
> [apache-cassandra-3.11.2.jar:3.11.2]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Resolved] (CASSANDRA-14401) Attempted serializing to buffer exceeded maximum of 65535 bytes
[
https://issues.apache.org/jira/browse/CASSANDRA-14401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Greaves resolved CASSANDRA-14401.
--
Resolution: Invalid
> Attempted serializing to buffer exceeded maximum of 65535 bytes
>
>
> Key: CASSANDRA-14401
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14401
> Project: Cassandra
> Issue Type: Bug
>Reporter: Artem Rokhin
>Priority: Major
>
> Cassandra version: 3.11.2
> 3 nodes cluster
> The following exception appears on all 3 nodes and after awhile cluster
> becomes unreposnsive
>
> {code}
> java.lang.AssertionError: Attempted serializing to buffer exceeded maximum of
> 65535 bytes: 67661
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithShortLength(ByteBufferUtil.java:309)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.db.filter.RowFilter$Expression$Serializer.serialize(RowFilter.java:547)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.db.filter.RowFilter$Serializer.serialize(RowFilter.java:1143)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.db.ReadCommand$Serializer.serialize(ReadCommand.java:726)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.db.ReadCommand$Serializer.serialize(ReadCommand.java:683)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.io.ForwardingVersionedSerializer.serialize(ForwardingVersionedSerializer.java:45)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:385)
> [apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:337)
> [apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:263)
> [apache-cassandra-3.11.2.jar:3.11.2]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Resolved] (CASSANDRA-14424) Gossip EchoMessages not being handled somewhere after node restart
[
https://issues.apache.org/jira/browse/CASSANDRA-14424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Greaves resolved CASSANDRA-14424.
--
Resolution: Duplicate
> Gossip EchoMessages not being handled somewhere after node restart
> --
>
> Key: CASSANDRA-14424
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14424
> Project: Cassandra
> Issue Type: Bug
> Environment: cassandra 3.11.2 - brand new ring - 18 nodes.
> ubuntu 16.04
> AWS - cross AZ, with GossipingPropertyFileSnitch setting the rack to the AZs.
>Reporter: Jason Harvey
>Priority: Major
> Fix For: 3.11.x, 4.x
>
>
> Noticing this behaviour on a brand new 3.11.2 ring:
> # Restart a random node in the ring.
> # When that node comes back up, around 30% of the time it sees a single
> other node down. No other node in the ring sees that node is down.
> # After 10-20 minutes, the DOWN node suddenly appears UP to the restarted
> node.
>
> After digging through tracing logs, here's what I know:
>
> The node seen as DOWN has not gone down, but simply hasn't been seen as UP
> yet. The restarted node is attempting to `markAlive()` the target node.
> Relevant logs from the restarted node's POV:
>
> {{INFO [GossipStage:1] 2018-04-27 14:03:50,950 Gossiper.java:1053 - Node
> /10.0.225.147 has restarted, now UP}}
> {{INFO [GossipStage:1] 2018-04-27 14:03:50,969 StorageService.java:2292 -
> Node /10.0.225.147 state jump to NORMAL}}
> {{INFO [HANDSHAKE-/10.0.225.147] 2018-04-27 14:03:50,976
> OutboundTcpConnection.java:560 - Handshaking version with /10.0.225.147}}
> {{INFO [GossipStage:1] 2018-04-27 14:03:50,977 TokenMetadata.java:479 -
> Updating topology for /10.0.225.147}}
> {{INFO [GossipStage:1] 2018-04-27 14:03:50,977 TokenMetadata.java:479 -
> Updating topology for /10.0.225.147}}
>
> (note that despite the Gossip seeing the DOWN node as 'UP', nodetool status
> still shows it as 'DOWN', as markAlive has not completed, and will not
> actually be seen as 'UP' for 20 more minutes)
>
> The restarted node is repeatedly sending Echo messages to the DOWN node as
> part of the `markAlive()` call. The DOWN node is receiving those, and claims
> to be sending a response. However, the restarted node is not marking the DOWN
> node as UP even after the DOWN node sends the Echo response.
>
> Relevant logs from the restarted node's POV:
>
> {{TRACE [GossipStage:1] 2018-04-27 14:11:28,792 MessagingService.java:945 -
> 10.0.103.45 sending ECHO to 99248@/10.0.225.147}}
> {{TRACE [GossipTasks:1] 2018-04-27 14:11:29,792 MessagingService.java:945 -
> 10.0.103.45 sending GOSSIP_DIGEST_SYN to 99631@/10.0.225.147}}
> {{TRACE [GossipStage:1] 2018-04-27 14:11:29,792 MessagingService.java:945 -
> 10.0.103.45 sending ECHO to 99632@/10.0.225.147}}
> {{TRACE [GossipStage:1] 2018-04-27 14:11:29,793 MessagingService.java:945 -
> 10.0.103.45 sending GOSSIP_DIGEST_ACK2 to 99633@/10.0.225.147}}
> {{TRACE [GossipStage:1] 2018-04-27 14:11:29,793 MessagingService.java:945 -
> 10.0.103.45 sending ECHO to 99635@/10.0.225.147}}
> {{TRACE [GossipStage:1] 2018-04-27 14:11:31,794 MessagingService.java:945 -
> 10.0.103.45 sending ECHO to 100348@/10.0.225.147}}
> {{TRACE [GossipStage:1] 2018-04-27 14:11:33,750 MessagingService.java:945 -
> 10.0.103.45 sending ECHO to 101157@/10.0.225.147}}
> {{TRACE [GossipStage:1] 2018-04-27 14:11:35,412 MessagingService.java:945 -
> 10.0.103.45 sending ECHO to 101753@/10.0.225.147}}
>
>
> Relevant logs from the DOWN node's POV:
>
> {{TRACE [GossipStage:1] 2018-04-27 14:18:16,500 EchoVerbHandler.java:39 -
> Sending a EchoMessage reply /10.0.103.45}}
> {{TRACE [GossipStage:1] 2018-04-27 14:18:16,500 MessagingService.java:945 -
> 10.0.225.147 sending REQUEST_RESPONSE to 328389@/10.0.103.45}}
> {{TRACE [GossipStage:1] 2018-04-27 14:18:17,679 EchoVerbHandler.java:39 -
> Sending a EchoMessage reply /10.0.103.45}}
> {{TRACE [GossipStage:1] 2018-04-27 14:18:17,679 MessagingService.java:945 -
> 10.0.225.147 sending REQUEST_RESPONSE to 329412@/10.0.103.45}}
> {{TRACE [GossipStage:1] 2018-04-27 14:18:18,680 EchoVerbHandler.java:39 -
> Sending a EchoMessage reply /10.0.103.45}}
> {{TRACE [GossipStage:1] 2018-04-27 14:18:18,680 MessagingService.java:945 -
> 10.0.225.147 sending REQUEST_RESPONSE to 330185@/10.0.103.45}}
>
>
> The metrics on the restarted node show that the MessagingService has a large
> number of TimeoutsPerHost for the DOWN node, and all other nodes have 0
> timeouts.
>
>
> Eventually, `realMarkAlive()` is called and the restarted node finally sees
> DOWN node as coming up, and it spams several UP messages when this happens:
>
>
> {{INFO [RequestResponseStage-7] 2018-04-27 14:19:27,210 Gossiper.java:1019 -
> InetAddress /10.0.225.147 is now UP}}
>
[jira] [Updated] (CASSANDRA-14476) ShortType and ByteType are incorrectly considered variable-length types
[ https://issues.apache.org/jira/browse/CASSANDRA-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Greaves updated CASSANDRA-14476: - Labels: lhf (was: ) > ShortType and ByteType are incorrectly considered variable-length types > --- > > Key: CASSANDRA-14476 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14476 > Project: Cassandra > Issue Type: Bug >Reporter: Vladimir Krivopalov >Priority: Minor > Labels: lhf > > The AbstractType class has a method valueLengthIfFixed() that returns -1 for > data types with a variable length and a positive value for types with a fixed > length. This is primarily used for efficient serialization and > deserialization. > > It turns out that there is an inconsistency in types ShortType and ByteType > as those are in fact fixed-length types (2 bytes and 1 byte, respectively) > but they don't have the valueLengthIfFixed() method overloaded and it returns > -1 as if they were of variable length. > > It would be good to fix that at some appropriate point, for example, when > introducing a new version of SSTables format, to keep the meaning of the > function consistent across data types. Saving some bytes in serialized format > is a minor but pleasant bonus. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14483) Bootstrap stream fails with Configuration exception merging remote schema
[
https://issues.apache.org/jira/browse/CASSANDRA-14483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552665#comment-16552665
]
Kurt Greaves commented on CASSANDRA-14483:
--
Was that the only error? The ColumnFamily ID mismatch may not be related to the
streaming failure. Did you try resume bootstrap after the failure and did it
complete?
> Bootstrap stream fails with Configuration exception merging remote schema
> -
>
> Key: CASSANDRA-14483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14483
> Project: Cassandra
> Issue Type: Bug
> Components: Configuration
>Reporter: Yongxin Cen
>Priority: Major
> Fix For: 3.11.2
>
>
> I configured yaml file for a seed node, and start it up, cqlsh into it, and
> create keyspace kong with replication =
> \{'class':'SimpleStrategy','replication_factor':2};
> create user kong with password 'xxx';
> And create tables in keyspace kong.
>
> Then, in another Cassandra node, point to the seed, and start Cassandra
> service in the new node
> Run command "nodetool status kong" shows the new node Owns ?, seed owns 100%.
> Run command "nodetool bootstrap resume",
> Resuming bootstrap
> [2018-05-31 04:15:57,807] prepare with IP_Seed complete (progress: 0%)
> [2018-05-31 04:15:57,921] received file system_auth/roles (progress: 50%)
> [2018-05-31 04:15:57,960] session with IP_Seed complete (progress: 50%)
> [2018-05-31 04:15:57,965] Stream failed
> [2018-05-31 04:15:57,966] Error during bootstrap: Stream failed
> [2018-05-31 04:15:57,966] Resume bootstrap complete
> At the end of /var/log/cassandra/cassandra.log, there are errors:
> ERROR [InternalResponseStage:2] 2018-05-31 00:02:30,559 MigrationTask.java:95
> - Configuration exception merging remote schema
> org.apache.cassandra.exceptions.ConfigurationException: Column family ID
> mismatch (found cce68250-63d6-11e8-b887-09f7d93c2253; expected
> 41679dd0-2804-11e8-a8d4-cd6631f48e81)
> at
> org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaData.java:941)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:895)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at org.apache.cassandra.config.Schema.updateTable(Schema.java:687)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1464)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1420)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1389)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1366)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.service.MigrationTask$1.response(MigrationTask.java:91)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53)
> [apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
> [apache-cassandra-3.11.2.jar:3.11.2]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_161]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [na:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [na:1.8.0_161]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> [apache-cassandra-3.11.2.jar:3.11.2]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_161]
> ERROR [main] 2018-05-31 00:02:58,417 StorageService.java:1524 - Error
> while waiting on bootstrap to complete. Bootstrap will have to be restarted.
> java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1519)
> [apache-cassandra-3.11.2.jar:3.11.2]
> at
>
[jira] [Commented] (CASSANDRA-14584) insert if not exists, with replication factor of 2 doesn't work
[ https://issues.apache.org/jira/browse/CASSANDRA-14584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552633#comment-16552633 ] Kurt Greaves commented on CASSANDRA-14584: -- You'll need multiple nodes if you want to actually have RF=2 and do QUORUM. You should either add more nodes or drop your RF to 1. > insert if not exists, with replication factor of 2 doesn't work > --- > > Key: CASSANDRA-14584 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14584 > Project: Cassandra > Issue Type: Bug >Reporter: arik >Priority: Major > > Running with a single node cluster. > My keyspace has a replication factor of 2. > Insert if not exists doesn't work on that setup. > Produce the following error: > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:720) > Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All > host(s) tried for query failed (tried: cassandra-service/10.23.251.29:9042 > (com.datastax.driver.core.exceptions.UnavailableException: Not enough > replicas available for query at consistency QUORUM (2 required but only 1 > alive))) at > com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:223) > at > com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:41) > at > com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:309) > at > com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:477) > at > com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:455) > at > com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:686) > at > com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1091) > at > com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1008) > at > io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) > at > io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) > at > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) > at > io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) > at > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) > at > io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) > at > com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:29) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) > at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1273) at > io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1084) at > io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) > at > io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) > at > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) > at >
[jira] [Updated] (CASSANDRA-14559) Check for endpoint collision with hibernating nodes
[ https://issues.apache.org/jira/browse/CASSANDRA-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Greaves updated CASSANDRA-14559: - Assignee: Vincent White Status: Patch Available (was: Open) > Check for endpoint collision with hibernating nodes > > > Key: CASSANDRA-14559 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14559 > Project: Cassandra > Issue Type: Bug >Reporter: Vincent White >Assignee: Vincent White >Priority: Major > > I ran across an edge case when replacing a node with the same address. This > issue results in the node(and its tokens) being unsafely removed from gossip. > Steps to replicate: > 1. Create 3 node cluster. > 2. Stop a node > 3. Replace the stopped node with a node using the same address using the > replace_address flag > 4. Stop the node before it finishes bootstrapping > 5. Remove the replace_address flag and restart the node to resume > bootstrapping (if the data dir is also cleared at this point the node will > also generate new tokens when it starts) > 6. Stop the node before it finishes bootstrapping again > 7. 30 Seconds later the node will be removed from gossip because it now > matches the check for a FatClient > I think this is only an issue when replacing a node with the same address > because other replacements now use STATUS_BOOTSTRAPPING_REPLACE and leave the > dead node unchanged. > I believe the simplest fix for this is to add a check that prevents a > non-bootstrapped node (without the replaces_address flag) starting if there > is a gossip entry for the same address in the hibernate state. > [3.11 PoC > |https://github.com/apache/cassandra/compare/trunk...vincewhite:check_for_hibernate_on_start] > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-10540) RangeAwareCompaction
[ https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550267#comment-16550267 ] Kurt Greaves commented on CASSANDRA-10540: -- [~krummas] We didn't run into any issues after applying the repair patch from CASSANDRA-13938. We'll probably look at doing a few more tests with repair to measure repair timings with RACS enabled. Let me know if there's anything you need help with to get this in by 4.0. > RangeAwareCompaction > > > Key: CASSANDRA-10540 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10540 > Project: Cassandra > Issue Type: New Feature >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Major > Labels: compaction, lcs, vnodes > Fix For: 4.x > > > Broken out from CASSANDRA-6696, we should split sstables based on ranges > during compaction. > Requirements; > * dont create tiny sstables - keep them bunched together until a single vnode > is big enough (configurable how big that is) > * make it possible to run existing compaction strategies on the per-range > sstables > We should probably add a global compaction strategy parameter that states > whether this should be enabled or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13464) Failed to create Materialized view with a specific token range
[
https://issues.apache.org/jira/browse/CASSANDRA-13464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Greaves updated CASSANDRA-13464:
-
Issue Type: Bug (was: Improvement)
> Failed to create Materialized view with a specific token range
> --
>
> Key: CASSANDRA-13464
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13464
> Project: Cassandra
> Issue Type: Bug
>Reporter: Natsumi Kojima
>Assignee: Krishna Dattu Koneru
>Priority: Minor
> Labels: materializedviews
>
> Failed to create Materialized view with a specific token range.
> Example :
> {code:java}
> $ ccm create "MaterializedView" -v 3.0.13
> $ ccm populate -n 3
> $ ccm start
> $ ccm status
> Cluster: 'MaterializedView'
> ---
> node1: UP
> node3: UP
> node2: UP
> $ccm node1 cqlsh
> Connected to MaterializedView at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 3.0.13 | CQL spec 3.4.0 | Native protocol v4]
> Use HELP for help.
> cqlsh> CREATE KEYSPACE test WITH replication = {'class':'SimpleStrategy',
> 'replication_factor':3};
> cqlsh> CREATE TABLE test.test ( id text PRIMARY KEY , value1 text , value2
> text, value3 text);
> $ccm node1 ring test
> Datacenter: datacenter1
> ==
> AddressRackStatus State LoadOwns
> Token
>
> 3074457345618258602
> 127.0.0.1 rack1 Up Normal 64.86 KB100.00%
> -9223372036854775808
> 127.0.0.2 rack1 Up Normal 86.49 KB100.00%
> -3074457345618258603
> 127.0.0.3 rack1 Up Normal 89.04 KB100.00%
> 3074457345618258602
> $ ccm node1 cqlsh
> cqlsh> INSERT INTO test.test (id, value1 , value2, value3 ) VALUES ('aaa',
> 'aaa', 'aaa' ,'aaa');
> cqlsh> INSERT INTO test.test (id, value1 , value2, value3 ) VALUES ('bbb',
> 'bbb', 'bbb' ,'bbb');
> cqlsh> SELECT token(id),id,value1 FROM test.test;
> system.token(id) | id | value1
> --+-+
> -4737872923231490581 | aaa |aaa
> -3071845237020185195 | bbb |bbb
> (2 rows)
> cqlsh> CREATE MATERIALIZED VIEW test.test_view AS SELECT value1, id FROM
> test.test WHERE id IS NOT NULL AND value1 IS NOT NULL AND TOKEN(id) >
> -9223372036854775808 AND TOKEN(id) < -3074457345618258603 PRIMARY KEY(value1,
> id) WITH CLUSTERING ORDER BY (id ASC);
> ServerError: java.lang.ClassCastException:
> org.apache.cassandra.cql3.TokenRelation cannot be cast to
> org.apache.cassandra.cql3.SingleColumnRelation
> {code}
> Stacktrace :
> {code:java}
> INFO [MigrationStage:1] 2017-04-19 18:32:48,131 ColumnFamilyStore.java:389 -
> Initializing test.test
> WARN [SharedPool-Worker-1] 2017-04-19 18:44:07,263 FBUtilities.java:337 -
> Trigger directory doesn't exist, please create it and try again.
> ERROR [SharedPool-Worker-1] 2017-04-19 18:46:10,072 QueryMessage.java:128 -
> Unexpected error during query
> java.lang.ClassCastException: org.apache.cassandra.cql3.TokenRelation cannot
> be cast to org.apache.cassandra.cql3.SingleColumnRelation
> at
> org.apache.cassandra.db.view.View.relationsToWhereClause(View.java:275)
> ~[apache-cassandra-3.0.13.jar:3.0.13]
> at
> org.apache.cassandra.cql3.statements.CreateViewStatement.announceMigration(CreateViewStatement.java:219)
> ~[apache-cassandra-3.0.13.jar:3.0.13]
> at
> org.apache.cassandra.cql3.statements.SchemaAlteringStatement.execute(SchemaAlteringStatement.java:93)
> ~[apache-cassandra-3.0.13.jar:3.0.13]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206)
> ~[apache-cassandra-3.0.13.jar:3.0.13]
> at
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:237)
> ~[apache-cassandra-3.0.13.jar:3.0.13]
> at
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:222)
> ~[apache-cassandra-3.0.13.jar:3.0.13]
> at
> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115)
> ~[apache-cassandra-3.0.13.jar:3.0.13]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513)
> [apache-cassandra-3.0.13.jar:3.0.13]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:407)
> [apache-cassandra-3.0.13.jar:3.0.13]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
>
[jira] [Commented] (CASSANDRA-14525) streaming failure during bootstrap makes new node into inconsistent state
[
https://issues.apache.org/jira/browse/CASSANDRA-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550237#comment-16550237
]
Kurt Greaves commented on CASSANDRA-14525:
--
Thanks heaps [~djoshi3], I'll look into it.
> streaming failure during bootstrap makes new node into inconsistent state
> -
>
> Key: CASSANDRA-14525
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14525
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Major
> Fix For: 4.0, 2.2.x, 3.0.x
>
>
> If bootstrap fails for newly joining node (most common reason is due to
> streaming failure) then Cassandra state remains in {{joining}} state which is
> fine but Cassandra also enables Native transport which makes overall state
> inconsistent. This further creates NullPointer exception if auth is enabled
> on the new node, please find reproducible steps here:
> For example if bootstrap fails due to streaming errors like
> {quote}java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1256)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:894)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:660)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:573)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:330)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:567)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:695)
> [apache-cassandra-3.0.16.jar:3.0.16]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:187)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:440)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:540)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:307)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
> {quote}
> then variable [StorageService.java::dataAvailable
> |https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageService.java#L892]
> will be {{false}}. Since {{dataAvailable}} is {{false}} hence it will not
> call [StorageService.java::finishJoiningRing
> |https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageService.java#L933]
> and as a result
> [StorageService.java::doAuthSetup|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageService.java#L999]
> will not be invoked.
> API
[jira] [Comment Edited] (CASSANDRA-14525) streaming failure during bootstrap makes new node into inconsistent state
[
https://issues.apache.org/jira/browse/CASSANDRA-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544176#comment-16544176
]
Kurt Greaves edited comment on CASSANDRA-14525 at 7/14/18 11:03 AM:
So I messed up a merge to 3.11 and upon doing further dtests found some issues
and inefficiencies. These have all been fixed up now and branches above should
be all up to date and passing tests so going to mark this as ready to commit.
||2.2||3.0||3.11||trunk||dtests||
|[branch|https://github.com/apache/cassandra/compare/cassandra-2.2...kgreav:14525-2.2]|[branch|https://github.com/apache/cassandra/compare/cassandra-3.0...kgreav:14525-3.0]|[branch|https://github.com/apache/cassandra/compare/cassandra-3.11...kgreav:14525-3.11]|[branch|https://github.com/apache/cassandra/compare/trunk...kgreav:14525-trunk]|[branch|https://github.com/apache/cassandra-dtest/compare/master...kgreav:14526-trunk]|
|[!https://circleci.com/gh/kgreav/cassandra/tree/14525-2.2.svg?style=svg!
|https://circleci.com/gh/kgreav/cassandra/202]|[!https://circleci.com/gh/kgreav/cassandra/tree/14525-3.0.svg?style=svg!
|https://circleci.com/gh/kgreav/cassandra/199]|[!https://circleci.com/gh/kgreav/cassandra/tree/14525-3.11.svg?style=svg!
|https://circleci.com/gh/kgreav/cassandra/201]|[!https://circleci.com/gh/kgreav/cassandra/tree/14525-trunk.svg?style=svg!
|https://circleci.com/gh/kgreav/cassandra/200]|
was (Author: kurtg):
So I messed up a merge to 3.11 and upon doing further dtests found some issues
and inefficiencies. These have all been fixed up now and branches above should
be all up to date and passing tests so going to mark this as ready to commit.
||2.2||3.0||3.11||trunk||dtests||
|[branch|https://github.com/apache/cassandra/compare/cassandra-2.2...kgreav:14525-2.2]|[branch|https://github.com/apache/cassandra/compare/cassandra-3.0...kgreav:14525-3.0]|[branch|https://github.com/apache/cassandra/compare/cassandra-3.11...kgreav:14525-3.11]|[branch|https://github.com/apache/cassandra/compare/trunk...kgreav:14525-trunk]|[branch|https://github.com/apache/cassandra-dtest/compare/master...kgreav:14526-trunk-k]|
|[!https://circleci.com/gh/kgreav/cassandra/tree/14525-2.2.svg?style=svg!
|https://circleci.com/gh/kgreav/cassandra/202]|[!https://circleci.com/gh/kgreav/cassandra/tree/14525-3.0.svg?style=svg!
|https://circleci.com/gh/kgreav/cassandra/199]|[!https://circleci.com/gh/kgreav/cassandra/tree/14525-3.11.svg?style=svg!
|https://circleci.com/gh/kgreav/cassandra/201]|[!https://circleci.com/gh/kgreav/cassandra/tree/14525-trunk.svg?style=svg!
|https://circleci.com/gh/kgreav/cassandra/200]|
> streaming failure during bootstrap makes new node into inconsistent state
> -
>
> Key: CASSANDRA-14525
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14525
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Major
> Fix For: 4.0, 2.2.x, 3.0.x
>
>
> If bootstrap fails for newly joining node (most common reason is due to
> streaming failure) then Cassandra state remains in {{joining}} state which is
> fine but Cassandra also enables Native transport which makes overall state
> inconsistent. This further creates NullPointer exception if auth is enabled
> on the new node, please find reproducible steps here:
> For example if bootstrap fails due to streaming errors like
> {quote}java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1256)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:894)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:660)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:573)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:330)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:567)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:695)
>
[jira] [Updated] (CASSANDRA-14525) streaming failure during bootstrap makes new node into inconsistent state
[
https://issues.apache.org/jira/browse/CASSANDRA-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Greaves updated CASSANDRA-14525:
-
Status: Ready to Commit (was: Patch Available)
> streaming failure during bootstrap makes new node into inconsistent state
> -
>
> Key: CASSANDRA-14525
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14525
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Major
> Fix For: 4.0, 2.2.x, 3.0.x
>
>
> If bootstrap fails for newly joining node (most common reason is due to
> streaming failure) then Cassandra state remains in {{joining}} state which is
> fine but Cassandra also enables Native transport which makes overall state
> inconsistent. This further creates NullPointer exception if auth is enabled
> on the new node, please find reproducible steps here:
> For example if bootstrap fails due to streaming errors like
> {quote}java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1256)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:894)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:660)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:573)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:330)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:567)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:695)
> [apache-cassandra-3.0.16.jar:3.0.16]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:187)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:440)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:540)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:307)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
> {quote}
> then variable [StorageService.java::dataAvailable
> |https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageService.java#L892]
> will be {{false}}. Since {{dataAvailable}} is {{false}} hence it will not
> call [StorageService.java::finishJoiningRing
> |https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageService.java#L933]
> and as a result
> [StorageService.java::doAuthSetup|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageService.java#L999]
> will not be invoked.
> API [StorageService.java::joinTokenRing
>
[jira] [Commented] (CASSANDRA-14525) streaming failure during bootstrap makes new node into inconsistent state
[
https://issues.apache.org/jira/browse/CASSANDRA-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544176#comment-16544176
]
Kurt Greaves commented on CASSANDRA-14525:
--
So I messed up a merge to 3.11 and upon doing further dtests found some issues
and inefficiencies. These have all been fixed up now and branches above should
be all up to date and passing tests so going to mark this as ready to commit.
||2.2||3.0||3.11||trunk||dtests||
|[branch|https://github.com/apache/cassandra/compare/cassandra-2.2...kgreav:14525-2.2]|[branch|https://github.com/apache/cassandra/compare/cassandra-3.0...kgreav:14525-3.0]|[branch|https://github.com/apache/cassandra/compare/cassandra-3.11...kgreav:14525-3.11]|[branch|https://github.com/apache/cassandra/compare/trunk...kgreav:14525-trunk]|[branch|https://github.com/apache/cassandra-dtest/compare/master...kgreav:14526-trunk-k]|
|[!https://circleci.com/gh/kgreav/cassandra/tree/14525-2.2.svg?style=svg!
|https://circleci.com/gh/kgreav/cassandra/202]|[!https://circleci.com/gh/kgreav/cassandra/tree/14525-3.0.svg?style=svg!
|https://circleci.com/gh/kgreav/cassandra/199]|[!https://circleci.com/gh/kgreav/cassandra/tree/14525-3.11.svg?style=svg!
|https://circleci.com/gh/kgreav/cassandra/201]|[!https://circleci.com/gh/kgreav/cassandra/tree/14525-trunk.svg?style=svg!
|https://circleci.com/gh/kgreav/cassandra/200]|
> streaming failure during bootstrap makes new node into inconsistent state
> -
>
> Key: CASSANDRA-14525
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14525
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Major
> Fix For: 4.0, 2.2.x, 3.0.x
>
>
> If bootstrap fails for newly joining node (most common reason is due to
> streaming failure) then Cassandra state remains in {{joining}} state which is
> fine but Cassandra also enables Native transport which makes overall state
> inconsistent. This further creates NullPointer exception if auth is enabled
> on the new node, please find reproducible steps here:
> For example if bootstrap fails due to streaming errors like
> {quote}java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1256)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:894)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:660)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:573)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:330)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:567)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:695)
> [apache-cassandra-3.0.16.jar:3.0.16]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:187)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:440)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
[jira] [Commented] (CASSANDRA-14558) dtest: log-watching thread leak and thread starvation
[
https://issues.apache.org/jira/browse/CASSANDRA-14558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536509#comment-16536509
]
Kurt Greaves commented on CASSANDRA-14558:
--
Looks like it was something in my environment. Working perfectly now. +1 on
patch.
> dtest: log-watching thread leak and thread starvation
> -
>
> Key: CASSANDRA-14558
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14558
> Project: Cassandra
> Issue Type: Bug
> Components: Testing
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
>
> We get occasional build timeouts on b.a.o after pytest becomes unresponsive
> for over 20 minutes. This will result in a thread dump like this one:
> [https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-2.2-dtest/117/consoleFull]
> If you look for "Log-watching thread starting" messages and the dump, it
> becomes quite obvious whats the issue here.
> I had a quick look at the python3 / pytest related changes in CASSANDRA-14134
> and it seems that some of the handling around dtest_setup's
> {{log_watch_thread}} var has been changed in a way that would prevent
> eventually yielding the allocated thread.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14558) dtest: log-watching thread leak and thread starvation
[
https://issues.apache.org/jira/browse/CASSANDRA-14558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535506#comment-16535506
]
Kurt Greaves commented on CASSANDRA-14558:
--
Turns out I was imagining a return statement when I glanced quickly yesterday.
Patch is fine and fixes the described issue, but not sure why I'm still getting
the same errors after applying the patch. I'll do a couple more runs and see if
it's something environment related.
> dtest: log-watching thread leak and thread starvation
> -
>
> Key: CASSANDRA-14558
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14558
> Project: Cassandra
> Issue Type: Bug
> Components: Testing
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
>
> We get occasional build timeouts on b.a.o after pytest becomes unresponsive
> for over 20 minutes. This will result in a thread dump like this one:
> [https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-2.2-dtest/117/consoleFull]
> If you look for "Log-watching thread starting" messages and the dump, it
> becomes quite obvious whats the issue here.
> I had a quick look at the python3 / pytest related changes in CASSANDRA-14134
> and it seems that some of the handling around dtest_setup's
> {{log_watch_thread}} var has been changed in a way that would prevent
> eventually yielding the allocated thread.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14525) streaming failure during bootstrap makes new node into inconsistent state
[
https://issues.apache.org/jira/browse/CASSANDRA-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533547#comment-16533547
]
Kurt Greaves commented on CASSANDRA-14525:
--
Thanks [~chovatia.jayd...@gmail.com]. I figured it was unfair making you do all
the work so I made some changes myself. Basically we were still silently
failing with {{nodetool join}}, so I've added {{isBootstrapMode()}} to
{{StorageServiceMBean}} and {{NodeProbe}} and added a check for it in
{{Join.java}}. I've also tried to simplify some of the if statements (edge
cases are killing me) and added a bit of documentation. I also created a 3.11
branch from our changes as there were a few conflicts from 3.0.
Finally, I extended your dtest from CASSANDRA-14526 to cover the write_survey,
join and resume cases to be in line with the changes we've made, and also
changed the byteman rule to trigger on {{maybeCompleted()}} as it wasn't
consistently triggering on {{startStreamingFiles()}}
Now I think we're pretty much ready to go, but if you want to give my changes a
once over to make sure I haven't missed anything as I'm sure you're aware the
startup code is an absolute nightmare.
Also, I haven't run the whole dtest suite, but I've at least run all the
bootstrap dtests and they are all passing, so I'm taking that as a good sign.
||2.2||3.0||3.11||trunk||dtests||
|[branch|https://github.com/apache/cassandra/compare/cassandra-2.2...kgreav:14525-2.2]|[branch|https://github.com/apache/cassandra/compare/cassandra-3.0...kgreav:14525-3.0]|[branch|https://github.com/apache/cassandra/compare/cassandra-3.11...kgreav:14525-3.11]|[branch|https://github.com/apache/cassandra/compare/trunk...kgreav:14525-trunk]|[branch|https://github.com/apache/cassandra-dtest/compare/master...kgreav:14526-trunk-k]|
|[!https://circleci.com/gh/kgreav/cassandra/tree/14525-2.2.svg?style=svg!
|https://circleci.com/gh/kgreav/cassandra/178]|[!https://circleci.com/gh/kgreav/cassandra/tree/14525-3.0.svg?style=svg!
|https://circleci.com/gh/kgreav/cassandra/181]|[!https://circleci.com/gh/kgreav/cassandra/tree/14525-3.11.svg?style=svg!
|https://circleci.com/gh/kgreav/cassandra/180]|[!https://circleci.com/gh/kgreav/cassandra/tree/14525-trunk.svg?style=svg!
|https://circleci.com/gh/kgreav/cassandra/183]|
> streaming failure during bootstrap makes new node into inconsistent state
> -
>
> Key: CASSANDRA-14525
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14525
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Major
> Fix For: 4.0, 2.2.x, 3.0.x
>
>
> If bootstrap fails for newly joining node (most common reason is due to
> streaming failure) then Cassandra state remains in {{joining}} state which is
> fine but Cassandra also enables Native transport which makes overall state
> inconsistent. This further creates NullPointer exception if auth is enabled
> on the new node, please find reproducible steps here:
> For example if bootstrap fails due to streaming errors like
> {quote}java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1256)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:894)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:660)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:573)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:330)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:567)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:695)
> [apache-cassandra-3.0.16.jar:3.0.16]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
>
[jira] [Assigned] (CASSANDRA-14291) Nodetool command to recreate SSTable components
[
https://issues.apache.org/jira/browse/CASSANDRA-14291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Greaves reassigned CASSANDRA-14291:
Assignee: Alexander Ivakov
> Nodetool command to recreate SSTable components
> ---
>
> Key: CASSANDRA-14291
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14291
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Kurt Greaves
>Assignee: Alexander Ivakov
>Priority: Minor
>
> Need a JMX/Nodetool command to recreate components for SSTables without
> re-writing the data files.
> Possible implementation idea:
> Create a {{nodetool (recreate|regen)component}} command that would enable you
> to recreate specific components of an SSTable, and also allow specifying
> SSTables or columnfamilies.
> I'd say a flag for a list of components and a flag for SSTables with
> keyspace.columnfamilies as positional arguments would work
> Alternatively this could become part of upgradesstables, but would likely
> make that command a bit bloated.
> Background:
> In CASSANDRA-11163 we changed it so summaries and bloomfilters were not
> regenerated or persisted on startup. This means we would rely on
> compactions/upgrades to regenerate the bloomfilter (or other components)
> after a configuration change. While this works, it's pretty inefficient on
> large tables just because you changed the bloomfilter size or summary chunk
> sizes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14443) Improvements for running dtests
[
https://issues.apache.org/jira/browse/CASSANDRA-14443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530863#comment-16530863
]
Kurt Greaves commented on CASSANDRA-14443:
--
Thanks for the review, good catch on the upgrade checks... I was bound to screw
that up because it's hard to test when they are very broken. Messed up the
resource intensive ones a number of times too.. hah.
bq. Is it intentional to use print instead of the loggers in conftest.py?
conftest.pyL492 may also be verbose. Similarly, run_dtests.py was changed to
use loggers instead of print on L73.
So I'm hoping to be proven wrong here, but as far as I'm aware, logger just
simply doesn't work in {{pytest_collection_modifyitems}}. I might be missing
something simple, but no combination of logger settings and debug args I could
find would make those log outputs actually print to stdout. {{print}}, on the
other hand, works flawlessly. [~mkjellman] maybe you can comment? Do you know
some way to get the logger output to actually print to stdout?
As for run_dtests.py, {{logger}} should be fine. I'll double check however.
bq. Logging is useful because it reduces the amount of time to find the issue
and fix it. Failing fast, with an option to override, speeds that up more but
I'm not strongly for it, if you prefer to leave it as is.
Understandable, I'm just dreaming of when we get to a point and you can just
run dtests without having to think about a load of different configurations. In
my experience, extra flags just leads to people ignoring them. Especially as
resource intensive don't run by default, so the check only makes sense if you
already specified resource intensive, which means the user has already
acknowledged they are running something that will chew up all their resources.
On that note, I'll add something more explicit to the README about this.
> Improvements for running dtests
> ---
>
> Key: CASSANDRA-14443
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14443
> Project: Cassandra
> Issue Type: Improvement
> Components: Testing
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Major
> Labels: dtest
>
> We currently hardcode a requirement that you need at least 27gb of memory to
> run the resource intensive tests. This is rather annoying as there isn't
> really a strict hardware requirement and tests can run on smaller machines in
> a lot of cases (especially if you mess around with HEAP).
> We've already got the command line argument
> {{--force-resource-intensive-tests}}, we don't need additional restrictions
> in place to stop people who shouldn't be running the tests from running them.
> We also don't have a way to run _only_ the resource-intensive dtests or
> _only_ the upgrade tests
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14443) Improvements for running dtests
[
https://issues.apache.org/jira/browse/CASSANDRA-14443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16529755#comment-16529755
]
Kurt Greaves commented on CASSANDRA-14443:
--
Sorry for the delay... Finally got a chance to get back to this.
Thanks heaps for the review [~jrwest]
So while addressing your comments I did some further cleaning up and got rid of
the {{--skip-resource-intensive-tests}} arg, as it appears to be completely
pointless. Resource intensive options all default to False, so there's never a
real need to explicitly skip them.
{quote}How about exiting instead of logging a warning when insufficient
resources are available for running resource intensive tests
(conftest.py#L460), unless --force-resource-intensive-tests is set? Behaviour
would be unchanged otherwise and users who want to force run only the resource
intensive tests on insufficient hardware can use
-force-resource-intensive-tests --resource-intensive-tests-only.
{quote}
I personally think this is just annoying. TBH I think that the whole resource
check can go, and this kind of information is more suitable for the
documentation. We don't put resource limits on Cassandra and I don't think we
should do it for the dtests. If people want to run it without enough resources,
by all means let them, they'll quickly find out what's wrong.
To clarify, I don't think we have to be so user friendly that we stop people
doing things that might not work, but I do think we should avoid having to
string together series of massive arguments. The config for running the dtests
at the moment is pretty complicated, and I think we can do away with some of
that.
Having said that, I don't think what I have in this patch is perfect. In fact
some serious refactoring needs to be done around the whole test selection
business, but that's not going to happen for a while and I'm hoping this will
at least make it a bit more usable in the meantime.
{quote} run_dtests.py: some things I noticed that may be worth cleaning up
while we’re here
1. original_raw_cmd_args is only used in one place now. Consider removing it.
2. comment on line 120 is stale{quote}
Fixed
Updated branch is
[here|https://github.com/apache/cassandra-dtest/compare/master...kgreav:14443]
> Improvements for running dtests
> ---
>
> Key: CASSANDRA-14443
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14443
> Project: Cassandra
> Issue Type: Improvement
> Components: Testing
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Major
> Labels: dtest
>
> We currently hardcode a requirement that you need at least 27gb of memory to
> run the resource intensive tests. This is rather annoying as there isn't
> really a strict hardware requirement and tests can run on smaller machines in
> a lot of cases (especially if you mess around with HEAP).
> We've already got the command line argument
> {{--force-resource-intensive-tests}}, we don't need additional restrictions
> in place to stop people who shouldn't be running the tests from running them.
> We also don't have a way to run _only_ the resource-intensive dtests or
> _only_ the upgrade tests
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14443) Improvements for running dtests
[
https://issues.apache.org/jira/browse/CASSANDRA-14443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470217#comment-16470217
]
Kurt Greaves edited comment on CASSANDRA-14443 at 7/2/18 11:49 AM:
---
[patch for
dtests|https://github.com/apache/cassandra-dtest/compare/master...kgreav:14443]
So I initially was only going to make that one small fix, but then I realised
we currently don't have a way to run _only_ the resource intensive tests, and I
ended up implementing a {{--resource-intensive-tests-only}} command... And then
I did the same for upgrade tests.
Seems that our jenkins tests will be doubling up on a lot of tests because the
"large" tests are running everything - not just resource-intensive. Once this
flag gets in we should use it for any "large" dtest runs.
was (Author: kurtg):
[patch for
dtests|https://github.com/apache/cassandra-dtest/compare/master...kgreav:remove_system_resource_check]
So I initially was only going to make that one small fix, but then I realised
we currently don't have a way to run _only_ the resource intensive tests, and I
ended up implementing a {{--resource-intensive-tests-only}} command... And then
I did the same for upgrade tests.
Seems that our jenkins tests will be doubling up on a lot of tests because the
"large" tests are running everything - not just resource-intensive. Once this
flag gets in we should use it for any "large" dtest runs.
> Improvements for running dtests
> ---
>
> Key: CASSANDRA-14443
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14443
> Project: Cassandra
> Issue Type: Improvement
> Components: Testing
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Major
> Labels: dtest
>
> We currently hardcode a requirement that you need at least 27gb of memory to
> run the resource intensive tests. This is rather annoying as there isn't
> really a strict hardware requirement and tests can run on smaller machines in
> a lot of cases (especially if you mess around with HEAP).
> We've already got the command line argument
> {{--force-resource-intensive-tests}}, we don't need additional restrictions
> in place to stop people who shouldn't be running the tests from running them.
> We also don't have a way to run _only_ the resource-intensive dtests or
> _only_ the upgrade tests
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14525) streaming failure during bootstrap makes new node into inconsistent state
[
https://issues.apache.org/jira/browse/CASSANDRA-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16529525#comment-16529525
]
Kurt Greaves commented on CASSANDRA-14525:
--
{quote}Also I've discovered another bug exists in current open source code in
which if isSurveyMode is true and streaming fails (i.e. isBootstrapMode is
true) then also one can call nodetool join without nodetool bootstrap resume
and have that node join the ring.
{quote}
Great catch. I found a couple more small issue w.r.t {{nodetool join}} as well
while I was testing this.
# If in write_survey and you join the ring after bootstrap, transports won't
be enabled. can we call {{CassandraDaemon#start()}} here?
# nodetool join fails silently if write_survey is true and we haven't
completed bootstrapping, but server log prints the following
{code:java}
WARN [RMI TCP Connection(5)-127.0.0.1] 2018-06-29 12:39:49,735
StorageService.java:1008 - Some data streaming failed. Use nodetool to check
bootstrap state and resume. For more, see `nodetool help bootstrap`. IN_PROGRESS
{code}
nodetool join should say something along the lines of "{{Can't join the ring
because in write_survey mode and bootstrap hasn't completed}}"
Also another minor nit w.r.t logging; you can get the following log message
after successfully bootstrapping if you were in write survey mode:
{code:java}
INFO [main] 2018-06-29 12:12:39,071 CassandraDaemon.java:479 - Not starting
client transports as bootstrap has not completed
{code}
Probably better to split CassandraDaemon.start() if block so that we print
"{{Not starting client transports as write_survey mode is enabled.}}"
And finally, there's still 2 occurences of "bootstraped" in the exception
messages in {{startNativeTransport}} and {{startRPCServer}}.
> streaming failure during bootstrap makes new node into inconsistent state
> -
>
> Key: CASSANDRA-14525
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14525
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Major
> Fix For: 4.0, 2.2.x, 3.0.x
>
>
> If bootstrap fails for newly joining node (most common reason is due to
> streaming failure) then Cassandra state remains in {{joining}} state which is
> fine but Cassandra also enables Native transport which makes overall state
> inconsistent. This further creates NullPointer exception if auth is enabled
> on the new node, please find reproducible steps here:
> For example if bootstrap fails due to streaming errors like
> {quote}java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1256)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:894)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:660)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:573)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:330)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:567)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:695)
> [apache-cassandra-3.0.16.jar:3.0.16]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> ~[guava-18.0.jar:na]
> at
>
[jira] [Updated] (CASSANDRA-14423) SSTables stop being compacted
[
https://issues.apache.org/jira/browse/CASSANDRA-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Greaves updated CASSANDRA-14423:
-
Reproduced In: 3.11.2, 3.11.0 (was: 3.11.0, 3.11.2)
Status: Patch Available (was: In Progress)
> SSTables stop being compacted
> -
>
> Key: CASSANDRA-14423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14423
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Blocker
> Fix For: 2.2.13, 3.0.17, 3.11.3
>
>
> So seeing a problem in 3.11.0 where SSTables are being lost from the view and
> not being included in compactions/as candidates for compaction. It seems to
> get progressively worse until there's only 1-2 SSTables in the view which
> happen to be the most recent SSTables and thus compactions completely stop
> for that table.
> The SSTables seem to still be included in reads, just not compactions.
> The issue can be fixed by restarting C*, as it will reload all SSTables into
> the view, but this is only a temporary fix. User defined/major compactions
> still work - not clear if they include the result back in the view but is not
> a good work around.
> This also results in a discrepancy between SSTable count and SSTables in
> levels for any table using LCS.
> {code:java}
> Keyspace : xxx
> Read Count: 57761088
> Read Latency: 0.10527088681224288 ms.
> Write Count: 2513164
> Write Latency: 0.018211106398149903 ms.
> Pending Flushes: 0
> Table: xxx
> SSTable count: 10
> SSTables in each level: [2, 0, 0, 0, 0, 0, 0, 0, 0]
> Space used (live): 894498746
> Space used (total): 894498746
> Space used by snapshots (total): 0
> Off heap memory used (total): 11576197
> SSTable Compression Ratio: 0.6956629530569777
> Number of keys (estimate): 3562207
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 87
> Local read count: 57761088
> Local read latency: 0.108 ms
> Local write count: 2513164
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 86.33
> Bloom filter false positives: 43
> Bloom filter false ratio: 0.0
> Bloom filter space used: 8046104
> Bloom filter off heap memory used: 8046024
> Index summary off heap memory used: 3449005
> Compression metadata off heap memory used: 81168
> Compacted partition minimum bytes: 104
> Compacted partition maximum bytes: 5722
> Compacted partition mean bytes: 175
> Average live cells per slice (last five minutes): 1.0
> Maximum live cells per slice (last five minutes): 1
> Average tombstones per slice (last five minutes): 1.0
> Maximum tombstones per slice (last five minutes): 1
> Dropped Mutations: 0
> {code}
> Also for STCS we've confirmed that SSTable count will be different to the
> number of SSTables reported in the Compaction Bucket's. In the below example
> there's only 3 SSTables in a single bucket - no more are listed for this
> table. Compaction thresholds haven't been modified for this table and it's a
> very basic KV schema.
> {code:java}
> Keyspace : yyy
> Read Count: 30485
> Read Latency: 0.06708991307200263 ms.
> Write Count: 57044
> Write Latency: 0.02204061776873992 ms.
> Pending Flushes: 0
> Table: yyy
> SSTable count: 19
> Space used (live): 18195482
> Space used (total): 18195482
> Space used by snapshots (total): 0
> Off heap memory used (total): 747376
> SSTable Compression Ratio: 0.7607394576769735
> Number of keys (estimate): 116074
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 39
> Local read count: 30485
> Local read latency: NaN ms
> Local write count: 57044
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 79.76
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 690912
> Bloom filter off heap memory used: 690760
> Index summary off heap memory used: 54736
> Compression metadata off heap memory used: 1880
> Compacted partition minimum bytes: 73
> Compacted partition maximum bytes: 124
> Compacted partition mean bytes: 96
> Average live cells per slice (last five minutes): NaN
> Maximum live cells per slice (last five minutes): 0
> Average tombstones per slice (last five minutes): NaN
> Maximum tombstones per slice (last five minutes): 0
> Dropped Mutations: 0
> {code}
> {code:java}
> Apr 27 03:10:39 cassandra[9263]: TRACE o.a.c.d.c.SizeTieredCompactionStrategy
> Compaction buckets are
>
[jira] [Commented] (CASSANDRA-14423) SSTables stop being compacted
[
https://issues.apache.org/jira/browse/CASSANDRA-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526987#comment-16526987
]
Kurt Greaves commented on CASSANDRA-14423:
--
Created CASSANDRA-14550 for the trunk port.
> SSTables stop being compacted
> -
>
> Key: CASSANDRA-14423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14423
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Blocker
> Fix For: 2.2.13, 3.0.17, 3.11.3
>
>
> So seeing a problem in 3.11.0 where SSTables are being lost from the view and
> not being included in compactions/as candidates for compaction. It seems to
> get progressively worse until there's only 1-2 SSTables in the view which
> happen to be the most recent SSTables and thus compactions completely stop
> for that table.
> The SSTables seem to still be included in reads, just not compactions.
> The issue can be fixed by restarting C*, as it will reload all SSTables into
> the view, but this is only a temporary fix. User defined/major compactions
> still work - not clear if they include the result back in the view but is not
> a good work around.
> This also results in a discrepancy between SSTable count and SSTables in
> levels for any table using LCS.
> {code:java}
> Keyspace : xxx
> Read Count: 57761088
> Read Latency: 0.10527088681224288 ms.
> Write Count: 2513164
> Write Latency: 0.018211106398149903 ms.
> Pending Flushes: 0
> Table: xxx
> SSTable count: 10
> SSTables in each level: [2, 0, 0, 0, 0, 0, 0, 0, 0]
> Space used (live): 894498746
> Space used (total): 894498746
> Space used by snapshots (total): 0
> Off heap memory used (total): 11576197
> SSTable Compression Ratio: 0.6956629530569777
> Number of keys (estimate): 3562207
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 87
> Local read count: 57761088
> Local read latency: 0.108 ms
> Local write count: 2513164
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 86.33
> Bloom filter false positives: 43
> Bloom filter false ratio: 0.0
> Bloom filter space used: 8046104
> Bloom filter off heap memory used: 8046024
> Index summary off heap memory used: 3449005
> Compression metadata off heap memory used: 81168
> Compacted partition minimum bytes: 104
> Compacted partition maximum bytes: 5722
> Compacted partition mean bytes: 175
> Average live cells per slice (last five minutes): 1.0
> Maximum live cells per slice (last five minutes): 1
> Average tombstones per slice (last five minutes): 1.0
> Maximum tombstones per slice (last five minutes): 1
> Dropped Mutations: 0
> {code}
> Also for STCS we've confirmed that SSTable count will be different to the
> number of SSTables reported in the Compaction Bucket's. In the below example
> there's only 3 SSTables in a single bucket - no more are listed for this
> table. Compaction thresholds haven't been modified for this table and it's a
> very basic KV schema.
> {code:java}
> Keyspace : yyy
> Read Count: 30485
> Read Latency: 0.06708991307200263 ms.
> Write Count: 57044
> Write Latency: 0.02204061776873992 ms.
> Pending Flushes: 0
> Table: yyy
> SSTable count: 19
> Space used (live): 18195482
> Space used (total): 18195482
> Space used by snapshots (total): 0
> Off heap memory used (total): 747376
> SSTable Compression Ratio: 0.7607394576769735
> Number of keys (estimate): 116074
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 39
> Local read count: 30485
> Local read latency: NaN ms
> Local write count: 57044
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 79.76
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 690912
> Bloom filter off heap memory used: 690760
> Index summary off heap memory used: 54736
> Compression metadata off heap memory used: 1880
> Compacted partition minimum bytes: 73
> Compacted partition maximum bytes: 124
> Compacted partition mean bytes: 96
> Average live cells per slice (last five minutes): NaN
> Maximum live cells per slice (last five minutes): 0
> Average tombstones per slice (last five minutes): NaN
> Maximum tombstones per slice (last five minutes): 0
> Dropped Mutations: 0
> {code}
> {code:java}
> Apr 27 03:10:39 cassandra[9263]: TRACE o.a.c.d.c.SizeTieredCompactionStrategy
> Compaction buckets are
>
[jira] [Created] (CASSANDRA-14550) Port changes for CASSANDRA-14423 to trunk
Kurt Greaves created CASSANDRA-14550: Summary: Port changes for CASSANDRA-14423 to trunk Key: CASSANDRA-14550 URL: https://issues.apache.org/jira/browse/CASSANDRA-14550 Project: Cassandra Issue Type: Bug Reporter: Kurt Greaves Assignee: Kurt Greaves In CASSANDRA-14423 we fixed the behaviour of CASSANDRA-13153 so that full repairs don't remove SSTables from the compaction strategy. Trunk was unaffected by the bug, however it should still have at the very least the test from CASSANDRA-14423 ported to trunk and any necessary improvements. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14423) SSTables stop being compacted
[
https://issues.apache.org/jira/browse/CASSANDRA-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526981#comment-16526981
]
Kurt Greaves commented on CASSANDRA-14423:
--
[~spo...@gmail.com], not at all. Probably won't have time to look at trunk
until next week, and I'm fairly sure it'll just be a case of making the test
work so I'll create another ticket for trunk.
> SSTables stop being compacted
> -
>
> Key: CASSANDRA-14423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14423
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Blocker
> Fix For: 2.2.13, 3.0.17, 3.11.3
>
>
> So seeing a problem in 3.11.0 where SSTables are being lost from the view and
> not being included in compactions/as candidates for compaction. It seems to
> get progressively worse until there's only 1-2 SSTables in the view which
> happen to be the most recent SSTables and thus compactions completely stop
> for that table.
> The SSTables seem to still be included in reads, just not compactions.
> The issue can be fixed by restarting C*, as it will reload all SSTables into
> the view, but this is only a temporary fix. User defined/major compactions
> still work - not clear if they include the result back in the view but is not
> a good work around.
> This also results in a discrepancy between SSTable count and SSTables in
> levels for any table using LCS.
> {code:java}
> Keyspace : xxx
> Read Count: 57761088
> Read Latency: 0.10527088681224288 ms.
> Write Count: 2513164
> Write Latency: 0.018211106398149903 ms.
> Pending Flushes: 0
> Table: xxx
> SSTable count: 10
> SSTables in each level: [2, 0, 0, 0, 0, 0, 0, 0, 0]
> Space used (live): 894498746
> Space used (total): 894498746
> Space used by snapshots (total): 0
> Off heap memory used (total): 11576197
> SSTable Compression Ratio: 0.6956629530569777
> Number of keys (estimate): 3562207
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 87
> Local read count: 57761088
> Local read latency: 0.108 ms
> Local write count: 2513164
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 86.33
> Bloom filter false positives: 43
> Bloom filter false ratio: 0.0
> Bloom filter space used: 8046104
> Bloom filter off heap memory used: 8046024
> Index summary off heap memory used: 3449005
> Compression metadata off heap memory used: 81168
> Compacted partition minimum bytes: 104
> Compacted partition maximum bytes: 5722
> Compacted partition mean bytes: 175
> Average live cells per slice (last five minutes): 1.0
> Maximum live cells per slice (last five minutes): 1
> Average tombstones per slice (last five minutes): 1.0
> Maximum tombstones per slice (last five minutes): 1
> Dropped Mutations: 0
> {code}
> Also for STCS we've confirmed that SSTable count will be different to the
> number of SSTables reported in the Compaction Bucket's. In the below example
> there's only 3 SSTables in a single bucket - no more are listed for this
> table. Compaction thresholds haven't been modified for this table and it's a
> very basic KV schema.
> {code:java}
> Keyspace : yyy
> Read Count: 30485
> Read Latency: 0.06708991307200263 ms.
> Write Count: 57044
> Write Latency: 0.02204061776873992 ms.
> Pending Flushes: 0
> Table: yyy
> SSTable count: 19
> Space used (live): 18195482
> Space used (total): 18195482
> Space used by snapshots (total): 0
> Off heap memory used (total): 747376
> SSTable Compression Ratio: 0.7607394576769735
> Number of keys (estimate): 116074
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 39
> Local read count: 30485
> Local read latency: NaN ms
> Local write count: 57044
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 79.76
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 690912
> Bloom filter off heap memory used: 690760
> Index summary off heap memory used: 54736
> Compression metadata off heap memory used: 1880
> Compacted partition minimum bytes: 73
> Compacted partition maximum bytes: 124
> Compacted partition mean bytes: 96
> Average live cells per slice (last five minutes): NaN
> Maximum live cells per slice (last five minutes): 0
> Average tombstones per slice (last five minutes): NaN
> Maximum tombstones per slice (last five minutes): 0
> Dropped Mutations: 0
> {code}
> {code:java}
> Apr 27
[jira] [Comment Edited] (CASSANDRA-14423) SSTables stop being compacted
[
https://issues.apache.org/jira/browse/CASSANDRA-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16524885#comment-16524885
]
Kurt Greaves edited comment on CASSANDRA-14423 at 6/27/18 10:51 AM:
[~krummas] I've suggested that we at the very least provide a flag to skip
anti-compaction from full repairs before, but it was all deemed too
[complicated|https://issues.apache.org/jira/browse/CASSANDRA-13885?focusedCommentId=16206922=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16206922].
Regardless, I think a flag would be perfectly fine and it's still desirable so
at least people can go back to at the very least running full repairs
successfully without having to worry about SSTables being marked repaired.
However, I don't think we can go and change the default behaviour, purely
because people _could_ still be running full repairs on earlier versions of
3.x/3.0 before this bug came along.
was (Author: kurtg):
[~krummas] I've suggested that we at the very least provide a flag to skip
anti-compaction from full repairs before, but it was all deemed too
complicated. Regardless, I think a flag would be perfectly fine and it's still
desirable so at least people can go back to at the very least running full
repairs successfully without having to worry about SSTables being marked
repaired. However, I don't think we can go and change the default behaviour,
purely because people _could_ still be running full repairs on earlier versions
of 3.x/3.0 before this bug came along.
> SSTables stop being compacted
> -
>
> Key: CASSANDRA-14423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14423
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Blocker
> Fix For: 2.2.13, 3.0.17, 3.11.3
>
>
> So seeing a problem in 3.11.0 where SSTables are being lost from the view and
> not being included in compactions/as candidates for compaction. It seems to
> get progressively worse until there's only 1-2 SSTables in the view which
> happen to be the most recent SSTables and thus compactions completely stop
> for that table.
> The SSTables seem to still be included in reads, just not compactions.
> The issue can be fixed by restarting C*, as it will reload all SSTables into
> the view, but this is only a temporary fix. User defined/major compactions
> still work - not clear if they include the result back in the view but is not
> a good work around.
> This also results in a discrepancy between SSTable count and SSTables in
> levels for any table using LCS.
> {code:java}
> Keyspace : xxx
> Read Count: 57761088
> Read Latency: 0.10527088681224288 ms.
> Write Count: 2513164
> Write Latency: 0.018211106398149903 ms.
> Pending Flushes: 0
> Table: xxx
> SSTable count: 10
> SSTables in each level: [2, 0, 0, 0, 0, 0, 0, 0, 0]
> Space used (live): 894498746
> Space used (total): 894498746
> Space used by snapshots (total): 0
> Off heap memory used (total): 11576197
> SSTable Compression Ratio: 0.6956629530569777
> Number of keys (estimate): 3562207
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 87
> Local read count: 57761088
> Local read latency: 0.108 ms
> Local write count: 2513164
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 86.33
> Bloom filter false positives: 43
> Bloom filter false ratio: 0.0
> Bloom filter space used: 8046104
> Bloom filter off heap memory used: 8046024
> Index summary off heap memory used: 3449005
> Compression metadata off heap memory used: 81168
> Compacted partition minimum bytes: 104
> Compacted partition maximum bytes: 5722
> Compacted partition mean bytes: 175
> Average live cells per slice (last five minutes): 1.0
> Maximum live cells per slice (last five minutes): 1
> Average tombstones per slice (last five minutes): 1.0
> Maximum tombstones per slice (last five minutes): 1
> Dropped Mutations: 0
> {code}
> Also for STCS we've confirmed that SSTable count will be different to the
> number of SSTables reported in the Compaction Bucket's. In the below example
> there's only 3 SSTables in a single bucket - no more are listed for this
> table. Compaction thresholds haven't been modified for this table and it's a
> very basic KV schema.
> {code:java}
> Keyspace : yyy
> Read Count: 30485
> Read Latency: 0.06708991307200263 ms.
> Write Count: 57044
> Write Latency: 0.02204061776873992 ms.
> Pending Flushes: 0
> Table: yyy
> SSTable count: 19
> Space used (live): 18195482
> Space used (total): 18195482
> Space used by snapshots (total): 0
> Off heap memory used (total): 747376
>
[jira] [Comment Edited] (CASSANDRA-14543) Hinted handoff to replay purgeable tombstones
[
https://issues.apache.org/jira/browse/CASSANDRA-14543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523302#comment-16523302
]
Kurt Greaves edited comment on CASSANDRA-14543 at 6/27/18 10:48 AM:
{quote}But hints-handoff will only happen once, and we know the target node is
missing that deletion. It may not need it if the tombstone is repaired from
another node within GCGS, otherwise, it's the best (only) way to delete the
data on that node.
{quote}
-Well, it could create more unnecessary read-repair in the following scenario
(which is the only case where HH of purgeable tombstones comes into play):-
-3 nodes, A, B, C.-
# -Insert as per your example-
# -Node B goes down-
# -Delete partition-
# -GCGS passes-
# -A and C compact away partition deletion-
# -B comes back up-
# -A/C HH tombstone to B-
-Any further reads for that partition will now cause a RR where the tombstone
is not propagated-
^ clearly NFI what I'm talking about.
But really, we'd be only addressing the case where a deletion is performed
within the HH window and then the node stays down until GCGS passes. This seems
like a really narrow use case here, especially because if a node is down for
GCGS you're going to have problems anyway (unless there's something I'm missing
here).
was (Author: kurtg):
{quote}But hints-handoff will only happen once, and we know the target node is
missing that deletion. It may not need it if the tombstone is repaired from
another node within GCGS, otherwise, it's the best (only) way to delete the
data on that node.
{quote}
Well, it could create more unnecessary read-repair in the following scenario
(which is the only case where HH of purgeable tombstones comes into play):
3 nodes, A, B, C.
# Insert as per your example
# Node B goes down
# Delete partition
# GCGS passes
# A and C compact away partition deletion
# B comes back up
# A/C HH tombstone to B
Any further reads for that partition will now cause a RR where the tombstone is
not propagated
But really, we'd be only addressing the case where a deletion is performed
within the HH window and then the node stays down until GCGS passes. This seems
like a really narrow use case here, especially because if a node is down for
GCGS you're going to have problems anyway (unless there's something I'm missing
here).
> Hinted handoff to replay purgeable tombstones
> --
>
> Key: CASSANDRA-14543
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14543
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Jay Zhuang
>Priority: Minor
>
> Hinted-handoff currently only dispatches and applies the mutations that are
> within GCGS:
> [{{Hint.java:97}}|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/hints/Hint.java#L97].
> Which is to make sure it won't resurrect any deleted data.
> But replaying tombstones should be safe, it could reduce the chance to have
> [un-repairable inconsistent
> data|https://lists.apache.org/thread.html/2d3d39d960143d4d2146ed2530821504ff855e832713dec7d0afd8ac@%3Cdev.cassandra.apache.org%3E].
> Here is the user scenario it tries to fix:
> {noformat}
> 1. Create a 3 nodes cluster
> 2. Create a table with small gc_grace_seconds (for reproducing purpose):
> CREATE KEYSPACE foo WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': 3};
> CREATE TABLE foo.bar (
> id int PRIMARY KEY,
> name text
> ) WITH gc_grace_seconds=30;
> 3. Insert data with consistency all:
> INSERT INTO foo.bar (id, name) VALUES(1, 'cstar');
> 4. stop 1 node
> $ ccm node2 stop
> 5. Delete the data with consistency quorum:
> DELETE FROM foo.bar WHERE id=1;
> 6. Wait 30 seconds and then start node2:
> $ ccm node2 start
> {noformat}
> Now, node2 has the data, node1/node3 have the purgeable tombstone. It
> triggers RR every time which sends data from node2 to node1/node3 but repairs
> nothing.
> With purgeable tombstones hints handoff, it at least will dispatch the
> tombstone and delete the data on node2. It won't fix the root cause but
> reduce the chance to have this issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14543) Hinted handoff to replay purgeable tombstones
[
https://issues.apache.org/jira/browse/CASSANDRA-14543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16524896#comment-16524896
]
Kurt Greaves commented on CASSANDRA-14543:
--
[~jay.zhuang], right, sorry my misunderstanding. But unless I'm mistaken this
still only helps the case where a node is down for the entirety of GCGS
correct? Because that's the only way you'll ever hint a purgeable tombstone. In
which case you'll be in for a real bad time anyway and hinting tombstones from
a small portion of the downtime isn't going to give any real benefit?
> Hinted handoff to replay purgeable tombstones
> --
>
> Key: CASSANDRA-14543
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14543
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Jay Zhuang
>Priority: Minor
>
> Hinted-handoff currently only dispatches and applies the mutations that are
> within GCGS:
> [{{Hint.java:97}}|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/hints/Hint.java#L97].
> Which is to make sure it won't resurrect any deleted data.
> But replaying tombstones should be safe, it could reduce the chance to have
> [un-repairable inconsistent
> data|https://lists.apache.org/thread.html/2d3d39d960143d4d2146ed2530821504ff855e832713dec7d0afd8ac@%3Cdev.cassandra.apache.org%3E].
> Here is the user scenario it tries to fix:
> {noformat}
> 1. Create a 3 nodes cluster
> 2. Create a table with small gc_grace_seconds (for reproducing purpose):
> CREATE KEYSPACE foo WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': 3};
> CREATE TABLE foo.bar (
> id int PRIMARY KEY,
> name text
> ) WITH gc_grace_seconds=30;
> 3. Insert data with consistency all:
> INSERT INTO foo.bar (id, name) VALUES(1, 'cstar');
> 4. stop 1 node
> $ ccm node2 stop
> 5. Delete the data with consistency quorum:
> DELETE FROM foo.bar WHERE id=1;
> 6. Wait 30 seconds and then start node2:
> $ ccm node2 start
> {noformat}
> Now, node2 has the data, node1/node3 have the purgeable tombstone. It
> triggers RR every time which sends data from node2 to node1/node3 but repairs
> nothing.
> With purgeable tombstones hints handoff, it at least will dispatch the
> tombstone and delete the data on node2. It won't fix the root cause but
> reduce the chance to have this issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14423) SSTables stop being compacted
[
https://issues.apache.org/jira/browse/CASSANDRA-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16524885#comment-16524885
]
Kurt Greaves commented on CASSANDRA-14423:
--
[~krummas] I've suggested that we at the very least provide a flag to skip
anti-compaction from full repairs before, but it was all deemed too
complicated. Regardless, I think a flag would be perfectly fine and it's still
desirable so at least people can go back to at the very least running full
repairs successfully without having to worry about SSTables being marked
repaired. However, I don't think we can go and change the default behaviour,
purely because people _could_ still be running full repairs on earlier versions
of 3.x/3.0 before this bug came along.
> SSTables stop being compacted
> -
>
> Key: CASSANDRA-14423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14423
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Blocker
> Fix For: 2.2.13, 3.0.17, 3.11.3
>
>
> So seeing a problem in 3.11.0 where SSTables are being lost from the view and
> not being included in compactions/as candidates for compaction. It seems to
> get progressively worse until there's only 1-2 SSTables in the view which
> happen to be the most recent SSTables and thus compactions completely stop
> for that table.
> The SSTables seem to still be included in reads, just not compactions.
> The issue can be fixed by restarting C*, as it will reload all SSTables into
> the view, but this is only a temporary fix. User defined/major compactions
> still work - not clear if they include the result back in the view but is not
> a good work around.
> This also results in a discrepancy between SSTable count and SSTables in
> levels for any table using LCS.
> {code:java}
> Keyspace : xxx
> Read Count: 57761088
> Read Latency: 0.10527088681224288 ms.
> Write Count: 2513164
> Write Latency: 0.018211106398149903 ms.
> Pending Flushes: 0
> Table: xxx
> SSTable count: 10
> SSTables in each level: [2, 0, 0, 0, 0, 0, 0, 0, 0]
> Space used (live): 894498746
> Space used (total): 894498746
> Space used by snapshots (total): 0
> Off heap memory used (total): 11576197
> SSTable Compression Ratio: 0.6956629530569777
> Number of keys (estimate): 3562207
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 87
> Local read count: 57761088
> Local read latency: 0.108 ms
> Local write count: 2513164
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 86.33
> Bloom filter false positives: 43
> Bloom filter false ratio: 0.0
> Bloom filter space used: 8046104
> Bloom filter off heap memory used: 8046024
> Index summary off heap memory used: 3449005
> Compression metadata off heap memory used: 81168
> Compacted partition minimum bytes: 104
> Compacted partition maximum bytes: 5722
> Compacted partition mean bytes: 175
> Average live cells per slice (last five minutes): 1.0
> Maximum live cells per slice (last five minutes): 1
> Average tombstones per slice (last five minutes): 1.0
> Maximum tombstones per slice (last five minutes): 1
> Dropped Mutations: 0
> {code}
> Also for STCS we've confirmed that SSTable count will be different to the
> number of SSTables reported in the Compaction Bucket's. In the below example
> there's only 3 SSTables in a single bucket - no more are listed for this
> table. Compaction thresholds haven't been modified for this table and it's a
> very basic KV schema.
> {code:java}
> Keyspace : yyy
> Read Count: 30485
> Read Latency: 0.06708991307200263 ms.
> Write Count: 57044
> Write Latency: 0.02204061776873992 ms.
> Pending Flushes: 0
> Table: yyy
> SSTable count: 19
> Space used (live): 18195482
> Space used (total): 18195482
> Space used by snapshots (total): 0
> Off heap memory used (total): 747376
> SSTable Compression Ratio: 0.7607394576769735
> Number of keys (estimate): 116074
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 39
> Local read count: 30485
> Local read latency: NaN ms
> Local write count: 57044
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 79.76
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 690912
> Bloom filter off heap memory used: 690760
> Index summary off heap memory used: 54736
> Compression metadata off heap memory used: 1880
> Compacted partition minimum bytes: 73
> Compacted partition maximum
[jira] [Commented] (CASSANDRA-14525) streaming failure during bootstrap makes new node into inconsistent state
[
https://issues.apache.org/jira/browse/CASSANDRA-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16524862#comment-16524862
]
Kurt Greaves commented on CASSANDRA-14525:
--
Mostly looks good, however we're still leaving write_survey mode after a resume
bootstrap completes when we were started in write survey mode. Also just
noticed but as we hackily re-use isSurveyMode when resuming a bootstrap we
always log the following message regardless of if we were in write survey mode
originally or not.
{code}Leaving write survey mode and joining ring at operator request{code}
I think at this point we could solve these 2 problems by simply calling
{{finishJoiningRing}} explicitly when we successfully bootstrap after a resume
in {{resumeBootstrap}}, rather than indirectly through {{joinRing}, and also
handle write_survey in the same place.
Also, another small nit, can we change the spelling of {{bootstraped}} to
{{bootstrapped}} in the exception messages?
> streaming failure during bootstrap makes new node into inconsistent state
> -
>
> Key: CASSANDRA-14525
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14525
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Major
> Fix For: 4.0, 2.2.x, 3.0.x
>
>
> If bootstrap fails for newly joining node (most common reason is due to
> streaming failure) then Cassandra state remains in {{joining}} state which is
> fine but Cassandra also enables Native transport which makes overall state
> inconsistent. This further creates NullPointer exception if auth is enabled
> on the new node, please find reproducible steps here:
> For example if bootstrap fails due to streaming errors like
> {quote}java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1256)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:894)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:660)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:573)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:330)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:567)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:695)
> [apache-cassandra-3.0.16.jar:3.0.16]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:187)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:440)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:540)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:307)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
>
[jira] [Commented] (CASSANDRA-14423) SSTables stop being compacted
[
https://issues.apache.org/jira/browse/CASSANDRA-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16524292#comment-16524292
]
Kurt Greaves commented on CASSANDRA-14423:
--
Sorry for the delay, was looking at this yesterday but yeah, it turned out the
assert failing was just me imagining things. That all seems good to me
[~spo...@gmail.com].
> SSTables stop being compacted
> -
>
> Key: CASSANDRA-14423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14423
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Blocker
> Fix For: 2.2.13, 3.0.17, 3.11.3
>
>
> So seeing a problem in 3.11.0 where SSTables are being lost from the view and
> not being included in compactions/as candidates for compaction. It seems to
> get progressively worse until there's only 1-2 SSTables in the view which
> happen to be the most recent SSTables and thus compactions completely stop
> for that table.
> The SSTables seem to still be included in reads, just not compactions.
> The issue can be fixed by restarting C*, as it will reload all SSTables into
> the view, but this is only a temporary fix. User defined/major compactions
> still work - not clear if they include the result back in the view but is not
> a good work around.
> This also results in a discrepancy between SSTable count and SSTables in
> levels for any table using LCS.
> {code:java}
> Keyspace : xxx
> Read Count: 57761088
> Read Latency: 0.10527088681224288 ms.
> Write Count: 2513164
> Write Latency: 0.018211106398149903 ms.
> Pending Flushes: 0
> Table: xxx
> SSTable count: 10
> SSTables in each level: [2, 0, 0, 0, 0, 0, 0, 0, 0]
> Space used (live): 894498746
> Space used (total): 894498746
> Space used by snapshots (total): 0
> Off heap memory used (total): 11576197
> SSTable Compression Ratio: 0.6956629530569777
> Number of keys (estimate): 3562207
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 87
> Local read count: 57761088
> Local read latency: 0.108 ms
> Local write count: 2513164
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 86.33
> Bloom filter false positives: 43
> Bloom filter false ratio: 0.0
> Bloom filter space used: 8046104
> Bloom filter off heap memory used: 8046024
> Index summary off heap memory used: 3449005
> Compression metadata off heap memory used: 81168
> Compacted partition minimum bytes: 104
> Compacted partition maximum bytes: 5722
> Compacted partition mean bytes: 175
> Average live cells per slice (last five minutes): 1.0
> Maximum live cells per slice (last five minutes): 1
> Average tombstones per slice (last five minutes): 1.0
> Maximum tombstones per slice (last five minutes): 1
> Dropped Mutations: 0
> {code}
> Also for STCS we've confirmed that SSTable count will be different to the
> number of SSTables reported in the Compaction Bucket's. In the below example
> there's only 3 SSTables in a single bucket - no more are listed for this
> table. Compaction thresholds haven't been modified for this table and it's a
> very basic KV schema.
> {code:java}
> Keyspace : yyy
> Read Count: 30485
> Read Latency: 0.06708991307200263 ms.
> Write Count: 57044
> Write Latency: 0.02204061776873992 ms.
> Pending Flushes: 0
> Table: yyy
> SSTable count: 19
> Space used (live): 18195482
> Space used (total): 18195482
> Space used by snapshots (total): 0
> Off heap memory used (total): 747376
> SSTable Compression Ratio: 0.7607394576769735
> Number of keys (estimate): 116074
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 39
> Local read count: 30485
> Local read latency: NaN ms
> Local write count: 57044
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 79.76
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 690912
> Bloom filter off heap memory used: 690760
> Index summary off heap memory used: 54736
> Compression metadata off heap memory used: 1880
> Compacted partition minimum bytes: 73
> Compacted partition maximum bytes: 124
> Compacted partition mean bytes: 96
> Average live cells per slice (last five minutes): NaN
> Maximum live cells per slice (last five minutes): 0
> Average tombstones per slice (last five minutes): NaN
> Maximum tombstones per slice (last five minutes): 0
> Dropped Mutations: 0
> {code}
> {code:java}
> Apr 27 03:10:39 cassandra[9263]:
[jira] [Commented] (CASSANDRA-14543) Hinted handoff to replay purgeable tombstones
[
https://issues.apache.org/jira/browse/CASSANDRA-14543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523302#comment-16523302
]
Kurt Greaves commented on CASSANDRA-14543:
--
{quote}But hints-handoff will only happen once, and we know the target node is
missing that deletion. It may not need it if the tombstone is repaired from
another node within GCGS, otherwise, it's the best (only) way to delete the
data on that node.
{quote}
Well, it could create more unnecessary read-repair in the following scenario
(which is the only case where HH of purgeable tombstones comes into play):
3 nodes, A, B, C.
# Insert as per your example
# Node B goes down
# Delete partition
# GCGS passes
# A and C compact away partition deletion
# B comes back up
# A/C HH tombstone to B
Any further reads for that partition will now cause a RR where the tombstone is
not propagated
But really, we'd be only addressing the case where a deletion is performed
within the HH window and then the node stays down until GCGS passes. This seems
like a really narrow use case here, especially because if a node is down for
GCGS you're going to have problems anyway (unless there's something I'm missing
here).
> Hinted handoff to replay purgeable tombstones
> --
>
> Key: CASSANDRA-14543
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14543
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Jay Zhuang
>Priority: Minor
>
> Hinted-handoff currently only dispatches and applies the mutations that are
> within GCGS:
> [{{Hint.java:97}}|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/hints/Hint.java#L97].
> Which is to make sure it won't resurrect any deleted data.
> But replaying tombstones should be safe, it could reduce the chance to have
> [un-repairable inconsistent
> data|https://lists.apache.org/thread.html/2d3d39d960143d4d2146ed2530821504ff855e832713dec7d0afd8ac@%3Cdev.cassandra.apache.org%3E].
> Here is the user scenario it tries to fix:
> {noformat}
> 1. Create a 3 nodes cluster
> 2. Create a table with small gc_grace_seconds (for reproducing purpose):
> CREATE KEYSPACE foo WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': 3};
> CREATE TABLE foo.bar (
> id int PRIMARY KEY,
> name text
> ) WITH gc_grace_seconds=30;
> 3. Insert data with consistency all:
> INSERT INTO foo.bar (id, name) VALUES(1, 'cstar');
> 4. stop 1 node
> $ ccm node2 stop
> 5. Delete the data with consistency quorum:
> DELETE FROM foo.bar WHERE id=1;
> 6. Wait 30 seconds and then start node2:
> $ ccm node2 start
> {noformat}
> Now, node2 has the data, node1/node3 have the purgeable tombstone. It
> triggers RR every time which sends data from node2 to node1/node3 but repairs
> nothing.
> With purgeable tombstones hints handoff, it at least will dispatch the
> tombstone and delete the data on node2. It won't fix the root cause but
> reduce the chance to have this issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14525) streaming failure during bootstrap makes new node into inconsistent state
[
https://issues.apache.org/jira/browse/CASSANDRA-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523222#comment-16523222
]
Kurt Greaves commented on CASSANDRA-14525:
--
A few things:
If we fail streaming and {{isSurveyMode}} is true we still get the NPE if auth
is enabled when trying to connect to C* on that node. Not much we can do about
this because auth isn't initialised until we join the ring, but I'm not sure
why we should handle this situation differently, and also it's currently kind
of broken. At the moment if you resume bootstrap after a streaming failure
_while in write survey mode_, you will leave write survey mode on completion of
bootstrapping (ouch).
I think we should handle write survey bootstrapping the same as normal
bootstrap, where if we get an error during streaming we don't start transports.
Then, on resume, handle survey mode so that we _don't_ join the ring on
completion of bootstrapping, but we do still start transports.
On top of that, seeing as we're in this code anyway, I think it would be
reasonable if we could look at handling the auth case a bit better when write
survey is enabled as well. Ideally, if auth is required I see no point in
starting the transports seeing as you'll always get an NPE, so maybe we can add
a check for that in {{CassandraDaemon#start()?}}
{{DatabaseDescriptor.getAuthenticator().requireAuthentication()}} should be
enough here I think.
Some things regarding the error message:
We've got repeated information in our error:
{code:java}
WARN [main] 2018-06-25 09:13:24,136 StorageService.java:935 - Some data
streaming failed. Use nodetool to check bootstrap state and resume. For more,
see `nodetool help bootstrap`. IN_PROGRESS
ERROR [main] 2018-06-25 09:13:32,190 CassandraDaemon.java:445 - Node is not yet
bootstraped hence not enabling native transport. Use nodetool to check
bootstrap state and resume. For more, see `nodetool helpbootstrap`
{code}
I think our new message should either be INFO or WARN (INFO is in line with
other messages in {{start()}}, and I think it would make more sense if the
original message in \{{StorageService}} was ERROR. We could change the message
in CassandraDaemon to:
{code:java}
Not starting client transports as bootstrap has not completed.{code}
or something similar, to be more in line with the other info messages.
Finally, with your patch if we resume bootstrap we don't start thrift. As per
Vince's patch, daemon.start() is desirable here over startNativeTransport so
that we always start thrift and CQL.
> streaming failure during bootstrap makes new node into inconsistent state
> -
>
> Key: CASSANDRA-14525
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14525
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Major
> Fix For: 4.0, 2.2.x, 3.0.x
>
>
> If bootstrap fails for newly joining node (most common reason is due to
> streaming failure) then Cassandra state remains in {{joining}} state which is
> fine but Cassandra also enables Native transport which makes overall state
> inconsistent. This further creates NullPointer exception if auth is enabled
> on the new node, please find reproducible steps here:
> For example if bootstrap fails due to streaming errors like
> {quote}java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1256)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:894)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:660)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:573)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:330)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:567)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:695)
> [apache-cassandra-3.0.16.jar:3.0.16]
> Caused by:
[jira] [Commented] (CASSANDRA-14543) Hinted handoff to replay purgeable tombstones
[
https://issues.apache.org/jira/browse/CASSANDRA-14543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523042#comment-16523042
]
Kurt Greaves commented on CASSANDRA-14543:
--
I'm not sure about this because it only "works" if you assume the delete
happened within the HH window, and then the node was down for an additional
(*GCGS - HH*). It does reduce the number of cases where users could be bitten
by this, but not significantly, and I suspect this will become another layer of
confusion when someone does hit the problem.
I think also [~slebresne]'s comment
[here|https://issues.apache.org/jira/browse/CASSANDRA-14532?focusedCommentId=16519023=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16519023]
does a good job of explaining why sending tombstones after GCGS is a bad idea.
While I don't think the constant unnecessary read repair is good, there's not
really many good solutions here. Ideally the read repair wouldn't happen in the
first place and we exclude post gcgs tombstones from digest calc but that's
likely very risky. So far it seems the best of a bunch of terrible solutions
here is to either repair within GCGS, or never delete things. Going forward
only_purge_repaired_tombstones w/ incremental repair should fix the underlying
design problem here.
> Hinted handoff to replay purgeable tombstones
> --
>
> Key: CASSANDRA-14543
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14543
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Jay Zhuang
>Priority: Minor
>
> Hinted-handoff currently only dispatches and applies the mutations that are
> within GCGS:
> [{{Hint.java:97}}|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/hints/Hint.java#L97].
> Which is to make sure it won't resurrect any deleted data.
> But replaying tombstones should be safe, it could reduce the chance to have
> [un-repairable inconsistent
> data|https://lists.apache.org/thread.html/2d3d39d960143d4d2146ed2530821504ff855e832713dec7d0afd8ac@%3Cdev.cassandra.apache.org%3E].
> Here is the user scenario it tries to fix:
> {noformat}
> 1. Create a 3 nodes cluster
> 2. Create a table with small gc_grace_seconds (for reproducing purpose):
> CREATE KEYSPACE foo WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': 3};
> CREATE TABLE foo.bar (
> id int PRIMARY KEY,
> name text
> ) WITH gc_grace_seconds=30;
> 3. Insert data with consistency all:
> INSERT INTO foo.bar (id, name) VALUES(1, 'cstar');
> 4. stop 1 node
> $ ccm node2 stop
> 5. Delete the data with consistency quorum:
> DELETE FROM foo.bar WHERE id=1;
> 6. Wait 30 seconds and then start node2:
> $ ccm node2 start
> {noformat}
> Now, node2 has the data, node1/node3 have the purgeable tombstone. It
> triggers RR every time which sends data from node2 to node1/node3 but repairs
> nothing.
> With purgeable tombstones hints handoff, it at least will dispatch the
> tombstone and delete the data on node2. It won't fix the root cause but
> reduce the chance to have this issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Resolved] (CASSANDRA-14470) Repair validation failed/unable to create merkle tree
[
https://issues.apache.org/jira/browse/CASSANDRA-14470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Greaves resolved CASSANDRA-14470.
--
Resolution: Duplicate
> Repair validation failed/unable to create merkle tree
> -
>
> Key: CASSANDRA-14470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14470
> Project: Cassandra
> Issue Type: Bug
>Reporter: Harry Hough
>Priority: Major
>
> I had trouble repairing with a full repair across all nodes and keyspaces so
> I swapped to doing table by table. This table will not repair even after
> scrub/restart of all nodes. I am using command:
> {code:java}
> nodetool repair -full -seq keyspace table
> {code}
> {code:java}
> [2018-05-25 19:26:36,525] Repair session 0198ee50-6050-11e8-a3b7-9d0793eab507
> for range [(165598500763544933,166800441975877433],
> (-5455068259072262254,-5445777107512274819],
> (-4614366950466274594,-4609359222424798148],
> (3417371506258365094,3421921915575816226],
> (5221788898381458942,5222846663270250559],
> (3421921915575816226,3429175540277204991],
> (3276484330153091115,3282213186258578546],
> (-3306169730424140596,-3303439264231406101],
> (5228704360821395206,5242415853745535023],
> (5808045095951939338,5808562658315740708],
> (-3303439264231406101,-3302592736123212969]] finished (progress: 1%)
> [2018-05-25 19:27:23,848] Repair session 0180f980-6050-11e8-a3b7-9d0793eab507
> for range [(-8495158945319933291,-8482949618583319581],
> (1803296697741516342,1805330812863783941],
> (8633191319643427141,8637771071728131257],
> (2214097236323810344,2218253238829661319],
> (8637771071728131257,8639627594735133685],
> (2195525904029414718,2214097236323810344],
> (-8500127431270773970,-8495158945319933291],
> (7151693083782264341,7152162989417914407],
> (-8482949618583319581,-8481973749935314249]] finished (progress: 1%)
> [2018-05-25 19:30:32,590] Repair session 01ac9d62-6050-11e8-a3b7-9d0793eab507
> for range [(7887346492105510731,7893062759268864220],
> (-153277717939330979,-151986584968539220],
> (-6351665356961460262,-6336288442758847669],
> (7881942012672602731,7887346492105510731],
> (-5884528383037906783,-5878097817437987368],
> (6054625594262089428,6060773114960761336],
> (-6354401100436622515,-6351665356961460262],
> (3358411934943460772,336336663817876],
> (6255644242745576360,6278718135193665575],
> (-6321106762570843270,-6316788220143151823],
> (1754319239259058661,1759314644652031521],
> (7893062759268864220,7894890594190784729],
> (-8012293411840276426,-8011781808288431224]] failed with error [repair
> #01ac9d62-6050-11e8-a3b7-9d0793eab507 on keyspace/table,
> [(7887346492105510731,7893062759268864220],
> (-153277717939330979,-151986584968539220],
> (-6351665356961460262,-6336288442758847669],
> (7881942012672602731,7887346492105510731],
> (-5884528383037906783,-5878097817437987368],
> (6054625594262089428,6060773114960761336],
> (-6354401100436622515,-6351665356961460262],
> (3358411934943460772,336336663817876],
> (6255644242745576360,6278718135193665575],
> (-6321106762570843270,-6316788220143151823],
> (1754319239259058661,1759314644652031521],
> (7893062759268864220,7894890594190784729],
> (-8012293411840276426,-8011781808288431224]]] Validation failed in
> /192.168.8.64 (progress: 1%)
> [2018-05-25 19:30:38,744] Repair session 01ab16c1-6050-11e8-a3b7-9d0793eab507
> for range [(4474598255414218354,4477186372547790770],
> (-8368931070988054567,-8367389908801757978],
> (4445104759712094068,4445123832517144036],
> (6749641233379918040,6749879473217708908],
> (717627050679001698,729408043324000761],
> (8984622403893999385,8990662643404904110],
> (4457612694557846994,4474598255414218354],
> (5589049422573545528,5593079877787783784],
> (3609693317839644945,3613727999875360405],
> (8499016262183246473,8504603366117127178],
> (-5421277973540712245,-5417725796037372830],
> (5586405751301680690,5589049422573545528],
> (-2611069890590917549,-2603911539353128123],
> (2424772330724108233,2427564448454334730],
> (3172651438220766183,3175226710613527829],
> (4445123832517144036,4457612694557846994],
> (-6827531712183440570,-6800863837312326365],
> (5593079877787783784,5596020904874304252],
> (716705770783505310,717627050679001698],
> (115377252345874298,119626359210683992],
> (239394377432130766,240250561347730054]] failed with error [repair
> #01ab16c1-6050-11e8-a3b7-9d0793eab507 on keyspace/table,
> [(4474598255414218354,4477186372547790770],
> (-8368931070988054567,-8367389908801757978],
> (4445104759712094068,4445123832517144036],
> (6749641233379918040,6749879473217708908],
> (717627050679001698,729408043324000761],
> (8984622403893999385,8990662643404904110],
> (4457612694557846994,4474598255414218354],
>
[jira] [Commented] (CASSANDRA-14525) streaming failure during bootstrap makes new node into inconsistent state
[
https://issues.apache.org/jira/browse/CASSANDRA-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521843#comment-16521843
]
Kurt Greaves commented on CASSANDRA-14525:
--
Sorry about that, we had a pretty busy week last week and Vince probably won't
have time. I'll review.
> streaming failure during bootstrap makes new node into inconsistent state
> -
>
> Key: CASSANDRA-14525
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14525
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Major
> Fix For: 4.0, 2.2.x, 3.0.x
>
>
> If bootstrap fails for newly joining node (most common reason is due to
> streaming failure) then Cassandra state remains in {{joining}} state which is
> fine but Cassandra also enables Native transport which makes overall state
> inconsistent. This further creates NullPointer exception if auth is enabled
> on the new node, please find reproducible steps here:
> For example if bootstrap fails due to streaming errors like
> {quote}java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1256)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:894)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:660)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:573)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:330)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:567)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:695)
> [apache-cassandra-3.0.16.jar:3.0.16]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:187)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:440)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:540)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:307)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
> {quote}
> then variable [StorageService.java::dataAvailable
> |https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageService.java#L892]
> will be {{false}}. Since {{dataAvailable}} is {{false}} hence it will not
> call [StorageService.java::finishJoiningRing
> |https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageService.java#L933]
> and as a result
>
[jira] [Assigned] (CASSANDRA-13857) Allow MV with only partition key
[
https://issues.apache.org/jira/browse/CASSANDRA-13857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Greaves reassigned CASSANDRA-13857:
Assignee: Alexander Ivakov
> Allow MV with only partition key
>
>
> Key: CASSANDRA-13857
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13857
> Project: Cassandra
> Issue Type: Bug
> Components: Materialized Views
>Reporter: Kurt Greaves
>Assignee: Alexander Ivakov
>Priority: Major
>
> We currently disallow creation of a view that has the exact same primary key
> as the base where no clustering keys are present, however a potential use
> case would be a view where part of the PK is filtered so as to have a subset
> of data in the view which is faster for range queries. We actually currently
> allow this, but only if you have a clustering key defined. If you only have a
> partitioning key it's not possible.
> From the mailing list, the below example works:
> {code:java}
> CREATE TABLE users (
> site_id int,
> user_id text,
> n int,
> data set>,
> PRIMARY KEY ((site_id, user_id), n));
> user data is updated and read by PK and sometimes I have to fetch all user
> for some specific site_id. It appeared that full scan by
> token(site_id,user_id) filtered by WHERE site_id = works much
> slower than unfiltered full scan on
> CREATE MATERIALIZED VIEW users_1 AS
> SELECT site_id, user_id, n, data
> FROM users
> WHERE site_id = 1 AND user_id IS NOT NULL AND n IS NOT NULL
> PRIMARY KEY ((site_id, user_id), n);
> {code}
> However the following does not:
> {code:java}
> CREATE TABLE users (
> site_id int,
> user_id text,
> data set,
> PRIMARY KEY ((site_id, user_id)));
> CREATE MATERIALIZED VIEW users_1 AS
> SELECT site_id, user_id, data
> FROM users
> WHERE site_id = 1 AND user_id IS NOT NULL
> PRIMARY KEY ((site_id, user_id));
> InvalidRequest: Error from server: code=2200 [Invalid query] message="No
> columns are defined for Materialized View other than primary key"
> {code}
> This is because if the clustering key is empty we assume they've only defined
> the primary key in the partition key and we haven't accounted for this use
> case.
> On that note, we also don't allow the following narrowing of the partition
> key:
> {code}
> CREATE TABLE kurt.base (
> id int,
> uid text,
> data text,
> PRIMARY KEY (id, uid)
> )
> CREATE MATERIALIZED VIEW kurt.mv2 AS SELECT * from kurt.base where id IS NOT
> NULL and uid='1' PRIMARY KEY ((id, uid));
> InvalidRequest: Error from server: code=2200 [Invalid query] message="No
> columns are defined for Materialized View other than primary key"
> {code}
> But we do allow the following, which works because there is still a
> clustering key, despite not changing the PK.
> {code}
> CREATE MATERIALIZED VIEW kurt.mv2 AS SELECT * from kurt.base where id IS NOT
> NULL and uid='1' PRIMARY KEY (id, uid);
> {code}
> And we also allow the following, which is a narrowing of the partition key as
> above, but with an extra clustering key.
> {code}
> create table kurt.base3 (id int, uid int, clus1 int, clus2 int, data text,
> PRIMARY KEY ((id, uid), clus1, clus2));
> CREATE MATERIALIZED VIEW kurt.mv4 AS SELECT * from kurt.base3 where id IS NOT
> NULL and uid IS NOT NULL and clus1 IS NOT NULL AND clus2 IS NOT NULL PRIMARY
> KEY ((id, uid, clus1), clus2);
> {code}
> I _think_ supporting these cases is trivial and mostly already handled in the
> underlying MV write path, so we might be able to get away with just a simple
> change of [this
> condition|https://github.com/apache/cassandra/blob/83822d12d87dcb3aaad2b1e670e57ebef4ab1c36/src/java/org/apache/cassandra/cql3/statements/CreateViewStatement.java#L291].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14532) Partition level deletions past GCGS are not propagated/merged on read
[
https://issues.apache.org/jira/browse/CASSANDRA-14532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Greaves updated CASSANDRA-14532:
-
Priority: Minor (was: Major)
> Partition level deletions past GCGS are not propagated/merged on read
> -
>
> Key: CASSANDRA-14532
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14532
> Project: Cassandra
> Issue Type: Bug
> Components: Documentation and Website
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Minor
>
> So as [~jay.zhuang] mentioned on the mailing list
> [here|http://mail-archives.us.apache.org/mod_mbox/cassandra-dev/201806.mbox/],
> it appears that partition deletions that have passed GCGS are not
> propagated/merged properly on read, and also not repaired via read repair.
> Steps to reproduce:
> {code}
> create keyspace test WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': 3};
> create table test.test (id int PRIMARY KEY , data text) WITH gc_grace_seconds
> = 10;
> CONSISTENCY ALL;
> INSERT INTO test.test (id, data) values (1, 'test');
> ccm node2 stop
> CONSISTENCY QUORUM;
> DELETE from test.test where id = 1; // wait 10 seconds so HH doesn't
> propagate tombstone when starting node2
> select * from test.test where id = 1 ;
> id | data
> +--
> (0 rows)
> ccm node2 start
> CONSISTENCY ALL;
> select * from test.test where id = 1 ;
> id | data
> +--
> 1 | test
> alter table test.test WITH gc_grace_seconds = 10; // GC
> select * from test.test where id = 1 ;
> id | data
> +--
> (0 rows)
> {code}
> We've also found a seemingly related issue in compaction where trying to
> compact an SSTable which contains the partition deletion post GCGS, the
> partition deletion won't be removed via compaction. Likely the same code is
> causing both bugs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14532) Partition level deletions past GCGS are not propagated/merged on read
[
https://issues.apache.org/jira/browse/CASSANDRA-14532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Greaves updated CASSANDRA-14532:
-
Component/s: Documentation and Website
> Partition level deletions past GCGS are not propagated/merged on read
> -
>
> Key: CASSANDRA-14532
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14532
> Project: Cassandra
> Issue Type: Bug
> Components: Documentation and Website
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Minor
>
> So as [~jay.zhuang] mentioned on the mailing list
> [here|http://mail-archives.us.apache.org/mod_mbox/cassandra-dev/201806.mbox/],
> it appears that partition deletions that have passed GCGS are not
> propagated/merged properly on read, and also not repaired via read repair.
> Steps to reproduce:
> {code}
> create keyspace test WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': 3};
> create table test.test (id int PRIMARY KEY , data text) WITH gc_grace_seconds
> = 10;
> CONSISTENCY ALL;
> INSERT INTO test.test (id, data) values (1, 'test');
> ccm node2 stop
> CONSISTENCY QUORUM;
> DELETE from test.test where id = 1; // wait 10 seconds so HH doesn't
> propagate tombstone when starting node2
> select * from test.test where id = 1 ;
> id | data
> +--
> (0 rows)
> ccm node2 start
> CONSISTENCY ALL;
> select * from test.test where id = 1 ;
> id | data
> +--
> 1 | test
> alter table test.test WITH gc_grace_seconds = 10; // GC
> select * from test.test where id = 1 ;
> id | data
> +--
> (0 rows)
> {code}
> We've also found a seemingly related issue in compaction where trying to
> compact an SSTable which contains the partition deletion post GCGS, the
> partition deletion won't be removed via compaction. Likely the same code is
> causing both bugs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14532) Partition level deletions past GCGS are not propagated/merged on read
[
https://issues.apache.org/jira/browse/CASSANDRA-14532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519843#comment-16519843
]
Kurt Greaves commented on CASSANDRA-14532:
--
bq. That's not really a bug in that this is working as designed. The very
reason for gc grace is to be a long enough time that we can guarantee any data
(including tombstone) has been propagated to all replica, and that's why you
must run repair within the gc grace window (otherwise other mechanism don't
truly guarantee that). So we should not have to propagate anything past gcgs
and doing so is at best an inefficiency.
Yeah we've come to that conclusion on the ML already. I haven't updated here
but I think this ticket really becomes a case of documenting the behaviour a
bit more explicitly. It's pretty hard to reason through, and having already
reasoned through it in the past and forgotten, and that it's happened to other
people seems to make this a good case for documentation. I'd like to put a
better explanation in the code, but more importantly, in the docs somewhere.
I'll look at that in the near future, but regarding docs we don't really have
much (anything?) on tombstones at the moment and it would be terribly out of
place to document this without a whole lot of general/background knowledge on
tombstones. Anyway, I'll get around to writing that up as well at some point...
bq. That would be a real bug, though - worth opening a JIRA there.
bq. Possibly. But remember that being post GCGS is not the only condition on
tombstone for them to be purged: compaction also needs to be able to prove the
tombstone cannot shadow something in another, non-compacted, sstable. So the
sentence is not precise enough to say this is a bug for sure either.
So I'll get exact reproduce steps sorted, but I left out some details in my
original comment which makes it significantly more minor. In this case what we
see is that you can create a single SSTable containing only a partition
deletion with no shadowed data in any other SSTable and when you compact the
tombstone won't be removed. However, as soon as you insert more data that
shadows the tombstone and compact with the SSTable it will be removed (there
may be more to it than this, have to test again).
Either way we figured that's not a major bug in the scheme of things, but I'll
get a test case up and running so can investigate a bit more.
> Partition level deletions past GCGS are not propagated/merged on read
> -
>
> Key: CASSANDRA-14532
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14532
> Project: Cassandra
> Issue Type: Bug
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Major
>
> So as [~jay.zhuang] mentioned on the mailing list
> [here|http://mail-archives.us.apache.org/mod_mbox/cassandra-dev/201806.mbox/],
> it appears that partition deletions that have passed GCGS are not
> propagated/merged properly on read, and also not repaired via read repair.
> Steps to reproduce:
> {code}
> create keyspace test WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': 3};
> create table test.test (id int PRIMARY KEY , data text) WITH gc_grace_seconds
> = 10;
> CONSISTENCY ALL;
> INSERT INTO test.test (id, data) values (1, 'test');
> ccm node2 stop
> CONSISTENCY QUORUM;
> DELETE from test.test where id = 1; // wait 10 seconds so HH doesn't
> propagate tombstone when starting node2
> select * from test.test where id = 1 ;
> id | data
> +--
> (0 rows)
> ccm node2 start
> CONSISTENCY ALL;
> select * from test.test where id = 1 ;
> id | data
> +--
> 1 | test
> alter table test.test WITH gc_grace_seconds = 10; // GC
> select * from test.test where id = 1 ;
> id | data
> +--
> (0 rows)
> {code}
> We've also found a seemingly related issue in compaction where trying to
> compact an SSTable which contains the partition deletion post GCGS, the
> partition deletion won't be removed via compaction. Likely the same code is
> causing both bugs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14532) Partition level deletions past GCGS are not propagated/merged on read
[
https://issues.apache.org/jira/browse/CASSANDRA-14532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Greaves updated CASSANDRA-14532:
-
Description:
So as [~jay.zhuang] mentioned on the mailing list
[here|http://mail-archives.us.apache.org/mod_mbox/cassandra-dev/201806.mbox/],
it appears that partition deletions that have passed GCGS are not
propagated/merged properly on read, and also not repaired via read repair.
Steps to reproduce:
{code}
create keyspace test WITH replication = {'class': 'SimpleStrategy',
'replication_factor': 3};
create table test.test (id int PRIMARY KEY , data text) WITH gc_grace_seconds =
10;
CONSISTENCY ALL;
INSERT INTO test.test (id, data) values (1, 'test');
ccm node2 stop
CONSISTENCY QUORUM;
DELETE from test.test where id = 1; // wait 10 seconds so HH doesn't propagate
tombstone when starting node2
select * from test.test where id = 1 ;
id | data
+--
(0 rows)
ccm node2 start
CONSISTENCY ALL;
select * from test.test where id = 1 ;
id | data
+--
1 | test
alter table test.test WITH gc_grace_seconds = 10; // GC
select * from test.test where id = 1 ;
id | data
+--
(0 rows)
{code}
We've also found a seemingly related issue in compaction where trying to
compact an SSTable which contains the partition deletion post GCGS, the
partition deletion won't be removed via compaction. Likely the same code is
causing both bugs.
was:
So as [~jay.zhuang] mentioned on the mailing list here, it appears that
partition deletions that have passed GCGS are not propagated/merged properly on
read, and also not repaired via read repair.
Steps to reproduce:
{code}
create keyspace test WITH replication = {'class': 'SimpleStrategy',
'replication_factor': 3};
create table test.test (id int PRIMARY KEY , data text) WITH gc_grace_seconds =
10;
CONSISTENCY ALL;
INSERT INTO test.test (id, data) values (1, 'test');
ccm node2 stop
CONSISTENCY QUORUM;
DELETE from test.test where id = 1; // wait 10 seconds so HH doesn't propagate
tombstone when starting node2
select * from test.test where id = 1 ;
id | data
+--
(0 rows)
ccm node2 start
CONSISTENCY ALL;
select * from test.test where id = 1 ;
id | data
+--
1 | test
alter table test.test WITH gc_grace_seconds = 10; // GC
select * from test.test where id = 1 ;
id | data
+--
(0 rows)
{code}
We've also found a seemingly related issue in compaction where trying to
compact an SSTable which contains the partition deletion post GCGS, the
partition deletion won't be removed via compaction. Likely the same code is
causing both bugs.
> Partition level deletions past GCGS are not propagated/merged on read
> -
>
> Key: CASSANDRA-14532
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14532
> Project: Cassandra
> Issue Type: Bug
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Major
>
> So as [~jay.zhuang] mentioned on the mailing list
> [here|http://mail-archives.us.apache.org/mod_mbox/cassandra-dev/201806.mbox/],
> it appears that partition deletions that have passed GCGS are not
> propagated/merged properly on read, and also not repaired via read repair.
> Steps to reproduce:
> {code}
> create keyspace test WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': 3};
> create table test.test (id int PRIMARY KEY , data text) WITH gc_grace_seconds
> = 10;
> CONSISTENCY ALL;
> INSERT INTO test.test (id, data) values (1, 'test');
> ccm node2 stop
> CONSISTENCY QUORUM;
> DELETE from test.test where id = 1; // wait 10 seconds so HH doesn't
> propagate tombstone when starting node2
> select * from test.test where id = 1 ;
> id | data
> +--
> (0 rows)
> ccm node2 start
> CONSISTENCY ALL;
> select * from test.test where id = 1 ;
> id | data
> +--
> 1 | test
> alter table test.test WITH gc_grace_seconds = 10; // GC
> select * from test.test where id = 1 ;
> id | data
> +--
> (0 rows)
> {code}
> We've also found a seemingly related issue in compaction where trying to
> compact an SSTable which contains the partition deletion post GCGS, the
> partition deletion won't be removed via compaction. Likely the same code is
> causing both bugs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14532) Partition level deletions past GCGS are not propagated/merged on read
[
https://issues.apache.org/jira/browse/CASSANDRA-14532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Greaves updated CASSANDRA-14532:
-
Reproduced In: 3.11.3
> Partition level deletions past GCGS are not propagated/merged on read
> -
>
> Key: CASSANDRA-14532
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14532
> Project: Cassandra
> Issue Type: Bug
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Major
>
> So as [~jay.zhuang] mentioned on the mailing list here, it appears that
> partition deletions that have passed GCGS are not propagated/merged properly
> on read, and also not repaired via read repair.
> Steps to reproduce:
> {code}
> create keyspace test WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': 3};
> create table test.test (id int PRIMARY KEY , data text) WITH gc_grace_seconds
> = 10;
> CONSISTENCY ALL;
> INSERT INTO test.test (id, data) values (1, 'test');
> ccm node2 stop
> CONSISTENCY QUORUM;
> DELETE from test.test where id = 1; // wait 10 seconds so HH doesn't
> propagate tombstone when starting node2
> select * from test.test where id = 1 ;
> id | data
> +--
> (0 rows)
> ccm node2 start
> CONSISTENCY ALL;
> select * from test.test where id = 1 ;
> id | data
> +--
> 1 | test
> alter table test.test WITH gc_grace_seconds = 10; // GC
> select * from test.test where id = 1 ;
> id | data
> +--
> (0 rows)
> {code}
> We've also found a seemingly related issue in compaction where trying to
> compact an SSTable which contains the partition deletion post GCGS, the
> partition deletion won't be removed via compaction. Likely the same code is
> causing both bugs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14532) Partition level deletions past GCGS are not propagated/merged on read
Kurt Greaves created CASSANDRA-14532:
Summary: Partition level deletions past GCGS are not
propagated/merged on read
Key: CASSANDRA-14532
URL: https://issues.apache.org/jira/browse/CASSANDRA-14532
Project: Cassandra
Issue Type: Bug
Reporter: Kurt Greaves
Assignee: Kurt Greaves
So as [~jay.zhuang] mentioned on the mailing list here, it appears that
partition deletions that have passed GCGS are not propagated/merged properly on
read, and also not repaired via read repair.
Steps to reproduce:
{code}
create keyspace test WITH replication = {'class': 'SimpleStrategy',
'replication_factor': 3};
create table test.test (id int PRIMARY KEY , data text) WITH gc_grace_seconds =
10;
CONSISTENCY ALL;
INSERT INTO test.test (id, data) values (1, 'test');
ccm node2 stop
CONSISTENCY QUORUM;
DELETE from test.test where id = 1; // wait 10 seconds so HH doesn't propagate
tombstone when starting node2
select * from test.test where id = 1 ;
id | data
+--
(0 rows)
ccm node2 start
CONSISTENCY ALL;
select * from test.test where id = 1 ;
id | data
+--
1 | test
alter table test.test WITH gc_grace_seconds = 10; // GC
select * from test.test where id = 1 ;
id | data
+--
(0 rows)
{code}
We've also found a seemingly related issue in compaction where trying to
compact an SSTable which contains the partition deletion post GCGS, the
partition deletion won't be removed via compaction. Likely the same code is
causing both bugs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14423) SSTables stop being compacted
[
https://issues.apache.org/jira/browse/CASSANDRA-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16517011#comment-16517011
]
Kurt Greaves commented on CASSANDRA-14423:
--
OK. I've updated all 3 branches to use {{nonAnticompacting}} for filtering.
Personally, I'm not a huge fan of this but it avoids the behaviour change
issues previously mentioned and still fixes the problem. Also as I see it, a
cleaner fix would require some refactoring of {{submitAntiCompaction}} and
{{performAntiCompaction}} to decouple anti-compaction from repair, which is not
possible in current releases anyway (to clarify, I don't think
{{performAntiCompaction}} should care about whether SSTables are repaired or
not). I can see a future though where it's possible to anti-compact SSTables
separate to a repair, but I'll think about that for a separate ticket for trunk.
Side-notes related to patches: I took the liberty of making one log message
less spammy "SSTable x will be anticompacted on range..." - with many vnodes
this becomes ridiculously spammy. It's not a lot better in the patched version,
but at least it's a one line log message with no repeated information.
And I took the liberty of fixing the "anti-compacted from x to y SSTables" log
message in {{doAntiCompaction}}. This was broken as {{repaired.originals.size}}
could/would return a wrong size after \{{repaired.split(sstableGroup)}} was
called.
Also, the assert at the end of {{performAntiCompaction}} had to be removed as
we're removing {{nonAnticompacting}} from the transaction which violates the
assert, this is fine however as the assert was effectively incorrect in the
first place, and only valid due to the bug.
Finally, I split the tests out into a separate method ({{shouldAntiCompact}}),
and added more test cases
|[2.2|https://github.com/apache/cassandra/compare/cassandra-2.2...kgreav:14423-2.2]|[utests|https://circleci.com/gh/kgreav/cassandra/169]|
|[3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...kgreav:14423-3.0]|[utests|https://circleci.com/gh/kgreav/cassandra/171]|
|[3.11|https://github.com/apache/cassandra/compare/cassandra-3.11...kgreav:14423-3.11]|[utests|https://circleci.com/gh/kgreav/cassandra/173]|
I'm working on a unit test for trunk, but it's going to have to be a fair bit
different, as the repaired SSTables are filtered out prior to creation of the
transaction. I don't think it needs to block review of the other branches
however.
> SSTables stop being compacted
> -
>
> Key: CASSANDRA-14423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14423
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Major
> Fix For: 2.2.13, 3.0.17, 3.11.3
>
>
> So seeing a problem in 3.11.0 where SSTables are being lost from the view and
> not being included in compactions/as candidates for compaction. It seems to
> get progressively worse until there's only 1-2 SSTables in the view which
> happen to be the most recent SSTables and thus compactions completely stop
> for that table.
> The SSTables seem to still be included in reads, just not compactions.
> The issue can be fixed by restarting C*, as it will reload all SSTables into
> the view, but this is only a temporary fix. User defined/major compactions
> still work - not clear if they include the result back in the view but is not
> a good work around.
> This also results in a discrepancy between SSTable count and SSTables in
> levels for any table using LCS.
> {code:java}
> Keyspace : xxx
> Read Count: 57761088
> Read Latency: 0.10527088681224288 ms.
> Write Count: 2513164
> Write Latency: 0.018211106398149903 ms.
> Pending Flushes: 0
> Table: xxx
> SSTable count: 10
> SSTables in each level: [2, 0, 0, 0, 0, 0, 0, 0, 0]
> Space used (live): 894498746
> Space used (total): 894498746
> Space used by snapshots (total): 0
> Off heap memory used (total): 11576197
> SSTable Compression Ratio: 0.6956629530569777
> Number of keys (estimate): 3562207
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 87
> Local read count: 57761088
> Local read latency: 0.108 ms
> Local write count: 2513164
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 86.33
> Bloom filter false positives: 43
> Bloom filter false ratio: 0.0
> Bloom filter space used: 8046104
> Bloom filter off heap memory used: 8046024
> Index summary off heap memory used: 3449005
> Compression metadata off heap memory used: 81168
> Compacted partition minimum bytes: 104
> Compacted partition maximum bytes: 5722
> Compacted partition mean bytes: 175
> Average live cells per slice (last five minutes): 1.0
> Maximum live cells per slice (last five
[jira] [Commented] (CASSANDRA-14525) streaming failure during bootstrap makes new node into inconsistent state
[
https://issues.apache.org/jira/browse/CASSANDRA-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516669#comment-16516669
]
Kurt Greaves commented on CASSANDRA-14525:
--
Thanks [~chovatia.jayd...@gmail.com], I agree. No fault on your part, more a
problem with the consistent lack of reviewers we have who can prioritise review
work. Just unfortunate that it's wasted more time than necessary for everyone.
I think [~VincentWhite] would appreciate the acknowledgement (especially after
such a long time) so FCFS makes sense to me, but there's no use doing the work
twice, just take into account the two patches slight discrepancies when
reviewing I guess.
> streaming failure during bootstrap makes new node into inconsistent state
> -
>
> Key: CASSANDRA-14525
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14525
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Major
> Fix For: 4.0, 2.2.x, 3.0.x
>
>
> If bootstrap fails for newly joining node (most common reason is due to
> streaming failure) then Cassandra state remains in {{joining}} state which is
> fine but Cassandra also enables Native transport which makes overall state
> inconsistent. This further creates NullPointer exception if auth is enabled
> on the new node, please find reproducible steps here:
> For example if bootstrap fails due to streaming errors like
> {quote}java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1256)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:894)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:660)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:573)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:330)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:567)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:695)
> [apache-cassandra-3.0.16.jar:3.0.16]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:187)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:440)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:540)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:307)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
> {quote}
> then variable [StorageService.java::dataAvailable
> |https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageService.java#L892]
> will be
[jira] [Comment Edited] (CASSANDRA-14525) streaming failure during bootstrap makes new node into inconsistent state
[
https://issues.apache.org/jira/browse/CASSANDRA-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515592#comment-16515592
]
Kurt Greaves edited comment on CASSANDRA-14525 at 6/18/18 11:15 AM:
We've already had a ticket (and a _very_ similar patch) for this since November
last year... CASSANDRA-14063
was (Author: kurtg):
We've already had a ticket (and a _very_ similar patch( for this since November
last year... CASSANDRA-14063
> streaming failure during bootstrap makes new node into inconsistent state
> -
>
> Key: CASSANDRA-14525
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14525
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Major
> Fix For: 4.0, 2.2.x, 3.0.x
>
>
> If bootstrap fails for newly joining node (most common reason is due to
> streaming failure) then Cassandra state remains in {{joining}} state which is
> fine but Cassandra also enables Native transport which makes overall state
> inconsistent. This further creates NullPointer exception if auth is enabled
> on the new node, please find reproducible steps here:
> For example if bootstrap fails due to streaming errors like
> {quote}java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1256)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:894)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:660)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:573)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:330)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:567)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:695)
> [apache-cassandra-3.0.16.jar:3.0.16]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:187)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:440)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:540)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:307)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
> {quote}
> then variable [StorageService.java::dataAvailable
> |https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageService.java#L892]
> will be {{false}}. Since {{dataAvailable}} is {{false}} hence it will not
> call [StorageService.java::finishJoiningRing
>
[jira] [Commented] (CASSANDRA-14525) streaming failure during bootstrap makes new node into inconsistent state
[
https://issues.apache.org/jira/browse/CASSANDRA-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515592#comment-16515592
]
Kurt Greaves commented on CASSANDRA-14525:
--
We've already had a ticket (and a _very_ similar patch( for this since November
last year... CASSANDRA-14063
> streaming failure during bootstrap makes new node into inconsistent state
> -
>
> Key: CASSANDRA-14525
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14525
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Major
> Fix For: 4.0, 2.2.x, 3.0.x
>
>
> If bootstrap fails for newly joining node (most common reason is due to
> streaming failure) then Cassandra state remains in {{joining}} state which is
> fine but Cassandra also enables Native transport which makes overall state
> inconsistent. This further creates NullPointer exception if auth is enabled
> on the new node, please find reproducible steps here:
> For example if bootstrap fails due to streaming errors like
> {quote}java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1256)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:894)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:660)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:573)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:330)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:567)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:695)
> [apache-cassandra-3.0.16.jar:3.0.16]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:187)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:440)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:540)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:307)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
> {quote}
> then variable [StorageService.java::dataAvailable
> |https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageService.java#L892]
> will be {{false}}. Since {{dataAvailable}} is {{false}} hence it will not
> call [StorageService.java::finishJoiningRing
> |https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageService.java#L933]
> and as a result
>
[jira] [Commented] (CASSANDRA-14423) SSTables stop being compacted
[
https://issues.apache.org/jira/browse/CASSANDRA-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515286#comment-16515286
]
Kurt Greaves commented on CASSANDRA-14423:
--
[~spo...@gmail.com] That change was to ensure that we didn't send repair status
change notifications where the SSTable was already marked as repaired. My
change doesn't revert that, as we no longer pass through the repaired SSTables
to {{performAntiCompaction}} so there is no need to filter them out. Granted it
does change the behaviour of {{performAntiCompaction}} and if someone were to
call it and pass in repaired SSTables, it would produce the old behaviour. But
arguably you should never be passing already repaired SSTables to
{{performAntiCompaction}}. At the moment {{performAntiCompaction}} is only ever
used by submitAntiCompaction in the codebase, so it's only a problem if 3rd
party tools are using it. If we're worried maybe adding a test to the start of
{{performAntiCompaction}} to check the SSTables aren't already repaired would
be the way to go?
utests:
[3.11|https://circleci.com/gh/kgreav/cassandra/163]
[3.0|https://circleci.com/gh/kgreav/cassandra/167]
[2.2|https://circleci.com/gh/kgreav/cassandra/165]
> SSTables stop being compacted
> -
>
> Key: CASSANDRA-14423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14423
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Major
> Fix For: 2.2.13, 3.0.17, 3.11.3
>
>
> So seeing a problem in 3.11.0 where SSTables are being lost from the view and
> not being included in compactions/as candidates for compaction. It seems to
> get progressively worse until there's only 1-2 SSTables in the view which
> happen to be the most recent SSTables and thus compactions completely stop
> for that table.
> The SSTables seem to still be included in reads, just not compactions.
> The issue can be fixed by restarting C*, as it will reload all SSTables into
> the view, but this is only a temporary fix. User defined/major compactions
> still work - not clear if they include the result back in the view but is not
> a good work around.
> This also results in a discrepancy between SSTable count and SSTables in
> levels for any table using LCS.
> {code:java}
> Keyspace : xxx
> Read Count: 57761088
> Read Latency: 0.10527088681224288 ms.
> Write Count: 2513164
> Write Latency: 0.018211106398149903 ms.
> Pending Flushes: 0
> Table: xxx
> SSTable count: 10
> SSTables in each level: [2, 0, 0, 0, 0, 0, 0, 0, 0]
> Space used (live): 894498746
> Space used (total): 894498746
> Space used by snapshots (total): 0
> Off heap memory used (total): 11576197
> SSTable Compression Ratio: 0.6956629530569777
> Number of keys (estimate): 3562207
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 87
> Local read count: 57761088
> Local read latency: 0.108 ms
> Local write count: 2513164
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 86.33
> Bloom filter false positives: 43
> Bloom filter false ratio: 0.0
> Bloom filter space used: 8046104
> Bloom filter off heap memory used: 8046024
> Index summary off heap memory used: 3449005
> Compression metadata off heap memory used: 81168
> Compacted partition minimum bytes: 104
> Compacted partition maximum bytes: 5722
> Compacted partition mean bytes: 175
> Average live cells per slice (last five minutes): 1.0
> Maximum live cells per slice (last five minutes): 1
> Average tombstones per slice (last five minutes): 1.0
> Maximum tombstones per slice (last five minutes): 1
> Dropped Mutations: 0
> {code}
> Also for STCS we've confirmed that SSTable count will be different to the
> number of SSTables reported in the Compaction Bucket's. In the below example
> there's only 3 SSTables in a single bucket - no more are listed for this
> table. Compaction thresholds haven't been modified for this table and it's a
> very basic KV schema.
> {code:java}
> Keyspace : yyy
> Read Count: 30485
> Read Latency: 0.06708991307200263 ms.
> Write Count: 57044
> Write Latency: 0.02204061776873992 ms.
> Pending Flushes: 0
> Table: yyy
> SSTable count: 19
> Space used (live): 18195482
> Space used (total): 18195482
> Space used by snapshots (total): 0
> Off heap memory used (total): 747376
> SSTable Compression Ratio: 0.7607394576769735
> Number of keys (estimate): 116074
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 39
> Local read count: 30485
> Local read latency: NaN ms
> Local write count: 57044
>
[jira] [Commented] (CASSANDRA-14423) SSTables stop being compacted
[
https://issues.apache.org/jira/browse/CASSANDRA-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513345#comment-16513345
]
Kurt Greaves commented on CASSANDRA-14423:
--
Patches for all branches:
|[2.2|https://github.com/apache/cassandra/compare/cassandra-2.2...kgreav:14423-2.2]|
|[3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...kgreav:14423-3.0]|
|[3.11|https://github.com/apache/cassandra/compare/cassandra-3.11...kgreav:14423-3.11]|
> SSTables stop being compacted
> -
>
> Key: CASSANDRA-14423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14423
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Major
> Fix For: 2.2.13, 3.0.17, 3.11.3
>
>
> So seeing a problem in 3.11.0 where SSTables are being lost from the view and
> not being included in compactions/as candidates for compaction. It seems to
> get progressively worse until there's only 1-2 SSTables in the view which
> happen to be the most recent SSTables and thus compactions completely stop
> for that table.
> The SSTables seem to still be included in reads, just not compactions.
> The issue can be fixed by restarting C*, as it will reload all SSTables into
> the view, but this is only a temporary fix. User defined/major compactions
> still work - not clear if they include the result back in the view but is not
> a good work around.
> This also results in a discrepancy between SSTable count and SSTables in
> levels for any table using LCS.
> {code:java}
> Keyspace : xxx
> Read Count: 57761088
> Read Latency: 0.10527088681224288 ms.
> Write Count: 2513164
> Write Latency: 0.018211106398149903 ms.
> Pending Flushes: 0
> Table: xxx
> SSTable count: 10
> SSTables in each level: [2, 0, 0, 0, 0, 0, 0, 0, 0]
> Space used (live): 894498746
> Space used (total): 894498746
> Space used by snapshots (total): 0
> Off heap memory used (total): 11576197
> SSTable Compression Ratio: 0.6956629530569777
> Number of keys (estimate): 3562207
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 87
> Local read count: 57761088
> Local read latency: 0.108 ms
> Local write count: 2513164
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 86.33
> Bloom filter false positives: 43
> Bloom filter false ratio: 0.0
> Bloom filter space used: 8046104
> Bloom filter off heap memory used: 8046024
> Index summary off heap memory used: 3449005
> Compression metadata off heap memory used: 81168
> Compacted partition minimum bytes: 104
> Compacted partition maximum bytes: 5722
> Compacted partition mean bytes: 175
> Average live cells per slice (last five minutes): 1.0
> Maximum live cells per slice (last five minutes): 1
> Average tombstones per slice (last five minutes): 1.0
> Maximum tombstones per slice (last five minutes): 1
> Dropped Mutations: 0
> {code}
> Also for STCS we've confirmed that SSTable count will be different to the
> number of SSTables reported in the Compaction Bucket's. In the below example
> there's only 3 SSTables in a single bucket - no more are listed for this
> table. Compaction thresholds haven't been modified for this table and it's a
> very basic KV schema.
> {code:java}
> Keyspace : yyy
> Read Count: 30485
> Read Latency: 0.06708991307200263 ms.
> Write Count: 57044
> Write Latency: 0.02204061776873992 ms.
> Pending Flushes: 0
> Table: yyy
> SSTable count: 19
> Space used (live): 18195482
> Space used (total): 18195482
> Space used by snapshots (total): 0
> Off heap memory used (total): 747376
> SSTable Compression Ratio: 0.7607394576769735
> Number of keys (estimate): 116074
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 39
> Local read count: 30485
> Local read latency: NaN ms
> Local write count: 57044
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 79.76
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 690912
> Bloom filter off heap memory used: 690760
> Index summary off heap memory used: 54736
> Compression metadata off heap memory used: 1880
> Compacted partition minimum bytes: 73
> Compacted partition maximum bytes: 124
> Compacted partition mean bytes: 96
> Average live cells per slice (last five minutes): NaN
> Maximum live cells per slice (last five minutes): 0
> Average tombstones per slice (last five minutes): NaN
> Maximum tombstones per slice
[jira] [Commented] (CASSANDRA-14470) Repair validation failed/unable to create merkle tree
[
https://issues.apache.org/jira/browse/CASSANDRA-14470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513335#comment-16513335
]
Kurt Greaves commented on CASSANDRA-14470:
--
Sounds likely, if you're happy with that do you mind if we close the ticket?
> Repair validation failed/unable to create merkle tree
> -
>
> Key: CASSANDRA-14470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14470
> Project: Cassandra
> Issue Type: Bug
>Reporter: Harry Hough
>Priority: Major
>
> I had trouble repairing with a full repair across all nodes and keyspaces so
> I swapped to doing table by table. This table will not repair even after
> scrub/restart of all nodes. I am using command:
> {code:java}
> nodetool repair -full -seq keyspace table
> {code}
> {code:java}
> [2018-05-25 19:26:36,525] Repair session 0198ee50-6050-11e8-a3b7-9d0793eab507
> for range [(165598500763544933,166800441975877433],
> (-5455068259072262254,-5445777107512274819],
> (-4614366950466274594,-4609359222424798148],
> (3417371506258365094,3421921915575816226],
> (5221788898381458942,5222846663270250559],
> (3421921915575816226,3429175540277204991],
> (3276484330153091115,3282213186258578546],
> (-3306169730424140596,-3303439264231406101],
> (5228704360821395206,5242415853745535023],
> (5808045095951939338,5808562658315740708],
> (-3303439264231406101,-3302592736123212969]] finished (progress: 1%)
> [2018-05-25 19:27:23,848] Repair session 0180f980-6050-11e8-a3b7-9d0793eab507
> for range [(-8495158945319933291,-8482949618583319581],
> (1803296697741516342,1805330812863783941],
> (8633191319643427141,8637771071728131257],
> (2214097236323810344,2218253238829661319],
> (8637771071728131257,8639627594735133685],
> (2195525904029414718,2214097236323810344],
> (-8500127431270773970,-8495158945319933291],
> (7151693083782264341,7152162989417914407],
> (-8482949618583319581,-8481973749935314249]] finished (progress: 1%)
> [2018-05-25 19:30:32,590] Repair session 01ac9d62-6050-11e8-a3b7-9d0793eab507
> for range [(7887346492105510731,7893062759268864220],
> (-153277717939330979,-151986584968539220],
> (-6351665356961460262,-6336288442758847669],
> (7881942012672602731,7887346492105510731],
> (-5884528383037906783,-5878097817437987368],
> (6054625594262089428,6060773114960761336],
> (-6354401100436622515,-6351665356961460262],
> (3358411934943460772,336336663817876],
> (6255644242745576360,6278718135193665575],
> (-6321106762570843270,-6316788220143151823],
> (1754319239259058661,1759314644652031521],
> (7893062759268864220,7894890594190784729],
> (-8012293411840276426,-8011781808288431224]] failed with error [repair
> #01ac9d62-6050-11e8-a3b7-9d0793eab507 on keyspace/table,
> [(7887346492105510731,7893062759268864220],
> (-153277717939330979,-151986584968539220],
> (-6351665356961460262,-6336288442758847669],
> (7881942012672602731,7887346492105510731],
> (-5884528383037906783,-5878097817437987368],
> (6054625594262089428,6060773114960761336],
> (-6354401100436622515,-6351665356961460262],
> (3358411934943460772,336336663817876],
> (6255644242745576360,6278718135193665575],
> (-6321106762570843270,-6316788220143151823],
> (1754319239259058661,1759314644652031521],
> (7893062759268864220,7894890594190784729],
> (-8012293411840276426,-8011781808288431224]]] Validation failed in
> /192.168.8.64 (progress: 1%)
> [2018-05-25 19:30:38,744] Repair session 01ab16c1-6050-11e8-a3b7-9d0793eab507
> for range [(4474598255414218354,4477186372547790770],
> (-8368931070988054567,-8367389908801757978],
> (4445104759712094068,4445123832517144036],
> (6749641233379918040,6749879473217708908],
> (717627050679001698,729408043324000761],
> (8984622403893999385,8990662643404904110],
> (4457612694557846994,4474598255414218354],
> (5589049422573545528,5593079877787783784],
> (3609693317839644945,3613727999875360405],
> (8499016262183246473,8504603366117127178],
> (-5421277973540712245,-5417725796037372830],
> (5586405751301680690,5589049422573545528],
> (-2611069890590917549,-2603911539353128123],
> (2424772330724108233,2427564448454334730],
> (3172651438220766183,3175226710613527829],
> (4445123832517144036,4457612694557846994],
> (-6827531712183440570,-6800863837312326365],
> (5593079877787783784,5596020904874304252],
> (716705770783505310,717627050679001698],
> (115377252345874298,119626359210683992],
> (239394377432130766,240250561347730054]] failed with error [repair
> #01ab16c1-6050-11e8-a3b7-9d0793eab507 on keyspace/table,
> [(4474598255414218354,4477186372547790770],
> (-8368931070988054567,-8367389908801757978],
> (4445104759712094068,4445123832517144036],
> (6749641233379918040,6749879473217708908],
> (717627050679001698,729408043324000761],
>
[jira] [Updated] (CASSANDRA-14423) SSTables stop being compacted
[
https://issues.apache.org/jira/browse/CASSANDRA-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Greaves updated CASSANDRA-14423:
-
Reproduced In: 3.11.2, 3.11.0 (was: 3.11.0)
> SSTables stop being compacted
> -
>
> Key: CASSANDRA-14423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14423
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Major
> Fix For: 2.2.13, 3.0.17, 3.11.3
>
>
> So seeing a problem in 3.11.0 where SSTables are being lost from the view and
> not being included in compactions/as candidates for compaction. It seems to
> get progressively worse until there's only 1-2 SSTables in the view which
> happen to be the most recent SSTables and thus compactions completely stop
> for that table.
> The SSTables seem to still be included in reads, just not compactions.
> The issue can be fixed by restarting C*, as it will reload all SSTables into
> the view, but this is only a temporary fix. User defined/major compactions
> still work - not clear if they include the result back in the view but is not
> a good work around.
> This also results in a discrepancy between SSTable count and SSTables in
> levels for any table using LCS.
> {code:java}
> Keyspace : xxx
> Read Count: 57761088
> Read Latency: 0.10527088681224288 ms.
> Write Count: 2513164
> Write Latency: 0.018211106398149903 ms.
> Pending Flushes: 0
> Table: xxx
> SSTable count: 10
> SSTables in each level: [2, 0, 0, 0, 0, 0, 0, 0, 0]
> Space used (live): 894498746
> Space used (total): 894498746
> Space used by snapshots (total): 0
> Off heap memory used (total): 11576197
> SSTable Compression Ratio: 0.6956629530569777
> Number of keys (estimate): 3562207
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 87
> Local read count: 57761088
> Local read latency: 0.108 ms
> Local write count: 2513164
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 86.33
> Bloom filter false positives: 43
> Bloom filter false ratio: 0.0
> Bloom filter space used: 8046104
> Bloom filter off heap memory used: 8046024
> Index summary off heap memory used: 3449005
> Compression metadata off heap memory used: 81168
> Compacted partition minimum bytes: 104
> Compacted partition maximum bytes: 5722
> Compacted partition mean bytes: 175
> Average live cells per slice (last five minutes): 1.0
> Maximum live cells per slice (last five minutes): 1
> Average tombstones per slice (last five minutes): 1.0
> Maximum tombstones per slice (last five minutes): 1
> Dropped Mutations: 0
> {code}
> Also for STCS we've confirmed that SSTable count will be different to the
> number of SSTables reported in the Compaction Bucket's. In the below example
> there's only 3 SSTables in a single bucket - no more are listed for this
> table. Compaction thresholds haven't been modified for this table and it's a
> very basic KV schema.
> {code:java}
> Keyspace : yyy
> Read Count: 30485
> Read Latency: 0.06708991307200263 ms.
> Write Count: 57044
> Write Latency: 0.02204061776873992 ms.
> Pending Flushes: 0
> Table: yyy
> SSTable count: 19
> Space used (live): 18195482
> Space used (total): 18195482
> Space used by snapshots (total): 0
> Off heap memory used (total): 747376
> SSTable Compression Ratio: 0.7607394576769735
> Number of keys (estimate): 116074
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 39
> Local read count: 30485
> Local read latency: NaN ms
> Local write count: 57044
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 79.76
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 690912
> Bloom filter off heap memory used: 690760
> Index summary off heap memory used: 54736
> Compression metadata off heap memory used: 1880
> Compacted partition minimum bytes: 73
> Compacted partition maximum bytes: 124
> Compacted partition mean bytes: 96
> Average live cells per slice (last five minutes): NaN
> Maximum live cells per slice (last five minutes): 0
> Average tombstones per slice (last five minutes): NaN
> Maximum tombstones per slice (last five minutes): 0
> Dropped Mutations: 0
> {code}
> {code:java}
> Apr 27 03:10:39 cassandra[9263]: TRACE o.a.c.d.c.SizeTieredCompactionStrategy
> Compaction buckets are
>
[jira] [Commented] (CASSANDRA-14423) SSTables stop being compacted
[
https://issues.apache.org/jira/browse/CASSANDRA-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513294#comment-16513294
]
Kurt Greaves commented on CASSANDRA-14423:
--
In
{{org.apache.cassandra.db.compaction.CompactionManager#submitAntiCompaction}}
we create a transaction over all SSTables included in the repair (including
repaired SSTables when doing full repair) and pass that through to
{{performAntiCompaction}} in which two things can happen:
1. The SSTable is fully contained within the repairing ranges, and in that case
we mutate repairedAt to the current time of repair and add it to
{{mutatedRepairStatuses}}
2. The SSTable isn't fully contained within the repairing ranges (highly likely
if vnodes or single tokens with >RF nodes). In this case we don't add the
_already repaired_ SSTable to {{mutatedRepairStatuses}}.
We then remove all SSTables from the transaction in {{mutatedRepairStatuses}}
[here|https://github.com/apache/cassandra/blob/191ad7b87a4ded26be4ab0bd192ef676f059276c/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L704].
If *2* occured, the already repaired SSTables were not in
{{mutatedRepairStatuses}} and thus didn't get removed from the transaction and
when {{txn.finish()}} is called they get removed from the CompactionStrategy's
view of sstables via
{{org.apache.cassandra.db.lifecycle.LifecycleTransaction#doCommit}} calling
{{Tracker#notifySSTablesChanged}} which will not include the already repaired
SSTables.
The reason CASSANDRA-13153 brought this bug to light was because up until that
point we _were_ anti-compacting already repaired SSTables, and thus upon
anti-compaction (rewrite) they would be added back into the transaction and the
old SSTable would be removed as usual and the new SSTable would take its place.
Seeing as the existing consensus seems to be that there's no real value at the
moment in mutating repaired times on already repaired SSTables I think the best
solution is to not include the repaired SSTables in the transaction in the
first place. This corresponds with how trunk currently works and also is a lot
cleaner, which is how it works in my patch mentioned above. The alternative
would be to remove them from the transaction regardless of if they were
mutated, but this seems pointless considering we don't do anything with it. If
we ever decide there is value in updating repairedAt on already repaired
SSTables, we can add it back and handle it then.
> SSTables stop being compacted
> -
>
> Key: CASSANDRA-14423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14423
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Major
> Fix For: 2.2.13, 3.0.17, 3.11.3
>
>
> So seeing a problem in 3.11.0 where SSTables are being lost from the view and
> not being included in compactions/as candidates for compaction. It seems to
> get progressively worse until there's only 1-2 SSTables in the view which
> happen to be the most recent SSTables and thus compactions completely stop
> for that table.
> The SSTables seem to still be included in reads, just not compactions.
> The issue can be fixed by restarting C*, as it will reload all SSTables into
> the view, but this is only a temporary fix. User defined/major compactions
> still work - not clear if they include the result back in the view but is not
> a good work around.
> This also results in a discrepancy between SSTable count and SSTables in
> levels for any table using LCS.
> {code:java}
> Keyspace : xxx
> Read Count: 57761088
> Read Latency: 0.10527088681224288 ms.
> Write Count: 2513164
> Write Latency: 0.018211106398149903 ms.
> Pending Flushes: 0
> Table: xxx
> SSTable count: 10
> SSTables in each level: [2, 0, 0, 0, 0, 0, 0, 0, 0]
> Space used (live): 894498746
> Space used (total): 894498746
> Space used by snapshots (total): 0
> Off heap memory used (total): 11576197
> SSTable Compression Ratio: 0.6956629530569777
> Number of keys (estimate): 3562207
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 87
> Local read count: 57761088
> Local read latency: 0.108 ms
> Local write count: 2513164
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 86.33
> Bloom filter false positives: 43
> Bloom filter false ratio: 0.0
> Bloom filter space used: 8046104
> Bloom filter off heap memory used: 8046024
> Index summary off heap memory used: 3449005
> Compression metadata off heap memory used: 81168
> Compacted partition minimum bytes: 104
> Compacted partition maximum bytes: 5722
> Compacted partition mean bytes: 175
> Average live cells per slice (last five minutes): 1.0
> Maximum live cells per slice
[jira] [Updated] (CASSANDRA-14423) SSTables stop being compacted
[
https://issues.apache.org/jira/browse/CASSANDRA-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Greaves updated CASSANDRA-14423:
-
Fix Version/s: 3.0.17
2.2.13
> SSTables stop being compacted
> -
>
> Key: CASSANDRA-14423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14423
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Major
> Fix For: 2.2.13, 3.0.17, 3.11.3
>
>
> So seeing a problem in 3.11.0 where SSTables are being lost from the view and
> not being included in compactions/as candidates for compaction. It seems to
> get progressively worse until there's only 1-2 SSTables in the view which
> happen to be the most recent SSTables and thus compactions completely stop
> for that table.
> The SSTables seem to still be included in reads, just not compactions.
> The issue can be fixed by restarting C*, as it will reload all SSTables into
> the view, but this is only a temporary fix. User defined/major compactions
> still work - not clear if they include the result back in the view but is not
> a good work around.
> This also results in a discrepancy between SSTable count and SSTables in
> levels for any table using LCS.
> {code:java}
> Keyspace : xxx
> Read Count: 57761088
> Read Latency: 0.10527088681224288 ms.
> Write Count: 2513164
> Write Latency: 0.018211106398149903 ms.
> Pending Flushes: 0
> Table: xxx
> SSTable count: 10
> SSTables in each level: [2, 0, 0, 0, 0, 0, 0, 0, 0]
> Space used (live): 894498746
> Space used (total): 894498746
> Space used by snapshots (total): 0
> Off heap memory used (total): 11576197
> SSTable Compression Ratio: 0.6956629530569777
> Number of keys (estimate): 3562207
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 87
> Local read count: 57761088
> Local read latency: 0.108 ms
> Local write count: 2513164
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 86.33
> Bloom filter false positives: 43
> Bloom filter false ratio: 0.0
> Bloom filter space used: 8046104
> Bloom filter off heap memory used: 8046024
> Index summary off heap memory used: 3449005
> Compression metadata off heap memory used: 81168
> Compacted partition minimum bytes: 104
> Compacted partition maximum bytes: 5722
> Compacted partition mean bytes: 175
> Average live cells per slice (last five minutes): 1.0
> Maximum live cells per slice (last five minutes): 1
> Average tombstones per slice (last five minutes): 1.0
> Maximum tombstones per slice (last five minutes): 1
> Dropped Mutations: 0
> {code}
> Also for STCS we've confirmed that SSTable count will be different to the
> number of SSTables reported in the Compaction Bucket's. In the below example
> there's only 3 SSTables in a single bucket - no more are listed for this
> table. Compaction thresholds haven't been modified for this table and it's a
> very basic KV schema.
> {code:java}
> Keyspace : yyy
> Read Count: 30485
> Read Latency: 0.06708991307200263 ms.
> Write Count: 57044
> Write Latency: 0.02204061776873992 ms.
> Pending Flushes: 0
> Table: yyy
> SSTable count: 19
> Space used (live): 18195482
> Space used (total): 18195482
> Space used by snapshots (total): 0
> Off heap memory used (total): 747376
> SSTable Compression Ratio: 0.7607394576769735
> Number of keys (estimate): 116074
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 39
> Local read count: 30485
> Local read latency: NaN ms
> Local write count: 57044
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 79.76
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 690912
> Bloom filter off heap memory used: 690760
> Index summary off heap memory used: 54736
> Compression metadata off heap memory used: 1880
> Compacted partition minimum bytes: 73
> Compacted partition maximum bytes: 124
> Compacted partition mean bytes: 96
> Average live cells per slice (last five minutes): NaN
> Maximum live cells per slice (last five minutes): 0
> Average tombstones per slice (last five minutes): NaN
> Maximum tombstones per slice (last five minutes): 0
> Dropped Mutations: 0
> {code}
> {code:java}
> Apr 27 03:10:39 cassandra[9263]: TRACE o.a.c.d.c.SizeTieredCompactionStrategy
> Compaction buckets are
>
[jira] [Commented] (CASSANDRA-14423) SSTables stop being compacted
[
https://issues.apache.org/jira/browse/CASSANDRA-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512316#comment-16512316
]
Kurt Greaves commented on CASSANDRA-14423:
--
got a patch & test for 3.11
[here|https://github.com/apache/cassandra/compare/cassandra-3.11...kgreav:14423-3.11].
Started on the wrong branch so will backport to 2.2 and 3.0 tomorrow...
> SSTables stop being compacted
> -
>
> Key: CASSANDRA-14423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14423
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Major
> Fix For: 3.11.3
>
>
> So seeing a problem in 3.11.0 where SSTables are being lost from the view and
> not being included in compactions/as candidates for compaction. It seems to
> get progressively worse until there's only 1-2 SSTables in the view which
> happen to be the most recent SSTables and thus compactions completely stop
> for that table.
> The SSTables seem to still be included in reads, just not compactions.
> The issue can be fixed by restarting C*, as it will reload all SSTables into
> the view, but this is only a temporary fix. User defined/major compactions
> still work - not clear if they include the result back in the view but is not
> a good work around.
> This also results in a discrepancy between SSTable count and SSTables in
> levels for any table using LCS.
> {code:java}
> Keyspace : xxx
> Read Count: 57761088
> Read Latency: 0.10527088681224288 ms.
> Write Count: 2513164
> Write Latency: 0.018211106398149903 ms.
> Pending Flushes: 0
> Table: xxx
> SSTable count: 10
> SSTables in each level: [2, 0, 0, 0, 0, 0, 0, 0, 0]
> Space used (live): 894498746
> Space used (total): 894498746
> Space used by snapshots (total): 0
> Off heap memory used (total): 11576197
> SSTable Compression Ratio: 0.6956629530569777
> Number of keys (estimate): 3562207
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 87
> Local read count: 57761088
> Local read latency: 0.108 ms
> Local write count: 2513164
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 86.33
> Bloom filter false positives: 43
> Bloom filter false ratio: 0.0
> Bloom filter space used: 8046104
> Bloom filter off heap memory used: 8046024
> Index summary off heap memory used: 3449005
> Compression metadata off heap memory used: 81168
> Compacted partition minimum bytes: 104
> Compacted partition maximum bytes: 5722
> Compacted partition mean bytes: 175
> Average live cells per slice (last five minutes): 1.0
> Maximum live cells per slice (last five minutes): 1
> Average tombstones per slice (last five minutes): 1.0
> Maximum tombstones per slice (last five minutes): 1
> Dropped Mutations: 0
> {code}
> Also for STCS we've confirmed that SSTable count will be different to the
> number of SSTables reported in the Compaction Bucket's. In the below example
> there's only 3 SSTables in a single bucket - no more are listed for this
> table. Compaction thresholds haven't been modified for this table and it's a
> very basic KV schema.
> {code:java}
> Keyspace : yyy
> Read Count: 30485
> Read Latency: 0.06708991307200263 ms.
> Write Count: 57044
> Write Latency: 0.02204061776873992 ms.
> Pending Flushes: 0
> Table: yyy
> SSTable count: 19
> Space used (live): 18195482
> Space used (total): 18195482
> Space used by snapshots (total): 0
> Off heap memory used (total): 747376
> SSTable Compression Ratio: 0.7607394576769735
> Number of keys (estimate): 116074
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 39
> Local read count: 30485
> Local read latency: NaN ms
> Local write count: 57044
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 79.76
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 690912
> Bloom filter off heap memory used: 690760
> Index summary off heap memory used: 54736
> Compression metadata off heap memory used: 1880
> Compacted partition minimum bytes: 73
> Compacted partition maximum bytes: 124
> Compacted partition mean bytes: 96
> Average live cells per slice (last five minutes): NaN
> Maximum live cells per slice (last five minutes): 0
> Average tombstones per slice (last five minutes): NaN
> Maximum tombstones per slice (last five minutes): 0
> Dropped Mutations: 0
> {code}
> {code:java}
> Apr 27 03:10:39 cassandra[9263]:
[jira] [Assigned] (CASSANDRA-14496) TWCS erroneously disabling tombstone compactions when unchecked_tombstone_compaction=true
[
https://issues.apache.org/jira/browse/CASSANDRA-14496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Greaves reassigned CASSANDRA-14496:
Assignee: Alexander Ivakov
> TWCS erroneously disabling tombstone compactions when
> unchecked_tombstone_compaction=true
> -
>
> Key: CASSANDRA-14496
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14496
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Robert Tarrall
>Assignee: Alexander Ivakov
>Priority: Minor
>
> This code:
> {code:java}
> this.options = new TimeWindowCompactionStrategyOptions(options);
> if
> (!options.containsKey(AbstractCompactionStrategy.TOMBSTONE_COMPACTION_INTERVAL_OPTION)
> &&
> !options.containsKey(AbstractCompactionStrategy.TOMBSTONE_THRESHOLD_OPTION))
> {
> disableTombstoneCompactions = true;
> logger.debug("Disabling tombstone compactions for TWCS");
> }
> else
> logger.debug("Enabling tombstone compactions for TWCS");
> }
> {code}
> ... in TimeWindowCompactionStrategy.java disables tombstone compactions in
> TWCS if you have not *explicitly* set either tombstone_compaction_interval or
> tombstone_threshold. Adding 'tombstone_compaction_interval': '86400' to the
> compaction stanza in a table definition has the (to me unexpected) side
> effect of enabling tombstone compactions.
> This is surprising and does not appear to be mentioned in the docs.
> I would suggest that tombstone compactions should be run unless these options
> are both set to 0.
> If the concern is that (as with DTCS in CASSANDRA-9234) we don't want to
> waste time on tombstone compactions when we expect the tables to eventually
> be expired away, perhaps we should also check unchecked_tombstone_compaction
> and still enable tombstone compactions if that's set to true.
> May also make sense to set defaults for interval & threshold to 0 & disable
> if they're nonzero so that setting non-default values, rather than setting
> ANY value, is what determines whether tombstone compactions are enabled?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14423) SSTables stop being compacted
[
https://issues.apache.org/jira/browse/CASSANDRA-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511934#comment-16511934
]
Kurt Greaves commented on CASSANDRA-14423:
--
Looks like we only _stopped_ anti-compacting repaired SSTables in
CASSANDRA-13153, so this bug only occurs since 2.2.10, 3.0.13, and 3.11.0.
> SSTables stop being compacted
> -
>
> Key: CASSANDRA-14423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14423
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Major
> Fix For: 3.11.3
>
>
> So seeing a problem in 3.11.0 where SSTables are being lost from the view and
> not being included in compactions/as candidates for compaction. It seems to
> get progressively worse until there's only 1-2 SSTables in the view which
> happen to be the most recent SSTables and thus compactions completely stop
> for that table.
> The SSTables seem to still be included in reads, just not compactions.
> The issue can be fixed by restarting C*, as it will reload all SSTables into
> the view, but this is only a temporary fix. User defined/major compactions
> still work - not clear if they include the result back in the view but is not
> a good work around.
> This also results in a discrepancy between SSTable count and SSTables in
> levels for any table using LCS.
> {code:java}
> Keyspace : xxx
> Read Count: 57761088
> Read Latency: 0.10527088681224288 ms.
> Write Count: 2513164
> Write Latency: 0.018211106398149903 ms.
> Pending Flushes: 0
> Table: xxx
> SSTable count: 10
> SSTables in each level: [2, 0, 0, 0, 0, 0, 0, 0, 0]
> Space used (live): 894498746
> Space used (total): 894498746
> Space used by snapshots (total): 0
> Off heap memory used (total): 11576197
> SSTable Compression Ratio: 0.6956629530569777
> Number of keys (estimate): 3562207
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 87
> Local read count: 57761088
> Local read latency: 0.108 ms
> Local write count: 2513164
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 86.33
> Bloom filter false positives: 43
> Bloom filter false ratio: 0.0
> Bloom filter space used: 8046104
> Bloom filter off heap memory used: 8046024
> Index summary off heap memory used: 3449005
> Compression metadata off heap memory used: 81168
> Compacted partition minimum bytes: 104
> Compacted partition maximum bytes: 5722
> Compacted partition mean bytes: 175
> Average live cells per slice (last five minutes): 1.0
> Maximum live cells per slice (last five minutes): 1
> Average tombstones per slice (last five minutes): 1.0
> Maximum tombstones per slice (last five minutes): 1
> Dropped Mutations: 0
> {code}
> Also for STCS we've confirmed that SSTable count will be different to the
> number of SSTables reported in the Compaction Bucket's. In the below example
> there's only 3 SSTables in a single bucket - no more are listed for this
> table. Compaction thresholds haven't been modified for this table and it's a
> very basic KV schema.
> {code:java}
> Keyspace : yyy
> Read Count: 30485
> Read Latency: 0.06708991307200263 ms.
> Write Count: 57044
> Write Latency: 0.02204061776873992 ms.
> Pending Flushes: 0
> Table: yyy
> SSTable count: 19
> Space used (live): 18195482
> Space used (total): 18195482
> Space used by snapshots (total): 0
> Off heap memory used (total): 747376
> SSTable Compression Ratio: 0.7607394576769735
> Number of keys (estimate): 116074
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 39
> Local read count: 30485
> Local read latency: NaN ms
> Local write count: 57044
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 79.76
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 690912
> Bloom filter off heap memory used: 690760
> Index summary off heap memory used: 54736
> Compression metadata off heap memory used: 1880
> Compacted partition minimum bytes: 73
> Compacted partition maximum bytes: 124
> Compacted partition mean bytes: 96
> Average live cells per slice (last five minutes): NaN
> Maximum live cells per slice (last five minutes): 0
> Average tombstones per slice (last five minutes): NaN
> Maximum tombstones per slice (last five minutes): 0
> Dropped Mutations: 0
> {code}
> {code:java}
> Apr 27 03:10:39 cassandra[9263]: TRACE o.a.c.d.c.SizeTieredCompactionStrategy
>
