[jira] [Comment Edited] (CASSANDRA-14624) Website: Add blog post for Faster Streaming
[ https://issues.apache.org/jira/browse/CASSANDRA-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572547#comment-16572547 ] Nate McCall edited comment on CASSANDRA-14624 at 8/8/18 2:18 AM: - Good content! Did an editing pass and looks fine. I can verify this publishes and looks as it should style wise locally. Adding to SVN and publishing (I think - we'll see). was (Author: zznate): I can verify this publishes and looks as it should locally. Adding to SVN and publishing (I think - we'll see). > Website: Add blog post for Faster Streaming > --- > > Key: CASSANDRA-14624 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14624 > Project: Cassandra > Issue Type: Task > Components: Documentation and Website >Reporter: Dinesh Joshi >Assignee: Dinesh Joshi >Priority: Major > Attachments: faster-streaming-blog.patch > > > Please add a new blog post entry on the Cassandra website. It describes the > recent work on performance optimizations in Cassandra related to streaming > enhancements. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
svn commit: r1837624 - in /cassandra/site/publish/blog: ./ 2018/ 2018/08/ 2018/08/07/ 2018/08/07/faster_streaming_in_cassandra.html index.html
Author: zznate Date: Wed Aug 8 01:44:58 2018 New Revision: 1837624 URL: http://svn.apache.org/viewvc?rev=1837624=rev Log: Actually add the pages for CASSANDRA-14624 blog post Added: cassandra/site/publish/blog/ cassandra/site/publish/blog/2018/ cassandra/site/publish/blog/2018/08/ cassandra/site/publish/blog/2018/08/07/ cassandra/site/publish/blog/2018/08/07/faster_streaming_in_cassandra.html cassandra/site/publish/blog/index.html Added: cassandra/site/publish/blog/2018/08/07/faster_streaming_in_cassandra.html URL: http://svn.apache.org/viewvc/cassandra/site/publish/blog/2018/08/07/faster_streaming_in_cassandra.html?rev=1837624=auto == --- cassandra/site/publish/blog/2018/08/07/faster_streaming_in_cassandra.html (added) +++ cassandra/site/publish/blog/2018/08/07/faster_streaming_in_cassandra.html Wed Aug 8 01:44:58 2018 @@ -0,0 +1,214 @@ + + + + + + + + + + + + + + + + Hardware-bound Zero Copy Streaming in Apache Cassandra 4.0 + + http://cassandra.apache.org/blog/2018/08/07/faster_streaming_in_cassandra.html;> + + https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css; integrity="sha384-1q8mTJOASx8j1Au+a5WDVnPi2lkFfwwEAa8hDDdjZlpLegxhjVME1fgjWPGmkzs7" crossorigin="anonymous"> + + + + + + + + + + + + + + + Apache Software Foundation + +http://www.apache.org;>Apache Homepage +http://www.apache.org/licenses/;>License +http://www.apache.org/foundation/sponsorship.html;>Sponsorship +http://www.apache.org/foundation/thanks.html;>Thanks +http://www.apache.org/security/;>Security + + + + + + Apache Cassandra + + + + +Hardware-bound Zero Copy Streaming in Apache Cassandra 4.0 + + + + + + + + + + + + + + + Toggle navigation + + + + + + + + + + Home + Download + Documentation + Community + Blog + + + + + + + + + + + Hardware-bound Zero Copy Streaming in Apache Cassandra 4.0 +Posted on August 07, 2018 by The Apache Cassandra Community + Back to the Apache Cassandra Blog + + Streaming in Apache Cassandra powers host replacement, range movements, and cluster expansions. Streaming plays a crucial role in the cluster and as such its performance is key to not only the speed of the operations its used in but the clusterâs health generally. In Apache Cassandra 4.0, we have introduced an improved streaming implementation that reduces GC pressure and increases throughput several folds and are now limited, in some cases, only by the disk / network IO (See: https://issues.apache.org/jira/browse/CASSANDRA-14556;>CASSANDRA-14556). + + To get an understanding of the impact of these changes, letâs first have a look at the current streaming code path. The diagram below illustrates the stream session setup when a node attempts to stream data from a peer. Letâs say, we have a 3 node cluster (Nodes A, B, C). Node C is being rebuilt and has to stream all data that it is responsible for from A B. C setups a streaming session with each of itâs peers (See: https://issues.apache.org/jira/browse/CASSANDRA-4650;>CASSANDRA-4560 how Cassandra applies https://en.wikipedia.org/wiki/Ford%E2%80%93Fulkerson_algorithm;>Ford Fulkerson to optimize streaming peers). It exchanges messages to request ranges and begins streaming data from the selected nodes. + +During the streaming phase, A collects all SSTables that have partitions in the requested ranges. It streams each SSTable by serializing individual partitions. Upon receiving the partition, node C reifies the data in memory and then writes it to disk. This is necessary to accurately transfer partitions from all possible SSTables for the requested ranges. This streaming path generates garbage and could be avoided in scenarios where all partitions within the SSTable need to be transmitted. This is common when youâre using LeveledCompactionStrategy or have enabled partitioning SSTables by token range (See: http://issues.apache.org/jira/browse/CASSANDRA-6696;>CASSANDRA-6696), etc. + +To solve this problem http://issues.apache.org/jira/browse/CASSANDRA-14556;>CASSANDRA-14556 adds a Zero Copy streaming path. This significantly speeds up the transfer of SSTables and reduces garbage and unnecessary object creation. It modifies the streaming path to add additional information into the streaming header and uses ZeroCopy APIs to transfer bytes to and from the network and disk. So now, an SSTable may be transferred using this strategy when Cassandra detects that a complete SSTable needs to be transferred. + +How do I use this
svn commit: r1837623 [3/5] - in /cassandra/site: publish/ publish/community/ publish/css/ publish/doc/4.0/ publish/doc/4.0/architecture/ publish/doc/4.0/configuration/ publish/doc/4.0/cql/ publish/doc
Modified: cassandra/site/publish/doc/4.0/tools/nodetool/stop.html URL: http://svn.apache.org/viewvc/cassandra/site/publish/doc/4.0/tools/nodetool/stop.html?rev=1837623=1837622=1837623=diff == --- cassandra/site/publish/doc/4.0/tools/nodetool/stop.html (original) +++ cassandra/site/publish/doc/4.0/tools/nodetool/stop.html Wed Aug 8 01:37:04 2018 @@ -2,12 +2,6 @@ - - - - - - @@ -89,6 +83,7 @@ Download Documentation Community + Blog @@ -208,7 +203,9 @@ - + + + Modified: cassandra/site/publish/doc/4.0/tools/nodetool/stopdaemon.html URL: http://svn.apache.org/viewvc/cassandra/site/publish/doc/4.0/tools/nodetool/stopdaemon.html?rev=1837623=1837622=1837623=diff == --- cassandra/site/publish/doc/4.0/tools/nodetool/stopdaemon.html (original) +++ cassandra/site/publish/doc/4.0/tools/nodetool/stopdaemon.html Wed Aug 8 01:37:04 2018 @@ -2,12 +2,6 @@ - - - - - - @@ -89,6 +83,7 @@ Download Documentation Community + Blog @@ -192,7 +187,9 @@ - + + + Modified: cassandra/site/publish/doc/4.0/tools/nodetool/tablehistograms.html URL: http://svn.apache.org/viewvc/cassandra/site/publish/doc/4.0/tools/nodetool/tablehistograms.html?rev=1837623=1837622=1837623=diff == --- cassandra/site/publish/doc/4.0/tools/nodetool/tablehistograms.html (original) +++ cassandra/site/publish/doc/4.0/tools/nodetool/tablehistograms.html Wed Aug 8 01:37:04 2018 @@ -2,12 +2,6 @@ - - - - - - @@ -89,6 +83,7 @@ Download Documentation Community + Blog @@ -201,7 +196,9 @@ - + + + Modified: cassandra/site/publish/doc/4.0/tools/nodetool/tablestats.html URL: http://svn.apache.org/viewvc/cassandra/site/publish/doc/4.0/tools/nodetool/tablestats.html?rev=1837623=1837622=1837623=diff == --- cassandra/site/publish/doc/4.0/tools/nodetool/tablestats.html (original) +++ cassandra/site/publish/doc/4.0/tools/nodetool/tablestats.html Wed Aug 8 01:37:04 2018 @@ -2,12 +2,6 @@ - - - - - - @@ -89,6 +83,7 @@ Download Documentation Community + Blog @@ -211,7 +206,9 @@ - + + + Modified: cassandra/site/publish/doc/4.0/tools/nodetool/toppartitions.html URL: http://svn.apache.org/viewvc/cassandra/site/publish/doc/4.0/tools/nodetool/toppartitions.html?rev=1837623=1837622=1837623=diff == --- cassandra/site/publish/doc/4.0/tools/nodetool/toppartitions.html (original) +++ cassandra/site/publish/doc/4.0/tools/nodetool/toppartitions.html Wed Aug 8 01:37:04 2018 @@ -2,12 +2,6 @@ - - - - - - @@ -89,6 +83,7 @@ Download Documentation Community + Blog @@ -212,7 +207,9 @@ - + + + Modified: cassandra/site/publish/doc/4.0/tools/nodetool/tpstats.html URL: http://svn.apache.org/viewvc/cassandra/site/publish/doc/4.0/tools/nodetool/tpstats.html?rev=1837623=1837622=1837623=diff == --- cassandra/site/publish/doc/4.0/tools/nodetool/tpstats.html (original) +++ cassandra/site/publish/doc/4.0/tools/nodetool/tpstats.html Wed Aug 8 01:37:04 2018 @@ -2,12 +2,6 @@ - - - - - - @@ -89,6 +83,7 @@ Download Documentation Community + Blog @@ -196,7 +191,9 @@ - + + + Modified: cassandra/site/publish/doc/4.0/tools/nodetool/truncatehints.html URL: http://svn.apache.org/viewvc/cassandra/site/publish/doc/4.0/tools/nodetool/truncatehints.html?rev=1837623=1837622=1837623=diff == --- cassandra/site/publish/doc/4.0/tools/nodetool/truncatehints.html (original) +++ cassandra/site/publish/doc/4.0/tools/nodetool/truncatehints.html Wed Aug 8 01:37:04 2018 @@ -2,12 +2,6 @@ - - - - - - @@ -89,6 +83,7 @@ Download Documentation Community + Blog @@ -203,7 +198,9 @@ - + + + Modified: cassandra/site/publish/doc/4.0/tools/nodetool/upgradesstables.html URL:
svn commit: r1837623 [2/5] - in /cassandra/site: publish/ publish/community/ publish/css/ publish/doc/4.0/ publish/doc/4.0/architecture/ publish/doc/4.0/configuration/ publish/doc/4.0/cql/ publish/doc
Modified: cassandra/site/publish/doc/4.0/tools/index.html URL: http://svn.apache.org/viewvc/cassandra/site/publish/doc/4.0/tools/index.html?rev=1837623=1837622=1837623=diff == --- cassandra/site/publish/doc/4.0/tools/index.html (original) +++ cassandra/site/publish/doc/4.0/tools/index.html Wed Aug 8 01:37:04 2018 @@ -2,12 +2,6 @@ - - - - - - @@ -89,6 +83,7 @@ Download Documentation Community + Blog @@ -185,7 +180,9 @@ - + + + Modified: cassandra/site/publish/doc/4.0/tools/nodetool/assassinate.html URL: http://svn.apache.org/viewvc/cassandra/site/publish/doc/4.0/tools/nodetool/assassinate.html?rev=1837623=1837622=1837623=diff == --- cassandra/site/publish/doc/4.0/tools/nodetool/assassinate.html (original) +++ cassandra/site/publish/doc/4.0/tools/nodetool/assassinate.html Wed Aug 8 01:37:04 2018 @@ -2,12 +2,6 @@ - - - - - - @@ -89,6 +83,7 @@ Download Documentation Community + Blog @@ -201,7 +196,9 @@ - + + + Modified: cassandra/site/publish/doc/4.0/tools/nodetool/bootstrap.html URL: http://svn.apache.org/viewvc/cassandra/site/publish/doc/4.0/tools/nodetool/bootstrap.html?rev=1837623=1837622=1837623=diff == --- cassandra/site/publish/doc/4.0/tools/nodetool/bootstrap.html (original) +++ cassandra/site/publish/doc/4.0/tools/nodetool/bootstrap.html Wed Aug 8 01:37:04 2018 @@ -2,12 +2,6 @@ - - - - - - @@ -89,6 +83,7 @@ Download Documentation Community + Blog @@ -200,7 +195,9 @@ - + + + Modified: cassandra/site/publish/doc/4.0/tools/nodetool/cleanup.html URL: http://svn.apache.org/viewvc/cassandra/site/publish/doc/4.0/tools/nodetool/cleanup.html?rev=1837623=1837622=1837623=diff == --- cassandra/site/publish/doc/4.0/tools/nodetool/cleanup.html (original) +++ cassandra/site/publish/doc/4.0/tools/nodetool/cleanup.html Wed Aug 8 01:37:04 2018 @@ -2,12 +2,6 @@ - - - - - - @@ -89,6 +83,7 @@ Download Documentation Community + Blog @@ -206,7 +201,9 @@ - + + + Modified: cassandra/site/publish/doc/4.0/tools/nodetool/clearsnapshot.html URL: http://svn.apache.org/viewvc/cassandra/site/publish/doc/4.0/tools/nodetool/clearsnapshot.html?rev=1837623=1837622=1837623=diff == --- cassandra/site/publish/doc/4.0/tools/nodetool/clearsnapshot.html (original) +++ cassandra/site/publish/doc/4.0/tools/nodetool/clearsnapshot.html Wed Aug 8 01:37:04 2018 @@ -2,12 +2,6 @@ - - - - - - @@ -89,6 +83,7 @@ Download Documentation Community + Blog @@ -206,7 +201,9 @@ - + + + Modified: cassandra/site/publish/doc/4.0/tools/nodetool/compact.html URL: http://svn.apache.org/viewvc/cassandra/site/publish/doc/4.0/tools/nodetool/compact.html?rev=1837623=1837622=1837623=diff == --- cassandra/site/publish/doc/4.0/tools/nodetool/compact.html (original) +++ cassandra/site/publish/doc/4.0/tools/nodetool/compact.html Wed Aug 8 01:37:04 2018 @@ -2,12 +2,6 @@ - - - - - - @@ -89,6 +83,7 @@ Download Documentation Community + Blog @@ -218,7 +213,9 @@ - + + + Modified: cassandra/site/publish/doc/4.0/tools/nodetool/compactionhistory.html URL: http://svn.apache.org/viewvc/cassandra/site/publish/doc/4.0/tools/nodetool/compactionhistory.html?rev=1837623=1837622=1837623=diff == --- cassandra/site/publish/doc/4.0/tools/nodetool/compactionhistory.html (original) +++ cassandra/site/publish/doc/4.0/tools/nodetool/compactionhistory.html Wed Aug 8 01:37:04 2018 @@ -2,12 +2,6 @@ - - - - - - @@ -89,6 +83,7 @@ Download Documentation Community + Blog @@ -196,7 +191,9 @@ - + + + Modified: cassandra/site/publish/doc/4.0/tools/nodetool/compactionstats.html URL: http://svn.apache.org/viewvc/cassandra/site/publish/doc/4.0/tools/nodetool/compactionstats.html?rev=1837623=1837622=1837623=diff
svn commit: r1837623 [5/5] - in /cassandra/site: publish/ publish/community/ publish/css/ publish/doc/4.0/ publish/doc/4.0/architecture/ publish/doc/4.0/configuration/ publish/doc/4.0/cql/ publish/doc
Added: cassandra/site/src/_posts/2018-08-06-faster_streaming_in_cassandra.markdown URL: http://svn.apache.org/viewvc/cassandra/site/src/_posts/2018-08-06-faster_streaming_in_cassandra.markdown?rev=1837623=auto == --- cassandra/site/src/_posts/2018-08-06-faster_streaming_in_cassandra.markdown (added) +++ cassandra/site/src/_posts/2018-08-06-faster_streaming_in_cassandra.markdown Wed Aug 8 01:37:04 2018 @@ -0,0 +1,77 @@ +--- +layout: post +title: "Hardware-bound Zero Copy Streaming in Apache Cassandra 4.0" +date: 2018-08-06 12:00:00 -0700 +author: The Apache Cassandra Community +categories: blog +--- + +Streaming in Apache Cassandra powers host replacement, range movements, and cluster expansions. Streaming plays a crucial role in the cluster and as such its performance is key to not only the speed of the operations its used in but the cluster's health generally. In Apache Cassandra 4.0, we have introduced an improved streaming implementation that reduces GC pressure and increases throughput several folds and are now limited, in some cases, only by the disk / network IO (See: [CASSANDRA-14556](https://issues.apache.org/jira/browse/CASSANDRA-14556)). + + +++ cassandra/site/publish/doc/latest/tools/nodetool/drain.html Wed Aug 8 01:37:04 2018 @@ -2,12 +2,6 @@ - - - - - - @@ -89,6 +83,7 @@ Download Documentation Community + Blog @@ -193,7 +188,9 @@ - + + + Modified: cassandra/site/publish/doc/latest/tools/nodetool/enableautocompaction.html URL: http://svn.apache.org/viewvc/cassandra/site/publish/doc/latest/tools/nodetool/enableautocompaction.html?rev=1837623=1837622=1837623=diff == --- cassandra/site/publish/doc/latest/tools/nodetool/enableautocompaction.html (original) +++ cassandra/site/publish/doc/latest/tools/nodetool/enableautocompaction.html Wed Aug 8 01:37:04 2018 @@ -2,12 +2,6 @@ - - - - - - @@ -89,6 +83,7 @@ Download Documentation Community + Blog @@ -202,7 +197,9 @@ - + + + Modified: cassandra/site/publish/doc/latest/tools/nodetool/enablebackup.html URL: http://svn.apache.org/viewvc/cassandra/site/publish/doc/latest/tools/nodetool/enablebackup.html?rev=1837623=1837622=1837623=diff == --- cassandra/site/publish/doc/latest/tools/nodetool/enablebackup.html (original) +++ cassandra/site/publish/doc/latest/tools/nodetool/enablebackup.html Wed Aug 8 01:37:04 2018 @@ -2,12 +2,6 @@ - - - - - - @@ -89,6 +83,7 @@ Download Documentation Community + Blog @@ -192,7 +187,9 @@ - + + + Modified: cassandra/site/publish/doc/latest/tools/nodetool/enablebinary.html URL: http://svn.apache.org/viewvc/cassandra/site/publish/doc/latest/tools/nodetool/enablebinary.html?rev=1837623=1837622=1837623=diff == --- cassandra/site/publish/doc/latest/tools/nodetool/enablebinary.html (original) +++ cassandra/site/publish/doc/latest/tools/nodetool/enablebinary.html Wed Aug 8 01:37:04 2018 @@ -2,12 +2,6 @@ - - - - - - @@ -89,6 +83,7 @@ Download Documentation Community + Blog @@ -192,7 +187,9 @@ - + + + Modified: cassandra/site/publish/doc/latest/tools/nodetool/enablegossip.html URL: http://svn.apache.org/viewvc/cassandra/site/publish/doc/latest/tools/nodetool/enablegossip.html?rev=1837623=1837622=1837623=diff == --- cassandra/site/publish/doc/latest/tools/nodetool/enablegossip.html (original) +++ cassandra/site/publish/doc/latest/tools/nodetool/enablegossip.html Wed Aug 8 01:37:04 2018 @@ -2,12 +2,6 @@ - - - - - - @@ -89,6 +83,7 @@ Download Documentation Community + Blog @@ -192,7 +187,9 @@ - + + + Modified: cassandra/site/publish/doc/latest/tools/nodetool/enablehandoff.html URL: http://svn.apache.org/viewvc/cassandra/site/publish/doc/latest/tools/nodetool/enablehandoff.html?rev=1837623=1837622=1837623=diff == --- cassandra/site/publish/doc/latest/tools/nodetool/enablehandoff.html (original) +++ cassandra/site/publish/doc/latest/tools/nodetool/enablehandoff.html Wed Aug 8 01:37:04 2018 @@ -2,12 +2,6 @@ - - - - - - @@ -89,6 +83,7 @@ Download Documentation Community + Blog @@ -193,7 +188,9 @@ - + + + Modified: cassandra/site/publish/doc/latest/tools/nodetool/enablehintsfordc.html URL: http://svn.apache.org/viewvc/cassandra/site/publish/doc/latest/tools/nodetool/enablehintsfordc.html?rev=1837623=1837622=1837623=diff == --- cassandra/site/publish/doc/latest/tools/nodetool/enablehintsfordc.html (original) +++ cassandra/site/publish/doc/latest/tools/nodetool/enablehintsfordc.html Wed Aug 8 01:37:04 2018 @@ -2,12 +2,6 @@ - - - - - - @@ -89,6 +83,7 @@ Download Documentation Community + Blog @@ -202,7 +197,9 @@ - + + + Modified:
svn commit: r1837623 [1/5] - in /cassandra/site: publish/ publish/community/ publish/css/ publish/doc/4.0/ publish/doc/4.0/architecture/ publish/doc/4.0/configuration/ publish/doc/4.0/cql/ publish/doc
Author: zznate Date: Wed Aug 8 01:37:04 2018 New Revision: 1837623 URL: http://svn.apache.org/viewvc?rev=1837623=rev Log: Add first blog post and run publish for such as part of CASSANDRA-14624 Added: cassandra/site/src/_posts/ cassandra/site/src/_posts/2018-08-06-faster_streaming_in_cassandra.markdown Modified: cassandra/site/publish/community/index.html cassandra/site/publish/css/style.css cassandra/site/publish/doc/4.0/architecture/dynamo.html cassandra/site/publish/doc/4.0/architecture/guarantees.html cassandra/site/publish/doc/4.0/architecture/index.html cassandra/site/publish/doc/4.0/architecture/overview.html cassandra/site/publish/doc/4.0/architecture/storage_engine.html cassandra/site/publish/doc/4.0/bugs.html cassandra/site/publish/doc/4.0/configuration/cassandra_config_file.html cassandra/site/publish/doc/4.0/configuration/index.html cassandra/site/publish/doc/4.0/contactus.html cassandra/site/publish/doc/4.0/cql/appendices.html cassandra/site/publish/doc/4.0/cql/changes.html cassandra/site/publish/doc/4.0/cql/ddl.html cassandra/site/publish/doc/4.0/cql/definitions.html cassandra/site/publish/doc/4.0/cql/dml.html cassandra/site/publish/doc/4.0/cql/functions.html cassandra/site/publish/doc/4.0/cql/index.html cassandra/site/publish/doc/4.0/cql/indexes.html cassandra/site/publish/doc/4.0/cql/json.html cassandra/site/publish/doc/4.0/cql/mvs.html cassandra/site/publish/doc/4.0/cql/operators.html cassandra/site/publish/doc/4.0/cql/security.html cassandra/site/publish/doc/4.0/cql/triggers.html cassandra/site/publish/doc/4.0/cql/types.html cassandra/site/publish/doc/4.0/data_modeling/index.html cassandra/site/publish/doc/4.0/development/code_style.html cassandra/site/publish/doc/4.0/development/documentation.html cassandra/site/publish/doc/4.0/development/gettingstarted.html cassandra/site/publish/doc/4.0/development/how_to_commit.html cassandra/site/publish/doc/4.0/development/how_to_review.html cassandra/site/publish/doc/4.0/development/ide.html cassandra/site/publish/doc/4.0/development/index.html cassandra/site/publish/doc/4.0/development/patches.html cassandra/site/publish/doc/4.0/development/testing.html cassandra/site/publish/doc/4.0/faq/index.html cassandra/site/publish/doc/4.0/getting_started/configuring.html cassandra/site/publish/doc/4.0/getting_started/drivers.html cassandra/site/publish/doc/4.0/getting_started/index.html cassandra/site/publish/doc/4.0/getting_started/installing.html cassandra/site/publish/doc/4.0/getting_started/querying.html cassandra/site/publish/doc/4.0/index.html cassandra/site/publish/doc/4.0/operating/backups.html cassandra/site/publish/doc/4.0/operating/bloom_filters.html cassandra/site/publish/doc/4.0/operating/bulk_loading.html cassandra/site/publish/doc/4.0/operating/cdc.html cassandra/site/publish/doc/4.0/operating/compaction.html cassandra/site/publish/doc/4.0/operating/compression.html cassandra/site/publish/doc/4.0/operating/hardware.html cassandra/site/publish/doc/4.0/operating/hints.html cassandra/site/publish/doc/4.0/operating/index.html cassandra/site/publish/doc/4.0/operating/metrics.html cassandra/site/publish/doc/4.0/operating/read_repair.html cassandra/site/publish/doc/4.0/operating/repair.html cassandra/site/publish/doc/4.0/operating/security.html cassandra/site/publish/doc/4.0/operating/snitch.html cassandra/site/publish/doc/4.0/operating/topo_changes.html cassandra/site/publish/doc/4.0/search.html cassandra/site/publish/doc/4.0/tools/cassandra_stress.html cassandra/site/publish/doc/4.0/tools/cqlsh.html cassandra/site/publish/doc/4.0/tools/index.html cassandra/site/publish/doc/4.0/tools/nodetool/assassinate.html cassandra/site/publish/doc/4.0/tools/nodetool/bootstrap.html cassandra/site/publish/doc/4.0/tools/nodetool/cleanup.html cassandra/site/publish/doc/4.0/tools/nodetool/clearsnapshot.html cassandra/site/publish/doc/4.0/tools/nodetool/compact.html cassandra/site/publish/doc/4.0/tools/nodetool/compactionhistory.html cassandra/site/publish/doc/4.0/tools/nodetool/compactionstats.html cassandra/site/publish/doc/4.0/tools/nodetool/decommission.html cassandra/site/publish/doc/4.0/tools/nodetool/describecluster.html cassandra/site/publish/doc/4.0/tools/nodetool/describering.html cassandra/site/publish/doc/4.0/tools/nodetool/disableautocompaction.html cassandra/site/publish/doc/4.0/tools/nodetool/disablebackup.html cassandra/site/publish/doc/4.0/tools/nodetool/disablebinary.html cassandra/site/publish/doc/4.0/tools/nodetool/disablegossip.html cassandra/site/publish/doc/4.0/tools/nodetool/disablehandoff.html cassandra/site/publish/doc/4.0/tools/nodetool/disablehintsfordc.html cassandra/site/publish/doc/4.0/tools/nodetool/drain.html
[jira] [Resolved] (CASSANDRA-14624) Website: Add blog post for Faster Streaming
[ https://issues.apache.org/jira/browse/CASSANDRA-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nate McCall resolved CASSANDRA-14624. - Resolution: Fixed > Website: Add blog post for Faster Streaming > --- > > Key: CASSANDRA-14624 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14624 > Project: Cassandra > Issue Type: Task > Components: Documentation and Website >Reporter: Dinesh Joshi >Assignee: Dinesh Joshi >Priority: Major > Attachments: faster-streaming-blog.patch > > > Please add a new blog post entry on the Cassandra website. It describes the > recent work on performance optimizations in Cassandra related to streaming > enhancements. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14624) Website: Add blog post for Faster Streaming
[ https://issues.apache.org/jira/browse/CASSANDRA-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572547#comment-16572547 ] Nate McCall commented on CASSANDRA-14624: - I can verify this publishes and looks as it should locally. Adding to SVN and publishing (I think - we'll see). > Website: Add blog post for Faster Streaming > --- > > Key: CASSANDRA-14624 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14624 > Project: Cassandra > Issue Type: Task > Components: Documentation and Website >Reporter: Dinesh Joshi >Assignee: Dinesh Joshi >Priority: Major > Attachments: faster-streaming-blog.patch > > > Please add a new blog post entry on the Cassandra website. It describes the > recent work on performance optimizations in Cassandra related to streaming > enhancements. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13457) Diag. Events: Add base classes
[
https://issues.apache.org/jira/browse/CASSANDRA-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572448#comment-16572448
]
Jason Brown commented on CASSANDRA-13457:
-
I was able to scrape a little bit of time to give this patch a semi-decent
read, but this is not a full, formal review. I'm leaving that to
[~michaelsembwever] and any others, but I have some comments/questions below.
First, though, to [~spo...@gmail.com]'s question: I'm largely fine with these
changes wrt runtime impact when the diagnostic events are disabled.
- Config.diagnostic_events_enabled needs to be volatile if you will modify it
at runtime via JMX.
- nit: TokenAllocatorEvent's ctor has a huge parameter list. Consider adding a
Builder as it's the pattern we are gnerally moving to, although I'm not sure
that's actually documented or communicated anywhere. HintEvent and SchemaEvent,
as well.
- DiagnosticEventService.publish() is invoked on the same thread as the caller.
What happens if one of the consumers slows down (because it relies on some kind
of blocking IO, disk, network, whatever). In the case of gossip that have
fairly bad consequences.
- Please document why the map's value type of the return from
DiagnosticEvent.toMap() is Serializable. (I looked at CASSANDRA-14435, and it's
to support transferring the maps via JMX). However, before I saw that ticket I
was quite surprised as to why Serializable as I usually assume "java object
serialization".
- I'm not sure what to think about DiagnosticEvent.getSource(). It's an Object,
which is rather non-descript, type-wise. What is the intention here? Do some of
the follow up tickets use this method? I assume they need to know
implementation details of the specific DiagnosticEvent implementation in order
to figure out qhat the type od the source is.
- Gossiper - I would rather have callers ask for immutable copies, via getter
methods, of essential gossip data structures, rather than change the visibility
of those structures. Directly shared those mutatble data structures makes me
very nervous.
- You've added toString() implementations to about 10 classes. Are those for
debugging help? If so, ignore the rest of this bullet point. If it's for
diagnostic events stuff, I'm worried that the use of toString() now ties any
debugging information or formatting we might want to add is now is bound to any
diagnostic consumers consumers - meaning, we couldn't be sure that in modifying
a class's {{toString()}} we wouldn't be breaking some consumer's logic.
- DiagnosticEvent uses millis for for the timestamp, and in particular,
System.currentTimeMillis(). That method is not guaranteed to be monotonic (nor
are wall clocks). I generally advise against using wall-clock timestamps, but
what are the requirements here?
> Diag. Events: Add base classes
> --
>
> Key: CASSANDRA-13457
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13457
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core, Observability
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
>
> Base ticket for adding classes that will allow you to implement and subscribe
> to events.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14557) Consider adding default and required keyspace replication options
[
https://issues.apache.org/jira/browse/CASSANDRA-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572332#comment-16572332
]
Jaydeepkumar Chovatia commented on CASSANDRA-14557:
---
LGTM
> Consider adding default and required keyspace replication options
> -
>
> Key: CASSANDRA-14557
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14557
> Project: Cassandra
> Issue Type: Improvement
> Components: Configuration
>Reporter: Sumanth Pasupuleti
>Assignee: Sumanth Pasupuleti
>Priority: Minor
> Labels: 4.0-feature-freeze-review-requested
> Fix For: 4.0
>
> Attachments: 14557-trunk.txt
>
>
> Ending up with a keyspace of RF=1 is unfortunately pretty easy in C* right
> now - the system_auth table for example is created with RF=1 (to take into
> account single node setups afaict from CASSANDRA-5112), and a user can
> further create a keyspace with RF=1 posing availability and streaming risks
> (e.g. rebuild).
> I propose we add two configuration options in cassandra.yaml:
> # {{default_keyspace_rf}} (default: 1) - If replication factors are not
> specified, use this number.
> # {{required_minimum_keyspace_rf}} (default: unset) - Prevent users from
> creating a keyspace with an RF less than what is configured
> These settings could further be re-used to:
> * Provide defaults for new keyspaces created with SimpleStrategy or
> NetworkTopologyStrategy (CASSANDRA-14303)
> * Make the automatic token [allocation
> algorithm|https://issues.apache.org/jira/browse/CASSANDRA-13701?focusedCommentId=16095662=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16095662]
> interface more intuitive allowing easy use of the new token allocation
> algorithm.
> At the end of the day, if someone really wants to allow RF=1, they simply
> don’t set the setting. For backwards compatibility the default remains 1 and
> C* would create with RF=1, and would default to current behavior of allowing
> any RF on keyspaces.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14557) Consider adding default and required keyspace replication options
[
https://issues.apache.org/jira/browse/CASSANDRA-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572163#comment-16572163
]
Sumanth Pasupuleti edited comment on CASSANDRA-14557 at 8/7/18 8:06 PM:
Thanks for reviewing [~chovatia.jayd...@gmail.com]
{quote}Don't we have to apply minimum keyspace RF check for system
keyspaces?{quote}
System keyspace creation *does* go through the minimum keyspace rf check. So,
if someone configures default_keyspace_rf = 1 and minimum_keyspace_rf=2,
Cassandra fails to start, since system keyspace creation fails because of the
check.
{quote}Think about renaming parsedRf to something more meaningful maybe
userSuppliedRF:{quote}
It could be both a user/system passed rf. Method parameter name has not been
refactored (rf) as part of this ticket. Named this as parsedRf because we are
parsing rf to int, and we wanted to avoid having to parse to int multiple times
in the code.
{quote} Have a consistent way of formatting string in Exception message{quote}
Fixed; pushed a commit
[7ee2827a3fd6a8f1550d9c0eec8f039b6ad6c74c|https://github.com/sumanth-pasupuleti/cassandra/commit/7ee2827a3fd6a8f1550d9c0eec8f039b6ad6c74c]
Updated attached patch file
was (Author: sumanth.pasupuleti):
Thanks for reviewing [~chovatia.jayd...@gmail.com]
{quote}Don't we have to apply minimum keyspace RF check for system
keyspaces?{quote}
System keyspace creation *does* go through the minimum keyspace rf check. So,
if someone configures default_keyspace_rf = 1 and minimum_keyspace_rf=2,
Cassandra fails to start, since system keyspace creation fails because of the
check.
{quote}Think about renaming parsedRf to something more meaningful maybe
userSuppliedRF:{quote}
It could be both a user/system passed rf. Method parameter name has not been
refactored (rf) as part of this ticket. Named this as parsedRf because we are
parsing rf to int, and we wanted to avoid having to parse to int multiple times
in the code.
{quote} Have a consistent way of formatting string in Exception message{quote}
Fixed; pushed a commit
[7ee2827a3fd6a8f1550d9c0eec8f039b6ad6c74c|https://github.com/sumanth-pasupuleti/cassandra/commit/7ee2827a3fd6a8f1550d9c0eec8f039b6ad6c74c]
> Consider adding default and required keyspace replication options
> -
>
> Key: CASSANDRA-14557
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14557
> Project: Cassandra
> Issue Type: Improvement
> Components: Configuration
>Reporter: Sumanth Pasupuleti
>Assignee: Sumanth Pasupuleti
>Priority: Minor
> Labels: 4.0-feature-freeze-review-requested
> Fix For: 4.0
>
> Attachments: 14557-trunk.txt
>
>
> Ending up with a keyspace of RF=1 is unfortunately pretty easy in C* right
> now - the system_auth table for example is created with RF=1 (to take into
> account single node setups afaict from CASSANDRA-5112), and a user can
> further create a keyspace with RF=1 posing availability and streaming risks
> (e.g. rebuild).
> I propose we add two configuration options in cassandra.yaml:
> # {{default_keyspace_rf}} (default: 1) - If replication factors are not
> specified, use this number.
> # {{required_minimum_keyspace_rf}} (default: unset) - Prevent users from
> creating a keyspace with an RF less than what is configured
> These settings could further be re-used to:
> * Provide defaults for new keyspaces created with SimpleStrategy or
> NetworkTopologyStrategy (CASSANDRA-14303)
> * Make the automatic token [allocation
> algorithm|https://issues.apache.org/jira/browse/CASSANDRA-13701?focusedCommentId=16095662=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16095662]
> interface more intuitive allowing easy use of the new token allocation
> algorithm.
> At the end of the day, if someone really wants to allow RF=1, they simply
> don’t set the setting. For backwards compatibility the default remains 1 and
> C* would create with RF=1, and would default to current behavior of allowing
> any RF on keyspaces.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14557) Consider adding default and required keyspace replication options
[
https://issues.apache.org/jira/browse/CASSANDRA-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sumanth Pasupuleti updated CASSANDRA-14557:
---
Attachment: 14557-trunk.txt
> Consider adding default and required keyspace replication options
> -
>
> Key: CASSANDRA-14557
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14557
> Project: Cassandra
> Issue Type: Improvement
> Components: Configuration
>Reporter: Sumanth Pasupuleti
>Assignee: Sumanth Pasupuleti
>Priority: Minor
> Labels: 4.0-feature-freeze-review-requested
> Fix For: 4.0
>
> Attachments: 14557-trunk.txt
>
>
> Ending up with a keyspace of RF=1 is unfortunately pretty easy in C* right
> now - the system_auth table for example is created with RF=1 (to take into
> account single node setups afaict from CASSANDRA-5112), and a user can
> further create a keyspace with RF=1 posing availability and streaming risks
> (e.g. rebuild).
> I propose we add two configuration options in cassandra.yaml:
> # {{default_keyspace_rf}} (default: 1) - If replication factors are not
> specified, use this number.
> # {{required_minimum_keyspace_rf}} (default: unset) - Prevent users from
> creating a keyspace with an RF less than what is configured
> These settings could further be re-used to:
> * Provide defaults for new keyspaces created with SimpleStrategy or
> NetworkTopologyStrategy (CASSANDRA-14303)
> * Make the automatic token [allocation
> algorithm|https://issues.apache.org/jira/browse/CASSANDRA-13701?focusedCommentId=16095662=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16095662]
> interface more intuitive allowing easy use of the new token allocation
> algorithm.
> At the end of the day, if someone really wants to allow RF=1, they simply
> don’t set the setting. For backwards compatibility the default remains 1 and
> C* would create with RF=1, and would default to current behavior of allowing
> any RF on keyspaces.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14557) Consider adding default and required keyspace replication options
[
https://issues.apache.org/jira/browse/CASSANDRA-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sumanth Pasupuleti updated CASSANDRA-14557:
---
Attachment: (was: 14557-trunk.txt)
> Consider adding default and required keyspace replication options
> -
>
> Key: CASSANDRA-14557
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14557
> Project: Cassandra
> Issue Type: Improvement
> Components: Configuration
>Reporter: Sumanth Pasupuleti
>Assignee: Sumanth Pasupuleti
>Priority: Minor
> Labels: 4.0-feature-freeze-review-requested
> Fix For: 4.0
>
> Attachments: 14557-trunk.txt
>
>
> Ending up with a keyspace of RF=1 is unfortunately pretty easy in C* right
> now - the system_auth table for example is created with RF=1 (to take into
> account single node setups afaict from CASSANDRA-5112), and a user can
> further create a keyspace with RF=1 posing availability and streaming risks
> (e.g. rebuild).
> I propose we add two configuration options in cassandra.yaml:
> # {{default_keyspace_rf}} (default: 1) - If replication factors are not
> specified, use this number.
> # {{required_minimum_keyspace_rf}} (default: unset) - Prevent users from
> creating a keyspace with an RF less than what is configured
> These settings could further be re-used to:
> * Provide defaults for new keyspaces created with SimpleStrategy or
> NetworkTopologyStrategy (CASSANDRA-14303)
> * Make the automatic token [allocation
> algorithm|https://issues.apache.org/jira/browse/CASSANDRA-13701?focusedCommentId=16095662=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16095662]
> interface more intuitive allowing easy use of the new token allocation
> algorithm.
> At the end of the day, if someone really wants to allow RF=1, they simply
> don’t set the setting. For backwards compatibility the default remains 1 and
> C* would create with RF=1, and would default to current behavior of allowing
> any RF on keyspaces.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-7066) Simplify (and unify) cleanup of compaction leftovers
[ https://issues.apache.org/jira/browse/CASSANDRA-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tania S Engel updated CASSANDRA-7066: - Attachment: Hide.url > Simplify (and unify) cleanup of compaction leftovers > > > Key: CASSANDRA-7066 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7066 > Project: Cassandra > Issue Type: Improvement > Components: Local Write-Read Paths >Reporter: Benedict >Assignee: Stefania >Priority: Minor > Labels: benedict-to-commit, compaction > Fix For: 3.0 alpha 1 > > Attachments: 7066.txt, Hide.url > > > Currently we manage a list of in-progress compactions in a system table, > which we use to cleanup incomplete compactions when we're done. The problem > with this is that 1) it's a bit clunky (and leaves us in positions where we > can unnecessarily cleanup completed files, or conversely not cleanup files > that have been superceded); and 2) it's only used for a regular compaction - > no other compaction types are guarded in the same way, so can result in > duplication if we fail before deleting the replacements. > I'd like to see each sstable store in its metadata its direct ancestors, and > on startup we simply delete any sstables that occur in the union of all > ancestor sets. This way as soon as we finish writing we're capable of > cleaning up any leftovers, so we never get duplication. It's also much easier > to reason about. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13304) Add checksumming to the native protocol
[
https://issues.apache.org/jira/browse/CASSANDRA-13304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572196#comment-16572196
]
Tom van der Woerdt edited comment on CASSANDRA-13304 at 8/7/18 7:35 PM:
My opinion on this hasn't changed: this kind of thing should be handled on a
lower level than the application protocol. TLS with a NULL cipher will achieve
the same thing, with significantly less code, at virtually no performance
overhead (because it disables the "heavy" crypto bits of TLS).
We've been running Cassandra in production, also with hundreds of billions of
requests per day, and thanks to TLS never had to worry about bit flips.
The IETF writes ([https://tools.ietf.org/html/rfc5246):]
{quote}The primary goal of the TLS protocol is to provide privacy and data
integrity between two communicating applications
{quote}
Data integrity is a primary goal of TLS: the two are not orthogonal, it's a
design goal.
wrt your patch: I don't know about other drivers, but I can tell you about the
driver I maintain. It disables compression for any frame less than a few
hundred bytes, as it's more likely to hurt performance than benefit it.
The patch also doesn't cover the stream ID, which is a field in which a bitflip
can cause significant harm. In my opinion any patch to add checksumming should
cover at least the stream ID.
Hope that helps!
was (Author: tvdw):
My opinion on this hasn't changed: this kind of thing should be handled on a
lower level than the application protocol. TLS with a NULL cipher will achieve
the same thing, with significantly less code, at virtually no performance
overhead (because it disables the "heavy" crypto bits of TLS).
We've been running Cassandra in production, also with hundreds of billions of
requests per day, and thanks to TLS never had to worry about bit flips.
The IETF writes (https://tools.ietf.org/html/rfc5246):
The primary goal of the TLS protocol is to provide privacy and data
integrity between two communicating applications
Data integrity is a primary goal of TLS: the two are not orthogonal, it's a
design goal.
wrt your patch: I don't know about other drivers, but I can tell you about the
driver I maintain. It disables compression for any frame less than a few
hundred bytes, as it's more likely to hurt performance than benefit it.
The patch also doesn't cover the stream ID, which is a field in which a bitflip
can cause significant harm. In my opinion any patch to add checksumming should
cover at least the stream ID.
Hope that helps!
> Add checksumming to the native protocol
> ---
>
> Key: CASSANDRA-13304
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13304
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Michael Kjellman
>Assignee: Sam Tunnicliffe
>Priority: Blocker
> Labels: client-impacting
> Fix For: 4.x
>
> Attachments: 13304_v1.diff, boxplot-read-throughput.png,
> boxplot-write-throughput.png
>
>
> The native binary transport implementation doesn't include checksums. This
> makes it highly susceptible to silently inserting corrupted data either due
> to hardware issues causing bit flips on the sender/client side, C*/receiver
> side, or network in between.
> Attaching an implementation that makes checksum'ing mandatory (assuming both
> client and server know about a protocol version that supports checksums) --
> and also adds checksumming to clients that request compression.
> The serialized format looks something like this:
> {noformat}
> * 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
> * 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Number of Compressed Chunks | Compressed Length (e1)/
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * / Compressed Length cont. (e1) |Uncompressed Length (e1) /
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length cont. (e1)| CRC32 Checksum of Lengths (e1)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Checksum of Lengths cont. (e1)|Compressed Bytes (e1)+//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e1) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (e2)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> *
[jira] [Commented] (CASSANDRA-13304) Add checksumming to the native protocol
[
https://issues.apache.org/jira/browse/CASSANDRA-13304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572196#comment-16572196
]
Tom van der Woerdt commented on CASSANDRA-13304:
My opinion on this hasn't changed: this kind of thing should be handled on a
lower level than the application protocol. TLS with a NULL cipher will achieve
the same thing, with significantly less code, at virtually no performance
overhead (because it disables the "heavy" crypto bits of TLS).
We've been running Cassandra in production, also with hundreds of billions of
requests per day, and thanks to TLS never had to worry about bit flips.
The IETF writes (https://tools.ietf.org/html/rfc5246):
The primary goal of the TLS protocol is to provide privacy and data
integrity between two communicating applications
Data integrity is a primary goal of TLS: the two are not orthogonal, it's a
design goal.
wrt your patch: I don't know about other drivers, but I can tell you about the
driver I maintain. It disables compression for any frame less than a few
hundred bytes, as it's more likely to hurt performance than benefit it.
The patch also doesn't cover the stream ID, which is a field in which a bitflip
can cause significant harm. In my opinion any patch to add checksumming should
cover at least the stream ID.
Hope that helps!
> Add checksumming to the native protocol
> ---
>
> Key: CASSANDRA-13304
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13304
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Michael Kjellman
>Assignee: Sam Tunnicliffe
>Priority: Blocker
> Labels: client-impacting
> Fix For: 4.x
>
> Attachments: 13304_v1.diff, boxplot-read-throughput.png,
> boxplot-write-throughput.png
>
>
> The native binary transport implementation doesn't include checksums. This
> makes it highly susceptible to silently inserting corrupted data either due
> to hardware issues causing bit flips on the sender/client side, C*/receiver
> side, or network in between.
> Attaching an implementation that makes checksum'ing mandatory (assuming both
> client and server know about a protocol version that supports checksums) --
> and also adds checksumming to clients that request compression.
> The serialized format looks something like this:
> {noformat}
> * 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
> * 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Number of Compressed Chunks | Compressed Length (e1)/
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * / Compressed Length cont. (e1) |Uncompressed Length (e1) /
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length cont. (e1)| CRC32 Checksum of Lengths (e1)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Checksum of Lengths cont. (e1)|Compressed Bytes (e1)+//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e1) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (e2)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |CRC32 Checksum of Lengths (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Compressed Bytes (e2) +//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e2) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (en) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (en)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |CRC32 Checksum of Lengths (en) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Compressed Bytes (en) +//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (en) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> {noformat}
> The first pass here adds checksums only to the actual contents of the frame
> body itself
[jira] [Commented] (CASSANDRA-13304) Add checksumming to the native protocol
[
https://issues.apache.org/jira/browse/CASSANDRA-13304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572177#comment-16572177
]
Michael Kjellman commented on CASSANDRA-13304:
--
we’ve got almost a year of testing on the current code with hundreds of
billions of requests and we even caught (and protected against) a host flipping
bits in the process..
> Add checksumming to the native protocol
> ---
>
> Key: CASSANDRA-13304
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13304
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Michael Kjellman
>Assignee: Sam Tunnicliffe
>Priority: Blocker
> Labels: client-impacting
> Fix For: 4.x
>
> Attachments: 13304_v1.diff, boxplot-read-throughput.png,
> boxplot-write-throughput.png
>
>
> The native binary transport implementation doesn't include checksums. This
> makes it highly susceptible to silently inserting corrupted data either due
> to hardware issues causing bit flips on the sender/client side, C*/receiver
> side, or network in between.
> Attaching an implementation that makes checksum'ing mandatory (assuming both
> client and server know about a protocol version that supports checksums) --
> and also adds checksumming to clients that request compression.
> The serialized format looks something like this:
> {noformat}
> * 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
> * 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Number of Compressed Chunks | Compressed Length (e1)/
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * / Compressed Length cont. (e1) |Uncompressed Length (e1) /
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length cont. (e1)| CRC32 Checksum of Lengths (e1)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Checksum of Lengths cont. (e1)|Compressed Bytes (e1)+//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e1) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (e2)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |CRC32 Checksum of Lengths (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Compressed Bytes (e2) +//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e2) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (en) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (en)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |CRC32 Checksum of Lengths (en) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Compressed Bytes (en) +//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (en) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> {noformat}
> The first pass here adds checksums only to the actual contents of the frame
> body itself (and doesn't actually checksum lengths and headers). While it
> would be great to fully add checksuming across the entire protocol, the
> proposed implementation will ensure we at least catch corrupted data and
> likely protect ourselves pretty well anyways.
> I didn't go to the trouble of implementing a Snappy Checksum'ed Compressor
> implementation as it's been deprecated for a while -- is really slow and
> crappy compared to LZ4 -- and we should do everything in our power to make
> sure no one in the community is still using it. I left it in (for obvious
> backwards compatibility aspects) old for clients that don't know about the
> new protocol.
> The current protocol has a 256MB (max) frame body -- where the serialized
> contents are simply written in to the frame body.
> If the client sends a compression option in the startup, we will install a
> FrameCompressor inline. Unfortunately, we went with a decision to treat the
> frame body separately
[jira] [Commented] (CASSANDRA-13304) Add checksumming to the native protocol
[
https://issues.apache.org/jira/browse/CASSANDRA-13304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572174#comment-16572174
]
Michael Kjellman commented on CASSANDRA-13304:
--
and the last patch i did already made it possible to opt out...
> Add checksumming to the native protocol
> ---
>
> Key: CASSANDRA-13304
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13304
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Michael Kjellman
>Assignee: Sam Tunnicliffe
>Priority: Blocker
> Labels: client-impacting
> Fix For: 4.x
>
> Attachments: 13304_v1.diff, boxplot-read-throughput.png,
> boxplot-write-throughput.png
>
>
> The native binary transport implementation doesn't include checksums. This
> makes it highly susceptible to silently inserting corrupted data either due
> to hardware issues causing bit flips on the sender/client side, C*/receiver
> side, or network in between.
> Attaching an implementation that makes checksum'ing mandatory (assuming both
> client and server know about a protocol version that supports checksums) --
> and also adds checksumming to clients that request compression.
> The serialized format looks something like this:
> {noformat}
> * 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
> * 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Number of Compressed Chunks | Compressed Length (e1)/
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * / Compressed Length cont. (e1) |Uncompressed Length (e1) /
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length cont. (e1)| CRC32 Checksum of Lengths (e1)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Checksum of Lengths cont. (e1)|Compressed Bytes (e1)+//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e1) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (e2)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |CRC32 Checksum of Lengths (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Compressed Bytes (e2) +//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e2) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (en) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (en)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |CRC32 Checksum of Lengths (en) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Compressed Bytes (en) +//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (en) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> {noformat}
> The first pass here adds checksums only to the actual contents of the frame
> body itself (and doesn't actually checksum lengths and headers). While it
> would be great to fully add checksuming across the entire protocol, the
> proposed implementation will ensure we at least catch corrupted data and
> likely protect ourselves pretty well anyways.
> I didn't go to the trouble of implementing a Snappy Checksum'ed Compressor
> implementation as it's been deprecated for a while -- is really slow and
> crappy compared to LZ4 -- and we should do everything in our power to make
> sure no one in the community is still using it. I left it in (for obvious
> backwards compatibility aspects) old for clients that don't know about the
> new protocol.
> The current protocol has a 256MB (max) frame body -- where the serialized
> contents are simply written in to the frame body.
> If the client sends a compression option in the startup, we will install a
> FrameCompressor inline. Unfortunately, we went with a decision to treat the
> frame body separately from the header bits etc in a given message. So,
> instead we put a compressor implementation in the options and
[jira] [Commented] (CASSANDRA-13304) Add checksumming to the native protocol
[
https://issues.apache.org/jira/browse/CASSANDRA-13304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572170#comment-16572170
]
Michael Kjellman commented on CASSANDRA-13304:
--
given we really should be looking to move to zstandard i’m not sure why we’d
double down on something specific to lz4. i’m not sure i agree this is that
much of a burden... it’s not like the logic is that complicated.
> Add checksumming to the native protocol
> ---
>
> Key: CASSANDRA-13304
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13304
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Michael Kjellman
>Assignee: Sam Tunnicliffe
>Priority: Blocker
> Labels: client-impacting
> Fix For: 4.x
>
> Attachments: 13304_v1.diff, boxplot-read-throughput.png,
> boxplot-write-throughput.png
>
>
> The native binary transport implementation doesn't include checksums. This
> makes it highly susceptible to silently inserting corrupted data either due
> to hardware issues causing bit flips on the sender/client side, C*/receiver
> side, or network in between.
> Attaching an implementation that makes checksum'ing mandatory (assuming both
> client and server know about a protocol version that supports checksums) --
> and also adds checksumming to clients that request compression.
> The serialized format looks something like this:
> {noformat}
> * 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
> * 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Number of Compressed Chunks | Compressed Length (e1)/
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * / Compressed Length cont. (e1) |Uncompressed Length (e1) /
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length cont. (e1)| CRC32 Checksum of Lengths (e1)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Checksum of Lengths cont. (e1)|Compressed Bytes (e1)+//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e1) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (e2)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |CRC32 Checksum of Lengths (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Compressed Bytes (e2) +//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e2) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (en) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (en)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |CRC32 Checksum of Lengths (en) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Compressed Bytes (en) +//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (en) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> {noformat}
> The first pass here adds checksums only to the actual contents of the frame
> body itself (and doesn't actually checksum lengths and headers). While it
> would be great to fully add checksuming across the entire protocol, the
> proposed implementation will ensure we at least catch corrupted data and
> likely protect ourselves pretty well anyways.
> I didn't go to the trouble of implementing a Snappy Checksum'ed Compressor
> implementation as it's been deprecated for a while -- is really slow and
> crappy compared to LZ4 -- and we should do everything in our power to make
> sure no one in the community is still using it. I left it in (for obvious
> backwards compatibility aspects) old for clients that don't know about the
> new protocol.
> The current protocol has a 256MB (max) frame body -- where the serialized
> contents are simply written in to the frame body.
> If the client sends a compression option in the startup, we will install a
> FrameCompressor inline. Unfortunately, we went with a
[jira] [Commented] (CASSANDRA-14557) Consider adding default and required keyspace replication options
[
https://issues.apache.org/jira/browse/CASSANDRA-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572163#comment-16572163
]
Sumanth Pasupuleti commented on CASSANDRA-14557:
Thanks for reviewing [~chovatia.jayd...@gmail.com]
{quote}Don't we have to apply minimum keyspace RF check for system
keyspaces?{quote}
System keyspace creation *does* go through the minimum keyspace rf check. So,
if someone configures default_keyspace_rf = 1 and minimum_keyspace_rf=2,
Cassandra fails to start, since system keyspace creation fails because of the
check.
{quote}Think about renaming parsedRf to something more meaningful maybe
userSuppliedRF:{quote}
It could be both a user/system passed rf. Method parameter name has not been
refactored (rf) as part of this ticket. Named this as parsedRf because we are
parsing rf to int, and we wanted to avoid having to parse to int multiple times
in the code.
{quote} Have a consistent way of formatting string in Exception message{quote}
Fixed; pushed a commit
[7ee2827a3fd6a8f1550d9c0eec8f039b6ad6c74c|https://github.com/sumanth-pasupuleti/cassandra/commit/7ee2827a3fd6a8f1550d9c0eec8f039b6ad6c74c]
> Consider adding default and required keyspace replication options
> -
>
> Key: CASSANDRA-14557
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14557
> Project: Cassandra
> Issue Type: Improvement
> Components: Configuration
>Reporter: Sumanth Pasupuleti
>Assignee: Sumanth Pasupuleti
>Priority: Minor
> Labels: 4.0-feature-freeze-review-requested
> Fix For: 4.0
>
> Attachments: 14557-trunk.txt
>
>
> Ending up with a keyspace of RF=1 is unfortunately pretty easy in C* right
> now - the system_auth table for example is created with RF=1 (to take into
> account single node setups afaict from CASSANDRA-5112), and a user can
> further create a keyspace with RF=1 posing availability and streaming risks
> (e.g. rebuild).
> I propose we add two configuration options in cassandra.yaml:
> # {{default_keyspace_rf}} (default: 1) - If replication factors are not
> specified, use this number.
> # {{required_minimum_keyspace_rf}} (default: unset) - Prevent users from
> creating a keyspace with an RF less than what is configured
> These settings could further be re-used to:
> * Provide defaults for new keyspaces created with SimpleStrategy or
> NetworkTopologyStrategy (CASSANDRA-14303)
> * Make the automatic token [allocation
> algorithm|https://issues.apache.org/jira/browse/CASSANDRA-13701?focusedCommentId=16095662=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16095662]
> interface more intuitive allowing easy use of the new token allocation
> algorithm.
> At the end of the day, if someone really wants to allow RF=1, they simply
> don’t set the setting. For backwards compatibility the default remains 1 and
> C* would create with RF=1, and would default to current behavior of allowing
> any RF on keyspaces.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13304) Add checksumming to the native protocol
[
https://issues.apache.org/jira/browse/CASSANDRA-13304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Tunnicliffe updated CASSANDRA-13304:
Status: Patch Available (was: In Progress)
> Add checksumming to the native protocol
> ---
>
> Key: CASSANDRA-13304
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13304
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Michael Kjellman
>Assignee: Sam Tunnicliffe
>Priority: Blocker
> Labels: client-impacting
> Fix For: 4.x
>
> Attachments: 13304_v1.diff, boxplot-read-throughput.png,
> boxplot-write-throughput.png
>
>
> The native binary transport implementation doesn't include checksums. This
> makes it highly susceptible to silently inserting corrupted data either due
> to hardware issues causing bit flips on the sender/client side, C*/receiver
> side, or network in between.
> Attaching an implementation that makes checksum'ing mandatory (assuming both
> client and server know about a protocol version that supports checksums) --
> and also adds checksumming to clients that request compression.
> The serialized format looks something like this:
> {noformat}
> * 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
> * 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Number of Compressed Chunks | Compressed Length (e1)/
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * / Compressed Length cont. (e1) |Uncompressed Length (e1) /
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length cont. (e1)| CRC32 Checksum of Lengths (e1)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Checksum of Lengths cont. (e1)|Compressed Bytes (e1)+//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e1) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (e2)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |CRC32 Checksum of Lengths (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Compressed Bytes (e2) +//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e2) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (en) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (en)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |CRC32 Checksum of Lengths (en) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Compressed Bytes (en) +//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (en) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> {noformat}
> The first pass here adds checksums only to the actual contents of the frame
> body itself (and doesn't actually checksum lengths and headers). While it
> would be great to fully add checksuming across the entire protocol, the
> proposed implementation will ensure we at least catch corrupted data and
> likely protect ourselves pretty well anyways.
> I didn't go to the trouble of implementing a Snappy Checksum'ed Compressor
> implementation as it's been deprecated for a while -- is really slow and
> crappy compared to LZ4 -- and we should do everything in our power to make
> sure no one in the community is still using it. I left it in (for obvious
> backwards compatibility aspects) old for clients that don't know about the
> new protocol.
> The current protocol has a 256MB (max) frame body -- where the serialized
> contents are simply written in to the frame body.
> If the client sends a compression option in the startup, we will install a
> FrameCompressor inline. Unfortunately, we went with a decision to treat the
> frame body separately from the header bits etc in a given message. So,
> instead we put a compressor implementation in the options and then if it's
> not null, we push the serialized bytes for
[jira] [Updated] (CASSANDRA-13304) Add checksumming to the native protocol
[
https://issues.apache.org/jira/browse/CASSANDRA-13304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Tunnicliffe updated CASSANDRA-13304:
Reviewer: (was: Sam Tunnicliffe)
> Add checksumming to the native protocol
> ---
>
> Key: CASSANDRA-13304
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13304
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Michael Kjellman
>Assignee: Sam Tunnicliffe
>Priority: Blocker
> Labels: client-impacting
> Fix For: 4.x
>
> Attachments: 13304_v1.diff, boxplot-read-throughput.png,
> boxplot-write-throughput.png
>
>
> The native binary transport implementation doesn't include checksums. This
> makes it highly susceptible to silently inserting corrupted data either due
> to hardware issues causing bit flips on the sender/client side, C*/receiver
> side, or network in between.
> Attaching an implementation that makes checksum'ing mandatory (assuming both
> client and server know about a protocol version that supports checksums) --
> and also adds checksumming to clients that request compression.
> The serialized format looks something like this:
> {noformat}
> * 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
> * 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Number of Compressed Chunks | Compressed Length (e1)/
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * / Compressed Length cont. (e1) |Uncompressed Length (e1) /
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length cont. (e1)| CRC32 Checksum of Lengths (e1)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Checksum of Lengths cont. (e1)|Compressed Bytes (e1)+//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e1) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (e2)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |CRC32 Checksum of Lengths (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Compressed Bytes (e2) +//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e2) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (en) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (en)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |CRC32 Checksum of Lengths (en) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Compressed Bytes (en) +//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (en) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> {noformat}
> The first pass here adds checksums only to the actual contents of the frame
> body itself (and doesn't actually checksum lengths and headers). While it
> would be great to fully add checksuming across the entire protocol, the
> proposed implementation will ensure we at least catch corrupted data and
> likely protect ourselves pretty well anyways.
> I didn't go to the trouble of implementing a Snappy Checksum'ed Compressor
> implementation as it's been deprecated for a while -- is really slow and
> crappy compared to LZ4 -- and we should do everything in our power to make
> sure no one in the community is still using it. I left it in (for obvious
> backwards compatibility aspects) old for clients that don't know about the
> new protocol.
> The current protocol has a 256MB (max) frame body -- where the serialized
> contents are simply written in to the frame body.
> If the client sends a compression option in the startup, we will install a
> FrameCompressor inline. Unfortunately, we went with a decision to treat the
> frame body separately from the header bits etc in a given message. So,
> instead we put a compressor implementation in the options and then if it's
> not null, we push the serialized bytes for the frame
[jira] [Commented] (CASSANDRA-13304) Add checksumming to the native protocol
[
https://issues.apache.org/jira/browse/CASSANDRA-13304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572164#comment-16572164
]
Sam Tunnicliffe commented on CASSANDRA-13304:
-
I've just come back to this with the intention of getting it committed before
the 4.0 freeze. It's a bit late in the cycle, but I can't help but be concerned
that the protocol changes are a bit too invasive and that pushing requirement
to implement the chunking and checksumming logic out onto all client
implementations is a bit burdensome. I've put together a patch which keeps
everything as is, but adds a new compression scheme which uses the [LZ4 Frame
format|https://github.com/lz4/lz4/wiki/lz4_Frame_format.md]. The most obvious
benefit of this over rolling our own format is that it's more or less a tiny
change from the driver perspective. The frame format is designed for interop
and is implemented by most LZ4 libs in most languages. Another is that by
adding this as a new compression scheme, it's completely trivial to roll it out
without a protocol bump.
I've haven't benchmarked or profiled this yet, only hacked a couple of
python/java driver patches to validate functionality. I also haven't updated
the protocol spec with details of the new format yet.
Patch is at: https://github.com/beobal/cassandra/commits/cci/13304-lz4-frame
[~adutra], [~aholmber], [~tvdw], [~mkjellman], any thoughts on this?
> Add checksumming to the native protocol
> ---
>
> Key: CASSANDRA-13304
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13304
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Michael Kjellman
>Assignee: Sam Tunnicliffe
>Priority: Blocker
> Labels: client-impacting
> Fix For: 4.x
>
> Attachments: 13304_v1.diff, boxplot-read-throughput.png,
> boxplot-write-throughput.png
>
>
> The native binary transport implementation doesn't include checksums. This
> makes it highly susceptible to silently inserting corrupted data either due
> to hardware issues causing bit flips on the sender/client side, C*/receiver
> side, or network in between.
> Attaching an implementation that makes checksum'ing mandatory (assuming both
> client and server know about a protocol version that supports checksums) --
> and also adds checksumming to clients that request compression.
> The serialized format looks something like this:
> {noformat}
> * 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
> * 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Number of Compressed Chunks | Compressed Length (e1)/
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * / Compressed Length cont. (e1) |Uncompressed Length (e1) /
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length cont. (e1)| CRC32 Checksum of Lengths (e1)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Checksum of Lengths cont. (e1)|Compressed Bytes (e1)+//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e1) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (e2)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |CRC32 Checksum of Lengths (e2) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Compressed Bytes (e2) +//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (e2) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |Compressed Length (en) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Uncompressed Length (en)|
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * |CRC32 Checksum of Lengths (en) |
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | Compressed Bytes (en) +//
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> * | CRC32 Checksum (en) ||
> * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> {noformat}
> The first
[jira] [Assigned] (CASSANDRA-14408) Transient Replication: Incremental & Validation repair handling of transient replicas
[ https://issues.apache.org/jira/browse/CASSANDRA-14408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Blake Eggleston reassigned CASSANDRA-14408: --- Assignee: Blake Eggleston > Transient Replication: Incremental & Validation repair handling of transient > replicas > - > > Key: CASSANDRA-14408 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14408 > Project: Cassandra > Issue Type: Sub-task > Components: Repair >Reporter: Ariel Weisberg >Assignee: Blake Eggleston >Priority: Major > Fix For: 4.0 > > > At transient replicas anti-compaction shouldn't output any data for transient > ranges as the data will be dropped after repair. > Transient replicas should also never have data streamed to them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14563) Add animalsniffer to build to ensure runtime jdk compatbility
[ https://issues.apache.org/jira/browse/CASSANDRA-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571865#comment-16571865 ] Sumanth Pasupuleti commented on CASSANDRA-14563: [~michaelsembwever] sure, [CASSANDRA-14625|https://issues.apache.org/jira/browse/CASSANDRA-14625] > Add animalsniffer to build to ensure runtime jdk compatbility > - > > Key: CASSANDRA-14563 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14563 > Project: Cassandra > Issue Type: Improvement > Components: Build >Reporter: mck >Assignee: Sumanth Pasupuleti >Priority: Minor > Labels: lhf > > Cassandra-2.2 still supports running on JDK1.7 > No tests check this though, as all build and test with JDK1.8 > Adding the ant animalsniffer task can check that jdk1.8 classes or methods > are not used accidentally. > ref: http://www.mojohaus.org/animal-sniffer/animal-sniffer/index.html -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14626) Expose buffer cache metrics in caches virtual table
[ https://issues.apache.org/jira/browse/CASSANDRA-14626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeremy Hanna updated CASSANDRA-14626: - Labels: virtual-tables (was: ) > Expose buffer cache metrics in caches virtual table > --- > > Key: CASSANDRA-14626 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14626 > Project: Cassandra > Issue Type: New Feature >Reporter: Benjamin Lerer >Assignee: Aleksey Yeschenko >Priority: Minor > Labels: virtual-tables > Fix For: 4.0 > > > As noted by [~blerer] in CASSANDRA-14538, we should expose buffer cache > metrics in the caches virtual table. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14538) Add virtual table to view cache information
[
https://issues.apache.org/jira/browse/CASSANDRA-14538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571671#comment-16571671
]
Aleksey Yeschenko commented on CASSANDRA-14538:
---
[~blerer] Good point. Agreed. Created CASSANDRA-14626 to take care of it.
> Add virtual table to view cache information
> ---
>
> Key: CASSANDRA-14538
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14538
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Chris Lohfink
>Assignee: Chris Lohfink
>Priority: Minor
> Labels: virtual-tables
> Fix For: 4.0
>
>
> Add a table for displaying the general cache information (like in {{nodetool
> info}})
> {code}
> cqlsh> SELECT * FROM system_views.caches;
> name| entries | hit_ratio | hits | recent_hits_per_sec |
> recent_requests_per_sec | requests | size_max | size_used
> -+-+---+--+-+-+--+--+---
> counter | 0 | NaN |0 | 0 |
> 0 |0 | 12582912 | 0
> key | 19 | 0.922509 | 250 | 11 |
> 13 | 271 | 25165824 | 1728
> row | 0 | NaN |0 | 0 |
> 0 |0 | 16777216 | 0
> (3 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14626) Expose buffer cache metrics in caches virtual table
Aleksey Yeschenko created CASSANDRA-14626: - Summary: Expose buffer cache metrics in caches virtual table Key: CASSANDRA-14626 URL: https://issues.apache.org/jira/browse/CASSANDRA-14626 Project: Cassandra Issue Type: New Feature Reporter: Benjamin Lerer Assignee: Aleksey Yeschenko Fix For: 4.0 As noted by [~blerer] in CASSANDRA-14538, we should expose buffer cache metrics in the caches virtual table. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13668) Diag. events for user audit logging
[ https://issues.apache.org/jira/browse/CASSANDRA-13668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mck updated CASSANDRA-13668: Reviewers: mck > Diag. events for user audit logging > --- > > Key: CASSANDRA-13668 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13668 > Project: Cassandra > Issue Type: Improvement > Components: Observability >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Major > Fix For: 4.x > > > With the availability of CASSANDRA-13459, any native transport enabled client > will be able to subscribe to internal Cassandra events. External tools can > take advantage by monitoring these events in various ways. Use-cases for this > can be e.g. auditing tools for compliance and security purposes. > The scope of this ticket is to add diagnostic events that are raised around > authentication and CQL operations. These events can then be consumed and used > by external tools to implement a Cassandra user auditing solution. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13668) Diag. events for user audit logging
[ https://issues.apache.org/jira/browse/CASSANDRA-13668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571577#comment-16571577 ] mck commented on CASSANDRA-13668: - reviewed, +1 from me. > Diag. events for user audit logging > --- > > Key: CASSANDRA-13668 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13668 > Project: Cassandra > Issue Type: Improvement > Components: Observability >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Major > Fix For: 4.x > > > With the availability of CASSANDRA-13459, any native transport enabled client > will be able to subscribe to internal Cassandra events. External tools can > take advantage by monitoring these events in various ways. Use-cases for this > can be e.g. auditing tools for compliance and security purposes. > The scope of this ticket is to add diagnostic events that are raised around > authentication and CQL operations. These events can then be consumed and used > by external tools to implement a Cassandra user auditing solution. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14435) Diag. Events: JMX events
[ https://issues.apache.org/jira/browse/CASSANDRA-14435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mck updated CASSANDRA-14435: Reviewers: mck > Diag. Events: JMX events > > > Key: CASSANDRA-14435 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14435 > Project: Cassandra > Issue Type: New Feature > Components: Observability >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Major > Fix For: 4.x > > > Nodes currently use JMX events for progress reporting on bootstrap and > repairs. This might also be an option to expose diagnostic events to external > subscribers. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14435) Diag. Events: JMX events
[ https://issues.apache.org/jira/browse/CASSANDRA-14435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571508#comment-16571508 ] mck commented on CASSANDRA-14435: - reviewed. +1 from me. > Diag. Events: JMX events > > > Key: CASSANDRA-14435 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14435 > Project: Cassandra > Issue Type: New Feature > Components: Observability >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Major > Fix For: 4.x > > > Nodes currently use JMX events for progress reporting on bootstrap and > repairs. This might also be an option to expose diagnostic events to external > subscribers. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-11429) DROP TABLE IF EXISTS fails against table with similar name
[
https://issues.apache.org/jira/browse/CASSANDRA-11429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571478#comment-16571478
]
Chakravarthi Manepalli commented on CASSANDRA-11429:
I am creating the tables concurrently, and i faced the similar issue but in the
reverse manner. I have got the entry in system_schema keyspace about the index,
but the index is not getting created.
Then, I stopped the node and cleared data folder and started cassandra again
and loaded the schema loader and I faced the same issue again. !column family
mismatch.png!
> DROP TABLE IF EXISTS fails against table with similar name
> --
>
> Key: CASSANDRA-11429
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11429
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Sotirios Delimanolis
>Priority: Major
> Attachments: column family mismatch.png
>
>
> We had a table named {{our_keyspace.native_address_book_uploads_cache}} (note
> the uploads*) which we dropped. We then created a new table named
> {{our_keyspace.native_address_book_upload_cache}} (note the upload*).
> We have a patching component that applies commands to prepare the schema
> using the C# driver. When we deploy, it tries to execute
> {noformat}
> DROP TABLE IF NOT EXISTS our_keyspace.native_address_book_uploads_cache;
> {noformat}
> This fails with
> {noformat}
> Caught an exception Cassandra.ServerErrorException:
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.ConfigurationException: Column family ID
> mismatch (found c712a590-f194-11e5-891d-2d7ca98597ba; expected
> b8b40ed0-f194-11e5-b481-d944f7ad0ce3)
> {noformat}
> showing the Cassandra Java exception through the C# driver. Note the
> {{found}} cf_id of {{c712a590-f194-11e5-891d-2d7ca98597ba}}.
> I can reproduce this with {{cqlsh}}.
> {noformat}
> selimanolis$ cqlsh
> Connected to Default Cluster at hostname:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.13-SNAPSHOT | CQL spec 3.2.1 | Native protocol
> v3]
> Use HELP for help.
> cqlsh> SELECT cf_id from system.schema_columnfamilies where keyspace_name =
> 'our_keyspace' and columnfamily_name ='native_address_book_uploads_cache';
> keyspace_name | columnfamily_name | bloom_filter_fp_chance | caching | cf_id
> | column_aliases | comment | compaction_strategy_class |
> compaction_strategy_options | comparator | compression_parameters |
> default_time_to_live | default_validator | dropped_columns | gc_grace_seconds
> | index_interval | is_dense | key_aliases | key_validator |
> local_read_repair_chance | max_compaction_threshold | max_index_interval |
> memtable_flush_period_in_ms | min_compaction_threshold | min_index_interval |
> read_repair_chance | speculative_retry | subcomparator | type | value_alias
> ---+---++-+---++-+---+-+++--+---+-+--++--+-+---+--+--++-+--+++---+---+--+-
> (0 rows)
> cqlsh> SELECT cf_id from system.schema_columnfamilies where keyspace_name =
> 'our_keyspace' and columnfamily_name ='native_address_book_upload_cache';
> cf_id
> --
> c712a590-f194-11e5-891d-2d7ca98597ba
> (1 rows)
> cqlsh> drop TABLE IF EXISTS our_keyspace.native_address_book_uploads_cache;
> InvalidRequest: code=2200 [Invalid query] message="No keyspace has been
> specified. USE a keyspace, or explicitly specify keyspace.tablename"
> cqlsh> drop TABLE IF EXISTS our_keyspace.native_address_book_uploads_cache;
> ServerError: message="java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.ConfigurationException: Column family ID
> mismatch (found c712a590-f194-11e5-891d-2d7ca98597ba; expected
> b8b40ed0-f194-11e5-b481-d944f7ad0ce3)">
> cqlsh>
> {noformat}
> The table doesn't exist. A table that has a similar name does. You'll notice
> that the new table has same {{cf_id}} found in the error message above. Why
> does Cassandra confuse the two?
> Our expectation is for the {{DROP TABLE IF EXISTS}} to silently succeed.
> Similarly, we expect a {{DROP TABLE}} to fail because the table doesn't
> exist. That's not what happens if you see below
> {noformat}
> cqlsh> DROP TABLE
[jira] [Updated] (CASSANDRA-11429) DROP TABLE IF EXISTS fails against table with similar name
[
https://issues.apache.org/jira/browse/CASSANDRA-11429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chakravarthi Manepalli updated CASSANDRA-11429:
---
Attachment: column family mismatch.png
> DROP TABLE IF EXISTS fails against table with similar name
> --
>
> Key: CASSANDRA-11429
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11429
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Sotirios Delimanolis
>Priority: Major
> Attachments: column family mismatch.png
>
>
> We had a table named {{our_keyspace.native_address_book_uploads_cache}} (note
> the uploads*) which we dropped. We then created a new table named
> {{our_keyspace.native_address_book_upload_cache}} (note the upload*).
> We have a patching component that applies commands to prepare the schema
> using the C# driver. When we deploy, it tries to execute
> {noformat}
> DROP TABLE IF NOT EXISTS our_keyspace.native_address_book_uploads_cache;
> {noformat}
> This fails with
> {noformat}
> Caught an exception Cassandra.ServerErrorException:
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.ConfigurationException: Column family ID
> mismatch (found c712a590-f194-11e5-891d-2d7ca98597ba; expected
> b8b40ed0-f194-11e5-b481-d944f7ad0ce3)
> {noformat}
> showing the Cassandra Java exception through the C# driver. Note the
> {{found}} cf_id of {{c712a590-f194-11e5-891d-2d7ca98597ba}}.
> I can reproduce this with {{cqlsh}}.
> {noformat}
> selimanolis$ cqlsh
> Connected to Default Cluster at hostname:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.13-SNAPSHOT | CQL spec 3.2.1 | Native protocol
> v3]
> Use HELP for help.
> cqlsh> SELECT cf_id from system.schema_columnfamilies where keyspace_name =
> 'our_keyspace' and columnfamily_name ='native_address_book_uploads_cache';
> keyspace_name | columnfamily_name | bloom_filter_fp_chance | caching | cf_id
> | column_aliases | comment | compaction_strategy_class |
> compaction_strategy_options | comparator | compression_parameters |
> default_time_to_live | default_validator | dropped_columns | gc_grace_seconds
> | index_interval | is_dense | key_aliases | key_validator |
> local_read_repair_chance | max_compaction_threshold | max_index_interval |
> memtable_flush_period_in_ms | min_compaction_threshold | min_index_interval |
> read_repair_chance | speculative_retry | subcomparator | type | value_alias
> ---+---++-+---++-+---+-+++--+---+-+--++--+-+---+--+--++-+--+++---+---+--+-
> (0 rows)
> cqlsh> SELECT cf_id from system.schema_columnfamilies where keyspace_name =
> 'our_keyspace' and columnfamily_name ='native_address_book_upload_cache';
> cf_id
> --
> c712a590-f194-11e5-891d-2d7ca98597ba
> (1 rows)
> cqlsh> drop TABLE IF EXISTS our_keyspace.native_address_book_uploads_cache;
> InvalidRequest: code=2200 [Invalid query] message="No keyspace has been
> specified. USE a keyspace, or explicitly specify keyspace.tablename"
> cqlsh> drop TABLE IF EXISTS our_keyspace.native_address_book_uploads_cache;
> ServerError: message="java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.ConfigurationException: Column family ID
> mismatch (found c712a590-f194-11e5-891d-2d7ca98597ba; expected
> b8b40ed0-f194-11e5-b481-d944f7ad0ce3)">
> cqlsh>
> {noformat}
> The table doesn't exist. A table that has a similar name does. You'll notice
> that the new table has same {{cf_id}} found in the error message above. Why
> does Cassandra confuse the two?
> Our expectation is for the {{DROP TABLE IF EXISTS}} to silently succeed.
> Similarly, we expect a {{DROP TABLE}} to fail because the table doesn't
> exist. That's not what happens if you see below
> {noformat}
> cqlsh> DROP TABLE our_keyspace.native_address_book_uploads_cache;
> ServerError: message="java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.ConfigurationException: Column family ID
> mismatch (found c712a590-f194-11e5-891d-2d7ca98597ba; expected
> b8b40ed0-f194-11e5-b481-d944f7ad0ce3)">
> cqlsh> DROP TABLE
[jira] [Commented] (CASSANDRA-13457) Diag. Events: Add base classes
[ https://issues.apache.org/jira/browse/CASSANDRA-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571334#comment-16571334 ] mck commented on CASSANDRA-13457: - reviewed again. +1 from me. > Diag. Events: Add base classes > -- > > Key: CASSANDRA-13457 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13457 > Project: Cassandra > Issue Type: Sub-task > Components: Core, Observability >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Major > > Base ticket for adding classes that will allow you to implement and subscribe > to events. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14538) Add virtual table to view cache information
[
https://issues.apache.org/jira/browse/CASSANDRA-14538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571241#comment-16571241
]
Benjamin Lerer commented on CASSANDRA-14538:
Even if it is slightly different. It seems to me that the view should have also
exposed the chunk cache.
> Add virtual table to view cache information
> ---
>
> Key: CASSANDRA-14538
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14538
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Chris Lohfink
>Assignee: Chris Lohfink
>Priority: Minor
> Labels: virtual-tables
> Fix For: 4.0
>
>
> Add a table for displaying the general cache information (like in {{nodetool
> info}})
> {code}
> cqlsh> SELECT * FROM system_views.caches;
> name| entries | hit_ratio | hits | recent_hits_per_sec |
> recent_requests_per_sec | requests | size_max | size_used
> -+-+---+--+-+-+--+--+---
> counter | 0 | NaN |0 | 0 |
> 0 |0 | 12582912 | 0
> key | 19 | 0.922509 | 250 | 11 |
> 13 | 271 | 25165824 | 1728
> row | 0 | NaN |0 | 0 |
> 0 |0 | 16777216 | 0
> (3 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14625) Add Java7 CircleCI workflow for C* 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-14625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571172#comment-16571172 ] Sumanth Pasupuleti commented on CASSANDRA-14625: Pull request: https://github.com/apache/cassandra/pull/243 > Add Java7 CircleCI workflow for C* 2.2 > -- > > Key: CASSANDRA-14625 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14625 > Project: Cassandra > Issue Type: Improvement > Components: Build >Reporter: Sumanth Pasupuleti >Priority: Minor > Fix For: 2.2.0 > > > C* 2.2 supports Java 7, in addition to Java 8. Adding a workflow to build > against Java7 (in addition to existing Java8 build) would ensure any release > we make on 2.2 does not break the contract of Java 7 compatibility. > There are multiple approaches of solving this - one is CircleCi build (this), > and other is to add a step into the development phase for a much faster > feedback loop [14563|https://issues.apache.org/jira/browse/CASSANDRA-14563] > [~michaelsembwever] fyi -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14625) Add Java7 CircleCI workflow for C* 2.2
Sumanth Pasupuleti created CASSANDRA-14625: -- Summary: Add Java7 CircleCI workflow for C* 2.2 Key: CASSANDRA-14625 URL: https://issues.apache.org/jira/browse/CASSANDRA-14625 Project: Cassandra Issue Type: Improvement Components: Build Reporter: Sumanth Pasupuleti Fix For: 2.2.0 C* 2.2 supports Java 7, in addition to Java 8. Adding a workflow to build against Java7 (in addition to existing Java8 build) would ensure any release we make on 2.2 does not break the contract of Java 7 compatibility. There are multiple approaches of solving this - one is CircleCi build (this), and other is to add a step into the development phase for a much faster feedback loop [14563|https://issues.apache.org/jira/browse/CASSANDRA-14563] [~michaelsembwever] fyi -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14436) Add sampler for query time and expose with nodetool
[
https://issues.apache.org/jira/browse/CASSANDRA-14436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571162#comment-16571162
]
Chris Lohfink commented on CASSANDRA-14436:
---
While I think we can do something like creating a concurrent set of Samplers
for each SamplerType that we tie to a Sampler session and flag it to start at
same time I dont think its necessary. The current use of top partitions has
never had a reported issue with people trying to concurrently run profiling
sessions so it can be a new feature to add in another ticket at sometime but I
dont think its needed enough here.
In meantime I added a strict restriction on a single at a time, raising an
exception if someone tries to kick off a 2nd one. Also the sampling will
timeout at the end of the duration so if the finish is never called it wont
spin forever.
I did write some basic jmh benchmarks but i didnt want to make insert()
accessible and the {{.*microbench.*}} in build.xml makes default visibility not
an option so... yeah. Ultimately (when on) its just ThreadExecuterPool.submit()
on the addSample in read/write path which is pretty straight forward limitation
on the contention on the queue but i saw 100-300nanosecond -ish. Going into the
actual guts, the frequency sampler being a wrapper around the addthis
StreamSummary - which there might be something better out there now but its
seemed to do fine so far. In some worst case JMH benchmarks I was able to see
this hit 3us or so, which could conceivably underperform writes which would
cause a backup. The MaxSampler uses MinMaxPriorityQueue, which after
PriorityQueue(comparator) becomes available (post java8) that can be replaced
to be more performant, but that rarely breaks a microsecond even with top 1024.
Just incase as a catchall I added the same as the trace executor - a throwaway
loadshedding incase the sampler executor does get backed up. This includes some
plumbing so its reported appropriately in metrics.
> Add sampler for query time and expose with nodetool
> ---
>
> Key: CASSANDRA-14436
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14436
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Chris Lohfink
>Assignee: Chris Lohfink
>Priority: Major
>
> Create a new {{nodetool profileload}} that functions just like toppartitions
> but with more data, returning the slowest local reads and writes on the host
> during a given duration and highest frequency touched partitions (same as
> {{nodetool toppartitions}}). Refactor included to extend use of the sampler
> for uses outside of top frequency (max instead of total sample values).
> Future work to this is to include top cpu and allocations by query and
> possibly tasks/cpu/allocations by stage during time window.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
