[jira] [Created] (CASSANDRA-15210) Streaming with CDC does not honor cdc_enabled
Andrew Prudhomme created CASSANDRA-15210:
Summary: Streaming with CDC does not honor cdc_enabled
Key: CASSANDRA-15210
URL: https://issues.apache.org/jira/browse/CASSANDRA-15210
Project: Cassandra
Issue Type: Bug
Components: Consistency/Streaming, Feature/Change Data Capture
Reporter: Andrew Prudhomme
When SSTables are streamed for a CDC enabled table, the updates are processed
through the write path to ensure they are made available through the commit
log. However, currently only the CDC state of the table is checked. Since CDC
is enabled at both the node and table level, a node with CDC disabled (with
cdc_enabled: false) will unnecessarily send updates through the write path if
CDC is enabled on the table. This seems like an oversight.
I'd imagine the fix would be something like
{code:java}
- hasCDC = cfs.metadata.params.cdc;
+ hasCDC = cfs.metadata.params.cdc && DatabaseDescriptor.isCDCEnabled();{code}
in
org.apache.cassandra.db.streaming.CassandraStreamReceiver (4)
org.apache.cassandra.streaming.StreamReceiveTask (3.11)
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14948) Backport dropped column checks to 3.11
[ https://issues.apache.org/jira/browse/CASSANDRA-14948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko updated CASSANDRA-14948: -- Reviewers: Aleksey Yeschenko Fix Version/s: 3.0.x > Backport dropped column checks to 3.11 > -- > > Key: CASSANDRA-14948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14948 > Project: Cassandra > Issue Type: Bug >Reporter: Vincent White >Assignee: Vincent White >Priority: Low > Fix For: 3.0.x, 3.11.x > > > This is a follow on from CASSANDRA-14913 and CASSANDRA-14843 that introduced > some fixes to prevent and mitigate data corruption caused by dropping a > column then re-adding it with the same name but an incompatible type (e.g. > simple int to a complex map<>) or different kind (regular/static). > This patch backports the checks that now exist in trunk. This does include > adding a column to the dropped_columns table to keep track of static columns > like trunk, not sure it we are able to make that change in 3.11.x. > Also not sure what our stance on backporting just the isValueCompatibleWith > check to 3.0 is. I'd be for it since it prevents recreating a simple column > as a map (or vice-versa) which will basically always lead to corruption. > ||C* 3.11.x|| > |[Patch|https://github.com/vincewhite/cassandra/commit/3986b53b8acaf1d3691f9b35fd098a40667c520f]| -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15097) Avoid updating unchanged gossip state
[ https://issues.apache.org/jira/browse/CASSANDRA-15097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885427#comment-16885427 ] Jay Zhuang edited comment on CASSANDRA-15097 at 7/15/19 5:29 PM: - Thanks [~samt]. The patch is rebased and tests are passed in circleci: | Branch | uTest (circleci) | | [15097-3.0|https://github.com/instagram/cassandra/tree/15097-3.0] | [pass|https://circleci.com/gh/Instagram/cassandra/tree/15097-3.0] | | [15097-3.11|https://github.com/instagram/cassandra/tree/15097-3.11] | [pass|https://circleci.com/gh/Instagram/cassandra/tree/15097-3.11] | | [15097-trunk|https://github.com/instagram/cassandra/tree/15097-trunk] | [pass|https://circleci.com/gh/Instagram/cassandra/tree/15097-trunk] | was (Author: jay.zhuang): Here is a patch to filter out updated states: | Branch | uTest (circleci) | | [15097-3.0|https://github.com/instagram/cassandra/tree/15097-3.0] | [pass|https://circleci.com/gh/Instagram/cassandra/tree/15097-3.0] | | [15097-3.11|https://github.com/instagram/cassandra/tree/15097-3.11] | [pass|https://circleci.com/gh/Instagram/cassandra/tree/15097-3.11] | | [15097-trunk|https://github.com/instagram/cassandra/tree/15097-trunk] | [pass|https://circleci.com/gh/Instagram/cassandra/tree/15097-trunk] | > Avoid updating unchanged gossip state > - > > Key: CASSANDRA-15097 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15097 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Gossip >Reporter: Jay Zhuang >Assignee: Jay Zhuang >Priority: Normal > > The node might get unchanged gossip states, the state might be just updated > after sending a GOSSIP_SYN, then it will get the state that is already up to > date. If the heartbeat in the GOSSIP_ACK message is updated, it will > unnecessary re-apply the same state again, which could be costly like > updating token change. > It's very likely to happen for large cluster when a node startup, as the > first gossip message will sync all endpoints tokens, it could take some time > (in our case about 200 seconds), during that time, it keeps gossip with other > node and get the full token states. Which causes lots of pending gossip tasks. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15097) Avoid updating unchanged gossip state
[ https://issues.apache.org/jira/browse/CASSANDRA-15097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Zhuang updated CASSANDRA-15097: --- Status: Review In Progress (was: Changes Suggested) > Avoid updating unchanged gossip state > - > > Key: CASSANDRA-15097 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15097 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Gossip >Reporter: Jay Zhuang >Assignee: Jay Zhuang >Priority: Normal > > The node might get unchanged gossip states, the state might be just updated > after sending a GOSSIP_SYN, then it will get the state that is already up to > date. If the heartbeat in the GOSSIP_ACK message is updated, it will > unnecessary re-apply the same state again, which could be costly like > updating token change. > It's very likely to happen for large cluster when a node startup, as the > first gossip message will sync all endpoints tokens, it could take some time > (in our case about 200 seconds), during that time, it keeps gossip with other > node and get the full token states. Which causes lots of pending gossip tasks. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15097) Avoid updating unchanged gossip state
[ https://issues.apache.org/jira/browse/CASSANDRA-15097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885427#comment-16885427 ] Jay Zhuang commented on CASSANDRA-15097: Here is a patch to filter out updated states: | Branch | uTest (circleci) | | [15097-3.0|https://github.com/instagram/cassandra/tree/15097-3.0] | [pass|https://circleci.com/gh/Instagram/cassandra/tree/15097-3.0] | | [15097-3.11|https://github.com/instagram/cassandra/tree/15097-3.11] | [pass|https://circleci.com/gh/Instagram/cassandra/tree/15097-3.11] | | [15097-trunk|https://github.com/instagram/cassandra/tree/15097-trunk] | [pass|https://circleci.com/gh/Instagram/cassandra/tree/15097-trunk] | > Avoid updating unchanged gossip state > - > > Key: CASSANDRA-15097 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15097 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Gossip >Reporter: Jay Zhuang >Assignee: Jay Zhuang >Priority: Normal > > The node might get unchanged gossip states, the state might be just updated > after sending a GOSSIP_SYN, then it will get the state that is already up to > date. If the heartbeat in the GOSSIP_ACK message is updated, it will > unnecessary re-apply the same state again, which could be costly like > updating token change. > It's very likely to happen for large cluster when a node startup, as the > first gossip message will sync all endpoints tokens, it could take some time > (in our case about 200 seconds), during that time, it keeps gossip with other > node and get the full token states. Which causes lots of pending gossip tasks. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15202) Deserialize merkle trees off-heap
[ https://issues.apache.org/jira/browse/CASSANDRA-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-15202: - Status: Changes Suggested (was: Review In Progress) > Deserialize merkle trees off-heap > - > > Key: CASSANDRA-15202 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15202 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Repair >Reporter: Jeff Jirsa >Assignee: Aleksey Yeschenko >Priority: Normal > Fix For: 4.0 > > Attachments: offheap-mts-gc.png > > > CASSANDRA-14096 made the first step to address the heavy on-heap footprint of > merkle trees on repair coordinators - by reducing the time frame over which > they are referenced, and by more intelligently limiting depth of the trees > based on available heap size. > That alone improves GC profile and prevents OOMs, but doesn’t address the > issue entirely. The coordinator still must hold all the trees on heap at once > until it’s done diffing them with each other, which has a negative effect, > and, by reducing depth, we lose precision and thus cause more overstreaming > than before. > One way to improve the situation further is to build on CASSANDRA-14096 and > move the trees entirely off-heap. This is a trivial endeavor, given that we > are dealing with what should be full binary trees (though in practice aren’t > quite, yet). This JIRA makes the first step towards there - by moving just > deserialisation off-heap, leaving construction on the replicas on-heap still. > Additionally, the proposed patch fixes the issue of replica coordinators > sending merkle trees to itself over loopback, costing us a ser/deser loop per > tree. > Please note that there is more room for improvement here, and depending on > 4.0 timeline those improvements may or may not land in time. To name a few: > - with some minor modifications to init(), we can make sure that no matter > the range, the tree is *always* perfectly full; this would allow us to get > rid of child pointers in inner nodes, as child node addresses will be > trivially calculatable given fixed size of nodes > - the trees can be easily constructed off-heap so long as you run init() to > pre-size the tree to find out how large a buffer you need > - on-wire format doesn’t need to stream inner nodes, only leaves, and, > really, only the hashes of the leaves -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15202) Deserialize merkle trees off-heap
[ https://issues.apache.org/jira/browse/CASSANDRA-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-15202: - Status: Review In Progress (was: Patch Available) > Deserialize merkle trees off-heap > - > > Key: CASSANDRA-15202 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15202 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Repair >Reporter: Jeff Jirsa >Assignee: Aleksey Yeschenko >Priority: Normal > Fix For: 4.0 > > Attachments: offheap-mts-gc.png > > > CASSANDRA-14096 made the first step to address the heavy on-heap footprint of > merkle trees on repair coordinators - by reducing the time frame over which > they are referenced, and by more intelligently limiting depth of the trees > based on available heap size. > That alone improves GC profile and prevents OOMs, but doesn’t address the > issue entirely. The coordinator still must hold all the trees on heap at once > until it’s done diffing them with each other, which has a negative effect, > and, by reducing depth, we lose precision and thus cause more overstreaming > than before. > One way to improve the situation further is to build on CASSANDRA-14096 and > move the trees entirely off-heap. This is a trivial endeavor, given that we > are dealing with what should be full binary trees (though in practice aren’t > quite, yet). This JIRA makes the first step towards there - by moving just > deserialisation off-heap, leaving construction on the replicas on-heap still. > Additionally, the proposed patch fixes the issue of replica coordinators > sending merkle trees to itself over loopback, costing us a ser/deser loop per > tree. > Please note that there is more room for improvement here, and depending on > 4.0 timeline those improvements may or may not land in time. To name a few: > - with some minor modifications to init(), we can make sure that no matter > the range, the tree is *always* perfectly full; this would allow us to get > rid of child pointers in inner nodes, as child node addresses will be > trivially calculatable given fixed size of nodes > - the trees can be easily constructed off-heap so long as you run init() to > pre-size the tree to find out how large a buffer you need > - on-wire format doesn’t need to stream inner nodes, only leaves, and, > really, only the hashes of the leaves -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15202) Deserialize merkle trees off-heap
[
https://issues.apache.org/jira/browse/CASSANDRA-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885370#comment-16885370
]
Benedict commented on CASSANDRA-15202:
--
LGTM. Some minor suggestions:
* Use {{ByteOrder.LITTLE_ENDIAN}} for off heap?
* {{RandomPartitioner.MAXIMUM_TOKEN_SIZE}}: use {{(bitLength + 7) / 8}}?
* {{differenceHelper}}: replace int return values with explicit Enum?
* {{OffHeapNode.attachRef}}: use {{null}} referent? Should probably (perhaps
another time) introduce a special variant for these pure leak-tracking use
cases.
* {{OnHeapLeaf.deserialize}}: verify {{in.readByte() == HASH_SIZE}}?
* Extract shared method {{OffHeapInner.child}} from {{left}} and {{right}}?
* Extract shared method for off heap disabled warning?
* {{prefer_offheap_merkle_trees}} - why prefer?
* {{FBUtilities.xor}} and {{FBUtilities.xorOntoLeft}} - move to {{MerkleTree}}?
* Missing line break: {{MerkleTree.release}}, {{OffHeapInner.maxOffHeapSize}}?
* Rename test-only methods to {{unsafeMethodName}} and disappear to bottom of
class?
* "TODO: reset computed flag on OnHeapInners" - simply ignore/remove TODO if
renamed to {{unsafeInvalidateHelper}}?
> Deserialize merkle trees off-heap
> -
>
> Key: CASSANDRA-15202
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15202
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Repair
>Reporter: Jeff Jirsa
>Assignee: Aleksey Yeschenko
>Priority: Normal
> Fix For: 4.0
>
> Attachments: offheap-mts-gc.png
>
>
> CASSANDRA-14096 made the first step to address the heavy on-heap footprint of
> merkle trees on repair coordinators - by reducing the time frame over which
> they are referenced, and by more intelligently limiting depth of the trees
> based on available heap size.
> That alone improves GC profile and prevents OOMs, but doesn’t address the
> issue entirely. The coordinator still must hold all the trees on heap at once
> until it’s done diffing them with each other, which has a negative effect,
> and, by reducing depth, we lose precision and thus cause more overstreaming
> than before.
> One way to improve the situation further is to build on CASSANDRA-14096 and
> move the trees entirely off-heap. This is a trivial endeavor, given that we
> are dealing with what should be full binary trees (though in practice aren’t
> quite, yet). This JIRA makes the first step towards there - by moving just
> deserialisation off-heap, leaving construction on the replicas on-heap still.
> Additionally, the proposed patch fixes the issue of replica coordinators
> sending merkle trees to itself over loopback, costing us a ser/deser loop per
> tree.
> Please note that there is more room for improvement here, and depending on
> 4.0 timeline those improvements may or may not land in time. To name a few:
> - with some minor modifications to init(), we can make sure that no matter
> the range, the tree is *always* perfectly full; this would allow us to get
> rid of child pointers in inner nodes, as child node addresses will be
> trivially calculatable given fixed size of nodes
> - the trees can be easily constructed off-heap so long as you run init() to
> pre-size the tree to find out how large a buffer you need
> - on-wire format doesn’t need to stream inner nodes, only leaves, and,
> really, only the hashes of the leaves
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15013) Prevent client requests from blocking on executor task queue
[ https://issues.apache.org/jira/browse/CASSANDRA-15013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885313#comment-16885313 ] Sumanth Pasupuleti commented on CASSANDRA-15013: Thanks for the feedback and review [~benedict]. I will incorporate this feedback into my future commits and submit a patch for the documentation if necessary. > Prevent client requests from blocking on executor task queue > > > Key: CASSANDRA-15013 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15013 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Client >Reporter: Sumanth Pasupuleti >Assignee: Sumanth Pasupuleti >Priority: Normal > Labels: pull-request-available > Fix For: 3.0.19, 3.11.5, 4.0 > > Attachments: 15013-3.0.txt, 15013-3.11.txt, 15013-trunk.txt, > BlockedEpollEventLoopFromHeapDump.png, > BlockedEpollEventLoopFromThreadDump.png, RequestExecutorQueueFull.png, heap > dump showing each ImmediateFlusher taking upto 600MB.png, > perftest2_15013_base_flamegraph.svg, perftest2_15013_patch_flamegraph.svg, > perftest2_blocked_threadpool.png, perftest2_cpu_usage.png, > perftest2_heap.png, perftest2_read_latency_99th.png, > perftest2_read_latency_avg.png, perftest2_readops.png, > perftest2_write_latency_99th.png, perftest2_write_latency_avg.png, > perftest2_writeops.png, perftest_blockedthreads.png, > perftest_connections_count.png, perftest_cpu_usage.png, > perftest_heap_usage.png, perftest_readlatency_99th.png, > perftest_readlatency_avg.png, perftest_readops.png, > perftest_writelatency_99th.png, perftest_writelatency_avg.png, > perftest_writeops.png > > > This is a follow-up ticket out of CASSANDRA-14855, to make the Flusher queue > bounded, since, in the current state, items get added to the queue without > any checks on queue size, nor with any checks on netty outbound buffer to > check the isWritable state. > We are seeing this issue hit our production 3.0 clusters quite often. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated (149caf0 -> 8a04204)
This is an automated email from the ASF dual-hosted git repository. benedict pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git. from 149caf0 CassandraNetworkAuthorizer gets login privilege from RolesCache add 5a03898 Prevent client requests from blocking on executor task queue add f5fe483 Merge branch 'cassandra-3.0' into cassandra-3.11 new 8a04204 Merge branch 'cassandra-3.11' into trunk The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt| 1 + doc/native_protocol_v4.spec| 4 + src/java/org/apache/cassandra/config/Config.java | 2 + .../cassandra/config/DatabaseDescriptor.java | 30 +++ .../apache/cassandra/metrics/ClientMetrics.java| 14 ++ .../org/apache/cassandra/net/ResourceLimits.java | 70 -- .../cassandra/service/NativeTransportService.java | 17 +- .../org/apache/cassandra/transport/Connection.java | 11 + src/java/org/apache/cassandra/transport/Frame.java | 8 +- .../org/apache/cassandra/transport/Message.java| 146 +++- .../transport/RequestThreadPoolExecutor.java | 96 .../org/apache/cassandra/transport/Server.java | 64 +++-- .../apache/cassandra/transport/SimpleClient.java | 8 + .../transport/messages/StartupMessage.java | 3 + test/unit/org/apache/cassandra/cql3/CQLTester.java | 9 +- .../service/NativeTransportServiceTest.java| 3 +- .../InflightRequestPayloadTrackerTest.java | 258 + 17 files changed, 592 insertions(+), 152 deletions(-) delete mode 100644 src/java/org/apache/cassandra/transport/RequestThreadPoolExecutor.java create mode 100644 test/unit/org/apache/cassandra/transport/InflightRequestPayloadTrackerTest.java - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-3.11' into trunk
This is an automated email from the ASF dual-hosted git repository.
benedict pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit 8a04204c10eafc2c1051a336d82211899889e276
Merge: 149caf0 f5fe483
Author: Benedict Elliott Smith
AuthorDate: Mon Jul 15 15:13:53 2019 +0100
Merge branch 'cassandra-3.11' into trunk
CHANGES.txt| 1 +
doc/native_protocol_v4.spec| 4 +
src/java/org/apache/cassandra/config/Config.java | 2 +
.../cassandra/config/DatabaseDescriptor.java | 30 +++
.../apache/cassandra/metrics/ClientMetrics.java| 14 ++
.../org/apache/cassandra/net/ResourceLimits.java | 70 --
.../cassandra/service/NativeTransportService.java | 17 +-
.../org/apache/cassandra/transport/Connection.java | 11 +
src/java/org/apache/cassandra/transport/Frame.java | 8 +-
.../org/apache/cassandra/transport/Message.java| 146 +++-
.../transport/RequestThreadPoolExecutor.java | 96
.../org/apache/cassandra/transport/Server.java | 64 +++--
.../apache/cassandra/transport/SimpleClient.java | 8 +
.../transport/messages/StartupMessage.java | 3 +
test/unit/org/apache/cassandra/cql3/CQLTester.java | 9 +-
.../service/NativeTransportServiceTest.java| 3 +-
.../InflightRequestPayloadTrackerTest.java | 258 +
17 files changed, 592 insertions(+), 152 deletions(-)
diff --cc src/java/org/apache/cassandra/config/Config.java
index 6b487fe,1d79e2a..34a5ce8
--- a/src/java/org/apache/cassandra/config/Config.java
+++ b/src/java/org/apache/cassandra/config/Config.java
@@@ -182,11 -160,13 +182,13 @@@ public class Confi
public int native_transport_max_frame_size_in_mb = 256;
public volatile long native_transport_max_concurrent_connections = -1L;
public volatile long native_transport_max_concurrent_connections_per_ip =
-1L;
-public boolean native_transport_flush_in_batches_legacy = true;
+public boolean native_transport_flush_in_batches_legacy = false;
+public volatile boolean native_transport_allow_older_protocols = true;
+public int native_transport_frame_block_size_in_kb = 32;
+ public volatile long
native_transport_max_concurrent_requests_in_bytes_per_ip = -1L;
+ public volatile long native_transport_max_concurrent_requests_in_bytes =
-1L;
-@Deprecated
-public int thrift_max_message_length_in_mb = 16;
/**
* Max size of values in SSTables, in MegaBytes.
* Default is the same as the native protocol frame limit: 256Mb.
diff --cc src/java/org/apache/cassandra/config/DatabaseDescriptor.java
index bb92716,75296b6..0166c5f
--- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
+++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
@@@ -2010,31 -1849,36 +2020,51 @@@ public class DatabaseDescripto
return conf.native_transport_flush_in_batches_legacy;
}
-public static double getCommitLogSyncBatchWindow()
+public static boolean getNativeTransportAllowOlderProtocols()
{
-return conf.commitlog_sync_batch_window_in_ms;
+return conf.native_transport_allow_older_protocols;
}
-public static void setCommitLogSyncBatchWindow(double windowMillis)
+public static void setNativeTransportAllowOlderProtocols(boolean
isEnabled)
{
-conf.commitlog_sync_batch_window_in_ms = windowMillis;
+conf.native_transport_allow_older_protocols = isEnabled;
+}
+
+public static int getNativeTransportFrameBlockSize()
+{
+return conf.native_transport_frame_block_size_in_kb * 1024;
+}
+
+public static double getCommitLogSyncGroupWindow()
+{
+return conf.commitlog_sync_group_window_in_ms;
+}
+
+public static void setCommitLogSyncGroupWindow(double windowMillis)
+{
+conf.commitlog_sync_group_window_in_ms = windowMillis;
}
+ public static long getNativeTransportMaxConcurrentRequestsInBytesPerIp()
+ {
+ return conf.native_transport_max_concurrent_requests_in_bytes_per_ip;
+ }
+
+ public static void
setNativeTransportMaxConcurrentRequestsInBytesPerIp(long
maxConcurrentRequestsInBytes)
+ {
+ conf.native_transport_max_concurrent_requests_in_bytes_per_ip =
maxConcurrentRequestsInBytes;
+ }
+
+ public static long getNativeTransportMaxConcurrentRequestsInBytes()
+ {
+ return conf.native_transport_max_concurrent_requests_in_bytes;
+ }
+
+ public static void setNativeTransportMaxConcurrentRequestsInBytes(long
maxConcurrentRequestsInBytes)
+ {
+ conf.native_transport_max_concurrent_requests_in_bytes =
maxConcurrentRequestsInBytes;
+ }
+
public static int getCommitLogSyncPeriod()
{
return conf.commitlog_sync_period_in_ms;
diff --cc
[jira] [Commented] (CASSANDRA-15013) Prevent client requests from blocking on executor task queue
[ https://issues.apache.org/jira/browse/CASSANDRA-15013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885255#comment-16885255 ] Benedict commented on CASSANDRA-15013: -- Thanks! Great work, I've merged it up. For future reference, it's super helpful if you can also modify CHANGES.txt, to avoid unnecessary rebasing and merging. If you can also stick to the commit message format (that I'm unsure if we've codified anywhere, and we should probably rectify), it also makes merging easier: " patch by ; reviewed by for CASSANDRA-# " > Prevent client requests from blocking on executor task queue > > > Key: CASSANDRA-15013 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15013 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Client >Reporter: Sumanth Pasupuleti >Assignee: Sumanth Pasupuleti >Priority: Normal > Labels: pull-request-available > Fix For: 3.0.19, 3.11.5, 4.0 > > Attachments: 15013-3.0.txt, 15013-3.11.txt, 15013-trunk.txt, > BlockedEpollEventLoopFromHeapDump.png, > BlockedEpollEventLoopFromThreadDump.png, RequestExecutorQueueFull.png, heap > dump showing each ImmediateFlusher taking upto 600MB.png, > perftest2_15013_base_flamegraph.svg, perftest2_15013_patch_flamegraph.svg, > perftest2_blocked_threadpool.png, perftest2_cpu_usage.png, > perftest2_heap.png, perftest2_read_latency_99th.png, > perftest2_read_latency_avg.png, perftest2_readops.png, > perftest2_write_latency_99th.png, perftest2_write_latency_avg.png, > perftest2_writeops.png, perftest_blockedthreads.png, > perftest_connections_count.png, perftest_cpu_usage.png, > perftest_heap_usage.png, perftest_readlatency_99th.png, > perftest_readlatency_avg.png, perftest_readops.png, > perftest_writelatency_99th.png, perftest_writelatency_avg.png, > perftest_writeops.png > > > This is a follow-up ticket out of CASSANDRA-14855, to make the Flusher queue > bounded, since, in the current state, items get added to the queue without > any checks on queue size, nor with any checks on netty outbound buffer to > check the isWritable state. > We are seeing this issue hit our production 3.0 clusters quite often. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15013) Prevent client requests from blocking on executor task queue
[ https://issues.apache.org/jira/browse/CASSANDRA-15013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-15013: - Fix Version/s: (was: 3.11.x) (was: 3.0.x) 3.11.5 3.0.19 Source Control Link: [5a03898c680ed6ada63901e8a4b278ccc8070717|https://github.com/apache/cassandra/tree/5a03898c680ed6ada63901e8a4b278ccc8070717] Since Version: 2.0.0 Status: Resolved (was: Ready to Commit) Resolution: Fixed > Prevent client requests from blocking on executor task queue > > > Key: CASSANDRA-15013 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15013 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Client >Reporter: Sumanth Pasupuleti >Assignee: Sumanth Pasupuleti >Priority: Normal > Labels: pull-request-available > Fix For: 3.0.19, 3.11.5, 4.0 > > Attachments: 15013-3.0.txt, 15013-3.11.txt, 15013-trunk.txt, > BlockedEpollEventLoopFromHeapDump.png, > BlockedEpollEventLoopFromThreadDump.png, RequestExecutorQueueFull.png, heap > dump showing each ImmediateFlusher taking upto 600MB.png, > perftest2_15013_base_flamegraph.svg, perftest2_15013_patch_flamegraph.svg, > perftest2_blocked_threadpool.png, perftest2_cpu_usage.png, > perftest2_heap.png, perftest2_read_latency_99th.png, > perftest2_read_latency_avg.png, perftest2_readops.png, > perftest2_write_latency_99th.png, perftest2_write_latency_avg.png, > perftest2_writeops.png, perftest_blockedthreads.png, > perftest_connections_count.png, perftest_cpu_usage.png, > perftest_heap_usage.png, perftest_readlatency_99th.png, > perftest_readlatency_avg.png, perftest_readops.png, > perftest_writelatency_99th.png, perftest_writelatency_avg.png, > perftest_writeops.png > > > This is a follow-up ticket out of CASSANDRA-14855, to make the Flusher queue > bounded, since, in the current state, items get added to the queue without > any checks on queue size, nor with any checks on netty outbound buffer to > check the isWritable state. > We are seeing this issue hit our production 3.0 clusters quite often. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-3.0 updated: Prevent client requests from blocking on executor task queue
This is an automated email from the ASF dual-hosted git repository.
benedict pushed a commit to branch cassandra-3.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/cassandra-3.0 by this push:
new 5a03898 Prevent client requests from blocking on executor task queue
5a03898 is described below
commit 5a03898c680ed6ada63901e8a4b278ccc8070717
Author: sumanthpasupuleti
AuthorDate: Mon Mar 25 08:06:13 2019 -0700
Prevent client requests from blocking on executor task queue
patch by Sumanth Pasupuleti, reviewed by Benedict for CASSANDRA-15013
---
CHANGES.txt| 1 +
doc/native_protocol_v4.spec| 4 +
src/java/org/apache/cassandra/config/Config.java | 3 +
.../cassandra/config/DatabaseDescriptor.java | 31 +++
.../apache/cassandra/metrics/ClientMetrics.java| 56 -
.../org/apache/cassandra/net/ResourceLimits.java | 245
.../cassandra/service/NativeTransportService.java | 26 +--
.../org/apache/cassandra/transport/Connection.java | 11 +
src/java/org/apache/cassandra/transport/Frame.java | 12 +-
.../org/apache/cassandra/transport/Message.java| 146 +++-
.../transport/RequestThreadPoolExecutor.java | 96
.../org/apache/cassandra/transport/Server.java | 64 --
.../apache/cassandra/transport/SimpleClient.java | 10 +
.../transport/messages/StartupMessage.java | 3 +
test/unit/org/apache/cassandra/cql3/CQLTester.java | 2 +
.../service/NativeTransportServiceTest.java| 3 +-
.../InflightRequestPayloadTrackerTest.java | 248 +
17 files changed, 811 insertions(+), 150 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index c8bd30d..68d309c 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
3.0.19
+ * Prevent client requests from blocking on executor task queue
(CASSANDRA-15013)
* Toughen up column drop/recreate type validations (CASSANDRA-15204)
* LegacyLayout should handle paging states that cross a collection column
(CASSANDRA-15201)
* Prevent RuntimeException when username or password is empty/null
(CASSANDRA-15198)
diff --git a/doc/native_protocol_v4.spec b/doc/native_protocol_v4.spec
index 02802a7..8beb77b 100644
--- a/doc/native_protocol_v4.spec
+++ b/doc/native_protocol_v4.spec
@@ -275,6 +275,9 @@ Table of Contents
mode. This mode will make all Thrift and Compact Tables to be exposed as
if
they were CQL Tables. This is optional; if not specified, the option will
not be used.
+- "THROW_ON_OVERLOAD": In case of server overloaded with too many
requests, by default the server puts
+back pressure on the client connection. Instead, the server can
send an OverloadedException error message back to
+the client if this option is set to true.
4.1.2. AUTH_RESPONSE
@@ -1175,3 +1178,4 @@ Table of Contents
* The returned in the v4 protocol is not compatible with the
v3
protocol. In other words, a returned by a node using
protocol v4
should not be used to query a node using protocol v3 (and vice-versa).
+ * Added THROW_ON_OVERLOAD startup option (Section 4.1.1).
diff --git a/src/java/org/apache/cassandra/config/Config.java
b/src/java/org/apache/cassandra/config/Config.java
index de158bd..830d3e1 100644
--- a/src/java/org/apache/cassandra/config/Config.java
+++ b/src/java/org/apache/cassandra/config/Config.java
@@ -154,6 +154,9 @@ public class Config
public volatile Long native_transport_max_concurrent_connections = -1L;
public volatile Long native_transport_max_concurrent_connections_per_ip =
-1L;
public boolean native_transport_flush_in_batches_legacy = true;
+public volatile long
native_transport_max_concurrent_requests_in_bytes_per_ip = -1L;
+public volatile long native_transport_max_concurrent_requests_in_bytes =
-1L;
+
@Deprecated
public Integer thrift_max_message_length_in_mb = 16;
diff --git a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
index db55c20..8417c39 100644
--- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
+++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
@@ -470,6 +470,17 @@ public class DatabaseDescriptor
{
throw new ConfigurationException("Missing endpoint_snitch
directive", false);
}
+
+if (conf.native_transport_max_concurrent_requests_in_bytes <= 0)
+{
+conf.native_transport_max_concurrent_requests_in_bytes =
Runtime.getRuntime().maxMemory() / 10;
+}
+
+if (conf.native_transport_max_concurrent_requests_in_bytes_per_ip <= 0)
+{
+conf.native_transport_max_concurrent_requests_in_bytes_per_ip =
Runtime.getRuntime().maxMemory() / 40;
+}
+
snitch =
[cassandra] branch cassandra-3.11 updated (ecdbef3 -> f5fe483)
This is an automated email from the ASF dual-hosted git repository. benedict pushed a change to branch cassandra-3.11 in repository https://gitbox.apache.org/repos/asf/cassandra.git. from ecdbef3 Merge branch 'cassandra-3.0' into cassandra-3.11 new 5a03898 Prevent client requests from blocking on executor task queue new f5fe483 Merge branch 'cassandra-3.0' into cassandra-3.11 The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt| 1 + doc/native_protocol_v4.spec| 4 + src/java/org/apache/cassandra/config/Config.java | 3 + .../cassandra/config/DatabaseDescriptor.java | 30 +++ .../org/apache/cassandra/metrics/AuthMetrics.java | 4 +- .../apache/cassandra/metrics/ClientMetrics.java| 52 - .../org/apache/cassandra/net/ResourceLimits.java | 245 .../cassandra/service/NativeTransportService.java | 24 +- .../org/apache/cassandra/transport/Connection.java | 11 + src/java/org/apache/cassandra/transport/Frame.java | 8 +- .../org/apache/cassandra/transport/Message.java| 146 +++- .../transport/RequestThreadPoolExecutor.java | 96 .../org/apache/cassandra/transport/Server.java | 64 -- .../apache/cassandra/transport/SimpleClient.java | 10 + .../transport/messages/StartupMessage.java | 3 + test/unit/org/apache/cassandra/cql3/CQLTester.java | 2 + .../service/NativeTransportServiceTest.java| 3 +- .../InflightRequestPayloadTrackerTest.java | 248 + 18 files changed, 805 insertions(+), 149 deletions(-) create mode 100644 src/java/org/apache/cassandra/net/ResourceLimits.java delete mode 100644 src/java/org/apache/cassandra/transport/RequestThreadPoolExecutor.java create mode 100644 test/unit/org/apache/cassandra/transport/InflightRequestPayloadTrackerTest.java - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-3.0' into cassandra-3.11
This is an automated email from the ASF dual-hosted git repository. benedict pushed a commit to branch cassandra-3.11 in repository https://gitbox.apache.org/repos/asf/cassandra.git commit f5fe483d0d8f967f3c52a60178df3ea7d43443be Merge: ecdbef3 5a03898 Author: Benedict Elliott Smith AuthorDate: Mon Jul 15 14:41:22 2019 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 CHANGES.txt| 1 + doc/native_protocol_v4.spec| 4 + src/java/org/apache/cassandra/config/Config.java | 3 + .../cassandra/config/DatabaseDescriptor.java | 30 +++ .../org/apache/cassandra/metrics/AuthMetrics.java | 4 +- .../apache/cassandra/metrics/ClientMetrics.java| 52 - .../org/apache/cassandra/net/ResourceLimits.java | 245 .../cassandra/service/NativeTransportService.java | 24 +- .../org/apache/cassandra/transport/Connection.java | 11 + src/java/org/apache/cassandra/transport/Frame.java | 8 +- .../org/apache/cassandra/transport/Message.java| 146 +++- .../transport/RequestThreadPoolExecutor.java | 96 .../org/apache/cassandra/transport/Server.java | 64 -- .../apache/cassandra/transport/SimpleClient.java | 10 + .../transport/messages/StartupMessage.java | 3 + test/unit/org/apache/cassandra/cql3/CQLTester.java | 2 + .../service/NativeTransportServiceTest.java| 3 +- .../InflightRequestPayloadTrackerTest.java | 248 + 18 files changed, 805 insertions(+), 149 deletions(-) diff --cc CHANGES.txt index 406246e,68d309c..7f5d52a --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,8 -1,5 +1,9 @@@ -3.0.19 +3.11.5 + * Fix cassandra-env.sh to use $CASSANDRA_CONF to find cassandra-jaas.config (CASSANDRA-14305) + * Fixed nodetool cfstats printing index name twice (CASSANDRA-14903) + * Add flag to disable SASI indexes, and warnings on creation (CASSANDRA-14866) +Merged from 3.0: + * Prevent client requests from blocking on executor task queue (CASSANDRA-15013) * Toughen up column drop/recreate type validations (CASSANDRA-15204) * LegacyLayout should handle paging states that cross a collection column (CASSANDRA-15201) * Prevent RuntimeException when username or password is empty/null (CASSANDRA-15198) diff --cc src/java/org/apache/cassandra/config/Config.java index 1976b95,830d3e1..1d79e2a --- a/src/java/org/apache/cassandra/config/Config.java +++ b/src/java/org/apache/cassandra/config/Config.java @@@ -137,33 -130,36 +137,36 @@@ public class Confi /* intentionally left set to true, despite being set to false in stock 2.2 cassandra.yaml we don't want to surprise Thrift users who have the setting blank in the yaml during 2.1->2.2 upgrade */ -public Boolean start_rpc = true; +public boolean start_rpc = true; public String rpc_address; public String rpc_interface; -public Boolean rpc_interface_prefer_ipv6 = false; +public boolean rpc_interface_prefer_ipv6 = false; public String broadcast_rpc_address; -public Integer rpc_port = 9160; -public Integer rpc_listen_backlog = 50; +public int rpc_port = 9160; +public int rpc_listen_backlog = 50; public String rpc_server_type = "sync"; -public Boolean rpc_keepalive = true; -public Integer rpc_min_threads = 16; -public Integer rpc_max_threads = Integer.MAX_VALUE; +public boolean rpc_keepalive = true; +public int rpc_min_threads = 16; +public int rpc_max_threads = Integer.MAX_VALUE; public Integer rpc_send_buff_size_in_bytes; public Integer rpc_recv_buff_size_in_bytes; -public Integer internode_send_buff_size_in_bytes; -public Integer internode_recv_buff_size_in_bytes; +public int internode_send_buff_size_in_bytes = 0; +public int internode_recv_buff_size_in_bytes = 0; -public Boolean start_native_transport = false; -public Integer native_transport_port = 9042; +public boolean start_native_transport = false; +public int native_transport_port = 9042; public Integer native_transport_port_ssl = null; -public Integer native_transport_max_threads = 128; -public Integer native_transport_max_frame_size_in_mb = 256; -public volatile Long native_transport_max_concurrent_connections = -1L; -public volatile Long native_transport_max_concurrent_connections_per_ip = -1L; +public int native_transport_max_threads = 128; +public int native_transport_max_frame_size_in_mb = 256; +public volatile long native_transport_max_concurrent_connections = -1L; +public volatile long native_transport_max_concurrent_connections_per_ip = -1L; public boolean native_transport_flush_in_batches_legacy = true; + public volatile long native_transport_max_concurrent_requests_in_bytes_per_ip = -1L; + public volatile long native_transport_max_concurrent_requests_in_bytes = -1L; +
[jira] [Updated] (CASSANDRA-15013) Prevent client requests from blocking on executor task queue
[ https://issues.apache.org/jira/browse/CASSANDRA-15013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-15013: - Summary: Prevent client requests from blocking on executor task queue (was: Message Flusher queue can grow unbounded, potentially running JVM out of memory) > Prevent client requests from blocking on executor task queue > > > Key: CASSANDRA-15013 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15013 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Client >Reporter: Sumanth Pasupuleti >Assignee: Sumanth Pasupuleti >Priority: Normal > Labels: pull-request-available > Fix For: 4.0, 3.0.x, 3.11.x > > Attachments: 15013-3.0.txt, 15013-3.11.txt, 15013-trunk.txt, > BlockedEpollEventLoopFromHeapDump.png, > BlockedEpollEventLoopFromThreadDump.png, RequestExecutorQueueFull.png, heap > dump showing each ImmediateFlusher taking upto 600MB.png, > perftest2_15013_base_flamegraph.svg, perftest2_15013_patch_flamegraph.svg, > perftest2_blocked_threadpool.png, perftest2_cpu_usage.png, > perftest2_heap.png, perftest2_read_latency_99th.png, > perftest2_read_latency_avg.png, perftest2_readops.png, > perftest2_write_latency_99th.png, perftest2_write_latency_avg.png, > perftest2_writeops.png, perftest_blockedthreads.png, > perftest_connections_count.png, perftest_cpu_usage.png, > perftest_heap_usage.png, perftest_readlatency_99th.png, > perftest_readlatency_avg.png, perftest_readops.png, > perftest_writelatency_99th.png, perftest_writelatency_avg.png, > perftest_writeops.png > > > This is a follow-up ticket out of CASSANDRA-14855, to make the Flusher queue > bounded, since, in the current state, items get added to the queue without > any checks on queue size, nor with any checks on netty outbound buffer to > check the isWritable state. > We are seeing this issue hit our production 3.0 clusters quite often. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15209) nodetool tablestats indexes name is written twice
Lapo Luchini created CASSANDRA-15209:
Summary: nodetool tablestats indexes name is written twice
Key: CASSANDRA-15209
URL: https://issues.apache.org/jira/browse/CASSANDRA-15209
Project: Cassandra
Issue Type: Bug
Components: Tool/nodetool
Reporter: Lapo Luchini
Tables names are printed OK, but index names always seems to be printed twice
in a row:
{code}
# nodetool tablestats -H | fgrep '(index)'
Table (index): fs_content.fsbyhashfs_content.fsbyhash
Table (index): documents.document_statusdocuments.document_status
Table (index): documents.document_statusdocuments.document_status
Table (index): certificates.certificates_cfcertificates.certificates_cf
Table (index): documents.document_id_dossierdocuments.document_id_dossier
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
