[
https://issues.apache.org/jira/browse/CASSANDRA-16103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217144#comment-17217144
]
David Capwell commented on CASSANDRA-16103:
-------------------------------------------
[~aholmber]
bq. For example the schema, workload, and topology? Was the cluster facing
outages or adverse conditions?
Looking at the clusters impacted, we have 3 categories: diff test (clone
cluster in 3.0 and 4.0, and diff 100% of rows), ad-hoc (random people deploying
random stuff), test cluster (user tests). Out of these 3 categories the ad-hoc
clusters hit this the most, followed by diff test (read only); the test cluster
is low traffic and only seen this a small number of times in the past week.
Diff is heavy read, that is the only workload I can talk about (because I don't
know what people do on the ad-hoc clusters).
Now, for the first test that saw this issue, I think it was the following:
* tip-stress
https://github.com/thelastpickle/tlp-stress/blob/master/src/main/kotlin/com/thelastpickle/tlpstress/profiles/KeyValue.kt
* 6 node cluster, single DC
* rate 800 qps
* concurrency 100
* consistency level LOCAL_ONE
I do not remember the read/write percentage, but seeing how this happens for
all response types... I don't think it matters
I sadly don't remember well if this is correct or not (mostly the profile used
is likely to change, the other variables are my defaults), I didn't document
this in CASSANDRA-16097 =(
> Invalid serialized size for responses
> -------------------------------------
>
> Key: CASSANDRA-16103
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16103
> Project: Cassandra
> Issue Type: Bug
> Components: Messaging/Internode
> Reporter: David Capwell
> Priority: Normal
> Fix For: 4.0-beta
>
>
> Was running a benchmark at LOCAL_ONE and eventually saw the below exception;
> this is related to CASSANDRA-16097 as it was found during the same test.
> {code}
> org.apache.cassandra.net.InvalidSerializedSizeException: Invalid serialized
> size; expected 14, actual size at least 13, for verb HINT_RSP
> at
> org.apache.cassandra.net.OutboundConnection$EventLoopDelivery.doRun(OutboundConnection.java:816)
> at
> org.apache.cassandra.net.OutboundConnection$Delivery.run(OutboundConnection.java:687)
> at
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {code}
> {code}
> org.apache.cassandra.net.InvalidSerializedSizeException: Invalid serialized
> size; expected 14, actual size at least 13, for verb MUTATION_RSP
> at
> org.apache.cassandra.net.OutboundConnection$EventLoopDelivery.doRun(OutboundConnection.java:816)
> at
> org.apache.cassandra.net.OutboundConnection$Delivery.run(OutboundConnection.java:687)
> at
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
