[jira] [Commented] (CASSANDRA-10779) Mutations do not block for completion under view lock contention
[
https://issues.apache.org/jira/browse/CASSANDRA-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15130504#comment-15130504
]
Carl Yeksigian commented on CASSANDRA-10779:
I mistakenly wrapped the ExecutionException's cause in a RuntimeException, when
we should have been propogating it. Changed to use Throwables.propagate instead
so that we get back the WriteTimeoutException instead of a RE.
> Mutations do not block for completion under view lock contention
>
>
> Key: CASSANDRA-10779
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10779
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Windows 7 64-bit, Cassandra v3.0.0, Java 1.8u60
>Reporter: Will Zhang
>Assignee: Carl Yeksigian
> Fix For: 3.0.3, 3.3
>
>
> Hi guys,
> I encountered the following warning message when I was testing to upgrade
> from v2.2.2 to v3.0.0.
> It looks like a write time-out but in an uncaught exception. Could this be an
> easy fix?
> Log file section below. Thank you!
> {code}
> WARN [SharedPool-Worker-64] 2015-11-26 14:04:24,678
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-64,10,main]: {}
> org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out -
> received only 0 responses.
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:427)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:386)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:205)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.Keyspace.lambda$apply$59(Keyspace.java:435)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.0.0.jar:3.0.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> INFO [IndexSummaryManager:1] 2015-11-26 14:41:10,527
> IndexSummaryManager.java:257 - Redistributing index summaries
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10779) Mutations do not block for completion under view lock contention
[
https://issues.apache.org/jira/browse/CASSANDRA-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15130418#comment-15130418
]
Sylvain Lebresne commented on CASSANDRA-10779:
--
bq. Agreed the test is a non-issue +1
I'm not sure I agree. That message is logged due to a timeout, which is not a
real error as far as C* is concerned, and logging like that could easily flood
the log with a not too useful message. In fact, it appears the place where the
exception is throw was explicitly trying to ignore timeouts, but the timeout is
wrapped by a {{RuntimeException}} so it get through. I think we better fix that.
In general, the MV tests don't do anything terribly nasty, so having them log
anything at ERROR is definitively a bad smell.
> Mutations do not block for completion under view lock contention
>
>
> Key: CASSANDRA-10779
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10779
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Windows 7 64-bit, Cassandra v3.0.0, Java 1.8u60
>Reporter: Will Zhang
>Assignee: Carl Yeksigian
> Fix For: 3.0.3, 3.3
>
>
> Hi guys,
> I encountered the following warning message when I was testing to upgrade
> from v2.2.2 to v3.0.0.
> It looks like a write time-out but in an uncaught exception. Could this be an
> easy fix?
> Log file section below. Thank you!
> {code}
> WARN [SharedPool-Worker-64] 2015-11-26 14:04:24,678
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-64,10,main]: {}
> org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out -
> received only 0 responses.
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:427)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:386)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:205)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.Keyspace.lambda$apply$59(Keyspace.java:435)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.0.0.jar:3.0.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> INFO [IndexSummaryManager:1] 2015-11-26 14:41:10,527
> IndexSummaryManager.java:257 - Redistributing index summaries
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10779) Mutations do not block for completion under view lock contention
[
https://issues.apache.org/jira/browse/CASSANDRA-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15130387#comment-15130387
]
T Jake Luciani commented on CASSANDRA-10779:
Agreed the test is a non-issue +1
> Mutations do not block for completion under view lock contention
>
>
> Key: CASSANDRA-10779
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10779
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Windows 7 64-bit, Cassandra v3.0.0, Java 1.8u60
>Reporter: Will Zhang
>Assignee: Carl Yeksigian
> Fix For: 3.0.3, 3.3
>
>
> Hi guys,
> I encountered the following warning message when I was testing to upgrade
> from v2.2.2 to v3.0.0.
> It looks like a write time-out but in an uncaught exception. Could this be an
> easy fix?
> Log file section below. Thank you!
> {code}
> WARN [SharedPool-Worker-64] 2015-11-26 14:04:24,678
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-64,10,main]: {}
> org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out -
> received only 0 responses.
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:427)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:386)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:205)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.Keyspace.lambda$apply$59(Keyspace.java:435)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.0.0.jar:3.0.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> INFO [IndexSummaryManager:1] 2015-11-26 14:41:10,527
> IndexSummaryManager.java:257 - Redistributing index summaries
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10779) Mutations do not block for completion under view lock contention
[
https://issues.apache.org/jira/browse/CASSANDRA-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15130363#comment-15130363
]
Carl Yeksigian commented on CASSANDRA-10779:
The test runs look good. I've opened a PR for the
[dtest|https://github.com/riptano/cassandra-dtest/pull/782] to ignore that
error; the test itself completed successfully.
> Mutations do not block for completion under view lock contention
>
>
> Key: CASSANDRA-10779
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10779
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Windows 7 64-bit, Cassandra v3.0.0, Java 1.8u60
>Reporter: Will Zhang
>Assignee: Carl Yeksigian
> Fix For: 3.0.3, 3.3
>
>
> Hi guys,
> I encountered the following warning message when I was testing to upgrade
> from v2.2.2 to v3.0.0.
> It looks like a write time-out but in an uncaught exception. Could this be an
> easy fix?
> Log file section below. Thank you!
> {code}
> WARN [SharedPool-Worker-64] 2015-11-26 14:04:24,678
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-64,10,main]: {}
> org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out -
> received only 0 responses.
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:427)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:386)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:205)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.Keyspace.lambda$apply$59(Keyspace.java:435)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.0.0.jar:3.0.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> INFO [IndexSummaryManager:1] 2015-11-26 14:41:10,527
> IndexSummaryManager.java:257 - Redistributing index summaries
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10779) Mutations do not block for completion under view lock contention
[
https://issues.apache.org/jira/browse/CASSANDRA-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15128424#comment-15128424
]
Sylvain Lebresne commented on CASSANDRA-10779:
--
I don't know if that's why you re-opened, but this seems to have broken
{{materialized_views_test.TestMaterializedViewsConsistency.single_partition_consistent_reads_after_write_test}}.
> Mutations do not block for completion under view lock contention
>
>
> Key: CASSANDRA-10779
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10779
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Windows 7 64-bit, Cassandra v3.0.0, Java 1.8u60
>Reporter: Will Zhang
>Assignee: Carl Yeksigian
> Fix For: 3.0.3, 3.3
>
>
> Hi guys,
> I encountered the following warning message when I was testing to upgrade
> from v2.2.2 to v3.0.0.
> It looks like a write time-out but in an uncaught exception. Could this be an
> easy fix?
> Log file section below. Thank you!
> {code}
> WARN [SharedPool-Worker-64] 2015-11-26 14:04:24,678
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-64,10,main]: {}
> org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out -
> received only 0 responses.
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:427)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:386)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:205)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.Keyspace.lambda$apply$59(Keyspace.java:435)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.0.0.jar:3.0.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> INFO [IndexSummaryManager:1] 2015-11-26 14:41:10,527
> IndexSummaryManager.java:257 - Redistributing index summaries
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10779) Mutations do not block for completion under view lock contention
[
https://issues.apache.org/jira/browse/CASSANDRA-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15126929#comment-15126929
]
Carl Yeksigian commented on CASSANDRA-10779:
Also CommitLogReplay; made those changes and am rerunning the tests now.
> Mutations do not block for completion under view lock contention
>
>
> Key: CASSANDRA-10779
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10779
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Windows 7 64-bit, Cassandra v3.0.0, Java 1.8u60
>Reporter: Will Zhang
>Assignee: Tyler Hobbs
> Fix For: 3.0.x, 3.x
>
>
> Hi guys,
> I encountered the following warning message when I was testing to upgrade
> from v2.2.2 to v3.0.0.
> It looks like a write time-out but in an uncaught exception. Could this be an
> easy fix?
> Log file section below. Thank you!
> {code}
> WARN [SharedPool-Worker-64] 2015-11-26 14:04:24,678
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-64,10,main]: {}
> org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out -
> received only 0 responses.
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:427)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:386)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:205)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.Keyspace.lambda$apply$59(Keyspace.java:435)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.0.0.jar:3.0.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> INFO [IndexSummaryManager:1] 2015-11-26 14:41:10,527
> IndexSummaryManager.java:257 - Redistributing index summaries
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10779) Mutations do not block for completion under view lock contention
[
https://issues.apache.org/jira/browse/CASSANDRA-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15126617#comment-15126617
]
Tyler Hobbs commented on CASSANDRA-10779:
-
There's still one more call path that needs to be updated:
{{PaxosState.commit()}} calls {{Keyspace.apply()}} and doesn't block on the
future. Other than that, the latest patch looks good to me.
> Mutations do not block for completion under view lock contention
>
>
> Key: CASSANDRA-10779
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10779
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Windows 7 64-bit, Cassandra v3.0.0, Java 1.8u60
>Reporter: Will Zhang
>Assignee: Tyler Hobbs
> Fix For: 3.0.x, 3.x
>
>
> Hi guys,
> I encountered the following warning message when I was testing to upgrade
> from v2.2.2 to v3.0.0.
> It looks like a write time-out but in an uncaught exception. Could this be an
> easy fix?
> Log file section below. Thank you!
> {code}
> WARN [SharedPool-Worker-64] 2015-11-26 14:04:24,678
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-64,10,main]: {}
> org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out -
> received only 0 responses.
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:427)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:386)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:205)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.Keyspace.lambda$apply$59(Keyspace.java:435)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.0.0.jar:3.0.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> INFO [IndexSummaryManager:1] 2015-11-26 14:41:10,527
> IndexSummaryManager.java:257 - Redistributing index summaries
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10779) Mutations do not block for completion under view lock contention
[
https://issues.apache.org/jira/browse/CASSANDRA-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15124163#comment-15124163
]
T Jake Luciani commented on CASSANDRA-10779:
No difference in the tests with the new changes. Likely due to the fact the
mvbench test runs at quorum so the coordinator was waiting on the verb response
> Mutations do not block for completion under view lock contention
>
>
> Key: CASSANDRA-10779
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10779
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Windows 7 64-bit, Cassandra v3.0.0, Java 1.8u60
>Reporter: Will Zhang
>Assignee: Tyler Hobbs
> Fix For: 3.0.x, 3.x
>
>
> Hi guys,
> I encountered the following warning message when I was testing to upgrade
> from v2.2.2 to v3.0.0.
> It looks like a write time-out but in an uncaught exception. Could this be an
> easy fix?
> Log file section below. Thank you!
> {code}
> WARN [SharedPool-Worker-64] 2015-11-26 14:04:24,678
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-64,10,main]: {}
> org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out -
> received only 0 responses.
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:427)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:386)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:205)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.Keyspace.lambda$apply$59(Keyspace.java:435)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.0.0.jar:3.0.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> INFO [IndexSummaryManager:1] 2015-11-26 14:41:10,527
> IndexSummaryManager.java:257 - Redistributing index summaries
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10779) Mutations do not block for completion under view lock contention
[
https://issues.apache.org/jira/browse/CASSANDRA-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15124020#comment-15124020
]
Carl Yeksigian commented on CASSANDRA-10779:
I pushed a change so that we only use the CompletableFuture in mutation verb
handler; otherwise we call {{.get()}} to turn it back into a synchronous call
for the other places.
[~tjake] kicked off new runs, so we'll see how it fares. We should get one or
the other committed before we code freeze 3.3.
> Mutations do not block for completion under view lock contention
>
>
> Key: CASSANDRA-10779
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10779
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Windows 7 64-bit, Cassandra v3.0.0, Java 1.8u60
>Reporter: Will Zhang
>Assignee: Tyler Hobbs
> Fix For: 3.0.x, 3.x
>
>
> Hi guys,
> I encountered the following warning message when I was testing to upgrade
> from v2.2.2 to v3.0.0.
> It looks like a write time-out but in an uncaught exception. Could this be an
> easy fix?
> Log file section below. Thank you!
> {code}
> WARN [SharedPool-Worker-64] 2015-11-26 14:04:24,678
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-64,10,main]: {}
> org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out -
> received only 0 responses.
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:427)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:386)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:205)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.Keyspace.lambda$apply$59(Keyspace.java:435)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.0.0.jar:3.0.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> INFO [IndexSummaryManager:1] 2015-11-26 14:41:10,527
> IndexSummaryManager.java:257 - Redistributing index summaries
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10779) Mutations do not block for completion under view lock contention
[
https://issues.apache.org/jira/browse/CASSANDRA-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15123891#comment-15123891
]
Tyler Hobbs commented on CASSANDRA-10779:
-
Thanks for the benchmarks, [~tjake].
I'm good with going with passing the CompletableFuture up the stack, we just
need to make sure all of the calling locations are blocking when necessary. It
looks like we're missing a few.
> Mutations do not block for completion under view lock contention
>
>
> Key: CASSANDRA-10779
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10779
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Windows 7 64-bit, Cassandra v3.0.0, Java 1.8u60
>Reporter: Will Zhang
>Assignee: Tyler Hobbs
> Fix For: 3.0.x, 3.x
>
>
> Hi guys,
> I encountered the following warning message when I was testing to upgrade
> from v2.2.2 to v3.0.0.
> It looks like a write time-out but in an uncaught exception. Could this be an
> easy fix?
> Log file section below. Thank you!
> {code}
> WARN [SharedPool-Worker-64] 2015-11-26 14:04:24,678
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-64,10,main]: {}
> org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out -
> received only 0 responses.
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:427)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:386)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:205)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.Keyspace.lambda$apply$59(Keyspace.java:435)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.0.0.jar:3.0.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> INFO [IndexSummaryManager:1] 2015-11-26 14:41:10,527
> IndexSummaryManager.java:257 - Redistributing index summaries
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10779) Mutations do not block for completion under view lock contention
[
https://issues.apache.org/jira/browse/CASSANDRA-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122464#comment-15122464
]
T Jake Luciani commented on CASSANDRA-10779:
I ran highly contended mvbench runs with 3.0 branch, Tyler's v2 branch and
Carl's completable future test.
Both thobbs and carls fixes the timeout issue, but when I add a stress write
workload running along side the view benchmark Tylers branch causes stress to
timeout because the write stage is blocked waiting on the contended locks. In
Carl's version this doesn't happen since the mutations are put back on the
queue freeing threads to work on other things. So I think we should go with
Carl's version.
In the case of an uncontended mvbench run (no lock contention) there is no
difference in performance between the current 3.0 branch and Carl's branch.
In the highly contended case as we are now waiting before acking the write the
performance has dropped to be 50% worse than non-mv. However I think we can
fix this by improving lock contention CASSANDRA-10307
> Mutations do not block for completion under view lock contention
>
>
> Key: CASSANDRA-10779
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10779
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Windows 7 64-bit, Cassandra v3.0.0, Java 1.8u60
>Reporter: Will Zhang
>Assignee: Tyler Hobbs
> Fix For: 3.0.x, 3.x
>
>
> Hi guys,
> I encountered the following warning message when I was testing to upgrade
> from v2.2.2 to v3.0.0.
> It looks like a write time-out but in an uncaught exception. Could this be an
> easy fix?
> Log file section below. Thank you!
> {code}
> WARN [SharedPool-Worker-64] 2015-11-26 14:04:24,678
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-64,10,main]: {}
> org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out -
> received only 0 responses.
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:427)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:386)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:205)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.Keyspace.lambda$apply$59(Keyspace.java:435)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.0.0.jar:3.0.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> INFO [IndexSummaryManager:1] 2015-11-26 14:41:10,527
> IndexSummaryManager.java:257 - Redistributing index summaries
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10779) Mutations do not block for completion under view lock contention
[
https://issues.apache.org/jira/browse/CASSANDRA-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122100#comment-15122100
]
Carl Yeksigian commented on CASSANDRA-10779:
I put together a limited change to use completable futures only from the
Mutation Verb handler, which seems to allow other mutations through while
fixing the issue of not blocking with contention.
[branch|https://github.com/carlyeks/cassandra/tree/ticket/10779/completableFuture]
[utest|http://cassci.datastax.com/view/Dev/view/carlyeks/job/carlyeks-ticket-10779-completableFuture-testall/]
[dtest|http://cassci.datastax.com/view/Dev/view/carlyeks/job/carlyeks-ticket-10779-completableFuture-dtest/]
> Mutations do not block for completion under view lock contention
>
>
> Key: CASSANDRA-10779
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10779
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Windows 7 64-bit, Cassandra v3.0.0, Java 1.8u60
>Reporter: Will Zhang
>Assignee: Tyler Hobbs
> Fix For: 3.0.x, 3.x
>
>
> Hi guys,
> I encountered the following warning message when I was testing to upgrade
> from v2.2.2 to v3.0.0.
> It looks like a write time-out but in an uncaught exception. Could this be an
> easy fix?
> Log file section below. Thank you!
> {code}
> WARN [SharedPool-Worker-64] 2015-11-26 14:04:24,678
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-64,10,main]: {}
> org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out -
> received only 0 responses.
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:427)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:386)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:205)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.Keyspace.lambda$apply$59(Keyspace.java:435)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.0.0.jar:3.0.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> INFO [IndexSummaryManager:1] 2015-11-26 14:41:10,527
> IndexSummaryManager.java:257 - Redistributing index summaries
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10779) Mutations do not block for completion under view lock contention
[
https://issues.apache.org/jira/browse/CASSANDRA-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15115679#comment-15115679
]
T Jake Luciani commented on CASSANDRA-10779:
I'm trying to explain the reason for non-blocking the Mutation stage. For
example this patch would block all non-mv mutations (including system
keyspaces) if we encounter (write stage pool size MV writes on the same lock).
If we decide to go with the non completable future approach then I would
suggest we reintroduce the MV Write stage. So we don't block non MV mutations.
> Mutations do not block for completion under view lock contention
>
>
> Key: CASSANDRA-10779
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10779
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Windows 7 64-bit, Cassandra v3.0.0, Java 1.8u60
>Reporter: Will Zhang
>Assignee: Tyler Hobbs
> Fix For: 3.0.x, 3.x
>
>
> Hi guys,
> I encountered the following warning message when I was testing to upgrade
> from v2.2.2 to v3.0.0.
> It looks like a write time-out but in an uncaught exception. Could this be an
> easy fix?
> Log file section below. Thank you!
> {code}
> WARN [SharedPool-Worker-64] 2015-11-26 14:04:24,678
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-64,10,main]: {}
> org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out -
> received only 0 responses.
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:427)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:386)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:205)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.Keyspace.lambda$apply$59(Keyspace.java:435)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.0.0.jar:3.0.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> INFO [IndexSummaryManager:1] 2015-11-26 14:41:10,527
> IndexSummaryManager.java:257 - Redistributing index summaries
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10779) Mutations do not block for completion under view lock contention
[
https://issues.apache.org/jira/browse/CASSANDRA-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15112923#comment-15112923
]
Carl Yeksigian commented on CASSANDRA-10779:
This ticket isn't about solving the contention, but about making the behavior
correct in the face of contention. This patch fixes that issue by making sure
we don't respond back before we have actually completed the work.
While moving to CompletableFutures would solve both, it's a far more invasive
change than this one, so it makes sense to know what the change in performance
is.
> Mutations do not block for completion under view lock contention
>
>
> Key: CASSANDRA-10779
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10779
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Windows 7 64-bit, Cassandra v3.0.0, Java 1.8u60
>Reporter: Will Zhang
>Assignee: Tyler Hobbs
> Fix For: 3.0.x, 3.x
>
>
> Hi guys,
> I encountered the following warning message when I was testing to upgrade
> from v2.2.2 to v3.0.0.
> It looks like a write time-out but in an uncaught exception. Could this be an
> easy fix?
> Log file section below. Thank you!
> {code}
> WARN [SharedPool-Worker-64] 2015-11-26 14:04:24,678
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-64,10,main]: {}
> org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out -
> received only 0 responses.
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:427)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:386)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:205)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.Keyspace.lambda$apply$59(Keyspace.java:435)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.0.0.jar:3.0.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> INFO [IndexSummaryManager:1] 2015-11-26 14:41:10,527
> IndexSummaryManager.java:257 - Redistributing index summaries
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10779) Mutations do not block for completion under view lock contention
[
https://issues.apache.org/jira/browse/CASSANDRA-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15112920#comment-15112920
]
Tyler Hobbs commented on CASSANDRA-10779:
-
Well, the _real_ issue is responding before a mutation has been applied, which
is a correctness/durability issue. We just need to make sure to fix that in a
way that doesn't severely hamper performance. I was already talking with
[~carlyeks] about benchmarking these solutions before committing (which I think
we do need to do), but I'm not convinced that this approach will have worse
performance than the current solution under contention.
> Mutations do not block for completion under view lock contention
>
>
> Key: CASSANDRA-10779
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10779
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Windows 7 64-bit, Cassandra v3.0.0, Java 1.8u60
>Reporter: Will Zhang
>Assignee: Tyler Hobbs
> Fix For: 3.0.x, 3.x
>
>
> Hi guys,
> I encountered the following warning message when I was testing to upgrade
> from v2.2.2 to v3.0.0.
> It looks like a write time-out but in an uncaught exception. Could this be an
> easy fix?
> Log file section below. Thank you!
> {code}
> WARN [SharedPool-Worker-64] 2015-11-26 14:04:24,678
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-64,10,main]: {}
> org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out -
> received only 0 responses.
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:427)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:386)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:205)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.Keyspace.lambda$apply$59(Keyspace.java:435)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.0.0.jar:3.0.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> INFO [IndexSummaryManager:1] 2015-11-26 14:41:10,527
> IndexSummaryManager.java:257 - Redistributing index summaries
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10779) Mutations do not block for completion under view lock contention
[
https://issues.apache.org/jira/browse/CASSANDRA-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15112903#comment-15112903
]
T Jake Luciani commented on CASSANDRA-10779:
I don't see how this is going to help. The issue is lock contention and
blocking just makes it worse :)
I'll run some mvbench runs with your patch to back this up. I think we need to
make the completable future approach send the response as Carl mentioned above
> Mutations do not block for completion under view lock contention
>
>
> Key: CASSANDRA-10779
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10779
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Windows 7 64-bit, Cassandra v3.0.0, Java 1.8u60
>Reporter: Will Zhang
>Assignee: Tyler Hobbs
> Fix For: 3.0.x, 3.x
>
>
> Hi guys,
> I encountered the following warning message when I was testing to upgrade
> from v2.2.2 to v3.0.0.
> It looks like a write time-out but in an uncaught exception. Could this be an
> easy fix?
> Log file section below. Thank you!
> {code}
> WARN [SharedPool-Worker-64] 2015-11-26 14:04:24,678
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-64,10,main]: {}
> org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out -
> received only 0 responses.
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:427)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:386)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:205)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.Keyspace.lambda$apply$59(Keyspace.java:435)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.0.0.jar:3.0.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> INFO [IndexSummaryManager:1] 2015-11-26 14:41:10,527
> IndexSummaryManager.java:257 - Redistributing index summaries
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10779) Mutations do not block for completion under view lock contention
[
https://issues.apache.org/jira/browse/CASSANDRA-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15112892#comment-15112892
]
Tyler Hobbs commented on CASSANDRA-10779:
-
Since {{Striped}} actually uses {{ReentrantLocks}}, it turns out a blocking
solution works for my purposes. I've pushed a second branch with blocking lock
acquisition. As discussed, it uses {{bulkGet()}} to ensure lock acquisition
order is consistent (avoiding deadlock):
||branch||testall||dtest||
|[CASSANDRA-10779-v2|https://github.com/thobbs/cassandra/tree/CASSANDRA-10779-v2]|[testall|http://cassci.datastax.com/view/Dev/view/thobbs/job/thobbs-CASSANDRA-10779-v2-testall]|[dtest|http://cassci.datastax.com/view/Dev/view/thobbs/job/thobbs-CASSANDRA-10779-v2-dtest]|
|[CASSANDRA-10779-v2-3.3|https://github.com/thobbs/cassandra/tree/CASSANDRA-10779-v2-3.3]|[testall|http://cassci.datastax.com/view/Dev/view/thobbs/job/thobbs-CASSANDRA-10779-v2-3.3-testall]|[dtest|http://cassci.datastax.com/view/Dev/view/thobbs/job/thobbs-CASSANDRA-10779-v2-3.3-dtest]|
|[CASSANDRA-10779-v2-trunk|https://github.com/thobbs/cassandra/tree/CASSANDRA-10779-v2-trunk]|[testall|http://cassci.datastax.com/view/Dev/view/thobbs/job/thobbs-CASSANDRA-10779-v2-trunk-testall]|[dtest|http://cassci.datastax.com/view/Dev/view/thobbs/job/thobbs-CASSANDRA-10779-v2-trunk-dtest]|
The trunk merge was the only non-trivial one.
> Mutations do not block for completion under view lock contention
>
>
> Key: CASSANDRA-10779
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10779
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Windows 7 64-bit, Cassandra v3.0.0, Java 1.8u60
>Reporter: Will Zhang
>Assignee: Tyler Hobbs
> Fix For: 3.0.x, 3.x
>
>
> Hi guys,
> I encountered the following warning message when I was testing to upgrade
> from v2.2.2 to v3.0.0.
> It looks like a write time-out but in an uncaught exception. Could this be an
> easy fix?
> Log file section below. Thank you!
> {code}
> WARN [SharedPool-Worker-64] 2015-11-26 14:04:24,678
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-64,10,main]: {}
> org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out -
> received only 0 responses.
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:427)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:386)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:205)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.Keyspace.lambda$apply$59(Keyspace.java:435)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.0.0.jar:3.0.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> INFO [IndexSummaryManager:1] 2015-11-26 14:41:10,527
> IndexSummaryManager.java:257 - Redistributing index summaries
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10779) Mutations do not block for completion under view lock contention
[
https://issues.apache.org/jira/browse/CASSANDRA-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15112604#comment-15112604
]
Carl Yeksigian commented on CASSANDRA-10779:
This will still block the current thread since we are calling {{.get()}} on the
Future, and also reschedule the task to happen on the {{Mutation}} stage. This
is equivalent to having a synchronous wait for the view lock, but also adds
another task to the mutation queue, taking up another thread's work.
We need to push the {{CompletableFuture}} all the way up to the network reply,
and reply if it is null or use {{thenApply}} to reply after the
{{CompletableFuture}} is completed.
> Mutations do not block for completion under view lock contention
>
>
> Key: CASSANDRA-10779
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10779
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Windows 7 64-bit, Cassandra v3.0.0, Java 1.8u60
>Reporter: Will Zhang
>Assignee: Tyler Hobbs
> Fix For: 3.0.x, 3.x
>
>
> Hi guys,
> I encountered the following warning message when I was testing to upgrade
> from v2.2.2 to v3.0.0.
> It looks like a write time-out but in an uncaught exception. Could this be an
> easy fix?
> Log file section below. Thank you!
> {code}
> WARN [SharedPool-Worker-64] 2015-11-26 14:04:24,678
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-64,10,main]: {}
> org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out -
> received only 0 responses.
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:427)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:386)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:205)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.db.Keyspace.lambda$apply$59(Keyspace.java:435)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-3.0.0.jar:3.0.0]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.0.0.jar:3.0.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> INFO [IndexSummaryManager:1] 2015-11-26 14:41:10,527
> IndexSummaryManager.java:257 - Redistributing index summaries
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
