[jira] [Commented] (CASSANDRA-13862) Optimize Paxos prepare and propose stage for local requests
[
https://issues.apache.org/jira/browse/CASSANDRA-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172160#comment-16172160
]
Ariel Weisberg commented on CASSANDRA-13862:
I did 3 dtest runs and compared the results to trunk and they looked correct.
I'll commit when I get a chance.
> Optimize Paxos prepare and propose stage for local requests
>
>
> Key: CASSANDRA-13862
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13862
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 4.x
>
>
> Currently Paxos prepare and propose messages always go through entire
> MessagingService stack in Cassandra even if request is to be served locally,
> we can enhance and make local requests severed w/o involving
> MessagingService. Similar things are done at may [other places |
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageProxy.java#L1244]
> in Cassandra which skips MessagingService stage for local requests.
> This is what it looks like currently if we have tracing on and run Cassandra
> light weight transaction.
> {quote}
> Sending PAXOS_PREPARE message to /A.B.C.D
> [MessagingService-Outgoing-/A.B.C.D] | 2017-09-11 21:55:18.971000 | A.B.C.D
> | 15045
> …
>
> REQUEST_RESPONSE message received from /A.B.C.D
> [MessagingService-Incoming-/A.B.C.D] | 2017-09-11 21:55:18.976000 | A.B.C.D
> | 20270
> …
>
> Processing response from /A.B.C.D [SharedPool-Worker-4]
> | 2017-09-11 21:55:18.976000 | A.B.C.D | 20372
> {quote}
> Same thing applies for {{Propose stage}} as well.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13862) Optimize Paxos prepare and propose stage for local requests
[
https://issues.apache.org/jira/browse/CASSANDRA-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168362#comment-16168362
]
Jaydeepkumar Chovatia commented on CASSANDRA-13862:
---
Hi [~aweisberg]
Tests are failing because my branch was few commits behind the trunk, extremely
sorry about that.
I have just created another branch from latest trunk and put my change on top
of it. I have also tested locally and now all the materialized view tests are
passing on my new branch.
Could you please check on the following new branch?
|| Branch || uTest||
| [trunk |
https://github.com/apache/cassandra/compare/trunk...jaydeepkumar1984:13862-trunk2]
| [circleci | https://circleci.com/gh/jaydeepkumar1984/cassandra/20] |
Jaydeep
> Optimize Paxos prepare and propose stage for local requests
>
>
> Key: CASSANDRA-13862
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13862
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 4.x
>
>
> Currently Paxos prepare and propose messages always go through entire
> MessagingService stack in Cassandra even if request is to be served locally,
> we can enhance and make local requests severed w/o involving
> MessagingService. Similar things are done at may [other places |
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageProxy.java#L1244]
> in Cassandra which skips MessagingService stage for local requests.
> This is what it looks like currently if we have tracing on and run Cassandra
> light weight transaction.
> {quote}
> Sending PAXOS_PREPARE message to /A.B.C.D
> [MessagingService-Outgoing-/A.B.C.D] | 2017-09-11 21:55:18.971000 | A.B.C.D
> | 15045
> …
>
> REQUEST_RESPONSE message received from /A.B.C.D
> [MessagingService-Incoming-/A.B.C.D] | 2017-09-11 21:55:18.976000 | A.B.C.D
> | 20270
> …
>
> Processing response from /A.B.C.D [SharedPool-Worker-4]
> | 2017-09-11 21:55:18.976000 | A.B.C.D | 20372
> {quote}
> Same thing applies for {{Propose stage}} as well.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13862) Optimize Paxos prepare and propose stage for local requests
[
https://issues.apache.org/jira/browse/CASSANDRA-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168121#comment-16168121
]
Ariel Weisberg commented on CASSANDRA-13862:
It seems like maybe there is some instability with materialized view dtests?
This is trunk, generally all the tests are stable
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-trunk-dtest/289/testReport/materialized_views_test/TestMaterializedViews/
Several failing materialized view tests here
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/311/#showFailuresLink
For instance the LWT test has no history of failing
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-trunk-dtest/289/testReport/materialized_views_test/TestMaterializedViews/lwt_test/history/
Can you run trunk and your branch locally and see if the materialized view
tests have consistent pass fail behavior?
> Optimize Paxos prepare and propose stage for local requests
>
>
> Key: CASSANDRA-13862
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13862
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 4.x
>
>
> Currently Paxos prepare and propose messages always go through entire
> MessagingService stack in Cassandra even if request is to be served locally,
> we can enhance and make local requests severed w/o involving
> MessagingService. Similar things are done at may [other places |
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageProxy.java#L1244]
> in Cassandra which skips MessagingService stage for local requests.
> This is what it looks like currently if we have tracing on and run Cassandra
> light weight transaction.
> {quote}
> Sending PAXOS_PREPARE message to /A.B.C.D
> [MessagingService-Outgoing-/A.B.C.D] | 2017-09-11 21:55:18.971000 | A.B.C.D
> | 15045
> …
>
> REQUEST_RESPONSE message received from /A.B.C.D
> [MessagingService-Incoming-/A.B.C.D] | 2017-09-11 21:55:18.976000 | A.B.C.D
> | 20270
> …
>
> Processing response from /A.B.C.D [SharedPool-Worker-4]
> | 2017-09-11 21:55:18.976000 | A.B.C.D | 20372
> {quote}
> Same thing applies for {{Propose stage}} as well.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13862) Optimize Paxos prepare and propose stage for local requests
[
https://issues.apache.org/jira/browse/CASSANDRA-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166584#comment-16166584
]
Ariel Weisberg commented on CASSANDRA-13862:
[dtests|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/311/]
> Optimize Paxos prepare and propose stage for local requests
>
>
> Key: CASSANDRA-13862
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13862
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 4.x
>
>
> Currently Paxos prepare and propose messages always go through entire
> MessagingService stack in Cassandra even if request is to be served locally,
> we can enhance and make local requests severed w/o involving
> MessagingService. Similar things are done at may [other places |
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageProxy.java#L1244]
> in Cassandra which skips MessagingService stage for local requests.
> This is what it looks like currently if we have tracing on and run Cassandra
> light weight transaction.
> {quote}
> Sending PAXOS_PREPARE message to /A.B.C.D
> [MessagingService-Outgoing-/A.B.C.D] | 2017-09-11 21:55:18.971000 | A.B.C.D
> | 15045
> …
>
> REQUEST_RESPONSE message received from /A.B.C.D
> [MessagingService-Incoming-/A.B.C.D] | 2017-09-11 21:55:18.976000 | A.B.C.D
> | 20270
> …
>
> Processing response from /A.B.C.D [SharedPool-Worker-4]
> | 2017-09-11 21:55:18.976000 | A.B.C.D | 20372
> {quote}
> Same thing applies for {{Propose stage}} as well.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13862) Optimize Paxos prepare and propose stage for local requests
[
https://issues.apache.org/jira/browse/CASSANDRA-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165400#comment-16165400
]
Jaydeepkumar Chovatia commented on CASSANDRA-13862:
---
Hi [~aweisberg]
Thanks for code review! Please find my response here:
{quote}
Thanks Jaydeepkumar. Because this is a performance improvement it can only go
in trunk and not 3.0 or 3.11.
{quote}
Sure
{quote}
Playing devil's advocate does this do much for performance? PAXOS is already so
slow due to other aspects of the implementation and it's generally always going
to require waiting for remote responses anyways. If we had a better
implementation of the rest of it then I can definitely see how it's a bigger
advantage like it is with a regular mutation which is pretty lightweight in a
lot of cases.
{quote}
It does not make much difference if we test against low transaction rate (few
hundreds) but as we start hitting millions of transactions then it makes
significant improvement. I tried test with 40K lwt/second with and without this
change, and with this patch it definitely helps with latency specifically p99
latency. I believe if we go higher transaction rate (in millions) then impact
will be bigger.
{quote}
Should the exception path here invoke an error callback or message of some
sort? Are there counters that should be incremented? I'm looking at
StorageProxy.performLocally for clues as to the right way to do this.
{quote}
[PrepareCallback|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/paxos/PrepareCallback.java#L42]
and
[ProposeCallback|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/paxos/ProposeCallback.java#L45]
have been derived from [AbstractPaxosCallback |
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/paxos/AbstractPaxosCallback.java#L33]
which is further derived from
[IAsyncCallback|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/IAsyncCallback.java#L32]
and
[IAsyncCallback|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/IAsyncCallback.java#L32]
doesn’t have a support for error callback, hence we don’t need to worry about
it. If it would have been derived from
[IAsyncCallbackWithFailure|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/IAsyncCallbackWithFailure.java#L24]
then we have to handle error callback.
No counter increment also required here.
{quote}
Will passing it off to the other thread in this situation correctly preserve
any necessary thread local tracing state?
{quote}
Yes, It will be similar to the other places where we are doing this type of
local requests.
{quote}
Is it possible to re-use the existing remote message handling path to some
degree? Right now those verb handlers are pretty trivial, but I would like to
avoid the situation where the two diverge and no one
notices. Factoring it out into a common method makes it obvious this process
occurs in two places.
{quote}
I’ve modified code to re-use existing Verb handler code and made it a common,
please review it in the updated code:
||Branch||uTest||
|[trunk|https://github.com/apache/cassandra/compare/trunk...jaydeepkumar1984:13862-trunk]|[circleci
|https://circleci.com/gh/jaydeepkumar1984/cassandra/18]|
Jaydeep
> Optimize Paxos prepare and propose stage for local requests
>
>
> Key: CASSANDRA-13862
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13862
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> Currently Paxos prepare and propose messages always go through entire
> MessagingService stack in Cassandra even if request is to be served locally,
> we can enhance and make local requests severed w/o involving
> MessagingService. Similar things are done at may [other places |
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageProxy.java#L1244]
> in Cassandra which skips MessagingService stage for local requests.
> This is what it looks like currently if we have tracing on and run Cassandra
> light weight transaction.
> {quote}
> Sending PAXOS_PREPARE message to /A.B.C.D
> [MessagingService-Outgoing-/A.B.C.D] | 2017-09-11 21:55:18.971000 | A.B.C.D
> | 15045
> …
>
> REQUEST_RESPONSE message received from /A.B.C.D
> [MessagingService-Incoming-/A.B.C.D] | 2017-09-11 21:55:18.976000 |
[jira] [Commented] (CASSANDRA-13862) Optimize Paxos prepare and propose stage for local requests
[
https://issues.apache.org/jira/browse/CASSANDRA-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165010#comment-16165010
]
Ariel Weisberg commented on CASSANDRA-13862:
Thanks Jaydeepkumar. Because this is a performance improvement it can only go
in trunk and not 3.0 or 3.11.
Playing devil's advocate does this do much for performance? PAXOS is already so
slow due to other aspects of the implementation and it's generally always going
to require waiting for remote responses anyways. If we had a better
implementation of the rest of it then I can definitely see how it's a bigger
advantage like it is with a regular mutation which is pretty lightweight in a
lot of cases.
Should the exception path here invoke an error callback or message of some
sort? Are there counters that should be incremented? I'm looking at
{{StorageProxy.performLocally}} for clues as to the right way to do this.
Will passing it off to the other thread in this situation correctly preserve
any necessary thread local tracing state?
Is it possible to re-use the existing remote message handling path to some
degree? Right now those verb handlers are pretty trivial, but I would like to
avoid the situation where the two diverge and no one notices. Factoring it out
into a common method makes it obvious this process occurs in two places.
> Optimize Paxos prepare and propose stage for local requests
>
>
> Key: CASSANDRA-13862
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13862
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
>Reporter: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> Currently Paxos prepare and propose messages always go through entire
> MessagingService stack in Cassandra even if request is to be served locally,
> we can enhance and make local requests severed w/o involving
> MessagingService. Similar things are done at may [other places |
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageProxy.java#L1244]
> in Cassandra which skips MessagingService stage for local requests.
> This is what it looks like currently if we have tracing on and run Cassandra
> light weight transaction.
> {quote}
> Sending PAXOS_PREPARE message to /A.B.C.D
> [MessagingService-Outgoing-/A.B.C.D] | 2017-09-11 21:55:18.971000 | A.B.C.D
> | 15045
> …
>
> REQUEST_RESPONSE message received from /A.B.C.D
> [MessagingService-Incoming-/A.B.C.D] | 2017-09-11 21:55:18.976000 | A.B.C.D
> | 20270
> …
>
> Processing response from /A.B.C.D [SharedPool-Worker-4]
> | 2017-09-11 21:55:18.976000 | A.B.C.D | 20372
> {quote}
> Same thing applies for {{Propose stage}} as well.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13862) Optimize Paxos prepare and propose stage for local requests
[
https://issues.apache.org/jira/browse/CASSANDRA-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163791#comment-16163791
]
Jaydeepkumar Chovatia commented on CASSANDRA-13862:
---
Hi [~aweisberg]
I see you have done similar thing for paxos's commit phase, could you please
review my code change?
Jaydeep
> Optimize Paxos prepare and propose stage for local requests
>
>
> Key: CASSANDRA-13862
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13862
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
>Reporter: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 3.0.15, 4.0
>
>
> Currently Paxos prepare and propose messages always go through entire
> MessagingService stack in Cassandra even if request is to be served locally,
> we can enhance and make local requests severed w/o involving
> MessagingService. Similar things are done at may [other places |
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageProxy.java#L1244]
> in Cassandra which skips MessagingService stage for local requests.
> This is what it looks like currently if we have tracing on and run Cassandra
> light weight transaction.
> Sending PAXOS_PREPARE message to /A.B.C.D
> [MessagingService-Outgoing-/A.B.C.D] | 2017-09-11 21:55:18.971000 | A.B.C.D
> | 15045
> …
>
> REQUEST_RESPONSE message received from /A.B.C.D
> [MessagingService-Incoming-/A.B.C.D] | 2017-09-11 21:55:18.976000 | A.B.C.D
> | 20270
> …
>
> Processing response from /A.B.C.D [SharedPool-Worker-4]
> | 2017-09-11 21:55:18.976000 | A.B.C.D | 20372
> Same thing applies for {{Propose stage}} as well.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13862) Optimize Paxos prepare and propose stage for local requests
[
https://issues.apache.org/jira/browse/CASSANDRA-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162572#comment-16162572
]
Jaydeepkumar Chovatia commented on CASSANDRA-13862:
---
Please find patch details here:
||Branch||uTest||
|[3.0 |
https://github.com/apache/cassandra/compare/trunk...jaydeepkumar1984:13862-3.0]|[https://circleci.com/gh/jaydeepkumar1984/cassandra/14]|
|[trunk|https://github.com/apache/cassandra/compare/trunk...jaydeepkumar1984:13862-trunk]|[https://circleci.com/gh/jaydeepkumar1984/cassandra/13]|
> Optimize Paxos prepare and propose stage for local requests
>
>
> Key: CASSANDRA-13862
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13862
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
>Reporter: Jaydeepkumar Chovatia
>Priority: Minor
> Fix For: 3.0.15
>
>
> Currently Paxos prepare and propose messages always go through entire
> MessagingService stack in Cassandra even if request is to be served locally,
> we can enhance and make local requests severed w/o involving
> MessagingService. Similar things are done at may [other places |
> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageProxy.java#L1244]
> in Cassandra which skips MessagingService stage for local requests.
> This is what it looks like currently if we have tracing on and run Cassandra
> light weight transaction.
> Sending PAXOS_PREPARE message to /A.B.C.D
> [MessagingService-Outgoing-/A.B.C.D] | 2017-09-11 21:55:18.971000 | A.B.C.D
> | 15045
> …
>
> REQUEST_RESPONSE message received from /A.B.C.D
> [MessagingService-Incoming-/A.B.C.D] | 2017-09-11 21:55:18.976000 | A.B.C.D
> | 20270
> …
>
> Processing response from /A.B.C.D [SharedPool-Worker-4]
> | 2017-09-11 21:55:18.976000 | A.B.C.D | 20372
> Same thing applies for {{Propose stage}} as well.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
