[jira] [Updated] (CASSANDRA-12311) Propagate TombstoneOverwhelmingException to the client
[
https://issues.apache.org/jira/browse/CASSANDRA-12311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tyler Hobbs updated CASSANDRA-12311:
Component/s: Observability
> Propagate TombstoneOverwhelmingException to the client
> --
>
> Key: CASSANDRA-12311
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12311
> Project: Cassandra
> Issue Type: Improvement
> Components: Observability
>Reporter: Geoffrey Yu
>Assignee: Geoffrey Yu
>Priority: Minor
> Labels: client-impacting, doc-impacting
> Fix For: 3.10
>
> Attachments: 12311-dtest.txt, 12311-trunk-v2.txt, 12311-trunk-v3.txt,
> 12311-trunk-v4.txt, 12311-trunk-v5.txt, 12311-trunk.txt
>
>
> Right now if a data node fails to perform a read because it ran into a
> {{TombstoneOverwhelmingException}}, it only responds back to the coordinator
> node with a generic failure. Under this scheme, the coordinator won't be able
> to know exactly why the request failed and subsequently the client only gets
> a generic {{ReadFailureException}}. It would be useful to inform the client
> that their read failed because we read too many tombstones. We should have
> the data nodes reply with a failure type so the coordinator can pass this
> information to the client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12311) Propagate TombstoneOverwhelmingException to the client
[
https://issues.apache.org/jira/browse/CASSANDRA-12311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tyler Hobbs updated CASSANDRA-12311:
Resolution: Fixed
Fix Version/s: (was: 4.x)
3.10
Status: Resolved (was: Awaiting Feedback)
Okay, finally got a good dtest run. Everything looks good, so +1.
Committed as {{39df31a06a35b221f55f17ed20947a1a2e33ee1a}} to trunk. I've
opened a [dtest pull
request|https://github.com/riptano/cassandra-dtest/pull/1263] for your changes
there, which should be merged soon.
Thanks [~geoffxy]!
> Propagate TombstoneOverwhelmingException to the client
> --
>
> Key: CASSANDRA-12311
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12311
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Geoffrey Yu
>Assignee: Geoffrey Yu
>Priority: Minor
> Labels: client-impacting, doc-impacting
> Fix For: 3.10
>
> Attachments: 12311-dtest.txt, 12311-trunk-v2.txt, 12311-trunk-v3.txt,
> 12311-trunk-v4.txt, 12311-trunk-v5.txt, 12311-trunk.txt
>
>
> Right now if a data node fails to perform a read because it ran into a
> {{TombstoneOverwhelmingException}}, it only responds back to the coordinator
> node with a generic failure. Under this scheme, the coordinator won't be able
> to know exactly why the request failed and subsequently the client only gets
> a generic {{ReadFailureException}}. It would be useful to inform the client
> that their read failed because we read too many tombstones. We should have
> the data nodes reply with a failure type so the coordinator can pass this
> information to the client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12311) Propagate TombstoneOverwhelmingException to the client
[
https://issues.apache.org/jira/browse/CASSANDRA-12311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoffrey Yu updated CASSANDRA-12311:
Attachment: 12311-dtest.txt
Thanks for the help with the driver and the example test! I really appreciate
it. :)
I've attached a patch meant to be applied on top of your
{{CASSANDRA-12311-tests}} branch. It modifies
{{paging_test.py:TestPagingWithDeletions.test_failure_threshold_deletions}} and
{{write_failure_tests.py}} so that they check for the failure map when protocol
v5 is used. I also added another file, {{read_failure_tests.py}} to test read
failures due to reading too many tombstones. I modeled it after
{{write_failure_tests.py}} and added tests for protocol v3 and v4 as well.
> Propagate TombstoneOverwhelmingException to the client
> --
>
> Key: CASSANDRA-12311
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12311
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Geoffrey Yu
>Assignee: Geoffrey Yu
>Priority: Minor
> Labels: client-impacting, doc-impacting
> Fix For: 4.x
>
> Attachments: 12311-dtest.txt, 12311-trunk-v2.txt, 12311-trunk-v3.txt,
> 12311-trunk-v4.txt, 12311-trunk-v5.txt, 12311-trunk.txt
>
>
> Right now if a data node fails to perform a read because it ran into a
> {{TombstoneOverwhelmingException}}, it only responds back to the coordinator
> node with a generic failure. Under this scheme, the coordinator won't be able
> to know exactly why the request failed and subsequently the client only gets
> a generic {{ReadFailureException}}. It would be useful to inform the client
> that their read failed because we read too many tombstones. We should have
> the data nodes reply with a failure type so the coordinator can pass this
> information to the client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12311) Propagate TombstoneOverwhelmingException to the client
[
https://issues.apache.org/jira/browse/CASSANDRA-12311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoffrey Yu updated CASSANDRA-12311:
Attachment: 12311-trunk-v5.txt
Thanks! I've attached a patch with changes to the documentation and also unit
tests to cover the serialization and deserialization of read/write failure
error messages.
I took a look at the dtests but, since this changes the encoding for the client
facing protocol, won't the python driver will need to be changed first to
recognize the new failure code map?
> Propagate TombstoneOverwhelmingException to the client
> --
>
> Key: CASSANDRA-12311
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12311
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Geoffrey Yu
>Assignee: Geoffrey Yu
>Priority: Minor
> Labels: client-impacting, doc-impacting
> Fix For: 4.x
>
> Attachments: 12311-trunk-v2.txt, 12311-trunk-v3.txt,
> 12311-trunk-v4.txt, 12311-trunk-v5.txt, 12311-trunk.txt
>
>
> Right now if a data node fails to perform a read because it ran into a
> {{TombstoneOverwhelmingException}}, it only responds back to the coordinator
> node with a generic failure. Under this scheme, the coordinator won't be able
> to know exactly why the request failed and subsequently the client only gets
> a generic {{ReadFailureException}}. It would be useful to inform the client
> that their read failed because we read too many tombstones. We should have
> the data nodes reply with a failure type so the coordinator can pass this
> information to the client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12311) Propagate TombstoneOverwhelmingException to the client
[
https://issues.apache.org/jira/browse/CASSANDRA-12311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoffrey Yu updated CASSANDRA-12311:
Attachment: 12311-trunk-v4.txt
I've attached a patch with {{failures}} removed. I removed it from the
exceptions themselves, which does have the implication that we lose some
information when decoding an {{ErrorMessage}} while using protocol v4 (i.e. we
can't meaningfully re-create the failure reason code map with just the number
of failures). I feel that this is okay since as far as I'm aware, decoding the
number of failures is meaningful (in this codebase) only when it is actually
being used by {{o.a.c.transport.Client}} client-side. Let me know if I should
change this.
I'll get working on the dtests and update here once I have them done.
> Propagate TombstoneOverwhelmingException to the client
> --
>
> Key: CASSANDRA-12311
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12311
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Geoffrey Yu
>Assignee: Geoffrey Yu
>Priority: Minor
> Labels: client-impacting, doc-impacting
> Fix For: 4.x
>
> Attachments: 12311-trunk-v2.txt, 12311-trunk-v3.txt,
> 12311-trunk-v4.txt, 12311-trunk.txt
>
>
> Right now if a data node fails to perform a read because it ran into a
> {{TombstoneOverwhelmingException}}, it only responds back to the coordinator
> node with a generic failure. Under this scheme, the coordinator won't be able
> to know exactly why the request failed and subsequently the client only gets
> a generic {{ReadFailureException}}. It would be useful to inform the client
> that their read failed because we read too many tombstones. We should have
> the data nodes reply with a failure type so the coordinator can pass this
> information to the client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12311) Propagate TombstoneOverwhelmingException to the client
[
https://issues.apache.org/jira/browse/CASSANDRA-12311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoffrey Yu updated CASSANDRA-12311:
Attachment: 12311-trunk-v3.txt
Those ideas sound good to me. I can see how having extensibility in the failure
codes can be useful so that we don't need to wait for protocol version bumps.
Also passing back an endpoint to failure code map would be nice since we won't
need to interpret the potentially different responses from the replicas at the
coordinator to determine which (single) failure code should be used.
I attached a patch with those changes incorporated. Since we need to pass some
sort of failure code back from the replicas, I wanted to use the same set of
failure codes between nodes as between the client and coordinator. So I placed
the codes in a new enum {{RequestFailureReason}} and placed the map under
{{RequestFailureException}}, meaning {{WriteFailureException}}s will carry this
endpoint to failure code map as well. Please let me know what you think.
> Propagate TombstoneOverwhelmingException to the client
> --
>
> Key: CASSANDRA-12311
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12311
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Geoffrey Yu
>Assignee: Geoffrey Yu
>Priority: Minor
> Labels: client-impacting, doc-impacting
> Fix For: 4.x
>
> Attachments: 12311-trunk-v2.txt, 12311-trunk-v3.txt, 12311-trunk.txt
>
>
> Right now if a data node fails to perform a read because it ran into a
> {{TombstoneOverwhelmingException}}, it only responds back to the coordinator
> node with a generic failure. Under this scheme, the coordinator won't be able
> to know exactly why the request failed and subsequently the client only gets
> a generic {{ReadFailureException}}. It would be useful to inform the client
> that their read failed because we read too many tombstones. We should have
> the data nodes reply with a failure type so the coordinator can pass this
> information to the client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12311) Propagate TombstoneOverwhelmingException to the client
[
https://issues.apache.org/jira/browse/CASSANDRA-12311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tyler Hobbs updated CASSANDRA-12311:
Labels: client-impacting doc-impacting (was: )
> Propagate TombstoneOverwhelmingException to the client
> --
>
> Key: CASSANDRA-12311
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12311
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Geoffrey Yu
>Assignee: Geoffrey Yu
>Priority: Minor
> Labels: client-impacting, doc-impacting
> Fix For: 4.x
>
> Attachments: 12311-trunk-v2.txt, 12311-trunk.txt
>
>
> Right now if a data node fails to perform a read because it ran into a
> {{TombstoneOverwhelmingException}}, it only responds back to the coordinator
> node with a generic failure. Under this scheme, the coordinator won't be able
> to know exactly why the request failed and subsequently the client only gets
> a generic {{ReadFailureException}}. It would be useful to inform the client
> that their read failed because we read too many tombstones. We should have
> the data nodes reply with a failure type so the coordinator can pass this
> information to the client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12311) Propagate TombstoneOverwhelmingException to the client
[
https://issues.apache.org/jira/browse/CASSANDRA-12311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tyler Hobbs updated CASSANDRA-12311:
Status: Open (was: Patch Available)
> Propagate TombstoneOverwhelmingException to the client
> --
>
> Key: CASSANDRA-12311
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12311
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Geoffrey Yu
>Assignee: Geoffrey Yu
>Priority: Minor
> Labels: client-impacting, doc-impacting
> Fix For: 4.x
>
> Attachments: 12311-trunk-v2.txt, 12311-trunk.txt
>
>
> Right now if a data node fails to perform a read because it ran into a
> {{TombstoneOverwhelmingException}}, it only responds back to the coordinator
> node with a generic failure. Under this scheme, the coordinator won't be able
> to know exactly why the request failed and subsequently the client only gets
> a generic {{ReadFailureException}}. It would be useful to inform the client
> that their read failed because we read too many tombstones. We should have
> the data nodes reply with a failure type so the coordinator can pass this
> information to the client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12311) Propagate TombstoneOverwhelmingException to the client
[
https://issues.apache.org/jira/browse/CASSANDRA-12311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tyler Hobbs updated CASSANDRA-12311:
Status: Awaiting Feedback (was: Open)
> Propagate TombstoneOverwhelmingException to the client
> --
>
> Key: CASSANDRA-12311
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12311
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Geoffrey Yu
>Assignee: Geoffrey Yu
>Priority: Minor
> Labels: client-impacting, doc-impacting
> Fix For: 4.x
>
> Attachments: 12311-trunk-v2.txt, 12311-trunk.txt
>
>
> Right now if a data node fails to perform a read because it ran into a
> {{TombstoneOverwhelmingException}}, it only responds back to the coordinator
> node with a generic failure. Under this scheme, the coordinator won't be able
> to know exactly why the request failed and subsequently the client only gets
> a generic {{ReadFailureException}}. It would be useful to inform the client
> that their read failed because we read too many tombstones. We should have
> the data nodes reply with a failure type so the coordinator can pass this
> information to the client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12311) Propagate TombstoneOverwhelmingException to the client
[
https://issues.apache.org/jira/browse/CASSANDRA-12311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoffrey Yu updated CASSANDRA-12311:
Attachment: 12311-trunk-v2.txt
I've attached an updated patch that removes the new exception and instead adds
a new {{reason}} field within {{ReadFailureException}} that can be used to
indicate why the read query failed.
> Propagate TombstoneOverwhelmingException to the client
> --
>
> Key: CASSANDRA-12311
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12311
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Geoffrey Yu
>Assignee: Geoffrey Yu
>Priority: Minor
> Fix For: 4.x
>
> Attachments: 12311-trunk-v2.txt, 12311-trunk.txt
>
>
> Right now if a data node fails to perform a read because it ran into a
> {{TombstoneOverwhelmingException}}, it only responds back to the coordinator
> node with a generic failure. Under this scheme, the coordinator won't be able
> to know exactly why the request failed and subsequently the client only gets
> a generic {{ReadFailureException}}. It would be useful to inform the client
> that their read failed because we read too many tombstones. We should have
> the data nodes reply with a failure type so the coordinator can pass this
> information to the client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12311) Propagate TombstoneOverwhelmingException to the client
[
https://issues.apache.org/jira/browse/CASSANDRA-12311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joshua McKenzie updated CASSANDRA-12311:
Reviewer: Tyler Hobbs
> Propagate TombstoneOverwhelmingException to the client
> --
>
> Key: CASSANDRA-12311
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12311
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Geoffrey Yu
>Assignee: Geoffrey Yu
>Priority: Minor
> Fix For: 4.x
>
> Attachments: 12311-trunk.txt
>
>
> Right now if a data node fails to perform a read because it ran into a
> {{TombstoneOverwhelmingException}}, it only responds back to the coordinator
> node with a generic failure. Under this scheme, the coordinator won't be able
> to know exactly why the request failed and subsequently the client only gets
> a generic {{ReadFailureException}}. It would be useful to inform the client
> that their read failed because we read too many tombstones. We should have
> the data nodes reply with a failure type so the coordinator can pass this
> information to the client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12311) Propagate TombstoneOverwhelmingException to the client
[
https://issues.apache.org/jira/browse/CASSANDRA-12311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoffrey Yu updated CASSANDRA-12311:
Description: Right now if a data node fails to perform a read because it
ran into a {{TombstoneOverwhelmingException}}, it only responds back to the
coordinator node with a generic failure. Under this scheme, the coordinator
won't be able to know exactly why the request failed and subsequently the
client only gets a generic {{ReadFailureException}}. It would be useful to
inform the client that their read failed because we read too many tombstones.
We should have the data nodes reply with a failure type so the coordinator can
pass this information to the client. (was: Right now if a data node fails to
perform a read because it ran into a TombstoneOverwhelmingException, it only
responds back to the coordinator node with a generic failure. Under this
scheme, the coordinator won't be able to know exactly why the request failed
and subsequently the client only gets a generic ReadFailureException. It would
be useful to inform the client that their read failed because we read too many
tombstones. We should have the data nodes reply with a failure type so the
coordinator can pass this information to the client.)
> Propagate TombstoneOverwhelmingException to the client
> --
>
> Key: CASSANDRA-12311
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12311
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Geoffrey Yu
>Assignee: Geoffrey Yu
>Priority: Minor
> Fix For: 4.x
>
> Attachments: 12311-trunk.txt
>
>
> Right now if a data node fails to perform a read because it ran into a
> {{TombstoneOverwhelmingException}}, it only responds back to the coordinator
> node with a generic failure. Under this scheme, the coordinator won't be able
> to know exactly why the request failed and subsequently the client only gets
> a generic {{ReadFailureException}}. It would be useful to inform the client
> that their read failed because we read too many tombstones. We should have
> the data nodes reply with a failure type so the coordinator can pass this
> information to the client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12311) Propagate TombstoneOverwhelmingException to the client
[ https://issues.apache.org/jira/browse/CASSANDRA-12311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Yu updated CASSANDRA-12311: Attachment: 12311-trunk.txt > Propagate TombstoneOverwhelmingException to the client > -- > > Key: CASSANDRA-12311 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12311 > Project: Cassandra > Issue Type: Improvement >Reporter: Geoffrey Yu >Assignee: Geoffrey Yu >Priority: Minor > Attachments: 12311-trunk.txt > > > Right now if a data node fails to perform a read because it ran into a > TombstoneOverwhelmingException, it only responds back to the coordinator node > with a generic failure. Under this scheme, the coordinator won't be able to > know exactly why the request failed and subsequently the client only gets a > generic ReadFailureException. It would be useful to inform the client that > their read failed because we read too many tombstones. We should have the > data nodes reply with a failure type so the coordinator can pass this > information to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12311) Propagate TombstoneOverwhelmingException to the client
[ https://issues.apache.org/jira/browse/CASSANDRA-12311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Yu updated CASSANDRA-12311: Fix Version/s: 4.x Status: Patch Available (was: Open) I've attached a proposed patch that implements these changes. It adds a new exception code and also makes changes to internode messaging, so I've marked it for 4.x. > Propagate TombstoneOverwhelmingException to the client > -- > > Key: CASSANDRA-12311 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12311 > Project: Cassandra > Issue Type: Improvement >Reporter: Geoffrey Yu >Assignee: Geoffrey Yu >Priority: Minor > Fix For: 4.x > > Attachments: 12311-trunk.txt > > > Right now if a data node fails to perform a read because it ran into a > TombstoneOverwhelmingException, it only responds back to the coordinator node > with a generic failure. Under this scheme, the coordinator won't be able to > know exactly why the request failed and subsequently the client only gets a > generic ReadFailureException. It would be useful to inform the client that > their read failed because we read too many tombstones. We should have the > data nodes reply with a failure type so the coordinator can pass this > information to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
