[
https://issues.apache.org/jira/browse/CASSANDRA-11521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356834#comment-15356834
]
Stefania commented on CASSANDRA-11521:
--------------------------------------
Just a quick update since I haven't had a chance to update this ticket in over
a month.
I have an almost final implementation of the patch, I need to fix a few more
dtests, implement new unit tests to test some corner cases, and repeat the
benchmark to rule out performance regressions.
I hope to have something ready for review in the next few weeks, depending on
other tasks and the results of the benchmark and new unit tests.
The new delivery mechanism has been renamed from _streaming_ to _asynchronous
paging_.
In terms of implementation, I ended up having something somewhat more generic
than what originally stated. Specifically, optimized local reads will be
available regardless of the delivery mechanism to the client (page-by-page or
asynchronous paging). This means that CASSANDRA-11520 will be included in this
patch.
Also, asynchronous paging will be available for all queries at all consistency
levels, not just optimized load queries. This does mean however that we will
need to implement the protocol changes in the python driver as well, in order
to write dtests, cc [~aholmber] for a heads up.
When using the new asynchronous paging delivery mechanism, clients will be able
to page by rows and by bytes. If paging by bytes, the page size returned to the
client will be between:
* the requested size minus one avg row size
* the requested size plus \[the current CQL row size - an avg row size\].
The avg row size is refined for every page sent. At CL > 1, intra-node, we are
still sending requests using page sizes in rows, to calculate the number of
rows we divide the page size in bytes by an average row size.
As for the optimized local reads, for now they are simply done in
SelectStatement by releasing the op order periodically. Later, we may be able
to do something fancier if required, such as using only sstable references,
avoiding polluting the chunk cache, using a larger read ahead and so forth. For
now, I wanted to focus on keeping delivery and local reads orthogonal and
implementing the new delivery mechanism.
h3. Client protocol changes
A new flag has been added to query options:
{code}
+ 0x80: With extended flags. If set, <extended_flags> should be present.
+ - <extended_flags> is a [byte] whose bits define additional options for
this query and,
+ like <flags>, may influence the remainder of the message. Supported
extended flags:
+ 0x01: With paging options. If set, <paging_options> will be present
and the
+ query result will be pushed to the client asynchronously, and
according to
+ the paging options. Asynchronous paging can be interrupted via a
CANCEL request.
+ <paging_options> contains the following:
+ - <uuid>, a [UUID] that uniquely identifies an asynchronous
paging session.
+ - <page_size>, an [Int] indicating the size of each page in
the unit defined below.
+ This is a mandatory parameter that takes precedence over
<result_page_size>,
+ which is obsoleted by this parameter.
+ - <page_unit>, an [Int] indicating the page unit: 1 indicates
bytes and 2 indicates rows.
+ - <max_num_pages>, an [Int] indicating the maximum number of
pages to
+ receive in total, set this to zero to indicate no limit.
+ - <pages_per_second>, an [Int] indicating the maximum number
of pages to receive
+ per second, set this to zero to indicate no limit.
+
{code}
A new cancel message has been added:
{code}
+4.1.9. CANCEL
+
+ Request to cancel an asynchronous operation. The body of the message is:
+ - an [int] identifying the operation type:
+ - 1 for cancelling an async paging session
+ - a [uuid] that contains the unique identifier of the operation to cancel.
+
{code}
A new flag has been added to the response metadata:
{code}
+ 0x0005 With_asynchronous_paging: if set, this result is part of
an asynchronous
+ paging session and the <paging_params> will be present.
[...]
+ - <paging_params> contains:
+ - a [uuid] that uniquely identifies the asynchronous paging session
+ - an [int] that identifies the sequential number of this result in
the session
+ - a [boolean] that is true for the last message in the session
{code}
The paging state will be returned as usual, so clients will be able to resume
paging on any node using the old or new paging mechanism.
> Implement streaming for bulk read requests
> ------------------------------------------
>
> Key: CASSANDRA-11521
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11521
> Project: Cassandra
> Issue Type: Sub-task
> Components: Local Write-Read Paths
> Reporter: Stefania
> Assignee: Stefania
> Fix For: 3.x
>
>
> Allow clients to stream data from a C* host, bypassing the coordination layer
> and eliminating the need to query individual pages one by one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
