[
https://issues.apache.org/jira/browse/CASSANDRA-11745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18033579#comment-18033579
]
Nadav Har'El commented on CASSANDRA-11745:
------------------------------------------
This issue seems propose giving users the per-request option to choose page
size by bytes. I just noticed that Cassandra seems to be missing an even more
basic feature: The ability to automatically cut pages at some hard-coded
*reasonable* size limit, regardless of the requested page size. It doesn't even
matter what this limit is - it just needs to be reasonable. Let me explain:
Consider that you are using page_size=5000 (this is the default in the Python
driver), but each row size is pretty big - 100 KB. This may cause a SELECT of a
long partition to return a single page of size 500 MB. Reading a single 500 MB
page can cause Cassandra, the client, or both, or run out of memory while
holding the page of results in memory.
Both DynamoDB and ScyllaDB solved this problem by having a hard-coded page size
cutoff (both use 1MB as this cutoff). If the page grows beyond 1MB, the page is
returned even if it doesn't yet contain the number of rows requested by
page_size. Users don't care, because the whole point of paging is that you
continue to read additional pages if the has_more_pages flag is true.
Note that the "accuracy" of the page size cutoff is not critical. In DynamoDB
it is fairly accurate, but counts read size (pre-filtering, and including all
columns). In ScyllaDB it's always accurate - if you do a whole-table scan and
it reads multiple partitions in parallel, you may end up with a page somewhat
bigger than 1MB. But it's not important - the point of this cutoff is to leave
pages at reasonable sizes, not to reach any specific accurate page size.
By the way, I was surprised to discover in my tests that even
native_transport_max_frame_size doesn't seem to offer any protection against
absurdly large pages - I was able to request a page significantly larger than
native_transport_max_frame_size. When I continued to increase the size of the
returned page, I ended up crashing Cassandra with:
java.lang.RuntimeException:
org.apache.cassandra.db.rows.PartitionSerializationException: Failed to
serialize partition key '0' on table '...' in keyspace '...'.
> Add bytes limit to queries and paging
> -------------------------------------
>
> Key: CASSANDRA-11745
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11745
> Project: Apache Cassandra
> Issue Type: Improvement
> Components: Legacy/Core
> Reporter: Richard Low
> Assignee: Jacek Lewandowski
> Priority: Normal
> Fix For: 5.0.x, 5.x
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For some data models, values may be of very different sizes. When querying
> data, limit by count doesn’t work well and leads to timeouts. It would be
> much better to limit by size of the response, probably by stopping at the
> first row that goes above the limit. This applies to paging too so you can
> safely page through such data without timeout worries.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
