maedhroz commented on code in PR #2540:
URL: https://github.com/apache/cassandra/pull/2540#discussion_r1284018469
##########
src/java/org/apache/cassandra/index/sai/disk/v1/sortedterms/SortedTermsWriter.java:
##########
@@ -49,9 +43,9 @@
* <p>
* For documentation of the underlying on-disk data structures, see the
package documentation.
* <p>
- * The TERMS_DICT_ constants allow for quickly determining the id of the
current block based on a point id
- * or to check if we are exactly at the beginning of the block.
- * Terms data are organized in blocks of (2 ^ {@link #TERMS_DICT_BLOCK_SHIFT})
terms.
+ * The {@code cassandra.sai.sorted_terms_block_shift} property is used to
quickly determine the id of the current block
+ * based on a point id or to check if we are exactly at the beginning of the
block.
+ * Terms data are organized in blocks of (2 ^ {@link #blockShift}) terms.
* The blocks should not be too small because they allow prefix compression of
* the terms except the first term in a block.
Review Comment:
One of the things I've been mulling around with respect to the whole sorted
terms structure is that it seems like we miss some opportunities for
compression as partition size increases. Let's say we have 16 terms per block,
and a simple `(k, c) PRIMARY KEY`. If there are 16 rows in a partition at `k =
5`, we compress away those partitions optimally. If we have 64 rows in that
partition, we're now storing `5` four times. Of course, the identical partition
across the 4 blocks means we don't degrade potential prefix compression for
`c`, but there might be some ways we can do better?
This obviously complicates some things, but let's say we wrote two
structures: one for the partition key, and one for the clustering keys. We
would be able to control block size separately, if that had potential to prefix
compress partition keys better. The other thing, though, is that when we
seek/skip within a partition, we could read from only the clustering key
structure, given it's indexed the same way in row ID space.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
