anishshri-db commented on code in PR #45778: URL: https://github.com/apache/spark/pull/45778#discussion_r1547194931
########## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateEncoder.scala: ########## @@ -206,19 +208,24 @@ class PrefixKeyScanStateEncoder( * for the range scan into an UnsafeRow; we then rewrite that UnsafeRow's fields in BIG_ENDIAN * to allow for scanning keys in sorted order using the byte-wise comparison method that * RocksDB uses. + * * Then, for the rest of the fields, we project those into another UnsafeRow. * We then effectively join these two UnsafeRows together, and finally take those bytes * to get the resulting row. + * * We cannot support variable sized fields given the UnsafeRow format which stores variable * sized fields as offset and length pointers to the actual values, thereby changing the required * ordering. + * * Note that we also support "null" values being passed for these fixed size fields. We prepend * a single byte to indicate whether the column value is null or not. We cannot change the * nullability on the UnsafeRow itself as the expected ordering would change if non-first * columns are marked as null. If the first col is null, those entries will appear last in * the iterator. If non-first columns are null, ordering based on the previous columns will * still be honored. For rows with null column values, ordering for subsequent columns - * will also be maintained within those set of rows. + * will also be maintained within those set of rows. We use the same byte to also encode whether + * the value is negative or not. For negative float/double values, we flip all the bits to ensure + * the right lexicographical ordering. Review Comment: Done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
