HeartSaVioR commented on code in PR #45503: URL: https://github.com/apache/spark/pull/45503#discussion_r1536586286
########## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateEncoder.scala: ########## @@ -212,6 +212,12 @@ class PrefixKeyScanStateEncoder( * We cannot support variable sized fields given the UnsafeRow format which stores variable * sized fields as offset and length pointers to the actual values, thereby changing the required * ordering. + * Note that we also support "null" values being passed for these fixed size fields. We prepend + * a single byte to indicate whether the column value is null or not. We cannot change the + * nullability on the UnsafeRow itself as the expected ordering would change if non-first + * columns are marked as null. If the first col is null, those entries will appear last in + * the iterator. If non-first columns are null, ordering based on the previous columns will Review Comment: What I meant is, the columns after null values should be still ordered. For the test, we should still be able to iterate the data of (null, 1), (null, 3), (null, 2) as ordered, (null, 1), (null, 2), (null, 3). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
