anishshri-db commented on code in PR #45503: URL: https://github.com/apache/spark/pull/45503#discussion_r1537035515
########## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateEncoder.scala: ########## @@ -212,6 +212,12 @@ class PrefixKeyScanStateEncoder( * We cannot support variable sized fields given the UnsafeRow format which stores variable * sized fields as offset and length pointers to the actual values, thereby changing the required * ordering. + * Note that we also support "null" values being passed for these fixed size fields. We prepend + * a single byte to indicate whether the column value is null or not. We cannot change the + * nullability on the UnsafeRow itself as the expected ordering would change if non-first + * columns are marked as null. If the first col is null, those entries will appear last in + * the iterator. If non-first columns are null, ordering based on the previous columns will Review Comment: Done- added a new test, expanded the previous test and also updated the comment a bit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
