neilramaswamy commented on PR #45503: URL: https://github.com/apache/spark/pull/45503#issuecomment-2013940433
Just thought of a question around the null bitset. I believe the format of the encoded rows is as follows: `[length of range scan portion + version byte, version byte, range scan unsafe row contents, version byte, rest of row contents]` If the range scanner is comprised of two columns, perhaps `col1` and `col2`, if we have a row `r` for which `col1` is defined but `col2` is null, then doing a range scan of that will make the null bit-set of the range scan UnsafeRow greater than zero, which means that you will never get back anything, even though in that case you should probably get back all the records whose `col1` value is greater than `r["col1"]`. Is this true? Do we just assume that rows should have non-null values, or if they do, we don't have any guarantee? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
