srowen commented on issue #26548: [SPARK-29918][SQL] RecordBinaryComparator should check endianness when compared by long URL: https://github.com/apache/spark/pull/26548#issuecomment-554520499 Is https://issues.apache.org/jira/browse/SPARK-23207 actually the same issue? that was marked fixed. So, hm, do we not also have a subtle problem with the signed nature of bytes and longs? Putting aside endianness issues, if you compare two longs starting with bytes like: 0000 0000 1000 0000 ... 0000 0000 0000 0000 ... Then the first one is larger than the second of course. But comparing them byte by byte, the first byte matches, but then comparing the second bytes, we compare -128 to 0 and judge that the second is larger. The difference is the 1 bit is a leading (sign) bit when read as a byte, but doesn't work that way inside a 64-bit long. Would this also be an issue between aligned and unaligned machines? That said, I also don't know how common it is to mix this hardware? (I don't even know which machines do and don't support aligned access.) Is it common?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
