srowen commented on issue #26548: [SPARK-29918][SQL] RecordBinaryComparator should check endianness when compared by long URL: https://github.com/apache/spark/pull/26548#issuecomment-554645859 Yes, it isn't really related to endian-ness, although to fix this issue you have to know whether the bytes were read as little-endian or not. Here's a simple proof of concept: ``` long arrayOffset = 12; long[] arr1 = new long[2]; Platform.putLong(arr1, arrayOffset, 0xa000000000000000L); long[] arr2 = new long[2]; Platform.putLong(arr2, arrayOffset, 0x0000000000000000L); System.out.println(binaryComparator.compare(arr1, arrayOffset, 8, arr2, arrayOffset, 8)); long[] arr3 = new long[2]; Platform.putLong(arr3, arrayOffset + 4, 0xa000000000000000L); long[] arr4 = new long[2]; Platform.putLong(arr4, arrayOffset, 0x0000000000000000L); System.out.println(binaryComparator.compare(arr3, arrayOffset + 4, 8, arr4, arrayOffset, 8)); ``` This prints 1 and -1, but it's the same 8 bytes being compared with each other. In the second instance, one is offset by 4 bytes. I think this example is actually illustrating the other issue mentioned here, about unsigned long comparison. But I think you can construct a similar example to show the issue you report here. And yes this is just running locally on one machine even.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
