RexXiong commented on PR #2979: URL: https://github.com/apache/celeborn/pull/2979#issuecomment-2572752828
After a deep check Flink's implementation, I found that the header/data written by Flink is in big-endian format, while the header written by Spark using Platform(Unsafe) may be in little-endian format. This could lead to compatibility issues when the server reads the header in the same way. Possible solutions include selecting the appropriate way to read the data based on the PartitionType or adding a new field, byteOrder, to explicitly inform the server of the endianness used in writing. Personally I prefer the first solution. also cc @reswqa @codenohup -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
