thisisnic commented on issue #39399: URL: https://github.com/apache/arrow/issues/39399#issuecomment-6767450553
I compared the files in the reporter's `parquet-diff-test` repo (written with pyarrow 14.0.2 and pandas 2.1.4 on all three platforms). Apart from gzip, the Ubuntu and Windows files are identical, and the macOS files differ from them only inside the `ARROW:schema` metadata value, where the `pandas` key and its value are stored in the opposite order with the same content. It isn't a difference in encoding, compression or library version. That came from `AppendKeyValue` in `cpp/src/arrow/ipc/metadata_internal.cc`, where the order the two strings were created in depended on the compiler. It was fixed in #40392 (17.0.0 and later), which also added a test that checks the serialized bytes are the same on every platform. Re-serializing the reporter's schema with pyarrow 26.0.1 on Linux now gives exactly the bytes the macOS file has. gzip files will still differ by one byte per platform: the OS field in the gzip header is 3 on Linux, 10 on Windows and 19 on macOS. That is expected. I only checked the empty files in the repo. To see what differs in any other pair of files, base64-decode the `ARROW:schema` value from the Parquet key/value metadata of each and compare the bytes. @kevinisaac which pyarrow version are you seeing this with, and on which platforms? Note that the `pandas` metadata embeds the pandas and pyarrow versions, so files written with different library versions will differ. If you see a difference with the same versions on 17.0.0 or later, a pair of small files would let us find what still differs. If there's no further information in the next month, I'll close this as fixed by #40392. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
