manuzhang opened a new issue, #1000: URL: https://github.com/apache/iceberg-cpp/issues/1000
## Summary Junwang (@zhjwpku) raised the big-endian question in the [review of #998](https://github.com/apache/iceberg-cpp/pull/998#discussion_r4239955204), which prompted this investigation. An endian audit of `main` at `44e29e109e461e7c9069855c9a90442002580d23` found four compatibility problems on big-endian systems: 1. Bucket hashes differ from Iceberg's specified values. 2. Both Avro decimal writers produce incorrect big-endian decimal bytes. 3. The configured Avro dependency uses native byte order for float/double serialization. 4. The configured CRoaring dependency's portable serialization is little-endian-only, affecting deletion-vector bitmap payloads. These are pre-existing findings, independent of #998. Representative failures were reproduced by executing actual s390x big-endian binaries under QEMU. This was a focused reproduction, **not a full Iceberg build or integration test**. ## Verification scope - Ubuntu 22.04.5 ARM64 container on a Mac; QEMU 6.2.0; s390x GCC/G++ 11.4.0; statically linked target binaries. - ELF headers identify IBM S/390, big-endian, and the harness checks native byte order at runtime. No physical big-endian hardware was used. - Native little-endian control tests pass. - Murmur: complete, unchanged Iceberg source; blob `14cedabf4ac7b4ece349b52d05e254b86d2a5e8d` matches upstream. - Decimal: actual Arrow 25.0.0 `BasicDecimal128` storage plus the original statements from both Iceberg writers. Full Arrow arrays and Iceberg writers were not instantiated. - Avro: four unchanged float/double routine bodies from pinned commit `209a3735ec330679790824da54b7558db55e4e7f`, using byte-copy stream substitutes. The full Avro library was not linked. - CRoaring: complete, unchanged 4.4.3 library and its original C++ wrapper, with no stubs. Dependency findings below apply to the configured bundled versions. Custom dependency versions may differ. ## 1. Bucket hashing reads little-endian input as native words [`getblock32` and `getblock64`](https://github.com/apache/iceberg-cpp/blob/44e29e109e461e7c9069855c9a90442002580d23/src/iceberg/util/murmurhash3_internal.cc#L49-L56) return `p[i]` without endian conversion. The [x86_32 block loop](https://github.com/apache/iceberg-cpp/blob/44e29e109e461e7c9069855c9a90442002580d23/src/iceberg/util/murmurhash3_internal.cc#L99-L102) uses these native loads. Although `HashLong` prepares little-endian input, the block reads interpret it incorrectly on a big-endian host. Observed signed 32-bit hashes and `bucket[100]` results: | Input | Expected hash | Actual s390x hash | Expected bucket | Actual bucket | | --- | ---: | ---: | ---: | ---: | | int/long `34` | 2017239379 | 1403955681 | 79 | 81 | | binary `00 01 02 03` | -188683207 | 749933555 | 41 | 55 | | string `iceberg` | 1210000089 | 1578596666 | 89 | 66 | The UUID golden vector also fails; a three-byte tail-only control passes. The failure is consistent with native full-block loads, rather than the byte-based tail handling. This affects bucket transforms for numeric/date/time/timestamp/UUID inputs and byte/string inputs with full four-byte blocks. The same underlying hash code also affects object-store path hashing, with a lower interoperability impact. Suggested direction: explicit little-endian, alignment-safe block loads, consistent with Iceberg's Appendix B 32-bit hash definition. Merely converting the input buffer to little-endian does not fix the subsequent native loads. ## 2. Avro decimal writers unconditionally reverse native bytes Both [direct encoder](https://github.com/apache/iceberg-cpp/blob/44e29e109e461e7c9069855c9a90442002580d23/src/iceberg/avro/avro_direct_encoder.cc#L192-L202) and [data utility writer](https://github.com/apache/iceberg-cpp/blob/44e29e109e461e7c9069855c9a90442002580d23/src/iceberg/avro/avro_data_util.cc#L756-L764) reverse the native decimal bytes unconditionally. For a 16-byte decimal with unscaled value `1`: ```text Expected big-endian wire: 00000000000000000000000000000001 Actual s390x wire: 01000000000000000000000000000000 Actual interpreted value: 2^120 ``` On big-endian Arrow storage, the reversal corrupts the intended big-endian wire representation. Suggested direction: reverse only on little-endian hosts, or use an explicit big-endian conversion. The reader-side nanoarrow decimal copy is not the source of this failure. ## 3. Pinned Avro float/double routines use native wire order The configured Avro revision's [encoder](https://github.com/apache/avro/blob/209a3735ec330679790824da54b7558db55e4e7f/lang/c++/impl/BinaryEncoder.cc#L81-L88) and [decoder](https://github.com/apache/avro/blob/209a3735ec330679790824da54b7558db55e4e7f/lang/c++/impl/BinaryDecoder.cc#L92-L101) copy native floating-point bytes, while Avro requires little-endian wire order. | Value | Expected wire hex | Actual s390x wire hex | s390x decoding of valid little-endian wire | | --- | --- | --- | --- | | float `1.0` | `0000803f` | `3f800000` | `4.6006029882248069e-41` | | double `1.0` | `000000000000f03f` | `3ff0000000000000` | `3.0386519416174186e-319` | Same-host encode/decode round trips pass, hiding the cross-endian incompatibility. Both Iceberg Avro backends use the affected dependency. Suggested direction: fix/update the dependency or explicitly reject unsupported big-endian configurations until this is resolved. ## 4. CRoaring portable serialization is incompatible on big-endian hosts Iceberg [writes](https://github.com/apache/iceberg-cpp/blob/44e29e109e461e7c9069855c9a90442002580d23/src/iceberg/deletes/roaring_position_bitmap.cc#L62-L68) and [reads](https://github.com/apache/iceberg-cpp/blob/44e29e109e461e7c9069855c9a90442002580d23/src/iceberg/deletes/roaring_position_bitmap.cc#L253-L257) the inner bitmap through CRoaring's portable serialization APIs. With the complete CRoaring 4.4.3 library and bitmap `{1}`, the s390x execution: - emits bytes different from the valid little-endian portable fixture; - rejects that valid little-endian fixture when reading; - successfully round-trips its own output. CRoaring's [4.4.3 README](https://github.com/RoaringBitmap/CRoaring/blob/v4.4.3/README.md#L97-L99) explicitly limits serialization support to little-endian systems. Its [writer](https://github.com/RoaringBitmap/CRoaring/blob/v4.4.3/src/roaring_array.c#L467-L514) and [reader](https://github.com/RoaringBitmap/CRoaring/blob/v4.4.3/src/roaring_array.c#L620-L647) use native byte order. Iceberg's outer deletion-vector header endian handling is correct; the inner serialized bitmap remains incompatible. Suggested direction: guard big-endian use of this path until the dependency provides compatible serialization. ## Regression coverage Cross-endian golden fixtures are needed in addition to same-host round trips: bucket hash vectors, decimal wire bytes, Avro float/double fixtures, and deletion-vector bitmap fixtures. A full Iceberg integration run on a big-endian target remains untested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
