DuanRuixiao opened a new issue, #17501:
URL: https://github.com/apache/iceberg/issues/17501
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Snowflake
### Please describe the bug 🐞
## Bug description
`ValueWriters.FixedByteBufferWriter.write(ByteBuffer, Encoder)` calls
`encoder.writeBytes(bytes)`
instead of `encoder.writeFixed(arr, 0, length)`. For Avro `fixed[N]` fields
this is wrong:
`writeBytes` prepends a zigzag-encoded length (e.g. `0x06` for a 3-byte
value) before the data,
but the Avro reader consumes exactly N bytes for a `fixed` field and has no
length prefix to skip.
The stray prefix byte spills into the next field in the record.
**Affected version:** 1.11.0 (also present in prior versions)
## Where the bug is
`core/src/main/java/org/apache/iceberg/avro/ValueWriters.java`, inner class
`FixedByteBufferWriter`:
```java
private static class FixedByteBufferWriter implements
ValueWriter<ByteBuffer> {
private final int length;
@Override
public void write(ByteBuffer bytes, Encoder encoder) throws IOException {
Preconditions.checkArgument(bytes.remaining() == length, ...);
encoder.writeBytes(bytes); // ← BUG: should be writeFixed
}
}
```
The sibling class `FixedWriter` (which handles `byte[]`) already does it
correctly:
```java
private static class FixedWriter implements ValueWriter<byte[]> {
@Override
public void write(byte[] bytes, Encoder encoder) throws IOException {
Preconditions.checkArgument(bytes.length == length, ...);
encoder.writeFixed(bytes); // ← correct
}
}
```
## Concrete corruption example
Writing a `FIXED(3)` partition value `[AB CD EF]` via
`FixedByteBufferWriter`:
| Encoding | Output bytes | What the reader sees |
|---|---|---|
| `writeBytes` (current) | `06 AB CD EF` | `fixed[3]` consumes `06 AB CD`;
`EF` spills into the next field |
| `writeFixed` (correct) | `AB CD EF` | `fixed[3]` consumes `AB CD EF` ✓ |
When the spill byte `EF` prefixes the subsequent `record_count` varint `08`
(zigzag for 4 rows),
the combined varint `EF 08` decodes to raw value `1135`, which
zigzag-decodes to `-568`. Every
reader (Java, native, Spark) trusts this value from the manifest and sees
`record_count = -568`
for a file that has 4 rows.
## How to reproduce
1. Create a partitioned Iceberg table with a `FIXED(3)` identity-partition
column.
2. Write a data file via the Iceberg Java writer (triggers `InternalWriter`
→ `FixedByteBufferWriter`).
3. Read the written manifest Avro directly and inspect the
`data_file.record_count` field for the
partition entry — it will be a large negative number instead of the
actual row count.
4. Alternatively: observe that `data_file.partition.val8` in the manifest
contains `06abcd` instead
of `abcdef` for a `[AB CD EF]` partition value.
## Proposed fix
In `FixedByteBufferWriter.write()`, extract the bytes from the `ByteBuffer`
and call
`encoder.writeFixed()`:
```java
@Override
public void write(ByteBuffer bytes, Encoder encoder) throws IOException {
Preconditions.checkArgument(bytes.remaining() == length,
"Cannot write byte buffer of length %s as fixed[%s]",
bytes.remaining(), length);
byte[] arr = new byte[length];
bytes.duplicate().get(arr);
encoder.writeFixed(arr);
}
```
A regression test should assert that a round-trip write/read of a manifest
entry with a `FIXED(N)`
identity-partition column preserves the exact partition value bytes and the
correct `record_count`.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
