[
https://issues.apache.org/jira/browse/AVRO-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-4268:
---------------------------------
Labels: pull-request-available (was: )
> BytesWritableConverter serializes unused capacity bytes
> -------------------------------------------------------
>
> Key: AVRO-4268
> URL: https://issues.apache.org/jira/browse/AVRO-4268
> Project: Apache Avro
> Issue Type: Bug
> Components: java
> Reporter: Ruiqi Dong
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> *Summary*
> `AvroDatumConverterFactory.BytesWritableConverter` converts Hadoop
> `BytesWritable` to Avro `bytes` using `ByteBuffer.wrap(input.getBytes())`.
> `BytesWritable.getBytes()` returns the backing array, whose length can be
> larger than the logical value length reported by `getLength()`. The converter
> therefore serializes stale or unused capacity bytes.
>
> *Affected code*
> File:
> `lang/java/mapred/src/main/java/org/apache/avro/hadoop/io/AvroDatumConverterFactory.java`
> {code:java}
> @Override
> public ByteBuffer convert(BytesWritable input) {
> return ByteBuffer.wrap(input.getBytes());
> } {code}
> *Reproducer*
> Add this test to
> `lang/java/mapred/src/test/java/org/apache/avro/hadoop/io/TestAvroDatumConverterFactory.java`
> {code:java}
> @Test
> void convertBytesWritableRespectsLogicalLength() {
> AvroDatumConverter<BytesWritable, ByteBuffer> converter =
> mFactory.create(BytesWritable.class);
> BytesWritable writable = new BytesWritable(new byte[] { 1, 2, 3, 4, 5 });
> writable.setSize(3);
> ByteBuffer bytes = converter.convert(writable);
> assertEquals(3, bytes.remaining());
> assertEquals(1, bytes.get(0));
> assertEquals(2, bytes.get(1));
> assertEquals(3, bytes.get(2));
> } {code}
> Run:
> {code:java}
> MAVEN_SKIP_RC=true
> JAVA_HOME=/opt/homebrew/Cellar/openjdk@21/21.0.6/libexec/openjdk.jdk/Contents/Home
> \
> PATH=/opt/homebrew/Cellar/openjdk@21/21.0.6/libexec/openjdk.jdk/Contents/Home/bin:/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin
> \
> /opt/homebrew/bin/mvn -q -t toolchains-local.xml -pl lang/java/mapred -am \
>
> -Dtest=org.apache.avro.hadoop.io.TestAvroDatumConverterFactory#convertBytesWritableRespectsLogicalLength
> \
> -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false
> -Dinvoker.skip=true -Drat.skip=true test
> {code}
> *Observed behavior*
> The converted `ByteBuffer` has `remaining() == 5`.
> *Expected behavior*
> The converted `ByteBuffer` should have `remaining() == input.getLength()`,
> which is `3` in the reproducer.
> Hadoop `BytesWritable` separates capacity from logical length. Avro `bytes`
> should encode only the logical value, not unused backing-array capacity. The
> likely fix is `ByteBuffer.wrap(input.copyBytes())` or
> `ByteBuffer.wrap(input.getBytes(), 0, input.getLength())`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
