[
https://issues.apache.org/jira/browse/AVRO-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18097420#comment-18097420
]
ASF subversion and git services commented on AVRO-4268:
-------------------------------------------------------
Commit f3bb10eaea1f03eb17893bc779eb26213da8177c in avro's branch
refs/heads/dependabot/maven/lang/java/org.apache-apache-39 from Harsh Srivastava
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=f3bb10eaea ]
AVRO-4268: BytesWritableConverter should respect logical length (#3878)
BytesWritable.getBytes() returns the backing array, which can be larger
than the logical length reported by BytesWritable.getLength(). Wrap
input.getBytes() with the actual length so the Avro bytes value reflects
the caller's data, not unused backing-array capacity.
Adds convertBytesWritableRespectsLogicalLength to cover the regression.
Signed-off-by: Harsh Srivastava <[email protected]>
> BytesWritableConverter serializes unused capacity bytes
> -------------------------------------------------------
>
> Key: AVRO-4268
> URL: https://issues.apache.org/jira/browse/AVRO-4268
> Project: Apache Avro
> Issue Type: Bug
> Components: java
> Reporter: Ruiqi Dong
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0, 1.12.2
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> *Summary*
> `AvroDatumConverterFactory.BytesWritableConverter` converts Hadoop
> `BytesWritable` to Avro `bytes` using `ByteBuffer.wrap(input.getBytes())`.
> `BytesWritable.getBytes()` returns the backing array, whose length can be
> larger than the logical value length reported by `getLength()`. The converter
> therefore serializes stale or unused capacity bytes.
>
> *Affected code*
> File:
> `lang/java/mapred/src/main/java/org/apache/avro/hadoop/io/AvroDatumConverterFactory.java`
> {code:java}
> @Override
> public ByteBuffer convert(BytesWritable input) {
> return ByteBuffer.wrap(input.getBytes());
> } {code}
> *Reproducer*
> Add this test to
> `lang/java/mapred/src/test/java/org/apache/avro/hadoop/io/TestAvroDatumConverterFactory.java`
> {code:java}
> @Test
> void convertBytesWritableRespectsLogicalLength() {
> AvroDatumConverter<BytesWritable, ByteBuffer> converter =
> mFactory.create(BytesWritable.class);
> BytesWritable writable = new BytesWritable(new byte[] { 1, 2, 3, 4, 5 });
> writable.setSize(3);
> ByteBuffer bytes = converter.convert(writable);
> assertEquals(3, bytes.remaining());
> assertEquals(1, bytes.get(0));
> assertEquals(2, bytes.get(1));
> assertEquals(3, bytes.get(2));
> } {code}
> Run:
> {code:java}
> MAVEN_SKIP_RC=true
> JAVA_HOME=/opt/homebrew/Cellar/openjdk@21/21.0.6/libexec/openjdk.jdk/Contents/Home
> \
> PATH=/opt/homebrew/Cellar/openjdk@21/21.0.6/libexec/openjdk.jdk/Contents/Home/bin:/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin
> \
> /opt/homebrew/bin/mvn -q -t toolchains-local.xml -pl lang/java/mapred -am \
>
> -Dtest=org.apache.avro.hadoop.io.TestAvroDatumConverterFactory#convertBytesWritableRespectsLogicalLength
> \
> -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false
> -Dinvoker.skip=true -Drat.skip=true test
> {code}
> *Observed behavior*
> The converted `ByteBuffer` has `remaining() == 5`.
> *Expected behavior*
> The converted `ByteBuffer` should have `remaining() == input.getLength()`,
> which is `3` in the reproducer.
> Hadoop `BytesWritable` separates capacity from logical length. Avro `bytes`
> should encode only the logical value, not unused backing-array capacity. The
> likely fix is `ByteBuffer.wrap(input.copyBytes())` or
> `ByteBuffer.wrap(input.getBytes(), 0, input.getLength())`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
