[
https://issues.apache.org/jira/browse/AVRO-4361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Yokota updated AVRO-4361:
--------------------------------
Description:
The collection size guard added by AVRO-4241 (and extended to the fast reader
by AVRO-4300 and to ReflectDatumReader by AVRO-4324) computes the minimum
bytes
per array element, or map value, from the reader's schema. But the elements
were encoded with the writer's schema. When the reader's element type has a
larger minimum encoded size than the writer's, the guard rejects valid data
with an EOFException.
Schema resolution allows this in several compatible cases:
- The reader's record adds a field with a default. Example: the writer has
{{Item {id: int}}} and the reader has {{Item {id: int, qty: int (default
0)}}}.
- Type promotion: int, long or float read as double (8 bytes in the guard),
or int or long read as float (4 bytes).
- A non-union writer's element read as a branch of a reader's union. Example:
array<null>, which is 0 bytes per element, read as array<["null","int"]>,
which the guard counts as at least 1 byte.
- The same cases for map values.
These are backward-compatible schema changes, and 1.12.1 decodes the same
bytes. The guard is only active when the decoder knows how many bytes remain
(e.g. BinaryDecoder over a byte array), which is the common case for
message-oriented use such as Kafka.
Repro (classic and fast reader both fail):
{{ String w =
"{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"items\","
+
"\"type\":{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"Item\","
+ "\"fields\":[{\"name\":\"id\",\"type\":\"int\"}]}}}]}";
String r = w.replace("{\"name\":\"id\",\"type\":\"int\"}",
"{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"qty\",\"type\":\"int\",\"default\":0}");
Schema writer = new Schema.Parser().parse(w);
Schema reader = new Schema.Parser().parse(r);
// write an Order with 100 Items, each with id = 0 (1 byte per element)
...
new GenericDatumReader<>(writer, reader).read(null,
DecoderFactory.get().binaryDecoder(bytes, null)); }}
1.12.1: decodes 100 items.
1.12.2: java.io.EOFException: Collection claims 100 elements with at least 2
bytes
each, but only 101 bytes are available
Cause:
- GenericDatumReader.readArray/readMap and ReflectDatumReader.readArray pass
the
reader's element schema (expected.getElementType()) to
ensureAvailableCollectionBytes.
- FastReaderBuilder.createArrayReader does the same with
readerSchema.getElementType(), although the writer's schema is available
there as action.writer.
- The zero-byte allocation cap (isZeroByteSchema) is chosen from the reader's
schema in the same way.
was:
The collection size guard added by AVRO-4241 (and extended to the fast reader
by AVRO-4300 and to ReflectDatumReader by AVRO-4324) computes the minimum
bytes
per array element, or map value, from the reader's schema. But the elements
were encoded with the writer's schema. When the reader's element type has a
larger minimum encoded size than the writer's, the guard rejects valid data
with an EOFException.
Schema resolution allows this in several compatible cases:
- The reader's record adds a field with a default. Example: the writer has
Item {id: int} and the reader has Item {id: int, qty: int (default 0)}.
- Type promotion: int, long or float read as double (8 bytes in the guard),
or int or long read as float (4 bytes).
- A non-union writer's element read as a branch of a reader's union. Example:
array<null>, which is 0 bytes per element, read as array<["null","int"]>,
which the guard counts as at least 1 byte.
- The same cases for map values.
These are backward-compatible schema changes, and 1.12.1 decodes the same
bytes. The guard is only active when the decoder knows how many bytes remain
(e.g. BinaryDecoder over a byte array), which is the common case for
message-oriented use such as Kafka.
Repro (classic and fast reader both fail):
```
String w =
"{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"items\","
+
"\"type\":{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"Item\","
+ "\"fields\":[{\"name\":\"id\",\"type\":\"int\"}]}}}]}";
String r = w.replace("{\"name\":\"id\",\"type\":\"int\"}",
"{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"qty\",\"type\":\"int\",\"default\":0}");
Schema writer = new Schema.Parser().parse(w);
Schema reader = new Schema.Parser().parse(r);
// write an Order with 100 Items, each with id = 0 (1 byte per element)
...
new GenericDatumReader<>(writer, reader).read(null,
DecoderFactory.get().binaryDecoder(bytes, null));
```
1.12.1: decodes 100 items.
1.12.2: java.io.EOFException: Collection claims 100 elements with at least 2
bytes
each, but only 101 bytes are available
Cause:
- GenericDatumReader.readArray/readMap and ReflectDatumReader.readArray pass
the
reader's element schema (expected.getElementType()) to
ensureAvailableCollectionBytes.
- FastReaderBuilder.createArrayReader does the same with
readerSchema.getElementType(), although the writer's schema is available
there as action.writer.
- The zero-byte allocation cap (isZeroByteSchema) is chosen from the reader's
schema in the same way.
> [Java] Collection size guard rejects valid data when the reader's element
> schema encodes larger than the writer's
> -----------------------------------------------------------------------------------------------------------------
>
> Key: AVRO-4361
> URL: https://issues.apache.org/jira/browse/AVRO-4361
> Project: Apache Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.12.2
> Reporter: Robert Yokota
> Assignee: Robert Yokota
> Priority: Major
> Fix For: 1.13.0, 1.12.3
>
>
> The collection size guard added by AVRO-4241 (and extended to the fast reader
>
>
> by AVRO-4300 and to ReflectDatumReader by AVRO-4324) computes the minimum
> bytes
>
> per array element, or map value, from the reader's schema. But the elements
>
>
> were encoded with the writer's schema. When the reader's element type has a
>
>
> larger minimum encoded size than the writer's, the guard rejects valid data
>
>
> with an EOFException.
>
>
>
>
>
> Schema resolution allows this in several compatible cases:
>
>
>
>
>
> - The reader's record adds a field with a default. Example: the writer has
>
>
> {{Item {id: int}}} and the reader has {{Item {id: int, qty: int (default
> 0)}}}.
>
> - Type promotion: int, long or float read as double (8 bytes in the guard),
>
>
> or int or long read as float (4 bytes).
>
>
> - A non-union writer's element read as a branch of a reader's union.
> Example:
>
> array<null>, which is 0 bytes per element, read as array<["null","int"]>,
>
>
> which the guard counts as at least 1 byte.
>
>
> - The same cases for map values.
>
>
>
>
>
> These are backward-compatible schema changes, and 1.12.1 decodes the same
>
>
> bytes. The guard is only active when the decoder knows how many bytes
> remain
>
> (e.g. BinaryDecoder over a byte array), which is the common case for
>
>
> message-oriented use such as Kafka.
> Repro (classic and fast reader both fail):
>
>
>
>
>
> {{ String w =
> "{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"items\","
>
>
> +
> "\"type\":{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"Item\","
>
>
> + "\"fields\":[{\"name\":\"id\",\"type\":\"int\"}]}}}]}";
>
>
> String r = w.replace("{\"name\":\"id\",\"type\":\"int\"}",
>
>
>
> "{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"qty\",\"type\":\"int\",\"default\":0}");
>
>
> Schema writer = new Schema.Parser().parse(w);
>
>
> Schema reader = new Schema.Parser().parse(r);
>
>
> // write an Order with 100 Items, each with id = 0 (1 byte per element)
>
>
> ...
>
>
> new GenericDatumReader<>(writer, reader).read(null,
> DecoderFactory.get().binaryDecoder(bytes, null)); }}
>
>
>
>
> 1.12.1: decodes 100 items.
>
>
> 1.12.2: java.io.EOFException: Collection claims 100 elements with at least
> 2 bytes
>
> each, but only 101 bytes are available
>
>
>
>
>
> Cause:
>
>
> - GenericDatumReader.readArray/readMap and ReflectDatumReader.readArray
> pass the
>
> reader's element schema (expected.getElementType()) to
>
>
> ensureAvailableCollectionBytes.
>
>
> - FastReaderBuilder.createArrayReader does the same with
>
>
> readerSchema.getElementType(), although the writer's schema is available
>
>
> there as action.writer.
>
>
> - The zero-byte allocation cap (isZeroByteSchema) is chosen from the
> reader's
>
> schema in the same way.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
