[
https://issues.apache.org/jira/browse/AVRO-4361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Yokota updated AVRO-4361:
--------------------------------
Description:
The collection size guard added by AVRO-4241 (and extended to the fast reader
by AVRO-4300 and to {{ReflectDatumReader}} by AVRO-4324) computes the minimum
bytes per array
element, or map value, from the reader's schema. But the elements were
encoded with the writer's schema. When the reader's element type has a larger
minimum encoded size than the
writer's, the guard rejects valid data with an {{EOFException}}.
Schema resolution allows this in several compatible cases:
* The reader's record adds a field with a default. Example: the writer has
{{Item \{id: int\}}} and the reader has {{Item \{id: int, qty: int (default
0)\}}}.
* Type promotion: {{int}}, {{long}} or {{float}} read as {{double}} (8 bytes
in the guard), or {{int}} or {{long}} read as {{float}} (4 bytes).
* A non-union writer's element read as a branch of a reader's union. Example:
{{array<null>}}, which is 0 bytes per element, read as
{{array<["null","int"]>}}, which the guard
counts as at least 1 byte.
* The same cases for map values.
These are backward-compatible schema changes, and 1.12.1 decodes the same
bytes. The guard is only active when the decoder knows how many bytes remain
(e.g. {{BinaryDecoder}}
over a byte array), which is the common case for message-oriented use such as
Kafka.
Repro (classic and fast reader both fail):
{code:java}
String w =
"{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"items\","
+
"\"type\":{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"Item\","
+ "\"fields\":[{\"name\":\"id\",\"type\":\"int\"}]}}}]}";
String r = w.replace("{\"name\":\"id\",\"type\":\"int\"}",
"{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"qty\",\"type\":\"int\",\"default\":0}");
Schema writer = new Schema.Parser().parse(w);
Schema reader = new Schema.Parser().parse(r);
// write an Order with 100 Items, each with id = 0 (1 byte per element)
...
new GenericDatumReader<>(writer, reader).read(null,
DecoderFactory.get().binaryDecoder(bytes, null));
{code}
1.12.1 decodes 100 items. 1.12.2 fails with:
{noformat}
java.io.EOFException: Collection claims 100 elements with at least 2 bytes
each, but only 101 bytes are available
{noformat}
Cause:
* {{GenericDatumReader.readArray}}/{{readMap}} and
{{ReflectDatumReader.readArray}} pass the reader's element schema
({{expected.getElementType()}}) to
{{ensureAvailableCollectionBytes}}.
* {{FastReaderBuilder.createArrayReader}} does the same with
{{readerSchema.getElementType()}}, although the writer's schema is available
there as {{action.writer}}.
* The zero-byte allocation cap ({{isZeroByteSchema}}) is chosen from the
reader's schema in the same way.
was:
The collection size guard added by AVRO-4241 (and extended to the fast reader
by AVRO-4300 and to ReflectDatumReader by AVRO-4324) computes the minimum bytes
per array element, or map value, from the reader's schema. But the elements
were encoded with the writer's schema. When the reader's element type has a
larger minimum encoded size than the writer's, the guard rejects valid data
with an EOFException.
Schema resolution allows this in several compatible cases:
- The reader's record adds a field with a default. Example: the writer has
Item \{id: int}
and the reader has Item \{id: int, qty: int (default 0)}
- Type promotion: int, long or float read as double (8 bytes in the guard),
or int or long read as float (4 bytes).
- A non-union writer's element read as a branch of a reader's union. Example:
array<null>, which is 0 bytes per element, read as array<["null","int"]>,
which the guard counts as at least 1 byte.
- The same cases for map values.
These are backward-compatible schema changes, and 1.12.1 decodes the same
bytes. The guard is only active when the decoder knows how many bytes remain
(e.g. BinaryDecoder over a byte array), which is the common case for
message-oriented use such as Kafka.
Repro (classic and fast reader both fail):
String w =
"{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"items\","
+ "\"type\":{\"type\":\"array\",\"items\":
{\"type\":\"record\",\"name\":\"Item\","
+
"\"fields\":[\\{\"name\":\"id\",\"type\":\"int\"}
]}}}]}";
String r = w.replace("\{\"name\":\"id\",\"type\":\"int\"}",
"\{\"name\":\"id\",\"type\":\"int\"},\{\"name\":\"qty\",\"type\":\"int\",\"default\":0}");
Schema writer = new Schema.Parser().parse(w);
Schema reader = new Schema.Parser().parse(r);
// write an Order with 100 Items, each with id = 0 (1 byte per element)
...
new GenericDatumReader<>(writer, reader).read(null,
DecoderFactory.get().binaryDecoder(bytes, null));
1.12.1: decodes 100 items.
1.12.2: java.io.EOFException: Collection claims 100 elements with at least 2
bytes
each, but only 101 bytes are available
Cause:
- GenericDatumReader.readArray/readMap and ReflectDatumReader.readArray pass
the
reader's element schema (expected.getElementType()) to
ensureAvailableCollectionBytes.
- FastReaderBuilder.createArrayReader does the same with
readerSchema.getElementType(), although the writer's schema is available
there as action.writer.
- The zero-byte allocation cap (isZeroByteSchema) is chosen from the reader's
schema in the same way.
> [Java] Collection size guard rejects valid data when the reader's element
> schema encodes larger than the writer's
> -----------------------------------------------------------------------------------------------------------------
>
> Key: AVRO-4361
> URL: https://issues.apache.org/jira/browse/AVRO-4361
> Project: Apache Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.12.2
> Reporter: Robert Yokota
> Assignee: Robert Yokota
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0, 1.12.3
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The collection size guard added by AVRO-4241 (and extended to the fast reader
> by AVRO-4300 and to {{ReflectDatumReader}} by AVRO-4324) computes the minimum
> bytes per array
> element, or map value, from the reader's schema. But the elements were
> encoded with the writer's schema. When the reader's element type has a larger
> minimum encoded size than the
> writer's, the guard rejects valid data with an {{EOFException}}.
>
>
>
>
>
> Schema resolution allows this in several compatible cases:
>
>
> * The reader's record adds a field with a default. Example: the writer has
> {{Item \{id: int\}}} and the reader has {{Item \{id: int, qty: int (default
> 0)\}}}.
> * Type promotion: {{int}}, {{long}} or {{float}} read as {{double}} (8
> bytes in the guard), or {{int}} or {{long}} read as {{float}} (4 bytes).
>
> * A non-union writer's element read as a branch of a reader's union.
> Example: {{array<null>}}, which is 0 bytes per element, read as
> {{array<["null","int"]>}}, which the guard
> counts as at least 1 byte.
>
>
> * The same cases for map values.
>
>
>
>
>
> These are backward-compatible schema changes, and 1.12.1 decodes the same
> bytes. The guard is only active when the decoder knows how many bytes remain
> (e.g. {{BinaryDecoder}}
> over a byte array), which is the common case for message-oriented use such
> as Kafka.
>
>
>
>
> Repro (classic and fast reader both fail):
>
>
> {code:java}
>
>
> String w =
> "{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"items\","
>
>
> +
> "\"type\":{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"Item\","
>
>
> + "\"fields\":[{\"name\":\"id\",\"type\":\"int\"}]}}}]}";
>
>
> String r = w.replace("{\"name\":\"id\",\"type\":\"int\"}",
>
>
>
> "{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"qty\",\"type\":\"int\",\"default\":0}");
>
>
> Schema writer = new Schema.Parser().parse(w);
>
>
> Schema reader = new Schema.Parser().parse(r);
>
>
> // write an Order with 100 Items, each with id = 0 (1 byte per element)
>
>
> ...
>
>
> new GenericDatumReader<>(writer, reader).read(null,
> DecoderFactory.get().binaryDecoder(bytes, null));
>
> {code}
>
>
>
> 1.12.1 decodes 100 items. 1.12.2 fails with:
>
>
> {noformat}
>
>
> java.io.EOFException: Collection claims 100 elements with at least 2 bytes
> each, but only 101 bytes are available
>
> {noformat}
>
>
>
>
>
> Cause:
>
>
> * {{GenericDatumReader.readArray}}/{{readMap}} and
> {{ReflectDatumReader.readArray}} pass the reader's element schema
> ({{expected.getElementType()}}) to
> {{ensureAvailableCollectionBytes}}.
>
>
> * {{FastReaderBuilder.createArrayReader}} does the same with
> {{readerSchema.getElementType()}}, although the writer's schema is available
> there as {{action.writer}}.
> * The zero-byte allocation cap ({{isZeroByteSchema}}) is chosen from the
> reader's schema in the same way.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
