[
https://issues.apache.org/jira/browse/AVRO-4361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Yokota updated AVRO-4361:
--------------------------------
Description:
The collection size guard added by AVRO-4241 (and extended to the fast reader
by AVRO-4300 and to ReflectDatumReader by AVRO-4324) computes the minimum bytes
per array element, or map value, from the reader's schema. But the elements
were encoded with the writer's schema. When the reader's element type has a
larger minimum encoded size than the writer's, the guard rejects valid data
with an EOFException.
Schema resolution allows this in several compatible cases:
- The reader's record adds a field with a default. Example: the writer has
Item \{id: int}
and the reader has Item
{id: int, qty: int (default 0)}
.
- Type promotion: int, long or float read as double (8 bytes in the guard),
or int or long read as float (4 bytes).
- A non-union writer's element read as a branch of a reader's union. Example:
array<null>, which is 0 bytes per element, read as array<["null","int"]>,
which the guard counts as at least 1 byte.
- The same cases for map values.
These are backward-compatible schema changes, and 1.12.1 decodes the same
bytes. The guard is only active when the decoder knows how many bytes remain
(e.g. BinaryDecoder over a byte array), which is the common case for
message-oriented use such as Kafka.
Repro (classic and fast reader both fail):
String w =
"{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"items\","
+
"\"type\":{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"Item\","
+ "\"fields\":[
{\"name\":\"id\",\"type\":\"int\"}
]}}}]}";
String r = w.replace("
{\"name\":\"id\",\"type\":\"int\"}
",
"
{\"name\":\"id\",\"type\":\"int\"}
,
{\"name\":\"qty\",\"type\":\"int\",\"default\":0}
");
Schema writer = new Schema.Parser().parse(w);
Schema reader = new Schema.Parser().parse(r);
// write an Order with 100 Items, each with id = 0 (1 byte per element)
...
new GenericDatumReader<>(writer, reader).read(null,
DecoderFactory.get().binaryDecoder(bytes, null));
1.12.1: decodes 100 items.
1.12.2: java.io.EOFException: Collection claims 100 elements with at least 2
bytes
each, but only 101 bytes are available
Cause:
- GenericDatumReader.readArray/readMap and ReflectDatumReader.readArray pass
the
reader's element schema (expected.getElementType()) to
ensureAvailableCollectionBytes.
- FastReaderBuilder.createArrayReader does the same with
readerSchema.getElementType(), although the writer's schema is available
there as action.writer.
- The zero-byte allocation cap (isZeroByteSchema) is chosen from the reader's
schema in the same way.
was:
The collection size guard added by AVRO-4241 (and extended to the fast reader
by AVRO-4300 and to ReflectDatumReader by AVRO-4324) computes the minimum
bytes
per array element, or map value, from the reader's schema. But the elements
were encoded with the writer's schema. When the reader's element type has a
larger minimum encoded size than the writer's, the guard rejects valid data
with an EOFException.
Schema resolution allows this in several compatible cases:
- The reader's record adds a field with a default. Example: the writer has
Item {id: int} and the reader has Item {id: int, qty: int (default 0)}.
- Type promotion: int, long or float read as double (8 bytes in the guard),
or int or long read as float (4 bytes).
- A non-union writer's element read as a branch of a reader's union. Example:
array<null>, which is 0 bytes per element, read as array<["null","int"]>,
which the guard counts as at least 1 byte.
- The same cases for map values.
These are backward-compatible schema changes, and 1.12.1 decodes the same
bytes. The guard is only active when the decoder knows how many bytes remain
(e.g. BinaryDecoder over a byte array), which is the common case for
message-oriented use such as Kafka.
Repro (classic and fast reader both fail):
String w =
"{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"items\","
+
"\"type\":{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"Item\","
+ "\"fields\":[{\"name\":\"id\",\"type\":\"int\"}]}}}]}";
String r = w.replace("{\"name\":\"id\",\"type\":\"int\"}",
"{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"qty\",\"type\":\"int\",\"default\":0}");
Schema writer = new Schema.Parser().parse(w);
Schema reader = new Schema.Parser().parse(r);
// write an Order with 100 Items, each with id = 0 (1 byte per element)
...
new GenericDatumReader<>(writer, reader).read(null,
DecoderFactory.get().binaryDecoder(bytes, null));
1.12.1: decodes 100 items.
1.12.2: java.io.EOFException: Collection claims 100 elements with at least 2
bytes
each, but only 101 bytes are available
Cause:
- GenericDatumReader.readArray/readMap and ReflectDatumReader.readArray pass
the
reader's element schema (expected.getElementType()) to
ensureAvailableCollectionBytes.
- FastReaderBuilder.createArrayReader does the same with
readerSchema.getElementType(), although the writer's schema is available
there as action.writer.
- The zero-byte allocation cap (isZeroByteSchema) is chosen from the reader's
schema in the same way.
> [Java] Collection size guard rejects valid data when the reader's element
> schema encodes larger than the writer's
> -----------------------------------------------------------------------------------------------------------------
>
> Key: AVRO-4361
> URL: https://issues.apache.org/jira/browse/AVRO-4361
> Project: Apache Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.12.2
> Reporter: Robert Yokota
> Assignee: Robert Yokota
> Priority: Major
> Fix For: 1.13.0, 1.12.3
>
>
> The collection size guard added by AVRO-4241 (and extended to the fast reader
> by AVRO-4300 and to ReflectDatumReader by AVRO-4324) computes the minimum
> bytes
> per array element, or map value, from the reader's schema. But the elements
> were encoded with the writer's schema. When the reader's element type has a
> larger minimum encoded size than the writer's, the guard rejects valid data
> with an EOFException.
> Schema resolution allows this in several compatible cases:
> - The reader's record adds a field with a default. Example: the writer has
> Item \{id: int}
> and the reader has Item
> {id: int, qty: int (default 0)}
> .
> - Type promotion: int, long or float read as double (8 bytes in the guard),
> or int or long read as float (4 bytes).
> - A non-union writer's element read as a branch of a reader's union. Example:
> array<null>, which is 0 bytes per element, read as array<["null","int"]>,
> which the guard counts as at least 1 byte.
> - The same cases for map values.
> These are backward-compatible schema changes, and 1.12.1 decodes the same
> bytes. The guard is only active when the decoder knows how many bytes remain
> (e.g. BinaryDecoder over a byte array), which is the common case for
> message-oriented use such as Kafka.
> Repro (classic and fast reader both fail):
> String w =
> "{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"items\","
> +
> "\"type\":{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"Item\","
>
> + "\"fields\":[
> {\"name\":\"id\",\"type\":\"int\"}
> ]}}}]}";
> String r = w.replace("
> {\"name\":\"id\",\"type\":\"int\"}
> ",
> "
> {\"name\":\"id\",\"type\":\"int\"}
> ,
> {\"name\":\"qty\",\"type\":\"int\",\"default\":0}
> ");
> Schema writer = new Schema.Parser().parse(w);
> Schema reader = new Schema.Parser().parse(r);
> // write an Order with 100 Items, each with id = 0 (1 byte per element)
> ...
> new GenericDatumReader<>(writer, reader).read(null,
> DecoderFactory.get().binaryDecoder(bytes, null));
> 1.12.1: decodes 100 items.
> 1.12.2: java.io.EOFException: Collection claims 100 elements with at least 2
> bytes
> each, but only 101 bytes are available
> Cause:
> - GenericDatumReader.readArray/readMap and ReflectDatumReader.readArray pass
> the
> reader's element schema (expected.getElementType()) to
> ensureAvailableCollectionBytes.
> - FastReaderBuilder.createArrayReader does the same with
> readerSchema.getElementType(), although the writer's schema is available
> there as action.writer.
> - The zero-byte allocation cap (isZeroByteSchema) is chosen from the reader's
> schema in the same way.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
