LuciferYang opened a new issue, #9562: URL: https://github.com/apache/paimon/issues/9562
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master, `2788fe596` (2.1-SNAPSHOT). ### Compute Engine Flink and Spark, wherever Paimon's own vectorized Parquet reader is used on a file it did not write: a format table over a Parquet directory, or a table produced by `migrate_table` / clone, which renames the source files in place rather than rewriting them. ### Minimal reproduce step Read a Parquet file whose DECIMAL column is stored on the BINARY physical type, with dictionary encoding off and at least two rows: ```java MessageType schema = new MessageType( "root", Types.optional(PrimitiveTypeName.BINARY) .as(LogicalTypeAnnotation.decimalType(2, 5)) .named("price")); // three rows: unscaled 100, 200, 300, written with .withDictionaryEncoding(false) ``` Reading it as `DECIMAL(5, 2)` throws on the second row: ``` java.lang.ArrayIndexOutOfBoundsException: 1 at org.apache.paimon.format.parquet.reader.VectorizedPlainValuesReader.readBinary(VectorizedPlainValuesReader.java:324) at org.apache.paimon.format.parquet.reader.ParquetVectorUpdaterFactory$BinaryToDecimalUpdater.readValue(...) at org.apache.paimon.format.parquet.reader.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:238) ``` `BinaryToDecimalUpdater` allocates a scratch vector of capacity 1 and then indexes it with the target row's offset: ```java this.bytesVector = new HeapBytesVector(1); ... valuesReader.readBinary(1, bytesVector, offset); BigInteger value = new BigInteger(bytesVector.getBytes(offset).getBytes()); ``` `HeapBytesVector.putByteArray` writes `start[elementNum]`, and `start` has one slot, so offset 1 is out of bounds. Only a dictionary-encoded page escapes it, because that path goes through `decodeSingleDictionaryId` and never touches the scratch vector. ### What doesn't meet your expectations? The Parquet spec allows DECIMAL on BINARY, and this reader has an updater for it, so a file that uses it should read. Instead any batch with two or more rows fails, and the message says nothing about decimals or about the file. ### Anything else? Paimon's own writer emits decimals as INT32, INT64 or FIXED_LEN_BYTE_ARRAY (`ParquetSchemaConverter`), never BINARY, so this only shows up on externally written files. parquet-avro maps an Avro `bytes` field carrying a decimal logical type onto BINARY, which is what Debezium and Kafka Connect produce. The updater was introduced in #5582 (`d83c5caf8`), which is also when the scratch vector appeared. The tests added there generate random precisions, so they only ever exercise the INT32/INT64/FIXED_LEN_BYTE_ARRAY updaters and this class has had no coverage. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
