zhuxiangyi opened a new pull request, #9961: URL: https://github.com/apache/paimon/pull/9961
### Purpose `AvroFileFormat.createBlockWriter` passes `file.block-size` straight to Avro's `DataFileWriter.setSyncInterval`, which only accepts 32 bytes to 1 GiB. Any other value fails at write time on the executor with Avro's own message (`Invalid syncInterval value: 6`) or, above 2 GiB, with `ArithmeticException: integer overflow` from `Math.toIntExact` — neither names the option. This also makes `DeletionVectorTest."Paimon deletionVector: select with format filter push down"` flaky: it picks a random format and `Random.nextInt(10240) + 1` as the block size, so about 1 run in 1,000 lands on avro with a block size below 32 and fails with the message above. It hit the `Java / Spark 3 / Scala 2.13` job of #9953 (https://github.com/apache/paimon/actions/runs/35309357679/job/105488023917, block size 6) on a change that does not touch Avro or deletion vectors; every other job passed. A CI failure like this sends a reviewer to look for a regression that is not there, so it is worth removing rather than re-running around. ### Change - `AvroFileFormat`: check `file.block-size` against `[32 bytes, 1 GiB]` before calling `setSyncInterval`, and fail with `file.block-size for avro must be between 32 bytes and 1 gb, but was N bytes.` The check also covers the former integer-overflow case. Reads are unaffected; the check runs only when a writer is created. - `DeletionVectorTest`: draw the random block size from `[32, 10271]` instead of `[1, 10240]`. ### Tests `AvroFileFormatTest`: - `testFileBlockSizeOutOfAvroRange` (7 values: 1, 6, 31, 2^30+1, 2^31, 4 GiB+, `Long.MAX_VALUE`) asserts an `IllegalArgumentException` naming `file.block-size`, the value and the bounds. On master these fail with `Invalid syncInterval value: N` (first four) or `integer overflow` (last three); it replaces `testFileBlockSizeOverflow`, which pinned the raw overflow. - `testFileBlockSizeAtAvroRangeBounds` (32, 33, 2^30) writes successfully. - The existing `testFileBlockSize` cases (no block size, 1 kb … 128 kb) and `testManifestIgnoresDataFileBlockSize` are unchanged and pass. Results: `AvroFileFormatTest` 55/55, `org.apache.paimon.format.avro.*` 79/79, Spark `DeletionVectorTest` 17/17 (Spark 3, Scala 2.12, shaded build), checkstyle and spotless clean. ### API and Format No API or format changes. Behaviour change: an out-of-range avro `file.block-size` now fails with a message naming the option instead of an Avro-internal message or an `ArithmeticException`; it still fails at the same point (writer creation). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
