LuciferYang opened a new issue, #9572: URL: https://github.com/apache/paimon/issues/9572
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master, `2788fe596` (2.1-SNAPSHOT). ### Compute Engine Flink and Spark, on the paths that extract statistics from Avro files Paimon did not write: `migrate_table` / `migrate_database` over an Avro Hive table, and the clone action. Both hand every non-hidden file in the source directory to `AvroSimpleStatsExtractor`. ### Minimal reproduce step Point the extractor at a file that is not a readable Avro file: ```java AvroSimpleStatsExtractor extractor = new AvroSimpleStatsExtractor(rowType, collectors); extractor.extract(fileIO, pathToSomeNonAvroFile, length); // throws, and leaks the stream ``` `extractWithFileInfo` opens the stream itself and passes it on: ```java SeekableInputStream fileInputStream = fileIO.newInputStream(path); long rowCount = getRowCount(fileInputStream); ... private long getRowCount(InputStream inStream) throws IOException { try (DataFileStream<Object> streamReader = new DataFileStream<>(inStream, new GenericDatumReader<>())) { ``` The try-with-resources binds the `DataFileStream`, not the stream. If that constructor throws, the resource was never bound and no local variable holds the input any more, so nothing closes it. Three ways it throws: a non-Avro file fails the magic check, an empty or truncated file hits EOF inside `readMagic`, and an unrecognised `avro.codec` makes `CodecFactory.fromString` throw `AvroRuntimeException`. ### What doesn't meet your expectations? `AvroBulkFormat.createReaderFromPath` in the same package already handles this: it closes the stream itself when reader construction throws. The stats extractor should not be the one place that leaks a descriptor per corrupt file. The amount leaked is one descriptor per attempt, one HDFS `DFSInputStream` or one pooled S3 connection. Small, but a zero-byte file left behind by a failed task is a normal thing to find in a Hive table directory, since the migrate scan only filters names starting with `_` or `.`, so the corrupt-file path is not exotic. ### Anything else? Not on the write path. `RollingFileWriter.createStatsProducer` and `KeyValueFileWriterFactory.statsProducer` both build stats from the collector for avro rather than re-reading the file, so writing is unaffected. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
