LuciferYang opened a new issue, #9578: URL: https://github.com/apache/paimon/issues/9578
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master, `2788fe596` (2.1-SNAPSHOT). ### Compute Engine Flink and Spark, reading a CSV, JSON or text table (`AbstractTextFileReader`) whose file is compressed and unreadable. ### Minimal reproduce step Put a file that is not valid gzip at a `.gz` path and read it: ```java // content: "this is not gzip", path: a.csv.gz new CsvFileReader(fileIO, file, rowType, rowType, new CsvOptions(new Options()), 0, null); // throws java.io.IOException, and the stream opened inside is never closed ``` `AbstractTextFileReader`'s constructor: ```java InputStream decompressedStream = createDecompressedInputStream(fileIO.newInputStream(filePath), filePath); this.lineReader = TextLineReader.create(decompressedStream, delimiter, offset, length); ``` The stream is held only by the argument expression until `lineReader` is assigned. The wrapper is chosen from the path suffix, so a compressed path such as `.gz` or `.zst` goes through a Hadoop codec, and `StandardLineReader` reads in its own constructor (`readAtBeginning` calls `fillBuffer`), which is where a bad header fails. When that throws, the constructor never returns, so nothing can close what it opened: the file descriptor, and the decompressor the codec borrowed from `CodecPool`. This is the plain read path, default line delimiter and no offset. ### What doesn't meet your expectations? A reader that fails to construct should not keep the file open. It matters most under `scan.ignore-corrupt-files`, where the caller swallows the failure and moves to the next file, so a scan over a directory with several bad compressed files loses a descriptor and a pooled decompressor for each of them. ### Anything else? `TextLineReader.create` also throws for a custom line delimiter combined with an offset, but that is not reachable in production: `SplitEnumerator.preferToSplitFile` only splits CSV and JSON files that use the default delimiter and are not compressed, so no `FileMeta` with an offset is ever produced for such a file. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
