plusplusjiajia opened a new pull request, #11005: URL: https://github.com/apache/paimon/pull/11005
### Purpose PyPaimon swallowed errors when reading format tables: - An unreadable Parquet, ORC or CSV file read as no rows. So did any file that does not start with the Parquet magic. - A directory it failed to list read as empty. So a network or certificate error while listing OSS read the table as empty, with no error. For example, this happens on hosts where Python cannot find the system CA bundle. As in Java `FormatTableScan` and `PartitionPathUtils`, only a directory that no longer exists reads as empty, for example one deleted after its parent was listed. Other listing errors and file read errors now fail the read, and the error names the file. Zero-byte files follow Java: - A zero-byte Parquet file fails the read: Java's `ParquetFileReader` rejects it with `length is too low`. - A zero-byte ORC file still reads as empty: ORC's `ReaderImpl` reads it as an empty file, since Hive writes them. - A zero-byte CSV file still reads as empty. Like Java, the reader checks the Parquet magic at the end of the file, not at the start. ### Tests - `format_write_execution_test.py#test_format_read_raises_on_unreadable_file`: an unreadable Parquet, ORC or CSV file fails the read, and the error names the file. - `format_write_execution_test.py#test_format_reads_zero_byte_files_like_java`: a zero-byte Parquet file fails the read, and zero-byte ORC and CSV files read as empty. - `format_write_execution_test.py#test_format_scan_raises_listing_errors`: a listing error fails the scan. - 5 of these test cases fail on master. The format table tests pass: `format_write_execution_test.py`, `rest_format_table_test.py` and `cli_table_test.py`, 106 tests. flake8 is clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
