Gabriel39 opened a new pull request, #69167: URL: https://github.com/apache/doris/pull/69167
### What problem does this PR solve? `test_hive_compress_type_large_data` succeeds with the default file split size but fails after setting `file_split_size=8388608`: ``` [INTERNAL_ERROR] For now we do not support split compressed file ``` Hive's planner used the table-level TEXT/CSV splittability for every file. For example, a 20 MiB gzip file was assigned ranges starting at 0, 8 MiB and 16 MiB. Both native text/CSV readers reject compressed streams starting at a nonzero offset. The same partition can contain plain and compressed files, so a table-wide decision is insufficient. Determine splittability per file in the common range iterator used by synchronous, partition-batch and streaming scans. Recognized compressed TEXT/CSV files now produce one complete range starting at zero. Plain text still splits; ORC/Parquet keep their existing range planning. Hadoop codec splittability is not used because the native readers have different restart capabilities. Move the existing filename-to-codec inference into a shared, dependency-free foundation enum and delegate `Util.inferFileCompressTypeByPath` to it. This keeps planning and the codec sent to BE on the same suffix rules, without changing supported suffixes, Hive's LZ4 adjustment, LZO sidecar filtering, or the BE safety checks. ### Validation - The new mixed-file planner test fails on the original code (33 ranges instead of 13) and passes with the fix. It covers TEXT/CSV, ORC/Parquet, all recognized compression suffixes and uppercase gzip, empty files, plain-file offsets/lengths, and synchronous/batch/streaming enumeration. - `mvn -f fe/pom.xml -pl :fe-connector-hive -am test -Dmaven.build.cache.enabled=false`: 1,462 tests passed, zero failures/errors/skips; 510 tests in the Hive module. - Foundation tests cover unchanged codec recognition and unknown-suffix behavior; connector tests verify the shared enum names map to native Thrift codecs. - The existing large-data regression retains its cardinality/filter assertions, forces synchronous 8 MiB queries, and compares forced streaming queries with the default-size results. Groovy syntax compilation passed. - FE Checkstyle and `git diff --check` passed. A live Hive environment was not exercised locally. The external regression and full FE integration build remain for CI. ### Release note Fix Hive compressed-text queries failing when file split sizes are smaller than the compressed files. ### Check List (For Author) - [x] Unit Test - [x] Regression test extension - [x] Behavior changed: compressed native text files are read as whole streams - [x] No documentation needed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
