[
https://issues.apache.org/jira/browse/FLINK-33059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Etienne Chauchot updated FLINK-33059:
-------------------------------------
Description: Some FileInputFormats don't use FileInputFormat#createSplits
(that would detect that the file is non-splittable and deal with reading
boundaries correctly), they all create split manually from FileSourceSplit. If
input files are compressed, split length is determined by the compressed file
length leading to [this|https://issues.apache.org/jira/browse/FLINK-30314] bug.
We should force reading the whole file split (like it is done for binary input
formats) on compressed files. Parallelism is still done at the file level (as
now) (was: Delimited file input formats (contrary to binary input format
etc...) do not support compression via the existing decorator because split
length is determined by the compressed file length lead to
[this|https://issues.apache.org/jira/browse/FLINK-30314] bug . We should force
reading the whole file split (like it is done for binary input formats) on
compressed files. Parallelism is still done at the file level (as now))
> Support transparent compression for file-connector for all file input formats
> -----------------------------------------------------------------------------
>
> Key: FLINK-33059
> URL: https://issues.apache.org/jira/browse/FLINK-33059
> Project: Flink
> Issue Type: Technical Debt
> Components: Connectors / FileSystem
> Reporter: Etienne Chauchot
> Assignee: Etienne Chauchot
> Priority: Major
>
> Some FileInputFormats don't use FileInputFormat#createSplits (that would
> detect that the file is non-splittable and deal with reading boundaries
> correctly), they all create split manually from FileSourceSplit. If input
> files are compressed, split length is determined by the compressed file
> length leading to [this|https://issues.apache.org/jira/browse/FLINK-30314]
> bug. We should force reading the whole file split (like it is done for binary
> input formats) on compressed files. Parallelism is still done at the file
> level (as now)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
