[
https://issues.apache.org/jira/browse/FLINK-10203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksandr Nitavskyi updated FLINK-10203:
----------------------------------------
Description:
New StreamingFileSink ( introduced in 1.6 Flink version ) use
HadoopRecoverableFsDataOutputStream wrapper to write data in HDFS.
HadoopRecoverableFsDataOutputStream is a wrapper for FSDataOutputStream to have
an ability to restore from a certain point of the file after failure and
continue to write data. To achieve this recovery functionality the
HadoopRecoverableFsDataOutputStream uses "truncate" method which was introduced
only in Hadoop 2.7.
FLINK-14170 has enabled the usage of StreamingFileSink for
OnCheckpointRollingPolicy, but it is still not possible to use
StreamingFileSink with DefaultRollingPolicy, which makes writing of the data to
HDFS unpractical in scale for HDFS < 2.7.
Unfortunately, there are a few official Hadoop distributives which latest
version still use Hadoop 2.6 (This distributives: Cloudera, Pivotal HD ). As
the result Flinks Hadoop connector can't work with this distributives.
Flink declares that supported Hadoop from version 2.4.0 upwards
([https://ci.apache.org/projects/flink/flink-docs-release-1.6/start/building.html#hadoop-versions])
I guess we should emulate the functionality of "truncate" method for older
Hadoop versions.
was:
New StreamingFileSink ( introduced in 1.6 Flink version ) use
HadoopRecoverableFsDataOutputStream wrapper to write data in HDFS.
HadoopRecoverableFsDataOutputStream is a wrapper for FSDataOutputStream to have
an ability to restore from certain point of file after failure and continue
write data. To achieve this recover functionality the
HadoopRecoverableFsDataOutputStream use "truncate" method which was introduced
only in Hadoop 2.7 .
Unfortunately there are a few official Hadoop distributive which latest version
still use Hadoop 2.6 (This distributives: Cloudera, Pivotal HD ). As the result
Flinks Hadoop connector can't work with this distributives.
Flink declares that supported Hadoop from version 2.4.0 upwards
([https://ci.apache.org/projects/flink/flink-docs-release-1.6/start/building.html#hadoop-versions])
I guess we should emulate the functionality of "truncate" method for older
Hadoop versions.
> Support truncate method for old Hadoop versions in
> HadoopRecoverableFsDataOutputStream
> --------------------------------------------------------------------------------------
>
> Key: FLINK-10203
> URL: https://issues.apache.org/jira/browse/FLINK-10203
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Connectors / FileSystem
> Affects Versions: 1.6.0, 1.6.1, 1.7.0
> Reporter: Artsem Semianenka
> Assignee: Artsem Semianenka
> Priority: Major
> Labels: pull-request-available
> Attachments: legacy truncate logic.pdf
>
>
> New StreamingFileSink ( introduced in 1.6 Flink version ) use
> HadoopRecoverableFsDataOutputStream wrapper to write data in HDFS.
> HadoopRecoverableFsDataOutputStream is a wrapper for FSDataOutputStream to
> have an ability to restore from a certain point of the file after failure and
> continue to write data. To achieve this recovery functionality the
> HadoopRecoverableFsDataOutputStream uses "truncate" method which was
> introduced only in Hadoop 2.7.
> FLINK-14170 has enabled the usage of StreamingFileSink for
> OnCheckpointRollingPolicy, but it is still not possible to use
> StreamingFileSink with DefaultRollingPolicy, which makes writing of the data
> to HDFS unpractical in scale for HDFS < 2.7.
> Unfortunately, there are a few official Hadoop distributives which latest
> version still use Hadoop 2.6 (This distributives: Cloudera, Pivotal HD ). As
> the result Flinks Hadoop connector can't work with this distributives.
> Flink declares that supported Hadoop from version 2.4.0 upwards
> ([https://ci.apache.org/projects/flink/flink-docs-release-1.6/start/building.html#hadoop-versions])
> I guess we should emulate the functionality of "truncate" method for older
> Hadoop versions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
