[
https://issues.apache.org/jira/browse/FLINK-10203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Artsem Semianenka updated FLINK-10203:
--------------------------------------
Summary: Support truncate method for old Hadoop versions in
HadoopRecoverableFsDataOutputStream (was: HadoopRecoverableFsDataOutputStream
does not support Hadoop 2.6)
> Support truncate method for old Hadoop versions in
> HadoopRecoverableFsDataOutputStream
> --------------------------------------------------------------------------------------
>
> Key: FLINK-10203
> URL: https://issues.apache.org/jira/browse/FLINK-10203
> Project: Flink
> Issue Type: Bug
> Components: DataStream API, filesystem-connector
> Affects Versions: 1.6.0, 1.6.1, 1.7.0
> Reporter: Artsem Semianenka
> Priority: Major
>
> New StreamingFileSink ( introduced in 1.6 Flink version ) use
> HadoopRecoverableFsDataOutputStream wrapper to write data in HDFS.
> HadoopRecoverableFsDataOutputStream is an wrapper on FSDataOutputStream to
> have an ability to restore from certain point of file after failure and
> continue write data. To achive this recover functionality the
> HadoopRecoverableFsDataOutputStream use "truncate" method which was
> introduced only in Hadoop 2.7 .
> Unfortently there are a few official Hadoop distibutives which latast version
> still use Hadoop 2.6 (This distibutives: Cloudera, Pivotal HD ). As the
> result Flink can't work with this distibutives.
> Flink declares that supported Hadoop from version 2.4.0 upwards
> (https://ci.apache.org/projects/flink/flink-docs-release-1.6/start/building.html#hadoop-versions)
> I guess we should emulate the functionality of "truncate" method for older
> Hadoop versions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
