Artsem Semianenka created FLINK-10203:
-----------------------------------------
Summary: HadoopRecoverableWriter does not support Hadoop
Key: FLINK-10203
URL: https://issues.apache.org/jira/browse/FLINK-10203
Project: Flink
Issue Type: Bug
Components: DataStream API, filesystem-connector
Affects Versions: 1.6.0, 1.6.1, 1.7.0
Reporter: Artsem Semianenka
New StreamingFileSink ( introduced in 1.6 Flink version ) use
HadoopRecoverableFsDataOutputStream wrapper to write data in HDFS.
HadoopRecoverableFsDataOutputStream is an wrapper on FSDataOutputStream to have
an ability to restore from certain point of file after failure and continue
write data. To achive this recover functionality the
HadoopRecoverableFsDataOutputStream use "truncate" method which was introduced
only in Hadoop 2.7 .
Unfortently there are a few official Hadoop distibutives which latast version
still use Hadoop 2.6 (This distibutives: Cloudera, Pivotal HD ). As the result
Flink can't work with this distibutives.
Flink declares that supported Hadoop from version 2.4.0 upwards
(https://ci.apache.org/projects/flink/flink-docs-release-1.6/start/building.html#hadoop-versions)
I guess we should emulate the functionality of "truncate" method for older
Hadoop versions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
