[
https://issues.apache.org/jira/browse/FLINK-14170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16939210#comment-16939210
]
Kostas Kloudas commented on FLINK-14170:
----------------------------------------
I agree that when the {{OnCheckpointRollingPolicy}} is used, our check is
pretty restrictive. But I do not think that the solution of throwing an
exception at runtime is the way to go. For cases that we know we do not
support, we should fail at build time or at pre-flight time and not let the
user think that everything is fine although there is a time bomb waiting to
explode.
One potential solution could be that we implement the same strategy as the
pre-existing {{BucketingSink}} and we write a {{valid-length}} file that
contains the valid offset and we always role upon restore. I would be more than
happy if you have any more elegant and less code-heavy solutions.
> Support hadoop < 2.7 with StreamingFileSink.BulkFormatBuilder
> -------------------------------------------------------------
>
> Key: FLINK-14170
> URL: https://issues.apache.org/jira/browse/FLINK-14170
> Project: Flink
> Issue Type: Improvement
> Components: API / DataSet
> Affects Versions: 1.8.0, 1.8.1, 1.8.2, 1.9.0
> Reporter: Bhagavan
> Priority: Major
>
> Currently, StreamingFileSink is supported only with Hadoop >= 2.7
> irrespective of Row/bulk format builder. This restriction is due to truncate
> is not supported in Hadoop < 2.7
> However, BulkFormatBuilder does not use truncate method to restore the file.
> So the restricting StreamingFileSink.BulkFormatBuilder to be used only with
> Hadoop >= 2.7 is not necessary.
> So requested improvement is to remove the precondition on
> HadoopRecoverableWriter and allow BulkFormatBuilder (Parquet) to be used in
> Hadoop 2.6 ( Most of the enterprises still on CDH 5.x)
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
