[
https://issues.apache.org/jira/browse/FLINK-15777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17024385#comment-17024385
]
Kostas Kloudas edited comment on FLINK-15777 at 1/27/20 9:05 PM:
-----------------------------------------------------------------
[~arvid heise] the checks existed since the introduction of the
{{StreamingFileSink}}. The commit mentioned in the description just moved them
from the task initialisation, further "down" the task runtime, so this is a
regression compared to previous Flink versions, not related to
[FLINK-14170|https://issues.apache.org/jira/browse/FLINK-14170], but probably
to the new plugin mechanism integration with the StreamingFileSink.
was (Author: kkl0u):
[~arvid heise] well the checks existed since the introduction of the
{{StreamingFileSink}}. The commit mentioned in the description just moved them
from the task initialisation, further "down" the task runtime, so this is a
regression compared to previous Flink versions not related to
[FLINK-14170|https://issues.apache.org/jira/browse/FLINK-14170], but probably
to the new plugin mechanism integration with the StreamingFileSink.
> Truncate not working with plugins (as expected)
> -----------------------------------------------
>
> Key: FLINK-15777
> URL: https://issues.apache.org/jira/browse/FLINK-15777
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
> Affects Versions: 1.10.0, 1.11.0
> Reporter: Arvid Heise
> Assignee: Arvid Heise
> Priority: Critical
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> FLINK-14170 introduced Hadoop version check to support older versions with
> limited functionality. However, the fix checks the Hadoop version of the
> system classloader and not of the plugin (of the actual used filesystem).
> The [version
> checks|https://github.com/apache/flink/blob/a607bd9dbdda7d9925d6c351ba88d82edce2c571/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java#L197]
> will fail when there is a Hadoop version in the classpath of Flink,
> independent of the bundled version of the filesystem.
> On a recent EMR setup with lingering Hadoop 2.6 libraries, this mean that you
> cannot write to S3 with StreamingFileSink.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
