Martijn created FLINK-11378:
-------------------------------
Summary: Allow HadoopRecoverableWriter to write to Hadoop
compatible Filesystems.
Key: FLINK-11378
URL: https://issues.apache.org/jira/browse/FLINK-11378
Project: Flink
Issue Type: Improvement
Components: FileSystem
Reporter: Martijn
At a client we're using Flink jobs to read data from Kafka and writing it to
GCS. In earlier versions, we've used `BucketingFileSink` for this, but we want
to switch to the newer `StreamingFileSink`.
Since we're running Flink on Google's DataProc, we're using the Hadoop
compatible GCS
[connector|https://github.com/GoogleCloudPlatform/bigdata-interop] made by
Google. This currently doesn't work on Flink, because Flink checks for a HDFS
scheme at 'HadoopRecoverableWriter'.
We've successfully ran our jobs by creating a custom Flink Distro which has the
hdfs scheme check removed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
