Re: difference b/t filesystem and connector-filesystem

2018-01-18 Thread Gary Yao
Hi, to do that you can set the env variable HADOOP_CONF_DIR: https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#hdfs Best, Gary On Wed, Jan 17, 2018 at 2:27 AM, cw7k wrote: > Hi, question on this page: > "You need to point Flink to a valid Hadoop configuration..."https:

Re: difference b/t filesystem and connector-filesystem

2018-01-16 Thread cw7k
Hi, question on this page: "You need to point Flink to a valid Hadoop configuration..."https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/deployment/aws.html#s3-simple-storage-service How do you point Flink to the Hadoop config? On Saturday, January 13, 2018, 4:56:15 AM PST, Till

Re: difference b/t filesystem and connector-filesystem

2018-01-16 Thread cw7k
Thanks, is there a sample project that outputs a Flink example app (ie WordCount) to S3? On Saturday, January 13, 2018, 4:56:15 AM PST, Till Rohrmann wrote: Hi, the flink-connector-filesystem contains the BucketingSink which is a connector with which you can write your data to a file

Re: difference b/t filesystem and connector-filesystem

2018-01-13 Thread Till Rohrmann
Hi, the flink-connector-filesystem contains the BucketingSink which is a connector with which you can write your data to a file system. It provides exactly once processing guarantees and allows to write data to different buckets [1]. The flink-filesystem module contains different file system impl

difference b/t filesystem and connector-filesystem

2018-01-12 Thread cw7k
Hi, I'm trying to understand the difference between the flink-filesystem and flink-connector-filesystem.  How is each intended to be used? If adding support for a different storage provider that supports HDFS, should additions be made to one or the other, or both?  Thanks.