Re: Writing to an HDFS file from a Flink stream job

Thanks for all your directions. BucketingSink worked. Isuru On Thu, Oct 12, 2017 at 9:05 AM, Piotr Nowojski wrote: > I think the issue might be that writeAsText (TextOutputFormat) doesn’t > flush the data anywhere (only on close, which in streaming doesn’t happen). >

Re: Writing to an HDFS file from a Flink stream job

I think the issue might be that writeAsText (TextOutputFormat) doesn’t flush the data anywhere (only on close, which in streaming doesn’t happen). You would need to use custom output format, but as Aljoscha pointed out BucketingSink makes more sense for streaming applications. Piotrek > On 12

Re: Writing to an HDFS file from a Flink stream job

Hi, Maybe this is an access rights issue? Could you try to create and write to same file (same directory) in some other way (manually?), using the same user and the same machine as would Flink job do? Maybe there will be some hint in hdfs logs? Piotrek > On 12 Oct 2017, at 00:19, Isuru

Writing to an HDFS file from a Flink stream job

Hi all, I'm just trying to use an HDFS file as the sink for my flink stream job. I use the following line to do so. stream.writeAsText("hdfs://hadoop-master:9000/user/isuru/foo"); I have not set "fs.hdfs.hadoopconf" in my flink configuration as it should work with the full hdfs file name