Re: Move files read by flink

so, if you want to have your file parsed try to stay away from flink file parser (v1.4). Use nifi to parse files and then you could use Kafka or Flink. My data pipeline looks like: ftp <-> nifi <-> kafka <-> flink -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Move files read by flink

>From path flink thread it seems like there is thread showing this. But just wonder is this resolved?Where to find if final split is complete

Re: Move files read by flink

Alternatively static method FileSystem.get > On 10. Mar 2018, at 10:36, flinkuser101 wrote: > > Where does Flink expose filesystem? Is it from env? or inputstream? > > TextInputFormat format = new TextInputFormat(new > org.apache.flink.core.fs.Path(url.toString())); > DataStream inputStream = >

Re: Move files read by flink

Path has a method getFileSystem > On 10. Mar 2018, at 10:36, flinkuser101 wrote: > > Where does Flink expose filesystem? Is it from env? or inputstream? > > TextInputFormat format = new TextInputFormat(new > org.apache.flink.core.fs.Path(url.toString())); > DataStream inputStream = > env.readFi

Re: Move files read by flink

Where does Flink expose filesystem? Is it from env? or inputstream? TextInputFormat format = new TextInputFormat(new org.apache.flink.core.fs.Path(url.toString())); DataStream inputStream = env.readFile(format,url.toString(),FileProcessingMode.PROCESS_CONTINUOUSLY, 1); DataStream parsedStream =

Re: Move files read by flink

FileSystem class provides by Flink. https://ci.apache.org/projects/flink/flink-docs-master/api/java/ > On 10. Mar 2018, at 00:44, flinkuser101 wrote: > > Is there any way to do that? I have been searching for way to do that but in > vain. > > > > -- > Sent from: > http://apache-flink-user-m

Re: Move files read by flink

Is there any way to do that? I have been searching for way to do that but in vain. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Move files read by flink

Why don’t you let your flink job move them once it’s done? > On 9. Mar 2018, at 03:12, flinkuser101 wrote: > > I am reading files from a folder suppose > > /files/* > > Files are pushed into that folder. > > /files/file1_2018_03_09.csv > /files/file2_2018_03_09.csv > > Flink is reading fil