Re: FileBasedSource does not match files on HDFS

No problem, happy to help ;) Regards JB On 01/10/2018 11:15 AM, Shashank Prabhakara wrote: It was the missing hdfs filesystem extension dependency. Thanks Jean-Baptiste. Much appreciated. Regards, Shashank On Wed, Jan 10, 2018 at 2:09 PM, Jean-Baptiste Onofré > wrot

Re: FileBasedSource does not match files on HDFS

It was the missing hdfs filesystem extension dependency. Thanks Jean-Baptiste. Much appreciated. Regards, Shashank On Wed, Jan 10, 2018 at 2:09 PM, Jean-Baptiste Onofré wrote: > Hi > > Do you have the beam hdfs filesystem extension in the dependencies ? Did > you define the HADOOP_CONF_DIR env

Re: FileBasedSource does not match files on HDFS

Hi Do you have the beam hdfs filesystem extension in the dependencies ? Did you define the HADOOP_CONF_DIR env variable containing path to hdfs-site.xml ? Regards JB On 01/10/2018 08:55 AM, Shashank Prabhakara wrote: Hello, I'm testing some pipelines on a dataproc cluster with hadoop versio

FileBasedSource does not match files on HDFS

Hello, I'm testing some pipelines on a dataproc cluster with hadoop version 2.8.2, beam 2.3.0-SNAPSHOT. I have observed on our pipeline as well as the wordcount that ships with beam, that FileBasedSource does not "match" any files when using hdfs prefix - verified this with apex runner and direct