Hello Akanksha, Thank you for reporting this! It looks that website misses the details how to use “org.apache.beam.sdk.io.hdfs” with different FSs, so I created a Jira to improve this: https://issues.apache.org/jira/browse/BEAM-4397 <https://issues.apache.org/jira/browse/BEAM-4397>
WBR, Alexey > On 24 May 2018, at 10:36, Akanksha Sharma B <[email protected]> > wrote: > > Hi, > > Answering my own question 😊 > > writing to maprfs worked after I added following property to core-site.xml:- > > <property> > <name>fs.maprfs.impl</name> > <value>com.mapr.fs.MapRFileSystem</value> > </property> > > > Regards, > Akanksha > > From: Akanksha Sharma B > Sent: Thursday, May 24, 2018 9:24:25 AM > To: [email protected] > Subject: writing to maprfs? > > Hi All, > > I have a pipeline; it writes to hdfs, using "org.apache.beam.sdk.io.hdfs" > package. > I was hoping that it will work with maprfs as well. However I have been > debugging for some days, with no success. > > I do not provide hdfsConfiguration from command line, and instead use the > configuration file /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/core-site.xml > (please find it attached with the mail) > However I get the stack trace ("No FileSystem for scheme: maprfs"). Please > find the detailed trace attached. I made sure that I > have/opt/mapr/lib/maprfs-5.2.2-mapr.jar on classpath. I use spark submit to > run the pipeline. > > If you have already tried writing to maprfs from beam, ccould you please help? > > Regards, > Akanksha
