It should just work in PySpark, the same way it does in Java / Scala apps.
Matei
On Oct 1, 2014, at 4:12 PM, Sungwook Yoon <sy...@maprtech.com> wrote:
>
> Yes.. you should use maprfs://
>
> I personally haven't used pyspark, I just used scala shell or standalone with
> MapR.
>
> I think you need to set classpath right, adding jar like
> /opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-0.20.2-dev-core.jar to the
> classpath
> in the classpath.
>
> Sungwook
>
> On Wed, Oct 1, 2014 at 4:09 PM, Addanki, Santosh Kumar
> <santosh.kumar.adda...@sap.com> wrote:
> Hi
>
>
>
> We would like to do this in PySpark Environment
>
>
>
> i.e something like
>
>
>
> test = sc.textFile("maprfs:///user/root/test") or
>
> test = sc.textFile("hdfs:///user/root/test") or
>
>
>
>
>
> Currently when we try
>
> test = sc.textFile("maprfs:///user/root/test")
>
>
>
> It throws the error
>
> No File-System for scheme: maprfs
>
>
>
>
>
> Best Regards
>
> Santosh
>
>
>
>
>
>
>
> From: Vladimir Rodionov [mailto:vrodio...@splicemachine.com]
> Sent: Wednesday, October 01, 2014 3:59 PM
> To: Addanki, Santosh Kumar
> Cc: user@spark.apache.org
> Subject: Re: Spark And Mapr
>
>
>
> There is doc on MapR:
>
>
>
> http://doc.mapr.com/display/MapR/Accessing+MapR-FS+in+Java+Applications
>
>
>
> -Vladimir Rodionov
>
>
>
> On Wed, Oct 1, 2014 at 3:00 PM, Addanki, Santosh Kumar
> <santosh.kumar.adda...@sap.com> wrote:
>
> Hi
>
>
>
> We were using Horton 2.4.1 as our Hadoop distribution and now switched to MapR
>
>
>
> Previously to read a text file we would use :
>
>
>
> test = sc.textFile(\"hdfs://10.48.101.111:8020/user/hdfs/test\")"
>
>
>
>
>
> What would be the equivalent of the same for Mapr.
>
>
>
> Best Regards
>
> Santosh
>
>
>
>
