How does wholeTextFiles() work in Spark-Hadoop Cluster?

2016-09-08 Thread Nisha Menon
I have an RDD created as follows: *JavaPairRDD inputDataFiles = sparkContext.wholeTextFiles("hdfs://ip:8020/user/cdhuser/inputFolder/");* On this RDD I perform a map to process individual files and invoke a foreach to trigger the same map. * JavaRDD output =

How does wholeTextFiles() work in Spark-Hadoop Cluster?

2016-09-21 Thread Nisha Menon
he worker logs or the driver? > > > On Thursday, September 8, 2016, Nisha Menon <nisha.meno...@gmail.com> > wrote: > >> I have an RDD created as follows: >> >> *JavaPairRDD<String,String> inputDataFiles = >> sparkContext.wholeTextFiles("

Re: How does wholeTextFiles() work in Spark-Hadoop Cluster?

2016-09-21 Thread Nisha Menon
a.a...@gmail.com> wrote: > On yarn, logs are aggregated from each containers to hdfs. You can use > yarn CLI or ui to view. For spark, you would have a history server which > consolidate s the logs > On 21 Sep 2016 19:03, "Nisha Menon" <nisha.meno...@gmail.com> wrote