Re: ArrayIndexOutOfBoundsException when reading bzip2 files

2014-06-09 Thread sam
Any idea when they will release it? Also I'm uncertain what we will need to do to fix the shell? Will we have to reinstall spark? or reinstall hadoop? (i'm not a devops so maybe this question sounds silly) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Arr

Re: ArrayIndexOutOfBoundsException when reading bzip2 files

2014-06-09 Thread MEETHU MATHEW
Hi Sean, Thank you for the fast response.   Thanks & Regards, Meethu M On Monday, 9 June 2014 6:04 PM, Sean Owen wrote: Have a search online / at the Spark JIRA. This was a known upstream bug in Hadoop. https://issues.apache.org/jira/browse/SPARK-1861 On Mon, Jun 9, 2014 at 7:54 AM, MEE

Re: ArrayIndexOutOfBoundsException when reading bzip2 files

2014-06-09 Thread Sean Owen
Have a search online / at the Spark JIRA. This was a known upstream bug in Hadoop. https://issues.apache.org/jira/browse/SPARK-1861 On Mon, Jun 9, 2014 at 7:54 AM, MEETHU MATHEW wrote: > Hi, > I am getting ArrayIndexOutOfBoundsException while reading from bz2 files in > HDFS.I have come across

Re: ArrayIndexOutOfBoundsException when reading bzip2 files

2014-06-09 Thread MEETHU MATHEW
Hi Akhil, Plz find the code below.  x = sc.textFile("hdfs:///**")  x = x.filter(lambda z:z.split(",")[0]!=' ')  x = x.filter(lambda z:z.split(",")[3]!=' ')  z = x.reduce(add)   Thanks & Regards, Meethu M On Monday, 9 June 2014 5:52 PM, Akhil Das wrote: Can you paste the piece of code!?

Re: ArrayIndexOutOfBoundsException when reading bzip2 files

2014-06-09 Thread Akhil Das
Can you paste the piece of code!? Thanks Best Regards On Mon, Jun 9, 2014 at 5:24 PM, MEETHU MATHEW wrote: > Hi, > I am getting ArrayIndexOutOfBoundsException while reading from bz2 files > in HDFS.I have come across the same issue in JIRA at > https://issues.apache.org/jira/browse/SPARK-1861