Re: Pyspark dataframe read

2015-10-06 Thread Koert Kuipers
gt; >>> On Tue, Oct 6, 2015 at 3:02 AM, Blaž Šnuderl wrote: >>> >>>> Hello everyone. >>>> >>>> It seems pyspark dataframe read is broken for reading multiple files. >>>> >>>> sql.read.json( "file1,file2") fails with java.io.IOException: No input >>>> paths specified in job. >>>> >>>> This used to work in spark 1.4 and also still work with sc.textFile >>>> >>>> Blaž >>>> >>> >>> >>

Re: Pyspark dataframe read

2015-10-06 Thread Reynold Xin
21 AM, Koert Kuipers > wrote: > >> i ran into the same thing in scala api. we depend heavily on comma >> separated paths, and it no longer works. >> >> >> On Tue, Oct 6, 2015 at 3:02 AM, Blaž Šnuderl > > wrote: >> >>> Hello everyone. >>

Re: Pyspark dataframe read

2015-10-06 Thread Josh Rosen
2015 at 3:02 AM, Blaž Šnuderl wrote: > >> Hello everyone. >> >> It seems pyspark dataframe read is broken for reading multiple files. >> >> sql.read.json( "file1,file2") fails with java.io.IOException: No input >> paths specified in job. >>

Re: Pyspark dataframe read

2015-10-06 Thread Koert Kuipers
i ran into the same thing in scala api. we depend heavily on comma separated paths, and it no longer works. On Tue, Oct 6, 2015 at 3:02 AM, Blaž Šnuderl wrote: > Hello everyone. > > It seems pyspark dataframe read is broken for reading multiple files. > > sql.read.json( "

Pyspark dataframe read

2015-10-06 Thread Blaž Šnuderl
Hello everyone. It seems pyspark dataframe read is broken for reading multiple files. sql.read.json( "file1,file2") fails with java.io.IOException: No input paths specified in job. This used to work in spark 1.4 and also still work with sc.textFile Blaž