Re: combitedTextFile and CombineTextInputFormat

Hi Alex, >From my understanding the community is shifting the effort from RDD based APIs to Dataset/DataFrame based ones, so for me it is not so necessary to add a new RDD based API as I mentioned before. Also for the problem of so many partitions, I think there're many other solutions to handle

Re: combitedTextFile and CombineTextInputFormat

Saisai, Reynold, Thank you for your replies. I also think that many variation of textFile() methods might be confusing for users. Better to have just one good textFile() implementation. Do you think sc.textFile() should use CombineTextInputFormat instead of TextInputFormat?

Re: combitedTextFile and CombineTextInputFormat

>From my understanding I think newAPIHadoopFile or hadoopFIle is generic enough for you to support any InputFormat you wanted. IMO it is not so necessary to add a new API for this. On Fri, May 20, 2016 at 12:59 AM, Alexander Pivovarov wrote: > Spark users might not know

Re: combitedTextFile and CombineTextInputFormat

Spark users might not know about CombineTextInputFormat. They probably think that sc.textFile already implements the best way to read text files. I think CombineTextInputFormat can replace regular TextInputFormat in most of the cases. Maybe Spark 2.0 can use CombineTextInputFormat in sc.textFile

Re: combitedTextFile and CombineTextInputFormat

Not exacly the same as the one you suggested but you can chain it with flatMap to get what you want, if each file is not huge. On Thu, May 19, 2016, 8:41 AM Xiangrui Meng wrote: > This was implemented as sc.wholeTextFiles. > > On Thu, May 19, 2016, 2:43 AM Reynold Xin

Re: combitedTextFile and CombineTextInputFormat

It is different isn't it. Whole text files returns one element per file, whereas combined inout format is similar to coalescing partitions to bin pack into a certain size. On Thursday, May 19, 2016, Xiangrui Meng wrote: > This was implemented as sc.wholeTextFiles. > > On Thu,

Re: combitedTextFile and CombineTextInputFormat

This was implemented as sc.wholeTextFiles. On Thu, May 19, 2016, 2:43 AM Reynold Xin wrote: > Users would be able to run this already with the 3 lines of code you > supplied right? In general there are a lot of methods already on > SparkContext and we lean towards the more

Re: combitedTextFile and CombineTextInputFormat

Users would be able to run this already with the 3 lines of code you supplied right? In general there are a lot of methods already on SparkContext and we lean towards the more conservative side in introducing new API variants. Note that this is something we are doing automatically in Spark SQL