This is an interesting question. I’m curious to know as well how this problem can be approached.
Is there a way, perhaps, to ensure that each input file matching the glob expression gets mapped to exactly one partition? Then you could probably get what you want using RDD.mapPartitions(). Nick On Wed, Jul 30, 2014 at 12:02 AM, Fengyun RAO <[email protected]> wrote: > Hi, all > > We are migrating from mapreduce to spark, and encountered a problem. > > Our input files are IIS logs with file head. It's easy to get the file > head if we process only one file, e.g. > > val lines = sc.textFile('hdfs://*/u_ex14073011.log') > val head = lines.take(4) > > Then we can write our map method using this head. > > However, if we input multiple files, each of which may have a different > file head, how can we get file head for each partition? > > It seems we have two options: > > 1. still use textFile() to get lines. > > Since each partition may have a different "head", we have to write > mapPartitionsWithContext method. However we can't find a way to get the > "head" for each partition. > > In our former mapreduce program, we could simply use > > Path path = ((FileSplit) context.getInputSplit()).getPath() > > but there seems no way in spark, since HadoopPartition which wraps > InputSplit inside HadoopRDD is a private class. > > 2. use wholeTextFile() to get whole contents. > > It's easy to get file head for each file, but according to the document, > this API is better for small files. > > > *Any suggestions on how to process these files with heads?* >
