This is an interesting question. I’m curious to know as well how this
problem can be approached.

Is there a way, perhaps, to ensure that each input file matching the glob
expression gets mapped to exactly one partition? Then you could probably
get what you want using RDD.mapPartitions().

Nick
​


On Wed, Jul 30, 2014 at 12:02 AM, Fengyun RAO <[email protected]> wrote:

> Hi, all
>
> We are migrating from mapreduce to spark, and encountered a problem.
>
> Our input files are IIS logs with file head. It's easy to get the file
> head if we process only one file, e.g.
>
> val lines = sc.textFile('hdfs://*/u_ex14073011.log')
> val head = lines.take(4)
>
> Then we can write our map method using this head.
>
> However, if we input multiple files, each of which may have a different
> file head, how can we get file head for each partition?
>
> It seems we have two options:
>
> 1. still use textFile() to get lines.
>
> Since each partition may have a different "head", we have to write
> mapPartitionsWithContext method. However we can't find a way to get the
> "head" for each partition.
>
> In our former mapreduce program, we could simply use
>
> Path path = ((FileSplit) context.getInputSplit()).getPath()
>
> but there seems no way in spark, since HadoopPartition which wraps
> InputSplit inside HadoopRDD is a private class.
>
> 2. use wholeTextFile() to get whole contents.
>
>  It's easy to get file head for each file, but according to the document,
> this API is better for small files.
>
>
> *Any suggestions on how to process these files with heads?*
>

Reply via email to