On Sat, Sep 22, 2007 at 02:35:51AM +0530, Arun C Murthy wrote:
>On Fri, Sep 21, 2007 at 01:53:21PM -0700, Joydeep Sen Sarma wrote:
>>Why don't u create/write to hdfs files directly from reduce job (don't
>>depend on the default reduce output dir/files)?
>>
>>Like the cases where input is not homogenous, this seems (at least to
>>me) to be another common pattern (output is not homogenous). I have run
>>into this when loading data into hadoop (and wanting to organize
>>different types of records into different dirs/files).
>
>>Just make sure
>>(somehow), that different reduce jobs don't try to write to same file.
>>
>
>Quick note: as long as you create files in the 'mapred.output.dir' directory
>(via map/reduce tasks) on hdfs, the framework will handle issues with
>speculative tasks etc.
>
To clarify:
${mapred.output.dir} is the eventual output directory for the job
(JobConf.setOutputDir).
${taskid} is the actual id of the individual task-attempt (e.g.
task_200709221812_0001_m_000000_0), a TIP is a bunch of ${taskid}s (e.g.
task_200709221812_0001_m_000000).
With speculative execution *on*, one could face issues with 2 instances of the
same TIP (running simultaneously) trying to open/write-to the same file (path)
on hdfs. Hence the app-writer will have to pick unique names (e.g. using the
complete taskid i.e. task_200709221812_0001_m_000000_0) per task-attempt, not
just per TIP. (Clearly, this needs to be done even if the user doens't create
his files directly via reduce tasks.)
To get around this the framework helps the app-writer out by maintaining a
special ${mapred.output.dir}/_${taskid} sub-dir for each task-attempt on hdfs
where the output of the reduce task-attempt goes. On successful completion of
the task-attempt the files in the ${mapred.output.dir}/_${taskid} (of the
successful taskid only) are moved to ${mapred.output.dir}. Of course, the
framework discards the sub-dir of unsuccessful task-attempts. This is
completely transparent to the application.
The app-writer can take advantage of this by creating any side-files required
in ${mapred.output.dir} during execution of his reduce-task, and the framework
will move them out similarly - thus you don't have to pick unique paths per
task-attempt.
Fine-print: the value of ${mapred.output.dir} during execution of a particular
task-attempt is actually ${mapred.output.dir}/_{$taskid}, not the value set by
JobConf.setOutputDir(). So, just create any hdfs files you want in
${mapred.output.dir} from your reduce task to take advantage of this feature.
The entire discussion holds true for maps of jobs with reducer=NONE (i.e. 0
reduces) since output of the map, in that case, goes directly to hdfs.
Arun
>Arun
>
>>
>>
>>-----Original Message-----
>>From: C G [mailto:[EMAIL PROTECTED]
>>Sent: Friday, September 21, 2007 1:20 PM
>>To: [email protected]
>>Subject: Multiple output files, and controlling output file name...
>>
>>Hi All:
>>
>> In the context of using the aggregation classes, is there anyway to
>>send output to multiple files? In my case, I am processing columnar
>>records that are very wide. I have to do a variety of different
>>aggregations and the results of each type of aggregation is a set of
>>rows suitable for loading into a database. Rather than write all the
>>records to "part-00000", etc., I'd like to write them to a series of
>>files based. I don't see an obvious way to do this..is it possible?
>>
>> Also, for those of us that don't like "part-00000" and so forth as
>>naming conventions, is there a way to name the output? In my case,
>>incorporating a date/time stamp like "loadA-200709221600" would be very
>>useful.
>>
>> Thanks for any advice,
>> C G
>>
>>
>>---------------------------------
>>Tonight's top picks. What will you watch tonight? Preview the hottest
>>shows on Yahoo! TV.
