Arun, could you please port the discussion to wiki. That would be very
helpful. Thanks.
Arun C Murthy wrote:
On Sat, Sep 22, 2007 at 02:35:51AM +0530, Arun C Murthy wrote:
On Fri, Sep 21, 2007 at 01:53:21PM -0700, Joydeep Sen Sarma wrote:
Why don't u create/write to hdfs files directly from reduce job (don't
depend on the default reduce output dir/files)?
Like the cases where input is not homogenous, this seems (at least to
me) to be another common pattern (output is not homogenous). I have run
into this when loading data into hadoop (and wanting to organize
different types of records into different dirs/files).
Just make sure
(somehow), that different reduce jobs don't try to write to same file.
Quick note: as long as you create files in the 'mapred.output.dir' directory
(via map/reduce tasks) on hdfs, the framework will handle issues with
speculative tasks etc.
To clarify:
${mapred.output.dir} is the eventual output directory for the job
(JobConf.setOutputDir).
${taskid} is the actual id of the individual task-attempt (e.g.
task_200709221812_0001_m_000000_0), a TIP is a bunch of ${taskid}s (e.g.
task_200709221812_0001_m_000000).
With speculative execution *on*, one could face issues with 2 instances of the
same TIP (running simultaneously) trying to open/write-to the same file (path)
on hdfs. Hence the app-writer will have to pick unique names (e.g. using the
complete taskid i.e. task_200709221812_0001_m_000000_0) per task-attempt, not
just per TIP. (Clearly, this needs to be done even if the user doens't create
his files directly via reduce tasks.)
To get around this the framework helps the app-writer out by maintaining a special ${mapred.output.dir}/_${taskid} sub-dir for each task-attempt on hdfs where the output of the reduce task-attempt goes. On successful completion of the task-attempt the files in the ${mapred.output.dir}/_${taskid} (of the successful taskid only) are moved to ${mapred.output.dir}. Of course, the framework discards the sub-dir of unsuccessful task-attempts. This is completely transparent to the application.
The app-writer can take advantage of this by creating any side-files required in ${mapred.output.dir} during execution of his reduce-task, and the framework will move them out similarly - thus you don't have to pick unique paths per task-attempt.
Fine-print: the value of ${mapred.output.dir} during execution of a particular
task-attempt is actually ${mapred.output.dir}/_{$taskid}, not the value set by
JobConf.setOutputDir(). So, just create any hdfs files you want in
${mapred.output.dir} from your reduce task to take advantage of this feature.
The entire discussion holds true for maps of jobs with reducer=NONE (i.e. 0
reduces) since output of the map, in that case, goes directly to hdfs.
Arun
Arun
-----Original Message-----
From: C G [mailto:[EMAIL PROTECTED]
Sent: Friday, September 21, 2007 1:20 PM
To: [email protected]
Subject: Multiple output files, and controlling output file name...
Hi All:
In the context of using the aggregation classes, is there anyway to
send output to multiple files? In my case, I am processing columnar
records that are very wide. I have to do a variety of different
aggregations and the results of each type of aggregation is a set of
rows suitable for loading into a database. Rather than write all the
records to "part-00000", etc., I'd like to write them to a series of
files based. I don't see an obvious way to do this..is it possible?
Also, for those of us that don't like "part-00000" and so forth as
naming conventions, is there a way to name the output? In my case,
incorporating a date/time stamp like "loadA-200709221600" would be very
useful.
Thanks for any advice,
C G
---------------------------------
Tonight's top picks. What will you watch tonight? Preview the hottest
shows on Yahoo! TV.
