[
https://issues.apache.org/jira/browse/HADOOP-372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615185#action_12615185
]
Alejandro Abdelnur commented on HADOOP-372:
-------------------------------------------
I find this feature useful but I'm not sure about how it has been implemented
(apologies for the late comment, just become aware of it when seeing the Jira
Resolved email).
*{{FileInputFormat}} API changes:*
All the methods added to the {{FileInputFormat}} are not related to
{{InputFormat}} functionality.
In my opinion they should be in a separate class (something like
{{MultipleInputs}} )
*Package for all the patch classes:*
All these classes should be in {{org.apache.hadoop.mapred.lib}} thus keeping
the core {{mapred}} with exactly that, the core, this is a useful extension but
still is an extension.
*{{JobConf}} for the different mappers:*
The current code forces all mappers to have the same {{JobConf}}, this means
that mappers have to be written with knowledge of the other mappers to avoid
collision in the configuration keys.
How about passing along a {{JobConf}} instance when adding an input path with
its own {{Mapper}}, something like HADOOP-3702 is doing when adding a mapper to
the chain?
> should allow to specify different inputformat classes for different input
> dirs for Map/Reduce jobs
> --------------------------------------------------------------------------------------------------
>
> Key: HADOOP-372
> URL: https://issues.apache.org/jira/browse/HADOOP-372
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.4.0
> Environment: all
> Reporter: Runping Qi
> Assignee: Chris Smith
> Fix For: 0.19.0
>
> Attachments: hadoop-372.patch, hadoop-372.patch, hadoop-372.patch,
> hadoop-372.patch, hadoop-372.patch
>
>
> Right now, the user can specify multiple input directories for a map reduce
> job.
> However, the files under all the directories are assumed to be in the same
> format,
> with the same key/value classes. This proves to be a serious limit in many
> situations.
> Here is an example. Suppose I have three simple tables:
> one has URLs and their rank values (page ranks),
> another has URLs and their classification values,
> and the third one has the URL meta data such as crawl status, last crawl
> time, etc.
> Suppose now I need a job to generate a list of URLs to be crawled next.
> The decision depends on the info in all the three tables.
> Right now, there is no easy way to accomplish this.
> However, this job can be done if the framework allows to specify different
> inputformats for different input dirs.
> Suppose my three tables are in the following directory respectively:
> rankTable, classificationTable. and metaDataTable.
> If we extend JobConf class with the following method (as Owen suggested to
> me):
> addInputPath(aPath, anInputFormatClass, anInputKeyClass,
> anInputValueClass)
> Then I can specify my job as follows:
> addInputPath(rankTable, SequenceFileInputFormat.class, UTF8.class,
> DoubleWritable.class)
> addInputPath(classificationTable, TextInputFormat.class, UTF8,class,
> UTF8.class)
> addInputPath(metaDataTable, SequenceFileInputFormat.class, UTF8.class,
> MyRecord.class)
> If an input directory is added through the current API, it will have the same
> meaning as it is now.
> Thus this extension will not affect any applications that do not need this
> new feature.
> It is relatively easy for the M/R framework to create an appropriate record
> reader for a map task based on the above information.
> And that is the only change needed for supporting this extension.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
