You don't really need a custom input format, I don't think.

You should be able to just add multiple inputs, one at a time after
filtering them outside hadoop.


On 4/14/08 10:59 AM, "Alfonso Olias Sanz" <[EMAIL PROTECTED]>
wrote:

> ok thanks for the info :)
> 
> On 11/04/2008, Arun C Murthy <[EMAIL PROTECTED]> wrote:
>> 
>>  On Apr 11, 2008, at 10:21 AM, Amar Kamat wrote:
>> 
>> 
>>> A simpler way is to use
>> FileInputFormat.setInputPathFilter(JobConf, PathFilter).
>> Look at org.apache.hadoop.fs.PathFilter for details on PathFilter interface.
>>> 
>> 
>>  +1, although FileInputFormat.setInputPathFilter is
>> available only in hadoop-0.17 and above... like Amar mentioned previously,
>> you'd have to have a custom InputFormat prior to hadoop-0.17.
>> 
>>  Arun
>> 
>> 
>> 
>>> Amar
>>> Alfonso Olias Sanz wrote:
>>> 
>>>> Hi
>>>> I have a general purpose input folder that it is used as input in a
>>>> Map/Reduce task. That folder contains files grouped by names.
>>>> 
>>>> I want to configure the JobConf in a way I can filter the files that
>>>> have to be processed from that pass (ie  files which name starts by
>>>> Elementary, or Source etc)  So the task function will only process
>>>> those files.  So if the folder contains 1000 files and only 50 start
>>>> by Elementary. Only those 50 will be processed by my task.
>>>> 
>>>> I could set up different input folders and those containing the
>>>> different files, but I cannot do that.
>>>> 
>>>> 
>>>> Any idea?
>>>> 
>>>> thanks
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 

Reply via email to