[
https://issues.apache.org/jira/browse/CHUKWA-581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
IvyTang updated CHUKWA-581:
---------------------------
Tags: CustomReducer
Fix Version/s: 0.5.0
0.6.0
Labels: patch (was: )
Affects Version/s: 0.6.0
0.5.0
Status: Patch Available (was: Open)
The ReduceProcessorFactory inits the reducer class using the classname
"org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+ reduceType.So
the reducer class must be put in package
org.apache.hadoop.chukwa.extraction.demux.processor.reducer and named
reduceType.
This is not that custom.
So i chaneg the implementation of ReduceProcessorFactory inits the reducer
class ,let the ReduceProcessorFactory read the reducer class name from
chukwa-demux-conf.xml ,like the way how the MapperProcessorFactory handle the
custom mapper class.
So for the users, If you write your own mapper class ClassA and reducer class
ClassB. And you want your DataType1 log be processed by ClassA,and you set the
ChukwaRecordKey#ReduceType to be ReduceType1 in ClassA.process .All you need is
setting
<property>
<name>DataType1_mapper</name>
<value>ClassA's name</value>
<description></description>
</property>
<property>
<name>ReduceType1_reducer</name>
<value>ClassB's name</value>
<description></description>
</property>
.The the DataType1 log will be processed by MapperClass ClassA and ReducerClass
ClassB.
I
> Demux do not support user custom reducer package name.
> -------------------------------------------------------
>
> Key: CHUKWA-581
> URL: https://issues.apache.org/jira/browse/CHUKWA-581
> Project: Chukwa
> Issue Type: Improvement
> Components: MR Data Processors
> Affects Versions: 0.5.0, 0.4.0, 0.6.0
> Environment: Ubuntu 10.0.4,Hadoop 0.20
> Reporter: Yingzhong Xu
> Assignee: IvyTang
> Priority: Critical
> Labels: patch
> Fix For: 0.6.0, 0.5.0
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> In ReduceProcessorFactory.java,
> it only gets {reduceType}.class from this package,as following:
> bq. String path =
> "org.apache.hadoop.chukwa.extraction.demux.processor.reducer." + reduceType;
> So our custom reducer could only be placed in that package.
> Thus, When we add our {demux}.jar to Hadoop classpath, since the Hadoop
> framework will not be able to merge the class files from multiple jar files
> with the same package name, custom reduce class can not be found.
> I get a way to overcome this defect.Put custom reducer into a package named
> with "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+{custom}
> Here,{custom} is a part of package name.Then in parser,we can write like this:
> bq. Key.setReducerType("{custom}.YourReducer")
> In this way, Hadoop framework could be able to find our reduce class in this
> partial user defined new package.
> But I think it will be better to fix chukwa to remove the default package
> name and fully support custom package name for reducer.
> Thanks for Eric Yang's help.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
