Yes, Chunk.getDataType() is the right way to do it. Thanks regards, Eric
On Fri, Jul 12, 2013 at 2:01 AM, IvyTang <[email protected]> wrote: > Fix: > Don't need to add the argument dataType in > AbstractProcessor.parse(String recordEntry, > OutputCollector<ChukwaRecordKey, ChukwaRecord> output, Reporter reporter). > I can get the datatype from the chunk. > -- > Best regards, > > Ivy Tang > > On 2013-7-12, at 下午3:04, IvyTang <[email protected]> wrote: > > > Hi,Eric > > In order to applying to all multiple datatypes in Demux,init the Reducer > class depends on the ChukwaRecordKey's datatype . I need the datatype in > ChukwaRecordKey. > > > > The ChukwaRecordKey needs to add a field named dataType , showing this > ChukwaRecord's datatype . And also i need add an argument in > > parse(String recordEntry, OutputCollector<ChukwaRecordKey, ChukwaRecord> > output, Reporter reporter) > > > > An argument String dataType needs to be passed to this method to build > the ChukwaRecordKey. > > > > Adding a field in ChukwaRecordKey ,and add an argument in method > AbstractProcessor.parse(String recordEntry, > OutputCollector<ChukwaRecordKey, ChukwaRecord> output, Reporter reporter) > > > > Is that acceptable? This is the only way i can think of to make the > custom reducer more configurable. > > > > -- > > Best regards, > > > > Ivy Tang > > > > On 2013-7-10, at 下午12:01, Eric Yang <[email protected]> wrote: > > > >> Yes that will help. Please include test case as well. Thanks Ivy. > >> > >> regards, > >> Eric > >> > >> > >> On Mon, Jul 8, 2013 at 10:42 PM, IvyTang <[email protected]> wrote: > >> > >>> This issue has been reported > >>> https://issues.apache.org/jira/browse/CHUKWA-581. > >>> > >>> And i think i can submit a patch later . > >>> > >>> Should i open a new jira or just assign this CHUKWA-581 to myself? > >>> > >>> This can implement reducer outside chukwa. > >>> > >>> -- > >>> Best regards, > >>> > >>> Ivy Tang > >>> > >>> On 2013-7-9, at 下午1:22, Eric Yang <[email protected]> wrote: > >>> > >>>> The recommended change can improve usability to implement parsers > outside > >>>> of Chukwa packages. +1 on the change. Do you mind open a jira for > this? > >>>> Thanks > >>>> > >>>> regards, > >>>> Eric > >>>> > >>>> > >>>> On Mon, Jul 8, 2013 at 6:54 AM, 唐 颖 <[email protected]> wrote: > >>>> > >>>>> We can write our own mapper ,because MapProcessorFactory uses > >>>>> Class.forName(parserClass).getConstructor().newInstance(); > >>>>> since we config the parserClass in config file. > >>>>> > >>>>> But the ReduceProcessorFactory inits the reducer class using the > >>>>> classname > >>> "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+ > >>>>> reduceType. > >>>>> So the reducer class must be put in package > >>>>> org.apache.hadoop.chukwa.extraction.demux.processor.reducer and named > >>>>> reduceType. > >>>>> > >>>>> Why not we put the reducer class name in config file either? > >>>>> For exam, the {customdatatype_mapper} means mapper class > >>>>> ,{customdatatype_reducer} means reducer class. > >>>>> > >>>>> This can give chukka more flexibility. > >>>>> > >>>>> -- > >>>>> Best regards, > >>>>> > >>>>> Ivy Tang > >>>>> > >>>>> > >>> > >>> > > > >
