Hi Stuti, There's 2 output in Chukwa. 1- Collectors are writing SeqFile in this format: <ChukwaArchiveKey and ChunkImpl> 1.1- Archives are in the same format format: <ChukwaArchiveKey and ChunkImpl> 2- Demux output is in this format:<ChukwaRecordKey, ChukwaRecord>
So if you want to have your Demux output in Avro format then you need to have your own AvroOutputFormat in Demux. I've already done some work to be able to use any Hadoop output format at the demux level by I haven't publish my code yet. What is your time range? /Jerome. On 5/16/10 9:58 PM, "Stuti Awasthi" <stuti_awas...@persistent.co.in> wrote: > > > Hello Guys, > > > I am a newbie to chukwa and I am trying to convert the chukwa sequence file > produced by the demuxer(<ChukwaRecordKey, ChukwaRecord>) file format to avro > format. Currently I am using Chukwa 0.3.0 > > Could setup and run chukwa successfully on a ubuntu machine, the agent and > collector were started successfully and files were created in finalArchives > folder. > > The output format of the files in FinalArchives is of type <ChukwaArchiveKey > and ChunkImpl> but according to the chukwa document and my findings I think > that files should be of format <ChukwaRecordKey, ChukwaRecord>. > > I used chukwa-data-processors.sh to start the dataprocessor and change the > chukwa-demux-conf.xml property to Stream. > <property> > <name>archive.grouper</name> > <value>Stream</value> > <description>How to group archive files. Choices are Hourly, Daily, DataType, > and Stream.</description> > </property> > > I looked into the source code of demux.java which takes <ChukwaArchiveKey > and ChunkImpl> (o/p of archiver) as input and gives <ChukwaRecordKey, > ChukwaRecord> as output. But I am not sure why it is not happening in my case. > > I want to feed my final output to the MetricDataLoader class which takes > <ChukwaRecordKey, ChukwaRecord> as Input, Please let me know if I am missing > something here. > > What should be the correct behavior of this whole process. Any pointers > regarding this would be helpful > > > Thanks in advance, > > Stuti > > > > > DISCLAIMER > ========== > This e-mail may contain privileged and confidential information which is the > property of Persistent Systems Ltd. It is intended only for the use of the > individual or entity to which it is addressed. If you are not the intended > recipient, you are not authorized to read, retain, copy, print, distribute or > use this message. If you have received this communication in error, please > notify the sender and delete all copies of this message. Persistent Systems > Ltd. does not accept any liability for virus infected mails. >
