[
https://issues.apache.org/jira/browse/CHUKWA-26?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12696605#action_12696605
]
Jerome Boulon commented on CHUKWA-26:
-------------------------------------
Since we still don't have a wiki for Chukwa, I'll put more information here.
Corinne will work on documented this, like all the Chukwa documentation.
All new daemon are responsible for taking data from the previous step and
producing data for the next one.
Each one running asynchronously from the others
- Collector-> DataSink (input for DemuxManager)
- DemuxManager
|-> Demux output (ChukwaRecord, input for PostProcessorManager)
|-> move dataSink file to dataSinkArchive directory
- PostProcessorManager
|-> consume demux output, load to database
|-> move ChukwaRecord to /chukwa/repos/...
- ArchiveManager
|-> every 2 hours compact dataSink files
- HourlyRolling
|-> same as before except a fileName change, the filname now contains
"HourlyDone" so I can guarantee that the Hourly was done
- DailyRolling
|-> same as before except that we are now waiting for hourlyRolling to be
done before processing a day
>>What does DemuxManager do?
DemuxManager is a daemon process.
It takes care of scheduling Demux on DataSink files, limit the number of input
file to demux, force a reprocess of any dataSink files that were part of the
previous demux if DemuxManager has been killed and after 3 attempts to process
the same list of DataSink files, DemuxManager automatically move those faulty
dataSink file to an Error directory
>>What does PostProcessorManager do?
Load all demuxOutput to DB
>>Do I need Nagios?
-No, if you're not adding your nagios information to chukwa-demux-conf.xml,
DemuxManager will not send anything to Nagios
>>or I think I have it but it's down
-Nothing, DemuxManager will try to send an NSCA command via a socket
connection, this command has no impact on DemuxManager.
> Rewrite processSinkFiles.sh in java to have a better error handling
> -------------------------------------------------------------------
>
> Key: CHUKWA-26
> URL: https://issues.apache.org/jira/browse/CHUKWA-26
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: Data Processors
> Reporter: Jerome Boulon
> Assignee: Jerome Boulon
> Priority: Blocker
> Attachments: CHUKWA-26-2.patch, CHUKWA-26.patch,
> NagiosAppender-1.5.0.jar
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
