Track data loading from agent
-----------------------------
Key: CHUKWA-203
URL: https://issues.apache.org/jira/browse/CHUKWA-203
Project: Hadoop Chukwa
Issue Type: New Feature
Components: data collection, Data Processors
Reporter: Jerome Boulon
Priority: Critical
Chukwa needs to track progress on all files for completeness reason.
The first step could be to send adaptor information to the backend for
postprocess/storage.
This could be done at the same time of the writing checkpoint file by building
a chunk and post it to the queue.
In addition to that, we need to track all Add/Remove operations and the final
offset for all files, the easiest way to do this will be to generate this
information at the beginning and the end of each adaptor.
Based on that, we should be able to:
- track any file from the add to the remove,
- validate that all data has been sent
- track all files' rotation.
- record any permission issue (expiration policy)
- generate alerts
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
