[
https://issues.apache.org/jira/browse/CHUKWA-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717960#action_12717960
]
Ari Rabkin commented on CHUKWA-203:
-----------------------------------
I don't entirely understand the scope of this. I had thought our model was
that adaptors conceal rotation from stages farther up the line?
I definitely like the idea of issuing an "end of file, adaptor has
deregistered" chunk/marker.
> Track data loading from agent
> -----------------------------
>
> Key: CHUKWA-203
> URL: https://issues.apache.org/jira/browse/CHUKWA-203
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection, Data Processors
> Reporter: Jerome Boulon
> Priority: Critical
>
> Chukwa needs to track progress on all files for completeness reason.
> The first step could be to send adaptor information to the backend for
> postprocess/storage.
> This could be done at the same time of the writing checkpoint file by
> building a chunk and post it to the queue.
> In addition to that, we need to track all Add/Remove operations and the final
> offset for all files, the easiest way to do this will be to generate this
> information at the beginning and the end of each adaptor.
> Based on that, we should be able to:
> - track any file from the add to the remove,
> - validate that all data has been sent
> - track all files' rotation.
> - record any permission issue (expiration policy)
> - generate alerts
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
