[
https://issues.apache.org/jira/browse/CHUKWA-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710262#action_12710262
]
Ari Rabkin commented on CHUKWA-204:
-----------------------------------
My proposal is to bundle this functionality into an Adaptor -- I believe
Adaptors have, or should have, enough control over the Agent process to look
through the list of current adaptors, and stop the ones that need stopping.
And making this an Adaptor means that we can reuse the existing checkpointing
and adaptor configuration mechanisms.
> Agent side file expiration/removal
> ----------------------------------
>
> Key: CHUKWA-204
> URL: https://issues.apache.org/jira/browse/CHUKWA-204
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection, Data Processors
> Reporter: Jerome Boulon
> Priority: Critical
>
> We have a shell script in order to expire files older than x days.
> The idea is to move this to ChukwaAgent in order to reduce the number of
> daemon running and the overall complexity.
> This could be done in a background/low priority Cleanup Thread inside the
> agent.
> The input should be a configuration file with:
> - path + fileName + regex (similar to the expiration in our Log4J appender)
> - number of days (last accessed/modified? date)
> The initial version could be as simple as this but a more advanced version
> will be needed.
> The advanced version should be able to track data completeness based on
> CHUKWA-203 and only remove file that has been completely sent over or
> generate alerts.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
