[
https://issues.apache.org/jira/browse/CHUKWA-185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719664#action_12719664
]
Jerome Boulon commented on CHUKWA-185:
--------------------------------------
Ari, it will be good to have a better control on TerminatorThread ... maybe a
pool of TerminatorThread instead of creating a new one every time. A simpler
solution will be to limit the number of "running" TerminatorThread's
instances...
Also I'm not sure if the solution could so simple.
If the agent crash, it shouldn't resend something that has already been sent.
Here what I was thinking of:
- make the timeWindow mandatory, could default to XX minutes
- keep track of all files that are in the processing window (
file.lastModifiedDate > now - timeWindow), using a simple text file, (tracking
file)
- when the last modified date for a file exceed the timeWindow then:
---> do a shutdown on the adaptor for this file's entry
---> delete the file's entry from the tracking file
- keep the tracking file in a chukwa directory and reload it at agent re-start
to avoid sending the same file twice
How do you stop tailing a file? We cannot assume that we can delete a file so
we need to have that built in. My proposal is to use the last modified date and
the timeWindow to automatically remove adaptors.
> ability to tail a whole directory
> ---------------------------------
>
> Key: CHUKWA-185
> URL: https://issues.apache.org/jira/browse/CHUKWA-185
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection
> Affects Versions: 0.1.2, 0.2.0
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Fix For: 0.3.0
>
> Attachments: CHUKWA-185.patch
>
>
> Right now, FileTailingAdaptors watch particular files. It'd be great to be
> able to watch a whole path: to say something like /var/logs/*, where new logs
> created in that directory get picked up.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
