[
https://issues.apache.org/jira/browse/CHUKWA-580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12982995#action_12982995
]
Ari Rabkin commented on CHUKWA-580:
-----------------------------------
This isn't a bug. That's the intended behavior. The idea being that if
DirTailingAdaptor finds a file needing tailing we can checkpoint that locally.
The checkpoint position of other adaptors won't be increased. So there's no
risk of data loss here.
> Checkpoints increasing without sending data to collector
> --------------------------------------------------------
>
> Key: CHUKWA-580
> URL: https://issues.apache.org/jira/browse/CHUKWA-580
> Project: Chukwa
> Issue Type: Bug
> Affects Versions: 0.4.0
> Environment: RHEL
> Reporter: Stuti Awasthi
>
> I have a query regarding the checkpoints in chukwa. According to theory :
> Every few minutes, each agent process polls a collector to find the length of
> each file to which data is being written. The length of the file is then
> compared with the offset at which each chunk was to be written. If the file
> length exceeds this value, then the data has been committed and the agent
> process advances its checkpoint accordingly.(Note that the length returned by
> the filesystem is the amount of data that has been successfully replicated.)
> This means that chukwa_agent_checkpoint would increase only when the agent
> receivers and ack from the collectors. But in case of dirtailing adapter,
> this is not correct. I have done the following steps to test this :
> - Started agent with some dummy collector which was not present.
> - Added dirtailing adapter with Charfile tailing adapter
> I can see the following output in my checkpoint file :
> ADD adaptor_67653208e8dea46c798e46753fc19dad =
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8
> Stuti 0 /root/Stuti/yum.log 0
> ADD adaptor_b505db62647203ffa3cfe17374042870 =
> org.apache.hadoop.chukwa.datacollection.adaptor.DirTailingAdaptor Stuti
> /root/Stuti filetailer.CharFileTailingAdaptorUTF8 1295014173306
> Since data is not getting sent to collector, so checkpoints should not
> increase.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
