[
https://issues.apache.org/jira/browse/HADOOP-12759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15132741#comment-15132741
]
Daniel Templeton commented on HADOOP-12759:
-------------------------------------------
Thanks for the reviews, [~rkanter] and [~andrew.wang]!
I'll give it a go with a static variable and add some paragraph tags.
On the probing logic, the reason I do it that way is to get synchronization
across daemons. I let HDFS sort out who gets any given file name. If I list
files first, the list of files could change by the time I go to create the file.
On removing the throws clause, I'm pretty sure that has no adverse effect. It
still adhere's to the interface regardless of whether the throws cause is
contracted or the return type is more specific.
About sleeping in the flush test, let's talk alternatives. I did actually write
the test with a sleep. If I don't wait for the flusher thread to roll over and
flush, then how do I test it? I can force the flush to happen on demand, but
if I do that, I'm not testing the timing of the thread; I'm just testing that
the flush behaves as expected. Do we just assume that timing works and only
test the behavior?
> RollingFileSystemSink should eagerly rotate directories
> -------------------------------------------------------
>
> Key: HADOOP-12759
> URL: https://issues.apache.org/jira/browse/HADOOP-12759
> Project: Hadoop Common
> Issue Type: Improvement
> Affects Versions: 2.8.0
> Reporter: Daniel Templeton
> Assignee: Daniel Templeton
> Priority: Critical
> Attachments: YARN-4664.001.patch
>
>
> The RollingFileSystemSink only rolls over to a new directory if a new metrics
> record comes in. The issue is that HDFS does not update the file size until
> it's closed (HDFS-5478), and if no new metrics record comes in, then the file
> size will never be updated.
> This JIRA is to add a background thread to the sink that will eagerly close
> the file at the top of the hour.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
