[
https://issues.apache.org/jira/browse/YARN-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935524#comment-15935524
]
Varun Saxena commented on YARN-6357:
------------------------------------
By the way, coming to the patch, although almost all the writer implementations
will have asynchronous writes and some buffering but there is no guarantee
(unlikely but writer implementations can choose to persist data to backend
immediately and choose to implement flush as a no-op), so lets name the method
writeTimelineEntities instead of writeTimelineEntitiesAsync.
Also persistOutstandingTimelineEntities can be renamed to
persistBufferedTimelineEntities. Or simply name the method flushEntities.
Other than this patch looks fine to me.
> Implement TimelineCollector#putEntitiesAsync
> --------------------------------------------
>
> Key: YARN-6357
> URL: https://issues.apache.org/jira/browse/YARN-6357
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2, timelineserver
> Affects Versions: YARN-2928
> Reporter: Joep Rottinghuis
> Assignee: Haibo Chen
> Labels: yarn-5355-merge-blocker
> Attachments: YARN-6357.01.patch, YARN-6357.02.patch
>
>
> As discovered and discussed in YARN-5269 the
> TimelineCollector#putEntitiesAsync method is currently not implemented and
> TimelineCollector#putEntities is asynchronous.
> TimelineV2ClientImpl#putEntities vs TimelineV2ClientImpl#putEntitiesAsync
> correctly call TimelineEntityDispatcher#dispatchEntities(boolean sync,...
> with the correct argument. This argument does seem to make it into the
> params, and on the server side TimelineCollectorWebService#putEntities
> correctly pulls the async parameter from the rest call. See line 156:
> {code}
> boolean isAsync = async != null && async.trim().equalsIgnoreCase("true");
> {code}
> However, this is where the problem starts. It simply calls
> TimelineCollector#putEntities and ignores the value of isAsync. It should
> instead have called TimelineCollector#putEntitiesAsync, which is currently
> not implemented.
> putEntities should call putEntitiesAsync and then after that call
> writer.flush()
> The fact that we flush on close and we flush periodically should be more of a
> concern of avoiding data loss; close in case sync is never called and the
> periodic flush to guard against having data from slow writers get buffered
> for a long time and expose us to risk of loss in case the collector crashes
> with data in its buffers. Size-based flush is a different concern to avoid
> blowing up memory footprint.
> The spooling behavior is also somewhat separate.
> We have two separate methods on our API putEntities and putEntitiesAsync and
> they should have different behavior beyond waiting for the request to be sent.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
