[
https://issues.apache.org/jira/browse/YARN-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15936763#comment-15936763
]
Varun Saxena commented on YARN-6357:
------------------------------------
bq. Also, I am wondering if there is any way for the TimelineWriteResponse to
indicate that a sync call was invoked or if an aync call was invoked.
If I am not wrong, async and sync call would lead to 2 separate puts so client
should be knowing already which one is what so is it necessary to indicate so
in response?
In TimelineV2ClientImpl we return directly for an async call and for sync, we
wait. Some place you think this will be useful?
bq. In the write APIs, I think it will be good to have an explicitly named
async API and a sync API (rather than persistXX). I think the API doc should
explain that the sync one means syncing to call HBase flush and that aysnc
means it will be translated to hbase#flush after a time interval or at the end
of the application life.
[~vrushalic], are you referring to TimelineWriter interface when you talk about
persistXXX APIs'? Actually the comments above are for code in TimelineCollector.
Anyways coming to APIs' in TimelineWriter which Haibo referred to in his
comment above, TimelineWriter is an interface so we cannot really talk about
HBase flush functionality in the javadoc here. Probably can be mentioned as an
example.
Also, we can split the APIs' or even pass a flag to write API and explain in
javadoc that you should call flush mandatorily after write for sync calls and
that should also flush buffered async calls if there is any buffering.
But if this is what we want writer (which is a pluggable interface) to do
mandatorily and essentially writer implementation would also need to call write
followed by flush for sync, can't we simply ensure the sequence from
TimelineCollector itself? Is there some scenario where you think an explicit
separate API will be useful?
Thoughts?
> Implement TimelineCollector#putEntitiesAsync
> --------------------------------------------
>
> Key: YARN-6357
> URL: https://issues.apache.org/jira/browse/YARN-6357
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2, timelineserver
> Affects Versions: YARN-2928
> Reporter: Joep Rottinghuis
> Assignee: Haibo Chen
> Labels: yarn-5355-merge-blocker
> Attachments: YARN-6357.01.patch, YARN-6357.02.patch
>
>
> As discovered and discussed in YARN-5269 the
> TimelineCollector#putEntitiesAsync method is currently not implemented and
> TimelineCollector#putEntities is asynchronous.
> TimelineV2ClientImpl#putEntities vs TimelineV2ClientImpl#putEntitiesAsync
> correctly call TimelineEntityDispatcher#dispatchEntities(boolean sync,...
> with the correct argument. This argument does seem to make it into the
> params, and on the server side TimelineCollectorWebService#putEntities
> correctly pulls the async parameter from the rest call. See line 156:
> {code}
> boolean isAsync = async != null && async.trim().equalsIgnoreCase("true");
> {code}
> However, this is where the problem starts. It simply calls
> TimelineCollector#putEntities and ignores the value of isAsync. It should
> instead have called TimelineCollector#putEntitiesAsync, which is currently
> not implemented.
> putEntities should call putEntitiesAsync and then after that call
> writer.flush()
> The fact that we flush on close and we flush periodically should be more of a
> concern of avoiding data loss; close in case sync is never called and the
> periodic flush to guard against having data from slow writers get buffered
> for a long time and expose us to risk of loss in case the collector crashes
> with data in its buffers. Size-based flush is a different concern to avoid
> blowing up memory footprint.
> The spooling behavior is also somewhat separate.
> We have two separate methods on our API putEntities and putEntitiesAsync and
> they should have different behavior beyond waiting for the request to be sent.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
