[
https://issues.apache.org/jira/browse/YARN-3031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14336054#comment-14336054
]
Vrushali C commented on YARN-3031:
----------------------------------
Hi [~zjshen],
Thanks for the prompt review, appreciate it! These are very good points you
mention, let me add some more context around why these are coded this way right
now.
1. The reasoning behind having two more apis for writing metrics and events in
addition to the entity write is that, it would be good (efficient) to have the
option to write a single metric or a single event. For example, say a job has
many custom metrics and one particular metric is updated extremely frequently
but not the others. We may want to write out only that particular metric
without having to look through/write all other metrics and other information in
that entity. Similarly for events. Perhaps we could do it differently that what
is proposed in the patch, but the functionality of writing them individually
would help in performance I believe.
2. Having a separate write and aggregator API makes them independent of the
order in which the entity details and aggregation are invoked/stored and makes
them independent of each other. For instance, we may choose to invoke the
aggregation at a different frequency (more slower) than the regular entity
writes. Hence two apis.
3. The TimelineServiceWriteResponse has two error codes presently:
NO_START_TIME and IO_EXCEPTION. We can of course add in more error codes as we
proceed. The reason I chose these two for now is that each flow is inherently
associated with a submit timestamp (run id of that flow). In case, we don’t
find that timestamp, it would be difficult to write the flow information for
that run to the store - I think an error should be thrown with an error code.
The other one, IO_EXCEPTION is what I thought would help accounting for
write/put errors to the store- we should be able to indicate that the write did
not go through. We can rename these if these names don’t sound meaningful.
thanks
Vrushali
> [Storage abstraction] Create backing storage write interface for ATS writers
> ----------------------------------------------------------------------------
>
> Key: YARN-3031
> URL: https://issues.apache.org/jira/browse/YARN-3031
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
> Reporter: Sangjin Lee
> Assignee: Vrushali C
> Attachments: Sequence_diagram_write_interaction.2.png,
> Sequence_diagram_write_interaction.png, YARN-3031.01.patch, YARN-3031.02.patch
>
>
> Per design in YARN-2928, come up with the interface for the ATS writer to
> write to various backing storages. The interface should be created to capture
> the right level of abstractions so that it will enable all backing storage
> implementations to implement it efficiently.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
