[
https://issues.apache.org/jira/browse/YARN-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14654341#comment-14654341
]
Sangjin Lee commented on YARN-3049:
-----------------------------------
{quote}
I'm trying to understand the discussion here. Yes, and what we worked quite
hard to avoid is to identify the types of the incoming entities, in a writer,
so that we can apply different write code paths. If this is the case, maybe we
can refactor the write method so that it contains an expandable context object?
We can easily encapsulate flags in a BitSet-like object, and we may add more if
needed. The only problem I'm wondering about is, is it possible for the caller
to easily generate a context with all required information (such as isNewApp or
appFinish)?
BTW, I believe we need to refactor the interface of the read and write methods
to use some sorts of contexts anyways. Our current argument lists are not
expandable. So if this helps, maybe we can move forward by refactor the write
interfaces?
{quote}
Another place where {{HBaseTimelineWriterImpl}} would check for the entity type
(being the application) is splitting the application table (YARN-3906). The
current patch checks the type of the entity to be able to send writes to
different tables. So that would need to be included in the discussion as well.
I completely understand the desire that we want to make writers as much
agnostic about entity types and data as possible. However, since a lot of
things in the schema need to be based on the applications (flow context, the
application table, flow run aggregation, etc.), the need to support that
strongly is real. We can either go the route of having the write recognize
applications and some of their events strongly (at the expense of making the
separation between entities and writers a little weaker), or try to create a
context for this decision (as [~gtCarrera9] suggested) and have the writer act
on it.
As for the latter option, while it still shields the writer from knowing
details about entities, it would still need to know similar attributes (e.g.
"application created", "whether the entity is an application", etc.), only in a
more passive manner.
Thoughts?
> [Storage Implementation] Implement storage reader interface to fetch raw data
> from HBase backend
> ------------------------------------------------------------------------------------------------
>
> Key: YARN-3049
> URL: https://issues.apache.org/jira/browse/YARN-3049
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
> Reporter: Sangjin Lee
> Assignee: Zhijie Shen
> Attachments: YARN-3049-WIP.1.patch, YARN-3049-WIP.2.patch,
> YARN-3049-WIP.3.patch, YARN-3049-YARN-2928.2.patch,
> YARN-3049-YARN-2928.3.patch, YARN-3049-YARN-2928.4.patch,
> YARN-3049-YARN-2928.5.patch
>
>
> Implement existing ATS queries with the new ATS reader design.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
