Sangjin Lee commented on YARN-3049:

I'm trying to understand the discussion here. Yes, and what we worked quite 
hard to avoid is to identify the types of the incoming entities, in a writer, 
so that we can apply different write code paths. If this is the case, maybe we 
can refactor the write method so that it contains an expandable context object? 
We can easily encapsulate flags in a BitSet-like object, and we may add more if 
needed. The only problem I'm wondering about is, is it possible for the caller 
to easily generate a context with all required information (such as isNewApp or 
BTW, I believe we need to refactor the interface of the read and write methods 
to use some sorts of contexts anyways. Our current argument lists are not 
expandable. So if this helps, maybe we can move forward by refactor the write 

Another place where {{HBaseTimelineWriterImpl}} would check for the entity type 
(being the application) is splitting the application table (YARN-3906). The 
current patch checks the type of the entity to be able to send writes to 
different tables. So that would need to be included in the discussion as well.

I completely understand the desire that we want to make writers as much 
agnostic about entity types and data as possible. However, since a lot of 
things in the schema need to be based on the applications (flow context, the 
application table, flow run aggregation, etc.), the need to support that 
strongly is real. We can either go the route of having the write recognize 
applications and some of their events strongly (at the expense of making the 
separation between entities and writers a little weaker), or try to create a 
context for this decision (as [~gtCarrera9] suggested) and have the writer act 
on it.

As for the latter option, while it still shields the writer from knowing 
details about entities, it would still need to know similar attributes (e.g. 
"application created", "whether the entity is an application", etc.), only in a 
more passive manner.


> [Storage Implementation] Implement storage reader interface to fetch raw data 
> from HBase backend
> ------------------------------------------------------------------------------------------------
>                 Key: YARN-3049
>                 URL: https://issues.apache.org/jira/browse/YARN-3049
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Sangjin Lee
>            Assignee: Zhijie Shen
>         Attachments: YARN-3049-WIP.1.patch, YARN-3049-WIP.2.patch, 
> YARN-3049-WIP.3.patch, YARN-3049-YARN-2928.2.patch, 
> YARN-3049-YARN-2928.3.patch, YARN-3049-YARN-2928.4.patch, 
> YARN-3049-YARN-2928.5.patch
> Implement existing ATS queries with the new ATS reader design.

This message was sent by Atlassian JIRA

Reply via email to