[jira] [Commented] (YARN-3041) [Data Model] create the ATS entity/event API
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14325042#comment-14325042 ] Zhijie Shen commented on YARN-3041: --- bq. How stringent do we want to be on allowed parent-child relationships? It should be take care of consistently. Thanks for the remind! bq. I think we should rename Sounds good. bq. I believe we discussed this somewhere before and decided that not all Entities need a configuration property Oh, I thought Sangjin suggested them still being in the generic entity. But it's not a big issue. We can adjust it later. At this moment, let's try to get a working version in. bq. info should be named metadata bq. whats the purpose of info ? can we name it as metadata ? I think metadata and info mean the identical stuff. I switch to use info to be as close to the old data model as possible. bq. After having HierarchicalTimelineEntity do we require isRelatedToEntities relatesToEntities in TimelineEntity or vice versa? This is required to allow a generic entity to be associated to another, and not just limited to parent-child limitation. bq. If any Entity data cannot be updated on subsequent posts of time line Entities better to capture it before hand. It's a good question. For an entity, we allow users to keep appending data or updating the fields, but we cannot know it upfront, when composing the object. Server should be responsible for rejecting the request with the object, whose change on some field is invalid. bq. IIUC if time series metric then singleData will be null and timeSeries will have values and if its non time series then vice versa. It's good to behave like a C++ union type, but I don't want to have stringent check for this. For example, sometimes the user just wants to create the metric, but the data will come later. At one moment, it's possible that neither single data nor time series is available. bq. flow version is not captured as class member of FlowEntity Good catch! I added for now. But if it's not a critical attribute. We can put it in info section actually. bq. Is this valid ? i was under the assumption that only FlowRun and cluster will be having ApplicationEntity as child I thought FlowRun is only necessary if there's multiple levels of flows. No? I'm about to post a new patch. It makes changes according jackson ser/der requirement and adds a simple demo test. [Data Model] create the ATS entity/event API Key: YARN-3041 URL: https://issues.apache.org/jira/browse/YARN-3041 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Zhijie Shen Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, YARN-3041.preliminary.001.patch Per design in YARN-2928, create the ATS entity and events API. Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3041) [Data Model] create the ATS entity/event API
[
https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14323157#comment-14323157
]
Naganarasimha G R commented on YARN-3041:
-
# After having HierarchicalTimelineEntity do we require isRelatedToEntities
relatesToEntities in TimelineEntity or vice versa?
{quote}
private SetTimelineEntity isRelatedToEntities = new HashSet();
private SetTimelineEntity relatesToEntities = new HashSet();
{quote}
# If any Entity data cannot be updated on subsequent posts of time line
Entities better to capture it before hand. for example
if we are inserting configs of timeline entity only during creation of new
TimelineEntity...
# Regarding metrics ; TimelineEntity has set of TimelineMetric and
TimelineMetric has
{quote}
private String id;
private MapString, Object info = new HashMap();
private Object singleData;
private MapLong, Object timeSeries = new LinkedHashMap();
{quote}
#* whats the purpose of info ? can we name it as metadata ?
#* Are all objects stored in backend by serialization and deserialization or as
json strings ?
#* If metric value is object then how aggregations (primary) are done ? is info
responsible for capturing this information?
#* IIUC if time series metric then {{singleData}} will be null and
{{timeSeries}} will have values and if its non time series then vice versa.
[Data Model] create the ATS entity/event API
Key: YARN-3041
URL: https://issues.apache.org/jira/browse/YARN-3041
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Sangjin Lee
Assignee: Zhijie Shen
Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch,
YARN-3041.preliminary.001.patch
Per design in YARN-2928, create the ATS entity and events API.
Also, as part of this JIRA, create YARN system entities (e.g. cluster, user,
flow, flow run, YARN app, ...).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3041) [Data Model] create the ATS entity/event API
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14323706#comment-14323706 ] Naganarasimha G R commented on YARN-3041: - Few other minor comments : # flow version is not captured as class member of FlowEntity # For FlowEntity bq. ACCEPTABLE_ENTITY_TYPES.add(ApplicationEntity.TYPE); Is this valid ? i was under the assumption that only FlowRun and cluster will be having ApplicationEntity as child [Data Model] create the ATS entity/event API Key: YARN-3041 URL: https://issues.apache.org/jira/browse/YARN-3041 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Zhijie Shen Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, YARN-3041.preliminary.001.patch Per design in YARN-2928, create the ATS entity and events API. Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3041) [Data Model] create the ATS entity/event API
[
https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14321124#comment-14321124
]
Robert Kanter commented on YARN-3041:
-
Looks good overall. Thanks for picking this up from me; I kept not having time
to update it.
A few things:
- How stringent do we want to be on allowed parent-child relationships?
-- There's nothing stopping me from calling {{addChild}} on multiple parents
for some {{TimelineEntity}}
-- There's nothing stopping me from using {{setChildren}} to bypass the
acceptable entities check
-- Should {{addChild}} also take care of calling {{child.setParent(this)}}?
- I think we should rename {{getAcceptableEntityTypes()}} and
{{ACCEPTABLE_ENTITY_TYPES}} to {{getAcceptableChildEntityTypes}} and
{{ACCEPTABLE_CHILD_ENTITY_TYPES}}. I wasn't sure what these were until I saw
how they were used in the {{HierarchicalTimelineEntity}} class
- I believe we discussed this somewhere before and decided that not all
Entities need a configuration property
- {{info}} should be named {{metadata}}
[Data Model] create the ATS entity/event API
Key: YARN-3041
URL: https://issues.apache.org/jira/browse/YARN-3041
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Sangjin Lee
Assignee: Zhijie Shen
Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch,
YARN-3041.preliminary.001.patch
Per design in YARN-2928, create the ATS entity and events API.
Also, as part of this JIRA, create YARN system entities (e.g. cluster, user,
flow, flow run, YARN app, ...).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3041) [Data Model] create the ATS entity/event API
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319230#comment-14319230 ] Zhijie Shen commented on YARN-3041: --- Take it over, Thanks! - Zhijie [Data Model] create the ATS entity/event API Key: YARN-3041 URL: https://issues.apache.org/jira/browse/YARN-3041 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Zhijie Shen Attachments: YARN-3041.preliminary.001.patch Per design in YARN-2928, create the ATS entity and events API. Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
