[jira] [Commented] (YARN-4238) createdTime is not reported while publishing entities to ATSv2
[ https://issues.apache.org/jira/browse/YARN-4238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973257#comment-14973257 ] Varun Saxena commented on YARN-4238: Just noticed that created time was retrospectively added in YARN-4129 so just added code for modified time in RM. And updated test case in RM. Assumption is that container entity will either be updated from RM or NM. So updating created time from both places. Moreover added code in JobHistoryEventHandler as well. > createdTime is not reported while publishing entities to ATSv2 > -- > > Key: YARN-4238 > URL: https://issues.apache.org/jira/browse/YARN-4238 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-4238-YARN-2928.01.patch > > > While publishing entities from RM and elsewhere we are not sending created > time. For instance, created time in TimelineServiceV2Publisher class and for > other entities in other such similar classes is not updated. We can easily > update created time when sending application created event. Likewise for > modification time on every write. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4238) createdTime is not reported while publishing entities to ATSv2
[ https://issues.apache.org/jira/browse/YARN-4238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949959#comment-14949959 ] Varun Saxena commented on YARN-4238: [~sjlee0], [~djp], thoughts on this ? > createdTime is not reported while publishing entities to ATSv2 > -- > > Key: YARN-4238 > URL: https://issues.apache.org/jira/browse/YARN-4238 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > > While publishing entities from RM and elsewhere we are not sending created > time. For instance, created time in TimelineServiceV2Publisher class and for > other entities in other such similar classes is not updated. We can easily > update created time when sending application created event. Likewise for > modification time on every write. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4238) createdTime is not reported while publishing entities to ATSv2
[ https://issues.apache.org/jira/browse/YARN-4238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949064#comment-14949064 ] Varun Saxena commented on YARN-4238: Yes its there as part of the event inside entity. But its not being set explicitly in entity.setCreatedTime(). You are correct this was not being done in ATSv1 either. In ATSv1 though if start time was not sent for an entity, Leveldb implementation of timeline store will check the entity(if it does not exist) and parse through all the events. And the smallest one will be chosen as start time. And if that was not there, I think current system time was taken(will have to check on that one). Regardless, in ATSv2 we are neither setting created time in the client, nor do we have logic like ATSv1 in the HBase writer. So the end result is that created time is never updated in the backend. Either ways this has to be handled. Either from publishing side or the writer. I am not sure why the approach of fetching it from events was taken in ATSv1. As get and then put on every call can be expensive call from a HBase perspective, I think client can send it whenever it wants to. I do not see any issues around sending it from RM, NM,etc. from where entities are published. Otherwise we will have to check for specific events. I will check if there are any issues centering around sending it from client when I fix this. Now coming to what if client does not send it. This would be an issue if entities have to be returned sorted by created time or filtering on the basis of created time range has to be done. This can be explicitly stated for clients that if you do not report created time then we cannot guarantee order while fetching multiple entities. This will make it simple from an implementation viewpoint. If not, maybe we can cache it and check if entity has gone in to the backed or not and based on that, set created time. But in this case, issue is what if daemon(having the writer) goes down. Maybe we can store this info in a state store. But do we need to do that ? > createdTime is not reported while publishing entities to ATSv2 > -- > > Key: YARN-4238 > URL: https://issues.apache.org/jira/browse/YARN-4238 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > > While publishing entities from RM and elsewhere we are not sending created > time. For instance, created time in TimelineServiceV2Publisher class and for > other entities in other such similar classes is not updated. We can easily > update created time when sending application created event. Likewise for > modification time on every write. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4238) createdTime is not reported while publishing entities to ATSv2
[ https://issues.apache.org/jira/browse/YARN-4238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14948041#comment-14948041 ] Naganarasimha G R commented on YARN-4238: - Hi [~varun_saxena], Similar to V1 we have already captured these information in TimelineServiceV2Publisher too, Its been captured as part of events of those entities If you want it to be as part of Entity i can add it there but IMO it makes more sense to be as part of event, as different states of application( basically any entity) are captured as events. > createdTime is not reported while publishing entities to ATSv2 > -- > > Key: YARN-4238 > URL: https://issues.apache.org/jira/browse/YARN-4238 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > > While publishing entities from RM and elsewhere we are not sending created > time. For instance, created time in TimelineServiceV2Publisher class and for > other entities in other such similar classes is not updated. We can easily > update created time when sending application created event. Likewise for > modification time on every write. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
