Varun Saxena commented on YARN-4238:

Yes its there as part of the event inside entity. But its not being set 
explicitly in entity.setCreatedTime(). You are correct this was not being done 
in ATSv1 either. 
In ATSv1 though if start time was not sent for an entity, Leveldb 
implementation of timeline store will check the entity(if it does not exist) 
and parse through all the events. And the smallest one will be chosen as start 
time. And if that was not there, I think current system time was taken(will 
have to check on that one).

Regardless, in ATSv2 we are neither setting created time in the client, nor do 
we have logic like ATSv1 in the HBase writer. So the end result is that created 
time is never updated in the backend.
Either ways this has to be handled. Either from publishing side or the writer.
I am not sure why the approach of fetching it from events was taken in ATSv1. 
As get and then put on every call can be expensive call from a HBase 
perspective, I think client can send it whenever it wants to. I do not see any 
issues around sending it from RM, NM,etc. from where entities are published. 
Otherwise we will have to check for specific events. I will check if there are 
any issues centering around sending it from client when I fix this. 

Now coming to what if client does not send it. This would be an issue if 
entities have to be returned sorted by created time or filtering on the basis 
of created time range has to be done. This can be explicitly stated for clients 
that if you do not report created time then we cannot guarantee order while 
fetching multiple entities. This will make it simple from an implementation 
viewpoint. If not, maybe we can cache it and check if entity has gone in to the 
backed or not and based on that, set created time.
But in this case, issue is what if daemon(having the writer) goes down. Maybe 
we can store this info in a state store. But do we need to do that ?

> createdTime is not reported while publishing entities to ATSv2
> --------------------------------------------------------------
>                 Key: YARN-4238
>                 URL: https://issues.apache.org/jira/browse/YARN-4238
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Varun Saxena
>            Assignee: Varun Saxena
> While publishing entities from RM and elsewhere we are not sending created 
> time. For instance, created time in TimelineServiceV2Publisher class and for 
> other entities in other such similar classes is not updated. We can easily 
> update created time when sending application created event. Likewise for 
> modification time on every write.

This message was sent by Atlassian JIRA

Reply via email to