Varun Saxena commented on YARN-3051:

bq. we have chosen clusterId + appId to globally find a unique flow run. I 
think here we should do it similar by adding clusterId
The current FS implementation had cluster as part of the path. So there will a 
app_flow_mapping.csv for each cluster. So in a way it is part of the primary 
key even though its not there in app_flow_mapping.csv
I hope that is what your concern was.

bq. 1. Maybe we want to cache the mapping instead of reading it from the file 
for every query.
Yes, we should be doing so. Plan to do these optimizations in later JIRA. Also 
some optimizations are required as in we are using set instead of map for 
storing metrics and events. So I have to iterate over all of them. Any issue in 
turning them into map ?

bq. 2. limit should be push down into the for loop. It's unnecessary that if we 
want to just retrieve.
The issue here is that we want to have limit on entities but these should be 
latest entities(sorted descendingly by created time). Having created time in 
entity file name will help towards not reading all the files.

bq.3. We'd better avoid hard code "/" as the path separator, and we should use 
FileSystem interface to operate the files, such that the impl can also work 
with HDFS.

> [Storage abstraction] Create backing storage read interface for ATS readers
> ---------------------------------------------------------------------------
>                 Key: YARN-3051
>                 URL: https://issues.apache.org/jira/browse/YARN-3051
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>    Affects Versions: YARN-2928
>            Reporter: Sangjin Lee
>            Assignee: Varun Saxena
>         Attachments: YARN-3051-YARN-2928.003.patch, 
> YARN-3051-YARN-2928.03.patch, YARN-3051-YARN-2928.04.patch, 
> YARN-3051-YARN-2928.05.patch, YARN-3051-YARN-2928.06.patch, 
> YARN-3051-YARN-2928.07.patch, YARN-3051-YARN-2928.08.patch, 
> YARN-3051.Reader_API.patch, YARN-3051.Reader_API_1.patch, 
> YARN-3051.Reader_API_2.patch, YARN-3051.Reader_API_3.patch, 
> YARN-3051.Reader_API_4.patch, YARN-3051.wip.02.YARN-2928.patch, 
> YARN-3051.wip.patch, YARN-3051_temp.patch
> Per design in YARN-2928, create backing storage read interface that can be 
> implemented by multiple backing storage implementations.

This message was sent by Atlassian JIRA

Reply via email to