Li Lu updated YARN-4265:
    Attachment: YARN-4265-trunk.002.patch

Thanks [~djp] for the review! In the 002 patch I addressed most of the 
checkstyle problems, as well as most existing comments. Please feel free to add 
more. Some comments:

bq. I noticed that we are setting 1 minutes as default scan interval but 
original patch in HDFS-3942 is 5 minutes. Why shall we do any update here? 
For now I increased the default frequency to scan HDFS and pull timeline data. 
Having a 5-minute time interval means users are less likely to see any running 
status for apps that finish within 5 minutes. Right now I'm setting this value 
to 1 minute to reduce reader react time. 

bq. The same question on "app-cache-size", the default value in HDFS-3942 is 5 
but here is 10. Any reason to update the value?
In YARN-3942, caching is performed on application level. In this patch, caching 
is performed in entity groups. Each application may have a few to tens of 
entity groups. Normally, there are slightly more active entity groups than 
active applications in the system. For now, I'm increasing this default value 
to hold slightly more entity groups in cache. 

bq. Why we don't have any default value specified in property of 
Plugins are provided by third-party applications such as Tez. Right now we 
cannot assume which exact entity group plugin the user is using, therefore we 
have to conservatively leave this config as empty. 

bq. For EmptyTimelineEntityGroupPlugin.java, why we need this class? I didn't 
see any help provided even in tests. We should remove it if useless.
Ah, nice catch. Removed it. 

bq. Can we optimize the synchronization logic here? Like in synchronized method 
refreshCache, we are intialize/start/stop TimelineDataManager (and 
MemoryTimelineStore) which is quite expensive and unnecessary to block other 
synchronized operations. Shall we move these operations out of synchronized 
It's certainly doable. Right now I have yet to optimize this part because it's 
a little bit tricky to fine tune synchronization performance before we have a 
relatively stable starting point. Also, we're using fine-grained locking for 
each cached item in the reader cache, and cache refresh only happens 
infrequently (~10 secs by default), so maybe we'd like to stabilize the whole 
synchronization story before fine tune this part? 

> Provide new timeline plugin storage to support fine-grained entity caching
> --------------------------------------------------------------------------
>                 Key: YARN-4265
>                 URL: https://issues.apache.org/jira/browse/YARN-4265
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Li Lu
>            Assignee: Li Lu
>         Attachments: YARN-4265-trunk.001.patch, YARN-4265-trunk.002.patch, 
> YARN-4265.YARN-4234.001.patch, YARN-4265.YARN-4234.002.patch
> To support the newly proposed APIs in YARN-4234, we need to create a new 
> plugin timeline store. The store may have similar behavior as the 
> EntityFileTimelineStore proposed in YARN-3942, but cache date in cache id 
> granularity, instead of application id granularity. Let's have this storage 
> as a standalone one, instead of updating EntityFileTimelineStore, to keep the 
> existing store (EntityFileTimelineStore) stable. 

This message was sent by Atlassian JIRA

Reply via email to