Li Lu resolved YARN-3595.
    Resolution: Later

Since we're moving phoenix storage to aggregation side, we need to fix this 
after we have a full story of the offline (time-based) aggregation. 

> Performance optimization using connection cache of Phoenix timeline writer
> --------------------------------------------------------------------------
>                 Key: YARN-3595
>                 URL: https://issues.apache.org/jira/browse/YARN-3595
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Li Lu
>            Assignee: Li Lu
> The story about the connection cache in Phoenix timeline storage is a little 
> bit long. In YARN-3033 we planned to have shared writer layer for all 
> collectors in the same collector manager. In this way we can better reuse the 
> same heavy-weight storage layer connection, therefore it's more friendly to 
> conventional storage layer connections which are typically heavy-weight. 
> Phoenix, on the other hand, implements its own connection interface layer to 
> be light-weight, thread-unsafe. To make these connections work with our 
> "multiple collector, single writer" model, we're adding a thread indexed 
> connection cache. However, many performance critical factors are yet to be 
> tested. 
> In this JIRA we're tracing performance optimization efforts using this 
> connection cache. Previously we had a draft, but there was one implementation 
> challenge on cache evictions: There may be races between Guava cache's 
> removal listener calls (which close the connection) and normal references to 
> the connection. We need to carefully define the way they synchronize. 
> Performance-wise, at the very beginning stage we may need to understand:
> # If the current, thread-based indexing is an appropriate approach, or we can 
> use some better ways to index the connections. 
> # the best size of the cache, presumably as the proposed default value of a 
> configuration. 
> # how long we need to preserve a connection in the cache. 
> Please feel free to add this list. 

This message was sent by Atlassian JIRA

Reply via email to