Hello, On Wed, Aug 16, 2017 at 10:35 AM, Arijit Tarafdar <arij...@microsoft.com.invalid> wrote: > 1. Additional copy of states in Livy which can be queried from YARN on > request.
Not sure I follow. > 2. The design is not event driven and may waste querying YARN unnecessarily > when no actual user/external request is pending. You don't need to keep querying YARN if there are no apps to monitor. > 3. There will always be an issue with stale data and update latency between > actual YARN state and Livy state map. That is also the case with a thread pool that has less threads than the number of apps being monitored, if making one request per app. > 4. Size and latency of the response in bulk querying YARN is unknown. That is also unknown when making multiple requests from multiple threads, unless you investigate the internal implementation of both YARN clients and servers. > 5. YARN bulk API needs to support filtering at the query level. Yes, I mentioned that in my original response and really was just expecting Nan to do a quick investigation of that implementation option. He finally did and it seems that the API only exists through the REST interface, so this all is moot. -- Marcelo
