Hi, Marcelo, Yes, essentially it is using multiple threads talking with YARN.
The key design consideration here is that how you model the state of applications, if in actor, then there will be no synchronization involved and yielding a cleaner design; if in a shared data structure, you will have to be careful about coordinating threads here (we actually have a design based on shared data structure and we eventually discard to pursue a cleaner one). I think bulk API can make life easier comparing to the shared data structure, but it raises up two questions 1. Are we going to update all applications in the uniform pace, even they are submitted in different time? 2. Are we going to use a single thread for everything, including send/recv req/res and parse, etc. and we still need to deal with some synchronization, What do you think? Best, Nan On Tue, Aug 15, 2017 at 11:53 AM, Marcelo Vanzin <van...@cloudera.com> wrote: > Hmm, I remember this... it was left as a "todo" item when the app > monitoring was added. > > The document you wrote seems to be a long way of saying you'll have a > few threads talking to YARN and updating the state of application > handles in Livy. Is that right? > > I would investigate whether there's any API in YARN to do a bulk get > of running applications with a particular filter; then you could make > a single call to YARN periodically to get the state of all apps that > Livy started. > > > On Mon, Aug 14, 2017 at 2:35 PM, Nan Zhu <zhunanmcg...@gmail.com> wrote: > > Hi, all > > > > In HDInsight, we (Microsoft) use Livy as the Spark job submission > service. > > We keep seeing the customers fall into the problem when they submit many > > concurrent applications to the system, or recover livy from a state with > > many concurrent applications > > > > By looking at the code and the customers' exception stack, we lock down > the > > problem to the application monitoring module where a new thread is > created > > for each application. > > > > To resolve the issue, we propose a actor-based design of application > > monitoring module and share it here (as new JIRA seems not working > > yet) *https://docs.google.com/document/d/1yDl5_3wPuzyGyFmSOzxRp6P- > nbTQTdDFXl2XQhXDiwA/edit?usp=sharing > > <https://docs.google.com/document/d/1yDl5_3wPuzyGyFmSOzxRp6P- > nbTQTdDFXl2XQhXDiwA/edit?usp=sharing>* > > > > We are glad to hear feedbacks from the community and improve the design > > before we start implementing it! > > > > Best, > > > > Nan > > > > -- > Marcelo >
