yes, we finally converge on the idea how large the reply can be? if I have only one running applications and I still need to fetch 1000
on the other side I have 1000 running apps, what's the cost of sending 1000 requests even the thread pool and yarn client are shared? On Wed, Aug 16, 2017 at 12:36 PM, Marcelo Vanzin <[email protected]> wrote: > On Wed, Aug 16, 2017 at 12:27 PM, Nan Zhu <[email protected]> wrote: > > I am using your words *current*. What's the definition of "current" in > > livy? I think that's all application which still keep some records in the > > livy's process's memory space > > There are two views of what is current: Livy's and YARN's. They may > not be the same. > > From your reply below, you seem to want to query YARN for the state of > applications that are current to Livy. There's no API for that, as you > said. But that is not what I'm talking about. > > I'm saying that Livy should query YARN for YARN's current view of what > applications exist, and then match those against its own view. > > Again, it's all a question about what is cheaper: a single request to > YARN that results in a large reply, parts of which Livy will ignore > because it's not interested in the data, or hundreds of small requests > to YARN polling specific applications? > > > 1. How you express this "current" in a query to YARN? I think you have to > > use ApplicationID (maybe there are some other ways) in a query > > > > 2. The problem is that I didn't see such an API to make such a "big call" > > by passing in all applications's IDs > > > -- > Marcelo >
