Then which API you would use for *current* Apps? I think you have to define *current* with applicationIds? If that's true, you have to call https://hadoop.apache.org/docs/r2.7.0/api/src-html/org/apache/hadoop/yarn/client/api/YarnClient.html#line.181 ,
If I didn't miss anything, there is no API to pass in a list of app ids, as a result, you have to fire N requests (N is the number of current apps) to YARN Then the solution becomes using a single thread to fire N request instead of using M threads to fire N requests (ideally M << N) On Wed, Aug 16, 2017 at 11:41 AM, Marcelo Vanzin <[email protected]> wrote: > On Wed, Aug 16, 2017 at 11:34 AM, Nan Zhu <[email protected]> wrote: > > Yes, I know there is such an API, what I don't understand is what I > should > > pass in the filtering API you mentioned, say we query YARN for every 5 > > tickets > > > > 0: Query and get App A is running > > > > 4: App A is done > > > > 5: Query...so what I should fill as filtering parameters at 5 get capture > > the changes of App A's state? > > You don't query for app state *changes*. You query for the current app > state, and compare against what you have, and then you can detect > changes that way. The trick is how to filter to get the information > you want, so you limit how much data you request from YARN. > > I'm not aware of any YARN API to query for state changes like that. So > even in the individual request case, you'd have to get app A's state, > and update the Livy handle if the state has changed from what was > previously know. > > That's most probably why Meisam's PR only filters by app type. If > there are further filters than can be applied, then great, but you > still need logic in Livy to detect the state changes you want. > > > If you look at Meisam's PR, they can only filter based on appType > > https://github.com/apache/incubator-livy/pull/36/files#diff- > a3f879755cfe10a678cc08ddbe60a4d3R75 > > > -- > Marcelo >
