Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

are throwing some imaginations without any values Please go with direct discussion On Wed, Aug 16, 2017 at 9:11 AM, Marcelo Vanzin <van...@cloudera.com> wrote: > On Wed, Aug 16, 2017 at 9:06 AM, Nan Zhu <zhunanmcg...@gmail.com> wrote: > >> I'm not really sure what you

resolve the scalability problem caused by app monitoring in livy with an actor-based design

Hi, all In HDInsight, we (Microsoft) use Livy as the Spark job submission service. We keep seeing the customers fall into the problem when they submit many concurrent applications to the system, or recover livy from a state with many concurrent applications By looking at the code and the

Re: red flags in IntelliJ when importing from pom.xml

; plugin. Currently I don't have a good solution to handle it. > > On Mon, Aug 7, 2017 at 8:05 AM, Nan Zhu <zhunanmcg...@gmail.com> wrote: > > > Hi, all > > > > I just start looking at the source code of the project, it looks like > when > > importing src fro

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

at 12:36 PM, Marcelo Vanzin <van...@cloudera.com> wrote: > On Wed, Aug 16, 2017 at 12:27 PM, Nan Zhu <zhunanmcg...@gmail.com> wrote: > > I am using your words *current*. What's the definition of "current" in > > livy? I think that's all application which stil

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

yes, it is going to be Akka if moving forward (at least not going to introduce an actor framework to livy) On Wed, Aug 16, 2017 at 11:24 AM, Meisam Fathi wrote: > That is true, but I was under the impression that this will be implemented > with Akka (maybe because it is

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

com> wrote: > On Wed, Aug 16, 2017 at 11:17 AM, Nan Zhu <zhunanmcg...@gmail.com> wrote: > > Looks like non-REST API also contains this https://hadoop.apache. > > org/docs/r2.7.0/api/src-html/org/apache/hadoop/yarn/client/ > > api/YarnClient.html#line.225 > > > &

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

based on this result, I think we should follow the bulk operation pattern Shall we move forward with the PR from Paypal? Best, Nan On Mon, Aug 21, 2017 at 12:21 PM, Meisam Fathi wrote: > Bottom line up front: > 1. The cost of calling 1 individual REST calls is