On Wed, Aug 16, 2017 at 9:06 AM, Nan Zhu wrote:
>> I'm not really sure what you're talking about here, since I did not
> suggest a "shared data structure", and I'm not really sure what that
> means in this context.
>
> What you claimed is just monitoring/updating the state
> What I proposed is having a single request to YARN to get all applications'
statuses, if that's possible. You'd still have multiple application handles
that are independent of each other. They'd all be updated separately from
that one thread talking to YARN. This has nothing to do with a "shared
I like that approach on paper, although I currently don't have much
time to actually be able to review the PR and provide decent feedback.
I think that regardless of the approach, one goal should be to
probably separate what is being monitored from how it's being
monitored; that way you can later
Hi Nan,
In the highlighted line
>
> https://github.com/apache/incubator-livy/pull/36/files#diff-a3f879755cfe10a678cc08ddbe60a4d3R75
>
> I assume that it will get the reports of all applications in YARN, even
> they are finished?
That's right. That line will return reports for all Spark
That is true, but I was under the impression that this will be implemented
with Akka (maybe because it is mentioned in the design doc).
On Wed, Aug 16, 2017 at 11:21 AM Marcelo Vanzin wrote:
> On Wed, Aug 16, 2017 at 11:16 AM, Meisam Fathi
> wrote:
yes, we finally converge on the idea
how large the reply can be? if I have only one running applications and I
still need to fetch 1000
on the other side
I have 1000 running apps, what's the cost of sending 1000 requests even the
thread pool and yarn client are shared?
On Wed, Aug 16, 2017
On Wed, Aug 16, 2017 at 12:27 PM, Nan Zhu wrote:
> I am using your words *current*. What's the definition of "current" in
> livy? I think that's all application which still keep some records in the
> livy's process's memory space
There are two views of what is current:
yes, it is going to be Akka if moving forward (at least not going to
introduce an actor framework to livy)
On Wed, Aug 16, 2017 at 11:24 AM, Meisam Fathi
wrote:
> That is true, but I was under the impression that this will be implemented
> with Akka (maybe because it is
Yes, I know there is such an API, what I don't understand is what I should
pass in the filtering API you mentioned, say we query YARN for every 5
tickets
0: Query and get App A is running
4: App A is done
5: Query...so what I should fill as filtering parameters at 5 get capture
the changes of
On Wed, Aug 16, 2017 at 11:34 AM, Nan Zhu wrote:
> Yes, I know there is such an API, what I don't understand is what I should
> pass in the filtering API you mentioned, say we query YARN for every 5
> tickets
>
> 0: Query and get App A is running
>
> 4: App A is done
>
>
On Wed, Aug 16, 2017 at 11:16 AM, Meisam Fathi wrote:
> I do agree that actor based design is cleaner and more maintainable. But we
> had to discard it because it adds more dependencies to Livy.
I've been reading "actor system" as a design pattern, not as
introducing a
On Wed, Aug 16, 2017 at 12:57 PM, Nan Zhu wrote:
> yes, we finally converge on the idea
>
> how large the reply can be? if I have only one running applications and I
> still need to fetch 1000
>
> on the other side
>
> I have 1000 running apps, what's the cost of sending
As Meisam highlighted, in our case, we have Livy Multi-Node HA i.e livy
running on 6 servers for each cluster, load-balanced, sharing livy metadata
on zookeeper and running thousands of applications. With below changes, we
are seeing good improvements due to batching the requests (one per livy
13 matches
Mail list logo