Re: How to access Python script's stdout when running in Yarn cluster mode?

[Saisai Shao]

I think you have plenty of ways to get application log, either via command
line or programmatically, even with yarn RM UI. Since there already has
several ways to get application log, so currently we don't have a plan to
address this.

2017-11-27 20:04 GMT+08:00 Partridge, Lucas (GE Aviation) <
lucas.partri...@ge.com>:

> Thanks Jerry.
>
>
>
> “Currently there's no Livy api for you to get application log via REST
> API.”
>
> - That’s a real shame. Are there any plans to address this?  Because this
> really limits the usefulness of Livy when I’m using Yarn cluster mode.  It
> seems I have to make a choice between running my client in a scalable
> manner (yarn cluster) or being able to see the logs programmatically (yarn
> client), but not both.  My client is multi-threaded; I don’t want it to
> have to host multiple concurrent Spark driver applications but it looks
> like I might have no choice about this.
>
>
>
> Thanks, Lucas.
>
>
>
> *From:* Saisai Shao [mailto:sai.sai.s...@gmail.com]
> *Sent:* 27 November 2017 02:19
> *To:* user@livy.incubator.apache.org
> *Subject:* EXT: Re: How to access Python script's stdout when running in
> Yarn cluster mode?
>
>
>
> Since you're running with yarn cluster mode, the output from your python
> script should be part of your yarn application log. you can get it via yarn
> command like yarn log -applicationId , or others like Yarn
> UI. Currently there's no Livy api for you to get application log via REST
> API.
>
>
>
> Thanks
>
> Jerry
>
>
>
> 2017-11-24 20:27 GMT+08:00 Partridge, Lucas (GE Aviation) <
> lucas.partri...@ge.com>:
>
> Hi,
>
>
>
> I’m using Livy’s GET /batches/{batchId}/log method to fetch the log lines
> from a Python script I’m running on Spark in Yarn cluster mode.
> Unfortunately the stdout from my Python script is not included in the log
> lines returned by GET /batches/{batchId}/log!
>
>
>
> Is this by design, or an unfortunate by-product of running in Yarn cluster
> mode?
>
>
>
> If this is intentional does anyone know how I can access the stdout from
> my Python script via Livy please? Preferably without having to change my
> REST client (a Java app) to use Yarn’s client deployment mode.
>
>
>
> Thanks,
>
> Lucas.
>
>
>
>
>

How to access Python script's stdout when running in Yarn cluster mode?

[Partridge, Lucas (GE Aviation)]

Thanks Jerry.

“Currently there's no Livy api for you to get application log via REST API.”
- That’s a real shame. Are there any plans to address this?  Because this 
really limits the usefulness of Livy when I’m using Yarn cluster mode.  It 
seems I have to make a choice between running my client in a scalable manner 
(yarn cluster) or being able to see the logs programmatically (yarn client), 
but not both.  My client is multi-threaded; I don’t want it to have to host 
multiple concurrent Spark driver applications but it looks like I might have no 
choice about this.

Thanks, Lucas.

From: Saisai Shao [mailto:sai.sai.s...@gmail.com]
Sent: 27 November 2017 02:19
To: user@livy.incubator.apache.org
Subject: EXT: Re: How to access Python script's stdout when running in Yarn 
cluster mode?

Since you're running with yarn cluster mode, the output from your python script 
should be part of your yarn application log. you can get it via yarn command 
like yarn log -applicationId , or others like Yarn UI. Currently 
there's no Livy api for you to get application log via REST API.

Thanks
Jerry

2017-11-24 20:27 GMT+08:00 Partridge, Lucas (GE Aviation) 
>:
Hi,

I’m using Livy’s GET /batches/{batchId}/log method to fetch the log lines from 
a Python script I’m running on Spark in Yarn cluster mode. Unfortunately the 
stdout from my Python script is not included in the log lines returned by GET 
/batches/{batchId}/log!

Is this by design, or an unfortunate by-product of running in Yarn cluster mode?

If this is intentional does anyone know how I can access the stdout from my 
Python script via Livy please? Preferably without having to change my REST 
client (a Java app) to use Yarn’s client deployment mode.

Thanks,
Lucas.