Thanks Jerry. “Currently there's no Livy api for you to get application log via REST API.” - That’s a real shame. Are there any plans to address this? Because this really limits the usefulness of Livy when I’m using Yarn cluster mode. It seems I have to make a choice between running my client in a scalable manner (yarn cluster) or being able to see the logs programmatically (yarn client), but not both. My client is multi-threaded; I don’t want it to have to host multiple concurrent Spark driver applications but it looks like I might have no choice about this.
Thanks, Lucas. From: Saisai Shao [mailto:sai.sai.s...@gmail.com] Sent: 27 November 2017 02:19 To: user@livy.incubator.apache.org Subject: EXT: Re: How to access Python script's stdout when running in Yarn cluster mode? Since you're running with yarn cluster mode, the output from your python script should be part of your yarn application log. you can get it via yarn command like yarn log -applicationId <applictionId>, or others like Yarn UI. Currently there's no Livy api for you to get application log via REST API. Thanks Jerry 2017-11-24 20:27 GMT+08:00 Partridge, Lucas (GE Aviation) <lucas.partri...@ge.com<mailto:lucas.partri...@ge.com>>: Hi, I’m using Livy’s GET /batches/{batchId}/log method to fetch the log lines from a Python script I’m running on Spark in Yarn cluster mode. Unfortunately the stdout from my Python script is not included in the log lines returned by GET /batches/{batchId}/log! Is this by design, or an unfortunate by-product of running in Yarn cluster mode? If this is intentional does anyone know how I can access the stdout from my Python script via Livy please? Preferably without having to change my REST client (a Java app) to use Yarn’s client deployment mode. Thanks, Lucas.
