The logs are available through the Spark UI (which is the Application
Master link in YARN UI) on the Executors tab (stderr and stdout)
AFAIK, the log search can only index local files but you could have a
cronjob downloading then directly from hdfs.
On 22 February 2018 at 21:27, Lian Jiang <jiangok2...@gmail.com> wrote:
> Thanks for any help. What's the best way to get spark application logs?
> yarn command line is not ideal for spark streaming applications.
> On Wed, Feb 21, 2018 at 10:41 AM, Lian Jiang <jiangok2...@gmail.com>
>> Can I make my spark application log available for ambari log search so
>> that I can search them on log search UI? Or I have to setup another
>> solr/elastic search cluster for my application logs? Thanks for any hints.