collinmcnulty opened a new issue #17340: URL: https://github.com/apache/airflow/issues/17340
**Description** The Airflow logs generated by the Livy operator currently only state the status of the submitted batch. To view the logs from the job itself, one must go separately to the session logs. I think that Airflow should have the option (possibly on by default) that retrieves the session logs after the batch reaches a terminal state if a `polling_interval` has been set. **Use case / motivation** When debugging a task submitted via Livy, the session logs are the first place to check. For most other tasks, including SparkSubmitOperator, viewing the first-check logs can be done in the Airflow UI, but for Livy you must go externally or write a separate task to retrieve them. **Are you willing to submit a PR?** I don't yet have a good sense of how challenging this will be to set up and test. I can try but if anyone else wants to go for it, don't let my attempt stop you. **Related Issues** None I could find -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
