[
https://issues.apache.org/jira/browse/LIVY-336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gyorgy Gal updated LIVY-336:
----------------------------
Fix Version/s: 0.10.0
(was: 0.9.0)
This issue has been moved to the 0.10.0 release as part of a bulk update. If
you feel this is moved out inappropriately, feel free to provide justification
and reset the Fix Version to 0.9.0.
> Livy should not spawn one thread per job to track the job on Yarn
> -----------------------------------------------------------------
>
> Key: LIVY-336
> URL: https://issues.apache.org/jira/browse/LIVY-336
> Project: Livy
> Issue Type: Improvement
> Components: Server
> Affects Versions: 0.4.0
> Environment: only on Yarn clusters
> Reporter: Meisam
> Priority: Minor
> Fix For: 0.10.0
>
> Attachments: size_downloaded_line_plot.png,
> transfer_time_bar_plot.png, transfer_time_line_plot.png
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> {{SparkYarnApp}} spawns a new thread for each application. The thread that
> does the following tasks:
> # It waits for the exit code of {{spark-submit}}
> # It polls yarn to get the {{ApplicaionId}} from the {{ApplicationTag}}
> # It periodically polls Yarn and updates {{SessionManager}} with updates to
> the state of the application on Yarn.
> Yarn does not provide an API to do 2, so Livy gets the list of all Spark
> applications running on yarn and traverses all of them to find the
> Application with the desired {{ApplicationTag}}.
> This process can be improved in a few ways to use less resources,
> particularly threads, and to shorten the latency. Some of the improvements
> are straightforward to do, but others
> # Spawn only one thread to poll Yarn on behalf of all the applications to get
> the {{ApplicationId}}
> # Use a thread pool to check the status of running applications
> # Avoid launching {{spark-submit}}. Instead call
> {{org.apache.spark.SparkSubmit}}
> 1,2 are easy to do, but for 3 we should decide how to redirect the {{stdout}}
> and {{stderr}} and needs more discussion.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
