[
https://issues.apache.org/jira/browse/HIVE-15947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subramanyam Pattipaka updated HIVE-15947:
-----------------------------------------
Attachment: HIVE-15947.3.patch
Changed implementation to use Thread pools instead of semaphore. Also
implemented Thread pool time out functionality as well.
> Enhance Templeton service job operations reliability
> ----------------------------------------------------
>
> Key: HIVE-15947
> URL: https://issues.apache.org/jira/browse/HIVE-15947
> Project: Hive
> Issue Type: Bug
> Reporter: Subramanyam Pattipaka
> Assignee: Subramanyam Pattipaka
> Attachments: HIVE-15947.2.patch, HIVE-15947.3.patch, HIVE-15947.patch
>
>
> Currently Templeton service doesn't restrict number of job operation
> requests. It simply accepts and tries to run all operations. If more number
> of concurrent job submit requests comes then the time to submit job
> operations can increase significantly. Templetonused hdfs to store staging
> file for job. If HDFS storage can't respond to large number of requests and
> throttles then the job submission can take very large times in order of
> minutes.
> This behavior may not be suitable for all applications and client
> applications may be looking for predictable and low response for successful
> request or send throttle response to client to wait for some time before
> re-requesting job operation.
> In this JIRA, I am trying to address following job operations
> 1) Submit new Job
> 2) Get Job Status
> 3) List jobs
> These three operations has different complexity due to variance in use of
> cluster resources like YARN/HDFS.
> The idea is to introduce a new config templeton.job.submit.exec.max-procs
> which controls maximum number of concurrent active job submissions within
> Templeton and use this config to control better response times. If a new job
> submission request sees that there are already
> templeton.job.submit.exec.max-procs jobs getting submitted concurrently then
> the request will fail with Http error 503 with reason
> βToo many concurrent job submission requests received. Please wait for
> some time before retrying.β
>
> The client is expected to catch this response and retry after waiting for
> some time. The default value for the config
> templeton.job.submit.exec.max-procs is set to β0β. This means by default job
> submission requests are always accepted. The behavior needs to be enabled
> based on requirements.
> We can have similar behavior for Status and List operations with configs
> templeton.job.status.exec.max-procs and templeton.list.job.exec.max-procs
> respectively.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
