[
https://issues.apache.org/jira/browse/HIVE-15947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15871264#comment-15871264
]
Hive QA commented on HIVE-15947:
--------------------------------
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12853173/HIVE-15947.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 34 failed/errored test(s), 10236 tests
executed
*Failed tests:*
{noformat}
TestConcurrentJobRequestsBase - did not produce a TEST-*.xml file (likely timed
out) (batchId=171)
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_auto_join1]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join31] (batchId=81)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multiMapJoin2]
(batchId=152)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[create_external_acid]
(batchId=86)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[create_not_acid]
(batchId=86)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[create_view_failure1]
(batchId=85)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[create_view_failure2]
(batchId=85)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[external1]
(batchId=85)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=223)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=127)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join31]
(batchId=133)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testAlterTable
(batchId=194)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=194)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testAlterTable
(batchId=197)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testTransactionalValidation
(batchId=197)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testAlterTable
(batchId=193)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testTransactionalValidation
(batchId=193)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testAlterTable
(batchId=191)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testTransactionalValidation
(batchId=191)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testAlterTable
(batchId=202)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testTransactionalValidation
(batchId=202)
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
(batchId=210)
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
(batchId=208)
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
(batchId=220)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel
(batchId=211)
org.apache.hive.hcatalog.api.TestHCatClient.testBasicDDLCommands (batchId=170)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=170)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.createTable
(batchId=221)
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
(batchId=213)
org.apache.hive.service.cli.thrift.TestThriftCLIServiceWithBinary.testExecuteStatementAsync
(batchId=213)
org.apache.hive.service.cli.thrift.TestThriftCLIServiceWithHttp.testExecuteStatementAsync
(batchId=213)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3616/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3616/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3616/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 34 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12853173 - PreCommit-HIVE-Build
> Enhance Templeton service job operations reliability
> ----------------------------------------------------
>
> Key: HIVE-15947
> URL: https://issues.apache.org/jira/browse/HIVE-15947
> Project: Hive
> Issue Type: Bug
> Reporter: Subramanyam Pattipaka
> Assignee: Subramanyam Pattipaka
> Attachments: HIVE-15947.patch
>
>
> Currently Templeton service doesn't restrict number of job operation
> requests. It simply accepts and tries to run all operations. If more number
> of concurrent job submit requests comes then the time to submit job
> operations can increase significantly. Templetonused hdfs to store staging
> file for job. If HDFS storage can't respond to large number of requests and
> throttles then the job submission can take very large times in order of
> minutes.
> This behavior may not be suitable for all applications and client
> applications may be looking for predictable and low response for successful
> request or send throttle response to client to wait for some time before
> re-requesting job operation.
> In this JIRA, I am trying to address following job operations
> 1) Submit new Job
> 2) Get Job Status
> 3) List jobs
> These three operations has different complexity due to variance in use of
> cluster resources like YARN/HDFS.
> The idea is to introduce a new config templeton.job.submit.exec.max-procs
> which controls maximum number of concurrent active job submissions within
> Templeton and use this config to control better response times. If a new job
> submission request sees that there are already
> templeton.job.submit.exec.max-procs jobs getting submitted concurrently then
> the request will fail with Http error 503 with reason
> βToo many concurrent job submission requests received. Please wait for
> some time before retrying.β
>
> The client is expected to catch this response and retry after waiting for
> some time. The default value for the config
> templeton.job.submit.exec.max-procs is set to β0β. This means by default job
> submission requests are always accepted. The behavior needs to be enabled
> based on requirements.
> We can have similar behavior for Status and List operations with configs
> templeton.job.status.exec.max-procs and templeton.list.job.exec.max-procs
> respectively.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
