[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[ https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15539510#comment-15539510 ] Chaoyu Tang commented on HIVE-9423: --- Thanks [~leftylev] for catching this. My fault. I have updated the errata.txt for this missing JIRA in the commit message. See HIVE-14874. > HiveServer2: Provide the user with different error messages depending on the > Thrift client exception code > - > > Key: HIVE-9423 > URL: https://issues.apache.org/jira/browse/HIVE-9423 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0 >Reporter: Vaibhav Gumashta >Assignee: Peter Vary > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch, > HIVE-9423.5-branch-2.1.patch, HIVE-9423.5.patch, > HIVE-9423.6-branch-2.1.patch, HIVE-9423.patch > > > After verifying that the original problem is mostly solved by the Thrift > upgrade, I created a patch to provide better error message when possible > Original description for reference: > --- > An example of where it is needed: it has been reported that when # of client > connections is greater than {{hive.server2.thrift.max.worker.threads}}, > HiveServer2 stops accepting new connections and ends up having to be > restarted. This should be handled more gracefully by the server and the JDBC > driver, so that the end user gets aware of the problem and can take > appropriate steps (either close existing connections or bump of the config > value or use multiple server instances with dynamic service discovery > enabled). Similarly, we should also review the behaviour of background thread > pool to have a well defined behavior on the the pool getting exhausted. > Ideally implementing some form of general admission control will be a better > solution, so that we do not accept new work unless sufficient resources are > available and display graceful degradation under overload. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[ https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15537841#comment-15537841 ] Lefty Leverenz commented on HIVE-9423: -- [~ctang.ma], the commit to master omitted the JIRA number. Please update errata.txt for commit d16d4f1bcc43d6ebcab0eaf5bc635fb88b60be5f. Thanks. Example of updating errata.txt: HIVE-11876. > HiveServer2: Provide the user with different error messages depending on the > Thrift client exception code > - > > Key: HIVE-9423 > URL: https://issues.apache.org/jira/browse/HIVE-9423 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0 >Reporter: Vaibhav Gumashta >Assignee: Peter Vary > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch, > HIVE-9423.5-branch-2.1.patch, HIVE-9423.5.patch, > HIVE-9423.6-branch-2.1.patch, HIVE-9423.patch > > > After verifying that the original problem is mostly solved by the Thrift > upgrade, I created a patch to provide better error message when possible > Original description for reference: > --- > An example of where it is needed: it has been reported that when # of client > connections is greater than {{hive.server2.thrift.max.worker.threads}}, > HiveServer2 stops accepting new connections and ends up having to be > restarted. This should be handled more gracefully by the server and the JDBC > driver, so that the end user gets aware of the problem and can take > appropriate steps (either close existing connections or bump of the config > value or use multiple server instances with dynamic service discovery > enabled). Similarly, we should also review the behaviour of background thread > pool to have a well defined behavior on the the pool getting exhausted. > Ideally implementing some form of general admission control will be a better > solution, so that we do not accept new work unless sufficient resources are > available and display graceful degradation under overload. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[ https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15534883#comment-15534883 ] Hive QA commented on HIVE-9423: --- Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12830914/HIVE-9423.6-branch-2.1.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 252 failed/errored test(s), 10435 tests executed *Failed tests:* {noformat} 249_TestHWISessionManager - did not produce a TEST-*.xml file 392_TestMsgBusConnection - did not produce a TEST-*.xml file 792_TestJdbcWithMiniKdcSQLAuthHttp - did not produce a TEST-*.xml file 793_TestJdbcWithMiniKdc - did not produce a TEST-*.xml file 794_TestHs2HooksWithMiniKdc - did not produce a TEST-*.xml file 796_TestJdbcWithDBTokenStore - did not produce a TEST-*.xml file 797_TestJdbcWithMiniKdcCookie - did not produce a TEST-*.xml file 798_TestJdbcNonKrbSASLWithMiniKdc - did not produce a TEST-*.xml file 800_TestJdbcWithMiniKdcSQLAuthBinary - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_mapjoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_table_stats org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_explain org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_binary_output_format org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_outer_join_ppr org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_char_udf1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnStatsUpdateForStatsOptimizer_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnStatsUpdateForStatsOptimizer_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_describe_table org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_full org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_partial org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_partial_ndv org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fouter_join_ppr org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_map_ppr org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_map_ppr_multi_distinct org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_ppr org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_ppr_multi_distinct org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input23 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input42 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_orig_table_use_metadata org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join0 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join17 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join26 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32_lessSize org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join33 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join34 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join35 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_map_ppr org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_json_serde1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_14 org.apache.hadoop.hive.cli.TestCliDriver.tes
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[ https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15526618#comment-15526618 ] Peter Vary commented on HIVE-9423: -- Hi [~vgumashta], Ok - I kept the description to document that there is no hangs any more. I would be happy to help in that jira - maybe I was wrong when I thought that having hive.server2.thrift.max.worker.threads, hive.server2.thrift.min.worker.threads, hive.server2.thrift.exponential.backoff.slot.length and hive.server2.thrift.login.timeout to control the Thrift server connection settings is enough. Since 0.9.2 the Thrift code respect the configurations above - no hangs, and restarts are needed -, and with this patch I try to provide a meaningful error message. If you have something even better, I would be happy to help. Thanks to checking out this jira, Peter > HiveServer2: Provide the user with different error messages depending on the > Thrift client exception code > - > > Key: HIVE-9423 > URL: https://issues.apache.org/jira/browse/HIVE-9423 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0 >Reporter: Vaibhav Gumashta >Assignee: Peter Vary > Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch, > HIVE-9423.5.patch, HIVE-9423.patch > > > An example of where it is needed: it has been reported that when # of client > connections is greater than {{hive.server2.thrift.max.worker.threads}}, > HiveServer2 stops accepting new connections and ends up having to be > restarted. This should be handled more gracefully by the server and the JDBC > driver, so that the end user gets aware of the problem and can take > appropriate steps (either close existing connections or bump of the config > value or use multiple server instances with dynamic service discovery > enabled). Similarly, we should also review the behaviour of background thread > pool to have a well defined behavior on the the pool getting exhausted. > Ideally implementing some form of general admission control will be a better > solution, so that we do not accept new work unless sufficient resources are > available and display graceful degradation under overload. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[ https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15526557#comment-15526557 ] Vaibhav Gumashta commented on HIVE-9423: [~pvary] Thanks for the patch. I'll create another jira to reflect the original intent (admission control etc) for which this was created. You may want to modify the description of this one. > HiveServer2: Provide the user with different error messages depending on the > Thrift client exception code > - > > Key: HIVE-9423 > URL: https://issues.apache.org/jira/browse/HIVE-9423 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0 >Reporter: Vaibhav Gumashta >Assignee: Peter Vary > Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch, > HIVE-9423.5.patch, HIVE-9423.patch > > > An example of where it is needed: it has been reported that when # of client > connections is greater than {{hive.server2.thrift.max.worker.threads}}, > HiveServer2 stops accepting new connections and ends up having to be > restarted. This should be handled more gracefully by the server and the JDBC > driver, so that the end user gets aware of the problem and can take > appropriate steps (either close existing connections or bump of the config > value or use multiple server instances with dynamic service discovery > enabled). Similarly, we should also review the behaviour of background thread > pool to have a well defined behavior on the the pool getting exhausted. > Ideally implementing some form of general admission control will be a better > solution, so that we do not accept new work unless sufficient resources are > available and display graceful degradation under overload. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[ https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15526119#comment-15526119 ] Chaoyu Tang commented on HIVE-9423: --- LGTM, +1 > HiveServer2: Provide the user with different error messages depending on the > Thrift client exception code > - > > Key: HIVE-9423 > URL: https://issues.apache.org/jira/browse/HIVE-9423 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0 >Reporter: Vaibhav Gumashta >Assignee: Peter Vary > Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch, > HIVE-9423.5.patch, HIVE-9423.patch > > > An example of where it is needed: it has been reported that when # of client > connections is greater than {{hive.server2.thrift.max.worker.threads}}, > HiveServer2 stops accepting new connections and ends up having to be > restarted. This should be handled more gracefully by the server and the JDBC > driver, so that the end user gets aware of the problem and can take > appropriate steps (either close existing connections or bump of the config > value or use multiple server instances with dynamic service discovery > enabled). Similarly, we should also review the behaviour of background thread > pool to have a well defined behavior on the the pool getting exhausted. > Ideally implementing some form of general admission control will be a better > solution, so that we do not accept new work unless sufficient resources are > available and display graceful degradation under overload. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[ https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15525880#comment-15525880 ] Hive QA commented on HIVE-9423: --- Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12830462/HIVE-9423.5.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10641 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char] org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1312/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1312/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1312/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 6 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12830462 - PreCommit-HIVE-Build > HiveServer2: Provide the user with different error messages depending on the > Thrift client exception code > - > > Key: HIVE-9423 > URL: https://issues.apache.org/jira/browse/HIVE-9423 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0 >Reporter: Vaibhav Gumashta >Assignee: Peter Vary > Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch, > HIVE-9423.5.patch, HIVE-9423.patch > > > An example of where it is needed: it has been reported that when # of client > connections is greater than {{hive.server2.thrift.max.worker.threads}}, > HiveServer2 stops accepting new connections and ends up having to be > restarted. This should be handled more gracefully by the server and the JDBC > driver, so that the end user gets aware of the problem and can take > appropriate steps (either close existing connections or bump of the config > value or use multiple server instances with dynamic service discovery > enabled). Similarly, we should also review the behaviour of background thread > pool to have a well defined behavior on the the pool getting exhausted. > Ideally implementing some form of general admission control will be a better > solution, so that we do not accept new work unless sufficient resources are > available and display graceful degradation under overload. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[ https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15525465#comment-15525465 ] Peter Vary commented on HIVE-9423: -- Hi [~ctang.ma], I have tried to dig deeper about what happens when the server closes the connection, but the client still tries to read. I have not found a definitive answer for it, the best thing which I have found is a stackoverflow discussion: http://stackoverflow.com/questions/10240694/java-socket-api-how-to-tell-if-a-connection-has-been-closed This tells us, that if the server closes the connection, and nothing unplanned happens, then the client should not get an IOException. Is it possible to check the Thrift server output to find out more about that Sentry case? As for the error messages you are absolutely right, again. I will post a new patch with revised error messages, and if you still have better ones then feel free to suggest something (to tell the truth, until now phrasing messages was not in my expertise :) Thanks, Peter > HiveServer2: Provide the user with different error messages depending on the > Thrift client exception code > - > > Key: HIVE-9423 > URL: https://issues.apache.org/jira/browse/HIVE-9423 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0 >Reporter: Vaibhav Gumashta >Assignee: Peter Vary > Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch, > HIVE-9423.patch > > > An example of where it is needed: it has been reported that when # of client > connections is greater than {{hive.server2.thrift.max.worker.threads}}, > HiveServer2 stops accepting new connections and ends up having to be > restarted. This should be handled more gracefully by the server and the JDBC > driver, so that the end user gets aware of the problem and can take > appropriate steps (either close existing connections or bump of the config > value or use multiple server instances with dynamic service discovery > enabled). Similarly, we should also review the behaviour of background thread > pool to have a well defined behavior on the the pool getting exhausted. > Ideally implementing some form of general admission control will be a better > solution, so that we do not accept new work unless sufficient resources are > available and display graceful degradation under overload. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[ https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15525466#comment-15525466 ] Peter Vary commented on HIVE-9423: -- Hi [~ctang.ma], I have tried to dig deeper about what happens when the server closes the connection, but the client still tries to read. I have not found a definitive answer for it, the best thing which I have found is a stackoverflow discussion: http://stackoverflow.com/questions/10240694/java-socket-api-how-to-tell-if-a-connection-has-been-closed This tells us, that if the server closes the connection, and nothing unplanned happens, then the client should not get an IOException. Is it possible to check the Thrift server output to find out more about that Sentry case? As for the error messages you are absolutely right, again. I will post a new patch with revised error messages, and if you still have better ones then feel free to suggest something (to tell the truth, until now phrasing messages was not in my expertise :) Thanks, Peter > HiveServer2: Provide the user with different error messages depending on the > Thrift client exception code > - > > Key: HIVE-9423 > URL: https://issues.apache.org/jira/browse/HIVE-9423 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0 >Reporter: Vaibhav Gumashta >Assignee: Peter Vary > Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch, > HIVE-9423.patch > > > An example of where it is needed: it has been reported that when # of client > connections is greater than {{hive.server2.thrift.max.worker.threads}}, > HiveServer2 stops accepting new connections and ends up having to be > restarted. This should be handled more gracefully by the server and the JDBC > driver, so that the end user gets aware of the problem and can take > appropriate steps (either close existing connections or bump of the config > value or use multiple server instances with dynamic service discovery > enabled). Similarly, we should also review the behaviour of background thread > pool to have a well defined behavior on the the pool getting exhausted. > Ideally implementing some form of general admission control will be a better > solution, so that we do not accept new work unless sufficient resources are > available and display graceful degradation under overload. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[ https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15517298#comment-15517298 ] Chaoyu Tang commented on HIVE-9423: --- Yeah, it makes sense. This patch improves the usability though we might have missed some error case (e.g. TTransportException: java.net.SocketException: Connection reset") which is also resulted from exceeding the max worker #. +1 > HiveServer2: Provide the user with different error messages depending on the > Thrift client exception code > - > > Key: HIVE-9423 > URL: https://issues.apache.org/jira/browse/HIVE-9423 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0 >Reporter: Vaibhav Gumashta >Assignee: Peter Vary > Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch, > HIVE-9423.patch > > > An example of where it is needed: it has been reported that when # of client > connections is greater than {{hive.server2.thrift.max.worker.threads}}, > HiveServer2 stops accepting new connections and ends up having to be > restarted. This should be handled more gracefully by the server and the JDBC > driver, so that the end user gets aware of the problem and can take > appropriate steps (either close existing connections or bump of the config > value or use multiple server instances with dynamic service discovery > enabled). Similarly, we should also review the behaviour of background thread > pool to have a well defined behavior on the the pool getting exhausted. > Ideally implementing some form of general admission control will be a better > solution, so that we do not accept new work unless sufficient resources are > available and display graceful degradation under overload. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[ https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15516910#comment-15516910 ] Peter Vary commented on HIVE-9423: -- I will think about this on the weekend, but my first guess is, that if there is not enough connection in the TThreadPoolServer, then it accepts the connections, and after the timeout closes it. {code} TTransport client = serverTransport_.accept(); [..] client.close(); {code} Ideally the executor service will not write anything into it (since it will throw an RejectedExecutionException) - what is the configuration of the ThreadPool in Sentry? Does it throw a RejectedExecutionException when the Theadpool is exhausted? On the Thrift client side we have this code: {code} try { bytesRead = inputStream_.read(buf, off, len); } catch (IOException iox) { throw new TTransportException(TTransportException.UNKNOWN, iox); } if (bytesRead < 0) { throw new TTransportException(TTransportException.END_OF_FILE); } {code} Reading from the InputStream read function documentation: {code} If no byte is available because the stream is at end of file, the value -1 is returned; otherwise, at least one byte is read and stored into b. {code} {code} IOException - If the first byte cannot be read for any reason other than end of file, or if the input stream has been closed, or if some other I/O error occurs. {code} Reading this my guess this can be a timing issue here. Thinking :) > HiveServer2: Provide the user with different error messages depending on the > Thrift client exception code > - > > Key: HIVE-9423 > URL: https://issues.apache.org/jira/browse/HIVE-9423 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0 >Reporter: Vaibhav Gumashta >Assignee: Peter Vary > Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch, > HIVE-9423.patch > > > An example of where it is needed: it has been reported that when # of client > connections is greater than {{hive.server2.thrift.max.worker.threads}}, > HiveServer2 stops accepting new connections and ends up having to be > restarted. This should be handled more gracefully by the server and the JDBC > driver, so that the end user gets aware of the problem and can take > appropriate steps (either close existing connections or bump of the config > value or use multiple server instances with dynamic service discovery > enabled). Similarly, we should also review the behaviour of background thread > pool to have a well defined behavior on the the pool getting exhausted. > Ideally implementing some form of general admission control will be a better > solution, so that we do not accept new work unless sufficient resources are > available and display graceful degradation under overload. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[ https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15516513#comment-15516513 ] Chaoyu Tang commented on HIVE-9423: --- That was run into by the Sentry, I wonder if it could be a similar case to HS2: {code} Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:277) at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) at org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl$UgiSaslClientTransport.baseOpen(SentryPolicyServiceClientDefaultImpl.java:126) at org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl$UgiSaslClientTransport.access$000(SentryPolicyServiceClientDefaultImpl.java:85) at org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl$UgiSaslClientTransport$1.run(SentryPolicyServiceClientDefaultImpl.java:112) at org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl$UgiSaslClientTransport$1.run(SentryPolicyServiceClientDefaultImpl.java:110) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1714) ... 23 more Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:209) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) {code} > HiveServer2: Provide the user with different error messages depending on the > Thrift client exception code > - > > Key: HIVE-9423 > URL: https://issues.apache.org/jira/browse/HIVE-9423 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0 >Reporter: Vaibhav Gumashta >Assignee: Peter Vary > Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch, > HIVE-9423.patch > > > An example of where it is needed: it has been reported that when # of client > connections is greater than {{hive.server2.thrift.max.worker.threads}}, > HiveServer2 stops accepting new connections and ends up having to be > restarted. This should be handled more gracefully by the server and the JDBC > driver, so that the end user gets aware of the problem and can take > appropriate steps (either close existing connections or bump of the config > value or use multiple server instances with dynamic service discovery > enabled). Similarly, we should also review the behaviour of background thread > pool to have a well defined behavior on the the pool getting exhausted. > Ideally implementing some form of general admission control will be a better > solution, so that we do not accept new work unless sufficient resources are > available and display graceful degradation under overload. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[ https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15516381#comment-15516381 ] Peter Vary commented on HIVE-9423: -- Hi [~ctang.ma], I had finished my other tasks, and was about to collect the logs for you. HiveServer2: {code} 2016-09-23T14:59:23,142 WARN [Thread-7] server.TThreadPoolServer: Task has been rejected by ExecutorService 10 times till timedout, reason: java.util.concurrent.RejectedExecutionException: Task org.apache.thrift.server.TThreadPoolServer$WorkerProcess@798d4da3 rejected from org.apache.hive.service.cli.thrift.ThreadPoolExecutorWithOomHook@f7ed8ef[Running, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 1] {code} Beeline without patch, with --verbose=true: {code} Connecting to jdbc:hive2://localhost:1 16/09/23 14:59:23 [main]: WARN jdbc.HiveConnection: Failed to connect to localhost:1 HS2 may be unavailable, check server status Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:1: null (state=08S01,code=0) java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:1: null at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:209) at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) at java.sql.DriverManager.getConnection(DriverManager.java:664) at java.sql.DriverManager.getConnection(DriverManager.java:208) at org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145) at org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:209) at org.apache.hive.beeline.Commands.connect(Commands.java:1524) at org.apache.hive.beeline.Commands.connect(Commands.java:1419) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:56) at org.apache.hive.beeline.BeeLine.execCommandWithPrefix(BeeLine.java:1128) at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1167) at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:798) at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:886) at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:512) at org.apache.hive.beeline.BeeLine.main(BeeLine.java:495) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:307) at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:227) at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:182) ... 24 more Beeline version 2.2.0-SNAPSHOT by Apache Hive {code} In my test cases the server side of Thrift closes the connection without writing any data to the output in case of reaching the max # of thrift threads. As for your question, I have not tried kerberized Beeline/HS2. If there is another timeout somewhere, which shorter/equal than the one defined by hive.server2.thrift.login.timeout which is used by ThriftBinaryCLIService, then it might throw the TTransportException with an inner "Connection reset" IOException and with the type of TTransportException.UNKNOWN. It would be good to find the source of the Connection reset. Could you please post your stack trace? Meanwhile I am trying to reproduce your output. Thanks, Peter > HiveServer2: Provide the user with different error messages depending on the > Thrift client exception code > - > > Key: HIVE-9423 > URL: https://issues.apache.org/jira/browse/HIVE-9423 > Project: Hive > Issue Type: Bug >
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[ https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15516382#comment-15516382 ] Peter Vary commented on HIVE-9423: -- Hi [~ctang.ma], I had finished my other tasks, and was about to collect the logs for you. HiveServer2: {code} 2016-09-23T14:59:23,142 WARN [Thread-7] server.TThreadPoolServer: Task has been rejected by ExecutorService 10 times till timedout, reason: java.util.concurrent.RejectedExecutionException: Task org.apache.thrift.server.TThreadPoolServer$WorkerProcess@798d4da3 rejected from org.apache.hive.service.cli.thrift.ThreadPoolExecutorWithOomHook@f7ed8ef[Running, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 1] {code} Beeline without patch, with --verbose=true: {code} Connecting to jdbc:hive2://localhost:1 16/09/23 14:59:23 [main]: WARN jdbc.HiveConnection: Failed to connect to localhost:1 HS2 may be unavailable, check server status Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:1: null (state=08S01,code=0) java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:1: null at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:209) at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) at java.sql.DriverManager.getConnection(DriverManager.java:664) at java.sql.DriverManager.getConnection(DriverManager.java:208) at org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145) at org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:209) at org.apache.hive.beeline.Commands.connect(Commands.java:1524) at org.apache.hive.beeline.Commands.connect(Commands.java:1419) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:56) at org.apache.hive.beeline.BeeLine.execCommandWithPrefix(BeeLine.java:1128) at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1167) at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:798) at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:886) at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:512) at org.apache.hive.beeline.BeeLine.main(BeeLine.java:495) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:307) at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:227) at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:182) ... 24 more Beeline version 2.2.0-SNAPSHOT by Apache Hive {code} In my test cases the server side of Thrift closes the connection without writing any data to the output in case of reaching the max # of thrift threads. As for your question, I have not tried kerberized Beeline/HS2. If there is another timeout somewhere, which shorter/equal than the one defined by hive.server2.thrift.login.timeout which is used by ThriftBinaryCLIService, then it might throw the TTransportException with an inner "Connection reset" IOException and with the type of TTransportException.UNKNOWN. It would be good to find the source of the Connection reset. Could you please post your stack trace? Meanwhile I am trying to reproduce your output. Thanks, Peter > HiveServer2: Provide the user with different error messages depending on the > Thrift client exception code > - > > Key: HIVE-9423 > URL: https://issues.apache.org/jira/browse/HIVE-9423 > Project: Hive > Issue Type: Bug >
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[ https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15516278#comment-15516278 ] Chaoyu Tang commented on HIVE-9423: --- In some case when the configured max # of thrift thread is reached, I think the client might also run into the TTransportException wrapping the java.net.SocketException with message "Connection reset". I wonder if you have run into such a case or not, and have you tried that in kerberized Beeline/HS2? > HiveServer2: Provide the user with different error messages depending on the > Thrift client exception code > - > > Key: HIVE-9423 > URL: https://issues.apache.org/jira/browse/HIVE-9423 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0 >Reporter: Vaibhav Gumashta >Assignee: Peter Vary > Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch, > HIVE-9423.patch > > > An example of where it is needed: it has been reported that when # of client > connections is greater than {{hive.server2.thrift.max.worker.threads}}, > HiveServer2 stops accepting new connections and ends up having to be > restarted. This should be handled more gracefully by the server and the JDBC > driver, so that the end user gets aware of the problem and can take > appropriate steps (either close existing connections or bump of the config > value or use multiple server instances with dynamic service discovery > enabled). Similarly, we should also review the behaviour of background thread > pool to have a well defined behavior on the the pool getting exhausted. > Ideally implementing some form of general admission control will be a better > solution, so that we do not accept new work unless sufficient resources are > available and display graceful degradation under overload. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[ https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514219#comment-15514219 ] Lefty Leverenz commented on HIVE-9423: -- +1 for the new error messages (patch 4) > HiveServer2: Provide the user with different error messages depending on the > Thrift client exception code > - > > Key: HIVE-9423 > URL: https://issues.apache.org/jira/browse/HIVE-9423 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0 >Reporter: Vaibhav Gumashta >Assignee: Peter Vary > Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch, > HIVE-9423.patch > > > An example of where it is needed: it has been reported that when # of client > connections is greater than {{hive.server2.thrift.max.worker.threads}}, > HiveServer2 stops accepting new connections and ends up having to be > restarted. This should be handled more gracefully by the server and the JDBC > driver, so that the end user gets aware of the problem and can take > appropriate steps (either close existing connections or bump of the config > value or use multiple server instances with dynamic service discovery > enabled). Similarly, we should also review the behaviour of background thread > pool to have a well defined behavior on the the pool getting exhausted. > Ideally implementing some form of general admission control will be a better > solution, so that we do not accept new work unless sufficient resources are > available and display graceful degradation under overload. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[ https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513905#comment-15513905 ] Hive QA commented on HIVE-9423: --- Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12829872/HIVE-9423.4.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10556 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char] org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1274/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1274/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1274/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 6 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12829872 - PreCommit-HIVE-Build > HiveServer2: Provide the user with different error messages depending on the > Thrift client exception code > - > > Key: HIVE-9423 > URL: https://issues.apache.org/jira/browse/HIVE-9423 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0 >Reporter: Vaibhav Gumashta >Assignee: Peter Vary > Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch, > HIVE-9423.patch > > > An example of where it is needed: it has been reported that when # of client > connections is greater than {{hive.server2.thrift.max.worker.threads}}, > HiveServer2 stops accepting new connections and ends up having to be > restarted. This should be handled more gracefully by the server and the JDBC > driver, so that the end user gets aware of the problem and can take > appropriate steps (either close existing connections or bump of the config > value or use multiple server instances with dynamic service discovery > enabled). Similarly, we should also review the behaviour of background thread > pool to have a well defined behavior on the the pool getting exhausted. > Ideally implementing some form of general admission control will be a better > solution, so that we do not accept new work unless sufficient resources are > available and display graceful degradation under overload. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[ https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513653#comment-15513653 ] Vihang Karajgaonkar commented on HIVE-9423: --- Thanks for the patch [~pvary]. This issue has been a pain point for beeline users and more user-friendly error messages helps a lot. Patch looks good to me. > HiveServer2: Provide the user with different error messages depending on the > Thrift client exception code > - > > Key: HIVE-9423 > URL: https://issues.apache.org/jira/browse/HIVE-9423 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0 >Reporter: Vaibhav Gumashta >Assignee: Peter Vary > Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch, > HIVE-9423.patch > > > An example of where it is needed: it has been reported that when # of client > connections is greater than {{hive.server2.thrift.max.worker.threads}}, > HiveServer2 stops accepting new connections and ends up having to be > restarted. This should be handled more gracefully by the server and the JDBC > driver, so that the end user gets aware of the problem and can take > appropriate steps (either close existing connections or bump of the config > value or use multiple server instances with dynamic service discovery > enabled). Similarly, we should also review the behaviour of background thread > pool to have a well defined behavior on the the pool getting exhausted. > Ideally implementing some form of general admission control will be a better > solution, so that we do not accept new work unless sufficient resources are > available and display graceful degradation under overload. -- This message was sent by Atlassian JIRA (v6.3.4#6332)