[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15539510#comment-15539510
]
Chaoyu Tang commented on HIVE-9423:
---
Thanks [~leftylev] for catching this. My fault. I have updated the errata.txt
for this missing JIRA in the commit message. See HIVE-14874.
> HiveServer2: Provide the user with different error messages depending on the
> Thrift client exception code
> -
>
> Key: HIVE-9423
> URL: https://issues.apache.org/jira/browse/HIVE-9423
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0
>Reporter: Vaibhav Gumashta
>Assignee: Peter Vary
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch,
> HIVE-9423.5-branch-2.1.patch, HIVE-9423.5.patch,
> HIVE-9423.6-branch-2.1.patch, HIVE-9423.patch
>
>
> After verifying that the original problem is mostly solved by the Thrift
> upgrade, I created a patch to provide better error message when possible
> Original description for reference:
> ---
> An example of where it is needed: it has been reported that when # of client
> connections is greater than {{hive.server2.thrift.max.worker.threads}},
> HiveServer2 stops accepting new connections and ends up having to be
> restarted. This should be handled more gracefully by the server and the JDBC
> driver, so that the end user gets aware of the problem and can take
> appropriate steps (either close existing connections or bump of the config
> value or use multiple server instances with dynamic service discovery
> enabled). Similarly, we should also review the behaviour of background thread
> pool to have a well defined behavior on the the pool getting exhausted.
> Ideally implementing some form of general admission control will be a better
> solution, so that we do not accept new work unless sufficient resources are
> available and display graceful degradation under overload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534883#comment-15534883
]
Hive QA commented on HIVE-9423:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12830914/HIVE-9423.6-branch-2.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 252 failed/errored test(s), 10435 tests
executed
*Failed tests:*
{noformat}
249_TestHWISessionManager - did not produce a TEST-*.xml file
392_TestMsgBusConnection - did not produce a TEST-*.xml file
792_TestJdbcWithMiniKdcSQLAuthHttp - did not produce a TEST-*.xml file
793_TestJdbcWithMiniKdc - did not produce a TEST-*.xml file
794_TestHs2HooksWithMiniKdc - did not produce a TEST-*.xml file
796_TestJdbcWithDBTokenStore - did not produce a TEST-*.xml file
797_TestJdbcWithMiniKdcCookie - did not produce a TEST-*.xml file
798_TestJdbcNonKrbSASLWithMiniKdc - did not produce a TEST-*.xml file
800_TestJdbcWithMiniKdcSQLAuthBinary - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_table_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_explain
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_binary_output_format
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_outer_join_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_char_udf1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnStatsUpdateForStatsOptimizer_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnStatsUpdateForStatsOptimizer_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_describe_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_full
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_partial
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_partial_ndv
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fouter_join_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_map_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_map_ppr_multi_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_ppr_multi_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input42
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_orig_table_use_metadata
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join0
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join17
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join26
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32_lessSize
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join33
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join34
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join35
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_map_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_json_serde1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_14
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526618#comment-15526618
]
Peter Vary commented on HIVE-9423:
--
Hi [~vgumashta],
Ok - I kept the description to document that there is no hangs any more. I
would be happy to help in that jira - maybe I was wrong when I thought that
having hive.server2.thrift.max.worker.threads,
hive.server2.thrift.min.worker.threads,
hive.server2.thrift.exponential.backoff.slot.length and
hive.server2.thrift.login.timeout to control the Thrift server connection
settings is enough.
Since 0.9.2 the Thrift code respect the configurations above - no hangs, and
restarts are needed -, and with this patch I try to provide a meaningful error
message.
If you have something even better, I would be happy to help.
Thanks to checking out this jira,
Peter
> HiveServer2: Provide the user with different error messages depending on the
> Thrift client exception code
> -
>
> Key: HIVE-9423
> URL: https://issues.apache.org/jira/browse/HIVE-9423
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0
>Reporter: Vaibhav Gumashta
>Assignee: Peter Vary
> Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch,
> HIVE-9423.5.patch, HIVE-9423.patch
>
>
> An example of where it is needed: it has been reported that when # of client
> connections is greater than {{hive.server2.thrift.max.worker.threads}},
> HiveServer2 stops accepting new connections and ends up having to be
> restarted. This should be handled more gracefully by the server and the JDBC
> driver, so that the end user gets aware of the problem and can take
> appropriate steps (either close existing connections or bump of the config
> value or use multiple server instances with dynamic service discovery
> enabled). Similarly, we should also review the behaviour of background thread
> pool to have a well defined behavior on the the pool getting exhausted.
> Ideally implementing some form of general admission control will be a better
> solution, so that we do not accept new work unless sufficient resources are
> available and display graceful degradation under overload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526557#comment-15526557
]
Vaibhav Gumashta commented on HIVE-9423:
[~pvary] Thanks for the patch. I'll create another jira to reflect the original
intent (admission control etc) for which this was created. You may want to
modify the description of this one.
> HiveServer2: Provide the user with different error messages depending on the
> Thrift client exception code
> -
>
> Key: HIVE-9423
> URL: https://issues.apache.org/jira/browse/HIVE-9423
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0
>Reporter: Vaibhav Gumashta
>Assignee: Peter Vary
> Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch,
> HIVE-9423.5.patch, HIVE-9423.patch
>
>
> An example of where it is needed: it has been reported that when # of client
> connections is greater than {{hive.server2.thrift.max.worker.threads}},
> HiveServer2 stops accepting new connections and ends up having to be
> restarted. This should be handled more gracefully by the server and the JDBC
> driver, so that the end user gets aware of the problem and can take
> appropriate steps (either close existing connections or bump of the config
> value or use multiple server instances with dynamic service discovery
> enabled). Similarly, we should also review the behaviour of background thread
> pool to have a well defined behavior on the the pool getting exhausted.
> Ideally implementing some form of general admission control will be a better
> solution, so that we do not accept new work unless sufficient resources are
> available and display graceful degradation under overload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526119#comment-15526119
]
Chaoyu Tang commented on HIVE-9423:
---
LGTM, +1
> HiveServer2: Provide the user with different error messages depending on the
> Thrift client exception code
> -
>
> Key: HIVE-9423
> URL: https://issues.apache.org/jira/browse/HIVE-9423
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0
>Reporter: Vaibhav Gumashta
>Assignee: Peter Vary
> Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch,
> HIVE-9423.5.patch, HIVE-9423.patch
>
>
> An example of where it is needed: it has been reported that when # of client
> connections is greater than {{hive.server2.thrift.max.worker.threads}},
> HiveServer2 stops accepting new connections and ends up having to be
> restarted. This should be handled more gracefully by the server and the JDBC
> driver, so that the end user gets aware of the problem and can take
> appropriate steps (either close existing connections or bump of the config
> value or use multiple server instances with dynamic service discovery
> enabled). Similarly, we should also review the behaviour of background thread
> pool to have a well defined behavior on the the pool getting exhausted.
> Ideally implementing some form of general admission control will be a better
> solution, so that we do not accept new work unless sufficient resources are
> available and display graceful degradation under overload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525880#comment-15525880
]
Hive QA commented on HIVE-9423:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12830462/HIVE-9423.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10641 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1312/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1312/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1312/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12830462 - PreCommit-HIVE-Build
> HiveServer2: Provide the user with different error messages depending on the
> Thrift client exception code
> -
>
> Key: HIVE-9423
> URL: https://issues.apache.org/jira/browse/HIVE-9423
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0
>Reporter: Vaibhav Gumashta
>Assignee: Peter Vary
> Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch,
> HIVE-9423.5.patch, HIVE-9423.patch
>
>
> An example of where it is needed: it has been reported that when # of client
> connections is greater than {{hive.server2.thrift.max.worker.threads}},
> HiveServer2 stops accepting new connections and ends up having to be
> restarted. This should be handled more gracefully by the server and the JDBC
> driver, so that the end user gets aware of the problem and can take
> appropriate steps (either close existing connections or bump of the config
> value or use multiple server instances with dynamic service discovery
> enabled). Similarly, we should also review the behaviour of background thread
> pool to have a well defined behavior on the the pool getting exhausted.
> Ideally implementing some form of general admission control will be a better
> solution, so that we do not accept new work unless sufficient resources are
> available and display graceful degradation under overload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525465#comment-15525465
]
Peter Vary commented on HIVE-9423:
--
Hi [~ctang.ma],
I have tried to dig deeper about what happens when the server closes the
connection, but the client still tries to read.
I have not found a definitive answer for it, the best thing which I have found
is a stackoverflow discussion:
http://stackoverflow.com/questions/10240694/java-socket-api-how-to-tell-if-a-connection-has-been-closed
This tells us, that if the server closes the connection, and nothing unplanned
happens, then the client should not get an IOException. Is it possible to check
the Thrift server output to find out more about that Sentry case?
As for the error messages you are absolutely right, again. I will post a new
patch with revised error messages, and if you still have better ones then feel
free to suggest something (to tell the truth, until now phrasing messages was
not in my expertise :)
Thanks,
Peter
> HiveServer2: Provide the user with different error messages depending on the
> Thrift client exception code
> -
>
> Key: HIVE-9423
> URL: https://issues.apache.org/jira/browse/HIVE-9423
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0
>Reporter: Vaibhav Gumashta
>Assignee: Peter Vary
> Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch,
> HIVE-9423.patch
>
>
> An example of where it is needed: it has been reported that when # of client
> connections is greater than {{hive.server2.thrift.max.worker.threads}},
> HiveServer2 stops accepting new connections and ends up having to be
> restarted. This should be handled more gracefully by the server and the JDBC
> driver, so that the end user gets aware of the problem and can take
> appropriate steps (either close existing connections or bump of the config
> value or use multiple server instances with dynamic service discovery
> enabled). Similarly, we should also review the behaviour of background thread
> pool to have a well defined behavior on the the pool getting exhausted.
> Ideally implementing some form of general admission control will be a better
> solution, so that we do not accept new work unless sufficient resources are
> available and display graceful degradation under overload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525466#comment-15525466
]
Peter Vary commented on HIVE-9423:
--
Hi [~ctang.ma],
I have tried to dig deeper about what happens when the server closes the
connection, but the client still tries to read.
I have not found a definitive answer for it, the best thing which I have found
is a stackoverflow discussion:
http://stackoverflow.com/questions/10240694/java-socket-api-how-to-tell-if-a-connection-has-been-closed
This tells us, that if the server closes the connection, and nothing unplanned
happens, then the client should not get an IOException. Is it possible to check
the Thrift server output to find out more about that Sentry case?
As for the error messages you are absolutely right, again. I will post a new
patch with revised error messages, and if you still have better ones then feel
free to suggest something (to tell the truth, until now phrasing messages was
not in my expertise :)
Thanks,
Peter
> HiveServer2: Provide the user with different error messages depending on the
> Thrift client exception code
> -
>
> Key: HIVE-9423
> URL: https://issues.apache.org/jira/browse/HIVE-9423
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0
>Reporter: Vaibhav Gumashta
>Assignee: Peter Vary
> Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch,
> HIVE-9423.patch
>
>
> An example of where it is needed: it has been reported that when # of client
> connections is greater than {{hive.server2.thrift.max.worker.threads}},
> HiveServer2 stops accepting new connections and ends up having to be
> restarted. This should be handled more gracefully by the server and the JDBC
> driver, so that the end user gets aware of the problem and can take
> appropriate steps (either close existing connections or bump of the config
> value or use multiple server instances with dynamic service discovery
> enabled). Similarly, we should also review the behaviour of background thread
> pool to have a well defined behavior on the the pool getting exhausted.
> Ideally implementing some form of general admission control will be a better
> solution, so that we do not accept new work unless sufficient resources are
> available and display graceful degradation under overload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517298#comment-15517298
]
Chaoyu Tang commented on HIVE-9423:
---
Yeah, it makes sense. This patch improves the usability though we might have
missed some error case (e.g. TTransportException: java.net.SocketException:
Connection reset") which is also resulted from exceeding the max worker #.
+1
> HiveServer2: Provide the user with different error messages depending on the
> Thrift client exception code
> -
>
> Key: HIVE-9423
> URL: https://issues.apache.org/jira/browse/HIVE-9423
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0
>Reporter: Vaibhav Gumashta
>Assignee: Peter Vary
> Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch,
> HIVE-9423.patch
>
>
> An example of where it is needed: it has been reported that when # of client
> connections is greater than {{hive.server2.thrift.max.worker.threads}},
> HiveServer2 stops accepting new connections and ends up having to be
> restarted. This should be handled more gracefully by the server and the JDBC
> driver, so that the end user gets aware of the problem and can take
> appropriate steps (either close existing connections or bump of the config
> value or use multiple server instances with dynamic service discovery
> enabled). Similarly, we should also review the behaviour of background thread
> pool to have a well defined behavior on the the pool getting exhausted.
> Ideally implementing some form of general admission control will be a better
> solution, so that we do not accept new work unless sufficient resources are
> available and display graceful degradation under overload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516910#comment-15516910
]
Peter Vary commented on HIVE-9423:
--
I will think about this on the weekend, but my first guess is, that if there is
not enough connection in the TThreadPoolServer, then it accepts the
connections, and after the timeout closes it.
{code}
TTransport client = serverTransport_.accept();
[..]
client.close();
{code}
Ideally the executor service will not write anything into it (since it will
throw an RejectedExecutionException) - what is the configuration of the
ThreadPool in Sentry? Does it throw a RejectedExecutionException when the
Theadpool is exhausted?
On the Thrift client side we have this code:
{code}
try {
bytesRead = inputStream_.read(buf, off, len);
} catch (IOException iox) {
throw new TTransportException(TTransportException.UNKNOWN, iox);
}
if (bytesRead < 0) {
throw new TTransportException(TTransportException.END_OF_FILE);
}
{code}
Reading from the InputStream read function documentation:
{code}
If no byte is available because the stream is at end of file, the value -1 is
returned; otherwise, at least one byte is read and stored into b.
{code}
{code}
IOException - If the first byte cannot be read for any reason other than end of
file, or if the input stream has been closed, or if some other I/O error occurs.
{code}
Reading this my guess this can be a timing issue here. Thinking :)
> HiveServer2: Provide the user with different error messages depending on the
> Thrift client exception code
> -
>
> Key: HIVE-9423
> URL: https://issues.apache.org/jira/browse/HIVE-9423
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0
>Reporter: Vaibhav Gumashta
>Assignee: Peter Vary
> Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch,
> HIVE-9423.patch
>
>
> An example of where it is needed: it has been reported that when # of client
> connections is greater than {{hive.server2.thrift.max.worker.threads}},
> HiveServer2 stops accepting new connections and ends up having to be
> restarted. This should be handled more gracefully by the server and the JDBC
> driver, so that the end user gets aware of the problem and can take
> appropriate steps (either close existing connections or bump of the config
> value or use multiple server instances with dynamic service discovery
> enabled). Similarly, we should also review the behaviour of background thread
> pool to have a well defined behavior on the the pool getting exhausted.
> Ideally implementing some form of general admission control will be a better
> solution, so that we do not accept new work unless sufficient resources are
> available and display graceful degradation under overload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516513#comment-15516513
]
Chaoyu Tang commented on HIVE-9423:
---
That was run into by the Sentry, I wonder if it could be a similar case to HS2:
{code}
Caused by: org.apache.thrift.transport.TTransportException:
java.net.SocketException: Connection reset
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at
org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:277)
at
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at
org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl$UgiSaslClientTransport.baseOpen(SentryPolicyServiceClientDefaultImpl.java:126)
at
org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl$UgiSaslClientTransport.access$000(SentryPolicyServiceClientDefaultImpl.java:85)
at
org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl$UgiSaslClientTransport$1.run(SentryPolicyServiceClientDefaultImpl.java:112)
at
org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl$UgiSaslClientTransport$1.run(SentryPolicyServiceClientDefaultImpl.java:110)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1714)
... 23 more
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:209)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
{code}
> HiveServer2: Provide the user with different error messages depending on the
> Thrift client exception code
> -
>
> Key: HIVE-9423
> URL: https://issues.apache.org/jira/browse/HIVE-9423
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0
>Reporter: Vaibhav Gumashta
>Assignee: Peter Vary
> Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch,
> HIVE-9423.patch
>
>
> An example of where it is needed: it has been reported that when # of client
> connections is greater than {{hive.server2.thrift.max.worker.threads}},
> HiveServer2 stops accepting new connections and ends up having to be
> restarted. This should be handled more gracefully by the server and the JDBC
> driver, so that the end user gets aware of the problem and can take
> appropriate steps (either close existing connections or bump of the config
> value or use multiple server instances with dynamic service discovery
> enabled). Similarly, we should also review the behaviour of background thread
> pool to have a well defined behavior on the the pool getting exhausted.
> Ideally implementing some form of general admission control will be a better
> solution, so that we do not accept new work unless sufficient resources are
> available and display graceful degradation under overload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516381#comment-15516381
]
Peter Vary commented on HIVE-9423:
--
Hi [~ctang.ma],
I had finished my other tasks, and was about to collect the logs for you.
HiveServer2:
{code}
2016-09-23T14:59:23,142 WARN [Thread-7] server.TThreadPoolServer: Task has
been rejected by ExecutorService 10 times till timedout, reason:
java.util.concurrent.RejectedExecutionException: Task
org.apache.thrift.server.TThreadPoolServer$WorkerProcess@798d4da3 rejected from
org.apache.hive.service.cli.thrift.ThreadPoolExecutorWithOomHook@f7ed8ef[Running,
pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 1]
{code}
Beeline without patch, with --verbose=true:
{code}
Connecting to jdbc:hive2://localhost:1
16/09/23 14:59:23 [main]: WARN jdbc.HiveConnection: Failed to connect to
localhost:1
HS2 may be unavailable, check server status
Error: Could not open client transport with JDBC Uri:
jdbc:hive2://localhost:1: null (state=08S01,code=0)
java.sql.SQLException: Could not open client transport with JDBC Uri:
jdbc:hive2://localhost:1: null
at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:209)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at
org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145)
at
org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:209)
at org.apache.hive.beeline.Commands.connect(Commands.java:1524)
at org.apache.hive.beeline.Commands.connect(Commands.java:1419)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:56)
at
org.apache.hive.beeline.BeeLine.execCommandWithPrefix(BeeLine.java:1128)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1167)
at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:798)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:886)
at
org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:512)
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:495)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.thrift.transport.TTransportException
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at
org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:307)
at
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at
org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:227)
at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:182)
... 24 more
Beeline version 2.2.0-SNAPSHOT by Apache Hive
{code}
In my test cases the server side of Thrift closes the connection without
writing any data to the output in case of reaching the max # of thrift threads.
As for your question, I have not tried kerberized Beeline/HS2. If there is
another timeout somewhere, which shorter/equal than the one defined by
hive.server2.thrift.login.timeout which is used by ThriftBinaryCLIService, then
it might throw the TTransportException with an inner "Connection reset"
IOException and with the type of TTransportException.UNKNOWN. It would be good
to find the source of the Connection reset. Could you please post your stack
trace? Meanwhile I am trying to reproduce your output.
Thanks,
Peter
> HiveServer2: Provide the user with different error messages depending on the
> Thrift client exception code
> -
>
> Key: HIVE-9423
> URL: https://issues.apache.org/jira/browse/HIVE-9423
> Project: Hive
> Issue Type: Bug
> Components:
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516382#comment-15516382
]
Peter Vary commented on HIVE-9423:
--
Hi [~ctang.ma],
I had finished my other tasks, and was about to collect the logs for you.
HiveServer2:
{code}
2016-09-23T14:59:23,142 WARN [Thread-7] server.TThreadPoolServer: Task has
been rejected by ExecutorService 10 times till timedout, reason:
java.util.concurrent.RejectedExecutionException: Task
org.apache.thrift.server.TThreadPoolServer$WorkerProcess@798d4da3 rejected from
org.apache.hive.service.cli.thrift.ThreadPoolExecutorWithOomHook@f7ed8ef[Running,
pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 1]
{code}
Beeline without patch, with --verbose=true:
{code}
Connecting to jdbc:hive2://localhost:1
16/09/23 14:59:23 [main]: WARN jdbc.HiveConnection: Failed to connect to
localhost:1
HS2 may be unavailable, check server status
Error: Could not open client transport with JDBC Uri:
jdbc:hive2://localhost:1: null (state=08S01,code=0)
java.sql.SQLException: Could not open client transport with JDBC Uri:
jdbc:hive2://localhost:1: null
at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:209)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at
org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145)
at
org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:209)
at org.apache.hive.beeline.Commands.connect(Commands.java:1524)
at org.apache.hive.beeline.Commands.connect(Commands.java:1419)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:56)
at
org.apache.hive.beeline.BeeLine.execCommandWithPrefix(BeeLine.java:1128)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1167)
at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:798)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:886)
at
org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:512)
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:495)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.thrift.transport.TTransportException
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at
org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:307)
at
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at
org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:227)
at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:182)
... 24 more
Beeline version 2.2.0-SNAPSHOT by Apache Hive
{code}
In my test cases the server side of Thrift closes the connection without
writing any data to the output in case of reaching the max # of thrift threads.
As for your question, I have not tried kerberized Beeline/HS2. If there is
another timeout somewhere, which shorter/equal than the one defined by
hive.server2.thrift.login.timeout which is used by ThriftBinaryCLIService, then
it might throw the TTransportException with an inner "Connection reset"
IOException and with the type of TTransportException.UNKNOWN. It would be good
to find the source of the Connection reset. Could you please post your stack
trace? Meanwhile I am trying to reproduce your output.
Thanks,
Peter
> HiveServer2: Provide the user with different error messages depending on the
> Thrift client exception code
> -
>
> Key: HIVE-9423
> URL: https://issues.apache.org/jira/browse/HIVE-9423
> Project: Hive
> Issue Type: Bug
> Components:
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516278#comment-15516278
]
Chaoyu Tang commented on HIVE-9423:
---
In some case when the configured max # of thrift thread is reached, I think the
client might also run into the TTransportException wrapping the
java.net.SocketException with message "Connection reset". I wonder if you have
run into such a case or not, and have you tried that in kerberized Beeline/HS2?
> HiveServer2: Provide the user with different error messages depending on the
> Thrift client exception code
> -
>
> Key: HIVE-9423
> URL: https://issues.apache.org/jira/browse/HIVE-9423
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0
>Reporter: Vaibhav Gumashta
>Assignee: Peter Vary
> Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch,
> HIVE-9423.patch
>
>
> An example of where it is needed: it has been reported that when # of client
> connections is greater than {{hive.server2.thrift.max.worker.threads}},
> HiveServer2 stops accepting new connections and ends up having to be
> restarted. This should be handled more gracefully by the server and the JDBC
> driver, so that the end user gets aware of the problem and can take
> appropriate steps (either close existing connections or bump of the config
> value or use multiple server instances with dynamic service discovery
> enabled). Similarly, we should also review the behaviour of background thread
> pool to have a well defined behavior on the the pool getting exhausted.
> Ideally implementing some form of general admission control will be a better
> solution, so that we do not accept new work unless sufficient resources are
> available and display graceful degradation under overload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514219#comment-15514219
]
Lefty Leverenz commented on HIVE-9423:
--
+1 for the new error messages (patch 4)
> HiveServer2: Provide the user with different error messages depending on the
> Thrift client exception code
> -
>
> Key: HIVE-9423
> URL: https://issues.apache.org/jira/browse/HIVE-9423
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0
>Reporter: Vaibhav Gumashta
>Assignee: Peter Vary
> Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch,
> HIVE-9423.patch
>
>
> An example of where it is needed: it has been reported that when # of client
> connections is greater than {{hive.server2.thrift.max.worker.threads}},
> HiveServer2 stops accepting new connections and ends up having to be
> restarted. This should be handled more gracefully by the server and the JDBC
> driver, so that the end user gets aware of the problem and can take
> appropriate steps (either close existing connections or bump of the config
> value or use multiple server instances with dynamic service discovery
> enabled). Similarly, we should also review the behaviour of background thread
> pool to have a well defined behavior on the the pool getting exhausted.
> Ideally implementing some form of general admission control will be a better
> solution, so that we do not accept new work unless sufficient resources are
> available and display graceful degradation under overload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513905#comment-15513905
]
Hive QA commented on HIVE-9423:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12829872/HIVE-9423.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10556 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1274/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1274/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1274/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12829872 - PreCommit-HIVE-Build
> HiveServer2: Provide the user with different error messages depending on the
> Thrift client exception code
> -
>
> Key: HIVE-9423
> URL: https://issues.apache.org/jira/browse/HIVE-9423
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0
>Reporter: Vaibhav Gumashta
>Assignee: Peter Vary
> Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch,
> HIVE-9423.patch
>
>
> An example of where it is needed: it has been reported that when # of client
> connections is greater than {{hive.server2.thrift.max.worker.threads}},
> HiveServer2 stops accepting new connections and ends up having to be
> restarted. This should be handled more gracefully by the server and the JDBC
> driver, so that the end user gets aware of the problem and can take
> appropriate steps (either close existing connections or bump of the config
> value or use multiple server instances with dynamic service discovery
> enabled). Similarly, we should also review the behaviour of background thread
> pool to have a well defined behavior on the the pool getting exhausted.
> Ideally implementing some form of general admission control will be a better
> solution, so that we do not accept new work unless sufficient resources are
> available and display graceful degradation under overload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513653#comment-15513653
]
Vihang Karajgaonkar commented on HIVE-9423:
---
Thanks for the patch [~pvary]. This issue has been a pain point for beeline
users and more user-friendly error messages helps a lot. Patch looks good to me.
> HiveServer2: Provide the user with different error messages depending on the
> Thrift client exception code
> -
>
> Key: HIVE-9423
> URL: https://issues.apache.org/jira/browse/HIVE-9423
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0
>Reporter: Vaibhav Gumashta
>Assignee: Peter Vary
> Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch,
> HIVE-9423.patch
>
>
> An example of where it is needed: it has been reported that when # of client
> connections is greater than {{hive.server2.thrift.max.worker.threads}},
> HiveServer2 stops accepting new connections and ends up having to be
> restarted. This should be handled more gracefully by the server and the JDBC
> driver, so that the end user gets aware of the problem and can take
> appropriate steps (either close existing connections or bump of the config
> value or use multiple server instances with dynamic service discovery
> enabled). Similarly, we should also review the behaviour of background thread
> pool to have a well defined behavior on the the pool getting exhausted.
> Ideally implementing some form of general admission control will be a better
> solution, so that we do not accept new work unless sufficient resources are
> available and display graceful degradation under overload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
