[jira] [Commented] (IMPALA-13034) Add logs for slow HTTP requests dumping the profile
[ https://issues.apache.org/jira/browse/IMPALA-13034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849073#comment-17849073 ] ASF subversion and git services commented on IMPALA-13034: -- Commit b975165a0acfe37af302dd7c007360633df54917 in impala's branch refs/heads/master from stiga-huang [ https://gitbox.apache.org/repos/asf?p=impala.git;h=b975165a0 ] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches There are several endpoints in WebUI that can dump a query profile: /query_profile, /query_profile_encoded, /query_profile_plain_text, /query_profile_json. The HTTP handler thread goes into ImpalaServer::GetRuntimeProfileOutput() which acquires lock of the ClientRequestState. This could block client requests in fetching query results. To help identify this issue, this patch adds warning logs when such profile dumping requests run slow and the query is still in-flight. Also adds a profile counter, GetInFlightProfileTimeStats, for the summary stats of this time. Dumping the profiles after the query is archived (e.g. closed) won't be tracked. Logs for slow http responses are also added. The thresholds are defined by two new flags, slow_profile_dump_warning_threshold_ms, and slow_http_response_warning_threshold_ms. Note that dumping the profile in-flight won't always block the query, e.g. if there are no client fetch requests or if the coordinator fragment is idle waiting for executor fragment instances. So a long time shown in GetInFlightProfileTimeStats doesn't mean it's hitting the issue. To better identify this issue, this patch adds another profile counter, ClientFetchLockWaitTimer, as the cumulative time client fetch requests waiting for locks. Also fixes false positive logs for complaining invalid query handles. Such logs are added in GetQueryHandle() when the query is not found in the active query map, but it could still exist in the query log. This removes the logs in GetQueryHandle() and lets the callers decide whether to log the error. Tests: - Added e2e test - Ran CORE tests Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc Reviewed-on: http://gerrit.cloudera.org:8080/21412 Reviewed-by: Impala Public Jenkins Tested-by: Michael Smith > Add logs for slow HTTP requests dumping the profile > --- > > Key: IMPALA-13034 > URL: https://issues.apache.org/jira/browse/IMPALA-13034 > Project: IMPALA > Issue Type: Bug > Components: Backend >Reporter: Quanlong Huang >Assignee: Quanlong Huang >Priority: Critical > Fix For: Impala 4.5.0 > > > There are several endpoints in WebUI that can dump a query profile: > /query_profile, /query_profile_encoded, /query_profile_plain_text, > /query_profile_json > The HTTP handler thread goes into ImpalaServer::GetRuntimeProfileOutput() > which acquires lock of the ClientRequestState. This could blocks client > requests in fetching query results. We should add warning logs when such HTTP > requests run slow (e.g. when the profile is too large to download in a short > time). IP address and other info of such requests should also be logged. > Related codes: > https://github.com/apache/impala/blob/f620e5d5c0bbdb0fd97bac31c7b7439cd13c6d08/be/src/service/impala-server.cc#L736 > https://github.com/apache/impala/blob/f620e5d5c0bbdb0fd97bac31c7b7439cd13c6d08/be/src/service/impala-beeswax-server.cc#L601 > https://github.com/apache/impala/blob/f620e5d5c0bbdb0fd97bac31c7b7439cd13c6d08/be/src/service/impala-hs2-server.cc#L207 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-13034) Add logs for slow HTTP requests dumping the profile
[ https://issues.apache.org/jira/browse/IMPALA-13034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844655#comment-17844655 ] Quanlong Huang commented on IMPALA-13034: - Uploaded a patch to add logs and counters first: [https://gerrit.cloudera.org/c/21412/] With that we can identify the issue and find users that abuse the HTTP requests. Filed IMPALA-13063 for a fix for the issue. We can discuss the solutions there. > Add logs for slow HTTP requests dumping the profile > --- > > Key: IMPALA-13034 > URL: https://issues.apache.org/jira/browse/IMPALA-13034 > Project: IMPALA > Issue Type: Bug > Components: Backend >Reporter: Quanlong Huang >Assignee: Quanlong Huang >Priority: Critical > > There are several endpoints in WebUI that can dump a query profile: > /query_profile, /query_profile_encoded, /query_profile_plain_text, > /query_profile_json > The HTTP handler thread goes into ImpalaServer::GetRuntimeProfileOutput() > which acquires lock of the ClientRequestState. This could blocks client > requests in fetching query results. We should add warning logs when such HTTP > requests run slow (e.g. when the profile is too large to download in a short > time). IP address and other info of such requests should also be logged. > Related codes: > https://github.com/apache/impala/blob/f620e5d5c0bbdb0fd97bac31c7b7439cd13c6d08/be/src/service/impala-server.cc#L736 > https://github.com/apache/impala/blob/f620e5d5c0bbdb0fd97bac31c7b7439cd13c6d08/be/src/service/impala-beeswax-server.cc#L601 > https://github.com/apache/impala/blob/f620e5d5c0bbdb0fd97bac31c7b7439cd13c6d08/be/src/service/impala-hs2-server.cc#L207 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-13034) Add logs for slow HTTP requests dumping the profile
[ https://issues.apache.org/jira/browse/IMPALA-13034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842406#comment-17842406 ] Kurt Deschler commented on IMPALA-13034: Changing the locking may be difficult since fetching updates the profile. I would suggest instead introducing a mechanism to prevent GetRuntimeProfileOutput() from monopolizing the lock time by making the request sleep internally when backoff is necessary. > Add logs for slow HTTP requests dumping the profile > --- > > Key: IMPALA-13034 > URL: https://issues.apache.org/jira/browse/IMPALA-13034 > Project: IMPALA > Issue Type: Bug > Components: Backend >Reporter: Quanlong Huang >Priority: Critical > > There are several endpoints in WebUI that can dump a query profile: > /query_profile, /query_profile_encoded, /query_profile_plain_text, > /query_profile_json > The HTTP handler thread goes into ImpalaServer::GetRuntimeProfileOutput() > which acquires lock of the ClientRequestState. This could blocks client > requests in fetching query results. We should add warning logs when such HTTP > requests run slow (e.g. when the profile is too large to download in a short > time). IP address and other info of such requests should also be logged. > Related codes: > https://github.com/apache/impala/blob/f620e5d5c0bbdb0fd97bac31c7b7439cd13c6d08/be/src/service/impala-server.cc#L736 > https://github.com/apache/impala/blob/f620e5d5c0bbdb0fd97bac31c7b7439cd13c6d08/be/src/service/impala-beeswax-server.cc#L601 > https://github.com/apache/impala/blob/f620e5d5c0bbdb0fd97bac31c7b7439cd13c6d08/be/src/service/impala-hs2-server.cc#L207 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-13034) Add logs for slow HTTP requests dumping the profile
[ https://issues.apache.org/jira/browse/IMPALA-13034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840598#comment-17840598 ] Quanlong Huang commented on IMPALA-13034: - Yeah, IMPALA-9380 just helps the part of finalizing (unregistering) a query. The HTTP requests come when the query is still running. We need a lock to protect such read requests from concurrent modification on the profile. Probably we can add a more fine-grained lock just for reading/writing the profile and don't block client fetching query results. > Add logs for slow HTTP requests dumping the profile > --- > > Key: IMPALA-13034 > URL: https://issues.apache.org/jira/browse/IMPALA-13034 > Project: IMPALA > Issue Type: Bug > Components: Backend >Reporter: Quanlong Huang >Priority: Critical > > There are several endpoints in WebUI that can dump a query profile: > /query_profile, /query_profile_encoded, /query_profile_plain_text, > /query_profile_json > The HTTP handler thread goes into ImpalaServer::GetRuntimeProfileOutput() > which acquires lock of the ClientRequestState. This could blocks client > requests in fetching query results. We should add warning logs when such HTTP > requests run slow (e.g. when the profile is too large to download in a short > time). IP address and other info of such requests should also be logged. > Related codes: > https://github.com/apache/impala/blob/f620e5d5c0bbdb0fd97bac31c7b7439cd13c6d08/be/src/service/impala-server.cc#L736 > https://github.com/apache/impala/blob/f620e5d5c0bbdb0fd97bac31c7b7439cd13c6d08/be/src/service/impala-beeswax-server.cc#L601 > https://github.com/apache/impala/blob/f620e5d5c0bbdb0fd97bac31c7b7439cd13c6d08/be/src/service/impala-hs2-server.cc#L207 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-13034) Add logs for slow HTTP requests dumping the profile
[ https://issues.apache.org/jira/browse/IMPALA-13034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840523#comment-17840523 ] Michael Smith commented on IMPALA-13034: Oh, we still take a handle lock in GetRuntimeProfileOutput. That's not great. > Add logs for slow HTTP requests dumping the profile > --- > > Key: IMPALA-13034 > URL: https://issues.apache.org/jira/browse/IMPALA-13034 > Project: IMPALA > Issue Type: Bug > Components: Backend >Reporter: Quanlong Huang >Priority: Critical > > There are several endpoints in WebUI that can dump a query profile: > /query_profile, /query_profile_encoded, /query_profile_plain_text, > /query_profile_json > The HTTP handler thread goes into ImpalaServer::GetRuntimeProfileOutput() > which acquires lock of the ClientRequestState. This could blocks client > requests in fetching query results. We should add warning logs when such HTTP > requests run slow (e.g. when the profile is too large to download in a short > time). IP address and other info of such requests should also be logged. > Related codes: > https://github.com/apache/impala/blob/f620e5d5c0bbdb0fd97bac31c7b7439cd13c6d08/be/src/service/impala-server.cc#L736 > https://github.com/apache/impala/blob/f620e5d5c0bbdb0fd97bac31c7b7439cd13c6d08/be/src/service/impala-beeswax-server.cc#L601 > https://github.com/apache/impala/blob/f620e5d5c0bbdb0fd97bac31c7b7439cd13c6d08/be/src/service/impala-hs2-server.cc#L207 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-13034) Add logs for slow HTTP requests dumping the profile
[ https://issues.apache.org/jira/browse/IMPALA-13034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840521#comment-17840521 ] Michael Smith commented on IMPALA-13034: This no longer requires a lock as of IMPALA-9380. I suspect logging has also been improved since Impala 3.2. > Add logs for slow HTTP requests dumping the profile > --- > > Key: IMPALA-13034 > URL: https://issues.apache.org/jira/browse/IMPALA-13034 > Project: IMPALA > Issue Type: Bug > Components: Backend >Reporter: Quanlong Huang >Priority: Critical > > There are several endpoints in WebUI that can dump a query profile: > /query_profile, /query_profile_encoded, /query_profile_plain_text, > /query_profile_json > The HTTP handler thread goes into ImpalaServer::GetRuntimeProfileOutput() > which acquires lock of the ClientRequestState. This could blocks client > requests in fetching query results. We should add warning logs when such HTTP > requests run slow (e.g. when the profile is too large to download in a short > time). IP address and other info of such requests should also be logged. > Related codes: > https://github.com/apache/impala/blob/f620e5d5c0bbdb0fd97bac31c7b7439cd13c6d08/be/src/service/impala-server.cc#L736 > https://github.com/apache/impala/blob/f620e5d5c0bbdb0fd97bac31c7b7439cd13c6d08/be/src/service/impala-beeswax-server.cc#L601 > https://github.com/apache/impala/blob/f620e5d5c0bbdb0fd97bac31c7b7439cd13c6d08/be/src/service/impala-hs2-server.cc#L207 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
