date:20240402

[Impala-ASF-CR] IMPALA-12602: Unregister queries on idle timeout

2024-04-02 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/21074 )

Change subject: IMPALA-12602: Unregister queries on idle timeout
..

IMPALA-12602: Unregister queries on idle timeout

Queries cancelled due to idle_query_timeout/QUERY_TIMEOUT_S are now also
Unregistered to free any remaining memory, as you cannot fetch results
from a cancelled query.

Adds a new structure - idle_query_statuses_ - to retain Status messages
for queries closed this way so that we can continue to return a clear
error message if the client returns and requests query status or
attempts to fetch results. This structure must be global because HS2
server can only identify a session ID from a query handle, and the query
handle no longer exists. SessionState tracks queries added to
idle_query_statuses_ so they can be cleared when the session is closed.

Also ensures MarkInactive is called in ClientRequestState when Wait()
completes. Previously WaitInternal would only MarkInactive on success,
leaving any failed requests in an active state until explicitly closed
or the session ended.

The beeswax get_log RPC will not return the preserved error message or
any warnings for these queries. It's also possible the summary and
profile are rotated out of query log as the query is no longer inflight.
This is an acceptable outcome as a client will likely not look for a
log/summary/profile after it times out.

Testing:
- updates test_query_expiration to verify number of waiting queries is
  only non-zero for queries cancelled by EXEC_TIME_LIMIT_S and not yet
  closed as an idle query
- modified test_retry_query_timeout to use exec_time_limit_s because
  queries closed by idle_timeout_s don't work with get_exec_summary

Change-Id: Iacfc285ed3587892c7ec6f7df3b5f71c9e41baf0
Reviewed-on: http://gerrit.cloudera.org:8080/21074
Reviewed-by: Michael Smith 
Tested-by: Impala Public Jenkins 
---
M be/src/service/client-request-state.cc
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M docs/topics/impala_timeouts.xml
M tests/custom_cluster/test_query_expiration.py
M tests/custom_cluster/test_query_retries.py
7 files changed, 140 insertions(+), 52 deletions(-)

Approvals:
  Michael Smith: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/21074
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Iacfc285ed3587892c7ec6f7df3b5f71c9e41baf0
Gerrit-Change-Number: 21074
Gerrit-PatchSet: 18
Gerrit-Owner: Michael Smith 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Jason Fehr 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Riza Suminto

[Impala-ASF-CR] IMPALA-12602: Unregister queries on idle timeout

2024-04-02 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21074 )

Change subject: IMPALA-12602: Unregister queries on idle timeout
..


Patch Set 17: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/21074
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iacfc285ed3587892c7ec6f7df3b5f71c9e41baf0
Gerrit-Change-Number: 21074
Gerrit-PatchSet: 17
Gerrit-Owner: Michael Smith 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Jason Fehr 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Comment-Date: Wed, 03 Apr 2024 03:25:09 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12291 impala checks hdfs ranger policy

2024-04-02 Thread Halim Kim (Code Review)

Halim Kim has posted comments on this change. (
http://gerrit.cloudera.org:8080/20221 )

Change subject: IMPALA-12291 impala checks hdfs ranger policy
..

Patch Set 9:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/20221/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:

http://gerrit.cloudera.org:8080/#/c/20221/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@852
PS5, Line 852: location = location.getParent();
> Thanks for the suggestion Quanlong!
Your test code will be very helpful. Thank you Fang-Yu.

I agree with the idea to hdfs permission check in query analysis when ranger is
enabled. however, It seems like query planning gets less thorough just because
ranger is enabled.

how about adding a flag to decide whether hdfs permission should be checked or
not like --hdfs_permission_check=true (or false)?

I think it is more reasonable to give user a choice. In other words, Asking
user if you want a AnalysisException or Runtime Exception.

Of course, we should add guidance to use --hdfs_permission_check flag.
Recommend to disable it when ranger is enabled.

http://gerrit.cloudera.org:8080/#/c/20221/5/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:

http://gerrit.cloudera.org:8080/#/c/20221/5/fe/src/main/java/org/apache/impala/service/Frontend.java@2109
PS5, Line 2109: LOG.warn("Analysis Exception query {}: {}",
: queryCtx.client_request.stmt, errorMsg);
> Not very sure if I missed something. Since we will throw the exception that
In retrospect, I think I found it difficult to know which query is in trouble
so that I put one more catch statement to catch the query.
But now, I realize that there is already AnalysisException message like below.

```
AnalysisException: Unable to INSERT into target table (testdb.testtable)
because Impala does not have WRITE access to HDFS location:
hdfs://testcluster/warehouse/testdb.db/testtable
```

To conclude, It is not necessary to catch like that because we can find which
query has permission issue by an AnalysisException thrown.

Thank you for commenting.

--
To view, visit http://gerrit.cloudera.org:8080/20221
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9
Gerrit-Change-Number: 20221
Gerrit-PatchSet: 9
Gerrit-Owner: Halim Kim
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Halim Kim
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Wed, 03 Apr 2024 02:53:02 +
Gerrit-HasComments: Yes

79 matches

Mail list logo