Hello Riza Suminto, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23558
to look at the new patch set (#8).
Change subject: IMPALA-12870: Tag query id for Java pool threads
......................................................................
IMPALA-12870: Tag query id for Java pool threads
Logs from Java threads running in ExecutorService are missing the query
id which is stored in the C++ thread-local ThreadDebugInfo variable.
This patch adds JNI calls for Java threads to manage the ThreadDebugInfo
variable. Currently two thread pools are changed:
- MissingTable loading pool in StmtMetadataLoader.parallelTableLoad().
- Table loading pool in TableLoadingMgr.
MissingTable loading pool only lives within the parallelTableLoad()
method. So we initialize ThreadDebugInfo with the queryId at the
beginning of the thread and delete it at the end of the thread. Note
that a thread might be reused to load different tables, but they all
belong to the same query.
Table loading pool is a long running pool in catalogd that never
shut down. Threads in it is used to load tables triggered by different
queries. We initialize ThreadDebugInfo as the above but update it when
the thread starts loading table for a different query id, and reset it
when the loading is done. The query id is passed down from the catalogd
RPC request headers.
Tests:
- Added e2e test to verify the logs.
- Ran existing CORE tests.
Change-Id: I83cca55edc72de35f5e8c5422efc104e6aa894c1
---
M be/src/service/fe-support.cc
M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/TableLoadingMgr.java
M
fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java
A fe/src/main/java/org/apache/impala/common/TaggedThreadFactory.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/FeSupport.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
M
fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java
M tests/custom_cluster/test_observability.py
11 files changed, 289 insertions(+), 59 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/23558/8
--
To view, visit http://gerrit.cloudera.org:8080/23558
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I83cca55edc72de35f5e8c5422efc104e6aa894c1
Gerrit-Change-Number: 23558
Gerrit-PatchSet: 8
Gerrit-Owner: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Riza Suminto <[email protected]>
