Quanlong Huang created IMPALA-13867:
---------------------------------------
Summary: Improves logging of GetTables on all dbs
Key: IMPALA-13867
URL: https://issues.apache.org/jira/browse/IMPALA-13867
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Reporter: Quanlong Huang
Assignee: Quanlong Huang
When authorization is enabled, GetTables or SHOW TABLES requests will check
whether a table is visible to the user before adding the table to the result
list. This requires checking the ownership info. However, when the table is
unloaded in catalog (i.e. only the table name is known), the ownership info is
missing. Coordinator will log these:
{noformat}
Owner of table_name is unknown due to msTable is unloaded
Table table_name not yet loaded, ignoring it in table listing.
{noformat}
Note that the last log is misleading. The table might still be returned if
privilege checks allowed it, e.g. when user has ALL privilege on SERVER. We
should only add these logs when the table is unloaded and actually ignored.
On the other hand, when there are lots of concurrent global GetTables requests
(i.e. on all dbs) and they have a huge result list (e.g. 1M tables), the
request could be blocked in logging:
{noformat}
"Thread-6985" Id=31986 BLOCKED on org.apache.log4j.spi.RootLogger@6bd98257
owned by "Thread-6968" Id=31969
at org.apache.log4j.Category.callAppenders(Category.java:204)
at org.apache.log4j.Category.forcedLog(Category.java:384)
at org.apache.log4j.Category.log(Category.java:810)
at
org.slf4j.reload4j.Reload4jLoggerAdapter.handleNormalizedLoggingCall(Reload4jLoggerAdapter.java:137)
at
org.slf4j.helpers.AbstractLogger.handle_1ArgsCall(AbstractLogger.java:386)
at org.slf4j.helpers.AbstractLogger.warn(AbstractLogger.java:269)
at
org.apache.impala.catalog.local.LocalTable.getOwnerUser(LocalTable.java:234) //
LOG.warn("Owner of {} is unknown due to msTable is unloaded"
at
org.apache.impala.catalog.local.LocalIncompleteTable.getOwnerUser(LocalIncompleteTable.java:34)
at
org.apache.impala.service.Frontend.filterTablesIfAuthNeeded(Frontend.java:1251)
at
org.apache.impala.service.Frontend.doGetCatalogTableNames(Frontend.java:1269)
at org.apache.impala.service.Frontend.getTableNames(Frontend.java:1181)
at org.apache.impala.service.Frontend.getTableNames(Frontend.java:1168)
at
org.apache.impala.service.MetadataOp.doGetDbsMetadata(MetadataOp.java:367)
at
org.apache.impala.service.MetadataOp.getDbsMetadata(MetadataOp.java:332)
at org.apache.impala.service.MetadataOp.getTables(MetadataOp.java:605)
at
org.apache.impala.catalog.Hive3MetastoreShimBase.execGetTables(Hive3MetastoreShimBase.java:243)
at
org.apache.impala.service.Frontend.execHiveServer2MetadataOp(Frontend.java:2947)
at
org.apache.impala.service.JniFrontend.execHiveServer2MetadataOp(JniFrontend.java:633){noformat}
Logging too many warnings for each unloaded tables might not be useful. We
should limit the logs, e.g. only for the first several tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
