Quanlong Huang created IMPALA-9703:
--------------------------------------
Summary: Skip loading partition meta and file meta for PB scale
tables
Key: IMPALA-9703
URL: https://issues.apache.org/jira/browse/IMPALA-9703
Project: IMPALA
Issue Type: New Feature
Components: Catalog
Reporter: Quanlong Huang
PB scale tables that have >100K partitions may hit catalog limitations. Caching
all the partitions is also a waste since usually only few of them are required.
Queries scanning all partitions probably fail with resource limitation errors
so it's not in our consideration.
This JIRA tracks the work to skip caching partition meta of a table. Catalogd
will only cache the HmsTable object and partition list (partition names, e.g.
"p1=a/p2=b" and internal partition ids generated by Impala). Coordinators fetch
the partition meta on-demand when compiling queries.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
