[
https://issues.apache.org/jira/browse/HIVE-26893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala reassigned HIVE-26893:
--------------------------------------------
Assignee: Sai Hemanth Gantasala
> Extend batch partition APIs to ignore partition schemas
> -------------------------------------------------------
>
> Key: HIVE-26893
> URL: https://issues.apache.org/jira/browse/HIVE-26893
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
> Reporter: Quanlong Huang
> Assignee: Sai Hemanth Gantasala
> Priority: Major
>
> There are several HMS APIs that return a list of partitions, e.g.
> get_partitions_ps(), get_partitions_by_names(), add_partitions_req() with
> needResult=true, etc. Each partition instance will have a unique list of
> FieldSchemas as the partition schema:
> {code:java}
> org.apache.hadoop.hive.metastore.api.Partition
> -> org.apache.hadoop.hive.metastore.api.StorageDescriptor
> -> cols: list<org.apache.hadoop.hive.metastore.api.FieldSchema> {code}
> This could occupy a large memory footprint for wide tables (e.g. with 2k
> cols). See the heap histogram in IMPALA-11812 as an example.
> Some engines like Impala doesn't actually use/respect the partition level
> schema. It's a waste of network/serde resource to transmit them. It'd be nice
> if these APIs provide an optional boolean flag for ignoring partition
> schemas. So HMS clients (e.g. Impala) don't need to clear them later (to save
> mem).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
