[
https://issues.apache.org/jira/browse/HIVE-26024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Samrat Deb updated HIVE-26024:
------------------------------
Summary: Metastore RDS CPU spike in ALTER TABLE COMMAND (was: Metastore
RDS CPU spike in DDL commands ())
> Metastore RDS CPU spike in ALTER TABLE COMMAND
> ----------------------------------------------
>
> Key: HIVE-26024
> URL: https://issues.apache.org/jira/browse/HIVE-26024
> Project: Hive
> Issue Type: Bug
> Reporter: Samrat Deb
> Priority: Major
>
> We are experiencing sudden spike in the cpu utilization of hive-metastore
> rds, whenever there is `ALTER TABLE ` queries in hive.
> There is sudden spike in the rds cpu (attached screenshot )
> This leads to non-responsive metastore service , and effects other
> jobs/queries
> Meta Information
> -> Hive version - 3.1.2
> -> Hive metastore version - 3.1.2
> -> Hadoop version - 3.2.2
> -> DB used for metastore - Postgres 42.2.18
> Observations :-
> 1) On running DDL queries like "ALTER TABLE " commands it fails in around
> 120s approx with the error message
> ```while processing statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. null```
> though it has returned error , we observed that
> even if the query has returned an error, the query still keeps running on the
> metastore via querying pg_stat_activity table .
> 2) the spike in metastore CPU is mostly related to partition stats
> collection, Tried running the queries via a hive cli in debug mode and
> noticed these logs when the CPU was spiking
> logs
> ```
> 2022-02-21T08:33:22,876 DEBUG [<object_id> main] metastore.ObjectStore:
> Open transaction: count = 3, isActive = true at:
>
> org.apache.hadoop.hive.metastore.ObjectStore.removeUnusedColumnDescriptor(ObjectStore.java:4232)
>
> org.apache.hadoop.hive.metastore.ObjectStore.alterPartition(ObjectStore.java:4129)
> sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
> 2022-02-21T08:33:22,876 DEBUG [<object_id> main] metastore.ObjectStore:
> execute removeUnusedColumnDescriptor
> 2022-02-21T08:33:24,482 DEBUG [70618618-a5a0-497a-abc3-6aacf390024e main]
> metastore.ObjectStore: Commit transaction: count = 2, isactive true at:
>
> org.apache.hadoop.hive.metastore.ObjectStore.removeUnusedColumnDescriptor(ObjectStore.java:4245)
>
> org.apache.hadoop.hive.metastore.ObjectStore.alterPartition(ObjectStore.java:4129)
> sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
> 2022-02-21T08:33:24,482 DEBUG [<object_id> main] metastore.ObjectStore:
> successfully deleted a CD in removeUnusedColumnDescriptor
> 2022-02-21T08:33:24,483 DEBUG [<object_id> main] metastore.ObjectStore:
> Commit transaction: count = 1, isactive true at:
>
> org.apache.hadoop.hive.metastore.ObjectStore.alterPartition(ObjectStore.java:4131)
> sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-02-21T08:33:24,483 DEBUG [<object_id> main] metastore.ObjectStore: Open
> transaction: count = 2, isActive = true at:
>
> org.apache.hadoop.hive.metastore.ObjectStore.alterPartition(ObjectStore.java:4127)
> sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-02-21T08:33:24,483 DEBUG [<object_id> main] metastore.ObjectStore: Open
> transaction: count = 3, isActive = true at:```
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
