Naresh P R created HIVE-27114:
---------------------------------
Summary: Provide a configurable filter for removing useless
properties from PartitionDesc objects from getPartitions HMS Calls
Key: HIVE-27114
URL: https://issues.apache.org/jira/browse/HIVE-27114
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
HMS API calls are throwing following exception because of thrift upgrade
{code:java}
org.apache.thrift.transport.TTransportException: MaxMessageSize reached
at
org.apache.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96)
at
org.apache.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97)
at
org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:390)
at
org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
at
org.apache.hadoop.hive.metastore.security.TFilterTransport.readAll(TFilterTransport.java:63)
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:417)
at
org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:411)
at
org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1286)
at
org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1205)
at
org.apache.hadoop.hive.metastore.api.Partition.read(Partition.java:1062)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result.read(ThriftHiveMetastore.java)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:88)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partitions(ThriftHiveMetastore.java:3290)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partitions(ThriftHiveMetastore.java:3275)
{code}
Large size partition metadata is causing this issue
eg., impala stores huge stats chunk in partitionMetadata with {*}param_keys =
(impala_intermediate_stats_chunk*){*}, these PARTITION_PARAM is not required
for Hive. These params should be skipped while preparing partition object from
HMS to HS2.
Similarly any user defined regex should be skipped in getPartitions HMS API
call. Similar to HIVE-25501
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
