[
https://issues.apache.org/jira/browse/HIVE-27114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naresh P R updated HIVE-27114:
------------------------------
Description:
HMS API calls are throwing following exception because of thrift upgrade
{code:java}
org.apache.thrift.transport.TTransportException: MaxMessageSize reached
at
org.apache.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96)
at
org.apache.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97)
at
org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:390)
at
org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
at
org.apache.hadoop.hive.metastore.security.TFilterTransport.readAll(TFilterTransport.java:63)
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:417)
at
org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:411)
at
org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1286)
at
org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1205)
at
org.apache.hadoop.hive.metastore.api.Partition.read(Partition.java:1062)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result.read(ThriftHiveMetastore.java)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:88)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partitions(ThriftHiveMetastore.java:3290)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partitions(ThriftHiveMetastore.java:3275)
{code}
Large size partition metadata is causing this issue
eg., impala stores huge stats chunk in partitionMetadata with {*}param_keys =
(impala_intermediate_stats_chunk*{*}), these PARTITION_PARAM_KEYS are not
required for Hive. These params should be skipped while preparing partition
object from HMS to HS2.
Similar to HIVE-25501, any user defined regex param_keys should be skipped in
getPartitions HMS API call response.
was:
HMS API calls are throwing following exception because of thrift upgrade
{code:java}
org.apache.thrift.transport.TTransportException: MaxMessageSize reached
at
org.apache.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96)
at
org.apache.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97)
at
org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:390)
at
org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
at
org.apache.hadoop.hive.metastore.security.TFilterTransport.readAll(TFilterTransport.java:63)
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:417)
at
org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:411)
at
org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1286)
at
org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1205)
at
org.apache.hadoop.hive.metastore.api.Partition.read(Partition.java:1062)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result.read(ThriftHiveMetastore.java)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:88)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partitions(ThriftHiveMetastore.java:3290)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partitions(ThriftHiveMetastore.java:3275)
{code}
Large size partition metadata is causing this issue
eg., impala stores huge stats chunk in partitionMetadata with {*}param_keys =
(impala_intermediate_stats_chunk*){*}, these PARTITION_PARAM is not required
for Hive. These params should be skipped while preparing partition object from
HMS to HS2.
Similarly any user defined regex should be skipped in getPartitions HMS API
call. Similar to HIVE-25501
> Provide a configurable filter for removing useless properties in Partition
> objects from getPartitions HMS Calls
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-27114
> URL: https://issues.apache.org/jira/browse/HIVE-27114
> Project: Hive
> Issue Type: Bug
> Reporter: Naresh P R
> Priority: Major
>
> HMS API calls are throwing following exception because of thrift upgrade
> {code:java}
> org.apache.thrift.transport.TTransportException: MaxMessageSize reached
> at
> org.apache.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96)
>
> at
> org.apache.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97)
>
> at
> org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:390)
> at
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39)
>
> at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
> at
> org.apache.hadoop.hive.metastore.security.TFilterTransport.readAll(TFilterTransport.java:63)
>
> at
> org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:417)
>
> at
> org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:411)
>
> at
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1286)
>
> at
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1205)
>
> at
> org.apache.hadoop.hive.metastore.api.Partition.read(Partition.java:1062)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.read(ThriftHiveMetastore.java)
>
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.read(ThriftHiveMetastore.java)
>
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result.read(ThriftHiveMetastore.java)
>
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:88)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partitions(ThriftHiveMetastore.java:3290)
>
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partitions(ThriftHiveMetastore.java:3275)
> {code}
> Large size partition metadata is causing this issue
> eg., impala stores huge stats chunk in partitionMetadata with {*}param_keys =
> (impala_intermediate_stats_chunk*{*}), these PARTITION_PARAM_KEYS are not
> required for Hive. These params should be skipped while preparing partition
> object from HMS to HS2.
> Similar to HIVE-25501, any user defined regex param_keys should be skipped in
> getPartitions HMS API call response.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
