[ https://issues.apache.org/jira/browse/HIVE-27114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Naresh P R updated HIVE-27114: ------------------------------ Description: HMS API calls are throwing following exception because of thrift upgrade {code:java} org.apache.thrift.transport.TTransportException: MaxMessageSize reached at org.apache.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96) at org.apache.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97) at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:390) at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109) at org.apache.hadoop.hive.metastore.security.TFilterTransport.readAll(TFilterTransport.java:63) at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:417) at org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:411) at org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1286) at org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1205) at org.apache.hadoop.hive.metastore.api.Partition.read(Partition.java:1062) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.read(ThriftHiveMetastore.java) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.read(ThriftHiveMetastore.java) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result.read(ThriftHiveMetastore.java) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:88) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partitions(ThriftHiveMetastore.java:3290) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partitions(ThriftHiveMetastore.java:3275) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitions(HiveMetaStoreClient.java:1782) at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.listPartitions(SessionHiveMetaStoreClient.java:1134) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitions(HiveMetaStoreClient.java:1775) at sun.reflect.GeneratedMethodAccessor169.invoke(Unknown Source) ~[?:?] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_311] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_311] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:213) at com.sun.proxy.$Proxy52.listPartitions(Unknown Source) ~[?:?] at sun.reflect.GeneratedMethodAccessor169.invoke(Unknown Source) ~[?:?] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_311] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_311] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:3550) at com.sun.proxy.$Proxy52.listPartitions(Unknown Source) ~[?:?] at org.apache.hadoop.hive.ql.metadata.Hive.getAllPartitionsOf(Hive.java:3793) at org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.getAllPartitions(PartitionPruner.java:485) {code} Large size partition metadata is causing this issue eg., impala stores huge stats chunk in partitionMetadata with {*}param_keys = (impala_intermediate_stats_chunk{*}{*}), these PARTITION_PARAM_KEYS are not required for Hive. These params should be skipped while preparing partition object from HMS to HS2. Similar to HIVE-25501, any user defined regex param_keys should be skipped in listPartitions HMS API call response. was: HMS API calls are throwing following exception because of thrift upgrade {code:java} org.apache.thrift.transport.TTransportException: MaxMessageSize reached at org.apache.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96) at org.apache.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97) at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:390) at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109) at org.apache.hadoop.hive.metastore.security.TFilterTransport.readAll(TFilterTransport.java:63) at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:417) at org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:411) at org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1286) at org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1205) at org.apache.hadoop.hive.metastore.api.Partition.read(Partition.java:1062) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.read(ThriftHiveMetastore.java) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.read(ThriftHiveMetastore.java) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result.read(ThriftHiveMetastore.java) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:88) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partitions(ThriftHiveMetastore.java:3290) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partitions(ThriftHiveMetastore.java:3275) {code} Large size partition metadata is causing this issue eg., impala stores huge stats chunk in partitionMetadata with {*}param_keys = (impala_intermediate_stats_chunk*{*}), these PARTITION_PARAM_KEYS are not required for Hive. These params should be skipped while preparing partition object from HMS to HS2. Similar to HIVE-25501, any user defined regex param_keys should be skipped in getPartitions HMS API call response. > Provide a configurable filter for removing useless properties in Partition > objects from getPartitions HMS Calls > --------------------------------------------------------------------------------------------------------------- > > Key: HIVE-27114 > URL: https://issues.apache.org/jira/browse/HIVE-27114 > Project: Hive > Issue Type: Bug > Reporter: Naresh P R > Priority: Major > > HMS API calls are throwing following exception because of thrift upgrade > {code:java} > org.apache.thrift.transport.TTransportException: MaxMessageSize reached > at > org.apache.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96) > > at > org.apache.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97) > > at > org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:390) > at > org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39) > > at > org.apache.thrift.transport.TTransport.readAll(TTransport.java:109) > at > org.apache.hadoop.hive.metastore.security.TFilterTransport.readAll(TFilterTransport.java:63) > > at > org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:417) > > at > org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:411) > > at > org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1286) > > at > org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1205) > > at > org.apache.hadoop.hive.metastore.api.Partition.read(Partition.java:1062) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.read(ThriftHiveMetastore.java) > > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.read(ThriftHiveMetastore.java) > > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result.read(ThriftHiveMetastore.java) > > at > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:88) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partitions(ThriftHiveMetastore.java:3290) > > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partitions(ThriftHiveMetastore.java:3275) > > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitions(HiveMetaStoreClient.java:1782) > > at > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.listPartitions(SessionHiveMetaStoreClient.java:1134) > > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitions(HiveMetaStoreClient.java:1775) > > at sun.reflect.GeneratedMethodAccessor169.invoke(Unknown Source) > ~[?:?] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[?:1.8.0_311] > at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_311] > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:213) > > at com.sun.proxy.$Proxy52.listPartitions(Unknown Source) ~[?:?] > at sun.reflect.GeneratedMethodAccessor169.invoke(Unknown Source) > ~[?:?] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[?:1.8.0_311] > at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_311] > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:3550) > > at com.sun.proxy.$Proxy52.listPartitions(Unknown Source) ~[?:?] > at > org.apache.hadoop.hive.ql.metadata.Hive.getAllPartitionsOf(Hive.java:3793) > at > org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.getAllPartitions(PartitionPruner.java:485) > {code} > Large size partition metadata is causing this issue > eg., impala stores huge stats chunk in partitionMetadata with {*}param_keys = > (impala_intermediate_stats_chunk{*}{*}), these PARTITION_PARAM_KEYS are not > required for Hive. These params should be skipped while preparing partition > object from HMS to HS2. > Similar to HIVE-25501, any user defined regex param_keys should be skipped in > listPartitions HMS API call response. -- This message was sent by Atlassian Jira (v8.20.10#820010)