[ https://issues.apache.org/jira/browse/HIVE-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-4322: ------------------------------ Attachment: HIVE-4322.HIVE-4322.HIVE-4322.HIVE-4322.D10203.1.patch sxyuan requested code review of "HIVE-4322 [jira] SkewedInfo in Metastore Thrift API cannot be deserialized in Python". Reviewers: gangtimliu See JIRA for description. TEST PLAN Ran both hadoop20 and hadoop23 tests. REVISION DETAIL https://reviews.facebook.net/D10203 AFFECTED FILES service/src/gen/thrift/gen-php/hive_service service/src/gen/thrift/gen-php/hive_service/ThriftHive.php service/src/gen/thrift/gen-php/hive_service/hive_service_types.php metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb metastore/src/gen/thrift/gen-php/hive_metastore_constants.php metastore/src/gen/thrift/gen-php/metastore/Types.php metastore/src/gen/thrift/gen-php/ThriftHiveMetastore.php metastore/src/gen/thrift/gen-php/hive_metastore_types.php metastore/src/gen/thrift/gen-php/hive_metastore metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedValueList.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java metastore/if/hive_metastore.thrift ql/src/java/org/apache/hadoop/hive/ql/optimizer/listbucketingpruner/ListBucketingPruner.java ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java ql/src/java/org/apache/hadoop/hive/ql/plan/ListBucketingCtx.java ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java ql/src/gen/thrift/gen-php/queryplan ql/src/gen/thrift/gen-php/queryplan/queryplan_types.php MANAGE HERALD RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/24381/ To: gangtimliu, sxyuan Cc: kevinwilfong, JIRA > SkewedInfo in Metastore Thrift API cannot be deserialized in Python > ------------------------------------------------------------------- > > Key: HIVE-4322 > URL: https://issues.apache.org/jira/browse/HIVE-4322 > Project: Hive > Issue Type: Bug > Components: Metastore, Thrift API > Affects Versions: 0.11.0 > Reporter: Samuel Yuan > Assignee: Samuel Yuan > Priority: Minor > Attachments: HIVE-4322.HIVE-4322.HIVE-4322.HIVE-4322.D10203.1.patch > > > The Thrift-generated Python code that deserializes Thrift objects fails > whenever a complex type is used as a map key, because by default mutable > Python objects such as lists do not have a hash function. See > https://issues.apache.org/jira/browse/THRIFT-162 for related discussion. > The SkewedInfo struct contains a map which uses a list as a key, breaking the > Python Thrift interface. It is not possible to specify the mapping from > Thrift types to Python types, or otherwise we could map Thrift lists to > Python tuples. Instead, the proposed workaround wraps the list inside a new > struct. This alone does not accomplish anything, but allows Python clients to > define a hash function for the struct class, e.g.: > def f(object): > return hash(tuple(object.skewedValueList)) > SkewedValueList.__hash__ = f > In practice a more efficient hash might be defined that does not involve > copying the list. The advantage of wrapping the list inside a struct is that > the client does not have to define the hash on the list itself, which would > change the behaviour of lists everywhere else in the code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira