[ https://issues.apache.org/jira/browse/HIVE-7223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14088926#comment-14088926 ]
Hive QA commented on HIVE-7223: ------------------------------- {color:red}Overall{color}: -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12660271/HIVE-7223.1.patch Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/202/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/202/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-202/ Messages: {noformat} **** This message was trimmed, see log for full details **** [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-metastore: Compilation failure: Compilation failure: [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[117,44] cannot find symbol [ERROR] symbol: class PartitionListComposingSpec [ERROR] location: package org.apache.hadoop.hive.metastore.api [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[118,44] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: package org.apache.hadoop.hive.metastore.api [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[119,44] cannot find symbol [ERROR] symbol: class PartitionSpecWithSharedSD [ERROR] location: package org.apache.hadoop.hive.metastore.api [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[120,44] cannot find symbol [ERROR] symbol: class PartitionWithoutSD [ERROR] location: package org.apache.hadoop.hive.metastore.api [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionSpecProxy.java:[5,44] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: package org.apache.hadoop.hive.metastore.api [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[1990,48] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[2004,58] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[2719,17] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[2779,18] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[2844,93] cannot find symbol [ERROR] symbol: class PartitionWithoutSD [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[2844,13] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[4073,17] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionSpecProxy.java:[55,24] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.partition.spec.PartitionSpecProxy [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:[100,44] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: package org.apache.hadoop.hive.metastore.api [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java:[31,44] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: package org.apache.hadoop.hive.metastore.api [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionListComposingSpecProxy.java:[5,44] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: package org.apache.hadoop.hive.metastore.api [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionListComposingSpecProxy.java:[16,11] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.partition.spec.PartitionListComposingSpecProxy [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionListComposingSpecProxy.java:[18,45] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.partition.spec.PartitionListComposingSpecProxy [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionListComposingSpecProxy.java:[40,15] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.partition.spec.PartitionListComposingSpecProxy [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionSpecWithSharedSDProxy.java:[5,44] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: package org.apache.hadoop.hive.metastore.api [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionSpecWithSharedSDProxy.java:[6,44] cannot find symbol [ERROR] symbol: class PartitionSpecWithSharedSD [ERROR] location: package org.apache.hadoop.hive.metastore.api [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionSpecWithSharedSDProxy.java:[7,44] cannot find symbol [ERROR] symbol: class PartitionWithoutSD [ERROR] location: package org.apache.hadoop.hive.metastore.api [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionSpecWithSharedSDProxy.java:[21,11] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.partition.spec.PartitionSpecWithSharedSDProxy [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionSpecWithSharedSDProxy.java:[23,41] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.partition.spec.PartitionSpecWithSharedSDProxy [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionSpecWithSharedSDProxy.java:[58,15] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.partition.spec.PartitionSpecWithSharedSDProxy [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionSpecWithSharedSDProxy.java:[74,13] cannot find symbol [ERROR] symbol: class PartitionSpecWithSharedSD [ERROR] location: class org.apache.hadoop.hive.metastore.partition.spec.PartitionSpecWithSharedSDProxy.Iterator [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionSpecProxy.java:[74,42] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.partition.spec.PartitionSpecProxy.Factory [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionSpecProxy.java:[97,47] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.partition.spec.PartitionSpecProxy.Factory [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/partition/spec/CompositePartitionSpecProxy.java:[7,44] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: package org.apache.hadoop.hive.metastore.api [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/partition/spec/CompositePartitionSpecProxy.java:[20,16] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.partition.spec.CompositePartitionSpecProxy [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/partition/spec/CompositePartitionSpecProxy.java:[24,46] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.partition.spec.CompositePartitionSpecProxy [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/partition/spec/CompositePartitionSpecProxy.java:[44,79] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.partition.spec.CompositePartitionSpecProxy [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/partition/spec/CompositePartitionSpecProxy.java:[200,15] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.partition.spec.CompositePartitionSpecProxy [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[1989,5] method does not override or implement a method from a supertype [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[2724,12] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[2733,11] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[2733,37] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[2734,38] cannot find symbol [ERROR] symbol: class PartitionListComposingSpec [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[2718,5] method does not override or implement a method from a supertype [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[2795,12] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[2795,53] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[2800,38] cannot find symbol [ERROR] symbol: class PartitionWithoutSD [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[2801,52] cannot find symbol [ERROR] symbol: class PartitionWithoutSD [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[2808,11] cannot find symbol [ERROR] symbol: class PartitionWithoutSD [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[2809,21] cannot find symbol [ERROR] symbol: class PartitionWithoutSD [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[2816,51] cannot find symbol [ERROR] symbol: class PartitionWithoutSD [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[2823,51] cannot find symbol [ERROR] symbol: class PartitionWithoutSD [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[2833,11] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[2833,44] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[2836,45] cannot find symbol [ERROR] symbol: class PartitionListComposingSpec [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[2848,7] cannot find symbol [ERROR] symbol: class PartitionSpecWithSharedSD [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[2849,15] cannot find symbol [ERROR] symbol: class PartitionSpecWithSharedSD [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[2851,7] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[2851,31] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[4079,12] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[4088,11] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[4088,37] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[4089,38] cannot find symbol [ERROR] symbol: class PartitionListComposingSpec [ERROR] location: class org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[4072,5] method does not override or implement a method from a supertype [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:[909,49] cannot find symbol [ERROR] symbol: method get_partitions_pspec(java.lang.String,java.lang.String,int) [ERROR] location: variable client of type org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore.Iface [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:[964,15] cannot find symbol [ERROR] symbol: method get_part_specs_by_filter(java.lang.String,java.lang.String,java.lang.String,int) [ERROR] location: variable client of type org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore.Iface [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionSpecWithSharedSDProxy.java:[102,7] cannot find symbol [ERROR] symbol: class PartitionWithoutSD [ERROR] location: class org.apache.hadoop.hive.metastore.partition.spec.PartitionSpecWithSharedSDProxy.Iterator [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/partition/spec/CompositePartitionSpecProxy.java:[34,12] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.partition.spec.CompositePartitionSpecProxy [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/metastore/src/java/org/apache/hadoop/hive/metastore/partition/spec/CompositePartitionSpecProxy.java:[49,10] cannot find symbol [ERROR] symbol: class PartitionSpec [ERROR] location: class org.apache.hadoop.hive.metastore.partition.spec.CompositePartitionSpecProxy [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :hive-metastore + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12660271 > Support generic PartitionSpecs in Metastore partition-functions > --------------------------------------------------------------- > > Key: HIVE-7223 > URL: https://issues.apache.org/jira/browse/HIVE-7223 > Project: Hive > Issue Type: Improvement > Components: HCatalog, Metastore > Affects Versions: 0.12.0, 0.13.0 > Reporter: Mithun Radhakrishnan > Assignee: Mithun Radhakrishnan > Attachments: HIVE-7223.1.patch > > > Currently, the functions in the HiveMetaStore API that handle multiple > partitions do so using List<Partition>. E.g. > {code} > public List<Partition> listPartitions(String db_name, String tbl_name, short > max_parts); > public List<Partition> listPartitionsByFilter(String db_name, String > tbl_name, String filter, short max_parts); > public int add_partitions(List<Partition> new_parts); > {code} > Partition objects are fairly heavyweight, since each Partition carries its > own copy of a StorageDescriptor, partition-values, etc. Tables with tens of > thousands of partitions take so long to have their partitions listed that the > client times out with default hive.metastore.client.socket.timeout. There is > the additional expense of serializing and deserializing metadata for large > sets of partitions, w.r.t time and heap-space. Reducing the thrift traffic > should help in this regard. > In a date-partitioned table, all sub-partitions for a particular date are > *likely* (but not expected) to have: > # The same base directory (e.g. {{/feeds/search/20140601/}}) > # Similar directory structure (e.g. {{/feeds/search/20140601/[US,UK,IN]}}) > # The same SerDe/StorageHandler/IOFormat classes > # Sorting/Bucketing/SkewInfo settings > In this “most likely” scenario (henceforth termed “normal”), it’s possible to > represent the partition-list (for a date) in a more condensed form: a list of > LighterPartition instances, all sharing a common StorageDescriptor whose > location points to the root directory. > We can go one better for the {{add_partitions()}} case: When adding all > partitions for a given date, the “normal” case affords us the ability to > specify the top-level date-directory, where sub-partitions can be inferred > from the HDFS directory-path. > These extensions are hard to introduce at the metastore-level, since > partition-functions explicitly specify {{List<Partition>}} arguments. I > wonder if a {{PartitionSpec}} interface might help: > {code} > public PartitionSpec listPartitions(db_name, tbl_name, max_parts) throws ... > ; > public int add_partitions( PartitionSpec new_parts ) throws … ; > {code} > where the PartitionSpec looks like: > {code} > public interface PartitionSpec { > public List<Partition> getPartitions(); > public List<String> getPartNames(); > public Iterator<Partition> getPartitionIter(); > public Iterator<String> getPartNameIter(); > } > {code} > For addPartitions(), an {{HDFSDirBasedPartitionSpec}} class could implement > {{PartitionSpec}}, store a top-level directory, and return Partition > instances from sub-directory names, while storing a single StorageDescriptor > for all of them. > Similarly, list_partitions() could return a List<PartitionSpec>, where each > PartitionSpec corresponds to a set or partitions that can share a > StorageDescriptor. > By exposing iterator semantics, neither the client nor the metastore need > instantiate all partitions at once. That should help with memory requirements. > In case no smart grouping is possible, we could just fall back on a > {{DefaultPartitionSpec}} which composes {{List<Partition>}}, and is no worse > than status quo. > PartitionSpec abstracts away how a set of partitions may be represented. A > tighter representation allows us to communicate metadata for a larger number > of Partitions, with less Thrift traffic. > Given that Thrift doesn’t support polymorphism, we’d have to implement the > PartitionSpec as a Thrift Union of supported implementations. (We could > convert from the Thrift PartitionSpec to the appropriate Java PartitionSpec > sub-class.) > Thoughts? -- This message was sent by Atlassian JIRA (v6.2#6252)