Re: Review Request: HIVE-2224 : Exposing add_partitions() from hive metastore, making it atomic
On 2011-07-14 23:39:58, Paul Yang wrote: http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, lines 1469-1471 https://reviews.apache.org/r/999/diff/1/?file=21960#file21960line1469 This part is a little unusual, as Entry objects are mostly used during iteration The reason for that was to return a Pair and not have to add on an implementation for a Pair, or to have a custom Object definition for that - do you have a preference in how else to do that? - Sushanth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/999/#review1068 --- On 2011-07-07 23:20:22, Sushanth Sowmyan wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/999/ --- (Updated 2011-07-07 23:20:22) Review request for hive. Summary --- As per HIVE-2224 ( https://issues.apache.org/jira/browse/HIVE-2224 ), this patch does the following: + Exposing add_partitions() from the thrift metastore api + Making add_partitions() atomic This addresses bug HIVE-2224. https://issues.apache.org/jira/browse/HIVE-2224 Diffs - http://svn.apache.org/repos/asf/hive/trunk/metastore/if/hive_metastore.thrift 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1142116 Diff: https://reviews.apache.org/r/999/diff Testing --- Modified TestHiveMetaStore.partitionTester() to add tests for the following scenarios: + add_partition(empty list) : no exceptions thrown : works + add_partitions(list containing 3 partitions) : works, verified that partitions exist + add_partitions(list containing 2 partitions, where one of them has keyvalues identical to original partition, and another partition has a directory already created, verified that the call causes an exception, and directories are unchanged(the one with the dir pre-existing continues to exist), and no additional partition was published. + add_partitions(list with a single partition) : works, does not fault on duplicate as a result of a partial publish above. + verified that all above created partitions exist : works All of the above is called from both TestEmbeddedHiveMetaStore and TestRemoteHiveMetaStore Thanks, Sushanth
Re: Review Request: HIVE-2224 : Exposing add_partitions() from hive metastore, making it atomic
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/999/#review1068 --- http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java https://reviews.apache.org/r/999/#comment2160 This part is a little unusual, as Entry objects are mostly used during iteration - Paul On 2011-07-07 23:20:22, Sushanth Sowmyan wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/999/ --- (Updated 2011-07-07 23:20:22) Review request for hive. Summary --- As per HIVE-2224 ( https://issues.apache.org/jira/browse/HIVE-2224 ), this patch does the following: + Exposing add_partitions() from the thrift metastore api + Making add_partitions() atomic This addresses bug HIVE-2224. https://issues.apache.org/jira/browse/HIVE-2224 Diffs - http://svn.apache.org/repos/asf/hive/trunk/metastore/if/hive_metastore.thrift 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1142116 Diff: https://reviews.apache.org/r/999/diff Testing --- Modified TestHiveMetaStore.partitionTester() to add tests for the following scenarios: + add_partition(empty list) : no exceptions thrown : works + add_partitions(list containing 3 partitions) : works, verified that partitions exist + add_partitions(list containing 2 partitions, where one of them has keyvalues identical to original partition, and another partition has a directory already created, verified that the call causes an exception, and directories are unchanged(the one with the dir pre-existing continues to exist), and no additional partition was published. + add_partitions(list with a single partition) : works, does not fault on duplicate as a result of a partial publish above. + verified that all above created partitions exist : works All of the above is called from both TestEmbeddedHiveMetaStore and TestRemoteHiveMetaStore Thanks, Sushanth
Re: Review Request: HIVE-2224 : Exposing add_partitions() from hive metastore, making it atomic
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/999/ --- (Updated 2011-07-07 23:20:22.001216) Review request for hive. Summary --- As per HIVE-2224 ( https://issues.apache.org/jira/browse/HIVE-2224 ), this patch does the following: + Exposing add_partitions() from the thrift metastore api + Making add_partitions() atomic This addresses bug HIVE-2224. https://issues.apache.org/jira/browse/HIVE-2224 Diffs - http://svn.apache.org/repos/asf/hive/trunk/metastore/if/hive_metastore.thrift 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1142116 Diff: https://reviews.apache.org/r/999/diff Testing --- Modified TestHiveMetaStore.partitionTester() to add tests for the following scenarios: + add_partition(empty list) : no exceptions thrown : works + add_partitions(list containing 3 partitions) : works, verified that partitions exist + add_partitions(list containing 2 partitions, where one of them has keyvalues identical to original partition, and another partition has a directory already created, verified that the call causes an exception, and directories are unchanged(the one with the dir pre-existing continues to exist), and no additional partition was published. + add_partitions(list with a single partition) : works, does not fault on duplicate as a result of a partial publish above. + verified that all above created partitions exist : works All of the above is called from both TestEmbeddedHiveMetaStore and TestRemoteHiveMetaStore Thanks, Sushanth
Re: Review Request: HIVE-2224 : Exposing add_partitions() from hive metastore, making it atomic
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/999/ --- (Updated 2011-07-03 23:29:52.277187) Review request for hive. Summary (updated) --- As per HIVE-2224 ( https://issues.apache.org/jira/browse/HIVE-2224 ), this patch does the following: + Exposing add_partitions() from the thrift metastore api + Making add_partitions() atomic Diffs - http://svn.apache.org/repos/asf/hive/trunk/metastore/if/hive_metastore.thrift 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1142116 Diff: https://reviews.apache.org/r/999/diff Testing --- Thanks, Sushanth
Re: Review Request: HIVE-2224 : Exposing add_partitions() from hive metastore, making it atomic
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/999/ --- (Updated 2011-07-03 23:37:20.552434) Review request for hive. Summary --- As per HIVE-2224 ( https://issues.apache.org/jira/browse/HIVE-2224 ), this patch does the following: + Exposing add_partitions() from the thrift metastore api + Making add_partitions() atomic Diffs - http://svn.apache.org/repos/asf/hive/trunk/metastore/if/hive_metastore.thrift 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1142116 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1142116 Diff: https://reviews.apache.org/r/999/diff Testing (updated) --- Modified TestHiveMetaStore.partitionTester() to add tests for the following scenarios: + add_partition(empty list) : no exceptions thrown : works + add_partitions(list containing 3 partitions) : works, verified that partitions exist + add_partitions(list containing 2 partitions, where one of them has keyvalues identical to original partition, and another partition has a directory already created, verified that the call causes an exception, and directories are unchanged(the one with the dir pre-existing continues to exist), and no additional partition was published. + add_partitions(list with a single partition) : works, does not fault on duplicate as a result of a partial publish above. + verified that all above created partitions exist : works All of the above is called from both TestEmbeddedHiveMetaStore and TestRemoteHiveMetaStore Thanks, Sushanth
