[jira] [Work logged] (GOBBLIN-861) Skip getPartition() call to Hive Metastore when a partition already exists

2019-08-19 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-861?focusedWorklogId=297291=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297291
 ]

ASF GitHub Bot logged work on GOBBLIN-861:
--

Author: ASF GitHub Bot
Created on: 19/Aug/19 17:01
Start Date: 19/Aug/19 17:01
Worklog Time Spent: 10m 
  Work Description: asfgit commented on pull request #2716: 
[GOBBLIN-861]Skip getPartition() call to Hive Metastore when a partition 
already e…
URL: https://github.com/apache/incubator-gobblin/pull/2716
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 297291)
Time Spent: 2h 10m  (was: 2h)

> Skip getPartition() call to Hive Metastore when a partition already exists
> --
>
> Key: GOBBLIN-861
> URL: https://issues.apache.org/jira/browse/GOBBLIN-861
> Project: Apache Gobblin
>  Issue Type: Task
>  Components: hive-registration
>Reporter: Zihan Li
>Assignee: Abhishek Tiwari
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Currently, we compute a diff between the current partition and an already 
> registered partition when a partition has already been registered in Hive. 
> This is done by calling getPartition() on the Hive metastore client, which 
> can be expensive. Since no time-varying attributes are stored in a Hive 
> partition, diff computation (and getPartition() call) can be skipped when a 
> partition already exists.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Work logged] (GOBBLIN-861) Skip getPartition() call to Hive Metastore when a partition already exists

2019-08-16 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-861?focusedWorklogId=296512=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296512
 ]

ASF GitHub Bot logged work on GOBBLIN-861:
--

Author: ASF GitHub Bot
Created on: 16/Aug/19 19:14
Start Date: 16/Aug/19 19:14
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2716: [GOBBLIN-861]Skip 
getPartition() call to Hive Metastore when a partition already e…
URL: 
https://github.com/apache/incubator-gobblin/pull/2716#issuecomment-522121077
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2716?src=pr=h1)
 Report
   > Merging 
[#2716](https://codecov.io/gh/apache/incubator-gobblin/pull/2716?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/c7e4a6efbb08c433c2be69958eebae3b328e347a?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `0%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2716/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2716?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2716  +/-   ##
   
   + Coverage 45.07%   45.07%   +<.01% 
   - Complexity 8738 8743   +5 
   
 Files  1879 1880   +1 
 Lines 7016270186  +24 
 Branches   7704 7707   +3 
   
   + Hits  3162531636  +11 
   - Misses3561735626   +9 
   - Partials   2920 2924   +4
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2716?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...lin/hive/metastore/HiveMetaStoreBasedRegister.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2716/diff?src=pr=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL21ldGFzdG9yZS9IaXZlTWV0YVN0b3JlQmFzZWRSZWdpc3Rlci5qYXZh)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2716/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh)
 | `61.42% <0%> (-1.43%)` | `4% <0%> (ø)` | |
   | 
[...e/gobblin/config/store/zip/ZipFileConfigStore.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2716/diff?src=pr=tree#diff-Z29iYmxpbi1jb25maWctbWFuYWdlbWVudC9nb2JibGluLWNvbmZpZy1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbmZpZy9zdG9yZS96aXAvWmlwRmlsZUNvbmZpZ1N0b3JlLmphdmE=)
 | `73.21% <0%> (-1.34%)` | `12% <0%> (ø)` | |
   | 
[...apache/gobblin/hive/avro/HiveAvroSerDeManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2716/diff?src=pr=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL2F2cm8vSGl2ZUF2cm9TZXJEZU1hbmFnZXIuamF2YQ==)
 | `52.17% <0%> (-0.77%)` | `8% <0%> (ø)` | |
   | 
[...g/apache/gobblin/metrics/event/EventSubmitter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2716/diff?src=pr=tree#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9ldmVudC9FdmVudFN1Ym1pdHRlci5qYXZh)
 | `37.03% <0%> (-0.7%)` | `3% <0%> (ø)` | |
   | 
[...che/gobblin/hive/metastore/HiveMetaStoreUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2716/diff?src=pr=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL21ldGFzdG9yZS9IaXZlTWV0YVN0b3JlVXRpbHMuamF2YQ==)
 | `31.83% <0%> (-0.15%)` | `12% <0%> (ø)` | |
   | 
[...onfig/store/api/PhysicalPathNotExistException.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2716/diff?src=pr=tree#diff-Z29iYmxpbi1jb25maWctbWFuYWdlbWVudC9nb2JibGluLWNvbmZpZy1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbmZpZy9zdG9yZS9hcGkvUGh5c2ljYWxQYXRoTm90RXhpc3RFeGNlcHRpb24uamF2YQ==)
 | `0% <0%> (ø)` | `0% <0%> (?)` | |
   | 
[.../org/apache/gobblin/hive/HiveRegistrationUnit.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2716/diff?src=pr=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL0hpdmVSZWdpc3RyYXRpb25Vbml0LmphdmE=)
 | `47.39% <0%> (+0.37%)` | `33% <0%> (+1%)` | :arrow_up: |
   | 

[jira] [Work logged] (GOBBLIN-861) Skip getPartition() call to Hive Metastore when a partition already exists

2019-08-16 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-861?focusedWorklogId=296436=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296436
 ]

ASF GitHub Bot logged work on GOBBLIN-861:
--

Author: ASF GitHub Bot
Created on: 16/Aug/19 17:14
Start Date: 16/Aug/19 17:14
Worklog Time Spent: 10m 
  Work Description: ZihanLi58 commented on pull request #2716: 
[GOBBLIN-861]Skip getPartition() call to Hive Metastore when a partition 
already e…
URL: https://github.com/apache/incubator-gobblin/pull/2716#discussion_r314811852
 
 

 ##
 File path: 
gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/metastore/HiveMetaStoreBasedRegister.java
 ##
 @@ -501,8 +482,30 @@ private void addOrAlterPartition(IMetaStoreClient client, 
Table table, HiveParti
   }
 }
   }
+  protected void OnPartitionExist(IMetaStoreClient client, Table table, 
HivePartition partition, Partition nativePartition, TException e) throws 
TException {
 
 Review comment:
   Because most of the logic are the same, ie. checking the key value size and 
use lock to call add_partition method. The only different is after it found the 
partition has already existed.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 296436)
Time Spent: 1h 50m  (was: 1h 40m)

> Skip getPartition() call to Hive Metastore when a partition already exists
> --
>
> Key: GOBBLIN-861
> URL: https://issues.apache.org/jira/browse/GOBBLIN-861
> Project: Apache Gobblin
>  Issue Type: Task
>  Components: hive-registration
>Reporter: Zihan Li
>Assignee: Abhishek Tiwari
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Currently, we compute a diff between the current partition and an already 
> registered partition when a partition has already been registered in Hive. 
> This is done by calling getPartition() on the Hive metastore client, which 
> can be expensive. Since no time-varying attributes are stored in a Hive 
> partition, diff computation (and getPartition() call) can be skipped when a 
> partition already exists.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Work logged] (GOBBLIN-861) Skip getPartition() call to Hive Metastore when a partition already exists

2019-08-15 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-861?focusedWorklogId=296037=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296037
 ]

ASF GitHub Bot logged work on GOBBLIN-861:
--

Author: ASF GitHub Bot
Created on: 16/Aug/19 03:32
Start Date: 16/Aug/19 03:32
Worklog Time Spent: 10m 
  Work Description: sv2000 commented on pull request #2716: 
[GOBBLIN-861]Skip getPartition() call to Hive Metastore when a partition 
already e…
URL: https://github.com/apache/incubator-gobblin/pull/2716#discussion_r314572693
 
 

 ##
 File path: 
gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/metastore/HiveMetaStoreRegisterWithoutComputingPartitionDiff.java
 ##
 @@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.hive.metastore;
+
+import com.google.common.base.Optional;
+import java.io.IOException;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.configuration.State;
+import org.apache.gobblin.hive.HivePartition;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.thrift.TException;
+
+
+@Slf4j
+public class HiveMetaStoreRegisterWithoutComputingPartitionDiff extends 
HiveMetaStoreBasedRegister {
 
 Review comment:
   Do we need a separate class? Can we have a skipDiffComputation flag that is 
set via config?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 296037)
Time Spent: 1h 20m  (was: 1h 10m)

> Skip getPartition() call to Hive Metastore when a partition already exists
> --
>
> Key: GOBBLIN-861
> URL: https://issues.apache.org/jira/browse/GOBBLIN-861
> Project: Apache Gobblin
>  Issue Type: Task
>  Components: hive-registration
>Reporter: Zihan Li
>Assignee: Abhishek Tiwari
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Currently, we compute a diff between the current partition and an already 
> registered partition when a partition has already been registered in Hive. 
> This is done by calling getPartition() on the Hive metastore client, which 
> can be expensive. Since no time-varying attributes are stored in a Hive 
> partition, diff computation (and getPartition() call) can be skipped when a 
> partition already exists.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Work logged] (GOBBLIN-861) Skip getPartition() call to Hive Metastore when a partition already exists

2019-08-15 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-861?focusedWorklogId=296038=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296038
 ]

ASF GitHub Bot logged work on GOBBLIN-861:
--

Author: ASF GitHub Bot
Created on: 16/Aug/19 03:32
Start Date: 16/Aug/19 03:32
Worklog Time Spent: 10m 
  Work Description: sv2000 commented on pull request #2716: 
[GOBBLIN-861]Skip getPartition() call to Hive Metastore when a partition 
already e…
URL: https://github.com/apache/incubator-gobblin/pull/2716#discussion_r314572770
 
 

 ##
 File path: 
gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/metastore/HiveMetaStoreRegisterWithoutComputingPartitionDiff.java
 ##
 @@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.hive.metastore;
+
+import com.google.common.base.Optional;
+import java.io.IOException;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.configuration.State;
+import org.apache.gobblin.hive.HivePartition;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.thrift.TException;
+
+
+@Slf4j
+public class HiveMetaStoreRegisterWithoutComputingPartitionDiff extends 
HiveMetaStoreBasedRegister {
+  public HiveMetaStoreRegisterWithoutComputingPartitionDiff(State state, 
Optional metastoreURI) throws IOException {
+super(state, metastoreURI);
+  }
+
+  @Override
+  protected void OnPartitionExist(IMetaStoreClient client, Table table, 
HivePartition partition,
+  Partition nativePartition, TException e) throws TException {
+if(e instanceof AlreadyExistsException) {
+  log.info(String.format("Partition %s in table %s with location %s 
already exists and no need to update",
 
 Review comment:
   +1.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 296038)
Time Spent: 1.5h  (was: 1h 20m)

> Skip getPartition() call to Hive Metastore when a partition already exists
> --
>
> Key: GOBBLIN-861
> URL: https://issues.apache.org/jira/browse/GOBBLIN-861
> Project: Apache Gobblin
>  Issue Type: Task
>  Components: hive-registration
>Reporter: Zihan Li
>Assignee: Abhishek Tiwari
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Currently, we compute a diff between the current partition and an already 
> registered partition when a partition has already been registered in Hive. 
> This is done by calling getPartition() on the Hive metastore client, which 
> can be expensive. Since no time-varying attributes are stored in a Hive 
> partition, diff computation (and getPartition() call) can be skipped when a 
> partition already exists.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Work logged] (GOBBLIN-861) Skip getPartition() call to Hive Metastore when a partition already exists

2019-08-15 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-861?focusedWorklogId=296036=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296036
 ]

ASF GitHub Bot logged work on GOBBLIN-861:
--

Author: ASF GitHub Bot
Created on: 16/Aug/19 03:32
Start Date: 16/Aug/19 03:32
Worklog Time Spent: 10m 
  Work Description: sv2000 commented on pull request #2716: 
[GOBBLIN-861]Skip getPartition() call to Hive Metastore when a partition 
already e…
URL: https://github.com/apache/incubator-gobblin/pull/2716#discussion_r314571989
 
 

 ##
 File path: 
gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/metastore/HiveMetaStoreBasedRegister.java
 ##
 @@ -472,26 +472,7 @@ private void addOrAlterPartition(IMetaStoreClient client, 
Table table, HiveParti
 table.getTableName(), nativePartition.getSd().getLocation()));
   } catch (TException e) {
 try {
-  HivePartition existingPartition;
-  try (Timer.Context context = 
this.metricContext.timer(GET_HIVE_PARTITION).time()) {
-existingPartition = HiveMetaStoreUtils.getHivePartition(
-client.getPartition(table.getDbName(), table.getTableName(), 
nativePartition.getValues()));
-  }
-
-  if (needToUpdatePartition(existingPartition, partition)) {
-log.info(String.format("Partition update required. 
ExistingPartition %s, newPartition %s",
-stringifyPartition(existingPartition), 
stringifyPartition(partition)));
-Partition newPartition = 
getPartitionWithCreateTime(nativePartition, existingPartition);
-log.info(String.format("Altering partition %s", newPartition));
-try (Timer.Context context = 
this.metricContext.timer(ALTER_PARTITION).time()) {
-  client.alter_partition(table.getDbName(), table.getTableName(), 
newPartition);
-}
-log.info(String.format("Updated partition %s in table %s with 
location %s", stringifyPartition(newPartition),
-table.getTableName(), nativePartition.getSd().getLocation()));
-  } else {
-log.info(String.format("Partition %s in table %s with location %s 
already exists and no need to update",
-stringifyPartition(nativePartition), table.getTableName(), 
nativePartition.getSd().getLocation()));
-  }
+  OnPartitionExist(client, table, partition, nativePartition, e);
 
 Review comment:
   we typically adopt the convention of having variable names and methods start 
with lower case. so onPartitionExists(...)?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 296036)
Time Spent: 1h 10m  (was: 1h)

> Skip getPartition() call to Hive Metastore when a partition already exists
> --
>
> Key: GOBBLIN-861
> URL: https://issues.apache.org/jira/browse/GOBBLIN-861
> Project: Apache Gobblin
>  Issue Type: Task
>  Components: hive-registration
>Reporter: Zihan Li
>Assignee: Abhishek Tiwari
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Currently, we compute a diff between the current partition and an already 
> registered partition when a partition has already been registered in Hive. 
> This is done by calling getPartition() on the Hive metastore client, which 
> can be expensive. Since no time-varying attributes are stored in a Hive 
> partition, diff computation (and getPartition() call) can be skipped when a 
> partition already exists.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Work logged] (GOBBLIN-861) Skip getPartition() call to Hive Metastore when a partition already exists

2019-08-15 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-861?focusedWorklogId=296039=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296039
 ]

ASF GitHub Bot logged work on GOBBLIN-861:
--

Author: ASF GitHub Bot
Created on: 16/Aug/19 03:32
Start Date: 16/Aug/19 03:32
Worklog Time Spent: 10m 
  Work Description: sv2000 commented on pull request #2716: 
[GOBBLIN-861]Skip getPartition() call to Hive Metastore when a partition 
already e…
URL: https://github.com/apache/incubator-gobblin/pull/2716#discussion_r314572835
 
 

 ##
 File path: 
gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/metastore/HiveMetaStoreRegisterWithoutComputingPartitionDiff.java
 ##
 @@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.hive.metastore;
+
+import com.google.common.base.Optional;
+import java.io.IOException;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.configuration.State;
+import org.apache.gobblin.hive.HivePartition;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.thrift.TException;
+
+
+@Slf4j
+public class HiveMetaStoreRegisterWithoutComputingPartitionDiff extends 
HiveMetaStoreBasedRegister {
+  public HiveMetaStoreRegisterWithoutComputingPartitionDiff(State state, 
Optional metastoreURI) throws IOException {
+super(state, metastoreURI);
+  }
+
+  @Override
+  protected void OnPartitionExist(IMetaStoreClient client, Table table, 
HivePartition partition,
 
 Review comment:
   onPartitionExists?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 296039)
Time Spent: 1h 40m  (was: 1.5h)

> Skip getPartition() call to Hive Metastore when a partition already exists
> --
>
> Key: GOBBLIN-861
> URL: https://issues.apache.org/jira/browse/GOBBLIN-861
> Project: Apache Gobblin
>  Issue Type: Task
>  Components: hive-registration
>Reporter: Zihan Li
>Assignee: Abhishek Tiwari
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Currently, we compute a diff between the current partition and an already 
> registered partition when a partition has already been registered in Hive. 
> This is done by calling getPartition() on the Hive metastore client, which 
> can be expensive. Since no time-varying attributes are stored in a Hive 
> partition, diff computation (and getPartition() call) can be skipped when a 
> partition already exists.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Work logged] (GOBBLIN-861) Skip getPartition() call to Hive Metastore when a partition already exists

2019-08-15 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-861?focusedWorklogId=295979=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-295979
 ]

ASF GitHub Bot logged work on GOBBLIN-861:
--

Author: ASF GitHub Bot
Created on: 16/Aug/19 02:46
Start Date: 16/Aug/19 02:46
Worklog Time Spent: 10m 
  Work Description: autumnust commented on pull request #2716: 
[GOBBLIN-861]Skip getPartition() call to Hive Metastore when a partition 
already e…
URL: https://github.com/apache/incubator-gobblin/pull/2716#discussion_r314566577
 
 

 ##
 File path: 
gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/metastore/HiveMetaStoreRegisterWithoutComputingPartitionDiff.java
 ##
 @@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.hive.metastore;
+
+import com.google.common.base.Optional;
+import java.io.IOException;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.configuration.State;
+import org.apache.gobblin.hive.HivePartition;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.thrift.TException;
+
+
+@Slf4j
+public class HiveMetaStoreRegisterWithoutComputingPartitionDiff extends 
HiveMetaStoreBasedRegister {
+  public HiveMetaStoreRegisterWithoutComputingPartitionDiff(State state, 
Optional metastoreURI) throws IOException {
+super(state, metastoreURI);
+  }
+
+  @Override
+  protected void OnPartitionExist(IMetaStoreClient client, Table table, 
HivePartition partition,
+  Partition nativePartition, TException e) throws TException {
+if(e instanceof AlreadyExistsException) {
 
 Review comment:
   nit-pick: pay attention to blank between character and bracket
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 295979)
Time Spent: 40m  (was: 0.5h)

> Skip getPartition() call to Hive Metastore when a partition already exists
> --
>
> Key: GOBBLIN-861
> URL: https://issues.apache.org/jira/browse/GOBBLIN-861
> Project: Apache Gobblin
>  Issue Type: Task
>  Components: hive-registration
>Reporter: Zihan Li
>Assignee: Abhishek Tiwari
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Currently, we compute a diff between the current partition and an already 
> registered partition when a partition has already been registered in Hive. 
> This is done by calling getPartition() on the Hive metastore client, which 
> can be expensive. Since no time-varying attributes are stored in a Hive 
> partition, diff computation (and getPartition() call) can be skipped when a 
> partition already exists.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Work logged] (GOBBLIN-861) Skip getPartition() call to Hive Metastore when a partition already exists

2019-08-15 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-861?focusedWorklogId=295982=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-295982
 ]

ASF GitHub Bot logged work on GOBBLIN-861:
--

Author: ASF GitHub Bot
Created on: 16/Aug/19 02:46
Start Date: 16/Aug/19 02:46
Worklog Time Spent: 10m 
  Work Description: autumnust commented on pull request #2716: 
[GOBBLIN-861]Skip getPartition() call to Hive Metastore when a partition 
already e…
URL: https://github.com/apache/incubator-gobblin/pull/2716#discussion_r314566652
 
 

 ##
 File path: 
gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/metastore/HiveMetaStoreRegisterWithoutComputingPartitionDiff.java
 ##
 @@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.hive.metastore;
+
+import com.google.common.base.Optional;
+import java.io.IOException;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.configuration.State;
+import org.apache.gobblin.hive.HivePartition;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.thrift.TException;
+
+
+@Slf4j
+public class HiveMetaStoreRegisterWithoutComputingPartitionDiff extends 
HiveMetaStoreBasedRegister {
+  public HiveMetaStoreRegisterWithoutComputingPartitionDiff(State state, 
Optional metastoreURI) throws IOException {
+super(state, metastoreURI);
+  }
+
+  @Override
+  protected void OnPartitionExist(IMetaStoreClient client, Table table, 
HivePartition partition,
+  Partition nativePartition, TException e) throws TException {
+if(e instanceof AlreadyExistsException) {
+  log.info(String.format("Partition %s in table %s with location %s 
already exists and no need to update",
 
 Review comment:
   this will end up flooding the log, suggest to use debug instead of info
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 295982)
Time Spent: 1h  (was: 50m)

> Skip getPartition() call to Hive Metastore when a partition already exists
> --
>
> Key: GOBBLIN-861
> URL: https://issues.apache.org/jira/browse/GOBBLIN-861
> Project: Apache Gobblin
>  Issue Type: Task
>  Components: hive-registration
>Reporter: Zihan Li
>Assignee: Abhishek Tiwari
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Currently, we compute a diff between the current partition and an already 
> registered partition when a partition has already been registered in Hive. 
> This is done by calling getPartition() on the Hive metastore client, which 
> can be expensive. Since no time-varying attributes are stored in a Hive 
> partition, diff computation (and getPartition() call) can be skipped when a 
> partition already exists.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Work logged] (GOBBLIN-861) Skip getPartition() call to Hive Metastore when a partition already exists

2019-08-15 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-861?focusedWorklogId=295981=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-295981
 ]

ASF GitHub Bot logged work on GOBBLIN-861:
--

Author: ASF GitHub Bot
Created on: 16/Aug/19 02:46
Start Date: 16/Aug/19 02:46
Worklog Time Spent: 10m 
  Work Description: autumnust commented on pull request #2716: 
[GOBBLIN-861]Skip getPartition() call to Hive Metastore when a partition 
already e…
URL: https://github.com/apache/incubator-gobblin/pull/2716#discussion_r314566770
 
 

 ##
 File path: 
gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/metastore/HiveMetaStoreRegisterWithoutComputingPartitionDiff.java
 ##
 @@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.hive.metastore;
+
+import com.google.common.base.Optional;
+import java.io.IOException;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.configuration.State;
+import org.apache.gobblin.hive.HivePartition;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.thrift.TException;
+
+
+@Slf4j
+public class HiveMetaStoreRegisterWithoutComputingPartitionDiff extends 
HiveMetaStoreBasedRegister {
 
 Review comment:
   add a javadoc for usage of this
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 295981)
Time Spent: 1h  (was: 50m)

> Skip getPartition() call to Hive Metastore when a partition already exists
> --
>
> Key: GOBBLIN-861
> URL: https://issues.apache.org/jira/browse/GOBBLIN-861
> Project: Apache Gobblin
>  Issue Type: Task
>  Components: hive-registration
>Reporter: Zihan Li
>Assignee: Abhishek Tiwari
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Currently, we compute a diff between the current partition and an already 
> registered partition when a partition has already been registered in Hive. 
> This is done by calling getPartition() on the Hive metastore client, which 
> can be expensive. Since no time-varying attributes are stored in a Hive 
> partition, diff computation (and getPartition() call) can be skipped when a 
> partition already exists.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Work logged] (GOBBLIN-861) Skip getPartition() call to Hive Metastore when a partition already exists

2019-08-15 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-861?focusedWorklogId=295980=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-295980
 ]

ASF GitHub Bot logged work on GOBBLIN-861:
--

Author: ASF GitHub Bot
Created on: 16/Aug/19 02:46
Start Date: 16/Aug/19 02:46
Worklog Time Spent: 10m 
  Work Description: autumnust commented on pull request #2716: 
[GOBBLIN-861]Skip getPartition() call to Hive Metastore when a partition 
already e…
URL: https://github.com/apache/incubator-gobblin/pull/2716#discussion_r314567012
 
 

 ##
 File path: 
gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/metastore/HiveMetaStoreBasedRegister.java
 ##
 @@ -501,8 +482,30 @@ private void addOrAlterPartition(IMetaStoreClient client, 
Table table, HiveParti
   }
 }
   }
+  protected void OnPartitionExist(IMetaStoreClient client, Table table, 
HivePartition partition, Partition nativePartition, TException e) throws 
TException {
 
 Review comment:
   Why not make `addOrAlterPartition` protected and overwrite that ?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 295980)
Time Spent: 50m  (was: 40m)

> Skip getPartition() call to Hive Metastore when a partition already exists
> --
>
> Key: GOBBLIN-861
> URL: https://issues.apache.org/jira/browse/GOBBLIN-861
> Project: Apache Gobblin
>  Issue Type: Task
>  Components: hive-registration
>Reporter: Zihan Li
>Assignee: Abhishek Tiwari
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Currently, we compute a diff between the current partition and an already 
> registered partition when a partition has already been registered in Hive. 
> This is done by calling getPartition() on the Hive metastore client, which 
> can be expensive. Since no time-varying attributes are stored in a Hive 
> partition, diff computation (and getPartition() call) can be skipped when a 
> partition already exists.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Work logged] (GOBBLIN-861) Skip getPartition() call to Hive Metastore when a partition already exists

2019-08-15 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-861?focusedWorklogId=295978=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-295978
 ]

ASF GitHub Bot logged work on GOBBLIN-861:
--

Author: ASF GitHub Bot
Created on: 16/Aug/19 02:46
Start Date: 16/Aug/19 02:46
Worklog Time Spent: 10m 
  Work Description: autumnust commented on pull request #2716: 
[GOBBLIN-861]Skip getPartition() call to Hive Metastore when a partition 
already e…
URL: https://github.com/apache/incubator-gobblin/pull/2716#discussion_r314565883
 
 

 ##
 File path: 
gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/metastore/HiveMetaStoreBasedRegister.java
 ##
 @@ -119,7 +119,7 @@
* when the first time a table/database is loaded into the cache, whether 
they existed on the remote hiveMetaStore side.
*/
   CacheLoader cacheLoader = new CacheLoader() {
-  @Override
+@Override
 
 Review comment:
   Additional blank
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 295978)
Time Spent: 0.5h  (was: 20m)

> Skip getPartition() call to Hive Metastore when a partition already exists
> --
>
> Key: GOBBLIN-861
> URL: https://issues.apache.org/jira/browse/GOBBLIN-861
> Project: Apache Gobblin
>  Issue Type: Task
>  Components: hive-registration
>Reporter: Zihan Li
>Assignee: Abhishek Tiwari
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Currently, we compute a diff between the current partition and an already 
> registered partition when a partition has already been registered in Hive. 
> This is done by calling getPartition() on the Hive metastore client, which 
> can be expensive. Since no time-varying attributes are stored in a Hive 
> partition, diff computation (and getPartition() call) can be skipped when a 
> partition already exists.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Work logged] (GOBBLIN-861) Skip getPartition() call to Hive Metastore when a partition already exists

2019-08-15 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-861?focusedWorklogId=295927=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-295927
 ]

ASF GitHub Bot logged work on GOBBLIN-861:
--

Author: ASF GitHub Bot
Created on: 16/Aug/19 01:29
Start Date: 16/Aug/19 01:29
Worklog Time Spent: 10m 
  Work Description: ZihanLi58 commented on issue #2716: [GOBBLIN-861]Skip 
getPartition() call to Hive Metastore when a partition already e…
URL: 
https://github.com/apache/incubator-gobblin/pull/2716#issuecomment-521852126
 
 
   @sv2000 @autumnust Can you take a look at this change? Thanks.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 295927)
Time Spent: 20m  (was: 10m)

> Skip getPartition() call to Hive Metastore when a partition already exists
> --
>
> Key: GOBBLIN-861
> URL: https://issues.apache.org/jira/browse/GOBBLIN-861
> Project: Apache Gobblin
>  Issue Type: Task
>  Components: hive-registration
>Reporter: Zihan Li
>Assignee: Abhishek Tiwari
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Currently, we compute a diff between the current partition and an already 
> registered partition when a partition has already been registered in Hive. 
> This is done by calling getPartition() on the Hive metastore client, which 
> can be expensive. Since no time-varying attributes are stored in a Hive 
> partition, diff computation (and getPartition() call) can be skipped when a 
> partition already exists.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Work logged] (GOBBLIN-861) Skip getPartition() call to Hive Metastore when a partition already exists

2019-08-15 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-861?focusedWorklogId=295926=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-295926
 ]

ASF GitHub Bot logged work on GOBBLIN-861:
--

Author: ASF GitHub Bot
Created on: 16/Aug/19 01:29
Start Date: 16/Aug/19 01:29
Worklog Time Spent: 10m 
  Work Description: ZihanLi58 commented on pull request #2716: 
[GOBBLIN-861]Skip getPartition() call to Hive Metastore when a partition 
already e…
URL: https://github.com/apache/incubator-gobblin/pull/2716
 
 
   …xists GOBBLIN-861
   
   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I 
have checked off all the steps below!
   
   
   ### JIRA
   - [ ] My PR addresses the following [Gobblin 
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references 
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
   - https://issues.apache.org/jira/browse/GOBBLIN-861
   
   
   ### Description
   - [ ] Here are some details about my PR, including screenshots (if 
applicable):
   Add a subclass of HiveMetaStoreBasedRegister to skip compute diff when the 
partition already exists.
   
   ### Tests
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   Test streaming job to make sure hove registration works well
   
   
   ### Commits
   - [ ] My commits all reference JIRA issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
   1. Subject is separated from body by a blank line
   2. Subject is limited to 50 characters
   3. Subject does not end with a period
   4. Subject uses the imperative mood ("add", not "adding")
   5. Body wraps at 72 characters
   6. Body explains "what" and "why", not "how"
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 295926)
Time Spent: 10m
Remaining Estimate: 0h

> Skip getPartition() call to Hive Metastore when a partition already exists
> --
>
> Key: GOBBLIN-861
> URL: https://issues.apache.org/jira/browse/GOBBLIN-861
> Project: Apache Gobblin
>  Issue Type: Task
>  Components: hive-registration
>Reporter: Zihan Li
>Assignee: Abhishek Tiwari
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Currently, we compute a diff between the current partition and an already 
> registered partition when a partition has already been registered in Hive. 
> This is done by calling getPartition() on the Hive metastore client, which 
> can be expensive. Since no time-varying attributes are stored in a Hive 
> partition, diff computation (and getPartition() call) can be skipped when a 
> partition already exists.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)