Re: Review Request 72815: Deletion of non existing hive entities

[Pinal Shah]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72815/
---

(Updated Sept. 11, 2020, 12:27 p.m.)


Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, Nixon 
Rodrigues, and Sarath Subramanian.


Changes
---

updating patch for branch-0.8


Repository: atlas


Description
---

**Problem:** Whenever database or table is dropped in hive, and HiveHook is not 
enabled, we dont have anyway to get the database and table sync with hive.

**Workaround:** Added support to delete hive entities in Atlas which are 
dropped in hive.

**Usage:** ./import-hive.sh -deleteNonExisting

**NOTE:** **atlas.hook.hive.page.limit** property is added to configure the 
pageSize/limit while fetching entities from Atlas


Diffs (updated)
-

  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 f18d01b7c 
  client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 6968e8358 


Diff: https://reviews.apache.org/r/72815/diff/4/

Changes: https://reviews.apache.org/r/72815/diff/3-4/


Testing
---

Manually tested


Thanks,

Pinal Shah

Re: Review Request 72815: Deletion of non existing hive entities

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72815/#review221808
---


Ship it!




Ship It!

- Madhan Neethiraj


On Sept. 5, 2020, 5:02 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72815/
> ---
> 
> (Updated Sept. 5, 2020, 5:02 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Problem:** Whenever database or table is dropped in hive, and HiveHook is 
> not enabled, we dont have anyway to get the database and table sync with hive.
> 
> **Workaround:** Added support to delete hive entities in Atlas which are 
> dropped in hive.
> 
> **Usage:** ./import-hive.sh -deleteNonExisting
> 
> **NOTE:** **atlas.hook.hive.page.limit** property is added to configure the 
> pageSize/limit while fetching entities from Atlas
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  e659ca041 
>   client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 
> 6968e8358 
> 
> 
> Diff: https://reviews.apache.org/r/72815/diff/3/
> 
> 
> Testing
> ---
> 
> Manually tested
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 72815: Deletion of non existing hive entities

[Pinal Shah]

> On Sept. 4, 2020, 4:04 p.m., Madhan Neethiraj wrote:
> > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
> > Lines 959 (patched)
> > 
> >
> > Returning after deleting one entity? Please review.

thanks for pointing, it is incorrect


- Pinal


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72815/#review221806
---


On Sept. 5, 2020, 5:02 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72815/
> ---
> 
> (Updated Sept. 5, 2020, 5:02 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Problem:** Whenever database or table is dropped in hive, and HiveHook is 
> not enabled, we dont have anyway to get the database and table sync with hive.
> 
> **Workaround:** Added support to delete hive entities in Atlas which are 
> dropped in hive.
> 
> **Usage:** ./import-hive.sh -deleteNonExisting
> 
> **NOTE:** **atlas.hook.hive.page.limit** property is added to configure the 
> pageSize/limit while fetching entities from Atlas
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  e659ca041 
>   client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 
> 6968e8358 
> 
> 
> Diff: https://reviews.apache.org/r/72815/diff/3/
> 
> 
> Testing
> ---
> 
> Manually tested
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 72815: Deletion of non existing hive entities

[Pinal Shah]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72815/
---

(Updated Sept. 5, 2020, 5:02 a.m.)


Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, Nixon 
Rodrigues, and Sarath Subramanian.


Changes
---

addressed comments


Repository: atlas


Description
---

**Problem:** Whenever database or table is dropped in hive, and HiveHook is not 
enabled, we dont have anyway to get the database and table sync with hive.

**Workaround:** Added support to delete hive entities in Atlas which are 
dropped in hive.

**Usage:** ./import-hive.sh -deleteNonExisting

**NOTE:** **atlas.hook.hive.page.limit** property is added to configure the 
pageSize/limit while fetching entities from Atlas


Diffs (updated)
-

  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 e659ca041 
  client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 6968e8358 


Diff: https://reviews.apache.org/r/72815/diff/3/

Changes: https://reviews.apache.org/r/72815/diff/2-3/


Testing
---

Manually tested


Thanks,

Pinal Shah

Re: Review Request 72815: Deletion of non existing hive entities

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72815/#review221806
---




addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
Lines 862 (patched)


Since there is only one criteria in the filter, there is no need for 
innerFc. Consider the following:

  SearchParameters.FilterCriteria fc = new 
SearchParameters.FilterCriteria();
 
  fc.setAttributeName(ATTRIBUTE_CLUSTER_NAME);
  fc.setAttributeValue(clusterName);
  fc.setOperator(SearchParameters.Operator.EQ);

Also, the filter criteira is the same across all iteration of the for loop 
at #858. Consider setting up the criteira before once before entering the for 
loop.



addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
Lines 959 (patched)


Returning after deleting one entity? Please review.



addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
Lines 968 (patched)


deleteHiveMetadata() => deleteEntitiesForNonExistingHiveMetadata()



addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
Lines 999 (patched)


There is only one parameter to replace, as LOG.error() will print execption 
details for 'e'. Please review and remove ', {}' at the end of the message.



addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
Lines 1006 (patched)


Consider replacing:
 Queue  => List
 LinkedList => ArrayList


- Madhan Neethiraj


On Sept. 3, 2020, 12:21 p.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72815/
> ---
> 
> (Updated Sept. 3, 2020, 12:21 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Problem:** Whenever database or table is dropped in hive, and HiveHook is 
> not enabled, we dont have anyway to get the database and table sync with hive.
> 
> **Workaround:** Added support to delete hive entities in Atlas which are 
> dropped in hive.
> 
> **Usage:** ./import-hive.sh -deleteNonExisting
> 
> **NOTE:** **atlas.hook.hive.page.limit** property is added to configure the 
> pageSize/limit while fetching entities from Atlas
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  e659ca041 
>   client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 
> 6968e8358 
> 
> 
> Diff: https://reviews.apache.org/r/72815/diff/2/
> 
> 
> Testing
> ---
> 
> Manually tested
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 72815: Deletion of non existing hive entities

[Pinal Shah]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72815/
---

(Updated Sept. 3, 2020, 12:21 p.m.)


Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, Nixon 
Rodrigues, and Sarath Subramanian.


Repository: atlas


Description (updated)
---

**Problem:** Whenever database or table is dropped in hive, and HiveHook is not 
enabled, we dont have anyway to get the database and table sync with hive.

**Workaround:** Added support to delete hive entities in Atlas which are 
dropped in hive.

**Usage:** ./import-hive.sh -deleteNonExisting

**NOTE:** **atlas.hook.hive.page.limit** property is added to configure the 
pageSize/limit while fetching entities from Atlas


Diffs
-

  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 e659ca041 
  client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 6968e8358 


Diff: https://reviews.apache.org/r/72815/diff/2/


Testing
---

Manually tested


Thanks,

Pinal Shah

Re: Review Request 72815: Deletion of non existing hive entities

[Pinal Shah]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72815/
---

(Updated Sept. 3, 2020, 10:26 a.m.)


Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, Nixon 
Rodrigues, and Sarath Subramanian.


Changes
---

Addressed review comments


Repository: atlas


Description
---

**Problem:** Whenever database or table is dropped in hive, and HiveHook is not 
enabled, we dont have anyway to get the database and table sync with hive.

**Workaround:** Added support to delete hive entities in Atlas which are 
dropped in hive.

**Usage:** ./import-hive.sh -deleteNonExisting


Diffs (updated)
-

  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 e659ca041 
  client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 6968e8358 


Diff: https://reviews.apache.org/r/72815/diff/2/

Changes: https://reviews.apache.org/r/72815/diff/1-2/


Testing
---

Manually tested


Thanks,

Pinal Shah

Re: Review Request 72815: Deletion of non existing hive entities

[Pinal Shah]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72815/
---

(Updated Aug. 31, 2020, 8:51 a.m.)


Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, Nixon 
Rodrigues, and Sarath Subramanian.


Summary (updated)
-

Deletion of non existing hive entities


Repository: atlas


Description
---

**Problem:** Whenever database or table is dropped in hive, and HiveHook is not 
enabled, we dont have anyway to get the database and table sync with hive.

**Workaround:** Added support to delete hive entities in Atlas which are 
dropped in hive.

**Usage:** ./import-hive.sh -deleteNonExisting


Diffs
-

  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 e659ca041 
  client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 6968e8358 


Diff: https://reviews.apache.org/r/72815/diff/1/


Testing
---

Manually tested


Thanks,

Pinal Shah