Review Request 72294: ATLAS-3712: Add 'isAppendOnPartialUpdate' option in ML model to append updates

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72294/
---

Review request for atlas, Ashutosh Mestry, Karthik Manamcheri, Sridhar K, 
Madhan Neethiraj, and Sarath Subramanian.


Bugs: atlas-3712
https://issues.apache.org/jira/browse/atlas-3712


Repository: atlas


Description
---

add 'isAppendOnPartialUpdate' option in ML model at 
https://github.com/apache/atlas/blob/master/addons/models/4000-MachineLearning/4010-ml_model.json
 to append updates using this new feature introduced in ATLAS-3700.


Diffs
-

  addons/models/4000-MachineLearning/4010-ml_model.json 
c4ba729b87cf581e7f9b23925968b275ae4a688b 


Diff: https://reviews.apache.org/r/72294/diff/1/


Testing
---


Thanks,

Na Li

Re: Review Request 72288: ATLAS-3700: updated attribute-def with an option to append value (instead of overwrite) during partial-updates

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72288/#review220156
---


Ship it!




Ship It!

- Na Li


On March 31, 2020, 8:13 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72288/
> ---
> 
> (Updated March 31, 2020, 8:13 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Na Li, Mandar Ambawane, mayank, 
> Nikhil Bonte, Nixon Rodrigues, Pinal Shah, Sarath Subramanian, and Sidharth 
> Mishra.
> 
> 
> Bugs: ATLAS-3700
> https://issues.apache.org/jira/browse/ATLAS-3700
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Added 'isAppendOnPartialUpdate' option in attribute-def, to enable appending 
> of values to array/map type attributes during partial-update operation. 
> Example:
> 
>   {
> "name":"metadata",
> "typeName":"map",
> "isIndexable": false,
> "isOptional":  true,
> "isUnique":false,
> "options": {
>   "isAppendOnPartialUpdate": "true"
> }
>   }
> 
> The value specified for this attribute in partial-update API calls will be 
> appended to existing value - if present. Full-update API calls will overwrite 
> the value of the attribute, as it does currently; there is no change in 
> full-update behavior.
> 
> 
> Diffs
> -
> 
>   
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java
>  1434a2459 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java 
> 1d4e37b05 
>   intg/src/test/java/org/apache/atlas/TestUtilsV2.java 7ec2b87e3 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  75b016cca 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasComplexAttributesTest.java
>  b682e8673 
> 
> 
> Diff: https://reviews.apache.org/r/72288/diff/1/
> 
> 
> Testing
> ---
> 
> - added tests to cover the new scenarios introduced by the new option:
> - pre-commit tests run: 
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1779/
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Review Request 72288: ATLAS-3700: updated attribute-def with an option to append value (instead of overwrite) during partial-updates

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72288/#review220154
---




repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
Lines 1357 (patched)


do we want to remove duplication?


- Na Li


On March 31, 2020, 8:13 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72288/
> ---
> 
> (Updated March 31, 2020, 8:13 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Na Li, Mandar Ambawane, mayank, 
> Nikhil Bonte, Nixon Rodrigues, Pinal Shah, Sarath Subramanian, and Sidharth 
> Mishra.
> 
> 
> Bugs: ATLAS-3700
> https://issues.apache.org/jira/browse/ATLAS-3700
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Added 'isAppendOnPartialUpdate' option in attribute-def, to enable appending 
> of values to array/map type attributes during partial-update operation. 
> Example:
> 
>   {
> "name":"metadata",
> "typeName":"map",
> "isIndexable": false,
> "isOptional":  true,
> "isUnique":false,
> "options": {
>   "isAppendOnPartialUpdate": "true"
> }
>   }
> 
> The value specified for this attribute in partial-update API calls will be 
> appended to existing value - if present. Full-update API calls will overwrite 
> the value of the attribute, as it does currently; there is no change in 
> full-update behavior.
> 
> 
> Diffs
> -
> 
>   
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java
>  1434a2459 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java 
> 1d4e37b05 
>   intg/src/test/java/org/apache/atlas/TestUtilsV2.java 7ec2b87e3 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  75b016cca 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasComplexAttributesTest.java
>  b682e8673 
> 
> 
> Diff: https://reviews.apache.org/r/72288/diff/1/
> 
> 
> Testing
> ---
> 
> - added tests to cover the new scenarios introduced by the new option:
> - pre-commit tests run: 
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1779/
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Review Request 72278: ATLAS-3700: Partial update to Atlas overwrite map entry with different keys

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72278/
---

(Updated March 30, 2020, 11:52 p.m.)


Review request for atlas, Ashutosh Mestry, Karthik Manamcheri, Sridhar K, and 
Sarath Subramanian.


Bugs: atlas-3700
https://issues.apache.org/jira/browse/atlas-3700


Repository: atlas


Description
---

1) The model definition is in 
"https://github.com/apache/atlas/blob/master/addons/models/4000-MachineLearning/4010-ml_model.json#L90;

2) I created a request of type EntityCreateRequestV2. It contains an entity of 
type ml_model_build with an attribute "metadata", which is a map, and contains 
key-value entries for "engineImageTag" and "engineImageName".

3) Then I created a request of type EntityPartialUpdateRequestV2. It contains 
an entity of type ml_model_build with an attribute "metadata", which is a map, 
and contains key-value entry for "updated_at".

4) In the properties of the entity ml_model_build, the * attribute "*metadata" 
only contains key-value entry for "updated_at".

Desired behavior:

The attribute "metadata" should contains key-value entries for 
"engineImageTag", "engineImageName" and "updated_at".

Current behavior:

the attribute "metadata" only contains key-value entry for "updated_at".


Diffs (updated)
-

  
intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java 
1434a24590c4f4172378f98d83ca8d9e9ec9c4d8 
  intg/src/main/java/org/apache/atlas/model/notification/HookNotification.java 
5b5fa04e26b17071bdc2160974d7005bc110de74 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
 75b016cca04801c4c1512c1527453fd59a11a6c4 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
 225b72cbef20b2329b36f550da60c1bbe1f95efa 
  server-api/src/main/java/org/apache/atlas/RequestContext.java 
282a66f1d466d58bce7a590513bc92f99733ba81 
  
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
 3f1ea05e17cded3433d050f891c807d55bdd022a 


Diff: https://reviews.apache.org/r/72278/diff/4/

Changes: https://reviews.apache.org/r/72278/diff/3-4/


Testing
---

add new test case for map attribute for partial update


Thanks,

Na Li

Re: Review Request 72278: ATLAS-3700: Partial update to Atlas overwrite map entry with different keys

[Na Li via Review Board]

> On March 30, 2020, 10:14 p.m., Sarath Subramanian wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
> > Lines 1300 (patched)
> > 
> >
> > The following cases are not handled:
> > 
> > |---|
> > | currValue |  newValue |
> > |---|
> > |  Null |   Null| => Will set to empty map 
> > |  Empty|   Null| => Will set to empty map 
> > |  Not Null |   Null| => Will set to empty map
> > |---|
> > 
> > Please review.
> 
> Karthik Manamcheri wrote:
> Remember to add unit tests for these cases as well.

fixed and added unit test


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72278/#review220127
---


On March 30, 2020, 11:52 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72278/
> ---
> 
> (Updated March 30, 2020, 11:52 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Karthik Manamcheri, Sridhar K, and 
> Sarath Subramanian.
> 
> 
> Bugs: atlas-3700
> https://issues.apache.org/jira/browse/atlas-3700
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1) The model definition is in 
> "https://github.com/apache/atlas/blob/master/addons/models/4000-MachineLearning/4010-ml_model.json#L90;
> 
> 2) I created a request of type EntityCreateRequestV2. It contains an entity 
> of type ml_model_build with an attribute "metadata", which is a map, and 
> contains key-value entries for "engineImageTag" and "engineImageName".
> 
> 3) Then I created a request of type EntityPartialUpdateRequestV2. It contains 
> an entity of type ml_model_build with an attribute "metadata", which is a 
> map, and contains key-value entry for "updated_at".
> 
> 4) In the properties of the entity ml_model_build, the * attribute 
> "*metadata" only contains key-value entry for "updated_at".
> 
> Desired behavior:
> 
> The attribute "metadata" should contains key-value entries for 
> "engineImageTag", "engineImageName" and "updated_at".
> 
> Current behavior:
> 
> the attribute "metadata" only contains key-value entry for "updated_at".
> 
> 
> Diffs
> -
> 
>   
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java
>  1434a24590c4f4172378f98d83ca8d9e9ec9c4d8 
>   
> intg/src/main/java/org/apache/atlas/model/notification/HookNotification.java 
> 5b5fa04e26b17071bdc2160974d7005bc110de74 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  75b016cca04801c4c1512c1527453fd59a11a6c4 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
>  225b72cbef20b2329b36f550da60c1bbe1f95efa 
>   server-api/src/main/java/org/apache/atlas/RequestContext.java 
> 282a66f1d466d58bce7a590513bc92f99733ba81 
>   
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  3f1ea05e17cded3433d050f891c807d55bdd022a 
> 
> 
> Diff: https://reviews.apache.org/r/72278/diff/4/
> 
> 
> Testing
> ---
> 
> add new test case for map attribute for partial update
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 72278: ATLAS-3700: Partial update to Atlas overwrite map entry with different keys

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72278/
---

(Updated March 30, 2020, 9:45 p.m.)


Review request for atlas, Ashutosh Mestry, Karthik Manamcheri, Sridhar K, and 
Sarath Subramanian.


Bugs: atlas-3700
https://issues.apache.org/jira/browse/atlas-3700


Repository: atlas


Description
---

1) The model definition is in 
"https://github.com/apache/atlas/blob/master/addons/models/4000-MachineLearning/4010-ml_model.json#L90;

2) I created a request of type EntityCreateRequestV2. It contains an entity of 
type ml_model_build with an attribute "metadata", which is a map, and contains 
key-value entries for "engineImageTag" and "engineImageName".

3) Then I created a request of type EntityPartialUpdateRequestV2. It contains 
an entity of type ml_model_build with an attribute "metadata", which is a map, 
and contains key-value entry for "updated_at".

4) In the properties of the entity ml_model_build, the * attribute "*metadata" 
only contains key-value entry for "updated_at".

Desired behavior:

The attribute "metadata" should contains key-value entries for 
"engineImageTag", "engineImageName" and "updated_at".

Current behavior:

the attribute "metadata" only contains key-value entry for "updated_at".


Diffs (updated)
-

  
intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java 
1434a24590c4f4172378f98d83ca8d9e9ec9c4d8 
  intg/src/main/java/org/apache/atlas/model/notification/HookNotification.java 
5b5fa04e26b17071bdc2160974d7005bc110de74 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
 75b016cca04801c4c1512c1527453fd59a11a6c4 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
 225b72cbef20b2329b36f550da60c1bbe1f95efa 
  server-api/src/main/java/org/apache/atlas/RequestContext.java 
282a66f1d466d58bce7a590513bc92f99733ba81 
  
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
 3f1ea05e17cded3433d050f891c807d55bdd022a 


Diff: https://reviews.apache.org/r/72278/diff/3/

Changes: https://reviews.apache.org/r/72278/diff/2-3/


Testing
---

add new test case for map attribute for partial update


Thanks,

Na Li

Re: Review Request 72278: ATLAS-3700: Partial update to Atlas overwrite map entry with different keys

[Na Li via Review Board]

> On March 30, 2020, 4:46 p.m., Karthik Manamcheri wrote:
> > You mention that the desired behavior is that the Map item should be 
> > appended (instead of overwritten). What do we do for the use-cases where 
> > the desired behavior is to over-write and not append? Does this change 
> > allow the flexibility to specify that?
> 
> Na Li wrote:
> The behavior is controled by the message type: 
> EntityPartialUpdateRequestV2. If the message type is EntityUpdateRequestV2, 
> overwriting will happen.
> 
> Madhan Neethiraj wrote:
> This change will cause existing use of EntityPartialUpdateRequestV2 to 
> break - as this will append to Map type attributes, instead of overwrite. To 
> avoid this issue, I suggest the following:
> - add a flag EntityPartialUpdateRequestV2.isAppendCollectionAttributes. 
> The default value should be false, which would retain the current behavior
> - add a flag in RequestContext.isAppendCollectionAttributes, and set this 
> value in NotificationHookConsumer where ENTITY_PARTIAL_UPDATE_V2 is handled
> - update EntityGraphMapper.mapMapValue() and 
> EntityGraphMapper.mapArrayValue() to perform append/overwrite depending on 
> value of RequestContext.isAppendCollectionAttributes

I have done the following

- add a flag EntityPartialUpdateRequestV2.isAppendCollectionAttributes. The 
default value should be false, which would retain the current behavior
- add a flag in RequestContext.isAppendCollectionAttributes, and set this value 
in NotificationHookConsumer where ENTITY_PARTIAL_UPDATE_V2 is handled
- update EntityGraphMapper.mapMapValue() to perform append/overwrite depending 
on value of RequestContext.isAppendCollectionAttributes

I did not update EntityGraphMapper.mapArrayValue() as it is not clear to me 
what's the reasonable behavior to append. In map, we have key in [key, value] 
to indicate what to append or replace.


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72278/#review220113
---


On March 30, 2020, 3:58 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72278/
> ---
> 
> (Updated March 30, 2020, 3:58 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Karthik Manamcheri, Sridhar K, and 
> Sarath Subramanian.
> 
> 
> Bugs: atlas-3700
> https://issues.apache.org/jira/browse/atlas-3700
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1) The model definition is in 
> "https://github.com/apache/atlas/blob/master/addons/models/4000-MachineLearning/4010-ml_model.json#L90;
> 
> 2) I created a request of type EntityCreateRequestV2. It contains an entity 
> of type ml_model_build with an attribute "metadata", which is a map, and 
> contains key-value entries for "engineImageTag" and "engineImageName".
> 
> 3) Then I created a request of type EntityPartialUpdateRequestV2. It contains 
> an entity of type ml_model_build with an attribute "metadata", which is a 
> map, and contains key-value entry for "updated_at".
> 
> 4) In the properties of the entity ml_model_build, the * attribute 
> "*metadata" only contains key-value entry for "updated_at".
> 
> Desired behavior:
> 
> The attribute "metadata" should contains key-value entries for 
> "engineImageTag", "engineImageName" and "updated_at".
> 
> Current behavior:
> 
> the attribute "metadata" only contains key-value entry for "updated_at".
> 
> 
> Diffs
> -
> 
>   
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java
>  1434a24590c4f4172378f98d83ca8d9e9ec9c4d8 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  75b016cca04801c4c1512c1527453fd59a11a6c4 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
>  225b72cbef20b2329b36f550da60c1bbe1f95efa 
> 
> 
> Diff: https://reviews.apache.org/r/72278/diff/2/
> 
> 
> Testing
> ---
> 
> add new test case for map attribute for partial update
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 72278: ATLAS-3700: Partial update to Atlas overwrite map entry with different keys

[Na Li via Review Board]

> On March 30, 2020, 4:59 p.m., Karthik Manamcheri wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
> > Lines 1301 (patched)
> > 
> >
> > You can use the .putAll function in HashMap. It does what you want it 
> > to do. It over-writes existing keys and adds non-existing keys.
> > 
> > ```
> >   HashMap currVal = ...
> >   // Add everything from newVal into currVal.
> >   // currVal becomes the merged map.
> >   currVal.putAll(newVal);
> >   ctx.getReferringVertex().setProperty(propertyName, currVal);
> > ```
> > 
> > 
> > https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html#putAll-java.util.Map-
> 
> Na Li wrote:
> currVal could be null
> 
> Karthik Manamcheri wrote:
> Sure then you can just check and create a new map right?
> 
> ```
> // If currVal is null, then there is nothing to merge with. Otherwise, 
> merge the new values
> // with current value.
> HashMap mergedVal = (currVal == null) ? newVal : 
> currVal.putAll(newVal);
> ctx.getReferringVertex().setProperty(propertyName, mergedVal);
> ```

Map newVal
Map mergedMap

Type does not match for putAll()


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72278/#review220114
---


On March 30, 2020, 3:58 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72278/
> ---
> 
> (Updated March 30, 2020, 3:58 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Karthik Manamcheri, Sridhar K, and 
> Sarath Subramanian.
> 
> 
> Bugs: atlas-3700
> https://issues.apache.org/jira/browse/atlas-3700
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1) The model definition is in 
> "https://github.com/apache/atlas/blob/master/addons/models/4000-MachineLearning/4010-ml_model.json#L90;
> 
> 2) I created a request of type EntityCreateRequestV2. It contains an entity 
> of type ml_model_build with an attribute "metadata", which is a map, and 
> contains key-value entries for "engineImageTag" and "engineImageName".
> 
> 3) Then I created a request of type EntityPartialUpdateRequestV2. It contains 
> an entity of type ml_model_build with an attribute "metadata", which is a 
> map, and contains key-value entry for "updated_at".
> 
> 4) In the properties of the entity ml_model_build, the * attribute 
> "*metadata" only contains key-value entry for "updated_at".
> 
> Desired behavior:
> 
> The attribute "metadata" should contains key-value entries for 
> "engineImageTag", "engineImageName" and "updated_at".
> 
> Current behavior:
> 
> the attribute "metadata" only contains key-value entry for "updated_at".
> 
> 
> Diffs
> -
> 
>   
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java
>  1434a24590c4f4172378f98d83ca8d9e9ec9c4d8 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  75b016cca04801c4c1512c1527453fd59a11a6c4 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
>  225b72cbef20b2329b36f550da60c1bbe1f95efa 
> 
> 
> Diff: https://reviews.apache.org/r/72278/diff/2/
> 
> 
> Testing
> ---
> 
> add new test case for map attribute for partial update
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 72278: ATLAS-3700: Partial update to Atlas overwrite map entry with different keys

[Na Li via Review Board]

> On March 30, 2020, 4:59 p.m., Karthik Manamcheri wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
> > Lines 1301 (patched)
> > 
> >
> > You can use the .putAll function in HashMap. It does what you want it 
> > to do. It over-writes existing keys and adds non-existing keys.
> > 
> > ```
> >   HashMap currVal = ...
> >   // Add everything from newVal into currVal.
> >   // currVal becomes the merged map.
> >   currVal.putAll(newVal);
> >   ctx.getReferringVertex().setProperty(propertyName, currVal);
> > ```
> > 
> > 
> > https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html#putAll-java.util.Map-

currVal could be null


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72278/#review220114
---


On March 30, 2020, 3:58 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72278/
> ---
> 
> (Updated March 30, 2020, 3:58 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Karthik Manamcheri, Sridhar K, and 
> Sarath Subramanian.
> 
> 
> Bugs: atlas-3700
> https://issues.apache.org/jira/browse/atlas-3700
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1) The model definition is in 
> "https://github.com/apache/atlas/blob/master/addons/models/4000-MachineLearning/4010-ml_model.json#L90;
> 
> 2) I created a request of type EntityCreateRequestV2. It contains an entity 
> of type ml_model_build with an attribute "metadata", which is a map, and 
> contains key-value entries for "engineImageTag" and "engineImageName".
> 
> 3) Then I created a request of type EntityPartialUpdateRequestV2. It contains 
> an entity of type ml_model_build with an attribute "metadata", which is a 
> map, and contains key-value entry for "updated_at".
> 
> 4) In the properties of the entity ml_model_build, the * attribute 
> "*metadata" only contains key-value entry for "updated_at".
> 
> Desired behavior:
> 
> The attribute "metadata" should contains key-value entries for 
> "engineImageTag", "engineImageName" and "updated_at".
> 
> Current behavior:
> 
> the attribute "metadata" only contains key-value entry for "updated_at".
> 
> 
> Diffs
> -
> 
>   
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java
>  1434a24590c4f4172378f98d83ca8d9e9ec9c4d8 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  75b016cca04801c4c1512c1527453fd59a11a6c4 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
>  225b72cbef20b2329b36f550da60c1bbe1f95efa 
> 
> 
> Diff: https://reviews.apache.org/r/72278/diff/2/
> 
> 
> Testing
> ---
> 
> add new test case for map attribute for partial update
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 72278: ATLAS-3700: Partial update to Atlas overwrite map entry with different keys

[Na Li via Review Board]

> On March 30, 2020, 4:46 p.m., Karthik Manamcheri wrote:
> > You mention that the desired behavior is that the Map item should be 
> > appended (instead of overwritten). What do we do for the use-cases where 
> > the desired behavior is to over-write and not append? Does this change 
> > allow the flexibility to specify that?

The behavior is controled by the message type: EntityPartialUpdateRequestV2. If 
the message type is EntityUpdateRequestV2, overwriting will happen.


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72278/#review220113
---


On March 30, 2020, 3:58 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72278/
> ---
> 
> (Updated March 30, 2020, 3:58 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Karthik Manamcheri, Sridhar K, and 
> Sarath Subramanian.
> 
> 
> Bugs: atlas-3700
> https://issues.apache.org/jira/browse/atlas-3700
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1) The model definition is in 
> "https://github.com/apache/atlas/blob/master/addons/models/4000-MachineLearning/4010-ml_model.json#L90;
> 
> 2) I created a request of type EntityCreateRequestV2. It contains an entity 
> of type ml_model_build with an attribute "metadata", which is a map, and 
> contains key-value entries for "engineImageTag" and "engineImageName".
> 
> 3) Then I created a request of type EntityPartialUpdateRequestV2. It contains 
> an entity of type ml_model_build with an attribute "metadata", which is a 
> map, and contains key-value entry for "updated_at".
> 
> 4) In the properties of the entity ml_model_build, the * attribute 
> "*metadata" only contains key-value entry for "updated_at".
> 
> Desired behavior:
> 
> The attribute "metadata" should contains key-value entries for 
> "engineImageTag", "engineImageName" and "updated_at".
> 
> Current behavior:
> 
> the attribute "metadata" only contains key-value entry for "updated_at".
> 
> 
> Diffs
> -
> 
>   
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java
>  1434a24590c4f4172378f98d83ca8d9e9ec9c4d8 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  75b016cca04801c4c1512c1527453fd59a11a6c4 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
>  225b72cbef20b2329b36f550da60c1bbe1f95efa 
> 
> 
> Diff: https://reviews.apache.org/r/72278/diff/2/
> 
> 
> Testing
> ---
> 
> add new test case for map attribute for partial update
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 72278: ATLAS-3700: Partial update to Atlas overwrite map entry with different keys

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72278/
---

(Updated March 30, 2020, 3:58 p.m.)


Review request for atlas, Ashutosh Mestry, Karthik Manamcheri, Sridhar K, and 
Sarath Subramanian.


Bugs: atlas-3700
https://issues.apache.org/jira/browse/atlas-3700


Repository: atlas


Description
---

1) The model definition is in 
"https://github.com/apache/atlas/blob/master/addons/models/4000-MachineLearning/4010-ml_model.json#L90;

2) I created a request of type EntityCreateRequestV2. It contains an entity of 
type ml_model_build with an attribute "metadata", which is a map, and contains 
key-value entries for "engineImageTag" and "engineImageName".

3) Then I created a request of type EntityPartialUpdateRequestV2. It contains 
an entity of type ml_model_build with an attribute "metadata", which is a map, 
and contains key-value entry for "updated_at".

4) In the properties of the entity ml_model_build, the * attribute "*metadata" 
only contains key-value entry for "updated_at".

Desired behavior:

The attribute "metadata" should contains key-value entries for 
"engineImageTag", "engineImageName" and "updated_at".

Current behavior:

the attribute "metadata" only contains key-value entry for "updated_at".


Diffs (updated)
-

  
intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java 
1434a24590c4f4172378f98d83ca8d9e9ec9c4d8 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
 75b016cca04801c4c1512c1527453fd59a11a6c4 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
 225b72cbef20b2329b36f550da60c1bbe1f95efa 


Diff: https://reviews.apache.org/r/72278/diff/2/

Changes: https://reviews.apache.org/r/72278/diff/1-2/


Testing
---

add new test case for map attribute for partial update


Thanks,

Na Li

Review Request 72278: ATLAS-3700: Partial update to Atlas overwrite map entry with different keys

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72278/
---

Review request for atlas, Ashutosh Mestry, Karthik Manamcheri, Sridhar K, and 
Sarath Subramanian.


Bugs: atlas-3700
https://issues.apache.org/jira/browse/atlas-3700


Repository: atlas


Description
---

1) The model definition is in 
"https://github.com/apache/atlas/blob/master/addons/models/4000-MachineLearning/4010-ml_model.json#L90;

2) I created a request of type EntityCreateRequestV2. It contains an entity of 
type ml_model_build with an attribute "metadata", which is a map, and contains 
key-value entries for "engineImageTag" and "engineImageName".

3) Then I created a request of type EntityPartialUpdateRequestV2. It contains 
an entity of type ml_model_build with an attribute "metadata", which is a map, 
and contains key-value entry for "updated_at".

4) In the properties of the entity ml_model_build, the * attribute "*metadata" 
only contains key-value entry for "updated_at".

Desired behavior:

The attribute "metadata" should contains key-value entries for 
"engineImageTag", "engineImageName" and "updated_at".

Current behavior:

the attribute "metadata" only contains key-value entry for "updated_at".


Diffs
-

  
intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java 
1434a24590c4f4172378f98d83ca8d9e9ec9c4d8 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
 75b016cca04801c4c1512c1527453fd59a11a6c4 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
 225b72cbef20b2329b36f550da60c1bbe1f95efa 


Diff: https://reviews.apache.org/r/72278/diff/1/


Testing
---

add new test case for map attribute for partial update


Thanks,

Na Li

Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance

[Na Li via Review Board]

> On Oct. 23, 2019, 10:20 p.m., Karthik Manamcheri wrote:
> > addons/models/4000-MachineLearning/4010-ml_model.json
> > Line 328 (original), 328 (patched)
> > 
> >
> > Why do we need this relationship? We can already trace the relationship 
> > through lineage?
> > 
> > Model Project -> Train Process -> Model Build
> > 
> > Same goes for the deployment. I don't think we need the relationship 
> > definition. Do we?
> > 
> > Model Build -> Deploy Process -> Model Deployment

That is what Atlas team preferred. It does no harm, and allow relationship 
graph to show.


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71619/#review218374
---


On Oct. 23, 2019, 9:22 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71619/
> ---
> 
> (Updated Oct. 23, 2019, 9:22 p.m.)
> 
> 
> Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, 
> Sridhar K, Madhan Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: atlas-3464
> https://issues.apache.org/jira/browse/atlas-3464
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Define entities used for Machine Learning Governance
> 
> 
> Diffs
> -
> 
>   addons/models/4000-MachineLearning/4010-ml_model.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/71619/diff/11/
> 
> 
> Testing
> ---
> 
> verified it is valid json file
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71619/
---

(Updated Oct. 23, 2019, 9:22 p.m.)


Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, 
Sridhar K, Madhan Neethiraj, and Sarath Subramanian.


Bugs: atlas-3464
https://issues.apache.org/jira/browse/atlas-3464


Repository: atlas


Description
---

Define entities used for Machine Learning Governance


Diffs (updated)
-

  addons/models/4000-MachineLearning/4010-ml_model.json PRE-CREATION 


Diff: https://reviews.apache.org/r/71619/diff/11/

Changes: https://reviews.apache.org/r/71619/diff/10-11/


Testing
---

verified it is valid json file


Thanks,

Na Li

Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71619/
---

(Updated Oct. 23, 2019, 6:29 p.m.)


Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, 
Sridhar K, Madhan Neethiraj, and Sarath Subramanian.


Bugs: atlas-3464
https://issues.apache.org/jira/browse/atlas-3464


Repository: atlas


Description
---

Define entities used for Machine Learning Governance


Diffs (updated)
-

  addons/models/4000-MachineLearning/4010-ml_model.json PRE-CREATION 


Diff: https://reviews.apache.org/r/71619/diff/10/

Changes: https://reviews.apache.org/r/71619/diff/9-10/


Testing
---

verified it is valid json file


Thanks,

Na Li

Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71619/
---

(Updated Oct. 22, 2019, 9:58 p.m.)


Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, 
Sridhar K, Madhan Neethiraj, and Sarath Subramanian.


Bugs: atlas-3464
https://issues.apache.org/jira/browse/atlas-3464


Repository: atlas


Description
---

Define entities used for Machine Learning Governance


Diffs (updated)
-

  addons/models/-Area0/0012-base_model.json PRE-CREATION 
  addons/models/4000-MachineLearning/4010-ml_model.json PRE-CREATION 


Diff: https://reviews.apache.org/r/71619/diff/9/

Changes: https://reviews.apache.org/r/71619/diff/8-9/


Testing
---

verified it is valid json file


Thanks,

Na Li

Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance

[Na Li via Review Board]

> On Oct. 22, 2019, 9:08 p.m., Sridhar K wrote:
> > addons/models/-Area0/0010-base_model.json
> > Lines 203 (patched)
> > 
> >
> > Instead of modifying addons/models/-Area0/0010-base_model.json, can 
> > you add this change in addons/models/-Area0/0011-base_model.json
> 
> Sridhar K wrote:
> Sorryinstead of modifying 
> addons/models/-Area0/0010-base_model.json, can you add this change in 
> addons/models/-Area0/0012-base_model.json
> 
> Sridhar K wrote:
> we want to add new changes model enhancements as new files with 
> increasing names.

it is done


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71619/#review218339
---


On Oct. 22, 2019, 9:58 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71619/
> ---
> 
> (Updated Oct. 22, 2019, 9:58 p.m.)
> 
> 
> Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, 
> Sridhar K, Madhan Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: atlas-3464
> https://issues.apache.org/jira/browse/atlas-3464
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Define entities used for Machine Learning Governance
> 
> 
> Diffs
> -
> 
>   addons/models/-Area0/0012-base_model.json PRE-CREATION 
>   addons/models/4000-MachineLearning/4010-ml_model.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/71619/diff/9/
> 
> 
> Testing
> ---
> 
> verified it is valid json file
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance

[Na Li via Review Board]

> On Oct. 22, 2019, 6:31 p.m., Sridhar K wrote:
> > addons/models/4000-MachineLearning/4010-ml_model.json
> > Lines 269 (patched)
> > 
> >
> > I am wondering if we should have a first class entity called "User" 
> > defined in base model and you have ml_user extend from it. what do you say?
> 
> Na Li wrote:
> what is parent for the entity "user"?

Made the changes 
1) define "AtlasUser" in base, derived from "Asset", it has "userName" attribute
2) define "ml_user", derived from "AtlasUser" and "DataSet". It only has 
"metaData" attibute


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71619/#review218332
---


On Oct. 22, 2019, 8:04 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71619/
> ---
> 
> (Updated Oct. 22, 2019, 8:04 p.m.)
> 
> 
> Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, 
> Sridhar K, Madhan Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: atlas-3464
> https://issues.apache.org/jira/browse/atlas-3464
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Define entities used for Machine Learning Governance
> 
> 
> Diffs
> -
> 
>   addons/models/-Area0/0010-base_model.json 2f5fdaf 
>   addons/models/4000-MachineLearning/4010-ml_model.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/71619/diff/8/
> 
> 
> Testing
> ---
> 
> verified it is valid json file
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71619/
---

(Updated Oct. 22, 2019, 8:04 p.m.)


Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, 
Sridhar K, Madhan Neethiraj, and Sarath Subramanian.


Bugs: atlas-3464
https://issues.apache.org/jira/browse/atlas-3464


Repository: atlas


Description
---

Define entities used for Machine Learning Governance


Diffs (updated)
-

  addons/models/-Area0/0010-base_model.json 2f5fdaf 
  addons/models/4000-MachineLearning/4010-ml_model.json PRE-CREATION 


Diff: https://reviews.apache.org/r/71619/diff/8/

Changes: https://reviews.apache.org/r/71619/diff/7-8/


Testing
---

verified it is valid json file


Thanks,

Na Li

Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance

[Na Li via Review Board]

> On Oct. 22, 2019, 5:43 p.m., Sridhar K wrote:
> > addons/models/4000-MachineLearning/4010-ml_model.json
> > Lines 46 (patched)
> > 
> >
> > is it different from Name?
> 
> Sridhar K wrote:
> We have name attribute already. Can we we use it instead?

good point. removed it.


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71619/#review218328
---


On Oct. 22, 2019, 3:42 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71619/
> ---
> 
> (Updated Oct. 22, 2019, 3:42 p.m.)
> 
> 
> Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, 
> Sridhar K, Madhan Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: atlas-3464
> https://issues.apache.org/jira/browse/atlas-3464
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Define entities used for Machine Learning Governance
> 
> 
> Diffs
> -
> 
>   addons/models/4000-MachineLearning/4010-ml_model.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/71619/diff/6/
> 
> 
> Testing
> ---
> 
> verified it is valid json file
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance

[Na Li via Review Board]

> On Oct. 22, 2019, 3:27 p.m., Karthik Manamcheri wrote:
> > addons/models/4000-MachineLearning/4010-ml_model.json
> > Lines 293 (patched)
> > 
> >
> > Should this also be an enum? What types are we envisioning?

removes this attributes as it is not clear how userful it is. User can specify 
the type in metadata attribute.


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71619/#review218322
---


On Oct. 22, 2019, 3:42 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71619/
> ---
> 
> (Updated Oct. 22, 2019, 3:42 p.m.)
> 
> 
> Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, 
> Sridhar K, Madhan Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: atlas-3464
> https://issues.apache.org/jira/browse/atlas-3464
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Define entities used for Machine Learning Governance
> 
> 
> Diffs
> -
> 
>   addons/models/4000-MachineLearning/4010-ml_model.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/71619/diff/6/
> 
> 
> Testing
> ---
> 
> verified it is valid json file
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71619/
---

(Updated Oct. 21, 2019, 6:51 p.m.)


Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, 
Sridhar K, Madhan Neethiraj, and Sarath Subramanian.


Bugs: atlas-3464
https://issues.apache.org/jira/browse/atlas-3464


Repository: atlas


Description
---

Define entities used for Machine Learning Governance


Diffs (updated)
-

  addons/models/4000-MachineLearning/4010-ml_model.json PRE-CREATION 


Diff: https://reviews.apache.org/r/71619/diff/5/

Changes: https://reviews.apache.org/r/71619/diff/4-5/


Testing
---

verified it is valid json file


Thanks,

Na Li

Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance

[Na Li via Review Board]

> On Oct. 21, 2019, 6:11 p.m., Sarath Subramanian wrote:
> > Move the ml model to a new directory in addons/models => 
> > 4000-MachineLearning
> > 
> > -ml_model.json => 4010-ml_model.json

done. Thanks!


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71619/#review218305
---


On Oct. 21, 2019, 6:51 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71619/
> ---
> 
> (Updated Oct. 21, 2019, 6:51 p.m.)
> 
> 
> Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, 
> Sridhar K, Madhan Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: atlas-3464
> https://issues.apache.org/jira/browse/atlas-3464
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Define entities used for Machine Learning Governance
> 
> 
> Diffs
> -
> 
>   addons/models/4000-MachineLearning/4010-ml_model.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/71619/diff/5/
> 
> 
> Testing
> ---
> 
> verified it is valid json file
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71619/
---

(Updated Oct. 21, 2019, 4:13 a.m.)


Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, 
Sridhar K, Madhan Neethiraj, and Sarath Subramanian.


Bugs: atlas-3464
https://issues.apache.org/jira/browse/atlas-3464


Repository: atlas


Description
---

Define entities used for Machine Learning Governance


Diffs (updated)
-

  addons/models/1000-Hadoop/-ml_model.json PRE-CREATION 


Diff: https://reviews.apache.org/r/71619/diff/4/

Changes: https://reviews.apache.org/r/71619/diff/3-4/


Testing
---

verified it is valid json file


Thanks,

Na Li

Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance

[Na Li via Review Board]

> On Oct. 18, 2019, 11:07 p.m., Anand Patil wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 27 (patched)
> > 
> >
> > Suggest aligning with the CML model deployment statuses for now: 
> > https://github.infra.cloudera.com/Sense/cloudera-sense/blob/master/services/proto/common.proto#L55
> 
> Na Li wrote:
> do you mean to change 
> 
> ml_model_deployment_status from
>   
> "elementDefs": [
> {
>   "value": "unknown",
>   "ordinal": 0
> },
> {
>   "value": "initializing",
>   "ordinal": 1
> },
> {
>   "value": "deployed",
>   "ordinal": 2
> },
> {
>   "value": "stopped",
>   "ordinal": 3
> }
>   ]
> 
> to  
>   "elementDefs": [
> {
>   "value": "deploying",
>   "ordinal": 0
> },
> {
>   "value": "deployed",
>   "ordinal": 1
> },
> {
>   "value": "stopping",
>   "ordinal": 2
> },
> {
>   "value": "stopped",
>   "ordinal": 3
> }
>   ]
>   
> ?
> 
> Anand Patil wrote:
> Yeah, but keep "unknown" too. What do you think?

agree.


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71619/#review218292
---


On Oct. 18, 2019, 2 a.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71619/
> ---
> 
> (Updated Oct. 18, 2019, 2 a.m.)
> 
> 
> Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, 
> Sridhar K, Madhan Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: atlas-3464
> https://issues.apache.org/jira/browse/atlas-3464
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Define entities used for Machine Learning Governance
> 
> 
> Diffs
> -
> 
>   addons/models/1000-Hadoop/-ml_model.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/71619/diff/3/
> 
> 
> Testing
> ---
> 
> verified it is valid json file
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance

[Na Li via Review Board]

> On Oct. 18, 2019, 11:07 p.m., Anand Patil wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 27 (patched)
> > 
> >
> > Suggest aligning with the CML model deployment statuses for now: 
> > https://github.infra.cloudera.com/Sense/cloudera-sense/blob/master/services/proto/common.proto#L55

do you mean to change 

ml_model_deployment_status from

"elementDefs": [
{
  "value": "unknown",
  "ordinal": 0
},
{
  "value": "initializing",
  "ordinal": 1
},
{
  "value": "deployed",
  "ordinal": 2
},
{
  "value": "stopped",
  "ordinal": 3
}
  ]

to
"elementDefs": [
{
  "value": "deploying",
  "ordinal": 0
},
{
  "value": "deployed",
  "ordinal": 1
},
{
  "value": "stopping",
  "ordinal": 2
},
{
  "value": "stopped",
  "ordinal": 3
}
  ]
  
?


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71619/#review218292
---


On Oct. 18, 2019, 2 a.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71619/
> ---
> 
> (Updated Oct. 18, 2019, 2 a.m.)
> 
> 
> Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, 
> Sridhar K, Madhan Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: atlas-3464
> https://issues.apache.org/jira/browse/atlas-3464
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Define entities used for Machine Learning Governance
> 
> 
> Diffs
> -
> 
>   addons/models/1000-Hadoop/-ml_model.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/71619/diff/3/
> 
> 
> Testing
> ---
> 
> verified it is valid json file
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71619/
---

(Updated Oct. 18, 2019, 2 a.m.)


Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, 
Sridhar K, Madhan Neethiraj, and Sarath Subramanian.


Bugs: atlas-3464
https://issues.apache.org/jira/browse/atlas-3464


Repository: atlas


Description
---

Define entities used for Machine Learning Governance


Diffs (updated)
-

  addons/models/1000-Hadoop/-ml_model.json PRE-CREATION 


Diff: https://reviews.apache.org/r/71619/diff/3/

Changes: https://reviews.apache.org/r/71619/diff/2-3/


Testing
---

verified it is valid json file


Thanks,

Na Li

Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance

[Na Li via Review Board]

> On Oct. 16, 2019, 4:28 p.m., Anand Patil wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 336 (patched)
> > 
> >
> > Like model builds, recommend adding an attribute for number of GPU's.

done


> On Oct. 16, 2019, 4:28 p.m., Anand Patil wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 339 (patched)
> > 
> >
> > Would this be things like pod ID's? If we want to expose this kind of 
> > low-level, debugging information then let's make it complete enough to 
> > support most debugging use cases. 
> > 
> > For example, in order to take action based on the pod ID's, a cluster 
> > admin would need to know the k8s API server API, the kubeconfig, and the 
> > namespace in which the model is running.
> > 
> > I think we could leave this field out for now, add an unstructured 
> > 'metadata' field and tackle the debugging flows later.

I removed it


> On Oct. 16, 2019, 4:28 p.m., Anand Patil wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 346 (patched)
> > 
> >
> > Should we add number of replicas?

done


> On Oct. 16, 2019, 4:28 p.m., Anand Patil wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 361 (patched)
> > 
> >
> > Should this be optional?

it is removed


> On Oct. 16, 2019, 4:28 p.m., Anand Patil wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 373 (patched)
> > 
> >
> > Is this for things like admin/full user/viewer only? If so, why make it 
> > non-optional?

changed it to optional. I am thinking the type is admin/user, mlgov/service etc.


> On Oct. 16, 2019, 4:28 p.m., Anand Patil wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 384-385 (patched)
> > 
> >
> > There currently is no such process in CML, projects are always created 
> > by user clicks in the UI.
> > 
> > Oh, is this just here to define lineage relationships betwene users and 
> > projects?

yes. The user is associated with a process, and the process creates a project. 
A user can have a lot of metadata. We don't want to duplicate the info to a 
project.


> On Oct. 16, 2019, 4:28 p.m., Anand Patil wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 415 (patched)
> > 
> >
> > Same question about whether uniqueId should really be optional.

removed.


> On Oct. 16, 2019, 4:28 p.m., Anand Patil wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 442 (patched)
> > 
> >
> > The name above is just ml_project_create_process.

This is relationship between ml_user and ml_project_create_process


> On Oct. 16, 2019, 4:28 p.m., Anand Patil wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 480 (patched)
> > 
> >
> > What's the reason for the ml_user prefix here and on 
> > ml_user_model_deploy_process?

it is a relationship involves two types


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71619/#review218231
---


On Oct. 16, 2019, 12:30 a.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71619/
> ---
> 
> (Updated Oct. 16, 2019, 12:30 a.m.)
> 
> 
> Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, 
> Sridhar K, Madhan Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: atlas-3464
> https://issues.apache.org/jira/browse/atlas-3464
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Define entities used for Machine Learning Governance
> 
> 
> Diffs
> -
> 
>   addons/models/1000-Hadoop/-ml_model.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/71619/diff/1/
> 
> 
> Testing
> ---
> 
> verified it is valid json file
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance

[Na Li via Review Board]

> On Oct. 16, 2019, 4:45 p.m., Karthik Manamcheri wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 286 (patched)
> > 
> >
> > What is the purpose of "updatedAt"?

I removed it. if user wants it, can add it in attribute "metadata"


> On Oct. 16, 2019, 4:45 p.m., Karthik Manamcheri wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 349 (patched)
> > 
> >
> > How is a ML User different from a regular user?

it is a user involved in ML actions. There is no base type we can re-use.

__AtlasUserProfile is used internally, totally different purpose from ours.


> On Oct. 16, 2019, 4:45 p.m., Karthik Manamcheri wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 447 (patched)
> > 
> >
> > I question the need for a ml_user at this point. I don't completely 
> > understand how or why a ml_user is different from any other user in the 
> > system?
> > 
> > Should this just be "user"?

I prefix each type with "ml_" to avoid type conflict. Other integration may 
change to use "user" for a totally different purpose. We do need to have a type 
for user.


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71619/#review218235
---


On Oct. 16, 2019, 12:30 a.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71619/
> ---
> 
> (Updated Oct. 16, 2019, 12:30 a.m.)
> 
> 
> Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, 
> Sridhar K, Madhan Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: atlas-3464
> https://issues.apache.org/jira/browse/atlas-3464
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Define entities used for Machine Learning Governance
> 
> 
> Diffs
> -
> 
>   addons/models/1000-Hadoop/-ml_model.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/71619/diff/1/
> 
> 
> Testing
> ---
> 
> verified it is valid json file
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance

[Na Li via Review Board]

> On Oct. 16, 2019, 7:36 p.m., Na Li wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 51 (patched)
> > 
> >
> > Those fields are optional. So if some projects cannot provide those 
> > fields, that is OK. The attributes I put here are used in several projects.
> > 
> > If we put several "key:value" pairs in a single string, we risk losing 
> > some info. 
> > 
> > I think the big drawback of putting several key:value pairs in a single 
> > json string is that:some can be lost by accident. For example the 
> > customAttributes = "{githubRepoURL:https://host1/project1 }". Then later 
> > on, an update comes in with customAttributes = "{mlFramework:tensorFlow}" 
> > because that application has no idea someone is tracking githubRepoURL, or 
> > did not set githubRepoURL because only mlFramework has changed. Then the 
> > previous info on where the source code is got lost by accident. It is hard 
> > to debug. Besides, each time someone wants to use the info, has to parse 
> > the string.
> > 
> > The counter argument can be how about putting these attributes here. If 
> > they are not popular, we just don't use them.
> > 
> > The following info is from Ashutosh. 
> > Having an attribute that contains json blob is not searchable at Atlas 
> > in general. 
> > 
> > Other benefit of adding an attribute to the model is that entities 
> > created will get validated for the type
> > because of validation, we can be certain about the data present in that 
> > field
> > 
> > A hacky approach to add json blob and make it searchable is to set them 
> > in customAttributes, which is system property. We cannot see what keys are 
> > used in customAttributes at model files. it is set by whoever creates the 
> > Atlas entity.
> 
> Anand Patil wrote:
> Instead of a string type containing json, what about a 
> map? I see that type used in some other attributes.

That is a really good idea without the risk of lossing info!

I tested Atlas behavior and Atlas can update part of an attribute's value if 
its type is map<>. I modified a HMS hook test in HiveMetastoreHookIT. 
Basically, the hive_table.parameters attribute's type is map. 
When I add another parameter after it has two parameters, all three parameters 
were stored and retrieved successfully.

I will reduce the attributes and have a general purpose attribute of 
map. So users can experiment and we can know what attributes 
are universally useful.


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71619/#review218237
---


On Oct. 16, 2019, 12:30 a.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71619/
> ---
> 
> (Updated Oct. 16, 2019, 12:30 a.m.)
> 
> 
> Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, 
> Sridhar K, Madhan Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: atlas-3464
> https://issues.apache.org/jira/browse/atlas-3464
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Define entities used for Machine Learning Governance
> 
> 
> Diffs
> -
> 
>   addons/models/1000-Hadoop/-ml_model.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/71619/diff/1/
> 
> 
> Testing
> ---
> 
> verified it is valid json file
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance

[Na Li via Review Board]

> On Oct. 16, 2019, 4:28 p.m., Anand Patil wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 165 (patched)
> > 
> >
> > Same comment about metadata: gitCommitId, hyperParameters, 
> > outcomeTypeDescription feel prematurely standardized. I feel that a 
> > map metadata field would be better suited to where we are 
> > now.

I will remove those attributes and add a general purpose attribute "metadata"


> On Oct. 16, 2019, 4:28 p.m., Anand Patil wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 175 (patched)
> > 
> >
> > It's slightly awkward to have to stringify all hyperparameter values. 
> > Does Atlas have option types?

it is removed


> On Oct. 16, 2019, 4:28 p.m., Anand Patil wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 197 (patched)
> > 
> >
> > If status is non-optional, it should probably be an enum or Altus 
> > equivalent.
> > 
> > Also, I'm not sure this field is required for model builds as opposed 
> > to model deployments. Builds' status is very simple - building, built or 
> > errored I suppose. We don't necessarily need to represent building or 
> > errored builds in Atlas.

It is removed.


> On Oct. 16, 2019, 4:28 p.m., Anand Patil wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 220 (patched)
> > 
> >
> > I'm not sure defaultReplicas belongs on the model build. The correct 
> > number of replicas depends on usage, it's not really a property of the 
> > model itself.

It is removed.


> On Oct. 16, 2019, 4:28 p.m., Anand Patil wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 228 (patched)
> > 
> >
> > "imageTag" would be more standard terminology. The tag typically 
> > includes the entire image URL.
> > 
> > Consider also adding the image hash, as tags/url's are not necessarily 
> > unique.

fixed


> On Oct. 16, 2019, 4:28 p.m., Anand Patil wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 268 (patched)
> > 
> >
> > We don't need a uniqueID for model deployments?

"qualifiedName" contains unique ID. no need for this.


> On Oct. 16, 2019, 4:28 p.m., Anand Patil wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 306 (patched)
> > 
> >
> > I don't think we shoud expose serviceIP. It may not be distinct from 
> > modelEndpointURL in non-Kubernetes serving environments and when it is, we 
> > may not want to encourage use of it. For example, when a model is being 
> > called from outside the k8s cluster where it is running, the service IP is 
> > not usable.

it is removed.


> On Oct. 16, 2019, 4:28 p.m., Anand Patil wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 319 (patched)
> > 
> >
> > If this is required I think it should be an enum or equivalent in Atlas.

changed to enum


> On Oct. 16, 2019, 4:28 p.m., Anand Patil wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 323 (patched)
> > 
> >
> > Use the same units for CPU and memory here as in the defaults for model 
> > build. I would go with millicores and bytes.

I changed the CPU unit to millicores. and still keep memory unit in Mb. Is a 
unit of bytes too small?


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71619/#review218231
---


On Oct. 16, 2019, 12:30 a.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71619/
> ---
> 
> (Updated Oct. 16, 2019, 12:30 a.m.)
> 
> 
> Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, 
> Sridhar K, Madhan Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: atlas-3464
> https://issues.apache.org/jira/browse/atlas-3464
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Define entities used for Machine Learning Governance
> 
> 
> Diffs
> -
> 
>   addons/models/1000-Hadoop/-ml_model.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/71619/diff/1/
> 
> 
> Testing
> ---
> 
> verified it is valid json file
> 
> 
> Thanks,
> 
> Na 

Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance

[Na Li via Review Board]

> On Oct. 16, 2019, 4:28 p.m., Anand Patil wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 10 (patched)
> > 
> >
> > Why make project a subtype of dataset?

Atlas server has a lot of code that build lineage based on Process and DataSet 
types

1) Process only handle types derives from `DataSet`
   {
  "name": "Process",
  "superTypes": [
"Asset"
  ],
  "serviceType": "atlas_core",
  "typeVersion": "1.1",
  "attributeDefs": [
{
  "name": "inputs",
  "typeName": "array",
  "cardinality": "SET",
  "isIndexable": false,
  "isOptional": true,
  "isUnique": false
},
{
  "name": "outputs",
  "typeName": "array",
  "cardinality": "SET",
  "isIndexable": false,
  "isOptional": true,
  "isUnique": false
}
  ]
}

2) Lineage is only built between Process type and DataSet type. 
https://github.com/apache/atlas/blob/master/repository/src/main/java/org/apache/atlas/discovery/EntityLineageService.java#L117
3) If we don't want to re-implement the lineage in Atlas, it is better to 
deriver action type from Process type, and data type from DataSet.


> On Oct. 16, 2019, 4:28 p.m., Anand Patil wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 51 (patched)
> > 
> >
> > businessUseCase, modelFramework, modelAlgorithms, githubRepoURL, 
> > notebookURL and resourceURL feel prematurely opinionated to me. Some 
> > projects will not use all these fields, and for some projects fields other 
> > than these will probably be more important.
> > 
> > Would it be possible in Atlas to move these to a single "metadata" 
> > attribute whose value is key-value pairs? We can then move to stronger 
> > typing as common patterns emerge.

replace those attributes by

{
  "name": "metadata",
  "description": "Contains key-value pairs that provide project 
metadata",
  "typeName": "map",
  "cardinality": "SINGLE",
  "isIndexable": false,
  "isOptional": true,
  "isUnique": false
}


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71619/#review218231
---


On Oct. 16, 2019, 12:30 a.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71619/
> ---
> 
> (Updated Oct. 16, 2019, 12:30 a.m.)
> 
> 
> Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, 
> Sridhar K, Madhan Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: atlas-3464
> https://issues.apache.org/jira/browse/atlas-3464
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Define entities used for Machine Learning Governance
> 
> 
> Diffs
> -
> 
>   addons/models/1000-Hadoop/-ml_model.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/71619/diff/1/
> 
> 
> Testing
> ---
> 
> verified it is valid json file
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance

[Na Li via Review Board]

> On Oct. 16, 2019, 4:28 p.m., Anand Patil wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 18 (patched)
> > 
> >
> > Can this really be optional?

I will drop this attribute, and user "qualifiedName", which must be unique.


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71619/#review218231
---


On Oct. 16, 2019, 12:30 a.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71619/
> ---
> 
> (Updated Oct. 16, 2019, 12:30 a.m.)
> 
> 
> Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, 
> Sridhar K, Madhan Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: atlas-3464
> https://issues.apache.org/jira/browse/atlas-3464
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Define entities used for Machine Learning Governance
> 
> 
> Diffs
> -
> 
>   addons/models/1000-Hadoop/-ml_model.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/71619/diff/1/
> 
> 
> Testing
> ---
> 
> verified it is valid json file
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance

[Na Li via Review Board]

> On Oct. 16, 2019, 7:36 p.m., Na Li wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 10 (patched)
> > 
> >
> > "DataSet" in Atlas has different meaning from the meaning in ML.  
> > 
> > DataSet: This type extends Referenceable and Asset. Conceptually, it 
> > can be used to represent an type that stores data. In Atlas, hive tables, 
> > Sqoop RDBMS tables etc are all types that extend from DataSet. Types that 
> > extend DataSet can be expected to have a Schema in the sense that they 
> > would have an attribute that defines attributes of that dataset. For e.g. 
> > the columns attribute in a hive_table. Also entities of types that extend 
> > DataSet participate in data transformation and this transformation can be 
> > captured by Atlas via lineage (or provenance) graphs. 
> > https://atlas.apache.org/0.8.1/TypeSystem.html
> 
> Anand Patil wrote:
> Hmm, it still feels like an odd fit to me, eg model builds and users 
> don't really have a schema. What would you think about instead making our 
> types inherit from Referenceable and Asset?

The type hierarchy is below

1) Asset derives from Referenceable
2) DataSet derives from Asset
3) Infrastructure derives from Asset
4) Process derives from Asset
5) ProcessExecution derives from Asset

It seems to me 
a) DataSet is used for storing data, 
b) Infrastructure is used as container, 
c) Process and ProcessExecution represent any data transformation operation or 
action

Deriving "model builds" or "users" from Asset directly violate this convention. 
I search the integration code from other modules, no type is derived from 
Referenceable or Asset directly.

{
  "name": "Referenceable",
  "superTypes": [],
  "serviceType": "atlas_core",
  "typeVersion": "1.0",
  "attributeDefs": [
{
  "name": "qualifiedName",
  "typeName": "string",
  "cardinality": "SINGLE",
  "isIndexable": true,
  "isOptional": false,
  "isUnique": true
}
  ]
},
{
  "name": "Asset",
  "superTypes": [
"Referenceable"
  ],
  "serviceType": "atlas_core",
  "typeVersion": "1.1",
  "attributeDefs": [
{
  "name": "name",
  "typeName": "string",
  "cardinality": "SINGLE",
  "isIndexable": true,
  "isOptional": false,
  "isUnique": false,
  "indexType": "STRING"
},
{
  "name": "DataSet",
  "superTypes": [
"Asset"
  ],
  "serviceType": "atlas_core",
  "typeVersion": "1.1",
  "attributeDefs": []
},
{
  "name": "Infrastructure",
  "description": "Infrastructure can be IT infrastructure, which contains 
hosts and servers. Infrastructure might not be IT orientated, such as 'Car' for 
IoT applications.",
  "superTypes": [
"Asset"
  ],
  "serviceType": "atlas_core",
  "typeVersion": "1.1",
  "attributeDefs": []
},
{
  "name": "Process",
  "superTypes": [
"Asset"
  ],
  "serviceType": "atlas_core",
  "typeVersion": "1.1",
  "attributeDefs": [
{
  "name": "inputs",
  "typeName": "array",
  "cardinality": "SET",
  "isIndexable": false,
  "isOptional": true,
  "isUnique": false
},
{
  "name": "outputs",
  "typeName": "array",
  "cardinality": "SET",
  "isIndexable": false,
  "isOptional": true,
  "isUnique": false
}
  ]
},
{
  "name": "ProcessExecution",
  "superTypes": [
"Asset"
  ],
  "serviceType": "atlas_core",
  "typeVersion": "1.0",
  "attributeDefs": []
}


> On Oct. 16, 2019, 7:36 p.m., Na Li wrote:
> > addons/models/1000-Hadoop/-ml_model.json
> > Lines 18 (patched)
> > 
> >
> > I don't want this to be optional. However, right now, some data source 
> > does not provide this info.
> 
> Anand Patil wrote:
> Could you give an example?

every Referenceable has an attribute "qualifiedName", which has to be unique. I 
am thinking we should get rid of this attribute "uniqueId", which duplicates 
the purpose of "qualifiedName". the attibute "name" can contain user-friendly 
name, and may not be unique.


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71619/#review218237
---


On Oct. 16, 2019, 12:30 a.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71619/
> ---
> 
> (Updated Oct. 16, 

Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71619/#review218238
---




addons/models/1000-Hadoop/-ml_model.json
Lines 135 (patched)


same answer as above



addons/models/1000-Hadoop/-ml_model.json
Lines 175 (patched)


do you want to define types that can be used for hyperparameters?

Can you give me some examples that we can define general purpose types to 
contain hyper parameters?



addons/models/1000-Hadoop/-ml_model.json
Lines 197 (patched)


the status can be "deprecated" for example. We need to look from ML 
operation point of view, not from how model build is created.

If operation engineer found some issue on a model build, which will then be 
marked as "deprecated", and some model may be in "dev" stage, and some are 
approved to be in "production". We need those info to help decide what model 
build to deploy.

For different deployment situations, the status can be of different values. 
Making it as enum may be too restrictive.



addons/models/1000-Hadoop/-ml_model.json
Lines 204 (patched)


good point. I will add the recommended gpu number. If it is 0, which means 
it does not need.



addons/models/1000-Hadoop/-ml_model.json
Lines 220 (patched)


OK. I will remove it.



addons/models/1000-Hadoop/-ml_model.json
Lines 228 (patched)


will add this attribute. Thanks!



addons/models/1000-Hadoop/-ml_model.json
Lines 248 (patched)


I am thinking about this. Will add example response to help caller process 
the response.



addons/models/1000-Hadoop/-ml_model.json
Lines 268 (patched)


We should. I intended to do that, but missed this one. Thanks!


- Na Li


On Oct. 16, 2019, 12:30 a.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71619/
> ---
> 
> (Updated Oct. 16, 2019, 12:30 a.m.)
> 
> 
> Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, 
> Sridhar K, Madhan Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: atlas-3464
> https://issues.apache.org/jira/browse/atlas-3464
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Define entities used for Machine Learning Governance
> 
> 
> Diffs
> -
> 
>   addons/models/1000-Hadoop/-ml_model.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/71619/diff/1/
> 
> 
> Testing
> ---
> 
> verified it is valid json file
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71619/#review218237
---




addons/models/1000-Hadoop/-ml_model.json
Lines 10 (patched)


"DataSet" in Atlas has different meaning from the meaning in ML.  

DataSet: This type extends Referenceable and Asset. Conceptually, it can be 
used to represent an type that stores data. In Atlas, hive tables, Sqoop RDBMS 
tables etc are all types that extend from DataSet. Types that extend DataSet 
can be expected to have a Schema in the sense that they would have an attribute 
that defines attributes of that dataset. For e.g. the columns attribute in a 
hive_table. Also entities of types that extend DataSet participate in data 
transformation and this transformation can be captured by Atlas via lineage (or 
provenance) graphs. https://atlas.apache.org/0.8.1/TypeSystem.html



addons/models/1000-Hadoop/-ml_model.json
Lines 18 (patched)


I don't want this to be optional. However, right now, some data source does 
not provide this info.



addons/models/1000-Hadoop/-ml_model.json
Lines 43 (patched)


One use case I can think of is when a user cannot access a project, it 
would be useful to know that the project is a private, and then the user should 
be a collaborator. If the project is public, then something else could go wrong.

For some business, it may be desirable for the project to be private.



addons/models/1000-Hadoop/-ml_model.json
Lines 51 (patched)


Those fields are optional. So if some projects cannot provide those fields, 
that is OK. The attributes I put here are used in several projects.

If we put several "key:value" pairs in a single string, we risk losing some 
info. 

I think the big drawback of putting several key:value pairs in a single 
json string is that:some can be lost by accident. For example the 
customAttributes = "{githubRepoURL:https://host1/project1 }". Then later on, an 
update comes in with customAttributes = "{mlFramework:tensorFlow}" because that 
application has no idea someone is tracking githubRepoURL, or did not set 
githubRepoURL because only mlFramework has changed. Then the previous info on 
where the source code is got lost by accident. It is hard to debug. Besides, 
each time someone wants to use the info, has to parse the string.

The counter argument can be how about putting these attributes here. If 
they are not popular, we just don't use them.

The following info is from Ashutosh. 
Having an attribute that contains json blob is not searchable at Atlas in 
general. 

Other benefit of adding an attribute to the model is that entities created 
will get validated for the type
because of validation, we can be certain about the data present in that 
field

A hacky approach to add json blob and make it searchable is to set them in 
customAttributes, which is system property. We cannot see what keys are used in 
customAttributes at model files. it is set by whoever creates the Atlas entity.


- Na Li


On Oct. 16, 2019, 12:30 a.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71619/
> ---
> 
> (Updated Oct. 16, 2019, 12:30 a.m.)
> 
> 
> Review request for atlas, Austin Nobis, Ashutosh Mestry, Karthik Manamcheri, 
> Sridhar K, Madhan Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: atlas-3464
> https://issues.apache.org/jira/browse/atlas-3464
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Define entities used for Machine Learning Governance
> 
> 
> Diffs
> -
> 
>   addons/models/1000-Hadoop/-ml_model.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/71619/diff/1/
> 
> 
> Testing
> ---
> 
> verified it is valid json file
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71619/
---

(Updated Oct. 15, 2019, 9:47 p.m.)


Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, and 
Sarath Subramanian.


Bugs: atlas-3464
https://issues.apache.org/jira/browse/atlas-3464


Repository: atlas


Description
---

Define entities used for Machine Learning Governance


Diffs
-

  addons/models/1000-Hadoop/-ml_model.json PRE-CREATION 


Diff: https://reviews.apache.org/r/71619/diff/1/


Testing
---

verified it is valid json file


Thanks,

Na Li

Review Request 71619: ATLAS-3464: Define Entities stored in Atlas for ML Governance

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71619/
---

Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, and 
Sarath Subramanian.


Repository: atlas


Description
---

Define entities used for Machine Learning Governance


Diffs
-

  addons/models/1000-Hadoop/-ml_model.json PRE-CREATION 


Diff: https://reviews.apache.org/r/71619/diff/1/


Testing
---

verified it is valid json file


Thanks,

Na Li

Re: Review Request 71568: ATLAS-3432: Add Machine Learning Governance integration

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71568/
---

(Updated Oct. 4, 2019, 8:21 p.m.)


Review request for atlas, Austin Nobis, Ashutosh Mestry, and Sarath Subramanian.


Repository: atlas


Description
---

add integration for ML


Diffs (updated)
-

  addons/hive-bridge/pom.xml 13384cb 
  addons/ml-bridge-shim/pom.xml PRE-CREATION 
  
addons/ml-bridge-shim/src/main/java/org/apache/atlas/ml/governance/hooks/GovernanceLineageHook.java
 PRE-CREATION 
  addons/ml-bridge/README.md PRE-CREATION 
  addons/ml-bridge/pom.xml PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/HookToolConfiguration.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/MlGovernanceHookTool.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/AtlasMlGovernanceProcessContext.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/GovernanceLineageHook.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/events/BaseMlGovernanceEvent.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/events/CreateMlGovernanceLineage.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlDataBase.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlDataType.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlLifecycleInfo.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModel.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModelBuild.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModelDeployment.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModelRef.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlProject.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlReplicaStatus.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlReplicationPolicy.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlReplicationStatus.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlResources.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlTrainingData.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlTrainingDataEntry.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlUser.java 
PRE-CREATION 
  addons/ml-bridge/src/main/resources/atlas-log4j.xml PRE-CREATION 
  addons/ml-bridge/src/main/resources/governance_hook.sh PRE-CREATION 
  
addons/ml-bridge/src/test/java/org/apache/atlas/ml/governance/GovernanceHookIT.java
 PRE-CREATION 
  
addons/ml-bridge/src/test/java/org/apache/atlas/ml/governance/GovernanceHookITBase.java
 PRE-CREATION 
  addons/ml-bridge/src/test/resources/atlas-application.properties PRE-CREATION 
  addons/ml-bridge/src/test/resources/atlas-log4j.xml PRE-CREATION 
  addons/ml-bridge/src/test/resources/mlModelDeployments.json PRE-CREATION 
  addons/ml-bridge/src/test/resources/mlModelDeploymentsTrainingData.json 
PRE-CREATION 
  addons/ml-bridge/src/test/resources/mlModels.json PRE-CREATION 
  addons/ml-bridge/src/test/resources/mlModelsTrainingData.json PRE-CREATION 
  addons/ml-bridge/src/test/resources/users-credentials.properties PRE-CREATION 
  addons/ml-hook-api/pom.xml PRE-CREATION 
  
addons/ml-hook-api/src/main/java/org/apache/mlops/governance/hooks/Entity.java 
PRE-CREATION 
  
addons/ml-hook-api/src/main/java/org/apache/mlops/governance/hooks/EntityType.java
 PRE-CREATION 
  
addons/ml-hook-api/src/main/java/org/apache/mlops/governance/hooks/GovernanceHook.java
 PRE-CREATION 
  
addons/ml-hook-api/src/main/java/org/apache/mlops/governance/hooks/HookContext.java
 PRE-CREATION 
  
addons/ml-hook-api/src/main/java/org/apache/mlops/governance/hooks/MethodType.java
 PRE-CREATION 
  addons/models/1000-Hadoop/1110-ml_model.json PRE-CREATION 
  pom.xml ce7d6fb 


Diff: https://reviews.apache.org/r/71568/diff/3/

Changes: https://reviews.apache.org/r/71568/diff/2-3/


Testing
---

add integration tests


Thanks,

Na Li

Re: Review Request 71568: ATLAS-3432: Add Machine Learning Governance integration

[Na Li via Review Board]

> On Oct. 3, 2019, 12:10 a.m., Austin Nobis wrote:
> > addons/ml-bridge-shim/src/main/java/org/apache/atlas/ml/governance/hooks/GovernanceLineageHook.java
> > Lines 35 (patched)
> > 
> >
> > How do we know the class name but need to load it via reflection? Where 
> > is this class defined and how is the JAR going to be included with Atlas at 
> > runtime?

the ml-bridge-shim module and ml-bridge module are implemented together. That 
is why we know the class name. The deployment will make sure the real implement 
jar will be put at the right folder. This is pattern in Atlas. It is to make 
sure the caller of the shim jar (which does not have real implementation, and 
only depends on Atlas classloader) won't load many Atlas classes and therefore 
causes issue for its class path.


> On Oct. 3, 2019, 12:10 a.m., Austin Nobis wrote:
> > addons/ml-bridge-shim/src/main/java/org/apache/atlas/ml/governance/hooks/GovernanceLineageHook.java
> > Lines 49 (patched)
> > 
> >
> > Is this style common in Atlas? I've seen it in Hive not sure about 
> > Atlas. If it is common why is it only present in this file and nowhere else?

I followed the style in hive-bridge. This style is used in functions that are 
entry point of processing, and makes the message stands out.


> On Oct. 3, 2019, 12:10 a.m., Austin Nobis wrote:
> > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/MlGovernanceHookTool.java
> > Lines 54-60 (patched)
> > 
> >
> > I think all of these could be placed into a Configuration class as 
> > opposed to bloating this class with configuration and functionality.

good point. done.


> On Oct. 3, 2019, 12:10 a.m., Austin Nobis wrote:
> > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/MlGovernanceHookTool.java
> > Lines 65 (patched)
> > 
> >
> > I think it is strange making references to closed source software 
> > (CDSW) in open source.

agree. it is replaced with general term.


> On Oct. 3, 2019, 12:10 a.m., Austin Nobis wrote:
> > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/MlGovernanceHookTool.java
> > Lines 94 (patched)
> > 
> >
> > I think it could cause issues if we return here instead of exiting with 
> > a non-zero exit code.

done.


> On Oct. 3, 2019, 12:10 a.m., Austin Nobis wrote:
> > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/MlGovernanceHookTool.java
> > Lines 106 (patched)
> > 
> >
> > I don't think this comment is useful. Add javadoc for this method if it 
> > is needed.

changed to javadoc


> On Oct. 3, 2019, 12:10 a.m., Austin Nobis wrote:
> > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/MlGovernanceHookTool.java
> > Lines 141 (patched)
> > 
> >
> > Remove comment or add java doc.

done


> On Oct. 3, 2019, 12:10 a.m., Austin Nobis wrote:
> > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/MlGovernanceHookTool.java
> > Lines 223 (patched)
> > 
> >
> > curl isn't installed on all machines by default. I think we should use 
> > the Java HTTP client to do this instead of spawning a separate process and 
> > using curl.

created "ATLAS-3440 Get ML metadata from ML service using HTTP client" to track 
this


> On Oct. 3, 2019, 12:10 a.m., Austin Nobis wrote:
> > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/MlGovernanceHookTool.java
> > Lines 226 (patched)
> > 
> >
> > Are we sending the unencrypted username and password? I don't think 
> > that is very secure. Is there no other way to do this communication?

this class is only used for testing. It should not be used for production. In 
production, the ML service will call the hook implementation to pass in the ML 
metadata.


> On Oct. 3, 2019, 12:10 a.m., Austin Nobis wrote:
> > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/AtlasMlGovernanceProcessContext.java
> > Lines 42 (patched)
> > 
> >
> > nit: why is there a big space after this.hook?

removed extra white space


> On Oct. 3, 2019, 12:10 a.m., Austin Nobis wrote:
> > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/AtlasMlGovernanceProcessContext.java
> > Lines 75 (patched)
> > 
> >

Re: Review Request 71568: ATLAS-3432: Add Machine Learning Governance integration

[Na Li via Review Board]

> On Oct. 3, 2019, 12:10 a.m., Austin Nobis wrote:
> > pom.xml
> > Lines 766 (patched)
> > 
> >
> > I would avoid using lombok if it is not already present in the project.
> 
> Na Li wrote:
> I am checking with Atlas team for preference

removed


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71568/#review218032
---


On Oct. 2, 2019, 11:31 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71568/
> ---
> 
> (Updated Oct. 2, 2019, 11:31 p.m.)
> 
> 
> Review request for atlas, Austin Nobis, Ashutosh Mestry, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> add integration for ML
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 13384cb 
>   addons/ml-bridge-shim/pom.xml PRE-CREATION 
>   
> addons/ml-bridge-shim/src/main/java/org/apache/atlas/ml/governance/hooks/GovernanceLineageHook.java
>  PRE-CREATION 
>   addons/ml-bridge/README.md PRE-CREATION 
>   addons/ml-bridge/pom.xml PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/MlGovernanceHookTool.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/AtlasMlGovernanceProcessContext.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/GovernanceLineageHook.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/events/BaseMlGovernanceEvent.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/events/CreateMlGovernanceLineage.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlDataBase.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlDataType.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlLifecycleInfo.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModel.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModelBuild.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModelDeployment.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModelRef.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlProject.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlReplicaStatus.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlReplicationPolicy.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlReplicationStatus.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlResources.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlTrainingData.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlTrainingDataEntry.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlUser.java
>  PRE-CREATION 
>   addons/ml-bridge/src/main/resources/atlas-log4j.xml PRE-CREATION 
>   addons/ml-bridge/src/main/resources/governance_hook.sh PRE-CREATION 
>   
> addons/ml-bridge/src/test/java/org/apache/atlas/ml/governance/GovernanceHookIT.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/test/java/org/apache/atlas/ml/governance/GovernanceHookITBase.java
>  PRE-CREATION 
>   addons/ml-bridge/src/test/resources/atlas-application.properties 
> PRE-CREATION 
>   addons/ml-bridge/src/test/resources/atlas-log4j.xml PRE-CREATION 
>   addons/ml-bridge/src/test/resources/cdswModelDeployments.json PRE-CREATION 
>   addons/ml-bridge/src/test/resources/cdswModelDeploymentsTrainingData.json 
> PRE-CREATION 
>   addons/ml-bridge/src/test/resources/cdswModels.json PRE-CREATION 
>   addons/ml-bridge/src/test/resources/cdswModelsTrainingData.json 
> PRE-CREATION 
>   addons/ml-bridge/src/test/resources/users-credentials.properties 
> PRE-CREATION 
>   addons/ml-hook-api/pom.xml PRE-CREATION 
>   
> addons/ml-hook-api/src/main/java/org/apache/mlops/governance/hooks/Entity.java
>  PRE-CREATION 
>   
> addons/ml-hook-api/src/main/java/org/apache/mlops/governance/hooks/EntityType.java
>  PRE-CREATION 
>   
> addons/ml-hook-api/src/main/java/org/apache/mlops/governance/hooks/GovernanceHook.java
>  PRE-CREATION 
>   
> 

Re: Review Request 71568: ATLAS-3432: Add Machine Learning Governance integration

[Na Li via Review Board]

> On Oct. 3, 2019, 4:55 a.m., radford nguyen wrote:
> > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/MlGovernanceHookTool.java
> > Lines 228 (patched)
> > 
> >
> > Apache commons has a built-in helper for printing usage text.  If you 
> > use it, then you don't have to duplicate the option descriptions in 
> > `printUsage`:
> > 
> > ```java
> > private static void printUsage(Options options) {
> > HelpFormatter formatter = new HelpFormatter();
> > formatter.printHelp("ml_hook.sh ", options);
> > }
> > ```
> > 
> > As an added bonus, if you ever change any options, you don't have to 
> > change `printUsage`

thanks for mentioning this. code is updated.


> On Oct. 3, 2019, 4:55 a.m., radford nguyen wrote:
> > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/AtlasMlGovernanceProcessContext.java
> > Lines 32 (patched)
> > 
> >
> > `hook` doesn't seem to be used in this class. Should it be removed?

done


> On Oct. 3, 2019, 4:55 a.m., radford nguyen wrote:
> > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/GovernanceLineageHook.java
> > Lines 125 (patched)
> > 
> >
> > This is a general commentary, but I am of the opinion that 
> > `AtlasMlGovernanceProcessContext` should be immutable, since it is passed 
> > around to many different consumers, and you generally want all of them to 
> > receive the exact same data (namely that which identifies the context).  
> > With a mutable object, the data can change at any point underneath you and 
> > you wouldn't know it, which opens up the possibility of bugs that would be 
> > hard to trace, especially in a multi-threaded environment.
> > 
> > With `AtlasMlGovernanceProcessContext` immutable, we would then 
> > refactor this method slightly to have the signature:
> > ```java
> > protected AtlasMlGovernanceProcessContext 
> > parseHookContext(HookContext context)
> > ```
> > The method signature itself would then identify the purpose of the 
> > method, which is to take a `HookContext` as _input_ and return a 
> > `AtlasMlGovernanceProcessContext` as _output_.

changed the signature.


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71568/#review218040
---


On Oct. 2, 2019, 11:31 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71568/
> ---
> 
> (Updated Oct. 2, 2019, 11:31 p.m.)
> 
> 
> Review request for atlas, Austin Nobis, Ashutosh Mestry, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> add integration for ML
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 13384cb 
>   addons/ml-bridge-shim/pom.xml PRE-CREATION 
>   
> addons/ml-bridge-shim/src/main/java/org/apache/atlas/ml/governance/hooks/GovernanceLineageHook.java
>  PRE-CREATION 
>   addons/ml-bridge/README.md PRE-CREATION 
>   addons/ml-bridge/pom.xml PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/MlGovernanceHookTool.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/AtlasMlGovernanceProcessContext.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/GovernanceLineageHook.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/events/BaseMlGovernanceEvent.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/events/CreateMlGovernanceLineage.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlDataBase.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlDataType.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlLifecycleInfo.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModel.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModelBuild.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModelDeployment.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModelRef.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlProject.java
>  PRE-CREATION 
>   
> 

Re: Review Request 71568: ATLAS-3432: Add Machine Learning Governance integration

[Na Li via Review Board]

> On Oct. 2, 2019, 7:56 p.m., Karthik Manamcheri wrote:
> > addons/ml-bridge-shim/src/main/java/org/apache/atlas/ml/governance/hooks/GovernanceLineageHook.java
> > Lines 28 (patched)
> > 
> >
> > This comment is confusing. What is the ML Governance server? I think 
> > that's an internal CDSW detail.
> 
> Na Li wrote:
> Does this comment works for you "This class is used to convert machine 
> learning metadata to lineage notifications
>  and send them to Atlas."?
>  
>  If not, what's your suggestion?
> 
> Karthik Manamcheri wrote:
> That works. Mention CDSW as well since this understands only CDSW-style 
> metadata. There is no universal machine learning metadata.
> 
> "This class is used to convert machine learning metadata from CDSW into 
> lineage notifications"

The feedback from Atlas team is to not mention CDSW.


> On Oct. 2, 2019, 7:56 p.m., Karthik Manamcheri wrote:
> > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlLifecycleInfo.java
> > Lines 33 (patched)
> > 
> >
> > A general comment applicable to your naming convention. Any object 
> > which represents a CDSW object should be prefixed with CDSW_something.
> > 
> > For example, this representation of "lifecycle" is very CDSW specific 
> > and we should call it that. For exmaple, CdswLifecycleInfo
> > 
> > This comment applies to all the objects which you are parsing out of 
> > CDSW Json.
> 
> Na Li wrote:
> the fields are in json string. Info from other sources can be formated to 
> a json string with those fields. That is why I named the classes with more 
> general names, not CDSW specific.

Based on feedback from Atlas team, we should not mention CDSW


> On Oct. 2, 2019, 7:56 p.m., Karthik Manamcheri wrote:
> > addons/models/1000-Hadoop/1110-ml_model.json
> > Lines 11 (patched)
> > 
> >
> > What is "serviceType" used for? In the future, if I want to extend this 
> > system for non-cdsw use-cases should I change this?

“serviceType” is a grouping of related entity typedefs e.g.“hive”, “impala”. I 
changed the service type to "ml". for other use-cases, we can use a different 
name.


> On Oct. 2, 2019, 7:56 p.m., Karthik Manamcheri wrote:
> > addons/models/1000-Hadoop/1110-ml_model.json
> > Lines 204 (patched)
> > 
> >
> > I don't think we should have training data represented here. We don't 
> > have a way to pull training data from CDSW. Let's not overdefine without 
> > implementation.

I removed the way to get training data.


> On Oct. 2, 2019, 7:56 p.m., Karthik Manamcheri wrote:
> > pom.xml
> > Lines 1574 (patched)
> > 
> >
> > Same goes here. This should be cdsw-bridge.

the hook API is general. should not be CDSW specific.


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71568/#review218027
---


On Oct. 2, 2019, 11:31 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71568/
> ---
> 
> (Updated Oct. 2, 2019, 11:31 p.m.)
> 
> 
> Review request for atlas, Austin Nobis, Ashutosh Mestry, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> add integration for ML
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 13384cb 
>   addons/ml-bridge-shim/pom.xml PRE-CREATION 
>   
> addons/ml-bridge-shim/src/main/java/org/apache/atlas/ml/governance/hooks/GovernanceLineageHook.java
>  PRE-CREATION 
>   addons/ml-bridge/README.md PRE-CREATION 
>   addons/ml-bridge/pom.xml PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/MlGovernanceHookTool.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/AtlasMlGovernanceProcessContext.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/GovernanceLineageHook.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/events/BaseMlGovernanceEvent.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/events/CreateMlGovernanceLineage.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlDataBase.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlDataType.java
>  PRE-CREATION 

Re: Review Request 71568: ATLAS-3432: Add Machine Learning Governance integration

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71568/
---

(Updated Oct. 2, 2019, 11:31 p.m.)


Review request for atlas, Austin Nobis, Ashutosh Mestry, and Sarath Subramanian.


Repository: atlas


Description
---

add integration for ML


Diffs (updated)
-

  addons/hive-bridge/pom.xml 13384cb 
  addons/ml-bridge-shim/pom.xml PRE-CREATION 
  
addons/ml-bridge-shim/src/main/java/org/apache/atlas/ml/governance/hooks/GovernanceLineageHook.java
 PRE-CREATION 
  addons/ml-bridge/README.md PRE-CREATION 
  addons/ml-bridge/pom.xml PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/MlGovernanceHookTool.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/AtlasMlGovernanceProcessContext.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/GovernanceLineageHook.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/events/BaseMlGovernanceEvent.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/events/CreateMlGovernanceLineage.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlDataBase.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlDataType.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlLifecycleInfo.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModel.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModelBuild.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModelDeployment.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModelRef.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlProject.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlReplicaStatus.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlReplicationPolicy.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlReplicationStatus.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlResources.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlTrainingData.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlTrainingDataEntry.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlUser.java 
PRE-CREATION 
  addons/ml-bridge/src/main/resources/atlas-log4j.xml PRE-CREATION 
  addons/ml-bridge/src/main/resources/governance_hook.sh PRE-CREATION 
  
addons/ml-bridge/src/test/java/org/apache/atlas/ml/governance/GovernanceHookIT.java
 PRE-CREATION 
  
addons/ml-bridge/src/test/java/org/apache/atlas/ml/governance/GovernanceHookITBase.java
 PRE-CREATION 
  addons/ml-bridge/src/test/resources/atlas-application.properties PRE-CREATION 
  addons/ml-bridge/src/test/resources/atlas-log4j.xml PRE-CREATION 
  addons/ml-bridge/src/test/resources/cdswModelDeployments.json PRE-CREATION 
  addons/ml-bridge/src/test/resources/cdswModelDeploymentsTrainingData.json 
PRE-CREATION 
  addons/ml-bridge/src/test/resources/cdswModels.json PRE-CREATION 
  addons/ml-bridge/src/test/resources/cdswModelsTrainingData.json PRE-CREATION 
  addons/ml-bridge/src/test/resources/users-credentials.properties PRE-CREATION 
  addons/ml-hook-api/pom.xml PRE-CREATION 
  
addons/ml-hook-api/src/main/java/org/apache/mlops/governance/hooks/Entity.java 
PRE-CREATION 
  
addons/ml-hook-api/src/main/java/org/apache/mlops/governance/hooks/EntityType.java
 PRE-CREATION 
  
addons/ml-hook-api/src/main/java/org/apache/mlops/governance/hooks/GovernanceHook.java
 PRE-CREATION 
  
addons/ml-hook-api/src/main/java/org/apache/mlops/governance/hooks/HookContext.java
 PRE-CREATION 
  
addons/ml-hook-api/src/main/java/org/apache/mlops/governance/hooks/MethodType.java
 PRE-CREATION 
  addons/models/1000-Hadoop/1110-ml_model.json PRE-CREATION 
  pom.xml ce7d6fb 


Diff: https://reviews.apache.org/r/71568/diff/2/

Changes: https://reviews.apache.org/r/71568/diff/1-2/


Testing
---

add integration tests


Thanks,

Na Li

Re: Review Request 71568: ATLAS-3432: Add Machine Learning Governance integration

[Na Li via Review Board]

> On Oct. 2, 2019, 7:56 p.m., Karthik Manamcheri wrote:
> > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/MlGovernanceHookTool.java
> > Lines 51 (patched)
> > 
> >
> > Where or why do we need this? and how do we use this. Please add 
> > comments here.

added comments


> On Oct. 2, 2019, 7:56 p.m., Karthik Manamcheri wrote:
> > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/GovernanceLineageHook.java
> > Lines 48 (patched)
> > 
> >
> > The "ML Governance server" is an internal detail of CDSW. We shouldn't 
> > add comments here about that. This should CDSWLineageHook.

In short term, it is. in long term, no


> On Oct. 2, 2019, 7:56 p.m., Karthik Manamcheri wrote:
> > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/GovernanceLineageHook.java
> > Lines 160 (patched)
> > 
> >
> > For all TODOs create JIRA tickets and attach them here.

ATLAS-3435 is created


> On Oct. 2, 2019, 7:56 p.m., Karthik Manamcheri wrote:
> > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/events/BaseMlGovernanceEvent.java
> > Lines 444 (patched)
> > 
> >
> > I don't think we should provide hacky implementations. Remove 
> > everything to do with training_data.

removed the hacky approach


> On Oct. 2, 2019, 7:56 p.m., Karthik Manamcheri wrote:
> > addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlLifecycleInfo.java
> > Lines 33 (patched)
> > 
> >
> > A general comment applicable to your naming convention. Any object 
> > which represents a CDSW object should be prefixed with CDSW_something.
> > 
> > For example, this representation of "lifecycle" is very CDSW specific 
> > and we should call it that. For exmaple, CdswLifecycleInfo
> > 
> > This comment applies to all the objects which you are parsing out of 
> > CDSW Json.

the fields are in json string. Info from other sources can be formated to a 
json string with those fields. That is why I named the classes with more 
general names, not CDSW specific.


> On Oct. 2, 2019, 7:56 p.m., Karthik Manamcheri wrote:
> > addons/ml-bridge/src/main/resources/atlas-log4j.xml
> > Lines 33 (patched)
> > 
> >
> > This will log ONLY from the class "MlGovernanceLineageHook". You should 
> > define a root logger to get logs out of the other classes as well.

I believe the behavior is that for GovernanceLineageHook, the log level is 
"info". For other classes, the log level is warning by default.every class can 
log messages. I changed the logger name to cover all atlas classes


> On Oct. 2, 2019, 7:56 p.m., Karthik Manamcheri wrote:
> > addons/ml-bridge/src/test/java/org/apache/atlas/ml/governance/GovernanceHookIT.java
> > Lines 35 (patched)
> > 
> >
> > What is "IT". Expand it. This is "IntegrationTest", right?

It is for integration test. Using "IT" is Atlas convention.


> On Oct. 2, 2019, 7:56 p.m., Karthik Manamcheri wrote:
> > addons/ml-hook-api/src/main/java/org/apache/mlops/governance/hooks/Entity.java
> > Lines 28 (patched)
> > 
> >
> > "Entity" has a specific meaning in Atlas. I think we should rename 
> > this. Maybe to "CdswObject" or something. As I understand, these are very 
> > specific CDSW defined objects, no?

This is not specific for CDSW. It contains just full name and json string.


> On Oct. 2, 2019, 7:56 p.m., Karthik Manamcheri wrote:
> > pom.xml
> > Lines 1568 (patched)
> > 
> >
> > I don't think we should call this "ml-hook-api". This is specifically 
> > targeting one product (CDSW) and so we should call this the cdsw-hook-api.

The hook is very general. Its data structure is not CDSW specific. Maybe we can 
call the hook implementation CDSW specific.


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71568/#review218027
---


On Oct. 1, 2019, 10:10 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71568/
> ---
> 
> (Updated Oct. 1, 2019, 10:10 p.m.)
> 
> 
> Review request for atlas, Austin Nobis, Ashutosh 

Re: Review Request 71568: ATLAS-3432: Add Machine Learning Governance integration

[Na Li via Review Board]

> On Oct. 2, 2019, 7:56 p.m., Karthik Manamcheri wrote:
> > addons/ml-bridge-shim/src/main/java/org/apache/atlas/ml/governance/hooks/GovernanceLineageHook.java
> > Lines 28 (patched)
> > 
> >
> > This comment is confusing. What is the ML Governance server? I think 
> > that's an internal CDSW detail.

Does this comment works for you "This class is used to convert machine learning 
metadata to lineage notifications
 and send them to Atlas."?
 
 If not, what's your suggestion?


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71568/#review218027
---


On Oct. 1, 2019, 10:10 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71568/
> ---
> 
> (Updated Oct. 1, 2019, 10:10 p.m.)
> 
> 
> Review request for atlas, Austin Nobis, Ashutosh Mestry, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> add integration for ML
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 13384cb 
>   addons/ml-bridge-shim/pom.xml PRE-CREATION 
>   
> addons/ml-bridge-shim/src/main/java/org/apache/atlas/ml/governance/hooks/GovernanceLineageHook.java
>  PRE-CREATION 
>   addons/ml-bridge/pom.xml PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/MlGovernanceHookTool.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/AtlasMlGovernanceProcessContext.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/GovernanceLineageHook.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/events/BaseMlGovernanceEvent.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/events/CreateMlGovernanceLineage.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlDataBase.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlDataType.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlLifecycleInfo.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModel.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModelBuild.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModelDeployment.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModelRef.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlProject.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlReplicaStatus.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlReplicationPolicy.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlReplicationStatus.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlResources.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlTrainingData.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlTrainingDataEntry.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlUser.java
>  PRE-CREATION 
>   addons/ml-bridge/src/main/resources/atlas-log4j.xml PRE-CREATION 
>   addons/ml-bridge/src/main/resources/governance_hook.sh PRE-CREATION 
>   
> addons/ml-bridge/src/test/java/org/apache/atlas/ml/governance/GovernanceHookIT.java
>  PRE-CREATION 
>   
> addons/ml-bridge/src/test/java/org/apache/atlas/ml/governance/GovernanceHookITBase.java
>  PRE-CREATION 
>   addons/ml-bridge/src/test/resources/atlas-application.properties 
> PRE-CREATION 
>   addons/ml-bridge/src/test/resources/atlas-log4j.xml PRE-CREATION 
>   addons/ml-bridge/src/test/resources/cdswModelDeployments.json PRE-CREATION 
>   addons/ml-bridge/src/test/resources/cdswModelDeploymentsTrainingData.json 
> PRE-CREATION 
>   addons/ml-bridge/src/test/resources/cdswModels.json PRE-CREATION 
>   addons/ml-bridge/src/test/resources/cdswModelsTrainingData.json 
> PRE-CREATION 
>   addons/ml-bridge/src/test/resources/users-credentials.properties 
> PRE-CREATION 
>   addons/ml-hook-api/pom.xml PRE-CREATION 
>   
> addons/ml-hook-api/src/main/java/org/apache/mlops/governance/hooks/Entity.java
>  PRE-CREATION 
>   
> 

Review Request 71568: ATLAS-3432: Add Machine Learning Governance integration

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71568/
---

Review request for atlas, Austin Nobis, Ashutosh Mestry, and Sarath Subramanian.


Repository: atlas


Description
---

add integration for ML


Diffs
-

  addons/hive-bridge/pom.xml 13384cb 
  addons/ml-bridge-shim/pom.xml PRE-CREATION 
  
addons/ml-bridge-shim/src/main/java/org/apache/atlas/ml/governance/hooks/GovernanceLineageHook.java
 PRE-CREATION 
  addons/ml-bridge/pom.xml PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/MlGovernanceHookTool.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/AtlasMlGovernanceProcessContext.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/GovernanceLineageHook.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/events/BaseMlGovernanceEvent.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/hooks/events/CreateMlGovernanceLineage.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlDataBase.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlDataType.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlLifecycleInfo.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModel.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModelBuild.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModelDeployment.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlModelRef.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlProject.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlReplicaStatus.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlReplicationPolicy.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlReplicationStatus.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlResources.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlTrainingData.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlTrainingDataEntry.java
 PRE-CREATION 
  
addons/ml-bridge/src/main/java/org/apache/atlas/ml/governance/model/MlUser.java 
PRE-CREATION 
  addons/ml-bridge/src/main/resources/atlas-log4j.xml PRE-CREATION 
  addons/ml-bridge/src/main/resources/governance_hook.sh PRE-CREATION 
  
addons/ml-bridge/src/test/java/org/apache/atlas/ml/governance/GovernanceHookIT.java
 PRE-CREATION 
  
addons/ml-bridge/src/test/java/org/apache/atlas/ml/governance/GovernanceHookITBase.java
 PRE-CREATION 
  addons/ml-bridge/src/test/resources/atlas-application.properties PRE-CREATION 
  addons/ml-bridge/src/test/resources/atlas-log4j.xml PRE-CREATION 
  addons/ml-bridge/src/test/resources/cdswModelDeployments.json PRE-CREATION 
  addons/ml-bridge/src/test/resources/cdswModelDeploymentsTrainingData.json 
PRE-CREATION 
  addons/ml-bridge/src/test/resources/cdswModels.json PRE-CREATION 
  addons/ml-bridge/src/test/resources/cdswModelsTrainingData.json PRE-CREATION 
  addons/ml-bridge/src/test/resources/users-credentials.properties PRE-CREATION 
  addons/ml-hook-api/pom.xml PRE-CREATION 
  
addons/ml-hook-api/src/main/java/org/apache/mlops/governance/hooks/Entity.java 
PRE-CREATION 
  
addons/ml-hook-api/src/main/java/org/apache/mlops/governance/hooks/EntityType.java
 PRE-CREATION 
  
addons/ml-hook-api/src/main/java/org/apache/mlops/governance/hooks/GovernanceHook.java
 PRE-CREATION 
  
addons/ml-hook-api/src/main/java/org/apache/mlops/governance/hooks/HookContext.java
 PRE-CREATION 
  
addons/ml-hook-api/src/main/java/org/apache/mlops/governance/hooks/MethodType.java
 PRE-CREATION 
  addons/models/1000-Hadoop/1110-ml_model.json PRE-CREATION 
  pom.xml ce7d6fb 


Diff: https://reviews.apache.org/r/71568/diff/1/


Testing
---

add integration tests


Thanks,

Na Li

Review Request 70880: ATLAS-3290: Impala Hook should get database name and table name from vertex metadata

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70880/
---

Review request for atlas, madhan and Sarath Subramanian.


Repository: atlas


Description
---

The column name in Impala lineage record may not contain its database name and 
its table name.

To get its its database name and its table name, we should use the metadata in 
a vertex, not assuming column name contains its database name and its table 
name.


Diffs
-

  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
 5b5f05a 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
 6d65ae0 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
 d437ba2 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
 6908191 
  
addons/impala-bridge/src/test/resources/impalaCreateTableAsSelectVertexIdNoTableName.json
 PRE-CREATION 


Diff: https://reviews.apache.org/r/70880/diff/1/


Testing
---

add new test case 
ImpalaLineageToolIT.testCreateTableAsSelectVertexIdNoTableNameFromFile for this 
scenario


Thanks,

Na Li

Re: Review Request 70759: ATLAS-3226: Add QueryText for hive_table and hive_db for Impala integration

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70759/
---

(Updated June 3, 2019, 7:08 p.m.)


Review request for atlas, madhan and Sarath Subramanian.


Repository: atlas


Description
---

ATLAS-3197 introduced a new feature on Hive integration. It introduced new 
model called Hive_table_ddl and hive_db_ddl which are used to track commands 
that are executed either on a table entity or bd entity. DDL entities will be 
added to a table/db entity through relationshipAttribute call HIVE_DB_TO_DDL 
and hive_tb_to_ddl. Every time when a new command is executed, a new ddl entity 
will be created, it will append to the existing ddlQueries field if a table/db 
has already been created. Once a table/db is deleted, all these relationship 
attributes will be deleted as well.

We need to support this new feature on Impala integration.


Diffs (updated)
-

  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
 7b4b893 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
 d241b6a 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java
 b7506a4 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaDataType.java
 10ce448 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
 b8cbf6b 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
 7f9a534 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
 6156208 
  addons/models/-Area0/0010-base_model.json d14b227 
  addons/models/1000-Hadoop/1030-hive_model.json 417c4c5 


Diff: https://reviews.apache.org/r/70759/diff/5/

Changes: https://reviews.apache.org/r/70759/diff/4-5/


Testing
---

add check for ddl entities in integration tests


Thanks,

Na Li

Re: Review Request 70759: ATLAS-3226: Add QueryText for hive_table and hive_db for Impala integration

[Na Li via Review Board]

> On May 31, 2019, 11:48 p.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
> > Lines 633 (patched)
> > 
> >
> > impalaDDL => hiveDDL

done


> On May 31, 2019, 11:48 p.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
> > Lines 646 (patched)
> > 
> >
> > add the corresponding entry in BaseHiveEvent to set serviceType to 
> > "hive" for hiveDDL entities

done


> On May 31, 2019, 11:48 p.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java
> > Lines 109 (patched)
> > 
> >
> > createImpalaDDLEntity => createHiveDDLEntity

done


> On May 31, 2019, 11:48 p.m., Sarath Subramanian wrote:
> > addons/models/-Area0/0010-base_model.json
> > Lines 88 (patched)
> > 
> >
> > 'serviceType' attribute is not defined here, value set in 
> > impalaDDL.setAttribute(ATTRIBUTE_SERVICE_TYPE, "impala"); will not be 
> > assigned to anything.

added


> On May 31, 2019, 11:48 p.m., Sarath Subramanian wrote:
> > addons/models/1000-Hadoop/1030-hive_model.json
> > Line 572 (original), 538 (patched)
> > 
> >
> > you don't need `db_ddl` as supertype, can consider adding supertype - 
> > "ddl" and remove types - db_ddl and table_ddl.

done


> On May 31, 2019, 11:48 p.m., Sarath Subramanian wrote:
> > addons/models/1000-Hadoop/1030-hive_model.json
> > Line 581 (original), 547 (patched)
> > 
> >
> > you don't need `table_ddl` as supertype, can consider adding supertype 
> > - "ddl"

done


> On May 31, 2019, 11:48 p.m., Sarath Subramanian wrote:
> > addons/models/1000-Hadoop/1030-hive_model.json
> > Line 727 (original), 693 (patched)
> > 
> >
> > type should still be "hive_table_ddl"

done


> On May 31, 2019, 11:48 p.m., Sarath Subramanian wrote:
> > addons/models/1000-Hadoop/1030-hive_model.json
> > Line 746 (original), 712 (patched)
> > 
> >
> > type should still be "hive_db_ddl"

done


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70759/#review215629
---


On May 31, 2019, 11:24 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70759/
> ---
> 
> (Updated May 31, 2019, 11:24 p.m.)
> 
> 
> Review request for atlas, madhan and Sarath Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-3197 introduced a new feature on Hive integration. It introduced new 
> model called Hive_table_ddl and hive_db_ddl which are used to track commands 
> that are executed either on a table entity or bd entity. DDL entities will be 
> added to a table/db entity through relationshipAttribute call HIVE_DB_TO_DDL 
> and hive_tb_to_ddl. Every time when a new command is executed, a new ddl 
> entity will be created, it will append to the existing ddlQueries field if a 
> table/db has already been created. Once a table/db is deleted, all these 
> relationship attributes will be deleted as well.
> 
> We need to support this new feature on Impala integration.
> 
> 
> Diffs
> -
> 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
>  d241b6a 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java
>  b7506a4 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaDataType.java
>  10ce448 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  b8cbf6b 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  7f9a534 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  6156208 
>   addons/models/-Area0/0010-base_model.json d14b227 
>   addons/models/1000-Hadoop/1030-hive_model.json 417c4c5 
> 
> 
> Diff: https://reviews.apache.org/r/70759/diff/4/
> 
> 
> Testing
> ---
> 
> add check for ddl entities in integration tests
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 70759: ATLAS-3226: Add QueryText for hive_table and hive_db for Impala integration

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70759/
---

(Updated May 31, 2019, 11:24 p.m.)


Review request for atlas, madhan and Sarath Subramanian.


Repository: atlas


Description
---

ATLAS-3197 introduced a new feature on Hive integration. It introduced new 
model called Hive_table_ddl and hive_db_ddl which are used to track commands 
that are executed either on a table entity or bd entity. DDL entities will be 
added to a table/db entity through relationshipAttribute call HIVE_DB_TO_DDL 
and hive_tb_to_ddl. Every time when a new command is executed, a new ddl entity 
will be created, it will append to the existing ddlQueries field if a table/db 
has already been created. Once a table/db is deleted, all these relationship 
attributes will be deleted as well.

We need to support this new feature on Impala integration.


Diffs (updated)
-

  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
 d241b6a 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java
 b7506a4 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaDataType.java
 10ce448 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
 b8cbf6b 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
 7f9a534 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
 6156208 
  addons/models/-Area0/0010-base_model.json d14b227 
  addons/models/1000-Hadoop/1030-hive_model.json 417c4c5 


Diff: https://reviews.apache.org/r/70759/diff/4/

Changes: https://reviews.apache.org/r/70759/diff/3-4/


Testing
---

add check for ddl entities in integration tests


Thanks,

Na Li

Re: Review Request 70759: ATLAS-3226: Add QueryText for hive_table and hive_db for Impala integration

[Na Li via Review Board]

> On May 31, 2019, 8:16 p.m., Sarath Subramanian wrote:
> > addons/models/1000-Hadoop/1090-impala_model.json
> > Lines 193 (patched)
> > 
> >
> > we don't need to define a new impala_db_ddl and impala_table_ddl, since 
> > impala ddls are hive ddls. We can add an attribute in hive_db_ddl and 
> > hive_table_ddl to specify the application/service which issued the DDL 
> > (Hive or Impala). Please review.

removed


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70759/#review215623
---


On May 31, 2019, 4:02 a.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70759/
> ---
> 
> (Updated May 31, 2019, 4:02 a.m.)
> 
> 
> Review request for atlas, madhan and Sarath Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-3197 introduced a new feature on Hive integration. It introduced new 
> model called Hive_table_ddl and hive_db_ddl which are used to track commands 
> that are executed either on a table entity or bd entity. DDL entities will be 
> added to a table/db entity through relationshipAttribute call HIVE_DB_TO_DDL 
> and hive_tb_to_ddl. Every time when a new command is executed, a new ddl 
> entity will be created, it will append to the existing ddlQueries field if a 
> table/db has already been created. Once a table/db is deleted, all these 
> relationship attributes will be deleted as well.
> 
> We need to support this new feature on Impala integration.
> 
> 
> Diffs
> -
> 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
>  d241b6a 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java
>  b7506a4 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaDataType.java
>  10ce448 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  b8cbf6b 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  7f9a534 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  6156208 
>   addons/models/-Area0/0010-base_model.json d14b227 
>   addons/models/1000-Hadoop/1030-hive_model.json 417c4c5 
>   addons/models/1000-Hadoop/1090-impala_model.json 24eb5db 
> 
> 
> Diff: https://reviews.apache.org/r/70759/diff/3/
> 
> 
> Testing
> ---
> 
> add check for ddl entities in integration tests
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 70759: ATLAS-3226: Add QueryText for hive_table and hive_db for Impala integration

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70759/#review215608
---




addons/models/1000-Hadoop/1030-hive_model.json
Line 727 (original), 693 (patched)


has to use common type for "hive_table_ddl" and "impala_table_ddl" in order 
for it to work for both hive and impala



addons/models/1000-Hadoop/1030-hive_model.json
Line 746 (original), 712 (patched)


the endpoint of the relationship has to be the common type of hive_db_ddl 
and impala_db_ddl in order for Atlas server to update the "ddlQueries" of a 
hive_db for both hive_db_ddl and impala_db_ddl


- Na Li


On May 31, 2019, 4:02 a.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70759/
> ---
> 
> (Updated May 31, 2019, 4:02 a.m.)
> 
> 
> Review request for atlas, madhan and Sarath Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-3197 introduced a new feature on Hive integration. It introduced new 
> model called Hive_table_ddl and hive_db_ddl which are used to track commands 
> that are executed either on a table entity or bd entity. DDL entities will be 
> added to a table/db entity through relationshipAttribute call HIVE_DB_TO_DDL 
> and hive_tb_to_ddl. Every time when a new command is executed, a new ddl 
> entity will be created, it will append to the existing ddlQueries field if a 
> table/db has already been created. Once a table/db is deleted, all these 
> relationship attributes will be deleted as well.
> 
> We need to support this new feature on Impala integration.
> 
> 
> Diffs
> -
> 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
>  d241b6a 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java
>  b7506a4 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaDataType.java
>  10ce448 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  b8cbf6b 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  7f9a534 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  6156208 
>   addons/models/-Area0/0010-base_model.json d14b227 
>   addons/models/1000-Hadoop/1030-hive_model.json 417c4c5 
>   addons/models/1000-Hadoop/1090-impala_model.json 24eb5db 
> 
> 
> Diff: https://reviews.apache.org/r/70759/diff/3/
> 
> 
> Testing
> ---
> 
> add check for ddl entities in integration tests
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 70759: ATLAS-3226: Add QueryText for hive_table and hive_db for Impala integration

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70759/
---

(Updated May 31, 2019, 4:02 a.m.)


Review request for atlas, madhan and Sarath Subramanian.


Repository: atlas


Description
---

ATLAS-3197 introduced a new feature on Hive integration. It introduced new 
model called Hive_table_ddl and hive_db_ddl which are used to track commands 
that are executed either on a table entity or bd entity. DDL entities will be 
added to a table/db entity through relationshipAttribute call HIVE_DB_TO_DDL 
and hive_tb_to_ddl. Every time when a new command is executed, a new ddl entity 
will be created, it will append to the existing ddlQueries field if a table/db 
has already been created. Once a table/db is deleted, all these relationship 
attributes will be deleted as well.

We need to support this new feature on Impala integration.


Diffs (updated)
-

  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
 d241b6a 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java
 b7506a4 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaDataType.java
 10ce448 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
 b8cbf6b 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
 7f9a534 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
 6156208 
  addons/models/-Area0/0010-base_model.json d14b227 
  addons/models/1000-Hadoop/1030-hive_model.json 417c4c5 
  addons/models/1000-Hadoop/1090-impala_model.json 24eb5db 


Diff: https://reviews.apache.org/r/70759/diff/3/

Changes: https://reviews.apache.org/r/70759/diff/2-3/


Testing
---

add check for ddl entities in integration tests


Thanks,

Na Li

Re: Review Request 70759: ATLAS-3226: Add QueryText for hive_table and hive_db for Impala integration

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70759/
---

(Updated May 30, 2019, 5:06 p.m.)


Review request for atlas, madhan and Sarath Subramanian.


Repository: atlas


Description
---

ATLAS-3197 introduced a new feature on Hive integration. It introduced new 
model called Hive_table_ddl and hive_db_ddl which are used to track commands 
that are executed either on a table entity or bd entity. DDL entities will be 
added to a table/db entity through relationshipAttribute call HIVE_DB_TO_DDL 
and hive_tb_to_ddl. Every time when a new command is executed, a new ddl entity 
will be created, it will append to the existing ddlQueries field if a table/db 
has already been created. Once a table/db is deleted, all these relationship 
attributes will be deleted as well.

We need to support this new feature on Impala integration.


Diffs (updated)
-

  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
 d241b6a 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java
 b7506a4 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaDataType.java
 10ce448 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
 b8cbf6b 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
 7f9a534 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
 6156208 
  addons/models/1000-Hadoop/1090-impala_model.json 24eb5db 


Diff: https://reviews.apache.org/r/70759/diff/2/

Changes: https://reviews.apache.org/r/70759/diff/1-2/


Testing
---

add check for ddl entities in integration tests


Thanks,

Na Li

Re: Review Request 70708: ATLAS-3185: Create process execution for Impala integration

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70708/
---

(Updated May 29, 2019, 3:29 p.m.)


Review request for atlas, Aadarsh Jajodia, Madhan Neethiraj, and Sarath 
Subramanian.


Repository: atlas


Description
---

ATLAS-3133 adds a new feature to track metadata for different executions of the 
same process in Atlas. Need to add this in Impala integration


Diffs (updated)
-

  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/ImpalaLineageTool.java
 7c9abc8 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
 88faace 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaIdentifierParser.java
 b9d6cbb 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
 232a569 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
 c7604ba 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java
 3071576 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
 0138d88 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
 033a518 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
 86801e3 
  
addons/impala-bridge/src/test/resources/impalaMultipleInsertIntoAsSelect1.json 
PRE-CREATION 
  
addons/impala-bridge/src/test/resources/impalaMultipleInsertIntoAsSelect2.json 
PRE-CREATION 


Diff: https://reviews.apache.org/r/70708/diff/3/

Changes: https://reviews.apache.org/r/70708/diff/2-3/


Testing
---

Integration tests


Thanks,

Na Li

Re: Review Request 70708: ATLAS-3185: Create process execution for Impala integration

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70708/
---

(Updated May 28, 2019, 9:19 p.m.)


Review request for atlas, Aadarsh Jajodia, Madhan Neethiraj, and Sarath 
Subramanian.


Repository: atlas


Description
---

ATLAS-3133 adds a new feature to track metadata for different executions of the 
same process in Atlas. Need to add this in Impala integration


Diffs (updated)
-

  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/ImpalaLineageTool.java
 7c9abc8 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
 88faace 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaIdentifierParser.java
 b9d6cbb 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
 232a569 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
 c7604ba 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java
 3071576 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
 0138d88 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
 033a518 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
 86801e3 
  
addons/impala-bridge/src/test/resources/impalaMultipleInsertIntoAsSelect1.json 
PRE-CREATION 
  
addons/impala-bridge/src/test/resources/impalaMultipleInsertIntoAsSelect2.json 
PRE-CREATION 


Diff: https://reviews.apache.org/r/70708/diff/2/

Changes: https://reviews.apache.org/r/70708/diff/1-2/


Testing
---

Integration tests


Thanks,

Na Li

Re: Review Request 70708: ATLAS-3185: Create process execution for Impala integration

[Na Li via Review Board]

> On May 24, 2019, 8:14 p.m., Aadarsh Jajodia wrote:
> > addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
> > Lines 201-205 (patched)
> > 
> >
> > Can we also add a unit test which produces more than one execution?

added new test with multiple process executions


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70708/#review215511
---


On May 28, 2019, 9:19 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70708/
> ---
> 
> (Updated May 28, 2019, 9:19 p.m.)
> 
> 
> Review request for atlas, Aadarsh Jajodia, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-3133 adds a new feature to track metadata for different executions of 
> the same process in Atlas. Need to add this in Impala integration
> 
> 
> Diffs
> -
> 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/ImpalaLineageTool.java
>  7c9abc8 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
>  88faace 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaIdentifierParser.java
>  b9d6cbb 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  232a569 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
>  c7604ba 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java
>  3071576 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  0138d88 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  033a518 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  86801e3 
>   
> addons/impala-bridge/src/test/resources/impalaMultipleInsertIntoAsSelect1.json
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/resources/impalaMultipleInsertIntoAsSelect2.json
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/70708/diff/2/
> 
> 
> Testing
> ---
> 
> Integration tests
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 70708: ATLAS-3185: Create process execution for Impala integration

[Na Li via Review Board]

> On May 28, 2019, 5:58 p.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
> > Lines 23 (patched)
> > 
> >
> > nit: unused imports, consider removing them.

done


> On May 28, 2019, 5:58 p.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
> > Lines 565 (patched)
> > 
> >
> > cast startTime as string value; same for line #568

directly casting it to long does not work. I change the type to "Long", and 
call its.toString() instead


> On May 28, 2019, 5:58 p.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java
> > Line 114 (original), 114 (patched)
> > 
> >
> > surround with LOG.isDebugEnabled(); same for line #121

see comment at 
https://stackoverflow.com/questions/105852/conditional-logging-with-minimal-cyclomatic-complexity/105908#105908
 and 
`https://stackoverflow.com/questions/963492/in-log4j-does-checking-isdebugenabled-before-logging-improve-performance`

`Better yet is to use a more up-to-date logging framework where the log 
statements take a format specification and a list of arguments to be 
substituted by the logger—but "lazily," only if the logger is enabled.` 
that is what I did. The overhead only happens if debug logging is enabled. So I 
don't need to check  `LOG.isDebugEnabled()`


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70708/#review215546
---


On May 23, 2019, 9:11 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70708/
> ---
> 
> (Updated May 23, 2019, 9:11 p.m.)
> 
> 
> Review request for atlas, Aadarsh Jajodia, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-3133 adds a new feature to track metadata for different executions of 
> the same process in Atlas. Need to add this in Impala integration
> 
> 
> Diffs
> -
> 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
>  88faace 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  232a569 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
>  63c5f87 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java
>  0dc520c 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  0138d88 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  033a518 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  86801e3 
> 
> 
> Diff: https://reviews.apache.org/r/70708/diff/1/
> 
> 
> Testing
> ---
> 
> Integration tests
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 70708: ATLAS-3185: Create process execution for Impala integration

[Na Li via Review Board]

> On May 24, 2019, 8:14 p.m., Aadarsh Jajodia wrote:
> > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
> > Lines 561-562 (patched)
> > 
> >
> > Maybe use long for both?

fixed


> On May 24, 2019, 8:14 p.m., Aadarsh Jajodia wrote:
> > addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java
> > Lines 120 (patched)
> > 
> >
> > Change to processExecution != null

good catch!


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70708/#review215511
---


On May 23, 2019, 9:11 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70708/
> ---
> 
> (Updated May 23, 2019, 9:11 p.m.)
> 
> 
> Review request for atlas, Aadarsh Jajodia, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-3133 adds a new feature to track metadata for different executions of 
> the same process in Atlas. Need to add this in Impala integration
> 
> 
> Diffs
> -
> 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
>  88faace 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  232a569 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
>  63c5f87 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java
>  0dc520c 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  0138d88 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  033a518 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  86801e3 
> 
> 
> Diff: https://reviews.apache.org/r/70708/diff/1/
> 
> 
> Testing
> ---
> 
> Integration tests
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 70710: ATLAS-3230: Change Impala Hook API Names

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70710/
---

(Updated May 24, 2019, 5:25 p.m.)


Review request for atlas and Sarath Subramanian.


Repository: atlas


Description
---

Impala has made the following changes:

1) "createTime" to "tableCreateTime"
2) "PostQueryHookContext" to "QueryCompleteContext"
3) "QueryExecHook" to "QueryEventHook"

Atlas needs to make corresponding changes to keep end-to-end work.


Diffs (updated)
-

  
addons/impala-bridge-shim/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
 b796f37 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaHook.java 
2afb4eb 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
 63c5f87 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java
 0dc520c 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/LineageVertexMetadata.java
 0bd236d 
  addons/impala-bridge/src/test/resources/impalaAlterViewAsSelect.json f8de7e3 
  addons/impala-bridge/src/test/resources/impalaCreateTableAsSelect.json 
b6ac516 
  addons/impala-bridge/src/test/resources/impalaCreateView.json ffdc658 
  addons/impala-bridge/src/test/resources/impalaInsertIntoAsSelect.json 140eced 
  
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/PostQueryHookContext.java
 c8a5b64 
  
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/QueryCompleteContext.java
 PRE-CREATION 
  
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/QueryEventHook.java
 PRE-CREATION 
  
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/QueryExecHook.java 
a7d84d4 


Diff: https://reviews.apache.org/r/70710/diff/3/

Changes: https://reviews.apache.org/r/70710/diff/2-3/


Testing
---


Thanks,

Na Li

Re: Review Request 70710: ATLAS-3230: Change Impala Hook API Names

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70710/
---

(Updated May 24, 2019, 2:49 a.m.)


Review request for atlas and Sarath Subramanian.


Repository: atlas


Description
---

Impala has made the following changes:

1) "createTime" to "tableCreateTime"
2) "PostQueryHookContext" to "QueryCompleteContext"
3) "QueryExecHook" to "QueryEventHook"

Atlas needs to make corresponding changes to keep end-to-end work.


Diffs (updated)
-

  
addons/impala-bridge-shim/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
 b796f37 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaHook.java 
2afb4eb 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
 63c5f87 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java
 0dc520c 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/LineageVertexMetadata.java
 0bd236d 
  
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/PostQueryHookContext.java
 c8a5b64 
  
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/QueryCompleteContext.java
 PRE-CREATION 
  
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/QueryEventHook.java
 PRE-CREATION 
  
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/QueryExecHook.java 
a7d84d4 


Diff: https://reviews.apache.org/r/70710/diff/2/

Changes: https://reviews.apache.org/r/70710/diff/1-2/


Testing
---


Thanks,

Na Li

Review Request 70710: ATLAS-3230: Change Impala Hook API Names

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70710/
---

Review request for atlas and Sarath Subramanian.


Repository: atlas


Description
---

Impala has made the following changes:

1) "createTime" to "tableCreateTime"
2) "PostQueryHookContext" to "QueryCompleteContext"
3) "QueryExecHook" to "QueryEventHook"

Atlas needs to make corresponding changes to keep end-to-end work.


Diffs
-

  
addons/impala-bridge-shim/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
 b796f37 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaHook.java 
2afb4eb 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
 63c5f87 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java
 0dc520c 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/LineageVertexMetadata.java
 0bd236d 
  
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/PostQueryHookContext.java
 c8a5b64 
  
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/QueryCompleteContext.java
 PRE-CREATION 
  
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/QueryEventHook.java
 PRE-CREATION 
  
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/QueryExecHook.java 
a7d84d4 


Diff: https://reviews.apache.org/r/70710/diff/1/


Testing
---


Thanks,

Na Li

Review Request 70708: ATLAS-3185: Create process execution for Impala integration

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70708/
---

Review request for atlas, Aadarsh Jajodia, Madhan Neethiraj, and Sarath 
Subramanian.


Repository: atlas


Description
---

ATLAS-3133 adds a new feature to track metadata for different executions of the 
same process in Atlas. Need to add this in Impala integration


Diffs
-

  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
 88faace 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
 232a569 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
 63c5f87 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java
 0dc520c 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
 0138d88 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
 033a518 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
 86801e3 


Diff: https://reviews.apache.org/r/70708/diff/1/


Testing
---

Integration tests


Thanks,

Na Li

Review Request 70701: ATLAS-3221: Refactor Impala integration code

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70701/
---

Review request for atlas and Sarath Subramanian.


Repository: atlas


Description
---

The code under impala-bridge module is under the folder 
"addons/impala-bridge/src/main/java/org.apache.atlas.impala/", they should be 
put under the folder 
"addons/impala-bridge/src/main/java/org/apache/atlas/impala/" to follow the 
convention.


Diffs
-

  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
 7c9abc8 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java
 88faace 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaHook.java 
2afb4eb 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaIdentifierParser.java
 b9d6cbb 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
 232a569 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java
 b9dd894 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
 63c5f87 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
 0dc520c 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataType.java
 10ce448 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDependencyType.java
 892ee9b 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java
 a3ddf53 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java
 a893b88 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaQuery.java
 27bdc72 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaVertexType.java
 8ec3f85 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java
 251507e 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
 0a664fc 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertexMetadata.java
 0bd236d 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/ImpalaLineageTool.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaHook.java 
PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaIdentifierParser.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaOperationParser.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaDataType.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaDependencyType.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaNode.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaOperationType.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaQuery.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaVertexType.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/LineageEdge.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/LineageVertex.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/LineageVertexMetadata.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/70701/diff/1/


Testing
---


Thanks,

Na Li

Re: Review Request 70619: ATLAS-3080: Integration with Impala Hook API

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70619/
---

(Updated May 22, 2019, 5:36 p.m.)


Review request for atlas, Fredy Wijaya, Madhan Neethiraj, and Sarath 
Subramanian.


Repository: atlas


Description
---

Integrate with Impala hook using Impala private jar with post exec hook

changes:

1) handle changed lineage record format change
The lineage record format has changed from having seperate table vertex 

  {  
 "id":4,
 "vertexType":"TABLE",
 "vertexId":"db_1.table_1",
 "createTime":155475
  }
  
To embed the table create time inside column vertex

{
  "id": 3,
  "vertexType": "COLUMN",
  "vertexId": "db_1.table_1.tinyint_col",
  "metadata": {
"createTime": 1557860978,
"tableName": "db_1.table_1"
  }
}

2) add Impala hook API code to allow Atlas integration code to compile without 
Impala updates


Diffs (updated)
-

  addons/impala-bridge-shim/pom.xml PRE-CREATION 
  
addons/impala-bridge-shim/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
 PRE-CREATION 
  addons/impala-bridge/pom.xml 8e7e952 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaHook.java 
PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
 fdb6748 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
 afe296c 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
 e4a38a3 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
 7c1103a 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
 82672c9 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertexMetadata.java
 PRE-CREATION 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
 cc62955 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
 6e4d321 
  addons/impala-bridge/src/test/resources/impala1.json 8f747f6 
  addons/impala-bridge/src/test/resources/impala2.json 239797b 
  addons/impala-bridge/src/test/resources/impala3.json 6a7d171 
  addons/impala-bridge/src/test/resources/impala5.json 854969b 
  addons/impala-bridge/src/test/resources/impala6.json f136180 
  addons/impala-bridge/src/test/resources/impala7.json f9ee670 
  addons/impala-bridge/src/test/resources/impalaAlterViewAsSelect.json 
PRE-CREATION 
  addons/impala-bridge/src/test/resources/impalaCreateTableAsSelect.json 
PRE-CREATION 
  addons/impala-bridge/src/test/resources/impalaCreateView.json PRE-CREATION 
  addons/impala-bridge/src/test/resources/impalaCreateViewNoCreateTime.json 
PRE-CREATION 
  addons/impala-bridge/src/test/resources/impalaInsertIntoAsSelect.json 
PRE-CREATION 
  addons/impala-hook-api/pom.xml PRE-CREATION 
  
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/PostQueryHookContext.java
 PRE-CREATION 
  
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/QueryExecHook.java 
PRE-CREATION 
  pom.xml 0a8f9b7 


Diff: https://reviews.apache.org/r/70619/diff/11/

Changes: https://reviews.apache.org/r/70619/diff/10-11/


Testing
---

add integration tests 
ImpalaLineageToolIT.testCreateViewFromFileWithoutCreateTime to handle the 
situation that there is no createTime in lineage record
change the file content for other integration tests in ImpalaLineageToolIT to 
handle new lineage record format


Thanks,

Na Li

Re: Review Request 70619: ATLAS-3080: Integration with Impala Hook API

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70619/
---

(Updated May 22, 2019, 2:35 a.m.)


Review request for atlas, Fredy Wijaya, Madhan Neethiraj, and Sarath 
Subramanian.


Repository: atlas


Description
---

Integrate with Impala hook using Impala private jar with post exec hook

changes:

1) handle changed lineage record format change
The lineage record format has changed from having seperate table vertex 

  {  
 "id":4,
 "vertexType":"TABLE",
 "vertexId":"db_1.table_1",
 "createTime":155475
  }
  
To embed the table create time inside column vertex

{
  "id": 3,
  "vertexType": "COLUMN",
  "vertexId": "db_1.table_1.tinyint_col",
  "metadata": {
"createTime": 1557860978,
"tableName": "db_1.table_1"
  }
}

2) add Impala hook API code to allow Atlas integration code to compile without 
Impala updates


Diffs (updated)
-

  addons/impala-bridge-shim/pom.xml PRE-CREATION 
  
addons/impala-bridge-shim/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
 PRE-CREATION 
  addons/impala-bridge/pom.xml 8e7e952 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaHook.java 
PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
 fdb6748 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
 afe296c 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
 e4a38a3 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
 7c1103a 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
 82672c9 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertexMetadata.java
 PRE-CREATION 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
 cc62955 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
 6e4d321 
  addons/impala-bridge/src/test/resources/impala1.json 8f747f6 
  addons/impala-bridge/src/test/resources/impala2.json 239797b 
  addons/impala-bridge/src/test/resources/impala3.json 6a7d171 
  addons/impala-bridge/src/test/resources/impala5.json 854969b 
  addons/impala-bridge/src/test/resources/impala6.json f136180 
  addons/impala-bridge/src/test/resources/impala7.json f9ee670 
  addons/impala-bridge/src/test/resources/impalaAlterViewAsSelect.json 
PRE-CREATION 
  addons/impala-bridge/src/test/resources/impalaCreateTableAsSelect.json 
PRE-CREATION 
  addons/impala-bridge/src/test/resources/impalaCreateView.json PRE-CREATION 
  addons/impala-bridge/src/test/resources/impalaCreateViewNoCreateTime.json 
PRE-CREATION 
  addons/impala-bridge/src/test/resources/impalaInsertIntoAsSelect.json 
PRE-CREATION 
  addons/impala-hook-api/pom.xml PRE-CREATION 
  
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/PostQueryHookContext.java
 PRE-CREATION 
  
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/QueryExecHook.java 
PRE-CREATION 
  pom.xml 98c2d69 


Diff: https://reviews.apache.org/r/70619/diff/10/

Changes: https://reviews.apache.org/r/70619/diff/9-10/


Testing
---

add integration tests 
ImpalaLineageToolIT.testCreateViewFromFileWithoutCreateTime to handle the 
situation that there is no createTime in lineage record
change the file content for other integration tests in ImpalaLineageToolIT to 
handle new lineage record format


Thanks,

Na Li

Re: Review Request 70619: ATLAS-3080: Integration with Impala Hook API

[Na Li via Review Board]

> On May 21, 2019, 7:09 p.m., Sridhar K wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaHook.java
> > Lines 19 (patched)
> > 
> >
> > Please package name w.r.t. directories. There are lot of classes in 
> > wrong folder strucutre. Please fix it.
> 
> Na Li wrote:
> will be fixed in https://issues.apache.org/jira/browse/ATLAS-3221

I have fixed the package issue in review version 9. However, do you really want 
to have integration change and fixing packaging in the same patch? As you can 
see, so many changes, the review gets really big, and you don't what has 
changed for integration with Impala.

That is preciously why I want to focus this issue ATLAS-3080 on integration, 
and ATLAS-3221 on fixing packaging issue.


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70619/#review215417
---


On May 22, 2019, 2:15 a.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70619/
> ---
> 
> (Updated May 22, 2019, 2:15 a.m.)
> 
> 
> Review request for atlas, Fredy Wijaya, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Integrate with Impala hook using Impala private jar with post exec hook
> 
> changes:
> 
> 1) handle changed lineage record format change
> The lineage record format has changed from having seperate table vertex 
> 
>   {  
>  "id":4,
>  "vertexType":"TABLE",
>  "vertexId":"db_1.table_1",
>  "createTime":155475
>   }
>   
> To embed the table create time inside column vertex
> 
> {
>   "id": 3,
>   "vertexType": "COLUMN",
>   "vertexId": "db_1.table_1.tinyint_col",
>   "metadata": {
> "createTime": 1557860978,
> "tableName": "db_1.table_1"
>   }
> }
> 
> 2) add Impala hook API code to allow Atlas integration code to compile 
> without Impala updates
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge-shim/pom.xml PRE-CREATION 
>   
> addons/impala-bridge-shim/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  PRE-CREATION 
>   addons/impala-bridge/pom.xml 8e7e952 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
>  7c9abc8 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java
>  88faace 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaIdentifierParser.java
>  b9d6cbb 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
>  fdb6748 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java
>  b9dd894 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
>  afe296c 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
>  e4a38a3 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
>  7c1103a 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataType.java
>  10ce448 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDependencyType.java
>  892ee9b 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java
>  a3ddf53 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java
>  a893b88 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaQuery.java
>  27bdc72 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaVertexType.java
>  8ec3f85 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java
>  251507e 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
>  82672c9 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/ImpalaLineageTool.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaIdentifierParser.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaOperationParser.java
>  PRE-CREATION 
>   
> 

Re: Review Request 70619: ATLAS-3080: Integration with Impala Hook API

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70619/
---

(Updated May 22, 2019, 2:15 a.m.)


Review request for atlas, Fredy Wijaya, Madhan Neethiraj, and Sarath 
Subramanian.


Repository: atlas


Description
---

Integrate with Impala hook using Impala private jar with post exec hook

changes:

1) handle changed lineage record format change
The lineage record format has changed from having seperate table vertex 

  {  
 "id":4,
 "vertexType":"TABLE",
 "vertexId":"db_1.table_1",
 "createTime":155475
  }
  
To embed the table create time inside column vertex

{
  "id": 3,
  "vertexType": "COLUMN",
  "vertexId": "db_1.table_1.tinyint_col",
  "metadata": {
"createTime": 1557860978,
"tableName": "db_1.table_1"
  }
}

2) add Impala hook API code to allow Atlas integration code to compile without 
Impala updates


Diffs (updated)
-

  addons/impala-bridge-shim/pom.xml PRE-CREATION 
  
addons/impala-bridge-shim/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
 PRE-CREATION 
  addons/impala-bridge/pom.xml 8e7e952 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
 7c9abc8 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java
 88faace 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaIdentifierParser.java
 b9d6cbb 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
 fdb6748 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java
 b9dd894 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
 afe296c 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
 e4a38a3 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
 7c1103a 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataType.java
 10ce448 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDependencyType.java
 892ee9b 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java
 a3ddf53 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java
 a893b88 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaQuery.java
 27bdc72 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaVertexType.java
 8ec3f85 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java
 251507e 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
 82672c9 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/ImpalaLineageTool.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaHook.java 
PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaIdentifierParser.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaOperationParser.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaDataType.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaDependencyType.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaNode.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaOperationType.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaQuery.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaVertexType.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/LineageEdge.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/LineageVertex.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/LineageVertexMetadata.java
 PRE-CREATION 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
 cc62955 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
 6e4d321 
  addons/impala-bridge/src/test/resources/impala1.json 8f747f6 
  

Re: Review Request 70619: ATLAS-3080: Integration with Impala Hook API

[Na Li via Review Board]

> On May 21, 2019, 7:27 p.m., Sridhar K wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
> > Lines 107 (patched)
> > 
> >
> > getTableNameFromColumn ==> getTableNameFromVertex
> 
> Na Li wrote:
> This is function to get table name from column vertext
> 
> Na Li wrote:
> public String getTableNameFromColumn(LineageVertex vertex) {
> if (vertex.getVertexType() == ImpalaVertexType.COLUMN) {
> LineageVertexMetadata metadata = vertex.getMetadata();
> 
> if (metadata != null) {
> return metadata.getTableName();
> }
> }
> 
> return getTableNameFromColumn(vertex.getVertexId());
> }

changed


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70619/#review215422
---


On May 21, 2019, 10:10 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70619/
> ---
> 
> (Updated May 21, 2019, 10:10 p.m.)
> 
> 
> Review request for atlas, Fredy Wijaya, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Integrate with Impala hook using Impala private jar with post exec hook
> 
> changes:
> 
> 1) handle changed lineage record format change
> The lineage record format has changed from having seperate table vertex 
> 
>   {  
>  "id":4,
>  "vertexType":"TABLE",
>  "vertexId":"db_1.table_1",
>  "createTime":155475
>   }
>   
> To embed the table create time inside column vertex
> 
> {
>   "id": 3,
>   "vertexType": "COLUMN",
>   "vertexId": "db_1.table_1.tinyint_col",
>   "metadata": {
> "createTime": 1557860978,
> "tableName": "db_1.table_1"
>   }
> }
> 
> 2) add Impala hook API code to allow Atlas integration code to compile 
> without Impala updates
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge-shim/pom.xml PRE-CREATION 
>   
> addons/impala-bridge-shim/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  PRE-CREATION 
>   addons/impala-bridge/pom.xml 8e7e952 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
>  fdb6748 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
>  afe296c 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
>  e4a38a3 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
>  7c1103a 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
>  82672c9 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertexMetadata.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  cc62955 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  6e4d321 
>   addons/impala-bridge/src/test/resources/impala3.json 6a7d171 
>   addons/impala-bridge/src/test/resources/impala4.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impala5.json 854969b 
>   addons/impala-bridge/src/test/resources/impala6.json f136180 
>   addons/impala-bridge/src/test/resources/impala7.json f9ee670 
>   addons/impala-hook-api/pom.xml PRE-CREATION 
>   
> addons/impala-hook-api/src/main/java/org/apache/impala/hooks/PostQueryHookContext.java
>  PRE-CREATION 
>   
> addons/impala-hook-api/src/main/java/org/apache/impala/hooks/QueryExecHook.java
>  PRE-CREATION 
>   pom.xml 98c2d69 
> 
> 
> Diff: https://reviews.apache.org/r/70619/diff/8/
> 
> 
> Testing
> ---
> 
> add integration tests 
> ImpalaLineageToolIT.testCreateViewFromFileWithoutCreateTime to handle the 
> situation that there is no createTime in lineage record
> change the file content for other integration tests in ImpalaLineageToolIT to 
> handle new lineage record format
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 70619: ATLAS-3080: Integration with Impala Hook API

[Na Li via Review Board]

> On May 21, 2019, 6:18 p.m., Sridhar K wrote:
> > addons/impala-bridge/pom.xml
> > Lines 59 (patched)
> > 
> >
> > Can we remove the project version hereand define dependency version 
> > in root pom.xml file's dependency management section.
> 
> Na Li wrote:
> No. I have to put version here to avoid compilation error (I don't know 
> why it is needed). "${project.version}" is the version defined in root 
> pom.xml at https://github.com/apache/atlas/blob/master/pom.xml#L32.
> 
> hive-bridge/pom.xml uses that too. 
> https://github.com/apache/atlas/blob/master/addons/hive-bridge/pom.xml#L123

done


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70619/#review215416
---


On May 21, 2019, 10:10 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70619/
> ---
> 
> (Updated May 21, 2019, 10:10 p.m.)
> 
> 
> Review request for atlas, Fredy Wijaya, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Integrate with Impala hook using Impala private jar with post exec hook
> 
> changes:
> 
> 1) handle changed lineage record format change
> The lineage record format has changed from having seperate table vertex 
> 
>   {  
>  "id":4,
>  "vertexType":"TABLE",
>  "vertexId":"db_1.table_1",
>  "createTime":155475
>   }
>   
> To embed the table create time inside column vertex
> 
> {
>   "id": 3,
>   "vertexType": "COLUMN",
>   "vertexId": "db_1.table_1.tinyint_col",
>   "metadata": {
> "createTime": 1557860978,
> "tableName": "db_1.table_1"
>   }
> }
> 
> 2) add Impala hook API code to allow Atlas integration code to compile 
> without Impala updates
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge-shim/pom.xml PRE-CREATION 
>   
> addons/impala-bridge-shim/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  PRE-CREATION 
>   addons/impala-bridge/pom.xml 8e7e952 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
>  fdb6748 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
>  afe296c 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
>  e4a38a3 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
>  7c1103a 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
>  82672c9 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertexMetadata.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  cc62955 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  6e4d321 
>   addons/impala-bridge/src/test/resources/impala3.json 6a7d171 
>   addons/impala-bridge/src/test/resources/impala4.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impala5.json 854969b 
>   addons/impala-bridge/src/test/resources/impala6.json f136180 
>   addons/impala-bridge/src/test/resources/impala7.json f9ee670 
>   addons/impala-hook-api/pom.xml PRE-CREATION 
>   
> addons/impala-hook-api/src/main/java/org/apache/impala/hooks/PostQueryHookContext.java
>  PRE-CREATION 
>   
> addons/impala-hook-api/src/main/java/org/apache/impala/hooks/QueryExecHook.java
>  PRE-CREATION 
>   pom.xml 98c2d69 
> 
> 
> Diff: https://reviews.apache.org/r/70619/diff/8/
> 
> 
> Testing
> ---
> 
> add integration tests 
> ImpalaLineageToolIT.testCreateViewFromFileWithoutCreateTime to handle the 
> situation that there is no createTime in lineage record
> change the file content for other integration tests in ImpalaLineageToolIT to 
> handle new lineage record format
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 70619: ATLAS-3080: Integration with Impala Hook API

[Na Li via Review Board]

> On May 21, 2019, 6:18 p.m., Sridhar K wrote:
> > addons/impala-bridge-shim/pom.xml
> > Lines 41 (patched)
> > 
> >
> > Can we remove the project version hereand define dependency version 
> > in root pom.xml file's dependency management section.
> 
> Na Li wrote:
> I tried to not put the version here, but did not work. Compilation failed 
> in dependency for not specifying version. 
> 
> 
> without this line "${project.version}", I got the 
> following error.
> 
> [ERROR]   The project org.apache.atlas:impala-bridge:3.0.0-SNAPSHOT 
> (/home/lina/sw/atlas/addons/impala-bridge/pom.xml) has 1 error
> [ERROR] 'dependencies.dependency.version' for 
> org.apache.atlas:impala-hook-api:jar is missing. @ 
> org.apache.atlas:impala-bridge:[unknown-version], 
> /home/lina/sw/atlas/addons/impala-bridge/pom.xml, line 56, column 17
> [ERROR]   
> [ERROR]   The project org.apache.atlas:impala-bridge-shim:3.0.0-SNAPSHOT 
> (/home/lina/sw/atlas/addons/impala-bridge-shim/pom.xml) has 1 error
> [ERROR] 'dependencies.dependency.version' for 
> org.apache.atlas:impala-hook-api:jar is missing. @ 
> org.apache.atlas:impala-bridge-shim:[unknown-version], 
> /home/lina/sw/atlas/addons/impala-bridge-shim/pom.xml, line 38, column 17
> 
> Sridhar K wrote:
> You should put the dependency root pom.xml file. Please check  root>/pom.xml for samples..here is a sample from root pom.xml file. Make sure 
> that you are adding in  dependencyManagement section..
> 
> 
> org.apache.atlas
> atlas-plugin-classloader
> ${project.version}
> 

done


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70619/#review215416
---


On May 21, 2019, 10:10 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70619/
> ---
> 
> (Updated May 21, 2019, 10:10 p.m.)
> 
> 
> Review request for atlas, Fredy Wijaya, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Integrate with Impala hook using Impala private jar with post exec hook
> 
> changes:
> 
> 1) handle changed lineage record format change
> The lineage record format has changed from having seperate table vertex 
> 
>   {  
>  "id":4,
>  "vertexType":"TABLE",
>  "vertexId":"db_1.table_1",
>  "createTime":155475
>   }
>   
> To embed the table create time inside column vertex
> 
> {
>   "id": 3,
>   "vertexType": "COLUMN",
>   "vertexId": "db_1.table_1.tinyint_col",
>   "metadata": {
> "createTime": 1557860978,
> "tableName": "db_1.table_1"
>   }
> }
> 
> 2) add Impala hook API code to allow Atlas integration code to compile 
> without Impala updates
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge-shim/pom.xml PRE-CREATION 
>   
> addons/impala-bridge-shim/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  PRE-CREATION 
>   addons/impala-bridge/pom.xml 8e7e952 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
>  fdb6748 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
>  afe296c 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
>  e4a38a3 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
>  7c1103a 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
>  82672c9 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertexMetadata.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  cc62955 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  6e4d321 
>   addons/impala-bridge/src/test/resources/impala3.json 6a7d171 
>   addons/impala-bridge/src/test/resources/impala4.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impala5.json 854969b 
>   addons/impala-bridge/src/test/resources/impala6.json f136180 
>   addons/impala-bridge/src/test/resources/impala7.json f9ee670 
>   addons/impala-hook-api/pom.xml PRE-CREATION 
>   
> addons/impala-hook-api/src/main/java/org/apache/impala/hooks/PostQueryHookContext.java
>  PRE-CREATION 
>   
> 

Re: Review Request 70619: ATLAS-3080: Integration with Impala Hook API

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70619/
---

(Updated May 21, 2019, 10:10 p.m.)


Review request for atlas, Fredy Wijaya, Madhan Neethiraj, and Sarath 
Subramanian.


Repository: atlas


Description
---

Integrate with Impala hook using Impala private jar with post exec hook

changes:

1) handle changed lineage record format change
The lineage record format has changed from having seperate table vertex 

  {  
 "id":4,
 "vertexType":"TABLE",
 "vertexId":"db_1.table_1",
 "createTime":155475
  }
  
To embed the table create time inside column vertex

{
  "id": 3,
  "vertexType": "COLUMN",
  "vertexId": "db_1.table_1.tinyint_col",
  "metadata": {
"createTime": 1557860978,
"tableName": "db_1.table_1"
  }
}

2) add Impala hook API code to allow Atlas integration code to compile without 
Impala updates


Diffs
-

  addons/impala-bridge-shim/pom.xml PRE-CREATION 
  
addons/impala-bridge-shim/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
 PRE-CREATION 
  addons/impala-bridge/pom.xml 8e7e952 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaHook.java 
PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
 fdb6748 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
 afe296c 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
 e4a38a3 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
 7c1103a 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
 82672c9 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertexMetadata.java
 PRE-CREATION 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
 cc62955 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
 6e4d321 
  addons/impala-bridge/src/test/resources/impala3.json 6a7d171 
  addons/impala-bridge/src/test/resources/impala4.json PRE-CREATION 
  addons/impala-bridge/src/test/resources/impala5.json 854969b 
  addons/impala-bridge/src/test/resources/impala6.json f136180 
  addons/impala-bridge/src/test/resources/impala7.json f9ee670 
  addons/impala-hook-api/pom.xml PRE-CREATION 
  
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/PostQueryHookContext.java
 PRE-CREATION 
  
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/QueryExecHook.java 
PRE-CREATION 
  pom.xml 98c2d69 


Diff: https://reviews.apache.org/r/70619/diff/8/


Testing
---

add integration tests 
ImpalaLineageToolIT.testCreateViewFromFileWithoutCreateTime to handle the 
situation that there is no createTime in lineage record
change the file content for other integration tests in ImpalaLineageToolIT to 
handle new lineage record format


Thanks,

Na Li

Re: Review Request 70512: ATLAS-3183: Read Impala lineage record for creating view and send to Atlas

[Na Li via Review Board]

> On May 7, 2019, 2:47 a.m., Aadarsh Jajodia wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
> > Lines 236 (patched)
> > 
> >
> > Add comment for method

done


> On May 7, 2019, 2:47 a.m., Aadarsh Jajodia wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
> > Lines 262 (patched)
> > 
> >
> > Add comment for method

done.


> On May 7, 2019, 2:47 a.m., Aadarsh Jajodia wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
> > Lines 304-310 (patched)
> > 
> >
> > Can be simplified
> > 
> > if (inputNodes.isEmpty() || ouputNodes.isEmpty()) {
> > return true;
> > }

done


> On May 7, 2019, 2:47 a.m., Aadarsh Jajodia wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDependencyType.java
> > Lines 21 (patched)
> > 
> >
> > extra space after enum

removed.


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70512/#review215078
---


On May 8, 2019, 9:36 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70512/
> ---
> 
> (Updated May 8, 2019, 9:36 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, madhan, Sarath 
> Subramanian, and Xinran Tinney.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Impala generates lineage records for its commands. This new feature will read 
> Impala lineage file, convert the lineage record to Atlas entities and send 
> them to Atlas. In this way, Atlas can get lineage of Impala operation.
> 
> The metadata referred in the lineage are captured in Hive Metastore hook and 
> sent to Atlas. This work is done in ATLAS-3148
> 
> This jira only supports the Impala command "create view". Following jira will 
> add support for more Impala commands.
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge/pom.xml PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaIdentifierParser.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataType.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDependencyType.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaQuery.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaVertexType.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
>  PRE-CREATION 
>   addons/impala-bridge/src/main/resources/atlas-log4j.xml PRE-CREATION 
>   addons/impala-bridge/src/main/resources/import-impala.sh PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  PRE-CREATION 
>   addons/impala-bridge/src/test/resources/atlas-application.properties 
> PRE-CREATION 
>   

Re: Review Request 70512: ATLAS-3183: Read Impala lineage record for creating view and send to Atlas

[Na Li via Review Board]

> On May 8, 2019, 7:35 p.m., Xinran Tinney wrote:
> > addons/impala-bridge/src/main/resources/import-impala.sh
> > Lines 88 (patched)
> > 
> >
> > should we remove this since it is printing too much information in the 
> > terminal after the shell is ran.

removed


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70512/#review215135
---


On May 8, 2019, 9:36 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70512/
> ---
> 
> (Updated May 8, 2019, 9:36 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, madhan, Sarath 
> Subramanian, and Xinran Tinney.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Impala generates lineage records for its commands. This new feature will read 
> Impala lineage file, convert the lineage record to Atlas entities and send 
> them to Atlas. In this way, Atlas can get lineage of Impala operation.
> 
> The metadata referred in the lineage are captured in Hive Metastore hook and 
> sent to Atlas. This work is done in ATLAS-3148
> 
> This jira only supports the Impala command "create view". Following jira will 
> add support for more Impala commands.
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge/pom.xml PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaIdentifierParser.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataType.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDependencyType.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaQuery.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaVertexType.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
>  PRE-CREATION 
>   addons/impala-bridge/src/main/resources/atlas-log4j.xml PRE-CREATION 
>   addons/impala-bridge/src/main/resources/import-impala.sh PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  PRE-CREATION 
>   addons/impala-bridge/src/test/resources/atlas-application.properties 
> PRE-CREATION 
>   addons/impala-bridge/src/test/resources/atlas-log4j.xml PRE-CREATION 
>   addons/impala-bridge/src/test/resources/hive-site.xml PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impala1.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impala2.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impala3.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/users-credentials.properties 
> PRE-CREATION 
>   addons/models/1000-Hadoop/1090-impala_model.json PRE-CREATION 
>   pom.xml ae4dfdc 
> 
> 
> Diff: https://reviews.apache.org/r/70512/diff/16/
> 
> 
> Testing
> ---
> 
> Run the tool in real cluster that has Atlas server with Impala lineage file 
> as input for creating view. The Atlas UI displays hive_lineage lineage and 
> hive_column_lineage.
> Add new integration tests and they pass
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 70512: ATLAS-3183: Read Impala lineage record for creating view and send to Atlas

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70512/
---

(Updated May 8, 2019, 9:36 p.m.)


Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, madhan, Sarath 
Subramanian, and Xinran Tinney.


Repository: atlas


Description
---

Impala generates lineage records for its commands. This new feature will read 
Impala lineage file, convert the lineage record to Atlas entities and send them 
to Atlas. In this way, Atlas can get lineage of Impala operation.

The metadata referred in the lineage are captured in Hive Metastore hook and 
sent to Atlas. This work is done in ATLAS-3148

This jira only supports the Impala command "create view". Following jira will 
add support for more Impala commands.


Diffs (updated)
-

  addons/impala-bridge/pom.xml PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaIdentifierParser.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataType.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDependencyType.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaQuery.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaVertexType.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
 PRE-CREATION 
  addons/impala-bridge/src/main/resources/atlas-log4j.xml PRE-CREATION 
  addons/impala-bridge/src/main/resources/import-impala.sh PRE-CREATION 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
 PRE-CREATION 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
 PRE-CREATION 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
 PRE-CREATION 
  addons/impala-bridge/src/test/resources/atlas-application.properties 
PRE-CREATION 
  addons/impala-bridge/src/test/resources/atlas-log4j.xml PRE-CREATION 
  addons/impala-bridge/src/test/resources/hive-site.xml PRE-CREATION 
  addons/impala-bridge/src/test/resources/impala1.json PRE-CREATION 
  addons/impala-bridge/src/test/resources/impala2.json PRE-CREATION 
  addons/impala-bridge/src/test/resources/impala3.json PRE-CREATION 
  addons/impala-bridge/src/test/resources/users-credentials.properties 
PRE-CREATION 
  addons/models/1000-Hadoop/1090-impala_model.json PRE-CREATION 
  pom.xml ae4dfdc 


Diff: https://reviews.apache.org/r/70512/diff/16/

Changes: https://reviews.apache.org/r/70512/diff/15-16/


Testing
---

Run the tool in real cluster that has Atlas server with Impala lineage file as 
input for creating view. The Atlas UI displays hive_lineage lineage and 
hive_column_lineage.
Add new integration tests and they pass


Thanks,

Na Li

Re: Review Request 70512: ATLAS-3183: Read Impala lineage record for creating view and send to Atlas

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70512/
---

(Updated May 8, 2019, 7:20 p.m.)


Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, madhan, Sarath 
Subramanian, and Xinran Tinney.


Repository: atlas


Description
---

Impala generates lineage records for its commands. This new feature will read 
Impala lineage file, convert the lineage record to Atlas entities and send them 
to Atlas. In this way, Atlas can get lineage of Impala operation.

The metadata referred in the lineage are captured in Hive Metastore hook and 
sent to Atlas. This work is done in ATLAS-3148

This jira only supports the Impala command "create view". Following jira will 
add support for more Impala commands.


Diffs (updated)
-

  addons/impala-bridge/pom.xml PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaIdentifierParser.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataType.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDependencyType.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaQuery.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaVertexType.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
 PRE-CREATION 
  addons/impala-bridge/src/main/resources/atlas-log4j.xml PRE-CREATION 
  addons/impala-bridge/src/main/resources/import-impala.sh PRE-CREATION 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
 PRE-CREATION 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
 PRE-CREATION 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
 PRE-CREATION 
  addons/impala-bridge/src/test/resources/atlas-application.properties 
PRE-CREATION 
  addons/impala-bridge/src/test/resources/atlas-log4j.xml PRE-CREATION 
  addons/impala-bridge/src/test/resources/hive-site.xml PRE-CREATION 
  addons/impala-bridge/src/test/resources/impala1.json PRE-CREATION 
  addons/impala-bridge/src/test/resources/impala2.json PRE-CREATION 
  addons/impala-bridge/src/test/resources/impala3.json PRE-CREATION 
  addons/impala-bridge/src/test/resources/users-credentials.properties 
PRE-CREATION 
  addons/models/1000-Hadoop/1090-impala_model.json PRE-CREATION 
  pom.xml ae4dfdc 


Diff: https://reviews.apache.org/r/70512/diff/15/

Changes: https://reviews.apache.org/r/70512/diff/14-15/


Testing
---

Run the tool in real cluster that has Atlas server with Impala lineage file as 
input for creating view. The Atlas UI displays hive_lineage lineage and 
hive_column_lineage.
Add new integration tests and they pass


Thanks,

Na Li

Re: Review Request 70512: ATLAS-3183: Read Impala lineage record for creating view and send to Atlas

[Na Li via Review Board]

> On May 7, 2019, 11:37 p.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/resources/import-impala.sh
> > Lines 60 (patched)
> > 
> >
> > why do we hardcode ATLAS lib path here?

changed to relative path


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70512/#review215106
---


On May 8, 2019, 7:20 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70512/
> ---
> 
> (Updated May 8, 2019, 7:20 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, madhan, Sarath 
> Subramanian, and Xinran Tinney.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Impala generates lineage records for its commands. This new feature will read 
> Impala lineage file, convert the lineage record to Atlas entities and send 
> them to Atlas. In this way, Atlas can get lineage of Impala operation.
> 
> The metadata referred in the lineage are captured in Hive Metastore hook and 
> sent to Atlas. This work is done in ATLAS-3148
> 
> This jira only supports the Impala command "create view". Following jira will 
> add support for more Impala commands.
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge/pom.xml PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaIdentifierParser.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataType.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDependencyType.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaQuery.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaVertexType.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
>  PRE-CREATION 
>   addons/impala-bridge/src/main/resources/atlas-log4j.xml PRE-CREATION 
>   addons/impala-bridge/src/main/resources/import-impala.sh PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  PRE-CREATION 
>   addons/impala-bridge/src/test/resources/atlas-application.properties 
> PRE-CREATION 
>   addons/impala-bridge/src/test/resources/atlas-log4j.xml PRE-CREATION 
>   addons/impala-bridge/src/test/resources/hive-site.xml PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impala1.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impala2.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impala3.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/users-credentials.properties 
> PRE-CREATION 
>   addons/models/1000-Hadoop/1090-impala_model.json PRE-CREATION 
>   pom.xml ae4dfdc 
> 
> 
> Diff: https://reviews.apache.org/r/70512/diff/15/
> 
> 
> Testing
> ---
> 
> Run the tool in real cluster that has Atlas server with Impala lineage file 
> as input for creating view. The Atlas UI displays hive_lineage lineage and 
> hive_column_lineage.
> Add new integration tests and they pass
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 70512: ATLAS-3183: Read Impala lineage record for creating view and send to Atlas

[Na Li via Review Board]

> On May 7, 2019, 11:37 p.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
> > Lines 48 (patched)
> > 
> >
> > we don't need 'args' to be stored as class variable. please review.

removed it as class variable. Pass it in as function input


> On May 7, 2019, 11:37 p.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
> > Lines 52 (patched)
> > 
> >
> > consider intializing values for directoryName, prefix in constructor 
> > instead of getCurrentFiles()

done


> On May 7, 2019, 11:37 p.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
> > Lines 57 (patched)
> > 
> >
> > why 'args' need to be passed in method, when its already initialized in 
> > constructor (line #53)?

removed.


> On May 7, 2019, 11:37 p.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
> > Lines 127 (patched)
> > 
> >
> > BasicParser() is deprecated; use DefaultParser() instead.

changed


> On May 7, 2019, 11:37 p.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
> > Lines 155 (patched)
> > 
> >
> > this method doesn't do anything to send lineage to kafka; consider 
> > renaming method to processImpalaLineageHook()

done.


> On May 7, 2019, 11:37 p.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
> > Lines 211 (patched)
> > 
> >
> > update logger message - Error sending impala lineage records to 
> > ImpalaHook

done


> On May 7, 2019, 11:37 p.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java
> > Lines 159 (patched)
> > 
> >
> > why table name needs to be checked for correctness? lineage query is 
> > written by impala after successfull query execution. Impala would have 
> > already validated the table name correctness. We don't need to re-validate 
> > here again. Consider removing ImpalaIdentifierParser class as well.

Impala output the column name in vertices. We need to derive the table name 
from that column name. In some vertex, the column name is not really a column 
name, so derived is not real table name. Like following example shows. Impala 
does not validate the table or column name correctness. That is why we need to 
do it here.

"queryText":"select sum(a.tinyint_col) over (partition by a.smallint_col order 
by a.id),\n  count(b.string_col), b.timestamp_col\nfrom functional.alltypes a 
join functional.alltypessmall b on (a.id = b.id)\nwhere a.year = 2010 and 
b.float_col > 0\ngroup by a.tinyint_col, a.smallint_col, a.id, b.string_col, 
b.timestamp_col, b.bigint_col\nhaving count(a.int_col) > 10\norder by 
b.bigint_col limit 10",


"vertices":[
{
"id":0,
"vertexType":"COLUMN",
"vertexId":"sum(a.tinyint_col) OVER(...)"
},


> On May 7, 2019, 11:37 p.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java
> > Lines 31 (patched)
> > 
> >
> > consider renaming method to getImpalaOperationType()

done


> On May 7, 2019, 11:37 p.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
> > Lines 78 (patched)
> > 
> >
> > depenendencyType => dependencyType

done


> On May 7, 2019, 11:37 p.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
> > Lines 79 (patched)
> > 
> >
> > unused variable - ATTRIBUTE_EXPRESSION

removed


> On May 7, 2019, 11:37 p.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
> > Lines 21 (patched)
> > 
> >
> > unused import; consider removing.

removed


> On May 7, 2019, 11:37 p.m., Sarath Subramanian wrote:
> > 

Re: Review Request 70512: ATLAS-3183: Read Impala lineage record for creating view and send to Atlas

[Na Li via Review Board]

> On May 7, 2019, 2:47 a.m., Aadarsh Jajodia wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
> > Lines 64 (patched)
> > 
> >
> > Replace with
> > LOG.info("Importing: {}", filename);

done


> On May 7, 2019, 2:47 a.m., Aadarsh Jajodia wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
> > Lines 65 (patched)
> > 
> >
> > What are we doing with the return value of this method?

nothing. I change the return to void since the list is processed inside this 
function.


> On May 7, 2019, 2:47 a.m., Aadarsh Jajodia wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
> > Lines 118 (patched)
> > 
> >
> > What if node itself is null?

add code to throw exception in this case.


> On May 7, 2019, 2:47 a.m., Aadarsh Jajodia wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
> > Lines 158 (patched)
> > 
> >
> > Do we need this check? We can just put LOG.debug. If log level is set 
> > to debug it will log it else it will not isnt it?

remove the check. The check is necessary for performance reason if it takes a 
lot of effort to construct the log message. It is not the case here, so remove 
the check.


> On May 7, 2019, 2:47 a.m., Aadarsh Jajodia wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
> > Lines 165 (patched)
> > 
> >
> > Remove extra line

done


> On May 7, 2019, 2:47 a.m., Aadarsh Jajodia wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
> > Lines 171 (patched)
> > 
> >
> > Remove extra line

done


> On May 7, 2019, 2:47 a.m., Aadarsh Jajodia wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
> > Lines 189 (patched)
> > 
> >
> > Remove extra line

done


> On May 7, 2019, 2:47 a.m., Aadarsh Jajodia wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
> > Lines 203 (patched)
> > 
> >
> > Change ":" to constant

done


> On May 7, 2019, 2:47 a.m., Aadarsh Jajodia wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
> > Lines 227 (patched)
> > 
> >
> > Do we need the isDebugEnabled check? We can just put LOG.debug. If log 
> > level is set to debug it will log it else it will not isnt it?
> > 
> > Do we also need the isColumnLineage check? I see that above we are 
> > creating objects of columnlineage and hence they will never be NULL. I feel 
> > this check can be avoided here

updated


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70512/#review215078
---


On May 4, 2019, 9:23 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70512/
> ---
> 
> (Updated May 4, 2019, 9:23 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, madhan, Sarath 
> Subramanian, and Xinran Tinney.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Impala generates lineage records for its commands. This new feature will read 
> Impala lineage file, convert the lineage record to Atlas entities and send 
> them to Atlas. In this way, Atlas can get lineage of Impala operation.
> 
> The metadata referred in the lineage are captured in Hive Metastore hook and 
> sent to Atlas. This work is done in ATLAS-3148
> 
> This jira only supports the Impala command "create view". Following jira will 
> add support for more Impala commands.
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge/pom.xml PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java
>  PRE-CREATION 
>   
> 

Re: Review Request 70512: ATLAS-3183: Read Impala lineage record for creating view and send to Atlas

[Na Li via Review Board]

> On May 7, 2019, 12:54 a.m., Sridhar K wrote:
> > addons/impala-bridge/pom.xml
> > Lines 65 (patched)
> > 
> >
> > Can we please move this version to a variable?

done


> On May 7, 2019, 12:54 a.m., Sridhar K wrote:
> > addons/impala-bridge/pom.xml
> > Lines 71 (patched)
> > 
> >
> > Can we please move this version to a variable? Use the same variable as 
> > above.

done


> On May 7, 2019, 12:54 a.m., Sridhar K wrote:
> > addons/impala-bridge/pom.xml
> > Lines 449 (patched)
> > 
> >
> > Please move to variable.

done


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70512/#review215076
---


On May 4, 2019, 9:23 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70512/
> ---
> 
> (Updated May 4, 2019, 9:23 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, madhan, Sarath 
> Subramanian, and Xinran Tinney.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Impala generates lineage records for its commands. This new feature will read 
> Impala lineage file, convert the lineage record to Atlas entities and send 
> them to Atlas. In this way, Atlas can get lineage of Impala operation.
> 
> The metadata referred in the lineage are captured in Hive Metastore hook and 
> sent to Atlas. This work is done in ATLAS-3148
> 
> This jira only supports the Impala command "create view". Following jira will 
> add support for more Impala commands.
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge/pom.xml PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaIdentifierParser.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataType.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDependencyType.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaQuery.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaVertexType.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
>  PRE-CREATION 
>   addons/impala-bridge/src/main/resources/atlas-log4j.xml PRE-CREATION 
>   addons/impala-bridge/src/main/resources/import-impala.sh PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  PRE-CREATION 
>   addons/impala-bridge/src/test/resources/atlas-application.properties 
> PRE-CREATION 
>   addons/impala-bridge/src/test/resources/atlas-log4j.xml PRE-CREATION 
>   addons/impala-bridge/src/test/resources/hive-site.xml PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impala1.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impala2.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impala3.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/users-credentials.properties 
> PRE-CREATION 
>   addons/models/1000-Hadoop/1090-impala_model.json PRE-CREATION 
>   pom.xml 7de5d31 
> 
> 
> Diff: 

Re: Review Request 70512: ATLAS-3183: Read Impala lineage record for creating view and send to Atlas

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70512/
---

(Updated May 4, 2019, 9:23 p.m.)


Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, madhan, Sarath 
Subramanian, and Xinran Tinney.


Repository: atlas


Description
---

Impala generates lineage records for its commands. This new feature will read 
Impala lineage file, convert the lineage record to Atlas entities and send them 
to Atlas. In this way, Atlas can get lineage of Impala operation.

The metadata referred in the lineage are captured in Hive Metastore hook and 
sent to Atlas. This work is done in ATLAS-3148

This jira only supports the Impala command "create view". Following jira will 
add support for more Impala commands.


Diffs (updated)
-

  addons/impala-bridge/pom.xml PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaIdentifierParser.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataType.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDependencyType.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaQuery.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaVertexType.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
 PRE-CREATION 
  addons/impala-bridge/src/main/resources/atlas-log4j.xml PRE-CREATION 
  addons/impala-bridge/src/main/resources/import-impala.sh PRE-CREATION 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
 PRE-CREATION 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
 PRE-CREATION 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
 PRE-CREATION 
  addons/impala-bridge/src/test/resources/atlas-application.properties 
PRE-CREATION 
  addons/impala-bridge/src/test/resources/atlas-log4j.xml PRE-CREATION 
  addons/impala-bridge/src/test/resources/hive-site.xml PRE-CREATION 
  addons/impala-bridge/src/test/resources/impala1.json PRE-CREATION 
  addons/impala-bridge/src/test/resources/impala2.json PRE-CREATION 
  addons/impala-bridge/src/test/resources/impala3.json PRE-CREATION 
  addons/impala-bridge/src/test/resources/users-credentials.properties 
PRE-CREATION 
  addons/models/1000-Hadoop/1090-impala_model.json PRE-CREATION 
  pom.xml 7de5d31 


Diff: https://reviews.apache.org/r/70512/diff/14/

Changes: https://reviews.apache.org/r/70512/diff/13-14/


Testing
---

Run the tool in real cluster that has Atlas server with Impala lineage file as 
input for creating view. The Atlas UI displays hive_lineage lineage and 
hive_column_lineage.
Add new integration tests and they pass


Thanks,

Na Li

Re: Review Request 70512: ATLAS-3183: Read Impala lineage record for creating view and send to Atlas

[Na Li via Review Board]

> On May 3, 2019, 2:01 a.m., Madhan Neethiraj wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
> > Lines 54 (patched)
> > 
> >
> > It may not be useful to designate a 'default-directoy'; the tool should 
> > process the files specified in the command-line. Please consider removing 
> > this.
> > 
> > Anyway, this doesn't seem to be used at all. Please review.

done


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70512/#review215010
---


On May 2, 2019, 8:31 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70512/
> ---
> 
> (Updated May 2, 2019, 8:31 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, madhan, Sarath 
> Subramanian, and Xinran Tinney.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Impala generates lineage records for its commands. This new feature will read 
> Impala lineage file, convert the lineage record to Atlas entities and send 
> them to Atlas. In this way, Atlas can get lineage of Impala operation.
> 
> The metadata referred in the lineage are captured in Hive Metastore hook and 
> sent to Atlas. This work is done in ATLAS-3148
> 
> This jira only supports the Impala command "create view". Following jira will 
> add support for more Impala commands.
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge-shim/pom.xml PRE-CREATION 
>   
> addons/impala-bridge-shim/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
>  PRE-CREATION 
>   addons/impala-bridge/pom.xml PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataTypes.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageQuery.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
>  PRE-CREATION 
>   addons/impala-bridge/src/main/resources/atlas-log4j.xml PRE-CREATION 
>   addons/impala-bridge/src/main/resources/import-impala.sh PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  PRE-CREATION 
>   addons/impala-bridge/src/test/resources/atlas-application.properties 
> PRE-CREATION 
>   addons/impala-bridge/src/test/resources/atlas-log4j.xml PRE-CREATION 
>   addons/impala-bridge/src/test/resources/hive-site.xml PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impala1.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impala2.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impala3.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/users-credentials.properties 
> PRE-CREATION 
>   pom.xml 7de5d31 
> 
> 
> Diff: https://reviews.apache.org/r/70512/diff/13/
> 
> 
> Testing
> ---
> 
> Run the tool in real cluster that has Atlas server with Impala lineage file 
> as input for creating view. The Atlas UI displays hive_lineage lineage and 
> hive_column_lineage.
> Add new integration tests and they pass
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 70512: ATLAS-3183: Read Impala lineage record for creating view and send to Atlas

[Na Li via Review Board]

> On May 3, 2019, 2:01 a.m., Madhan Neethiraj wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageQuery.java
> > Lines 35 (patched)
> > 
> >
> > Consider renaming LineageQuery to ImpalaQuery or ImpalaProcess

changed the type to ImpalaQuery


> On May 3, 2019, 2:01 a.m., Madhan Neethiraj wrote:
> > addons/impala-bridge/src/main/resources/import-impala.sh
> > Lines 48 (patched)
> > 
> >
> > Avoid references to paths like /usr/hdp/. Instead consider using paths 
> > relative to the script directoty i.e. ${BASEDIR}.

change it to use ${BASEDIR}


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70512/#review215010
---


On May 2, 2019, 8:31 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70512/
> ---
> 
> (Updated May 2, 2019, 8:31 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, madhan, Sarath 
> Subramanian, and Xinran Tinney.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Impala generates lineage records for its commands. This new feature will read 
> Impala lineage file, convert the lineage record to Atlas entities and send 
> them to Atlas. In this way, Atlas can get lineage of Impala operation.
> 
> The metadata referred in the lineage are captured in Hive Metastore hook and 
> sent to Atlas. This work is done in ATLAS-3148
> 
> This jira only supports the Impala command "create view". Following jira will 
> add support for more Impala commands.
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge-shim/pom.xml PRE-CREATION 
>   
> addons/impala-bridge-shim/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
>  PRE-CREATION 
>   addons/impala-bridge/pom.xml PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataTypes.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageQuery.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
>  PRE-CREATION 
>   addons/impala-bridge/src/main/resources/atlas-log4j.xml PRE-CREATION 
>   addons/impala-bridge/src/main/resources/import-impala.sh PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  PRE-CREATION 
>   addons/impala-bridge/src/test/resources/atlas-application.properties 
> PRE-CREATION 
>   addons/impala-bridge/src/test/resources/atlas-log4j.xml PRE-CREATION 
>   addons/impala-bridge/src/test/resources/hive-site.xml PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impala1.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impala2.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impala3.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/users-credentials.properties 
> PRE-CREATION 
>   pom.xml 7de5d31 
> 
> 
> Diff: https://reviews.apache.org/r/70512/diff/13/
> 
> 
> Testing
> ---
> 
> Run the tool in real cluster that has Atlas server with Impala lineage file 
> as input for creating view. The Atlas UI displays hive_lineage lineage and 
> hive_column_lineage.
> Add new integration tests and they pass
> 
> 
> Thanks,
> 

Re: Review Request 70512: ATLAS-3183: Read Impala lineage record for creating view and send to Atlas

[Na Li via Review Board]

> On May 3, 2019, 7:41 a.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
> > Lines 193 (patched)
> > 
> >
> > consider registering new typedef for impala_column_lineage instead of 
> > reusing hive_column_lineage.

done


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70512/#review215016
---


On May 2, 2019, 8:31 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70512/
> ---
> 
> (Updated May 2, 2019, 8:31 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, madhan, Sarath 
> Subramanian, and Xinran Tinney.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Impala generates lineage records for its commands. This new feature will read 
> Impala lineage file, convert the lineage record to Atlas entities and send 
> them to Atlas. In this way, Atlas can get lineage of Impala operation.
> 
> The metadata referred in the lineage are captured in Hive Metastore hook and 
> sent to Atlas. This work is done in ATLAS-3148
> 
> This jira only supports the Impala command "create view". Following jira will 
> add support for more Impala commands.
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge-shim/pom.xml PRE-CREATION 
>   
> addons/impala-bridge-shim/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
>  PRE-CREATION 
>   addons/impala-bridge/pom.xml PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataTypes.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageQuery.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
>  PRE-CREATION 
>   addons/impala-bridge/src/main/resources/atlas-log4j.xml PRE-CREATION 
>   addons/impala-bridge/src/main/resources/import-impala.sh PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  PRE-CREATION 
>   addons/impala-bridge/src/test/resources/atlas-application.properties 
> PRE-CREATION 
>   addons/impala-bridge/src/test/resources/atlas-log4j.xml PRE-CREATION 
>   addons/impala-bridge/src/test/resources/hive-site.xml PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impala1.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impala2.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impala3.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/users-credentials.properties 
> PRE-CREATION 
>   pom.xml 7de5d31 
> 
> 
> Diff: https://reviews.apache.org/r/70512/diff/13/
> 
> 
> Testing
> ---
> 
> Run the tool in real cluster that has Atlas server with Impala lineage file 
> as input for creating view. The Atlas UI displays hive_lineage lineage and 
> hive_column_lineage.
> Add new integration tests and they pass
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 70512: ATLAS-3183: Read Impala lineage record for creating view and send to Atlas

[Na Li via Review Board]

> On May 3, 2019, 7:38 a.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
> > Lines 39 (patched)
> > 
> >
> > make vertexType as enum

done


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70512/#review215013
---


On May 2, 2019, 8:31 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70512/
> ---
> 
> (Updated May 2, 2019, 8:31 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, madhan, Sarath 
> Subramanian, and Xinran Tinney.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Impala generates lineage records for its commands. This new feature will read 
> Impala lineage file, convert the lineage record to Atlas entities and send 
> them to Atlas. In this way, Atlas can get lineage of Impala operation.
> 
> The metadata referred in the lineage are captured in Hive Metastore hook and 
> sent to Atlas. This work is done in ATLAS-3148
> 
> This jira only supports the Impala command "create view". Following jira will 
> add support for more Impala commands.
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge-shim/pom.xml PRE-CREATION 
>   
> addons/impala-bridge-shim/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
>  PRE-CREATION 
>   addons/impala-bridge/pom.xml PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataTypes.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageQuery.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
>  PRE-CREATION 
>   addons/impala-bridge/src/main/resources/atlas-log4j.xml PRE-CREATION 
>   addons/impala-bridge/src/main/resources/import-impala.sh PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  PRE-CREATION 
>   addons/impala-bridge/src/test/resources/atlas-application.properties 
> PRE-CREATION 
>   addons/impala-bridge/src/test/resources/atlas-log4j.xml PRE-CREATION 
>   addons/impala-bridge/src/test/resources/hive-site.xml PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impala1.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impala2.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impala3.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/users-credentials.properties 
> PRE-CREATION 
>   pom.xml 7de5d31 
> 
> 
> Diff: https://reviews.apache.org/r/70512/diff/13/
> 
> 
> Testing
> ---
> 
> Run the tool in real cluster that has Atlas server with Impala lineage file 
> as input for creating view. The Atlas UI displays hive_lineage lineage and 
> hive_column_lineage.
> Add new integration tests and they pass
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 70512: ATLAS-3183: Read Impala lineage record for creating view and send to Atlas

[Na Li via Review Board]

> On May 3, 2019, 7:38 a.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
> > Lines 204 (patched)
> > 
> >
> > dependencyType can not always be be SIMPLE; 
> > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/hooks/LineageInfo.java#L59

Impala dependencyType can only be Projection or Predicate. We should skip the 
dependency for Predicate. So the dependency type for Impala column lineage will 
always Projection. Impala does not differentiate simple, expression or script 
as Hive does.

// based on 
https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java#L267
// There are two types of dependencies that are represented as 
edges in the column
// lineage graph:
//a) Projection dependency: This is a dependency between a set 
of source
//columns (base table columns) and a single target (result expr 
or table column).
//This dependency indicates that values of the target depend on 
the values of the source
//columns.
//b) Predicate dependency: This is a dependency between a set 
of target
//columns (or exprs) and a set of source columns (base table 
columns). It indicates that
//the source columns restrict the values of their targets (e.g. 
by participating in
//WHERE clause predicates).


> On May 3, 2019, 7:38 a.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java
> > Lines 26 (patched)
> > 
> >
> > consider removing explicit "private" modifier.

done


> On May 3, 2019, 7:38 a.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java
> > Lines 38 (patched)
> > 
> >
> > make edgeType as enum

done


> On May 3, 2019, 7:38 a.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
> > Lines 36 (patched)
> > 
> >
> > consider changing "id" type from Integer to Long.

good catch! done.

  // https://github.com/apache/impala/blob/master/be/src/util/lineage-util.h#L40
  // Impala id is int64. Therefore, define this field as Long


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70512/#review215013
---


On May 2, 2019, 8:31 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70512/
> ---
> 
> (Updated May 2, 2019, 8:31 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, madhan, Sarath 
> Subramanian, and Xinran Tinney.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Impala generates lineage records for its commands. This new feature will read 
> Impala lineage file, convert the lineage record to Atlas entities and send 
> them to Atlas. In this way, Atlas can get lineage of Impala operation.
> 
> The metadata referred in the lineage are captured in Hive Metastore hook and 
> sent to Atlas. This work is done in ATLAS-3148
> 
> This jira only supports the Impala command "create view". Following jira will 
> add support for more Impala commands.
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge-shim/pom.xml PRE-CREATION 
>   
> addons/impala-bridge-shim/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
>  PRE-CREATION 
>   addons/impala-bridge/pom.xml PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
>  PRE-CREATION 
>   
> 

Re: Review Request 70512: ATLAS-3183: Read Impala lineage record for creating view and send to Atlas

[Na Li via Review Board]

> On May 3, 2019, 7:38 a.m., Sarath Subramanian wrote:
> > addons/impala-bridge-shim/pom.xml
> > Lines 28 (patched)
> > 
> >
> > shim is not needed; since this will be used as a standalone utility.

removed


> On May 3, 2019, 7:38 a.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java
> > Lines 35 (patched)
> > 
> >
> > TEMP_TABLE_PREFIX is never used; consider removing it.

removed


> On May 3, 2019, 7:38 a.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
> > Lines 65 (patched)
> > 
> >
> > return after LOG.warn stmt; might hit NPE in line #73

fixed


> On May 3, 2019, 7:38 a.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java
> > Lines 33 (patched)
> > 
> >
> > why the following queries are not handled?
> > * CREATE TABLE
> > * INSERT INTO TABLE  SELECT
> > * INSERT OVERWRITE TABLE

This patch only supports "create view" command. More command support will be 
done in ATLAS-3184. The goal of this patch is to create Impala-bridge project 
and get integration work end-to-end.


> On May 3, 2019, 7:38 a.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
> > Lines 459 (patched)
> > 
> >
> > columnPosition never used? consider removing it.

removed


> On May 3, 2019, 7:38 a.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
> > Lines 490 (patched)
> > 
> >
> > can we have new model change to create "impala_process" typedef with 
> > supertype "process" as part of this patch? Will be good to see 
> > impala_process connecting hive entities in the server side.

added types for processes. Reuse table, db types

public enum ImpalaDataType {

IMPALA_PROCESS,
IMPALA_PROCESS_EXECUTION,
IMPALA_COLUMN_LINEAGE;

public String getName() {
return name().toLowerCase();
}
}


> On May 3, 2019, 7:38 a.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
> > Lines 71 (patched)
> > 
> >
> > what cases do we want to skip impala_process creation?

I changed the code. get inputs and outputs first. If one of them is empty, skip 
the process.


> On May 3, 2019, 7:38 a.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
> > Lines 133 (patched)
> > 
> >
> > remove edges==null check; CollectionUtils.isEmpty(edges) does null 
> > check as well.

removed


> On May 3, 2019, 7:38 a.m., Sarath Subramanian wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
> > Lines 138 (patched)
> > 
> >
> > lineageInputsCount is unused; consider removing if not used.

removed


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70512/#review215013
---


On May 2, 2019, 8:31 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70512/
> ---
> 
> (Updated May 2, 2019, 8:31 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, madhan, Sarath 
> Subramanian, and Xinran Tinney.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Impala generates lineage records for its commands. This new feature will read 
> Impala lineage file, convert the lineage record to Atlas entities and send 
> them to Atlas. In this way, Atlas can get lineage of Impala operation.
> 
> The metadata referred in the lineage are captured in Hive Metastore hook and 
> sent to Atlas. This work is done in ATLAS-3148
> 
> This jira only supports the Impala command "create view". Following jira will 
> add support for more Impala commands.
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge-shim/pom.xml 

Re: Review Request 70512: ATLAS-3183: Read Impala lineage record for creating view and send to Atlas

[Na Li via Review Board]

> On May 3, 2019, 2:01 a.m., Madhan Neethiraj wrote:
> > addons/impala-bridge-shim/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
> > Lines 35 (patched)
> > 
> >
> > Shim is not needed for ImpalaLineageTool. It is necessary only for the 
> > classes that run within host components (like Hive, HBase). Please remove 
> > this class.

done


> On May 3, 2019, 2:01 a.m., Madhan Neethiraj wrote:
> > addons/impala-bridge/pom.xml
> > Lines 47 (patched)
> > 
> >
> > Following libraries are not needed, as the implementation uses 
> > notifications, instead of REST APIs. Please review and remove these and 
> > other unused libraries from here:
> >   - atlas-client-v1
> >   - atlas-client-v2
> >   - hdfs-model

I need atlas-client-v2 for testing (fetching process entity from Atlas server). 
I mark the scope of libraries I need as "test"


> On May 3, 2019, 2:01 a.m., Madhan Neethiraj wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
> > Lines 52 (patched)
> > 
> >
> > 'implements Runnable' seems unnecessary here, as no threads are created 
> > to run the tool.

removed Runnable


> On May 3, 2019, 2:01 a.m., Madhan Neethiraj wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
> > Lines 59 (patched)
> > 
> >
> > Instead of using 'static' fields (args, instance), consider using class 
> > members and instantiate the class from main().
> > 
> > public class ImpalaLineageTool {
> >   public static void main(String[] args) {
> > ImpalaLineageTool instance = new ImpalaLineageTool(args);
> > 
> > instance.run();
> >   }
> > 
> >   private ImpalaLineageTool(String[] args) {
> > this.args = args;
> >   }
> > 
> >   public void run() {
> > // ..
> >   }
> > }

done.


> On May 3, 2019, 2:01 a.m., Madhan Neethiraj wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
> > Lines 61 (patched)
> > 
> >
> > Consider avoiding test specific details (like 'isUnitTest') from 
> > product code.

removed


> On May 3, 2019, 2:01 a.m., Madhan Neethiraj wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
> > Lines 80 (patched)
> > 
> >
> > log message seems incorrect. Please review the implementation to make 
> > sure that current directoy is read when arguments are not provided.

I changed the code to return if the desired number of input arguments are not 
provided


> On May 3, 2019, 2:01 a.m., Madhan Neethiraj wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
> > Lines 87 (patched)
> > 
> >
> > - args.length may be less than 2 here - see line #79 above
> > - instead of using args[1] here, consider initializing following 
> > variable after line #83 and use it here:
> >   String directoryName = args[1];

I add directoryName as instance variable, which are set in 
getCurrentFiles(args);. So following processing can use it directly.


> On May 3, 2019, 2:01 a.m., Madhan Neethiraj wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
> > Lines 91 (patched)
> > 
> >
> > info level log will be useful here.

changed to LOG.info


> On May 3, 2019, 2:01 a.m., Madhan Neethiraj wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
> > Lines 123 (patched)
> > 
> >
> > It will help to use consistent stype in use of '{'. Please consider 
> > moving '{' to end of previous line. And review other such occurrences as 
> > well.

done


> On May 3, 2019, 2:01 a.m., Madhan Neethiraj wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
> > Lines 124 (patched)
> > 
> >
> > It will be useful to include name of the file in the log:
> >   LOG.info("deleted lineage file {}", currentFile.getAbsolutePath());
> >   
> >   LOG.info("failed to delete lineage file {}", 
> > currentFile.getAbsolutePath());
> > 
> > Please review and update other such occurrences.

added names


> On May 3, 2019, 2:01 a.m., Madhan Neethiraj 

Re: Review Request 70512: ATLAS-3183: Read Impala lineage record for creating view and send to Atlas

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70512/
---

(Updated May 2, 2019, 8:31 p.m.)


Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, madhan, Sarath 
Subramanian, and Xinran Tinney.


Repository: atlas


Description
---

Impala generates lineage records for its commands. This new feature will read 
Impala lineage file, convert the lineage record to Atlas entities and send them 
to Atlas. In this way, Atlas can get lineage of Impala operation.

The metadata referred in the lineage are captured in Hive Metastore hook and 
sent to Atlas. This work is done in ATLAS-3148

This jira only supports the Impala command "create view". Following jira will 
add support for more Impala commands.


Diffs
-

  addons/impala-bridge-shim/pom.xml PRE-CREATION 
  
addons/impala-bridge-shim/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
 PRE-CREATION 
  addons/impala-bridge/pom.xml PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataTypes.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageQuery.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
 PRE-CREATION 
  addons/impala-bridge/src/main/resources/atlas-log4j.xml PRE-CREATION 
  addons/impala-bridge/src/main/resources/import-impala.sh PRE-CREATION 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
 PRE-CREATION 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
 PRE-CREATION 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
 PRE-CREATION 
  addons/impala-bridge/src/test/resources/atlas-application.properties 
PRE-CREATION 
  addons/impala-bridge/src/test/resources/atlas-log4j.xml PRE-CREATION 
  addons/impala-bridge/src/test/resources/hive-site.xml PRE-CREATION 
  addons/impala-bridge/src/test/resources/impala1.json PRE-CREATION 
  addons/impala-bridge/src/test/resources/impala2.json PRE-CREATION 
  addons/impala-bridge/src/test/resources/impala3.json PRE-CREATION 
  addons/impala-bridge/src/test/resources/users-credentials.properties 
PRE-CREATION 
  pom.xml 7de5d31 


Diff: https://reviews.apache.org/r/70512/diff/13/


Testing
---

Run the tool in real cluster that has Atlas server with Impala lineage file as 
input for creating view. The Atlas UI displays hive_lineage lineage and 
hive_column_lineage.
Add new integration tests and they pass


Thanks,

Na Li

Re: Review Request 70536: ATLAS-3157: Add Integration tests for Hive metastore hook

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70536/#review214855
---




addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java
Lines 110 (patched)


It is not obvious here why you do it?

Can you add comment that driver using this configuration will be used to 
test HiveHook, so HMS hook is not configurated?



addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java
Line 131 (original), 137 (patched)


add comment



addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java
Line 138 (original), 146 (patched)


to make the logic clear, you can make runCommand(String cmd) as abstract 
function. And the child class implement it.

For example, HiveHookIT implements with driver , and HiveMetastoreHookIT 
implements with driverWithoutContext.


- Na Li


On April 24, 2019, 1:34 a.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70536/
> ---
> 
> (Updated April 24, 2019, 1:34 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, keval bhatt, Na 
> Li, Le Ma, Madhan Neethiraj, Nikhil Bonte, and Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-3157
> https://issues.apache.org/jira/browse/ATLAS-3157
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Add Integration tests for Hive metastore hook
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 
> c8c53c473 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 677043a01 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveMetastoreHookIT.java
>  PRE-CREATION 
>   addons/hive-bridge/src/test/resources/hive-site.xml 4605ae322 
> 
> 
> Diff: https://reviews.apache.org/r/70536/diff/1/
> 
> 
> Testing
> ---
> 
> Precommit: 
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1048/console
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

Re: Review Request 70536: ATLAS-3157: Add Integration tests for Hive metastore hook

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70536/#review214853
---




addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java
Lines 349 (patched)


There could be a problem in calling waitFor() to verify that the entity is 
not registered.

Notice in waitFor(), the Predicate is first called. If not exception 
happens, the test returns. If there is not found exception, it waits and try 
again. It is to verify something eventually happens with delay.

To test entity not registered, we need to wait first, then test to avoid 
the situation that
1) query of entity happens first, not found, and test return
2) registration happens later
3) The test does not detact registration actually happens

protected void waitFor(int timeout, Predicate predicate) throws 
Exception {
ParamChecker.notNull(predicate, "predicate");
long mustEnd = System.currentTimeMillis() + timeout;

while (true) {
try {
predicate.evaluate();
return;
} catch(Error | Exception e) {
if (System.currentTimeMillis() >= mustEnd) {
fail("Assertions failed. Failing after waiting for 
timeout " + timeout + " msecs", e);
}
LOG.debug("Waiting up to {} msec as assertion failed", 
mustEnd - System.currentTimeMillis(), e);
Thread.sleep(5000);
}
}
}


- Na Li


On April 24, 2019, 1:34 a.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70536/
> ---
> 
> (Updated April 24, 2019, 1:34 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, keval bhatt, Na 
> Li, Le Ma, Madhan Neethiraj, Nikhil Bonte, and Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-3157
> https://issues.apache.org/jira/browse/ATLAS-3157
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Add Integration tests for Hive metastore hook
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 
> c8c53c473 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 677043a01 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveMetastoreHookIT.java
>  PRE-CREATION 
>   addons/hive-bridge/src/test/resources/hive-site.xml 4605ae322 
> 
> 
> Diff: https://reviews.apache.org/r/70536/diff/1/
> 
> 
> Testing
> ---
> 
> Precommit: 
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1048/console
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

Re: Review Request 70505: CDPD-252:parse metadata/Lineage files generated by Impala

[Na Li via Review Board]

> On April 19, 2019, 3:27 p.m., Na Li wrote:
> > addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/LineageVertex.java
> > Lines 27 (patched)
> > 
> >
> > can you make the type to be Integer? So it is consistent with the id 
> > type ion LineageEdge "private List sources;"
> 
> Xinran Tinney wrote:
> Thre are 3 elements in LieageVetext 
> private String id;
> private String vertexType;
> private String vertexId;
> 
> The one in private List sources is using the first one "id", I 
> can not make vertexId an Integer because it is not a number. e.g. 
> "vertexId":"db_1.view_1.count"
> 
> Xinran Tinney wrote:
> Do you mean the change "id" to Integer?

sorry. I meant "id" should be Integer. vertexId should be String


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70505/#review214761
---


On April 18, 2019, 8:33 p.m., Xinran Tinney wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70505/
> ---
> 
> (Updated April 18, 2019, 8:33 p.m.)
> 
> 
> Review request for atlas and Na Li.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> the code is implemented in Java.
> 
> 1) It takes Impala file location and file name prefix as input. 
> 2) It reads its own record-keeping file to get the offset of the Impala file 
> it has already read. Then read the new records from Impala file, update the 
> record-keeping file to append the new offset of the Impala file.
> 3) It parses the new records from Impala file into internal data structure as 
> output.
> 4) If the Impala file name has changed, start a new file for record-keeping 
> and remove the old record-keeping file.
> 5) Implemented unit test to verify the behavior.
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge/impala1.txt PRE-CREATION 
>   addons/impala-bridge/impala2.txt PRE-CREATION 
>   addons/impala-bridge/pom.xml PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/ImpalaBridge.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/LineageEdge.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/LineageQuery.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/LineageVertex.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/bridge/TestImpalaBridge.java
>  PRE-CREATION 
>   pom.xml 76341388b 
> 
> 
> Diff: https://reviews.apache.org/r/70505/diff/1/
> 
> 
> Testing
> ---
> 
> unit testing and mvn clean install -DskipTests
> 
> 
> Thanks,
> 
> Xinran Tinney
> 
>

Re: Review Request 70505: CDPD-252:parse metadata/Lineage files generated by Impala

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70505/#review214765
---




addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/LineageQuery.java
Lines 31 (patched)


Your code does not work for this lineage record. That is why it is critical 
to finish "Collect the lineage records for all operations that Impala generate 
lineage records" before you can claim you are done with this jira!

{  
   "1554132988258":{  
  "query_id":"544e8c5ab3b6a634:650a9d9e",
  "session_id":"044dbeb34e24aa5d:ad43167db6878180",
  "start_time":"2019-04-01 08:36:27.842860",
  "authorization_failure":false,
  "status":"",
  "user":"imp...@gce.cloudera.com",
  "impersonator":null,
  "statement_type":"CREATE_TABLE",
  "network_address":":::172.31.114.111:41200",
  "sql_statement":"CREATE EXTERNAL TABLE IF NOT EXISTS 
db_atlqkltata.tbl_atlqkltata_1(col_tbl_atlqkltata_1_id 
INT,col_tbl_atlqkltata_1_name STRING,col_tbl_atlqkltata_1_street 
STRING,col_tbl_atlqkltata_1_city STRING,col_tbl_atlqkltata_1_state 
STRING,col_tbl_atlqkltata_1_zipcode INT)",
  "catalog_objects":[  
 {  
"name":"db_atlqkltata.tbl_atlqkltata_1",
"object_type":"TABLE",
"privilege":"CREATE"
 }
  ]
   }
}


- Na Li


On April 19, 2019, 3:51 p.m., Xinran Tinney wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70505/
> ---
> 
> (Updated April 19, 2019, 3:51 p.m.)
> 
> 
> Review request for atlas and Na Li.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> the code is implemented in Java.
> 
> 1) It takes Impala file location and file name prefix as input. 
> 2) It reads its own record-keeping file to get the offset of the Impala file 
> it has already read. Then read the new records from Impala file, update the 
> record-keeping file to append the new offset of the Impala file.
> 3) It parses the new records from Impala file into internal data structure as 
> output.
> 4) If the Impala file name has changed, start a new file for record-keeping 
> and remove the old record-keeping file.
> 5) Implemented unit test to verify the behavior.
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge/impala1.txt PRE-CREATION 
>   addons/impala-bridge/impala2.txt PRE-CREATION 
>   addons/impala-bridge/pom.xml PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/ImpalaBridge.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/LineageEdge.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/LineageQuery.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/LineageVertex.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/bridge/TestImpalaBridge.java
>  PRE-CREATION 
>   pom.xml 76341388b 
> 
> 
> Diff: https://reviews.apache.org/r/70505/diff/2/
> 
> 
> Testing
> ---
> 
> unit testing and mvn clean install -DskipTests
> 
> 
> Thanks,
> 
> Xinran Tinney
> 
>

Re: Review Request 70505: CDPD-252:parse metadata/Lineage files generated by Impala

[Na Li via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70505/#review214761
---




addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/LineageVertex.java
Lines 27 (patched)


can you make the type to be Integer? So it is consistent with the id type 
ion LineageEdge "private List sources;"


- Na Li


On April 18, 2019, 8:33 p.m., Xinran Tinney wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70505/
> ---
> 
> (Updated April 18, 2019, 8:33 p.m.)
> 
> 
> Review request for atlas and Na Li.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> the code is implemented in Java.
> 
> 1) It takes Impala file location and file name prefix as input. 
> 2) It reads its own record-keeping file to get the offset of the Impala file 
> it has already read. Then read the new records from Impala file, update the 
> record-keeping file to append the new offset of the Impala file.
> 3) It parses the new records from Impala file into internal data structure as 
> output.
> 4) If the Impala file name has changed, start a new file for record-keeping 
> and remove the old record-keeping file.
> 5) Implemented unit test to verify the behavior.
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge/impala1.txt PRE-CREATION 
>   addons/impala-bridge/impala2.txt PRE-CREATION 
>   addons/impala-bridge/pom.xml PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/ImpalaBridge.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/LineageEdge.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/LineageQuery.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/bridge/LineageVertex.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/bridge/TestImpalaBridge.java
>  PRE-CREATION 
>   pom.xml 76341388b 
> 
> 
> Diff: https://reviews.apache.org/r/70505/diff/1/
> 
> 
> Testing
> ---
> 
> unit testing and mvn clean install -DskipTests
> 
> 
> Thanks,
> 
> Xinran Tinney
> 
>