Re: Review Request 73595: ATLAS-4428 : Enhance Atlas hook messages to capture hook version & message source

2021-10-06 Thread Sarath Subramanian 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73595/#review223579
---


Fix it, then Ship it!




Add tests for cross compatability. If older Atlas server can deserialize the 
new message structure added by new hook and the other way (older hook sending 
message to new Atlas server).


addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
Line 56 (original)


nit: revert new line change



addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
Line 259 (original)


nit: revert new line change



intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java
Lines 62 (patched)


move private methods after public methods


- Sarath Subramanian


On Oct. 6, 2021, 8:13 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73595/
> ---
> 
> (Updated Oct. 6, 2021, 8:13 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4428
> https://issues.apache.org/jira/browse/ATLAS-4428
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas hook messages doesn't specify the source information of the messages 
> coming i.e. which hook source - HiveServer2, HiveMetastore, Impala, Spark, 
> HBase, Kafka.
> 
> Each hook message already captures, the msgSourceIP, msgCreatedBy and 
> msgCreationTime info. In addition to these, we should capture the msgSource 
> information as well.
> 
> When messages come from different hook versions, Atlas server should be aware 
> of the version of different hook messages to process the message based on the 
> hook version.
> 
> 
> Diffs
> -
> 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java
>  8c09d3394 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  0ab06f2c5 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 3cc7b3bc6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveMetastoreHookImpl.java
>  6a492c214 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  10ae08f6d 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> 4785960a7 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  779c5cb20 
>   common/src/main/java/org/apache/atlas/repository/Constants.java aea0c1311 
>   
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationBaseMessage.java
>  ff45d5713 
>   
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java
>  5869910cb 
>   intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java 
> PRE-CREATION 
>   intg/src/main/resources/atlas-buildinfo.properties PRE-CREATION 
>   notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
>  c45a1da95 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  3d8d9cc0a 
>   
> notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
>  0c92c300e 
>   notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java 
> 1ae7c278c 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
>  8078a6ceb 
>   
> notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationDeserializerTest.java
>  13eafb6b1 
>   
> notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java
>  d04817024 
> 
> 
> Diff: https://reviews.apache.org/r/73595/diff/7/
> 
> 
> Testing
> ---
> 
> mvn clean install
> mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
> testcases running successfully
> 
> 
> Thanks,
> 
> chaitali
> 
>

Re: Review Request 73595: ATLAS-4428 : Enhance Atlas hook messages to capture hook version & message source

2021-10-06 Thread Ashutosh Mestry via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73595/#review223578
---


Fix it, then Ship it!




Ship It!


intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java
Lines 55 (patched)


Rename hookVersion to version



intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java
Lines 59 (patched)


Rename: storedHookVersion to storedVersion.



intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java
Lines 85 (patched)


setSource



intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java
Lines 87 (patched)


Rename: getVersion


- Ashutosh Mestry


On Oct. 6, 2021, 3:13 p.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73595/
> ---
> 
> (Updated Oct. 6, 2021, 3:13 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4428
> https://issues.apache.org/jira/browse/ATLAS-4428
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas hook messages doesn't specify the source information of the messages 
> coming i.e. which hook source - HiveServer2, HiveMetastore, Impala, Spark, 
> HBase, Kafka.
> 
> Each hook message already captures, the msgSourceIP, msgCreatedBy and 
> msgCreationTime info. In addition to these, we should capture the msgSource 
> information as well.
> 
> When messages come from different hook versions, Atlas server should be aware 
> of the version of different hook messages to process the message based on the 
> hook version.
> 
> 
> Diffs
> -
> 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java
>  8c09d3394 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  0ab06f2c5 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 3cc7b3bc6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveMetastoreHookImpl.java
>  6a492c214 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  10ae08f6d 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> 4785960a7 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  779c5cb20 
>   common/src/main/java/org/apache/atlas/repository/Constants.java aea0c1311 
>   
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationBaseMessage.java
>  ff45d5713 
>   
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java
>  5869910cb 
>   intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java 
> PRE-CREATION 
>   intg/src/main/resources/atlas-buildinfo.properties PRE-CREATION 
>   notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
>  c45a1da95 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  3d8d9cc0a 
>   
> notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
>  0c92c300e 
>   notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java 
> 1ae7c278c 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
>  8078a6ceb 
>   
> notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationDeserializerTest.java
>  13eafb6b1 
>   
> notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java
>  d04817024 
> 
> 
> Diff: https://reviews.apache.org/r/73595/diff/7/
> 
> 
> Testing
> ---
> 
> mvn clean install
> mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
> testcases running successfully
> 
> 
> Thanks,
> 
> chaitali
> 
>

[jira] [Commented] (ATLAS-4246) Make Kafka Interface aware of Kafka Schema Registry

2021-10-06 Thread Ashutosh Mestry (Jira) 


[ 
https://issues.apache.org/jira/browse/ATLAS-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425068#comment-17425068
 ] 

Ashutosh Mestry commented on ATLAS-4246:


[~aileeen] We will proceed with merging this. I will update RB, if we need any 
updates.

[~viktorsomogyi] Thanks for taking care of this.

> Make Kafka Interface aware of Kafka Schema Registry
> ---
>
> Key: ATLAS-4246
> URL: https://issues.apache.org/jira/browse/ATLAS-4246
> Project: Atlas
>  Issue Type: Improvement
>  Components: kafka-integration
>Affects Versions: 2.1.0, 3.0.0
>Reporter: Aileen Toleikis
>Assignee: Viktor Somogyi-Vass
>Priority: Major
>  Labels: Kafka
> Fix For: 3.0.0, 2.3.0
>
>
> Kafka Community is using Schema Registry more and more heavily but as Atlas 
> is currently unaware of this, this extension helps Atlas make use of the 
> Schemas.
>  
> We have tested this extension and we have production environments where Atlas 
> will not be allowed without schema registry access. We have received feedback 
> that this extension would be sufficient to allow production use.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-4246) Make Kafka Interface aware of Kafka Schema Registry

2021-10-06 Thread Viktor Somogyi-Vass (Jira) 


[ 
https://issues.apache.org/jira/browse/ATLAS-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425044#comment-17425044
 ] 

Viktor Somogyi-Vass commented on ATLAS-4246:


+1, reviewed it, looks good.

> Make Kafka Interface aware of Kafka Schema Registry
> ---
>
> Key: ATLAS-4246
> URL: https://issues.apache.org/jira/browse/ATLAS-4246
> Project: Atlas
>  Issue Type: Improvement
>  Components: kafka-integration
>Affects Versions: 2.1.0, 3.0.0
>Reporter: Aileen Toleikis
>Assignee: Viktor Somogyi-Vass
>Priority: Major
>  Labels: Kafka
> Fix For: 3.0.0, 2.3.0
>
>
> Kafka Community is using Schema Registry more and more heavily but as Atlas 
> is currently unaware of this, this extension helps Atlas make use of the 
> Schemas.
>  
> We have tested this extension and we have production environments where Atlas 
> will not be allowed without schema registry access. We have received feedback 
> that this extension would be sufficient to allow production use.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 73595: ATLAS-4428 : Enhance Atlas hook messages to capture hook version & message source

2021-10-06 Thread chaitali 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73595/
---

(Updated Oct. 6, 2021, 3:13 p.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and 
Sarath Subramanian.


Bugs: ATLAS-4428
https://issues.apache.org/jira/browse/ATLAS-4428


Repository: atlas


Description
---

Atlas hook messages doesn't specify the source information of the messages 
coming i.e. which hook source - HiveServer2, HiveMetastore, Impala, Spark, 
HBase, Kafka.

Each hook message already captures, the msgSourceIP, msgCreatedBy and 
msgCreationTime info. In addition to these, we should capture the msgSource 
information as well.

When messages come from different hook versions, Atlas server should be aware 
of the version of different hook messages to process the message based on the 
hook version.


Diffs (updated)
-

  
addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java 
8c09d3394 
  
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
 0ab06f2c5 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
3cc7b3bc6 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveMetastoreHookImpl.java
 6a492c214 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
 10ae08f6d 
  addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
4785960a7 
  
addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
 779c5cb20 
  common/src/main/java/org/apache/atlas/repository/Constants.java aea0c1311 
  
intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationBaseMessage.java
 ff45d5713 
  
intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java
 5869910cb 
  intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java 
PRE-CREATION 
  intg/src/main/resources/atlas-buildinfo.properties PRE-CREATION 
  notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144 
  
notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
 c45a1da95 
  
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
 3d8d9cc0a 
  
notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
 0c92c300e 
  notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java 1ae7c278c 
  
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
 8078a6ceb 
  
notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationDeserializerTest.java
 13eafb6b1 
  
notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java
 d04817024 


Diff: https://reviews.apache.org/r/73595/diff/7/

Changes: https://reviews.apache.org/r/73595/diff/6-7/


Testing
---

mvn clean install
mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
testcases running successfully


Thanks,

chaitali

[GitHub] [atlas] martin-g commented on pull request #147: ATLAS-4159 Update Apache Storm to 2.2.0

2021-10-06 Thread GitBox 


martin-g commented on pull request #147:
URL: https://github.com/apache/atlas/pull/147#issuecomment-935972369


   @ashutoshm The build passed here with Storm 2.3.0. I've updated this PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@atlas.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org