Re: Review Request 73791: ATLAS-4523 - Build failure on Atlas docs

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73791/#review223944
---


Ship it!




Ship It!

- Ashutosh Mestry


On Jan. 7, 2022, 12:20 p.m., Farhan Khan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73791/
> ---
> 
> (Updated Jan. 7, 2022, 12:20 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab and Prasad Pawar.
> 
> 
> Bugs: ATLAS-4523
> https://issues.apache.org/jira/browse/ATLAS-4523
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> docz has transitive dependencies which require a newer version of babel (at 
> least 7.13.0). Requiring these old versions ends up bringing in conflicting 
> versions of babel which breaks the atlas docs project.
> 
> 
> Diffs
> -
> 
>   docs/package.json f38a965ed 
> 
> 
> Diff: https://reviews.apache.org/r/73791/diff/1/
> 
> 
> Testing
> ---
> 
> Tested on local, build get success after the babel version upgrade.
> 
> 
> Thanks,
> 
> Farhan Khan
> 
>

Re: Review Request 73725: ATLAS-4493 : Upgrade netty to 4.1.68+

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73725/#review223767
---


Ship it!




Ship It!

- Ashutosh Mestry


On Nov. 23, 2021, 9:24 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73725/
> ---
> 
> (Updated Nov. 23, 2021, 9:24 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4493
> https://issues.apache.org/jira/browse/ATLAS-4493
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Upgrading netty to 4.1.69
> 
> 
> Diffs
> -
> 
>   pom.xml 86519daba 
> 
> 
> Diff: https://reviews.apache.org/r/73725/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean install
> mvn clean -DskipTests package -Pdist,embedded-hbase-solr
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/983/consoleFull
> 
> 
> Thanks,
> 
> chaitali
> 
>

Re: Review Request 73671: ATLAS-4463: Infinite loop at Index Health Monitor (IndexRecoveryService)

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73671/#review223686
---


Ship it!




Ship It!

- Ashutosh Mestry


On Oct. 28, 2021, 9:32 p.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73671/
> ---
> 
> (Updated Oct. 28, 2021, 9:32 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4463
> https://issues.apache.org/jira/browse/ATLAS-4463
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-4463: Fixed Infinite loop at Index Health Monitor (IndexRecoveryService)
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/IndexRecoveryService.java
>  2f1161067 
> 
> 
> Diff: https://reviews.apache.org/r/73671/diff/2/
> 
> 
> Testing
> ---
> 
> Run local UT/ITs and Manually ran jetty run and some IT to see the loop is 
> exiting at the end
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>

Re: Review Request 73670: ATLAS-4351 Maven-jetty throwing warnings when running Integration tests for conflicting jars in classpath

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73670/#review223685
---


Ship it!




Ship It!

- Ashutosh Mestry


On Oct. 28, 2021, 9:14 p.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73670/
> ---
> 
> (Updated Oct. 28, 2021, 9:14 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4351
> https://issues.apache.org/jira/browse/ATLAS-4351
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Reverting: ATLAS-4351
> Maven-jetty throwing warnings when running Integration tests for conflicting 
> jars in classpath
> 
> 
> Diffs
> -
> 
>   addons/falcon-bridge/pom.xml 9d56fd8d3 
>   addons/hbase-bridge/pom.xml 151a62a77 
>   addons/hbase-testing-util/pom.xml 8ca962adc 
>   addons/hive-bridge/pom.xml f62140918 
>   addons/impala-bridge/pom.xml 87650e1da 
>   addons/kafka-bridge/pom.xml 2ac19fd20 
>   addons/sqoop-bridge/pom.xml 03861df84 
>   addons/storm-bridge-shim/pom.xml e1e89ceda 
>   addons/storm-bridge/pom.xml 6bfe15b5b 
>   common/pom.xml ebf8ea5ae 
>   graphdb/janus-hbase2/pom.xml 9ad64062f 
>   graphdb/janus/pom.xml f38c4e9a3 
>   pom.xml 84630e76b 
>   test-tools/pom.xml 59c7a9dbf 
>   webapp/pom.xml 443559c8d 
> 
> 
> Diff: https://reviews.apache.org/r/73670/diff/1/
> 
> 
> Testing
> ---
> 
> As there are Precommit job failures I am revrting the changes. Later we can 
> fix the warning issue separately.
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>

Re: Review Request 73671: ATLAS-4463: Infinite loop at Index Health Monitor (IndexRecoveryService)

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73671/#review223684
---


Fix it, then Ship it!




Fix it, then Ship it!


repository/src/main/java/org/apache/atlas/repository/graph/IndexRecoveryService.java
Lines 185 (patched)


Use:

try {
LOG.info("==> Index Health Monitor: Shutdown: Starting...");

// handle the case where thread was not started at all
// and shutdown called
if (shouldRun.get() == false) {
return;
}

shouldRun.set(false);
}
finally {
LOG.info("<== Index Health Monitor: Shutdown: Done!");
}


- Ashutosh Mestry


On Oct. 28, 2021, 9:21 p.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73671/
> ---
> 
> (Updated Oct. 28, 2021, 9:21 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4463
> https://issues.apache.org/jira/browse/ATLAS-4463
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-4463: Fixed Infinite loop at Index Health Monitor (IndexRecoveryService)
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/IndexRecoveryService.java
>  2f1161067 
> 
> 
> Diff: https://reviews.apache.org/r/73671/diff/1/
> 
> 
> Testing
> ---
> 
> Run local UT/ITs and Manually ran jetty run and some IT to see the loop is 
> exiting at the end
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>

Re: Review Request 73669: ATLAS-4462: Atlas tests are failing and PreCommit tests are also failing

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73669/#review223683
---




notification/src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java
Lines 123 (patched)


nit: Fix formatting.



notification/src/main/java/org/apache/atlas/util/CommandHandlerUtility.java
Lines 33 (patched)


Change attemptCount to boolean. That way you caller can simply pass true or 
false.
The logic attemptCount != 0 will depend on caller.



notification/src/main/java/org/apache/atlas/util/CommandHandlerUtility.java
Lines 42 (patched)


Refactor promote to private static final field.



notification/src/main/java/org/apache/atlas/util/CommandHandlerUtility.java
Lines 74 (patched)


Change StringUtils.isBlank to StringUtils.isEmpty.



notification/src/main/java/org/apache/atlas/util/CommandHandlerUtility.java
Lines 78 (patched)


Refactor: Promote to private static final String.


- Ashutosh Mestry


On Oct. 28, 2021, 9:19 p.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73669/
> ---
> 
> (Updated Oct. 28, 2021, 9:19 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4462
> https://issues.apache.org/jira/browse/ATLAS-4462
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-4462: Updated the pom to stop jetty before start, Added daemon flag to 
> true to stop jetty at end of post integration, Added retry logic for Kafka 
> start having port bind issue, Fixed Infinite loop for Index Recovery Service
> 
> 
> Diffs
> -
> 
>   addons/falcon-bridge/pom.xml 9d56fd8d3 
>   addons/hbase-bridge/pom.xml 151a62a77 
>   addons/hive-bridge/pom.xml f62140918 
>   addons/impala-bridge/pom.xml 87650e1da 
>   addons/kafka-bridge/pom.xml 2ac19fd20 
>   addons/sqoop-bridge/pom.xml 03861df84 
>   addons/storm-bridge/pom.xml 6bfe15b5b 
>   notification/src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java 
> 19717fb2b 
>   notification/src/main/java/org/apache/atlas/util/CommandHandlerUtility.java 
> PRE-CREATION 
>   pom.xml 84630e76b 
>   webapp/pom.xml 443559c8d 
> 
> 
> Diff: https://reviews.apache.org/r/73669/diff/2/
> 
> 
> Testing
> ---
> 
> Manually tested the Infinite loop issue at Index Recovery Service, Kafka 
> retry logic for kafka start. Along with this I ran the local tests which 
> passed.
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>

Re: Review Request 73132: ATLAS-4079 : Atlas-Upgrade jackson to 2.11.0+

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73132/#review223681
---


Ship it!




Ship It!

- Ashutosh Mestry


On Jan. 12, 2021, 9:39 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73132/
> ---
> 
> (Updated Jan. 12, 2021, 9:39 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4079
> https://issues.apache.org/jira/browse/ATLAS-4079
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch updates jackson databind version to 2.11.3
> 
> 
> Diffs
> -
> 
>   pom.xml 7bfd68b94 
> 
> 
> Diff: https://reviews.apache.org/r/73132/diff/1/
> 
> 
> Testing
> ---
> 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/324/console
> mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
> Atlas server running
> 
> 
> Thanks,
> 
> chaitali
> 
>

Re: Review Request 73669: ATLAS-4462: Atlas tests are failing and PreCommit tests are also failing

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73669/#review223679
---




notification/src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java
Lines 184 (patched)


Instead of killing the process, why not attempt to use next available port.



notification/src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java
Lines 194 (patched)


This won't work on Windows.



notification/src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java
Lines 207 (patched)


Call this in finally



pom.xml
Line 1688 (original), 1680 (patched)


What is the reason for downgrading this?



repository/src/main/java/org/apache/atlas/repository/graph/IndexRecoveryService.java
Line 70 (original), 72 (patched)


If possible, package this in a separate patch.



repository/src/main/java/org/apache/atlas/repository/graph/IndexRecoveryService.java
Lines 185 (patched)


Change: Index Health Monitor: Shutdown: Starting...



repository/src/main/java/org/apache/atlas/repository/graph/IndexRecoveryService.java
Lines 195 (patched)


Change: Index Health Monitor: Shutdown: Done! Add this to finally block.


- Ashutosh Mestry


On Oct. 28, 2021, 5:08 a.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73669/
> ---
> 
> (Updated Oct. 28, 2021, 5:08 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4462
> https://issues.apache.org/jira/browse/ATLAS-4462
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-4462: Updated the pom to stop jetty before start, Added daemon flag to 
> true to stop jetty at end of post integration, Added retry logic for Kafka 
> start having port bind issue, Fixed Infinite loop for Index Recovery Service
> 
> 
> Diffs
> -
> 
>   addons/falcon-bridge/pom.xml 9d56fd8d3 
>   addons/hbase-bridge/pom.xml 151a62a77 
>   addons/hive-bridge/pom.xml f62140918 
>   addons/impala-bridge/pom.xml 87650e1da 
>   addons/kafka-bridge/pom.xml 2ac19fd20 
>   addons/sqoop-bridge/pom.xml 03861df84 
>   addons/storm-bridge/pom.xml 6bfe15b5b 
>   common/pom.xml ebf8ea5ae 
>   graphdb/janus/pom.xml f38c4e9a3 
>   notification/src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java 
> 19717fb2b 
>   pom.xml 84630e76b 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/IndexRecoveryService.java
>  2f1161067 
>   test-tools/pom.xml 59c7a9dbf 
>   webapp/pom.xml 443559c8d 
> 
> 
> Diff: https://reviews.apache.org/r/73669/diff/1/
> 
> 
> Testing
> ---
> 
> Manually tested the Infinite loop issue at Index Recovery Service, Kafka 
> retry logic for kafka start. Along with this I ran the local tests which 
> passed.
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>

Re: Review Request 73639: ATLAS-4446 : Atlas - Upgrade elasticsearch to 7.14.0

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73639/#review223653
---


Ship it!




Ship It!

- Ashutosh Mestry


On Oct. 21, 2021, 2:50 p.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73639/
> ---
> 
> (Updated Oct. 21, 2021, 2:50 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4446
> https://issues.apache.org/jira/browse/ATLAS-4446
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently Elastic search is at 6.8.17
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 078bb2e5c 
>   pom.xml 84630e76b 
> 
> 
> Diff: https://reviews.apache.org/r/73639/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean package -Pdist,berkeley-elasticsearch
> 
> An additional step is required for the binary built using this profile to be 
> used along with the Atlas distribution. Due to licensing requirements, Atlas 
> does not bundle the BerkeleyDB Java Edition in the tarball.
> 
> You can download the Berkeley DB jar file from the URL: 
> http://download.oracle.com/otn/berkeley-db/je-7.5.11.zip and copy the 
> je-7.5.11.jar to the ${atlas_home}/libext directory.
> 
> Tar can be found in 
> atlas/distro/target/apache-atlas-${project.version}-bin.tar.gz
> 
> 
> Thanks,
> 
> chaitali
> 
>

Re: Review Request 73641: Repair Composite indexes

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73641/#review223593
---


Ship it!




Ship It!

- Ashutosh Mestry


On Oct. 8, 2021, 1:38 p.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73641/
> ---
> 
> (Updated Oct. 8, 2021, 1:38 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> If by any chance, commposite index are not enabled, this patch fixes the 
> composite index from REGISTERED state to ENABLED state
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  7e3b2f440 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
>  1cc7f8b55 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java
>  d30971afc 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/RepairCompositeIndexPatch.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/73641/diff/1/
> 
> 
> Testing
> ---
> 
> manually tested
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 73595: ATLAS-4428 : Enhance Atlas hook messages to capture hook version & message source

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73595/#review223578
---


Fix it, then Ship it!




Ship It!


intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java
Lines 55 (patched)


Rename hookVersion to version



intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java
Lines 59 (patched)


Rename: storedHookVersion to storedVersion.



intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java
Lines 85 (patched)


setSource



intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java
Lines 87 (patched)


Rename: getVersion


- Ashutosh Mestry


On Oct. 6, 2021, 3:13 p.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73595/
> ---
> 
> (Updated Oct. 6, 2021, 3:13 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4428
> https://issues.apache.org/jira/browse/ATLAS-4428
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas hook messages doesn't specify the source information of the messages 
> coming i.e. which hook source - HiveServer2, HiveMetastore, Impala, Spark, 
> HBase, Kafka.
> 
> Each hook message already captures, the msgSourceIP, msgCreatedBy and 
> msgCreationTime info. In addition to these, we should capture the msgSource 
> information as well.
> 
> When messages come from different hook versions, Atlas server should be aware 
> of the version of different hook messages to process the message based on the 
> hook version.
> 
> 
> Diffs
> -
> 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java
>  8c09d3394 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  0ab06f2c5 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 3cc7b3bc6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveMetastoreHookImpl.java
>  6a492c214 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  10ae08f6d 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> 4785960a7 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  779c5cb20 
>   common/src/main/java/org/apache/atlas/repository/Constants.java aea0c1311 
>   
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationBaseMessage.java
>  ff45d5713 
>   
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java
>  5869910cb 
>   intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java 
> PRE-CREATION 
>   intg/src/main/resources/atlas-buildinfo.properties PRE-CREATION 
>   notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
>  c45a1da95 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  3d8d9cc0a 
>   
> notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
>  0c92c300e 
>   notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java 
> 1ae7c278c 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
>  8078a6ceb 
>   
> notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationDeserializerTest.java
>  13eafb6b1 
>   
> notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java
>  d04817024 
> 
> 
> Diff: https://reviews.apache.org/r/73595/diff/7/
> 
> 
> Testing
> ---
> 
> mvn clean install
> mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
> testcases running successfully
> 
> 
> Thanks,
> 
> chaitali
> 
>

Re: Review Request 73595: ATLAS-4428 : Enhance Atlas hook messages to capture hook version & message source

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73595/#review223575
---




intg/pom.xml
Lines 149 (patched)


Remove this and instead add atlas-buildinfo.properties here:

build.epoch=${timestamp}
project.version=${pom.version}
build.version=${pom.version}
vc.revision=${buildNumber}

This will avoid copying as build step and will keep the build version true 
to the project.



notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java
Line 139 (original), 141 (patched)


Add assertEquals verifying contents of MessageSource.


- Ashutosh Mestry


On Oct. 4, 2021, 3:59 p.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73595/
> ---
> 
> (Updated Oct. 4, 2021, 3:59 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4428
> https://issues.apache.org/jira/browse/ATLAS-4428
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas hook messages doesn't specify the source information of the messages 
> coming i.e. which hook source - HiveServer2, HiveMetastore, Impala, Spark, 
> HBase, Kafka.
> 
> Each hook message already captures, the msgSourceIP, msgCreatedBy and 
> msgCreationTime info. In addition to these, we should capture the msgSource 
> information as well.
> 
> When messages come from different hook versions, Atlas server should be aware 
> of the version of different hook messages to process the message based on the 
> hook version.
> 
> 
> Diffs
> -
> 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java
>  8c09d3394 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  0ab06f2c5 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 3cc7b3bc6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveMetastoreHookImpl.java
>  6a492c214 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  10ae08f6d 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> 4785960a7 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  779c5cb20 
>   common/src/main/java/org/apache/atlas/repository/Constants.java aea0c1311 
>   intg/pom.xml 691cdf0a3 
>   
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationBaseMessage.java
>  ff45d5713 
>   
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java
>  5869910cb 
>   intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java 
> PRE-CREATION 
>   notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
>  c45a1da95 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  3d8d9cc0a 
>   
> notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
>  0c92c300e 
>   notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java 
> 1ae7c278c 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
>  8078a6ceb 
>   
> notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationDeserializerTest.java
>  13eafb6b1 
>   
> notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java
>  d04817024 
> 
> 
> Diff: https://reviews.apache.org/r/73595/diff/6/
> 
> 
> Testing
> ---
> 
> mvn clean install
> mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
> testcases running successfully
> 
> 
> Thanks,
> 
> chaitali
> 
>

Re: Review Request 73628: ATLAS-4444: Atlas - Upgrade commons-io to 2.11.0

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73628/#review223573
---



https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/874/

- Ashutosh Mestry


On Oct. 5, 2021, 4:02 p.m., Disha Talreja wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73628/
> ---
> 
> (Updated Oct. 5, 2021, 4:02 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry and Sarath Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch upgrades commons-io 2.8 to 2.11.0
> 
> 
> Diffs
> -
> 
>   pom.xml 47768e9fa 
> 
> 
> Diff: https://reviews.apache.org/r/73628/diff/1/
> 
> 
> Testing
> ---
> 
> 1. Tested locally
> 2. Patched a cluster
> 
> 
> Thanks,
> 
> Disha Talreja
> 
>

Re: Review Request 73628: ATLAS-4444: Atlas - Upgrade commons-io to 2.11.0

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73628/#review223572
---


Ship it!




Ship It!

- Ashutosh Mestry


On Oct. 5, 2021, 4:02 p.m., Disha Talreja wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73628/
> ---
> 
> (Updated Oct. 5, 2021, 4:02 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry and Sarath Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch upgrades commons-io 2.8 to 2.11.0
> 
> 
> Diffs
> -
> 
>   pom.xml 47768e9fa 
> 
> 
> Diff: https://reviews.apache.org/r/73628/diff/1/
> 
> 
> Testing
> ---
> 
> 1. Tested locally
> 2. Patched a cluster
> 
> 
> Thanks,
> 
> Disha Talreja
> 
>

Re: Review Request 73286: Initial commit

[Ashutosh Mestry via Review Board]

> On Oct. 4, 2021, 3:42 p.m., Ashutosh Mestry wrote:
> > Can you please rebase this against latest master?
> 
> Aileen Toleikis wrote:
> Hi Ashutosh!
> We rebased our code to the latest master!
> And we also added schema evolution to the code, therefore there are more 
> new codelines in the diff.
> How do we proceed from here?
> Kind regards,
> Aileen

Thanks! I have asked Viktor from our Kafka team to review the changes.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73286/#review223560
---


On Oct. 5, 2021, 8:06 a.m., Aileen Toleikis wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73286/
> ---
> 
> (Updated Oct. 5, 2021, 8:06 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> The Kafka Community is using Schema Registry more and more heavily but as 
> Atlas is currently unaware of this, this extension helps Atlas make use of 
> the Schemas.
> 
> We have tested this extension and we have production environments where Atlas 
> will not be allowed without schema registry access. We have received feedback 
> that this extension would be sufficient to allow production use.
> 
> Changes:
> 
> initial commit
> 
> 
> added schema creation in Atlas
> 
> 
> added Schema creation, Schema linkage and started to work on versions
> 
> 
> avro_field creation is working. However, there is no connection to schemas yet
> 
> 
> smaller refactorings and improved logging
> 
> 
> added field reference in schemas
> 
> 
> deleted second README and added gitignore
> 
> 
> Smaller changes after rebasing on new version
> 
> 
> moved HTTP REST calls to a separate class (SchemaRegistryConnector.java)
> 
> 
> added testCreateSchema() as test case
> 
> 
> added testCreateField() and testUpdateField() as test case
> 
> 
> added testGetSchemas() and testGetSchemaVersions() as test case
> 
> 
> added testUpdateSchema() as test case
> 
> 
> removed unnecessary comments
> 
> 
> Diffs
> -
> 
>   addons/kafka-bridge/pom.xml 2ac19fd20e0c0322b0d104641ddd5a2ef89bf9d0 
>   
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
>  f9548244364a8f79f346411739348d0d53298c99 
>   
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java
>  PRE-CREATION 
>   
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java
>  0f81b4c37a1d8f84c587a914eb460b290ae36987 
>   
> addons/kafka-bridge/src/test/java/org/apache/atlas/kafka/bridge/KafkaBridgeTest.java
>  f86ceb58fd250d80f3ae3c71085c8c0adbc9116b 
>   pom.xml 47768e9fa7d8abcfc81cd148917400ac478a333a 
> 
> 
> Diff: https://reviews.apache.org/r/73286/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Aileen Toleikis
> 
>

Re: Review Request 73286: Initial commit

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73286/#review223560
---



Can you please rebase this against latest master?

- Ashutosh Mestry


On Sept. 3, 2021, 7:42 a.m., Aileen Toleikis wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73286/
> ---
> 
> (Updated Sept. 3, 2021, 7:42 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> The Kafka Community is using Schema Registry more and more heavily but as 
> Atlas is currently unaware of this, this extension helps Atlas make use of 
> the Schemas.
> 
> We have tested this extension and we have production environments where Atlas 
> will not be allowed without schema registry access. We have received feedback 
> that this extension would be sufficient to allow production use.
> 
> Changes:
> 
> initial commit
> 
> 
> added schema creation in Atlas
> 
> 
> added Schema creation, Schema linkage and started to work on versions
> 
> 
> avro_field creation is working. However, there is no connection to schemas yet
> 
> 
> smaller refactorings and improved logging
> 
> 
> added field reference in schemas
> 
> 
> deleted second README and added gitignore
> 
> 
> Smaller changes after rebasing on new version
> 
> 
> moved HTTP REST calls to a separate class (SchemaRegistryConnector.java)
> 
> 
> added testCreateSchema() as test case
> 
> 
> added testCreateField() and testUpdateField() as test case
> 
> 
> added testGetSchemas() and testGetSchemaVersions() as test case
> 
> 
> added testUpdateSchema() as test case
> 
> 
> removed unnecessary comments
> 
> 
> Diffs
> -
> 
>   addons/kafka-bridge/pom.xml 7fe97eb39897e32f5e2a4104cc06948e1ec48b27 
>   
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
>  bf74c67454120e3bdeebab0d7cc428d6bba8c021 
>   
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java
>  PRE-CREATION 
>   
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java
>  0f81b4c37a1d8f84c587a914eb460b290ae36987 
>   
> addons/kafka-bridge/src/test/java/org/apache/atlas/kafka/bridge/KafkaBridgeTest.java
>  f86ceb58fd250d80f3ae3c71085c8c0adbc9116b 
>   pom.xml 092ba09cee93ca703e29d8ae1afaa20559a06227 
> 
> 
> Diff: https://reviews.apache.org/r/73286/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Aileen Toleikis
> 
>

Re: Review Request 73586: ATLAS-4424: Enhance the Import hive utility to create export zip files and run bulk import

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73586/#review223551
---




addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java
Lines 130 (patched)


Rename: qualifiedNameGuidMap



addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java
Lines 300 (patched)


A get in to map with db as key should simplify this.



addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java
Lines 300 (patched)


A get in to map with db as key should simplify this.



addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java
Lines 415 (patched)


This will make large number of calls to Atlas thereby slowing the operation 
down.



addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java
Lines 529 (patched)


This method converts Table to AtlasEntity and writes to stream. Consider 
separating it in 2 distinct methods. Say 
AtlasEntityWithExtInfo toAtlasEntity(Table table) and write(OutputStream 
stream, AtlasEntityWithExtInfo entity)

The write method will be reused by all other operations.



intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java
Line 56 (original)


This is not necessary.



repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java
Lines 90 (patched)


What is the reason for this?



repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java
Lines 180 (patched)


This is not necessary.



repository/src/test/java/org/apache/atlas/repository/impexp/DataMigrationStatusServiceTest.java
Line 46 (original), 48 (patched)


What is the reason for doing this?


- Ashutosh Mestry


On Sept. 15, 2021, 11 p.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73586/
> ---
> 
> (Updated Sept. 15, 2021, 11 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4424
> https://issues.apache.org/jira/browse/ATLAS-4424
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-4424: Enhanced the Import hive utility to create export zip files and 
> run bulk import
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/src/bin/import-hive.sh fde486f28 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  08471b480 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java
>  PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
> 2c1870444 
>   
> intg/src/main/java/org/apache/atlas/model/migration/MigrationImportStatus.java
>  3430fdae8 
>   
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationStatusService.java
>  a22c68790 
>   
> repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java
>  d56261f78 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/DataMigrationStatusServiceTest.java
>  f1dc990e8 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
> baa040f4c 
> 
> 
> Diff: https://reviews.apache.org/r/73586/diff/1/
> 
> 
> Testing
> ---
> 
> Manually tested the new zip file creation and bulk import to atlas
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>

Re: Review Request 73585: ATLAS-4425: Migration import should be able to import multiple zip files present in a particular path

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73585/#review223550
---



User specifies:
atlas.migration.data.filename=/tmp/atlas-data/golden-?-of-2-atlas.zip

The directory contains:
golden-1-of-2-atlas.zip
golden-2-of-2-atlas.zip


Algo:
- RW permissions for /tmp/atlas-data/archive.
  - If permissions exist, sucessfully imported files are moved to archive 
directory.
  - If not, LOG.WARN is displayed. Implication of this is the subsequent 
startup in migration mode will restart import from the beginning. 
- - Wildcard gets expanded.
- Files are enumerated.
- They are sequentially fed to the importer.
- Successful files are moved to /tmp/atlas-data/archive folder.

- Ashutosh Mestry


On Sept. 23, 2021, 3:52 a.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73585/
> ---
> 
> (Updated Sept. 23, 2021, 3:52 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4425
> https://issues.apache.org/jira/browse/ATLAS-4425
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-442: Added support for Migration import to run on a directory with 
> multip[le zip files
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceDirect.java
>  04342fa52 
>   
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
>  0a2257eae 
>   
> repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java
>  d56261f78 
> 
> 
> Diff: https://reviews.apache.org/r/73585/diff/2/
> 
> 
> Testing
> ---
> 
> Manually tested with multiple zip files at a location which was imported 
> successfully.
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>

Re: Review Request 73585: ATLAS-4425: Migration import should be able to import multiple zip files present in a particular path

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73585/#review223548
---



I have tested this and it works quite well.


repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceDirect.java
Lines 260 (patched)


I think this change is not necessary given that the input will always have 
a typeDefinition.



repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
Line 68 (original), 68 (patched)


Doing this will impact branch-0.8 to 1.0 migration. The old and new 
migration relies on directory vs file existence check.



repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
Lines 93 (patched)


This won't be necessary as well.



repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java
Line 61 (original), 66 (patched)


If the file contains wildcards (? or '*'), expand them using listFiles.



repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java
Line 62 (original), 67 (patched)


Add check to detect archive folder can be created. If not LOG.WARN and 
proceed without archieving.



repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java
Lines 94 (patched)


Modify this to take the user-specified input, expand if necessary. Sort it 
and then return the list of files to be imported.



repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java
Lines 135 (patched)


Be defensive here.

if (this.archiveDir == null) {
return;
}

Doing this will add clarity.


- Ashutosh Mestry


On Sept. 23, 2021, 3:52 a.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73585/
> ---
> 
> (Updated Sept. 23, 2021, 3:52 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4425
> https://issues.apache.org/jira/browse/ATLAS-4425
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-442: Added support for Migration import to run on a directory with 
> multip[le zip files
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceDirect.java
>  04342fa52 
>   
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
>  0a2257eae 
>   
> repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java
>  d56261f78 
> 
> 
> Diff: https://reviews.apache.org/r/73585/diff/2/
> 
> 
> Testing
> ---
> 
> Manually tested with multiple zip files at a location which was imported 
> successfully.
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>

Re: Review Request 73595: ATLAS-4428 : Enhance Atlas hook messages to capture hook version & message source

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73595/#review223547
---



Why would you not include messageInfo in the message itself? That will make the 
message self-sufficient and prevent the ctor changes.


intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java
Lines 54 (patched)


Refactor: Extract method. Call it: getBuildVersion



intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java
Lines 80 (patched)


Since build version will remain the same for a deployment, it will help to 
store this into a static property.


- Ashutosh Mestry


On Sept. 29, 2021, 7:30 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73595/
> ---
> 
> (Updated Sept. 29, 2021, 7:30 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4428
> https://issues.apache.org/jira/browse/ATLAS-4428
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas hook messages doesn't specify the source information of the messages 
> coming i.e. which hook source - HiveServer2, HiveMetastore, Impala, Spark, 
> HBase, Kafka.
> 
> Each hook message already captures, the msgSourceIP, msgCreatedBy and 
> msgCreationTime info. In addition to these, we should capture the msgSource 
> information as well.
> 
> When messages come from different hook versions, Atlas server should be aware 
> of the version of different hook messages to process the message based on the 
> hook version.
> 
> 
> Diffs
> -
> 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java
>  8c09d3394 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  0ab06f2c5 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 3cc7b3bc6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveMetastoreHookImpl.java
>  6a492c214 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  10ae08f6d 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> 4785960a7 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  779c5cb20 
>   intg/pom.xml 05914 
>   
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationBaseMessage.java
>  ff45d5713 
>   
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java
>  5869910cb 
>   intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java 
> PRE-CREATION 
>   notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
>  c45a1da95 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  3d8d9cc0a 
>   
> notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
>  0c92c300e 
>   notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java 
> 1ae7c278c 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
>  8078a6ceb 
>   
> notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationDeserializerTest.java
>  13eafb6b1 
>   
> notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java
>  d04817024 
> 
> 
> Diff: https://reviews.apache.org/r/73595/diff/3/
> 
> 
> Testing
> ---
> 
> mvn clean install
> mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
> testcases running successfully
> 
> 
> Thanks,
> 
> chaitali
> 
>

Review Request 73606: AtlasPatchManager: Avoid Race Condition in HA Mode

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73606/
---

Review request for atlas, Jayendra Parab, Mandar Ambawane, Nikhil Bonte, Nixon 
Rodrigues, and Sarath Subramanian.


Bugs: ATLAS-4358
https://issues.apache.org/jira/browse/ATLAS-4358


Repository: atlas


Description
---

**Problem**
During start-up in HA mode, the _AtlasPatchManager_ tries to read the patch 
data while another instanace of Atlas is trying to create the indexes.

This causes indexes not to be created correctly. When this happens, the patch 
registry ends up doing a a full database scan. With sizable data, this can 
result in long startup times. This is detremental for setup that use 
containerized cluster setup.

**Approach**
Initialize the _PatchContext_ in the _applyAll_ method.


Diffs
-

  
repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java
 e2a38ab52 


Diff: https://reviews.apache.org/r/73606/diff/1/


Testing
---

**PC Build**
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/864/


Thanks,

Ashutosh Mestry

Re: Review Request 73599: Conditionally Disable 'Session Inactivity Timeout'

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73599/
---

(Updated Sept. 22, 2021, 8:35 p.m.)


Review request for atlas, Nikhil Bonte, Nixon Rodrigues, Prasad Pawar, and 
Sarath Subramanian.


Changes
---

Updates include: Minor refactoring for clarity.


Bugs: ATLAS-4435
https://issues.apache.org/jira/browse/ATLAS-4435


Repository: atlas


Description
---

Please see JIRA for details.

Additional info:
* UI pop-up dialog looks at the value passed in the response of 'admin/session' 
REST endpoint. Not passing the configuration value, disables the feature from 
UI (Smart!).


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java fa519ef5a 
  
webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
 b8d21b9bb 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
baa040f4c 
  
webapp/src/main/java/org/apache/atlas/web/security/AtlasAuthenticationSuccessHandler.java
 1b1a80826 


Diff: https://reviews.apache.org/r/73599/diff/3/

Changes: https://reviews.apache.org/r/73599/diff/2-3/


Testing
---

Manual testing:
* Verified against Knox SSO enabled cluster.
* User-name password enabled cluster.


Thanks,

Ashutosh Mestry

Re: Review Request 73599: Conditionally Disable 'Session Inactivity Timeout'

[Ashutosh Mestry via Review Board]

> On Sept. 21, 2021, 7:02 p.m., Sailaja Polavarapu wrote:
> > webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
> > Line 359 (original), 374 (patched)
> > 
> >
> > I think it will be intuitive to move logoutHandler != null check to 
> > inner if condition on line #375.

Please see my comment below.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73599/#review223509
---


On Sept. 22, 2021, 5:42 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73599/
> ---
> 
> (Updated Sept. 22, 2021, 5:42 p.m.)
> 
> 
> Review request for atlas, Nikhil Bonte, Nixon Rodrigues, Prasad Pawar, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4435
> https://issues.apache.org/jira/browse/ATLAS-4435
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Please see JIRA for details.
> 
> Additional info:
> * UI pop-up dialog looks at the value passed in the response of 
> 'admin/session' REST endpoint. Not passing the configuration value, disables 
> the feature from UI (Smart!).
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java fa519ef5a 
>   
> webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
>  b8d21b9bb 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
> baa040f4c 
>   
> webapp/src/main/java/org/apache/atlas/web/security/AtlasAuthenticationSuccessHandler.java
>  1b1a80826 
> 
> 
> Diff: https://reviews.apache.org/r/73599/diff/2/
> 
> 
> Testing
> ---
> 
> Manual testing:
> * Verified against Knox SSO enabled cluster.
> * User-name password enabled cluster.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 73599: Conditionally Disable 'Session Inactivity Timeout'

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73599/
---

(Updated Sept. 22, 2021, 5:42 p.m.)


Review request for atlas, Nikhil Bonte, Nixon Rodrigues, Prasad Pawar, and 
Sarath Subramanian.


Changes
---

Updates include: 
- Addressed review comments.
- Minor refactoring.


Bugs: ATLAS-4435
https://issues.apache.org/jira/browse/ATLAS-4435


Repository: atlas


Description
---

Please see JIRA for details.

Additional info:
* UI pop-up dialog looks at the value passed in the response of 'admin/session' 
REST endpoint. Not passing the configuration value, disables the feature from 
UI (Smart!).


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java fa519ef5a 
  
webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
 b8d21b9bb 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
baa040f4c 
  
webapp/src/main/java/org/apache/atlas/web/security/AtlasAuthenticationSuccessHandler.java
 1b1a80826 


Diff: https://reviews.apache.org/r/73599/diff/2/

Changes: https://reviews.apache.org/r/73599/diff/1-2/


Testing
---

Manual testing:
* Verified against Knox SSO enabled cluster.
* User-name password enabled cluster.


Thanks,

Ashutosh Mestry

Re: Review Request 73599: Conditionally Disable 'Session Inactivity Timeout'

[Ashutosh Mestry via Review Board]

> On Sept. 22, 2021, 4:58 p.m., Sarath Subramanian wrote:
> > webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
> > Line 311 (original), 325 (patched)
> > 
> >
> > would 'sessionTimeout' be initialized by then? I see it getting 
> > initialized only at getConfiguration(), so won't it be always take default 
> > ('0'). line 326 will never be reached.

This follows the AuthenticationFilter initialization path. All our secrets 
(which are also part of the config) get initialized here. In short, yes, it 
does get initialized.


> On Sept. 22, 2021, 4:58 p.m., Sarath Subramanian wrote:
> > webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
> > Line 360 (original), 375 (patched)
> > 
> >
> > add null check ('logoutHandler') to inner 'if' - closer to where it is 
> > used for better readabality.

Doing this will change the flow. I want to be able to resort to the state where 
this code did not exist. Hence I have made it this way.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73599/#review223517
---


On Sept. 21, 2021, 6:38 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73599/
> ---
> 
> (Updated Sept. 21, 2021, 6:38 p.m.)
> 
> 
> Review request for atlas, Nikhil Bonte, Nixon Rodrigues, Prasad Pawar, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4435
> https://issues.apache.org/jira/browse/ATLAS-4435
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Please see JIRA for details.
> 
> Additional info:
> * UI pop-up dialog looks at the value passed in the response of 
> 'admin/session' REST endpoint. Not passing the configuration value, disables 
> the feature from UI (Smart!).
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java fa519ef5a 
>   
> webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
>  b8d21b9bb 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
> baa040f4c 
>   
> webapp/src/main/java/org/apache/atlas/web/security/AtlasAuthenticationSuccessHandler.java
>  1b1a80826 
> 
> 
> Diff: https://reviews.apache.org/r/73599/diff/1/
> 
> 
> Testing
> ---
> 
> Manual testing:
> * Verified against Knox SSO enabled cluster.
> * User-name password enabled cluster.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 73589: Indexing Monitor and Recovery

[Ashutosh Mestry via Review Board]

> On Sept. 21, 2021, 7:35 p.m., Sarath Subramanian wrote:
> > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
> > Lines 215 (patched)
> > 
> >
> > in that case - only if user explicitly sets the config in atlas 
> > application properties to true/false we should change the global config.
> > 
> > It takes a default value here and always updates the global config even 
> > if user haven't set it.
> > 
> > global config is a site-wide setting for Atlas/Janusgraph and should 
> > not be changed unless explicitly specified.

E.g.: User sets the config then removes it. Unless we maintain a state across 
restarts, we won't know.

Another option is to read existing value and match it against incoming and 
update it only if there is a change.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73589/#review223508
---


On Sept. 21, 2021, 3:40 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73589/
> ---
> 
> (Updated Sept. 21, 2021, 3:40 p.m.)
> 
> 
> Review request for atlas, Nikhil Bonte, Nixon Rodrigues, Radhika Kundam, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4408
> https://issues.apache.org/jira/browse/ATLAS-4408
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This implementation builds up on Radhika's earlier implementation. 
> https://reviews.apache.org/r/73555/
> 
> Indexing
> 
> atlas.graph.index.status.check.frequency: Frequence of checking Solr for 
> health status.
> atlas.graph.index.recovery.start.time: User can set recovery time in UTC. 
> E.g. 
> ```
> atlas.graph.index.recovery.start.time=2014-12-11T02:35:58.440Z
> ```
> 
> **Implementation**
> New: _updateGlobalConfiguration_: Updating global configuration variables.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java aea0c1311 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndexClient.java
>  9960d8909 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  7e3b2f440 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
>  11267c438 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphIndexClient.java
>  ef42dbd3d 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
>  1cc7f8b55 
>   intg/src/main/java/org/apache/atlas/ApplicationProperties.java 682206dad 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  ddfb0089e 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/IndexRecoveryService.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/RecoveryInfoManagementTest.java
>  PRE-CREATION 
>   
> server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java
>  ba8f0886c 
> 
> 
> Diff: https://reviews.apache.org/r/73589/diff/2/
> 
> 
> Testing
> ---
> 
> Additional unit tests.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 73595: ATLAS-4428 : Enhance Atlas hook messages to capture hook version & message source

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73595/#review223510
---




intg/src/main/java/org/apache/atlas/model/notification/HookInfo.java
Lines 51 (patched)


Since this is a model, it is used for moving data around. Populate an 
instance of this via a helper method.



intg/src/main/java/org/apache/atlas/model/notification/HookInfo.java
Lines 54 (patched)


Refactor: Extract to field.


- Ashutosh Mestry


On Sept. 21, 2021, 10:23 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73595/
> ---
> 
> (Updated Sept. 21, 2021, 10:23 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4428
> https://issues.apache.org/jira/browse/ATLAS-4428
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas hook messages doesn't specify the source information of the messages 
> coming i.e. which hook source - HiveServer2, HiveMetastore, Impala, Spark, 
> HBase, Kafka.
> 
> Each hook message already captures, the msgSourceIP, msgCreatedBy and 
> msgCreationTime info. In addition to these, we should capture the msgSource 
> information as well.
> 
> When messages come from different hook versions, Atlas server should be aware 
> of the version of different hook messages to process the message based on the 
> hook version.
> 
> 
> Diffs
> -
> 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java
>  8c09d3394 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  0ab06f2c5 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 3cc7b3bc6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveMetastoreHookImpl.java
>  6a492c214 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  10ae08f6d 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> 4785960a7 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  779c5cb20 
>   intg/pom.xml 05914 
>   
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationBaseMessage.java
>  ff45d5713 
>   
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java
>  5869910cb 
>   intg/src/main/java/org/apache/atlas/model/notification/HookInfo.java 
> PRE-CREATION 
>   notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
>  c45a1da95 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  3d8d9cc0a 
>   
> notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
>  0c92c300e 
>   notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java 
> 1ae7c278c 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
>  8078a6ceb 
>   
> notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationDeserializerTest.java
>  13eafb6b1 
>   
> notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java
>  d04817024 
> 
> 
> Diff: https://reviews.apache.org/r/73595/diff/2/
> 
> 
> Testing
> ---
> 
> mvn clean install
> mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
> testcases running successfully
> 
> 
> Thanks,
> 
> chaitali
> 
>

Review Request 73599: Conditionally Disable 'Session Inactivity Timeout'

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73599/
---

Review request for atlas, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.


Bugs: ATLAS-4435
https://issues.apache.org/jira/browse/ATLAS-4435


Repository: atlas


Description
---

Please see JIRA for details.


Diffs
-

  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java fa519ef5a 
  
webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
 b8d21b9bb 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
baa040f4c 
  
webapp/src/main/java/org/apache/atlas/web/security/AtlasAuthenticationSuccessHandler.java
 1b1a80826 


Diff: https://reviews.apache.org/r/73599/diff/1/


Testing
---

Manual testing:
* Verified against Knox SSO enabled cluster.
* User-name password enabled cluster.


Thanks,

Ashutosh Mestry

Re: Review Request 73589: Indexing Monitor and Recovery

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73589/
---

(Updated Sept. 21, 2021, 3:40 p.m.)


Review request for atlas, Nikhil Bonte, Nixon Rodrigues, Radhika Kundam, and 
Sarath Subramanian.


Changes
---

Updates include: 
- Addressed review comments.
- Added description.


Summary (updated)
-

Indexing Monitor and Recovery


Bugs: ATLAS-4408
https://issues.apache.org/jira/browse/ATLAS-4408


Repository: atlas


Description (updated)
---

This implementation builds up on Radhika's earlier implementation. 
https://reviews.apache.org/r/73555/

Indexing

atlas.graph.index.status.check.frequency: Frequence of checking Solr for health 
status.
atlas.graph.index.recovery.start.time: User can set recovery time in UTC. E.g. 
```
atlas.graph.index.recovery.start.time=2014-12-11T02:35:58.440Z
```

**Implementation**
New: _updateGlobalConfiguration_: Updating global configuration variables.


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/repository/Constants.java aea0c1311 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndexClient.java
 9960d8909 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 7e3b2f440 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
 11267c438 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphIndexClient.java
 ef42dbd3d 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
 1cc7f8b55 
  intg/src/main/java/org/apache/atlas/ApplicationProperties.java 682206dad 
  
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
 ddfb0089e 
  
repository/src/main/java/org/apache/atlas/repository/graph/IndexRecoveryService.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/graph/RecoveryInfoManagementTest.java
 PRE-CREATION 
  
server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java
 ba8f0886c 


Diff: https://reviews.apache.org/r/73589/diff/2/

Changes: https://reviews.apache.org/r/73589/diff/1-2/


Testing (updated)
---

Additional unit tests.


Thanks,

Ashutosh Mestry

Re: Review Request 73593: ATLAS-4431: Random NPE when retrieving tasks

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73593/#review223503
---


Ship it!




Ship It!

- Ashutosh Mestry


On Sept. 20, 2021, 8:57 p.m., Disha Talreja wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73593/
> ---
> 
> (Updated Sept. 20, 2021, 8:57 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Sarath Subramanian, and Sidharth 
> Mishra.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> System tests fail randomly because retrieving tasks using the "admin/tasks" 
> endpoint sometimes returns HTTP 500. This patch includes a few null checks to 
> avoid NPE.
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java fae8a4fed 
> 
> 
> Diff: https://reviews.apache.org/r/73593/diff/1/
> 
> 
> Testing
> ---
> 
> Local Build Successful.
> 
> 
> Thanks,
> 
> Disha Talreja
> 
>

Re: Review Request 73584: ATLAS-4422 : Quick Search: Provide sorting

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73584/#review223491
---


Ship it!




Ship It!

- Ashutosh Mestry


On Sept. 15, 2021, 4:03 p.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73584/
> ---
> 
> (Updated Sept. 15, 2021, 4:03 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-4422
> https://issues.apache.org/jira/browse/ATLAS-4422
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Improvement:** 
> 
> This will add a sorting functionality in quick search API and get the result 
> list in Sorted manner.
> 
> **Note:** 
> 
> - Result will be sorted only when 'typeName' has a valid entity type name as 
> a value.
> - Only string attribute type (not TEXT) guarantees sorted results
> 
> **Example:** 
> 
> { "query":"sales", 
> "typeName":"Asset",
> "sortBy":"__typeName",
> "sortOrder":"ASCENDING" }
> 
> 
> Diffs
> -
> 
>   
> intg/src/main/java/org/apache/atlas/model/discovery/QuickSearchParameters.java
>  9e5e9778a 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  5b3b9486b 
>   
> repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
>  5e47295cb 
>   webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java e4c74a9d1 
> 
> 
> Diff: https://reviews.apache.org/r/73584/diff/2/
> 
> 
> Testing
> ---
> 
> added Unit tests
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 73580: ATLAS-4369: import-hive.sh does not write logs to /var/log/atlas/import-hive.log file, instead write into ${atlas.log.file} under the directory "${atlas.log.dir}"

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73580/#review223483
---


Ship it!




Ship It!

- Ashutosh Mestry


On Sept. 13, 2021, 5:13 p.m., Disha Talreja wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73580/
> ---
> 
> (Updated Sept. 13, 2021, 5:13 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Sarath Subramanian, and Sidharth 
> Mishra.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Due to the conflicting log4j jars from hive conf, import-hive.sh did not 
> write logs to /var/log/atlas/import-hive.log and wrote to 
> ${atlas.log.dir}/${atlas.log.file} instead. With this patch, the logs are 
> written to the correct/desired location (/var/log/atlas/import-hive.log).
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/src/bin/import-hive.sh 693a79bd0 
> 
> 
> Diff: https://reviews.apache.org/r/73580/diff/1/
> 
> 
> Testing
> ---
> 
> Patched a cluster
> 
> 
> Thanks,
> 
> Disha Talreja
> 
>

Re: Review Request 73528: ATLAS-4377 : Atlas - Upgrade Spring Security to 5.4.4+/5.3.8+/5.2.9+

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73528/#review223463
---


Ship it!




Ship It!


common/src/main/java/org/apache/atlas/utils/OnAtlasPropertyCondition.java
Line 40 (original), 40 (patched)


Good.


- Ashutosh Mestry


On Sept. 6, 2021, 8:45 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73528/
> ---
> 
> (Updated Sept. 6, 2021, 8:45 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4377
> https://issues.apache.org/jira/browse/ATLAS-4377
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas is pulling in Spring security 4.2.17.RELEASE upgrade to 5.4.4, 5.3.8, 
> or 5.2.9
> Upgrading spring-security version to 5.5.1 this is the latest
> Upgrading spring-core version to 5.3.8 current version 4.3.29.RELEASE
> 
> 
> Diffs
> -
> 
>   addons/falcon-bridge-shim/pom.xml 02da049db 
>   addons/falcon-bridge/pom.xml e0d2f3be7 
>   authorization/pom.xml 8e5a85db4 
>   common/src/main/java/org/apache/atlas/utils/OnAtlasPropertyCondition.java 
> ece6e370a 
>   pom.xml 8ae338cf2 
>   repository/pom.xml bf34efee2 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
> 343d00d72 
>   test-tools/pom.xml c861ea67f 
>   webapp/src/main/java/org/apache/atlas/web/dao/UserDao.java e16796ff6 
>   webapp/src/main/java/org/apache/atlas/web/util/Servlets.java 71eca2e50 
>   webapp/src/main/webapp/WEB-INF/web.xml 7fb56efde 
>   webapp/src/test/java/org/apache/atlas/web/security/UserDaoTest.java 
> 519be804e 
>   webapp/src/test/webapp/WEB-INF/web.xml 2388fbfbb 
> 
> 
> Diff: https://reviews.apache.org/r/73528/diff/3/
> 
> 
> Testing
> ---
> 
> mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
> Atlas server runnning
> mvn clean install  -DskipTests
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/811/console
> 
> 
> Thanks,
> 
> chaitali
> 
>

Re: Review Request 73536: ATLAS-4379 :- Atlas Filter changes for user inactivity on Atlas UI

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73536/#review223431
---


Ship it!




Ship It!

- Ashutosh Mestry


On Aug. 31, 2021, 12:49 p.m., Nixon Rodrigues wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73536/
> ---
> 
> (Updated Aug. 31, 2021, 12:49 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry and Jayendra Parab.
> 
> 
> Bugs: ATLAS-4379
> https://issues.apache.org/jira/browse/ATLAS-4379
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas server filter changes to support user inactivity on Atlas UI and logout 
> the user on UI invalidating the user session.
> This integration is required with Atlas with knox proxy.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2f2c8a540 
>   
> webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
>  d9b1c82b1 
>   webapp/src/main/java/org/apache/atlas/web/filters/RestUtil.java 
> PRE-CREATION 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
> 01fb8ec02 
>   
> webapp/src/main/java/org/apache/atlas/web/security/AtlasAuthenticationSuccessHandler.java
>  e7a5d668c 
> 
> 
> Diff: https://reviews.apache.org/r/73536/diff/4/
> 
> 
> Testing
> ---
> 
> Tested Atlas UI flow on kerberos with trusted proxy and simple authentication 
> flow.
> 
> PC
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/819/console
> 
> 
> Thanks,
> 
> Nixon Rodrigues
> 
>

Re: Review Request 73536: ATLAS-4379 :- Atlas Filter changes for user inactivity on Atlas UI

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73536/#review223428
---


Ship it!




Ship It!

- Ashutosh Mestry


On Aug. 30, 2021, 7:35 p.m., Nixon Rodrigues wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73536/
> ---
> 
> (Updated Aug. 30, 2021, 7:35 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry and Jayendra Parab.
> 
> 
> Bugs: ATLAS-4379
> https://issues.apache.org/jira/browse/ATLAS-4379
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas server filter changes to support user inactivity on Atlas UI and logout 
> the user on UI invalidating the user session.
> This integration is required with Atlas with knox proxy.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2f2c8a540 
>   
> webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
>  d9b1c82b1 
>   webapp/src/main/java/org/apache/atlas/web/filters/RestUtil.java 
> PRE-CREATION 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
> 01fb8ec02 
>   
> webapp/src/main/java/org/apache/atlas/web/security/AtlasAuthenticationSuccessHandler.java
>  e7a5d668c 
> 
> 
> Diff: https://reviews.apache.org/r/73536/diff/3/
> 
> 
> Testing
> ---
> 
> Tested Atlas UI flow on kerberos with trusted proxy and simple authentication 
> flow.
> 
> PC
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/819/console
> 
> 
> Thanks,
> 
> Nixon Rodrigues
> 
>

Re: Review Request 73528: ATLAS-4377 : Atlas - Upgrade Spring Security to 5.4.4+/5.3.8+/5.2.9+

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73528/#review223422
---




common/src/main/java/org/apache/atlas/utils/OnAtlasPropertyCondition.java
Line 40 (original), 41 (patched)


Can you please check type before cast.



webapp/src/main/java/org/apache/atlas/web/dao/UserDao.java
Lines 238 (patched)


Does this change the value returned. It would be worthwhile to add a UT 
verifying this before and after the version change.


- Ashutosh Mestry


On Aug. 24, 2021, 7:05 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73528/
> ---
> 
> (Updated Aug. 24, 2021, 7:05 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4377
> https://issues.apache.org/jira/browse/ATLAS-4377
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas is pulling in Spring security 4.2.17.RELEASE upgrade to 5.4.4, 5.3.8, 
> or 5.2.9
> Upgrading spring-security version to 5.5.1 this is the latest
> Upgrading spring-core version to 5.3.8 current version 4.3.29.RELEASE
> 
> 
> Diffs
> -
> 
>   addons/falcon-bridge-shim/pom.xml 02da049db 
>   addons/falcon-bridge/pom.xml e0d2f3be7 
>   authorization/pom.xml 8e5a85db4 
>   common/src/main/java/org/apache/atlas/utils/OnAtlasPropertyCondition.java 
> ece6e370a 
>   pom.xml cf0eb34a7 
>   repository/pom.xml bf34efee2 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
> 343d00d72 
>   test-tools/pom.xml c861ea67f 
>   webapp/src/main/java/org/apache/atlas/web/dao/UserDao.java e16796ff6 
>   webapp/src/main/java/org/apache/atlas/web/util/Servlets.java 71eca2e50 
>   webapp/src/main/webapp/WEB-INF/web.xml 7fb56efde 
>   webapp/src/test/webapp/WEB-INF/web.xml 2388fbfbb 
> 
> 
> Diff: https://reviews.apache.org/r/73528/diff/2/
> 
> 
> Testing
> ---
> 
> mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
> Atlas server runnning
> mvn clean install  -DskipTests
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/811/console
> 
> 
> Thanks,
> 
> chaitali
> 
>

Re: Review Request 73552: ATLAS-4406: Ignore configs:- A hive table DDL entity is created for ignored hive tables

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73552/#review223421
---


Ship it!




Ship It!

- Ashutosh Mestry


On Aug. 27, 2021, 3:54 p.m., Disha Talreja wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73552/
> ---
> 
> (Updated Aug. 27, 2021, 3:54 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Radhika Kundam, Sarath 
> Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-4406
> https://issues.apache.org/jira/browse/ATLAS-4406
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> When hive tables are ignored, a hive table DDL entity is created in Atlas for 
> the same. A shell entity is also created for the hive table associated with 
> the DDL operation. 
> 
> A DDL entity should not be created for ignored tables and this patch avoids 
> the creation of a DDL entity for ignored tables.
> 
> Ignore configs used for Atlas server: 
> atlas.notification.consumer.preprocess.hive_table.ignore.pattern=finance..*_tmp.*
> Ignore configs used for Hive Metastore server: 
> atlas.hook.hive.hive_table.ignore.pattern=finance..*_tmp.*.*
> 
> 
> Diffs
> -
> 
>   
> webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java
>  bf6a623fe 
> 
> 
> Diff: https://reviews.apache.org/r/73552/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean -DskipTests install 
> mvn clean -DskipTests package -Pdist,embedded-hbase-solr 
> Atlas server running.
> 
> 
> Thanks,
> 
> Disha Talreja
> 
>

Re: Review Request 73549: ATLAS-4403: Improve find entity by unique attributes logic - when multiple unique attributes exist for entity type

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73549/#review223419
---



Please add unit test with appropriate test data.

- Ashutosh Mestry


On Aug. 26, 2021, 8:32 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73549/
> ---
> 
> (Updated Aug. 26, 2021, 8:32 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Jayendra Parab, 
> Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, Radhika Kundam, and Sidharth 
> Mishra.
> 
> 
> Bugs: ATLAS-4403
> https://issues.apache.org/jira/browse/ATLAS-4403
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> When a entity type has multiple unique attributes defined:
> 
> unique_attr1
> unique_attr2
> During creation of entity, we check if entity with unique attributes is 
> already present and return/update them. The current logic check if entity is 
> present - one unique attribute at a time, if present will return immediately.
> 
> We should improve the lookup logic to look for entity with all unique 
> attributes specified and not just return on the first unique attribute value.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
>  e73f084b8 
> 
> 
> Diff: https://reviews.apache.org/r/73549/diff/5/
> 
> 
> Testing
> ---
> 
> Pre-commit: 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/822/consoleFull
>  (SUCCESS)
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

Re: Review Request 73552: ATLAS-4406: Ignore configs:- A hive table DDL entity is created for ignored hive tables

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73552/#review223411
---



Can you please add to description:
* ignore patter you used for hive_table.
* ignore patter you used for hive_db.

Did you verify the change with hive_db_ddl?

- Ashutosh Mestry


On Aug. 27, 2021, 2:15 a.m., Disha Talreja wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73552/
> ---
> 
> (Updated Aug. 27, 2021, 2:15 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Radhika Kundam, Sarath 
> Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-4406
> https://issues.apache.org/jira/browse/ATLAS-4406
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> When hive tables are ignored, a hive table DDL entity is created in Atlas for 
> the same. A shell entity is also created for the hive table associated with 
> the DDL operation. 
> 
> A DDL entity should not be created for ignored tables and this patch avoids 
> the creation of a DDL entity for ignored tables.
> 
> 
> Diffs
> -
> 
>   
> webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java
>  bf6a623fe 
> 
> 
> Diff: https://reviews.apache.org/r/73552/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean -DskipTests install 
> mvn clean -DskipTests package -Pdist,embedded-hbase-solr 
> Atlas server running.
> 
> 
> Thanks,
> 
> Disha Talreja
> 
>

Re: Review Request 73548: ATLAS-4402: Upgrade junit to 4.13.1

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73548/#review223397
---


Ship it!




Ship It!

- Ashutosh Mestry


On Aug. 25, 2021, 7:02 p.m., Disha Talreja wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73548/
> ---
> 
> (Updated Aug. 25, 2021, 7:02 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Radhika Kundam, Sarath 
> Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-4402
> https://issues.apache.org/jira/browse/ATLAS-4402
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch upgrades junit version 4.13 to 4.13.1
> 
> 
> Diffs
> -
> 
>   pom.xml 8ae338cf2 
> 
> 
> Diff: https://reviews.apache.org/r/73548/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean -DskipTests install 
> mvn clean -DskipTests package -Pdist,embedded-hbase-solr 
> Atlas server running.
> 
> 
> Thanks,
> 
> Disha Talreja
> 
>

Re: Review Request 73494: ATLAS-4376 : AtlasClientV2 getRelationshipDefByName does not work

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73494/#review223378
---


Ship it!




Ship It!

- Ashutosh Mestry


On Aug. 4, 2021, 11:38 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73494/
> ---
> 
> (Updated Aug. 4, 2021, 11:38 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Nikhil Bonte, Nixon Rodrigues, 
> Pinal Shah, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4376
> https://issues.apache.org/jira/browse/ATLAS-4376
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> When tried to get the typedef of a relationship by:
> 
> atlasClient.getRelationshipDefByName(name)
> This will get translated into the following REST request: 
> /api/atlas/v2/types/structdef/name/
> 
> {name}
> 
> The correct REST request should be: 
> /api/atlas/v2/types/relationshipdef/name/{name}
> The reason why it's querying a structdef and not a relationshipdef is due to 
> a bug in AtlasClientV2 line 1135:
> 
> } else if (AtlasStructDef.class.isAssignableFrom(typeDefClass))
> 
> { return "structdef"; }
> else if (AtlasRelationshipDef.class.isAssignableFrom(typeDefClass)) {
> return "relationshipdef";
> Since AtlasRelationshipDef extends AtlasStructDef, the first "if" branch will 
> always be true and the second "if" branch will never happen. The fix for this 
> issue is to reverse the two "if" branches.
> 
> provide test cases for all "if" branches to ensure the querying works 
> correctly
> 
> 
> Diffs
> -
> 
>   client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 
> eb0e630f9 
>   client/client-v2/src/test/java/org/apache/atlas/AtlasClientV2Test.java 
> 83a3391eb 
> 
> 
> Diff: https://reviews.apache.org/r/73494/diff/2/
> 
> 
> Testing
> ---
> 
> mvn clean install -Drat.skip
> mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
> testcases added
> 
> 
> Thanks,
> 
> chaitali
> 
>

Re: Review Request 73534: ATLAS-4385: Update version in branch-2.0 to 2.3.0-SNAPSHOT

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73534/#review223377
---


Ship it!




Ship It!

- Ashutosh Mestry


On Aug. 21, 2021, 12:12 a.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73534/
> ---
> 
> (Updated Aug. 21, 2021, 12:12 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Jayendra Parab, 
> Nikhil Bonte, Nixon Rodrigues, Pinal Shah, Radhika Kundam, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-4385
> https://issues.apache.org/jira/browse/ATLAS-4385
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-4385: Update version in branch-2.0 to 2.3.0-SNAPSHOT
> 
> 
> Diffs
> -
> 
>   addons/falcon-bridge-shim/pom.xml 674084472 
>   addons/falcon-bridge/pom.xml bcf01e734 
>   addons/hbase-bridge-shim/pom.xml a7ba0f6a4 
>   addons/hbase-bridge/pom.xml 866941c93 
>   addons/hbase-testing-util/pom.xml 3603e111a 
>   addons/hdfs-model/pom.xml cff4b58a8 
>   addons/hive-bridge-shim/pom.xml 5bba79845 
>   addons/hive-bridge/pom.xml ef6c578b0 
>   addons/impala-bridge-shim/pom.xml 16c696536 
>   addons/impala-bridge/pom.xml 0c873049b 
>   addons/impala-hook-api/pom.xml e5ed85fed 
>   addons/kafka-bridge/pom.xml 25b0a0539 
>   addons/sqoop-bridge-shim/pom.xml 715415463 
>   addons/sqoop-bridge/pom.xml 57badcc48 
>   addons/storm-bridge-shim/pom.xml 3e3cdce5c 
>   addons/storm-bridge/pom.xml f3977da53 
>   atlas-examples/pom.xml 8db3529ba 
>   atlas-examples/sample-app/pom.xml 716fc2df0 
>   authorization/pom.xml 5ff793f82 
>   build-tools/pom.xml 29117265d 
>   client/client-v1/pom.xml 8d4046a57 
>   client/client-v2/pom.xml 002fd3d5c 
>   client/common/pom.xml 12664b10a 
>   client/pom.xml 4b39e72e7 
>   common/pom.xml cda250ec3 
>   dashboardv2/pom.xml 867dac327 
>   dashboardv3/pom.xml 659908894 
>   distro/pom.xml 8570ad11c 
>   docs/pom.xml 02f3b43ba 
>   graphdb/api/pom.xml ac8f4de21 
>   graphdb/common/pom.xml 6afddbe48 
>   graphdb/graphdb-impls/pom.xml 13a8a2804 
>   graphdb/janus-hbase2/pom.xml 404c198b9 
>   graphdb/janus/pom.xml 95674066e 
>   graphdb/pom.xml 58b8b6015 
>   intg/pom.xml 1a6447a4c 
>   notification/pom.xml 2ed23cd3d 
>   plugin-classloader/pom.xml e2089e0ff 
>   pom.xml e47b74a3a 
>   repository/pom.xml 8bd4e3eb4 
>   server-api/pom.xml b47d90689 
>   test-tools/pom.xml bf060f1ce 
>   tools/atlas-index-repair/pom.xml fac343f6a 
>   tools/classification-updater/pom.xml f6923d162 
>   webapp/pom.xml 91002b0f3 
> 
> 
> Diff: https://reviews.apache.org/r/73534/diff/1/
> 
> 
> Testing
> ---
> 
> Local build and Precommit - 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/810/
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>

Re: Review Request 73532: ATLAS-4383 Update Atlas website for 2.2.0 release (Part 2)

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73532/#review223365
---


Ship it!




It would be good if you could format this consistently for all entries.

- Ashutosh Mestry


On Aug. 20, 2021, 1:47 a.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73532/
> ---
> 
> (Updated Aug. 20, 2021, 1:47 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4383
> https://issues.apache.org/jira/browse/ATLAS-4383
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-4383: Add 2.1.0 and 2.2.0 to list versions in the dropdown
> 
> 
> Diffs
> -
> 
>   docs/docz-lib/config/versions.js feefbf175 
> 
> 
> Diff: https://reviews.apache.org/r/73532/diff/1/
> 
> 
> Testing
> ---
> 
> Precomit - 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/806/
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>

Re: Review Request 73530: ATLAS-4394: Build error due to wrong version of atllass-buildtools at pom.xml (Part 2)

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73530/#review223364
---


Ship it!




Ship It!

- Ashutosh Mestry


On Aug. 19, 2021, 8:27 p.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73530/
> ---
> 
> (Updated Aug. 19, 2021, 8:27 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4394
> https://issues.apache.org/jira/browse/ATLAS-4394
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-4394: Build error due to wrong version of atllass-buildtools at 
> build-tools/pom.xml (2)
> 
> 
> Diffs
> -
> 
>   build-tools/pom.xml 976cd81c0 
> 
> 
> Diff: https://reviews.apache.org/r/73530/diff/1/
> 
> 
> Testing
> ---
> 
> Manul build and Precommit - 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/805/
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>

Re: Review Request 73529: ATLAS-4383: Update Atlas website for 2.2.0 release

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73529/#review223357
---


Ship it!




Ship It!

- Ashutosh Mestry


On Aug. 18, 2021, 10:36 p.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73529/
> ---
> 
> (Updated Aug. 18, 2021, 10:36 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4383
> https://issues.apache.org/jira/browse/ATLAS-4383
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-4383: Update Atlas website for 2.2.0 release
> 
> 
> Diffs
> -
> 
>   README.txt d8096f31e 
>   docs/docz-lib/config/menu.js d83518e3e 
>   docs/src/documents/Downloads/Downloads.md d1ab04e72 
>   docs/src/documents/Overview.md 824ec3c67 
>   docs/src/documents/Setup/InstallationInstruction.md 89ecdc533 
>   docs/src/documents/Whats-New/WhatsNew-2.2.md PRE-CREATION 
>   docs/src/site/site.xml b841dd439 
> 
> 
> Diff: https://reviews.apache.org/r/73529/diff/3/
> 
> 
> Testing
> ---
> 
> Manually checked the md files and formatting.
> Precommit - 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/800/
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>

Re: Review Request 73529: ATLAS-4383: Update Atlas website for 2.2.0 release

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73529/#review223355
---




docs/src/documents/Downloads/Downloads.md
Lines 76 (patched)


Fix case.
Have the 3 things as sub-bullets or 3 bullets.



docs/src/documents/Downloads/Downloads.md
Lines 92 (patched)


Fix case 'Fixed'



docs/src/documents/Downloads/Downloads.md
Lines 93 (patched)


fix case.


- Ashutosh Mestry


On Aug. 18, 2021, 9:44 p.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73529/
> ---
> 
> (Updated Aug. 18, 2021, 9:44 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4383
> https://issues.apache.org/jira/browse/ATLAS-4383
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-4383: Update Atlas website for 2.2.0 release
> 
> 
> Diffs
> -
> 
>   README.txt d8096f31e 
>   docs/docz-lib/config/menu.js d83518e3e 
>   docs/src/documents/Downloads/Downloads.md d1ab04e72 
>   docs/src/documents/Overview.md 824ec3c67 
>   docs/src/documents/Setup/InstallationInstruction.md 89ecdc533 
>   docs/src/documents/Whats-New/WhatsNew-2.2.md PRE-CREATION 
>   docs/src/site/site.xml b841dd439 
> 
> 
> Diff: https://reviews.apache.org/r/73529/diff/2/
> 
> 
> Testing
> ---
> 
> Manually checked the md files and formatting.
> Precommit - 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/800/
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>

Re: Review Request 73524: ATLAS-4394: Build error due to wrong version of atllass-buildtools at pom.xml

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73524/#review223349
---


Ship it!




Ship It!

- Ashutosh Mestry


On Aug. 17, 2021, 11:54 p.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73524/
> ---
> 
> (Updated Aug. 17, 2021, 11:54 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Jayendra Parab, 
> Nixon Rodrigues, Radhika Kundam, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4394
> https://issues.apache.org/jira/browse/ATLAS-4394
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-4394: Fixed Apache Parent POM artifact and atlas-buildtools version
> 
> 
> Diffs
> -
> 
>   pom.xml cf0eb34a7 
> 
> 
> Diff: https://reviews.apache.org/r/73524/diff/1/
> 
> 
> Testing
> ---
> 
> Manual successful Build and Precomit - 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/799/
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>

Re: Review Request 73503: ATLAS-4386 : Relationship search: Sorting does not work when passed relationship label

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73503/#review223325
---




repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java
Lines 1621 (patched)


What is the rationale for this?

The way I look at it is:
- Fetch edges with a label. (from line 1611)
- Checked if edges exists (line 1613)
- Found edges, get vertex that is not the passed vertex.
- Return the vertex.

I don't understand why StringUtils.equals is used. I think the outVertexId 
== entityVertexId should never be used.

Reason: entityVertexId can potentially have edges of different labels.


- Ashutosh Mestry


On Aug. 4, 2021, 2:55 p.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73503/
> ---
> 
> (Updated Aug. 4, 2021, 2:55 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4386
> https://issues.apache.org/jira/browse/ATLAS-4386
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Problem:**
> 
> Relationship Search is not able to sort when relationship label is passed as 
> a queryParam instead of attribute.
> 
> **Example:**
> 
> api/atlas/v2/search/relationship?limit=25&offset=0&guid=cca5c29d-307f-4a9b-ac6c-79f41a08a90b&relation=__hive_table.columns&sortBy=name&sortOrder=DESCENDING
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  f2290c672 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> ff7fd150e 
>   
> repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
>  a9fbd435a 
> 
> 
> Diff: https://reviews.apache.org/r/73503/diff/1/
> 
> 
> Testing
> ---
> 
> added test cases
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 73502: ATLAS-3917 While deleting parent tag, shows incorrect message.

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73502/#review223307
---




repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasTypeDefGraphStoreV2.java
Line 47 (original), 47 (patched)


Retain imports. (If using IntelliJ, change default setting.)



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasTypeDefGraphStoreV2.java
Line 56 (original), 51 (patched)


nit: Retain individual imports.



repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasClassificationDefStoreV2Test.java
Lines 82 (patched)


Good!



repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasClassificationDefStoreV2Test.java
Lines 91 (patched)


Consider replacing string with the error message mentioned in the 
AtlasErrorCode.


- Ashutosh Mestry


On Aug. 4, 2021, 2:11 p.m., Shraddha Pardeshi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73502/
> ---
> 
> (Updated Aug. 4, 2021, 2:11 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Nikhil Bonte, Nixon Rodrigues, and 
> Pinal Shah.
> 
> 
> Bugs: ATLAS-3917
> https://issues.apache.org/jira/browse/ATLAS-3917
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> -Actual Behaviour : While deleting parent tag if the tag is not associated 
> with any entity shows this message- "Given type {0} has reference."
> -Expected Result: There should be tag name instead of {0}
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasTypeDefGraphStoreV2.java
>  b9d41bbba 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasClassificationDefStoreV2Test.java
>  3242a33ad 
> 
> 
> Diff: https://reviews.apache.org/r/73502/diff/1/
> 
> 
> Testing
> ---
> 
> -Manual testing done on local server
> 
> -Test case added.
> 
> PC - 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/779/console
> 
> 
> Thanks,
> 
> Shraddha Pardeshi
> 
>

Re: Review Request 73478: ATLAS-4368: Upgrade elasticsearch version to 6.8.15

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73478/#review223284
---


Ship it!




Ship It!

- Ashutosh Mestry


On July 27, 2021, 8:45 p.m., Disha Talreja wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73478/
> ---
> 
> (Updated July 27, 2021, 8:45 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Radhika Kundam, Sarath 
> Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-4368
> https://issues.apache.org/jira/browse/ATLAS-4368
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently Atlas is pulling in Elasticsearch 6.8.14
> this patch upgrades Elasticsearch version to 6.8.15
> 
> 
> Diffs
> -
> 
>   pom.xml 19db43146 
> 
> 
> Diff: https://reviews.apache.org/r/73478/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean -DskipTests install 
> mvn clean -DskipTests package -Pdist,embedded-hbase-solr 
> Atlas server running.
> PC Build: 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/767/
> 
> 
> Thanks,
> 
> Disha Talreja
> 
>

Re: Review Request 73492: ATLAS-4372: Upgrade nimbus-jose-jwt to 9.8.1

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73492/#review223278
---


Ship it!




Ship It!

- Ashutosh Mestry


On July 27, 2021, 9:01 p.m., Disha Talreja wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73492/
> ---
> 
> (Updated July 27, 2021, 9:01 p.m.)
> 
> 
> Review request for atlas, Radhika Kundam and Sarath Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas is currently pulling in nimbus-jose-jwt 5.4. This patch upgrades it to 
> 9.8.1.
> 
> 
> Diffs
> -
> 
>   webapp/pom.xml ccbcc3141 
> 
> 
> Diff: https://reviews.apache.org/r/73492/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean -DskipTests install 
> mvn clean -DskipTests package -Pdist,embedded-hbase-solr 
> Atlas server running.
> PC Build: 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/774/
> 
> 
> Thanks,
> 
> Disha Talreja
> 
>

Re: Review Request 73430: ATLAS-4340: Set Solr wait-searcher property to false by default to make Solr commits async

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73430/#review223275
---


Ship it!




Ship It!

- Ashutosh Mestry


On July 27, 2021, 6:32 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73430/
> ---
> 
> (Updated July 27, 2021, 6:32 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
> Nikhil Bonte, and Pinal Shah.
> 
> 
> Bugs: ATLAS-4340
> https://issues.apache.org/jira/browse/ATLAS-4340
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> In Atlas when a transaction is committed, the entries are committed to HBase 
> (primary storage) and Solr (indexing storage). A transaction is rolled-back 
> if the primary storage commit fails, on the other hand when the secondary 
> commit fails (solr), the transaction is not-rolled back and logged as warning 
> and it is recommended to use reindex to repair the missing index documents. 
> This behavior is due to the fact that the primary storage is the source of 
> truth and indexes can be rebuild.
> 
> In Janusgraph, there is a property for Solr to make solr commits async. This 
> is set to true in Atlas making every commit to wait until the solr commit is 
> successful. This will have a negative impact on performance and is 
> recommended to be false by default.
> 
> Property: index.[X].solr.wait-searcher
> 
> When mutating - wait for the index to reflect new mutations before returning. 
> This can have a negative impact on performance.
>  
> 
> This Jira is about setting the default value for above property to FALSE and 
> can be overridden if need arises.
> 
> 
> Diffs
> -
> 
>   addons/falcon-bridge/src/test/resources/atlas-application.properties 
> 898b69c99 
>   addons/hbase-bridge/src/test/resources/atlas-application.properties 
> 898b69c99 
>   addons/kafka-bridge/src/test/resources/atlas-application.properties 
> 91fd8b092 
>   authorization/src/test/resources/atlas-application.properties 2e02678a6 
>   distro/pom.xml d84f5e7b1 
>   distro/src/bin/atlas_config.py 493a34ad8 
>   distro/src/bin/atlas_start.py 7cf35a92a 
>   distro/src/test/python/scripts/TestMetadata.py 662fbddba 
>   graphdb/janus/src/test/resources/atlas-application.properties a355234e9 
>   intg/src/main/java/org/apache/atlas/ApplicationProperties.java bf97ab146 
>   intg/src/test/resources/atlas-application.properties 50ce01e70 
>   webapp/src/test/resources/atlas-application.properties 1d45e78f3 
> 
> 
> Diff: https://reviews.apache.org/r/73430/diff/2/
> 
> 
> Testing
> ---
> 
> 1. Precommit Test: 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/688/
> 2. Build Atlas with embedded Hbase/Solr profile and validated basic sanity 
> tests - running quick start, basic search, tag propagation
> 3. Performance Test details:
> 
> 
> Run with default settings - Solr wait-searcher property enabled - true 
> (without patch)
> --
> Start Time : Tue Jun 15 22:26:58 PDT 2021
> End Time   : Fri Jun 18 02:32:34 PDT 2021
> Messages Processed : 91,225
> Time Taken : 52 hours 5 mins
> Rate   : ~ 29.2 messages/minute
> 
> 
> Run with disabled Solr wait-searcher property (will improve solr commit time 
> making it async) - with patch
> ---
> Start Time : Mon Jun 14 13:30:04 PDT 2021
> End Time   : Tue Jun 15 17:23:56 PDT 2021
> Messages Processed : 91,225
> Time Taken : 27 hours 54 mins
> Rate   : ~ 54.5 messages/minute
> 
> 
> We see almost 50% perf imrpovement with this change.
> 
> Precommit (updated): 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/772/console
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

Re: Review Request 73430: ATLAS-4340: Set Solr wait-searcher property to false by default to make Solr commits async

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73430/#review223274
---




distro/src/bin/atlas_start.py
Line 113 (original), 113 (patched)


nit: What is the reason for CRLF?



intg/src/main/java/org/apache/atlas/ApplicationProperties.java
Line 346 (original), 346 (patched)


This comment is no longer relevant.


- Ashutosh Mestry


On July 27, 2021, 6:32 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73430/
> ---
> 
> (Updated July 27, 2021, 6:32 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
> Nikhil Bonte, and Pinal Shah.
> 
> 
> Bugs: ATLAS-4340
> https://issues.apache.org/jira/browse/ATLAS-4340
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> In Atlas when a transaction is committed, the entries are committed to HBase 
> (primary storage) and Solr (indexing storage). A transaction is rolled-back 
> if the primary storage commit fails, on the other hand when the secondary 
> commit fails (solr), the transaction is not-rolled back and logged as warning 
> and it is recommended to use reindex to repair the missing index documents. 
> This behavior is due to the fact that the primary storage is the source of 
> truth and indexes can be rebuild.
> 
> In Janusgraph, there is a property for Solr to make solr commits async. This 
> is set to true in Atlas making every commit to wait until the solr commit is 
> successful. This will have a negative impact on performance and is 
> recommended to be false by default.
> 
> Property: index.[X].solr.wait-searcher
> 
> When mutating - wait for the index to reflect new mutations before returning. 
> This can have a negative impact on performance.
>  
> 
> This Jira is about setting the default value for above property to FALSE and 
> can be overridden if need arises.
> 
> 
> Diffs
> -
> 
>   addons/falcon-bridge/src/test/resources/atlas-application.properties 
> 898b69c99 
>   addons/hbase-bridge/src/test/resources/atlas-application.properties 
> 898b69c99 
>   addons/kafka-bridge/src/test/resources/atlas-application.properties 
> 91fd8b092 
>   authorization/src/test/resources/atlas-application.properties 2e02678a6 
>   distro/pom.xml d84f5e7b1 
>   distro/src/bin/atlas_config.py 493a34ad8 
>   distro/src/bin/atlas_start.py 7cf35a92a 
>   distro/src/test/python/scripts/TestMetadata.py 662fbddba 
>   graphdb/janus/src/test/resources/atlas-application.properties a355234e9 
>   intg/src/main/java/org/apache/atlas/ApplicationProperties.java bf97ab146 
>   intg/src/test/resources/atlas-application.properties 50ce01e70 
>   webapp/src/test/resources/atlas-application.properties 1d45e78f3 
> 
> 
> Diff: https://reviews.apache.org/r/73430/diff/2/
> 
> 
> Testing
> ---
> 
> 1. Precommit Test: 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/688/
> 2. Build Atlas with embedded Hbase/Solr profile and validated basic sanity 
> tests - running quick start, basic search, tag propagation
> 3. Performance Test details:
> 
> 
> Run with default settings - Solr wait-searcher property enabled - true 
> (without patch)
> --
> Start Time : Tue Jun 15 22:26:58 PDT 2021
> End Time   : Fri Jun 18 02:32:34 PDT 2021
> Messages Processed : 91,225
> Time Taken : 52 hours 5 mins
> Rate   : ~ 29.2 messages/minute
> 
> 
> Run with disabled Solr wait-searcher property (will improve solr commit time 
> making it async) - with patch
> ---
> Start Time : Mon Jun 14 13:30:04 PDT 2021
> End Time   : Tue Jun 15 17:23:56 PDT 2021
> Messages Processed : 91,225
> Time Taken : 27 hours 54 mins
> Rate   : ~ 54.5 messages/minute
> 
> 
> We see almost 50% perf imrpovement with this change.
> 
> Precommit (updated): 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/772/console
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

Re: Review Request 73475: ATLAS-4367: UI: Single quote in can't is being removed. Hence renaming the message to cannot

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73475/#review223262
---


Ship it!




Ship It!

- Ashutosh Mestry


On July 23, 2021, 3:51 p.m., Umesh Padashetty wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73475/
> ---
> 
> (Updated July 23, 2021, 3:51 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-4367
> https://issues.apache.org/jira/browse/ATLAS-4367
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> can't is misspelled as 'cant'. This is because single quote is being removed
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 2febff469 
> 
> 
> Diff: https://reviews.apache.org/r/73475/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Umesh Padashetty
> 
>

Re: Review Request 73423: ATLAS-4338: Import Service: Support change notifiers for migration import

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73423/#review223211
---


Ship it!




Ship It!

- Ashutosh Mestry


On June 30, 2021, 11:07 a.m., Nikhil Bonte wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73423/
> ---
> 
> (Updated June 30, 2021, 11:07 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
> Rodrigues, Sarath Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-4338
> https://issues.apache.org/jira/browse/ATLAS-4338
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently Migration import feature (MigrationImport.java) does not support 
> enabling notifiers (audits writer, notification to Atlas entities topic).
> 
> With this improvement, Atlas now will enable notifiers for Migration import, 
> based on a flag.
> 
>  
> 
> Approach:
> 
> * New option in AtlasImportRequest - sendNotifications (default : false)
> * MigrationImport will use the flag to select appropriate implementation 
> of IAtlasEntityChangeNotifier to create EntityCreationManager.
> * EntityCreationManager will then notify listeners.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
> 2c1870444 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
>  1d29bf833 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
>  99be169f2 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
>  8e17fd410 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/GuidMutationResponsePair.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java
>  d6f23d6e2 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java
>  b73988fd7 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityCreationManager.java
>  734add6d7 
> 
> 
> Diff: https://reviews.apache.org/r/73423/diff/3/
> 
> 
> Testing
> ---
> 
> * Tested Manually
> * Sample request
> {
> "options": {
> "migration": "true",
> "size": "100",
> "sendNotifications": "true",
> "batchSize": 500,
> "numWorkers": 40
> }
> }
> * took ~270 secs to import 12987 entities with notifiers enabled.
> * PC build : 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/679/console
> 
> 
> Thanks,
> 
> Nikhil Bonte
> 
>

Re: Review Request 73445: ATLAS-4350: Atlas - Jetty Version disclosure in http response

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73445/#review223202
---


Ship it!




For completeness, can you please add HTTP header before/after the change.

- Ashutosh Mestry


On July 8, 2021, 10:36 p.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73445/
> ---
> 
> (Updated July 8, 2021, 10:36 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Radhika Kundam, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-4350
> https://issues.apache.org/jira/browse/ATLAS-4350
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-4350: Atlas - Jetty Version disclosure in http response
> 
> 
> Diffs
> -
> 
>   webapp/src/main/java/org/apache/atlas/web/service/EmbeddedServer.java 
> 61aa3134c 
>   webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 
> 7a045749f 
> 
> 
> Diff: https://reviews.apache.org/r/73445/diff/1/
> 
> 
> Testing
> ---
> 
> Manually tested and confirmed that the version is not present as part for 
> response header
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>

Re: Review Request 73439: ATLAS-4347 : DSL Search : Caching Mechanism for translated dsl queries

[Ashutosh Mestry via Review Board]

> On July 7, 2021, 4:49 a.m., Sarath Subramanian wrote:
> > repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
> > Lines 169 (patched)
> > 
> >
> > only the query syntax should be cached so that it can be reused not the 
> > values
> > 
> > Your strategy to cache will generate 5 cache entry for same query 
> > pattern, which si not useful:
> > 
> > ive_column where table.name = \"sales_fact_daily_mv1\"
> > hive_column where table.name = \"sales_fact_daily_mv2\"
> > hive_column where table.name = \"sales_fact_daily_mv3\"
> > hive_column where table.name = \"sales_fact_daily_mv4\"
> > hive_column where table.name = \"sales_fact_daily_mv5\"

Actually this approach will benefit the integrators than the end users. End 
users who use DSL via UI may not fire same query. The LRU cache will handle 
that part. Integrating applications programatically generate same queries and 
keep executing it at a set frequency. Caching greatly benefits those users.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73439/#review223198
---


On July 7, 2021, 4:28 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73439/
> ---
> 
> (Updated July 7, 2021, 4:28 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4347
> https://issues.apache.org/jira/browse/ATLAS-4347
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> 
> - DSL Query is converted to Tinkerpop Traversals.
> - Caching traversal with respective to query saves time of translation.
> - Whenever same query with same limit and offset is fired again, it will pick 
> the translated query from cache.
> 
> **Config**
> 
> - atlas.dsl.cached.translator when 'true' uses caching mechanism (Default: 
> true)
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 9ef848752 
>   
> repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
>  724cb8244 
> 
> 
> Diff: https://reviews.apache.org/r/73439/diff/1/
> 
> 
> Testing
> ---
> 
> Precommit : 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/718/
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 73439: ATLAS-4347 : DSL Search : Caching Mechanism for translated dsl queries

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73439/#review223197
---


Ship it!




Ship It!

- Ashutosh Mestry


On July 6, 2021, noon, Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73439/
> ---
> 
> (Updated July 6, 2021, noon)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4347
> https://issues.apache.org/jira/browse/ATLAS-4347
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> 
> - DSL Query is converted to Tinkerpop Traversals.
> - Caching traversal with respective to query saves time of translation.
> - Whenever same query with same limit and offset is fired again, it will pick 
> the translated query from cache.
> 
> **Config**
> 
> - atlas.dsl.cached.translator when 'true' uses caching mechanism (Default: 
> true)
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 9ef848752 
>   
> repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
>  724cb8244 
> 
> 
> Diff: https://reviews.apache.org/r/73439/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 73423: ATLAS-4338: Import Service: Support change notifiers for migration import

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73423/#review223196
---




intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java
Lines 58 (patched)


Rename: OPTION_MIGRATION_WITH_AUDITS



intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java
Lines 170 (patched)


Rename: getOptionMigrationWithAudis



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/BulkImportResultItem.java
Lines 23 (patched)


Rename: GuidResponsePair


- Ashutosh Mestry


On June 23, 2021, 2:21 p.m., Nikhil Bonte wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73423/
> ---
> 
> (Updated June 23, 2021, 2:21 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
> Rodrigues, Sarath Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-4338
> https://issues.apache.org/jira/browse/ATLAS-4338
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently Migration import feature (MigrationImport.java) does not support 
> enabling notifiers (audits writer, notification to Atlas entities topic).
> 
> With this improvement, Atlas now will enable notifiers for Migration import, 
> based on a flag.
> 
>  
> 
> Approach:
> 
> * New option in AtlasImportRequest - sendNotifications (default : false)
> * MigrationImport will use the flag to select appropriate implementation 
> of IAtlasEntityChangeNotifier to create EntityCreationManager.
> * EntityCreationManager will then notify listeners.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
> 2c1870444 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
>  1d29bf833 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
>  99be169f2 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
>  8e17fd410 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/BulkImportResultItem.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java
>  d6f23d6e2 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java
>  b73988fd7 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityCreationManager.java
>  734add6d7 
> 
> 
> Diff: https://reviews.apache.org/r/73423/diff/2/
> 
> 
> Testing
> ---
> 
> * Tested Manually
> * Sample request
> {
> "options": {
> "migration": "true",
> "size": "100",
> "sendNotifications": "true",
> "batchSize": 500,
> "numWorkers": 40
> }
> }
> * took ~270 secs to import 12987 entities with notifiers enabled.
> * PC build : 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/679/console
> 
> 
> Thanks,
> 
> Nikhil Bonte
> 
>

Re: Review Request 73423: ATLAS-4338: Import Service: Support change notifiers for migration import

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73423/#review223184
---




repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java
Lines 118 (patched)


Refactor: Move to separate file.


- Ashutosh Mestry


On June 21, 2021, 2:06 p.m., Nikhil Bonte wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73423/
> ---
> 
> (Updated June 21, 2021, 2:06 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
> Rodrigues, Sarath Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-4338
> https://issues.apache.org/jira/browse/ATLAS-4338
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently Migration import feature (MigrationImport.java) does not support 
> enabling notifiers (audits writer, notification to Atlas entities topic).
> 
> With this improvement, Atlas now will enable notifiers for Migration import, 
> based on a flag.
> 
>  
> 
> Approach:
> 
> * New option in AtlasImportRequest - sendNotifications (default : false)
> * MigrationImport will use the flag to select appropriate implementation 
> of IAtlasEntityChangeNotifier to create EntityCreationManager.
> * EntityCreationManager will then notify listeners.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
> 2c1870444 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
>  1d29bf833 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
>  99be169f2 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
>  8e17fd410 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java
>  d6f23d6e2 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java
>  b73988fd7 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityCreationManager.java
>  734add6d7 
> 
> 
> Diff: https://reviews.apache.org/r/73423/diff/1/
> 
> 
> Testing
> ---
> 
> * Tested Manually
> * Sample request
> {
> "options": {
> "migration": "true",
> "size": "100",
> "sendNotifications": "true",
> "batchSize": 500,
> "numWorkers": 40
> }
> }
> * took ~270 secs to import 12987 entities with notifiers enabled.
> * PC build : 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/679/console
> 
> 
> Thanks,
> 
> Nikhil Bonte
> 
>

Re: Review Request 73431: ATLAS-4341 : Regression: DSL query with like operator and regex expression does not work

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73431/#review223181
---


Ship it!




Ship It!

- Ashutosh Mestry


On June 22, 2021, 5:21 p.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73431/
> ---
> 
> (Updated June 22, 2021, 5:21 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4341
> https://issues.apache.org/jira/browse/ATLAS-4341
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Issue:**
> 
> hdfs_path having 5 entities with suffix 0 to 5
> 
> /api/atlas/v2/search/dsl?typeName=hdfs_path&query=name%20like%20%22hdfs_path_vzxtp_*%22
> **Finds the result**
> 
> But
> 
> /api/atlas/v2/search/dsl?typeName=hdfs_path&query=name like 
> "hdfs_path_vzxtp_[0-9]"
> **Returns empty**
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java 
> d2906ea40 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 3404dc6e2 
> 
> 
> Diff: https://reviews.apache.org/r/73431/diff/2/
> 
> 
> Testing
> ---
> 
> Manually verified
> 
> Precommit : 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/692/ 
> (In Progress)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 73431: ATLAS-4341 : Regression: DSL query with like operator and regex expression does not work

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73431/#review223180
---




repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java
Line 39 (original), 39 (patched)


Can you please add GremlinQueriesComposerTest?


- Ashutosh Mestry


On June 22, 2021, 5:11 p.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73431/
> ---
> 
> (Updated June 22, 2021, 5:11 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4341
> https://issues.apache.org/jira/browse/ATLAS-4341
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Issue:**
> 
> hdfs_path having 5 entities with suffix 0 to 5
> 
> /api/atlas/v2/search/dsl?typeName=hdfs_path&query=name%20like%20%22hdfs_path_vzxtp_*%22
> **Finds the result**
> 
> But
> 
> /api/atlas/v2/search/dsl?typeName=hdfs_path&query=name like 
> "hdfs_path_vzxtp_[0-9]"
> **Returns empty**
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java 
> d2906ea40 
> 
> 
> Diff: https://reviews.apache.org/r/73431/diff/1/
> 
> 
> Testing
> ---
> 
> Manually verified
> 
> Precommit :
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 73430: ATLAS-4340: Set Solr wait-searcher property to false by default to make Solr commits async

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73430/#review223178
---




intg/src/main/java/org/apache/atlas/ApplicationProperties.java
Lines 349 (patched)


What is the need for this if condition:
boolean solrWaitSearcherValue = 
getBoolean(SOLR_WAIT_SEARCHER_CONF, DEFAULT_SOLR_WAIT_SEARCHER);



repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
Line 880 (original), 880 (patched)


Not necessary.



repository/src/test/java/org/apache/atlas/discovery/ClassificationSearchProcessorTest.java
Lines 76 (patched)


Sorround with: 
try {
LOG.debug("Waiting for indexes to catch-up: Starting...");
Thread.sleep(1);
} catch (InterruptedException e) {
e.printStackTrace();
}
finally {
LOG.debug("Waiting for indexes to catch-up: Done!");
}



repository/src/test/java/org/apache/atlas/discovery/EntitySearchProcessorTest.java
Line 146 (original), 146 (patched)


Not necessary.



repository/src/test/java/org/apache/atlas/repository/store/graph/v2/EntityCorrelationStoreTest.java
Line 57 (original), 57 (patched)


Not necessary.



webapp/src/test/java/org/apache/atlas/web/adapters/TypeDefsRESTTest.java
Line 80 (original), 80 (patched)


Not necessary.



webapp/src/test/java/org/apache/atlas/web/adapters/TypeDefsRESTTest.java
Line 143 (original), 145 (patched)


Not necessary.



webapp/src/test/java/org/apache/atlas/web/adapters/TypeDefsRESTTest.java
Lines 208 (patched)


Sorround with:
try {
LOG.debug("Waiting for indexes to catch-up: Starting...");
Thread.sleep(1);
} catch (InterruptedException e) {
e.printStackTrace();
}
finally {
LOG.debug("Waiting for indexes to catch-up: Done!");
}



webapp/src/test/java/org/apache/atlas/web/adapters/TypeDefsRESTTest.java
Line 221 (original), 224 (patched)


Not necessary.


- Ashutosh Mestry


On June 22, 2021, 4:29 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73430/
> ---
> 
> (Updated June 22, 2021, 4:29 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
> Nikhil Bonte, and Pinal Shah.
> 
> 
> Bugs: ATLAS-4340
> https://issues.apache.org/jira/browse/ATLAS-4340
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> In Atlas when a transaction is committed, the entries are committed to HBase 
> (primary storage) and Solr (indexing storage). A transaction is rolled-back 
> if the primary storage commit fails, on the other hand when the secondary 
> commit fails (solr), the transaction is not-rolled back and logged as warning 
> and it is recommended to use reindex to repair the missing index documents. 
> This behavior is due to the fact that the primary storage is the source of 
> truth and indexes can be rebuild.
> 
> In Janusgraph, there is a property for Solr to make solr commits async. This 
> is set to true in Atlas making every commit to wait until the solr commit is 
> successful. This will have a negative impact on performance and is 
> recommended to be false by default.
> 
> Property: index.[X].solr.wait-searcher
> 
> When mutating - wait for the index to reflect new mutations before returning. 
> This can have a negative impact on performance.
>  
> 
> This Jira is about setting the default value for above property to FALSE and 
> can be overridden if need arises.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/ApplicationProperties.java bf97ab146 
>   repository/src/test/java/org/apache/atlas/BasicTestSetup.java a1d7b62a5 
>   
> repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
>  a9fbd435a 
>   
> repository/src/test/java/org/apache/atlas/discovery/ClassificationSearchProcessorTest.java
>  121dca9f5 
>   
> repository/src/test/java/org/apache/atlas/discovery/EntitySearchProcessorTest.java
>  764568915 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/EntityCorrelationStoreTest.java
>  a3be5f4ed 
> 

Re: Review Request 73394: ATLAS-4317 : Upgrade curator jar version to 4.3.0

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73394/#review223166
---


Ship it!




Ship It!

- Ashutosh Mestry


On June 1, 2021, 4:06 p.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73394/
> ---
> 
> (Updated June 1, 2021, 4:06 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Nixon Rodrigues, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-4317
> https://issues.apache.org/jira/browse/ATLAS-4317
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Upgrade curator jar version to 4.3.0
> 
> 
> Diffs
> -
> 
>   pom.xml 91c23ee04 
> 
> 
> Diff: https://reviews.apache.org/r/73394/diff/1/
> 
> 
> Testing
> ---
> 
> Atlas server running successfully
> mvn clean install 
> mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
> 
> 
> Thanks,
> 
> chaitali
> 
>

Re: Review Request 73416: ATLAS-4336 Restrict attributes of type to be created starting with double underscore

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73416/#review223156
---


Ship it!




Ship It!

- Ashutosh Mestry


On June 15, 2021, 8:57 a.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73416/
> ---
> 
> (Updated June 15, 2021, 8:57 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4336
> https://issues.apache.org/jira/browse/ATLAS-4336
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently type can be updated with an attribute whose name starts with "__" 
> like __guid.
> 
> This gets mixed up with system attributes.
> 
> Hence attribute names should not be allowed to be started with double 
> underscore or following attribute names (system attribute names) should be 
> restricted when a type is created or updated :
> 
> __classificationNames
> __modifiedBy
> __createdBy
> __state
> __typeName
> __modificationTimestamp
> __propagatedClassificationNames
> __customAttributes
> __isIncomplete
> __guid
> __timestamp
> __labels
> 
> 
> Diffs
> -
> 
>   intg/src/test/java/org/apache/atlas/TestUtilsV2.java f0ec675 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasAbstractDefStoreV2.java
>  3dab120 
>   webapp/src/test/java/org/apache/atlas/web/adapters/TypeDefsRESTTest.java 
> 962345b 
> 
> 
> Diff: https://reviews.apache.org/r/73416/diff/4/
> 
> 
> Testing
> ---
> 
> PreCommit: 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/651/consoleFull
> 
> 
> Thanks,
> 
> Mandar Ambawane
> 
>

Re: Review Request 73416: ATLAS-4336 Restrict attributes of type to be created starting with double underscore

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73416/#review223143
---




repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasAbstractDefStoreV2.java
Line 117 (original), 117 (patched)


I don't think this condition will work. It should be:
if (!allowReservedKeywords && AtlasDSL.Parser.isKeyword(attrDef.getName()) 
|| isInternalAttributeName(attrDef.getName())


- Ashutosh Mestry


On June 14, 2021, 1:57 p.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73416/
> ---
> 
> (Updated June 14, 2021, 1:57 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4336
> https://issues.apache.org/jira/browse/ATLAS-4336
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently type can be updated with an attribute whose name starts with "__" 
> like __guid.
> 
> This gets mixed up with system attributes.
> 
> Hence attribute names should not be allowed to be started with double 
> underscore or following attribute names (system attribute names) should be 
> restricted when a type is created or updated :
> 
> __classificationNames
> __modifiedBy
> __createdBy
> __state
> __typeName
> __modificationTimestamp
> __propagatedClassificationNames
> __customAttributes
> __isIncomplete
> __guid
> __timestamp
> __labels
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasAbstractDefStoreV2.java
>  3dab120 
> 
> 
> Diff: https://reviews.apache.org/r/73416/diff/2/
> 
> 
> Testing
> ---
> 
> PreCommit: 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/651/consoleFull
> 
> 
> Thanks,
> 
> Mandar Ambawane
> 
>

Re: Review Request 73370: ATLAS-4300 : DSL Search : Support search by classification and its attribute

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73370/#review223142
---


Ship it!




Ship It!

- Ashutosh Mestry


On June 14, 2021, 2:21 p.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73370/
> ---
> 
> (Updated June 14, 2021, 2:21 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4300
> https://issues.apache.org/jira/browse/ATLAS-4300
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Problem:** Advanced Search support Searching entities by classification 
> Ex. Table isA Dimension
> 
> But user cannot search entities by classification attributes
> Ex. To search entities which has classification Dimension and Dimension.attr 
> = value
> 
> **Improvement:** User will be able to search entities through classification 
> attributes too.
> 
> **Example query:** "From Table where name = 'sales_fact' and Dimension.attr = 
> 'value' "
> 
> 
> Diffs
> -
> 
>   docs/src/documents/Search/SearchAdvanced.md 45e63675a 
>   repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 80250fb96 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> cff7afffc 
>   repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java 
> d2906ea40 
>   repository/src/main/java/org/apache/atlas/query/RegistryBasedLookup.java 
> eb3c349f6 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java 
> 4091fe982 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 
> 4bdf479f1 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java 
> 04f602c0f 
>   repository/src/test/java/org/apache/atlas/BasicTestSetup.java a821b2549 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 3404dc6e2 
>   
> repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
> 6220c23b9 
> 
> 
> Diff: https://reviews.apache.org/r/73370/diff/3/
> 
> 
> Testing
> ---
> 
> Unit test cases added
> Precommit : 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/659/
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 73370: ATLAS-4300 : DSL Search : Support search by classification and its attribute

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73370/#review223140
---



Can you please few lines that describe the syntax to documentation.

- Ashutosh Mestry


On June 7, 2021, 10:28 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73370/
> ---
> 
> (Updated June 7, 2021, 10:28 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4300
> https://issues.apache.org/jira/browse/ATLAS-4300
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Problem:** Advanced Search support Searching entities by classification 
> Ex. Table isA Dimension
> 
> But user cannot search entities by classification attributes
> Ex. To search entities which has classification Dimension and Dimension.attr 
> = value
> 
> **Improvement:** User will be able to search entities through classification 
> attributes too.
> 
> **Example query:** "From Table where name = 'sales_fact' and Dimension.attr = 
> 'value' "
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 80250fb96 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 320acbe22 
>   repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java 
> c53a324a1 
>   repository/src/main/java/org/apache/atlas/query/RegistryBasedLookup.java 
> eb3c349f6 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java 
> 4091fe982 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 
> 4bdf479f1 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java 
> 04f602c0f 
>   repository/src/test/java/org/apache/atlas/BasicTestSetup.java 99e075ac5 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 0c1dd59d2 
>   
> repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
> 3a2d66d8c 
> 
> 
> Diff: https://reviews.apache.org/r/73370/diff/2/
> 
> 
> Testing
> ---
> 
> Unit test cases added
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 73370: ATLAS-4300 : DSL Search : Support search by classification and its attribute

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73370/#review223139
---




repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java
Lines 363 (patched)


shouldn't new context be passed?
return new GremlinQueryComposer(lookup, new Context(lookup), 
queryMetadata, this.providedLimit, this.providedOffset, null);


- Ashutosh Mestry


On June 7, 2021, 10:28 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73370/
> ---
> 
> (Updated June 7, 2021, 10:28 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4300
> https://issues.apache.org/jira/browse/ATLAS-4300
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Problem:** Advanced Search support Searching entities by classification 
> Ex. Table isA Dimension
> 
> But user cannot search entities by classification attributes
> Ex. To search entities which has classification Dimension and Dimension.attr 
> = value
> 
> **Improvement:** User will be able to search entities through classification 
> attributes too.
> 
> **Example query:** "From Table where name = 'sales_fact' and Dimension.attr = 
> 'value' "
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 80250fb96 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 320acbe22 
>   repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java 
> c53a324a1 
>   repository/src/main/java/org/apache/atlas/query/RegistryBasedLookup.java 
> eb3c349f6 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java 
> 4091fe982 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 
> 4bdf479f1 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java 
> 04f602c0f 
>   repository/src/test/java/org/apache/atlas/BasicTestSetup.java 99e075ac5 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 0c1dd59d2 
>   
> repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
> 3a2d66d8c 
> 
> 
> Diff: https://reviews.apache.org/r/73370/diff/2/
> 
> 
> Testing
> ---
> 
> Unit test cases added
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 73416: ATLAS-4336 Restrict attributes of type to be created starting with double underscore

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73416/#review223138
---




repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasAbstractDefStoreV2.java
Line 117 (original), 117 (patched)


I think it would be possible to combine 117 and 120 in one clause.


- Ashutosh Mestry


On June 11, 2021, 1:09 p.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73416/
> ---
> 
> (Updated June 11, 2021, 1:09 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4336
> https://issues.apache.org/jira/browse/ATLAS-4336
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently type can be updated with an attribute whose name starts with "__" 
> like __guid.
> 
> This gets mixed up with system attributes.
> 
> Hence attribute names should not be allowed to be started with double 
> underscore or following attribute names (system attribute names) should be 
> restricted when a type is created or updated :
> 
> __classificationNames
> __modifiedBy
> __createdBy
> __state
> __typeName
> __modificationTimestamp
> __propagatedClassificationNames
> __customAttributes
> __isIncomplete
> __guid
> __timestamp
> __labels
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasAbstractDefStoreV2.java
>  3dab120 
> 
> 
> Diff: https://reviews.apache.org/r/73416/diff/1/
> 
> 
> Testing
> ---
> 
> PreCommit: 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/651/consoleFull
> 
> 
> Thanks,
> 
> Mandar Ambawane
> 
>

Re: Review Request 73412: ATLAS-4332 : DSL Query : query with like operator and "/" in search text throws 500

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73412/#review223130
---


Ship it!




Ship It!

- Ashutosh Mestry


On June 10, 2021, 11:32 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73412/
> ---
> 
> (Updated June 10, 2021, 11:32 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4332
> https://issues.apache.org/jira/browse/ATLAS-4332
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Problem:** In DSL query , searching with like operator in combination of 
> special characters like '/' in search text throws 500
> 
> **Example:** 
> hdfs_path name="/warehouse/tablespace/external/hive/db_fdchj.db" returns 
> right entity.
> 
> hdfs_path name like "/warehouse/*" throws following exception:
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 320acbe22 
>   repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java 
> c53a324a1 
>   repository/src/test/java/org/apache/atlas/BasicTestSetup.java 99e075ac5 
>   repository/src/test/java/org/apache/atlas/query/BaseDSLComposer.java 
> b10400157 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 0c1dd59d2 
>   
> repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
> 3a2d66d8c 
> 
> 
> Diff: https://reviews.apache.org/r/73412/diff/2/
> 
> 
> Testing
> ---
> 
> unit testcase added
> 
> Precommit : 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/640/console
>  (In Progress)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 73412: ATLAS-4332 : DSL Query : query with like operator and "/" in search text throws 500

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73412/#review223129
---




repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java
Lines 40 (patched)


Rename: ESCAPE_CHARACTERS_SET


- Ashutosh Mestry


On June 10, 2021, 11:32 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73412/
> ---
> 
> (Updated June 10, 2021, 11:32 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4332
> https://issues.apache.org/jira/browse/ATLAS-4332
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Problem:** In DSL query , searching with like operator in combination of 
> special characters like '/' in search text throws 500
> 
> **Example:** 
> hdfs_path name="/warehouse/tablespace/external/hive/db_fdchj.db" returns 
> right entity.
> 
> hdfs_path name like "/warehouse/*" throws following exception:
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 320acbe22 
>   repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java 
> c53a324a1 
>   repository/src/test/java/org/apache/atlas/BasicTestSetup.java 99e075ac5 
>   repository/src/test/java/org/apache/atlas/query/BaseDSLComposer.java 
> b10400157 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 0c1dd59d2 
>   
> repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
> 3a2d66d8c 
> 
> 
> Diff: https://reviews.apache.org/r/73412/diff/2/
> 
> 
> Testing
> ---
> 
> unit testcase added
> 
> Precommit : 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/640/console
>  (In Progress)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 73412: ATLAS-4332 : DSL Query : query with like operator and "/" in search text throws 500

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73412/#review223125
---



Can you please add unit test in GremlinQueryComposerTest.


repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java
Lines 137 (patched)


A static private final Set can be used here, instead.


- Ashutosh Mestry


On June 9, 2021, 2:47 p.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73412/
> ---
> 
> (Updated June 9, 2021, 2:47 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4332
> https://issues.apache.org/jira/browse/ATLAS-4332
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Problem:** In DSL query , searching with like operator in combination of 
> special characters like '/' in search text throws 500
> 
> **Example:** 
> hdfs_path name="/warehouse/tablespace/external/hive/db_fdchj.db" returns 
> right entity.
> 
> hdfs_path name like "/warehouse/*" throws following exception:
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 320acbe22 
>   repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java 
> c53a324a1 
>   repository/src/test/java/org/apache/atlas/BasicTestSetup.java 99e075ac5 
>   repository/src/test/java/org/apache/atlas/query/BaseDSLComposer.java 
> b10400157 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 0c1dd59d2 
> 
> 
> Diff: https://reviews.apache.org/r/73412/diff/1/
> 
> 
> Testing
> ---
> 
> unit testcase added
> 
> Precommit : 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/640/console
>  (In Progress)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 73404: ATLAS-4324: FS entity created for load data inpath is created as shell entity - Part 2

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73404/
---

(Updated June 8, 2021, 11:07 p.m.)


Review request for atlas, Mandar Ambawane, Radhika Kundam, and Sarath 
Subramanian.


Changes
---

Updates include: Added screen shot from functional testing.


Bugs: ATLAS-4324
https://issues.apache.org/jira/browse/ATLAS-4324


Repository: atlas


Description
---

**Approach**
Updates:
- Simplified approach.
- Maintained abstraction.
- Updated unit tests.


Diffs
-

  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
94ef22577 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilter.java
 737c637bf 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
 9163c4709 
  
addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilterTest.java
 4dde1dc2f 
  addons/hive-bridge/src/test/resources/json/hs2-create-db-v2.json 881ee105e 
  
addons/hive-bridge/src/test/resources/json/hs2-create-db-with-no-pathentities-to-retain-v2.json
 42553b5cf 
  
addons/hive-bridge/src/test/resources/json/hs2-create-db-with-no-pathentities-to-retain.json
 a5b810f72 
  addons/hive-bridge/src/test/resources/json/hs2-create-process-v2.json 
8e55b7282 
  addons/hive-bridge/src/test/resources/json/hs2-create-table-v2.json 801918e04 
  addons/hive-bridge/src/test/resources/json/hs2-load-inpath-v2.json dd31aa073 
  
addons/hive-bridge/src/test/resources/json/hs2-load-inpath-with-no-pathentities-to-retain-v2.json
 7f90d1986 
  
addons/hive-bridge/src/test/resources/json/hs2-load-inpath-with-no-pathentities-to-retain.json
 dd31aa073 
  addons/hive-bridge/src/test/resources/json/hs2-load-inpath.json dd31aa073 
  addons/hive-bridge/src/test/resources/json/hs2-table-rename-v2.json f2b2bf7a8 


Diff: https://reviews.apache.org/r/73404/diff/2/


Testing
---

**Unit tests**
- Additional unit test.

**PC Build**
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/617/


File Attachments (updated)


Screen Shot 2021-06-08 at 4.05.26 PM.png
  
https://reviews.apache.org/media/uploaded/files/2021/06/08/b15b33fc-007d-4ee3-8862-64750af16432__Screen_Shot_2021-06-08_at_4.05.26_PM.png


Thanks,

Ashutosh Mestry

Re: Review Request 73409: ATLAS-4330: Add Kafka topics lag information on metrics and log

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73409/#review223122
---



Can you please add log output?

- Ashutosh Mestry


On June 7, 2021, 11:12 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73409/
> ---
> 
> (Updated June 7, 2021, 11:12 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Deep Singh, Jayendra Parab, Madhan 
> Neethiraj, Nikhil Bonte, Nixon Rodrigues, Radhika Kundam, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-4330
> https://issues.apache.org/jira/browse/ATLAS-4330
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> In metrics page, in addition to other Notification details. Lag information 
> of kafka consumer topics will be useful to check the current backed up 
> messages in Atlas: 
> 
> The solution involves using the kafka admin client APIs to get log end offset 
> of a topic for the consumer ('atlas')
> 
> Config property:
> 
> atlas.notification.consumer.topic.report.frequency.seconds=60 (default value 
> 3600secs or 1 hour)
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/utils/KafkaUtils.java 167442259 
>   
> dashboardv2/public/js/templates/site/Statistics_Topic_Offset_table_tmpl.html 
> b4c8d8677 
>   dashboardv2/public/js/views/site/Statistics.js 32a66691c 
>   
> dashboardv3/public/js/templates/site/Statistics_Topic_Offset_table_tmpl.html 
> b4c8d8677 
>   dashboardv3/public/js/utils/Enums.js ee80313c2 
>   dashboardv3/public/js/views/site/Statistics.js 9eb714d45 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
> 96dc5856a 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java 
> af3727df4 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java
>  1fb9f9989 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
>  1b486e528 
>   repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java 
> beb90e67b 
>   repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 
> bea8eb8b5 
>   
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  5643af95c 
> 
> 
> Diff: https://reviews.apache.org/r/73409/diff/1/
> 
> 
> Testing
> ---
> 
> Precommit: 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/624/console
> 
> Manually verified lag information is updated correctly on metrics page and 
> periodically printed on log
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

Re: Review Request 73404: ATLAS-4324: FS entity created for load data inpath is created as shell entity - Part 2

[Ashutosh Mestry via Review Board]

> On June 4, 2021, 10:46 p.m., Radhika Kundam wrote:
> > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
> > Line 200 (original), 194 (patched)
> > 
> >
> > This will keep negative guid not only to Path entity types but for all 
> > typesToRetain.IdBasedResolver will not be able to find the entities like 
> > hive_table with -ve guid which are already created. We need to keep 
> > negative guid only for Path entity types.

The allowed types are OK to be resolved locally. Also, the way lineage entities 
are created, they don't refer to each other. The inputs and output refer to 
Asset types. Hence this is fine this way.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73404/#review223107
---


On June 4, 2021, 8:43 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73404/
> ---
> 
> (Updated June 4, 2021, 8:43 p.m.)
> 
> 
> Review request for atlas, Mandar Ambawane, Radhika Kundam, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-4324
> https://issues.apache.org/jira/browse/ATLAS-4324
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> Updates:
> - Simplified approach.
> - Maintained abstraction.
> - Updated unit tests.
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 94ef22577 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilter.java
>  737c637bf 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
>  9163c4709 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilterTest.java
>  4dde1dc2f 
>   addons/hive-bridge/src/test/resources/json/hs2-create-db-v2.json 881ee105e 
>   
> addons/hive-bridge/src/test/resources/json/hs2-create-db-with-no-pathentities-to-retain-v2.json
>  42553b5cf 
>   
> addons/hive-bridge/src/test/resources/json/hs2-create-db-with-no-pathentities-to-retain.json
>  a5b810f72 
>   addons/hive-bridge/src/test/resources/json/hs2-create-process-v2.json 
> 8e55b7282 
>   addons/hive-bridge/src/test/resources/json/hs2-create-table-v2.json 
> 801918e04 
>   addons/hive-bridge/src/test/resources/json/hs2-load-inpath-v2.json 
> dd31aa073 
>   
> addons/hive-bridge/src/test/resources/json/hs2-load-inpath-with-no-pathentities-to-retain-v2.json
>  7f90d1986 
>   
> addons/hive-bridge/src/test/resources/json/hs2-load-inpath-with-no-pathentities-to-retain.json
>  dd31aa073 
>   addons/hive-bridge/src/test/resources/json/hs2-load-inpath.json dd31aa073 
>   addons/hive-bridge/src/test/resources/json/hs2-table-rename-v2.json 
> f2b2bf7a8 
> 
> 
> Diff: https://reviews.apache.org/r/73404/diff/2/
> 
> 
> Testing
> ---
> 
> **Unit tests**
> - Additional unit test.
> 
> **PC Build**
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/617/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 73404: ATLAS-4324: FS entity created for load data inpath is created as shell entity - Part 2

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73404/
---

(Updated June 4, 2021, 8:43 p.m.)


Review request for atlas, Mandar Ambawane, Radhika Kundam, and Sarath 
Subramanian.


Changes
---

Updates include: Addressed review comments.


Bugs: ATLAS-4324
https://issues.apache.org/jira/browse/ATLAS-4324


Repository: atlas


Description
---

**Approach**
Updates:
- Simplified approach.
- Maintained abstraction.
- Updated unit tests.


Diffs (updated)
-

  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
94ef22577 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilter.java
 737c637bf 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
 9163c4709 
  
addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilterTest.java
 4dde1dc2f 
  addons/hive-bridge/src/test/resources/json/hs2-create-db-v2.json 881ee105e 
  
addons/hive-bridge/src/test/resources/json/hs2-create-db-with-no-pathentities-to-retain-v2.json
 42553b5cf 
  
addons/hive-bridge/src/test/resources/json/hs2-create-db-with-no-pathentities-to-retain.json
 a5b810f72 
  addons/hive-bridge/src/test/resources/json/hs2-create-process-v2.json 
8e55b7282 
  addons/hive-bridge/src/test/resources/json/hs2-create-table-v2.json 801918e04 
  addons/hive-bridge/src/test/resources/json/hs2-load-inpath-v2.json dd31aa073 
  
addons/hive-bridge/src/test/resources/json/hs2-load-inpath-with-no-pathentities-to-retain-v2.json
 7f90d1986 
  
addons/hive-bridge/src/test/resources/json/hs2-load-inpath-with-no-pathentities-to-retain.json
 dd31aa073 
  addons/hive-bridge/src/test/resources/json/hs2-load-inpath.json dd31aa073 
  addons/hive-bridge/src/test/resources/json/hs2-table-rename-v2.json f2b2bf7a8 


Diff: https://reviews.apache.org/r/73404/diff/2/

Changes: https://reviews.apache.org/r/73404/diff/1-2/


Testing (updated)
---

**Unit tests**
- Additional unit test.

**PC Build**
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/617/


Thanks,

Ashutosh Mestry

Re: Review Request 73405: ATLAS-4302: Migrated Data: Process Entity Name not set to QualifiedName

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73405/
---

(Updated June 4, 2021, 4:48 a.m.)


Review request for atlas, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.


Changes
---

Updates include: Addressed review comments.


Bugs: ATLAS-4302
https://issues.apache.org/jira/browse/ATLAS-4302


Repository: atlas


Description
---

**Approach**
Java patch is optionally run when 
```atlas.process.name.update.patch

is set to true.


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java e31afa89a 
  
repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java
 fae28c440 
  
repository/src/main/java/org/apache/atlas/repository/patches/ProcessNamePatch.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/73405/diff/2/

Changes: https://reviews.apache.org/r/73405/diff/1-2/


Testing
---

**Functional test**
- Validated with medium data set (5M entities).

**Volume test**
- Validated improvement for a lineage entity with 1200 inputs and 1700 outputs. 
With this change, the _AtlasEntityHeader_ payload size reduces by 90%.


Thanks,

Ashutosh Mestry

Review Request 73405: ATLAS-4302: Migrated Data: Process Entity Name not set to QualifiedName

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73405/
---

Review request for atlas, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.


Bugs: ATLAS-4302
https://issues.apache.org/jira/browse/ATLAS-4302


Repository: atlas


Description
---

**Approach**
Java patch is optionally run when 
```atlas.process.name.update.patch

is set to true.


Diffs
-

  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java e31afa89a 
  
repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java
 fae28c440 
  
repository/src/main/java/org/apache/atlas/repository/patches/ProcessNamePatch.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/73405/diff/1/


Testing
---

**Functional test**
- Validated with medium data set (5M entities).

**Volume test**
- Validated improvement for a lineage entity with 1200 inputs and 1700 outputs. 
With this change, the _AtlasEntityHeader_ payload size reduces by 90%.


Thanks,

Ashutosh Mestry

Review Request 73404: ATLAS-4324: FS entity created for load data inpath is created as shell entity - Part 2

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73404/
---

Review request for atlas and Radhika Kundam.


Bugs: ATLAS-4324
https://issues.apache.org/jira/browse/ATLAS-4324


Repository: atlas


Description
---

**Approach**
Updates:
- Simplified approach.
- Maintained abstraction.
- Updated unit tests.


Diffs
-

  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilter.java
 737c637bf 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
 9163c4709 
  
addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilterTest.java
 4dde1dc2f 
  
addons/hive-bridge/src/test/resources/json/hs2-create-db-with-no-pathentities-to-retain-v2.json
 42553b5cf 
  
addons/hive-bridge/src/test/resources/json/hs2-create-db-with-no-pathentities-to-retain.json
 a5b810f72 
  addons/hive-bridge/src/test/resources/json/hs2-load-inpath-v2.json dd31aa073 
  
addons/hive-bridge/src/test/resources/json/hs2-load-inpath-with-no-pathentities-to-retain-v2.json
 7f90d1986 
  
addons/hive-bridge/src/test/resources/json/hs2-load-inpath-with-no-pathentities-to-retain.json
 dd31aa073 
  addons/hive-bridge/src/test/resources/json/hs2-load-inpath.json dd31aa073 


Diff: https://reviews.apache.org/r/73404/diff/1/


Testing
---

**Unit tests**
- Additional unit test.


Thanks,

Ashutosh Mestry

Re: Review Request 73402: ATLAS-4301 Handle Test Case Failure on Pre-commit environment

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73402/#review223095
---


Ship it!




Can you please add details of end-to-end testing done for HMS and HS2 hooks? 
Essentially we should exercise the 'load inpath' queries.

Good work!

- Ashutosh Mestry


On June 3, 2021, 2:23 p.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73402/
> ---
> 
> (Updated June 3, 2021, 2:23 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4301
> https://issues.apache.org/jira/browse/ATLAS-4301
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Getting following error on Pre-commit build due to change in the file path
> 
> hive-bridge/target/logs/application.log
> 
> Wrong FS: 
> pfile:/home/jenkins/jenkins-agent/workspace/Atlas/PreCommit-ATLAS-Build-Test/addons/hive-bridge/target/partition-data-{dir},
>  expected: file:///
>  
> 
> Also,
> 
> There is Enum "Type" in the "Entity"
> 
> Class: org.apache.hadoop.hive.ql.hooks.Entity
> Enum: Type
> Enum "Type" has one constant "LOCAL_DIR"
> 
> Due to recent changes, we need to provide support for this constant 
> "LOCAL_DIR" in Testing Environment.
> 
> Without which following issues occuring on Testing Environment:
> 
> 1. While creating "hive_process" entity, The "outputs" attribute is not 
> getting set (Which is of type "hdfs_path").
> 
> 2. While setting the "qualifiedName" of "hive_process" entity, File path is 
> not getting appended.
> 
> This causing Failure of some Test cases.
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  21c0ae3 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 
> 7a11ec3 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 52684e7 
> 
> 
> Diff: https://reviews.apache.org/r/73402/diff/1/
> 
> 
> Testing
> ---
> 
> PreCommit: 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/612/consoleFull
> 
> 
> Thanks,
> 
> Mandar Ambawane
> 
>

Re: Review Request 73317: ATLAS-3812: Add schema for Apache Flink

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73317/#review223094
---


Ship it!




Ship It!

- Ashutosh Mestry


On June 1, 2021, 6:59 p.m., Josh Yeh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73317/
> ---
> 
> (Updated June 1, 2021, 6:59 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Pinal Shah, and Sarath Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-3812: Add schema for Apache Flink
> 
> 
> Diffs
> -
> 
>   addons/models/1000-Hadoop/1110-flink_model.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/73317/diff/4/
> 
> 
> Testing
> ---
> 
> Tested on my CDH cluster with flink example wordcount streaming job.
> 
> 
> File Attachments
> 
> 
> Screen Shot 2021-04-30 at 12.38.27 AM.png
>   
> https://reviews.apache.org/media/uploaded/files/2021/04/30/2ff59445-7237-4be1-a1a0-b6db70bc7827__Screen_Shot_2021-04-30_at_12.38.27_AM.png
> 
> 
> Thanks,
> 
> Josh Yeh
> 
>

Re: Review Request 73317: ATLAS-3812: Add schema for Apache Flink

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73317/#review223088
---



Thanks for the patch. The model looks good.
Can you please attach a screen shot (like you did before) that create entities 
using the new model? That will provide us with a good reference.
It will also help if you can attach the JSON that is used to create the 
entities.

- Ashutosh Mestry


On June 1, 2021, 6:59 p.m., Josh Yeh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73317/
> ---
> 
> (Updated June 1, 2021, 6:59 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Pinal Shah, and Sarath Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-3812: Add schema for Apache Flink
> 
> 
> Diffs
> -
> 
>   addons/models/1000-Hadoop/1110-flink_model.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/73317/diff/4/
> 
> 
> Testing
> ---
> 
> Tested on my CDH cluster with flink example wordcount streaming job.
> 
> 
> File Attachments
> 
> 
> Screen Shot 2021-04-30 at 12.38.27 AM.png
>   
> https://reviews.apache.org/media/uploaded/files/2021/04/30/2ff59445-7237-4be1-a1a0-b6db70bc7827__Screen_Shot_2021-04-30_at_12.38.27_AM.png
> 
> 
> Thanks,
> 
> Josh Yeh
> 
>

Re: Review Request 73394: ATLAS-4317 : Upgrade curator jar version to 4.3.0

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73394/#review223082
---


Ship it!




Can you please add PC build details.

- Ashutosh Mestry


On June 1, 2021, 4:06 p.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73394/
> ---
> 
> (Updated June 1, 2021, 4:06 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Nixon Rodrigues, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-4317
> https://issues.apache.org/jira/browse/ATLAS-4317
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Upgrade curator jar version to 4.3.0
> 
> 
> Diffs
> -
> 
>   pom.xml 91c23ee04 
> 
> 
> Diff: https://reviews.apache.org/r/73394/diff/1/
> 
> 
> Testing
> ---
> 
> Atlas server running successfully
> mvn clean install 
> mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
> 
> 
> Thanks,
> 
> chaitali
> 
>

Re: Review Request 73388: ATLAS-4174 : [Atlas: Glossary Term Bulk Import] When the file size is large (5 mb) bulk import times out

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73388/#review223066
---



Does 'timeout' of 0 indicate that no timeout is specified?

- Ashutosh Mestry


On May 27, 2021, 7:06 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73388/
> ---
> 
> (Updated May 27, 2021, 7:06 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4174
> https://issues.apache.org/jira/browse/ATLAS-4174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch has set timeout to 0 or we can say disabled the timeout which will 
> no longer throw the exception for request timeout
> 
> 
> Diffs
> -
> 
>   dashboardv2/public/js/views/import/ImportLayoutView.js 8f7524e34 
>   dashboardv3/public/js/views/import/ImportLayoutView.js 8f7524e34 
> 
> 
> Diff: https://reviews.apache.org/r/73388/diff/1/
> 
> 
> Testing
> ---
> 
> Atlas server runs successfully
> mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
> mvn clean install
> Checked with file 50 kb to 100kb no timeout error was seen.
> 
> 
> Thanks,
> 
> chaitali
> 
>

Re: Review Request 73376: ATLAS-4307: Atlas bulk clarification REST should be enhanced to associate classification to bulk entities with uniq attribute and type

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73376/#review223044
---




intg/src/main/java/org/apache/atlas/model/instance/ClassificationAssociateRequest.java
Lines 47 (patched)


Rename: entitiesUniqAttributes -> entityUniqAttributes


- Ashutosh Mestry


On May 26, 2021, 12:19 a.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73376/
> ---
> 
> (Updated May 26, 2021, 12:19 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Deep Singh, Radhika Kundam, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4307
> https://issues.apache.org/jira/browse/ATLAS-4307
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-4307: Atlas bulk clarification REST should be enhanced to associate 
> classification to bulk entities with uniq attribute and type
> 
> 
> Diffs
> -
> 
>   
> intg/src/main/java/org/apache/atlas/model/instance/ClassificationAssociateRequest.java
>  3f946ef7a 
>   webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java ec893f25c 
> 
> 
> Diff: https://reviews.apache.org/r/73376/diff/1/
> 
> 
> Testing
> ---
> 
> Manually tested. With new changes the three options will be:
> 
> Option 1 (Only with type name and list of uniq attributes)
> 
> curl --location --request POST 
> 'http://atlas-host:31000/api/atlas/v2/entity/bulk/classification?skipFailedEntities=True'
>  \
> --header 'Authorization: Basic YWRtaW46YWRtaW4xMjM=' \
> --header 'Content-Type: application/json' \
> --data-raw '{
>   "classification" : {
> "propagate" : false,
> "attributes" : {
>   "type" : "false"
> },
> "typeName" : "TEST_TAG_TABLE"
>   },
>   "entitiesUniqAttributes" : [
>   {"qualifiedName": "db1.table43@cm"},
>   {"qualifiedName": "db1.table7@cm"},
>   {"qualifiedName": "db1.table55@cm"}],
>   "entityTypeName" : "hive_table"
> }'
> 
> Option 2 (Only with entity Guids)
> curl --location --request POST 
> 'http://atlas-host:31000/api/atlas/v2/entity/bulk/classification?skipFailedEntities=True'
>  \
> --header 'Authorization: Basic YWRtaW46YWRtaW4xMjM=' \
> --header 'Content-Type: application/json' \
> --data-raw '{
>   "classification" : {
> "propagate" : false,
> "attributes" : {
>   "type" : "false"
> },
> "typeName" : "TEST_TAG_TABLE"
>   },
>   "entityGuids" : ["f7288e81-46e4-4c88-b6c7-da092d6b6e35", 
> "1793c6df-53a6-481b-a75c-c705f98d6900"],
>   "entityTypeName" : "hive_table"
> }'
> 
> Option 3 (With both entity GUIDs and type Name with list of uniq attributes)
> curl --location --request POST 
> 'http://atlas-host:31000/api/atlas/v2/entity/bulk/classification?skipFailedEntities=True'
>  \
> --header 'Authorization: Basic YWRtaW46YWRtaW4xMjM=' \
> --header 'Content-Type: application/json' \
> --data-raw '{
>   "classification" : {
> "propagate" : false,
> "attributes" : {
>   "type" : "false"
> },
> "typeName" : "TEST_TAG_TABLE"
>   },
>   "entitiesUniqAttributes" : [
>   {"qualifiedName": "db1.table43@cm"},
>   {"qualifiedName": "db1.table8@cm"},
>   {"qualifiedName": "db1.table55@cm"}],
>   "entityGuids" : [
>   "6d6ab082-9a62-4aea-97b7-2b7337173631",
>   "9a0d3275-eb42-4229-ac51-6db1ddf3ccc6"],
>   "entityTypeName" : "hive_table"
> }'
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>

Re: Review Request 73376: ATLAS-4307: Atlas bulk clarification REST should be enhanced to associate classification to bulk entities with uniq attribute and type

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73376/#review223043
---




webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java
Lines 820 (patched)


Refactor: Extract to method.


- Ashutosh Mestry


On May 26, 2021, 12:19 a.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73376/
> ---
> 
> (Updated May 26, 2021, 12:19 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Deep Singh, Radhika Kundam, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4307
> https://issues.apache.org/jira/browse/ATLAS-4307
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-4307: Atlas bulk clarification REST should be enhanced to associate 
> classification to bulk entities with uniq attribute and type
> 
> 
> Diffs
> -
> 
>   
> intg/src/main/java/org/apache/atlas/model/instance/ClassificationAssociateRequest.java
>  3f946ef7a 
>   webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java ec893f25c 
> 
> 
> Diff: https://reviews.apache.org/r/73376/diff/1/
> 
> 
> Testing
> ---
> 
> Manually tested. With new changes the three options will be:
> 
> Option 1 (Only with type name and list of uniq attributes)
> 
> curl --location --request POST 
> 'http://atlas-host:31000/api/atlas/v2/entity/bulk/classification?skipFailedEntities=True'
>  \
> --header 'Authorization: Basic YWRtaW46YWRtaW4xMjM=' \
> --header 'Content-Type: application/json' \
> --data-raw '{
>   "classification" : {
> "propagate" : false,
> "attributes" : {
>   "type" : "false"
> },
> "typeName" : "TEST_TAG_TABLE"
>   },
>   "entitiesUniqAttributes" : [
>   {"qualifiedName": "db1.table43@cm"},
>   {"qualifiedName": "db1.table7@cm"},
>   {"qualifiedName": "db1.table55@cm"}],
>   "entityTypeName" : "hive_table"
> }'
> 
> Option 2 (Only with entity Guids)
> curl --location --request POST 
> 'http://atlas-host:31000/api/atlas/v2/entity/bulk/classification?skipFailedEntities=True'
>  \
> --header 'Authorization: Basic YWRtaW46YWRtaW4xMjM=' \
> --header 'Content-Type: application/json' \
> --data-raw '{
>   "classification" : {
> "propagate" : false,
> "attributes" : {
>   "type" : "false"
> },
> "typeName" : "TEST_TAG_TABLE"
>   },
>   "entityGuids" : ["f7288e81-46e4-4c88-b6c7-da092d6b6e35", 
> "1793c6df-53a6-481b-a75c-c705f98d6900"],
>   "entityTypeName" : "hive_table"
> }'
> 
> Option 3 (With both entity GUIDs and type Name with list of uniq attributes)
> curl --location --request POST 
> 'http://atlas-host:31000/api/atlas/v2/entity/bulk/classification?skipFailedEntities=True'
>  \
> --header 'Authorization: Basic YWRtaW46YWRtaW4xMjM=' \
> --header 'Content-Type: application/json' \
> --data-raw '{
>   "classification" : {
> "propagate" : false,
> "attributes" : {
>   "type" : "false"
> },
> "typeName" : "TEST_TAG_TABLE"
>   },
>   "entitiesUniqAttributes" : [
>   {"qualifiedName": "db1.table43@cm"},
>   {"qualifiedName": "db1.table8@cm"},
>   {"qualifiedName": "db1.table55@cm"}],
>   "entityGuids" : [
>   "6d6ab082-9a62-4aea-97b7-2b7337173631",
>   "9a0d3275-eb42-4229-ac51-6db1ddf3ccc6"],
>   "entityTypeName" : "hive_table"
> }'
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>

Review Request 73375: Handle Spool Directory Creation by Different Users

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73375/
---

Review request for atlas, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.


Bugs: ATLAS-4306
https://issues.apache.org/jira/browse/ATLAS-4306


Repository: atlas


Description
---

Please see bug description.


Diffs
-

  
notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
 ea3128465 
  
notification/src/main/java/org/apache/atlas/notification/spool/IndexManagement.java
 f018983a4 
  
notification/src/main/java/org/apache/atlas/notification/spool/SpoolConfiguration.java
 76f05efea 
  
notification/src/main/java/org/apache/atlas/notification/spool/SpoolUtils.java 
abbe33d24 
  notification/src/test/java/org/apache/atlas/notification/spool/BaseTest.java 
304c82122 


Diff: https://reviews.apache.org/r/73375/diff/1/


Testing
---

**Functional tests**
Existing workflows.

**Unit tests**
None.


Thanks,

Ashutosh Mestry

Re: Review Request 73372: [Glossary Bulk Import] Rename template file name from "template" to "template_glossary" to maintain parity with Business Metadata Bulk Import

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73372/#review223037
---




webapp/src/main/java/org/apache/atlas/web/rest/GlossaryREST.java
Line 996 (original), 996 (patched)


Extract method: getGlossaryTemplateAsAttachment.



webapp/src/main/java/org/apache/atlas/web/rest/GlossaryREST.java
Line 1001 (original), 1001 (patched)


Refactor: Extract to const field. That way file name is at one place.


- Ashutosh Mestry


On May 24, 2021, 6:54 p.m., Umesh Padashetty wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73372/
> ---
> 
> (Updated May 24, 2021, 6:54 p.m.)
> 
> 
> Review request for atlas, Ashutosh Chauhan and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4305
> https://issues.apache.org/jira/browse/ATLAS-4305
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas provides support to Bulk Import Business Metadata and Glossary. THe 
> user can download the template file and then bulk upload the data to atlas. 
> 
> 
> While the Business Metadata template file is named as 
> "template_business_metadata", Glossary Bulk Import template file is just 
> named as "template". This leads to confusion when the user downloads both the 
> template files and he/she has to open the file to confirm the template type
> 
> 
> Proposal: Rename the template file to template_glossary
> 
> 
> Diffs
> -
> 
>   webapp/src/main/java/org/apache/atlas/web/rest/GlossaryREST.java c1de83be9 
> 
> 
> Diff: https://reviews.apache.org/r/73372/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Umesh Padashetty
> 
>

Re: Review Request 73329: Correlating Deleted Entities with Lineage

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73329/
---

(Updated May 20, 2021, 10:18 p.m.)


Review request for atlas, Radhika Kundam and Sarath Subramanian.


Changes
---

Updates include: Addressed review comments.


Bugs: ATLAS-4152
https://issues.apache.org/jira/browse/ATLAS-4152


Repository: atlas


Description
---

**Background**
As part of ATLAS-4204, HS2 notifications send entity-lineage only (provided the 
poperty is enabled).

When Spooling enabled the order of messages can potentially change. The 
notification messages coming from HS2 and HMS may not be in the same order as 
when they arrived with direct notification.

Problem: 
Consider the sequence of arriving messages:

This is the sequence of messages for Entity 1 (C = create, U = update, D = 
delete, L?x = Lineage of type 'x')
No problem: C1, U1, L1x, L1y, D1
Problem: C1, U1, D1, L1x, L1y

This implementation attempts to handle ths problem mentioned above. If the 
above case is not handled, it will end up creating shell entities, since 
deleted entities are not looked up as part of entity creation.

**Approach**
Used bounded stream approach where an incoming stream of messages is bounded 
with an indicator that it originates from spool. This helps makes localized 
decisions on the incoming stream of messages.

High-level approach:
- Messages when written to the spool are tagged with a timestamp.
- Deleted entities are maintained in a cache.
- Lineage-only message are checked if they refer to a deleted entity.
- If they refer to deleted entity, they are stitched to the one present in the 
cache only if it falls within the threshold.
- Using step-climbing approach for locating right entity to stitch lineage to.

New: _EntityCorrelationsManager_: Uses message timestamp and cached entity 
qualifiedName-GUID map.
Modifed: _NotificationHookConsumer_ Uses the new class.
New: _HiveDDLLineagePreprocess_ Uses entity-correlation to link to deleted 
entities.
Modified: _SpoolConfiguration_: Added new configuration to pause message 
sending after destination is available: 
_atlas.hook.spool.pause.before.send.sec_.
In-memory lookup approach changed to persistent loookup.


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/repository/Constants.java ffcec9743 
  
intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java
 810ba97c9 
  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
f7d9668ec 
  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java 
22bd79fdf 
  notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
3d1b3ccf1 
  
notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java
 3264e264c 
  
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
 edd8ed931 
  
notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
 2d7d19595 
  notification/src/main/java/org/apache/atlas/notification/spool/Publisher.java 
22242c933 
  
notification/src/main/java/org/apache/atlas/notification/spool/SpoolConfiguration.java
 a9a3a78cc 
  notification/src/main/java/org/apache/atlas/notification/spool/Spooler.java 
2cacaaadc 
  
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
 d7e4959f7 
  
notification/src/test/java/org/apache/atlas/notification/spool/AtlasFileSpoolTest.java
 167efbecc 
  
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
 cc727c6ba 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/EntityCorrelationStore.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
 0a9470815 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/EntityCorrelationStoreTest.java
 PRE-CREATION 
  
webapp/src/main/java/org/apache/atlas/notification/EntityCorrelationManager.java
 PRE-CREATION 
  
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
 84cc8d813 
  
webapp/src/main/java/org/apache/atlas/notification/preprocessor/EntityPreprocessor.java
 89568e236 
  
webapp/src/main/java/org/apache/atlas/notification/preprocessor/HiveDbDDLPreprocessor.java
 PRE-CREATION 
  
webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java
 86e3384ee 
  
webapp/src/main/java/org/apache/atlas/notification/preprocessor/HiveTableDDLPreprocessor.java
 PRE-CREATION 
  
webapp/src/main/java/org/apache/atlas/notification/preprocessor/PreprocessorContext.java
 608b4a304 
  
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java
 65e8b5001 
  
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java
 15a19005e 


Diff: https://reviews.apache.o

Re: Review Request 73329: Correlating Deleted Entities with Lineage

[Ashutosh Mestry via Review Board]

> On May 18, 2021, 6:05 a.m., Sarath Subramanian wrote:
> > webapp/src/main/java/org/apache/atlas/notification/EntityCorrelationManager.java
> > Lines 93 (patched)
> > 
> >
> > this checks for the first entry that is less than the spooledTimestamp. 
> > We should be fetching the entry(ts) having close proximity to 
> > spooledTimestamp.
> > 
> > CACHE:
> > ---
> > QName  | Guid | 
> > ---
> > T1@cl1 | [7:00: guid1], [7:40: guid2]
> > ---
> > 
> > 6:50 - CTAS (T5) FROM T1 (guid1)
> > 7:20 - CTAS (T6) FROM T1 (guid2)

This case gets addressed with current algorithm.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73329/#review223005
---


On May 20, 2021, 4 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73329/
> ---
> 
> (Updated May 20, 2021, 4 a.m.)
> 
> 
> Review request for atlas, Radhika Kundam and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4152
> https://issues.apache.org/jira/browse/ATLAS-4152
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> As part of ATLAS-4204, HS2 notifications send entity-lineage only (provided 
> the poperty is enabled).
> 
> When Spooling enabled the order of messages can potentially change. The 
> notification messages coming from HS2 and HMS may not be in the same order as 
> when they arrived with direct notification.
> 
> Problem: 
> Consider the sequence of arriving messages:
> 
> This is the sequence of messages for Entity 1 (C = create, U = update, D = 
> delete, L?x = Lineage of type 'x')
> No problem: C1, U1, L1x, L1y, D1
> Problem: C1, U1, D1, L1x, L1y
> 
> This implementation attempts to handle ths problem mentioned above. If the 
> above case is not handled, it will end up creating shell entities, since 
> deleted entities are not looked up as part of entity creation.
> 
> **Approach**
> Used bounded stream approach where an incoming stream of messages is bounded 
> with an indicator that it originates from spool. This helps makes localized 
> decisions on the incoming stream of messages.
> 
> High-level approach:
> - Messages when written to the spool are tagged with a timestamp.
> - Deleted entities are maintained in a cache.
> - Lineage-only message are checked if they refer to a deleted entity.
> - If they refer to deleted entity, they are stitched to the one present in 
> the cache only if it falls within the threshold.
> - Using step-climbing approach for locating right entity to stitch lineage to.
> 
> New: _EntityCorrelationsManager_: Uses message timestamp and cached entity 
> qualifiedName-GUID map.
> Modifed: _NotificationHookConsumer_ Uses the new class.
> New: _HiveDDLLineagePreprocess_ Uses entity-correlation to link to deleted 
> entities.
> Modified: _SpoolConfiguration_: Added new configuration to pause message 
> sending after destination is available: 
> _atlas.hook.spool.pause.before.send.sec_.
> In-memory lookup approach changed to persistent loookup.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java ffcec9743 
>   
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java
>  810ba97c9 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
> f7d9668ec 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java 
> 22bd79fdf 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 3d1b3ccf1 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java
>  3264e264c 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  edd8ed931 
>   
> notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
>  2d7d19595 
>   
> notification/src/main/java/org/apache/atlas/notification/spool/Publisher.java 
> 22242c933 
>   
> notification/src/main/java/org/apache/atlas/notification/spool/SpoolConfiguration.java
>  a9a3a78cc 
>   notification/src/main/java/org/apache/atlas/notification/spool/Spooler.java 
> 2cacaaadc 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
>  d7e4959f7 
>   
> notification/src/test/java/org/apache/atlas/notification/spool/AtlasFileSpoolTest.java
>  167efbecc 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  cc727c6ba 
>   
> repository/src/main/java/org/apache/atlas/repository/

Re: Review Request 73329: Correlating Deleted Entities with Lineage

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73329/
---

(Updated May 20, 2021, 4 a.m.)


Review request for atlas, Radhika Kundam and Sarath Subramanian.


Changes
---

Updates include: 
- Addressed review comments.
- Added persistent caching approach. 
- Additional unit tests.


Bugs: ATLAS-4152
https://issues.apache.org/jira/browse/ATLAS-4152


Repository: atlas


Description (updated)
---

**Background**
As part of ATLAS-4204, HS2 notifications send entity-lineage only (provided the 
poperty is enabled).

When Spooling enabled the order of messages can potentially change. The 
notification messages coming from HS2 and HMS may not be in the same order as 
when they arrived with direct notification.

Problem: 
Consider the sequence of arriving messages:

This is the sequence of messages for Entity 1 (C = create, U = update, D = 
delete, L?x = Lineage of type 'x')
No problem: C1, U1, L1x, L1y, D1
Problem: C1, U1, D1, L1x, L1y

This implementation attempts to handle ths problem mentioned above. If the 
above case is not handled, it will end up creating shell entities, since 
deleted entities are not looked up as part of entity creation.

**Approach**
Used bounded stream approach where an incoming stream of messages is bounded 
with an indicator that it originates from spool. This helps makes localized 
decisions on the incoming stream of messages.

High-level approach:
- Messages when written to the spool are tagged with a timestamp.
- Deleted entities are maintained in a cache.
- Lineage-only message are checked if they refer to a deleted entity.
- If they refer to deleted entity, they are stitched to the one present in the 
cache only if it falls within the threshold.
- Using step-climbing approach for locating right entity to stitch lineage to.

New: _EntityCorrelationsManager_: Uses message timestamp and cached entity 
qualifiedName-GUID map.
Modifed: _NotificationHookConsumer_ Uses the new class.
New: _HiveDDLLineagePreprocess_ Uses entity-correlation to link to deleted 
entities.
Modified: _SpoolConfiguration_: Added new configuration to pause message 
sending after destination is available: 
_atlas.hook.spool.pause.before.send.sec_.
In-memory lookup approach changed to persistent loookup.


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/repository/Constants.java ffcec9743 
  
intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java
 810ba97c9 
  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
f7d9668ec 
  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java 
22bd79fdf 
  notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
3d1b3ccf1 
  
notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java
 3264e264c 
  
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
 edd8ed931 
  
notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
 2d7d19595 
  notification/src/main/java/org/apache/atlas/notification/spool/Publisher.java 
22242c933 
  
notification/src/main/java/org/apache/atlas/notification/spool/SpoolConfiguration.java
 a9a3a78cc 
  notification/src/main/java/org/apache/atlas/notification/spool/Spooler.java 
2cacaaadc 
  
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
 d7e4959f7 
  
notification/src/test/java/org/apache/atlas/notification/spool/AtlasFileSpoolTest.java
 167efbecc 
  
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
 cc727c6ba 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/EntityCorrelationStore.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
 0a9470815 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/EntityCorrelationStoreTest.java
 PRE-CREATION 
  
webapp/src/main/java/org/apache/atlas/notification/EntityCorrelationManager.java
 PRE-CREATION 
  
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
 84cc8d813 
  
webapp/src/main/java/org/apache/atlas/notification/preprocessor/EntityPreprocessor.java
 89568e236 
  
webapp/src/main/java/org/apache/atlas/notification/preprocessor/HiveDbDDLPreprocessor.java
 PRE-CREATION 
  
webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java
 86e3384ee 
  
webapp/src/main/java/org/apache/atlas/notification/preprocessor/HiveTableDDLPreprocessor.java
 PRE-CREATION 
  
webapp/src/main/java/org/apache/atlas/notification/preprocessor/PreprocessorContext.java
 608b4a304 
  
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java
 65e8b5001 
  
webapp/src/test/java/org/apache/atlas/notification/Notifi

Re: Review Request 73329: Correlating Deleted Entities with Lineage

[Ashutosh Mestry via Review Board]

> On May 18, 2021, 4:34 a.m., Sarath Subramanian wrote:
> > webapp/src/main/java/org/apache/atlas/notification/EntityCorrelationManager.java
> > Lines 76 (patched)
> > 
> >
> > we don't need to reverse iterate the list, since we are iterating the 
> > entire list:
> > 
> > consider maintaining a minValue - the timestamp in cache should be the 
> > closest to the spooled timestamp: consider using the below method for 
> > better readability
> > 
> > ```
> > public String getGuidForDeletedEntity(String qualifiedName, long 
> > spooledMsgTimestamp) {
> > if (!this.entitiesDeletedByDelete.containsKey(qualifiedName) || 
> > spooledMsgTimestamp <= 0) {
> > return null;
> > }
> > 
> > String ret= null;
> > List> timestampGuidPairs = 
> > this.entitiesDeletedByDelete.get(qualifiedName);
> > long   minTimestamp   = 
> > Long.MAX_VALUE;
> > 
> > for (TypesUtil.Pair tsGuidPair : 
> > timestampGuidPairs) {
> > String entityGuid= tsGuidPair.right;
> > long   entityDeleteTimestamp = tsGuidPair.left;
> > long   timestampDifference   = 
> > Math.abs(entityDeleteTimestamp - spooledMsgTimestamp);
> > 
> >if (timestampDifference < minTimestamp) {
> >minTimestamp = timestampDifference;
> >ret  = entityGuid;
> >}
> > }
> > 
> > return ret;
> > }
> > ```

I modified the logic and added unit test to cover the logic. It is better than 
before and does not use Math.abs.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73329/#review223002
---


On May 18, 2021, 5:39 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73329/
> ---
> 
> (Updated May 18, 2021, 5:39 a.m.)
> 
> 
> Review request for atlas, Radhika Kundam and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4152
> https://issues.apache.org/jira/browse/ATLAS-4152
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> As part of ATLAS-4204, HS2 notifications send entity-lineage only (provided 
> the poperty is enabled).
> 
> When Spooling enabled the order of messages can potentially change. The 
> notification messages coming from HS2 and HMS may not be in the same order as 
> when they arrived with direct notification.
> 
> Problem: 
> Consider the sequence of arriving messages:
> 
> This is the sequence of messages for Entity 1 (C = create, U = update, D = 
> delete, L?x = Lineage of type 'x')
> No problem: C1, U1, L1x, L1y, D1
> Problem: C1, U1, D1, L1x, L1y
> 
> This implementation attempts to handle ths problem mentioned above. If the 
> above case is not handled, it will end up creating shell entities, since 
> deleted entities are not looked up as part of entity creation.
> 
> **Approach**
> Used bounded stream approach where an incoming stream of messages is bounded 
> with an indicator that it originates from spool. This helps makes localized 
> decisions on the incoming stream of messages.
> 
> High-level approach:
> - Messages when written to the spool are tagged with a timestamp.
> - Deleted entities are maintained in a cache.
> - Lineage-only message are checked if they refer to a deleted entity.
> - If they refer to deleted entity, they are stitched to the one present in 
> the cache only if it falls within the threshold.
> - Using step-climbing approach for locating right entity to stitch lineage to.
> 
> New: _EntityCorrelationsManager_: Uses message timestamp and cached entity 
> qualifiedName-GUID map.
> Modifed: _NotificationHookConsumer_ Uses the new class.
> New: _HiveDDLLineagePreprocess_ Uses entity-correlation to link to deleted 
> entities.
> Modified: _SpoolConfiguration_: Added new configuration to pause message 
> sending after destination is available: 
> _atlas.hook.spool.pause.before.send.sec_.
> 
> 
> Diffs
> -
> 
>   
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java
>  810ba97c9 
>   notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
> f7d9668ec 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java 
> 22bd79fdf 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 3d1b3ccf1 
>   
> notification/src/main/java/org/apache

Re: Review Request 73329: Correlating Deleted Entities with Lineage

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73329/
---

(Updated May 18, 2021, 5:39 a.m.)


Review request for atlas, Radhika Kundam and Sarath Subramanian.


Changes
---

Updates include: 
- Addressed review comments.
- Added unit test for _EntityCorrelationManager_.
- Removed need to set the new _atlas.hook.spool.pause.before.send.sec_ property 
for lineage-producing hooks.


Bugs: ATLAS-4152
https://issues.apache.org/jira/browse/ATLAS-4152


Repository: atlas


Description
---

**Background**
As part of ATLAS-4204, HS2 notifications send entity-lineage only (provided the 
poperty is enabled).

When Spooling enabled the order of messages can potentially change. The 
notification messages coming from HS2 and HMS may not be in the same order as 
when they arrived with direct notification.

Problem: 
Consider the sequence of arriving messages:

This is the sequence of messages for Entity 1 (C = create, U = update, D = 
delete, L?x = Lineage of type 'x')
No problem: C1, U1, L1x, L1y, D1
Problem: C1, U1, D1, L1x, L1y

This implementation attempts to handle ths problem mentioned above. If the 
above case is not handled, it will end up creating shell entities, since 
deleted entities are not looked up as part of entity creation.

**Approach**
Used bounded stream approach where an incoming stream of messages is bounded 
with an indicator that it originates from spool. This helps makes localized 
decisions on the incoming stream of messages.

High-level approach:
- Messages when written to the spool are tagged with a timestamp.
- Deleted entities are maintained in a cache.
- Lineage-only message are checked if they refer to a deleted entity.
- If they refer to deleted entity, they are stitched to the one present in the 
cache only if it falls within the threshold.
- Using step-climbing approach for locating right entity to stitch lineage to.

New: _EntityCorrelationsManager_: Uses message timestamp and cached entity 
qualifiedName-GUID map.
Modifed: _NotificationHookConsumer_ Uses the new class.
New: _HiveDDLLineagePreprocess_ Uses entity-correlation to link to deleted 
entities.
Modified: _SpoolConfiguration_: Added new configuration to pause message 
sending after destination is available: 
_atlas.hook.spool.pause.before.send.sec_.


Diffs (updated)
-

  
intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java
 810ba97c9 
  notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144 
  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
f7d9668ec 
  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java 
22bd79fdf 
  notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
3d1b3ccf1 
  
notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java
 3264e264c 
  
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
 edd8ed931 
  
notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
 2d7d19595 
  notification/src/main/java/org/apache/atlas/notification/spool/Publisher.java 
22242c933 
  
notification/src/main/java/org/apache/atlas/notification/spool/SpoolConfiguration.java
 a9a3a78cc 
  notification/src/main/java/org/apache/atlas/notification/spool/Spooler.java 
2cacaaadc 
  
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
 d7e4959f7 
  
notification/src/test/java/org/apache/atlas/notification/spool/AtlasFileSpoolTest.java
 167efbecc 
  
webapp/src/main/java/org/apache/atlas/notification/EntityCorrelationManager.java
 PRE-CREATION 
  
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
 84cc8d813 
  
webapp/src/main/java/org/apache/atlas/notification/preprocessor/EntityPreprocessor.java
 89568e236 
  
webapp/src/main/java/org/apache/atlas/notification/preprocessor/HiveDbDDLPreprocessor.java
 PRE-CREATION 
  
webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java
 86e3384ee 
  
webapp/src/main/java/org/apache/atlas/notification/preprocessor/HiveTableDDLPreprocessor.java
 PRE-CREATION 
  
webapp/src/main/java/org/apache/atlas/notification/preprocessor/PreprocessorContext.java
 608b4a304 
  
webapp/src/test/java/org/apache/atlas/notification/EntityCorrelationManagerTest.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/73329/diff/7/

Changes: https://reviews.apache.org/r/73329/diff/6-7/


Testing
---

**Functional tests**
Manual verification of scenarios.

**Test data**
11:55: Kafka Down!
12:00: create table t01(c01 string);
12:10: create view t06_vw as select * from t01;
12:20: create view t06_vw_1 as select * from t01;
12:30: create view t06_vw_2 as select * from t01;
12:40: create view t06_vw_3 as select * from t01;
12:50: DROP TABLE t

Re: Review Request 73356: Improve Tag Propagation in Entities Involved in Several Lineages

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73356/
---

(Updated May 14, 2021, 4:48 p.m.)


Review request for atlas, Jayendra Parab, Nikhil Bonte, and Sarath Subramanian.


Changes
---

Updates include: Addtional Information section.


Bugs: ATLAS-4285
https://issues.apache.org/jira/browse/ATLAS-4285


Repository: atlas


Description (updated)
---

**Steps to Duplicate**
_Setup_:
- Create a linear lineage using chained CTAS queries via Hive or REST APIs E.g. 
```
create view abcd_vw_1 as select * from abc;
create view abcd_vw_2 as select * from abcd_vw_1;
create view abcd_vw_3 as select * from abcd_vw_2;

.
.
.

```
This should result in a long lineage.
- Create 3 classifications, say C1, C2, C3.
- Assign C1 to abcd_vw_1, C2 to abcd_vw_2 and C3 to abcd_vw_3.

Expected result: abcd_vw_n should have all the 3 classifications propagated.

**Aproach**
Modify _propagateClassification_ to lock all the impacted vertices. Requests 
attempting to modify the same entity will thus be blocked.

**Additional Information**
- Discovered object creation sequence change for _TaskMangement_ and 
_TaskFactoryRegistry_. This causes pending tasks not to be scheduled.
- Removed _ClassificationPropagationTasks.Update_ since it is not used.


Diffs
-

  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
 7984a34a4 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagateTaskFactory.java
 8a81dc952 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java
 f86cbc722 
  repository/src/main/java/org/apache/atlas/tasks/TaskFactoryRegistry.java 
38f2cc9cd 
  repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 2756504da 
  
repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationWithTasksTest.java
 84aefc9ef 


Diff: https://reviews.apache.org/r/73356/diff/1/


Testing
---

**Manual tests**
Used the scenario described above. See the attached file used to generate 
lineage.


Thanks,

Ashutosh Mestry

Review Request 73356: Improve Tag Propagation in Entities Involved in Several Lineages

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73356/
---

Review request for atlas, Jayendra Parab, Nikhil Bonte, and Sarath Subramanian.


Bugs: ATLAS-4285
https://issues.apache.org/jira/browse/ATLAS-4285


Repository: atlas


Description
---

**Steps to Duplicate**
_Setup_:
- Create a linear lineage using chained CTAS queries via Hive or REST APIs E.g. 
```
create view abcd_vw_1 as select * from abc;
create view abcd_vw_2 as select * from abcd_vw_1;
create view abcd_vw_3 as select * from abcd_vw_2;

.
.
.

```
This should result in a long lineage.
- Create 3 classifications, say C1, C2, C3.
- Assign C1 to abcd_vw_1, C2 to abcd_vw_2 and C3 to abcd_vw_3.

Expected result: abcd_vw_n should have all the 3 classifications propagated.

**Aproach**
Modify _propagateClassification_ to lock all the impacted vertices. Requests 
attempting to modify the same entity will thus be blocked.


Diffs
-

  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
 7984a34a4 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagateTaskFactory.java
 8a81dc952 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java
 f86cbc722 
  repository/src/main/java/org/apache/atlas/tasks/TaskFactoryRegistry.java 
38f2cc9cd 
  repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 2756504da 
  
repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationWithTasksTest.java
 84aefc9ef 


Diff: https://reviews.apache.org/r/73356/diff/1/


Testing
---

**Manual tests**
Used the scenario described above. See the attached file used to generate 
lineage.


Thanks,

Ashutosh Mestry

Re: Review Request 73329: Correlating Deleted Entities with Lineage

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73329/
---

(Updated May 11, 2021, 5:22 a.m.)


Review request for atlas, Radhika Kundam and Sarath Subramanian.


Changes
---

Updates include: Simplified approach.


Bugs: ATLAS-4152
https://issues.apache.org/jira/browse/ATLAS-4152


Repository: atlas


Description (updated)
---

**Background**
As part of ATLAS-4204, HS2 notifications send entity-lineage only (provided the 
poperty is enabled).

When Spooling enabled the order of messages can potentially change. The 
notification messages coming from HS2 and HMS may not be in the same order as 
when they arrived with direct notification.

Problem: 
Consider the sequence of arriving messages:

This is the sequence of messages for Entity 1 (C = create, U = update, D = 
delete, L?x = Lineage of type 'x')
No problem: C1, U1, L1x, L1y, D1
Problem: C1, U1, D1, L1x, L1y

This implementation attempts to handle ths problem mentioned above. If the 
above case is not handled, it will end up creating shell entities, since 
deleted entities are not looked up as part of entity creation.

**Approach**
Used bounded stream approach where an incoming stream of messages is bounded 
with an indicator that it originates from spool. This helps makes localized 
decisions on the incoming stream of messages.

High-level approach:
- Messages when written to the spool are tagged with a timestamp.
- Deleted entities are maintained in a cache.
- Lineage-only message are checked if they refer to a deleted entity.
- If they refer to deleted entity, they are stitched to the one present in the 
cache only if it falls within the threshold.
- Using step-climbing approach for locating right entity to stitch lineage to.

New: _EntityCorrelationsManager_: Uses message timestamp and cached entity 
qualifiedName-GUID map.
Modifed: _NotificationHookConsumer_ Uses the new class.
New: _HiveDDLLineagePreprocess_ Uses entity-correlation to link to deleted 
entities.


Diffs (updated)
-

  
intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java
 810ba97c9 
  notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144 
  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
f7d9668ec 
  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java 
22bd79fdf 
  
notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
 c45a1da95 
  
notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java
 3264e264c 
  notification/src/main/java/org/apache/atlas/notification/spool/Spooler.java 
2cacaaadc 
  
webapp/src/main/java/org/apache/atlas/notification/EntityCorrelationManager.java
 PRE-CREATION 
  
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
 84cc8d813 
  
webapp/src/main/java/org/apache/atlas/notification/preprocessor/EntityPreprocessor.java
 89568e236 
  
webapp/src/main/java/org/apache/atlas/notification/preprocessor/HiveDdlLineagePreprocessor.java
 PRE-CREATION 
  
webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java
 e69d63e3a 
  
webapp/src/main/java/org/apache/atlas/notification/preprocessor/PreprocessorContext.java
 608b4a304 


Diff: https://reviews.apache.org/r/73329/diff/5/

Changes: https://reviews.apache.org/r/73329/diff/4-5/


Testing
---

**Functional tests**
Manual verification of scenarios.


Thanks,

Ashutosh Mestry

Re: Review Request 73317: ATLAS-3812: Add schema for Apache Flink

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73317/#review222968
---




addons/models/1000-Hadoop/1110-flink_model.json
Lines 46 (patched)


Please split this model in 2 different entities. One could be application 
and other should be process. This will help with rendering of linage and other 
features.


- Ashutosh Mestry


On April 30, 2021, 7:39 a.m., Josh Yeh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73317/
> ---
> 
> (Updated April 30, 2021, 7:39 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry and Sarath Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-3812: Add schema for Apache Flink
> 
> 
> Diffs
> -
> 
>   addons/models/1000-Hadoop/1110-flink_model.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/73317/diff/1/
> 
> 
> Testing
> ---
> 
> Tested on my CDH cluster with flink example wordcount streaming job.
> 
> 
> File Attachments
> 
> 
> Screen Shot 2021-04-30 at 12.38.27 AM.png
>   
> https://reviews.apache.org/media/uploaded/files/2021/04/30/2ff59445-7237-4be1-a1a0-b6db70bc7827__Screen_Shot_2021-04-30_at_12.38.27_AM.png
> 
> 
> Thanks,
> 
> Josh Yeh
> 
>

Re: Review Request 73341: ATLAS-4278 - Deferred Actions : Deleting a tag after disassociating it from a table which propagated fails

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73341/#review222967
---



Can you please update the jira/review with the error you saw in the log?

- Ashutosh Mestry


On May 9, 2021, 1:09 p.m., Jayendra Parab wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73341/
> ---
> 
> (Updated May 9, 2021, 1:09 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Nikhil Bonte, Nixon Rodrigues, 
> Pinal Shah, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4278
> https://issues.apache.org/jira/browse/ATLAS-4278
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> When deferred action is enabled, even after dissociating the tag from all 
> entities the classification vertex created while associating the tag still 
> exists.
> The error given "tag1 has references" is because of the classification vertex 
> created is still present in graphdb.
> 
> When deferred action is disable, at the time of dissociating the tag, the 
> classification vertex is deleted from DeleteHandlerV1.deleteEdgeReference.
> But when the deferred action is enabled, the classification vertex isn't 
> deleted because when DeleteHandlerV1.deleteEdgeReference is invoked, that 
> instance the classification vertex has references to the propagated entities. 
> In the background thread, which executes the dissociating of tags from 
> propagated entities, the classification vertex isn't deleted.
> To resolve this issue, the classification vertex is deleted once the tag is 
> disassoociated from propagated entities.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  02ba5ef8d 
> 
> 
> Diff: https://reviews.apache.org/r/73341/diff/1/
> 
> 
> Testing
> ---
> 
> With deferred action enabled:
> Test 1:
> - Associated a tag to an entity table1 which has lineage like table1 -> 
> process1 -> table2
> - Waited for tag propagation to complete
> - Once tag is propagation is complete, deleted the tag1 from table1
> - Once the tag is removed from all the entities in the lineage above, deleted 
> the tag
> - The tag is deleted properly
> 
> Test 2: 
> - Associated a tag to an entity table1 which has lineage like table1 -> 
> process1 -> table2
> - Waited for tag propagation to complete
> - Once tag is propagation is complete, deleted the tag1 from table1
> - Associated the same tag to an entity table3 which has lineage like table3 
> -> process1 -> table4
> - Once the tag is removed from all the entities in the lineage above, deleted 
> the tag
> - The tag is deleted properly
> 
> 
> Thanks,
> 
> Jayendra Parab
> 
>

Re: Review Request 73341: ATLAS-4278 - Deferred Actions : Deleting a tag after disassociating it from a table which propagated fails

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73341/#review222966
---


Ship it!




Ship It!

- Ashutosh Mestry


On May 9, 2021, 1:09 p.m., Jayendra Parab wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73341/
> ---
> 
> (Updated May 9, 2021, 1:09 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Nikhil Bonte, Nixon Rodrigues, 
> Pinal Shah, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4278
> https://issues.apache.org/jira/browse/ATLAS-4278
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> When deferred action is enabled, even after dissociating the tag from all 
> entities the classification vertex created while associating the tag still 
> exists.
> The error given "tag1 has references" is because of the classification vertex 
> created is still present in graphdb.
> 
> When deferred action is disable, at the time of dissociating the tag, the 
> classification vertex is deleted from DeleteHandlerV1.deleteEdgeReference.
> But when the deferred action is enabled, the classification vertex isn't 
> deleted because when DeleteHandlerV1.deleteEdgeReference is invoked, that 
> instance the classification vertex has references to the propagated entities. 
> In the background thread, which executes the dissociating of tags from 
> propagated entities, the classification vertex isn't deleted.
> To resolve this issue, the classification vertex is deleted once the tag is 
> disassoociated from propagated entities.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  02ba5ef8d 
> 
> 
> Diff: https://reviews.apache.org/r/73341/diff/1/
> 
> 
> Testing
> ---
> 
> With deferred action enabled:
> Test 1:
> - Associated a tag to an entity table1 which has lineage like table1 -> 
> process1 -> table2
> - Waited for tag propagation to complete
> - Once tag is propagation is complete, deleted the tag1 from table1
> - Once the tag is removed from all the entities in the lineage above, deleted 
> the tag
> - The tag is deleted properly
> 
> Test 2: 
> - Associated a tag to an entity table1 which has lineage like table1 -> 
> process1 -> table2
> - Waited for tag propagation to complete
> - Once tag is propagation is complete, deleted the tag1 from table1
> - Associated the same tag to an entity table3 which has lineage like table3 
> -> process1 -> table4
> - Once the tag is removed from all the entities in the lineage above, deleted 
> the tag
> - The tag is deleted properly
> 
> 
> Thanks,
> 
> Jayendra Parab
> 
>