Re: Review Request 74978: ATLAS-4861: Export/Import: add flag to skip updating replicated attributes

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74978/#review226444
---


Ship it!




Ship It!

- Ashutosh Mestry


On May 14, 2024, 4:52 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74978/
> ---
> 
> (Updated May 14, 2024, 4:52 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
> Nikhil Bonte, and Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-4861
> https://issues.apache.org/jira/browse/ATLAS-4861
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Replicated Attributes:**
>  
> Users want knowledge of how the entities have landed into Atlas instance, if 
> they were created via hook ingestion or whether they were imported from 
> another Atlas instance.
> 
> This is addressed by 2 new attributes that are now part of Referenceable 
> entity type viz. replicatedFrom and replicatedTo.
>  
> **Current Behaviour:**
> 
> At the end of export/import, the entities which are exported/imported are 
> updated with the replicationTo/replicationFrom  respectively with the value 
> of AtlasServer object 
> Hence updating each and every entity takes much time.
> 
> **Improvement:**
> 
> Add flag to skip such updates.
> 
> **Example request for Export:**
> 
> curl -v -X POST -u admin:admin -H "Content-Type: application/json"  
> "http://<>/api/atlas/admin/export" -d 
> '{"itemsToExport":[{"typeName":"hive_db","uniqueAttributes":{"qualifiedName":"bsadp@hpchdd2"}}],"options":{"fetchType":"full","replicatedTo":"cm",**"skipUpdateReplicationAttr":"true"**},"fetchTypeOptionValue":"full","skipLineageOptionValue":false,"changeTokenFromOptions":0}}'
>   > test_noupdate.zip
> 
> **Example import-options.json for Import:**
> 
> {
> "options": {
> "replicatedFrom":"cm",
> "skipUpdateReplicationAttr":true
> }
> }
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportRequest.java 
> 878b1d8bc 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
> cbc1aa938 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/AuditsWriter.java 
> c4de0ed27 
> 
> 
> Diff: https://reviews.apache.org/r/74978/diff/1/
> 
> 
> Testing
> ---
> 
> Tests(~3000 entity)  w/o updateswith updates
> Export1 11s  25s
> Export2 12s  27s
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 74750: ATLAS-4184: updated build/python scripts to replace references to python with python3

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74750/#review225994
---


Ship it!




Ship It!

- Ashutosh Mestry


On Nov. 22, 2023, 4:32 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74750/
> ---
> 
> (Updated Nov. 22, 2023, 4:32 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Nikhil Bonte, Nixon Rodrigues, 
> Pinal Shah, Radhika Kundam, Sarath Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-4184
> https://issues.apache.org/jira/browse/ATLAS-4184
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> - updated distro/pom.xml to replace reference to python with python3
> - updated Docker setup to not install python2
> 
> 
> Diffs
> -
> 
>   dev-support/atlas-docker/Dockerfile.atlas-base 4aa225c16 
>   distro/pom.xml 9df238eb3 
>   distro/src/bin/atlas_admin.py 795a5716f 
>   distro/src/bin/atlas_client_cmdline.py f05a3c8f9 
>   distro/src/bin/atlas_config.py 31e6fd043 
>   distro/src/bin/atlas_kafka_setup.py c13030e88 
>   distro/src/bin/atlas_kafka_setup_hook.py cd8c0c3ca 
>   distro/src/bin/atlas_start.py ffbe16271 
>   distro/src/bin/atlas_stop.py b9cd63689 
>   distro/src/bin/atlas_update_simple_auth_json.py 11c249d9b 
>   distro/src/bin/cputil.py 8e99f95bc 
>   distro/src/bin/quick_start.py 78517f865 
>   distro/src/bin/quick_start_v1.py 1c5775a68 
>   distro/src/test/mock/setup.py 3639df4a1 
>   distro/src/test/python/scripts/TestMetadata.py b813f927f 
>   distro/src/test/python/unitTests.py ec1a78cb9 
>   tools/atlas-index-repair/src/main/resources/repair_index.py 0333dadca 
>   tools/atlas-migration-exporter/atlas_migration_export.py e1b12f609 
> 
> 
> Diff: https://reviews.apache.org/r/74750/diff/1/
> 
> 
> Testing
> ---
> 
> - pre-commit tests run: 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1516/
> - successfully built and installed Atlas in docker containers having no 
> Python2, using scripts under dev-support/atlas-docker
> - verified quick_start.py
> - verified Hive and HBase hooks reported entities from respective components
> - verified Kafka bridge imported Kafka topics successfully
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Review Request 74587: ATLAS-4795: docker setup updated to support Apple M1

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74587/#review225736
---


Ship it!




Ship It!

- Ashutosh Mestry


On Sept. 9, 2023, 12:47 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74587/
> ---
> 
> (Updated Sept. 9, 2023, 12:47 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Nikhil Bonte, Nixon Rodrigues, 
> Radhika Kundam, Sarath Subramanian, Sharmadha Sainath, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-4795
> https://issues.apache.org/jira/browse/ATLAS-4795
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> - updated docker setup scripts to use TARGETARCH
> - added support for building and runing with JDK8 or JDK11
> 
> 
> Diffs
> -
> 
>   dev-support/atlas-docker/.env 440be2a46 
>   dev-support/atlas-docker/Dockerfile 7bd79651f 
>   dev-support/atlas-docker/Dockerfile.atlas 888779133 
>   dev-support/atlas-docker/Dockerfile.atlas-base e68daa69d 
>   dev-support/atlas-docker/Dockerfile.atlas-build 4b381c1d9 
>   dev-support/atlas-docker/docker-compose.atlas-base.yml 349c306d6 
>   dev-support/atlas-docker/docker-compose.atlas-build.yml d2ec7bda5 
>   dev-support/atlas-docker/docker-compose.atlas.yml aeb706e59 
>   dev-support/atlas-docker/scripts/atlas-build.sh 592a055f7 
> 
> 
> Diff: https://reviews.apache.org/r/74587/diff/1/
> 
> 
> Testing
> ---
> 
> - verified that Atlas builds and runs successfully via docker setup scripts 
> in MacBook Pro with Apple M1
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Review Request 74586: ATLAS-4794: fixed initialization failure in HBase, Hive hooks

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74586/#review225732
---


Ship it!




Ship It!

- Ashutosh Mestry


On Sept. 8, 2023, 9:16 p.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74586/
> ---
> 
> (Updated Sept. 8, 2023, 9:16 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Nikhil Bonte, Nixon Rodrigues, 
> Radhika Kundam, Sarath Subramanian, Sharmadha Sainath, Sidharth Mishra, and 
> Umesh Padashetty.
> 
> 
> Bugs: ATLAS-4794
> https://issues.apache.org/jira/browse/ATLAS-4794
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> - updated HBase hook packaging to include log4j-api and log4j-core libraries
> - updated Hive hook packaging to include log4j-api and log4j-core libraries
> 
> 
> Diffs
> -
> 
>   addons/hbase-bridge/pom.xml d78abbb37 
>   addons/hive-bridge/pom.xml 3464b65f0 
> 
> 
> Diff: https://reviews.apache.org/r/74586/diff/2/
> 
> 
> Testing
> ---
> 
> - verified that HBase and Hive hooks initialize and work successfully with 
> the fix
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Review Request 74452: ATLAS-4754 : Download Search with Basic Search gives java.io.FileNotFoundException

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74452/#review225548
---


Ship it!




Ship It!

- Ashutosh Mestry


On May 23, 2023, 7:56 a.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74452/
> ---
> 
> (Updated May 23, 2023, 7:56 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Pinal Shah, and Sheetal Shah.
> 
> 
> Bugs: ATLAS-4754
> https://issues.apache.org/jira/browse/ATLAS-4754
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Default value for the download directory will be read from system property 
> "user.dir" instead of property "atlas.home"
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java
>  fd90fd440 
> 
> 
> Diff: https://reviews.apache.org/r/74452/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Mandar Ambawane
> 
>

Re: Review Request 74289: ATLAS-4728 :- Upgrade tinkerpop to 3.5.2

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74289/#review225146
---


Ship it!




Ship It!

- Ashutosh Mestry


On Feb. 1, 2023, 11:51 a.m., Vinayak Marraiya wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74289/
> ---
> 
> (Updated Feb. 1, 2023, 11:51 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab and Pinal Shah.
> 
> 
> Bugs: ATLAS-4728
> https://issues.apache.org/jira/browse/ATLAS-4728
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently Atlas is using tinker pop version 3.5.2 upgrading it to 3.5.4
> 
> 
> Diffs
> -
> 
>   pom.xml 44e827ac1 
> 
> 
> Diff: https://reviews.apache.org/r/74289/diff/1/
> 
> 
> Testing
> ---
> 
> 1)mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
> 2)Quick started was passed successfully
> 3)Precommit - 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1318/console
> 
> 
> Thanks,
> 
> Vinayak Marraiya
> 
>

Re: Review Request 74269: ATLAS-4720 :- Atlas - Upgrade Netty to 4.1.86.Final

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74269/#review225056
---


Ship it!




Ship It!

- Ashutosh Mestry


On Jan. 3, 2023, 6:55 p.m., Vinayak Marraiya wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74269/
> ---
> 
> (Updated Jan. 3, 2023, 6:55 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab and Pinal Shah.
> 
> 
> Bugs: ATLAS-4720
> https://issues.apache.org/jira/browse/ATLAS-4720
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently atlas is using netty version 4.1.78.Final
> 
> 
> Diffs
> -
> 
>   pom.xml 125076def 
> 
> 
> Diff: https://reviews.apache.org/r/74269/diff/1/
> 
> 
> Testing
> ---
> 
> 1)mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
>   Atlas server runnning
> 2)Quick started was passed successfully
> 
> Pre-commit :- 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1299/console
> 
> 
> Thanks,
> 
> Vinayak Marraiya
> 
>

Re: Review Request 74241: ATLAS-4702: updated version in pom.xml from 2.3.0 to 2.4.0-SNAPSHOT

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74241/#review225033
---


Ship it!




Ship It!

- Ashutosh Mestry


On Dec. 7, 2022, 9:23 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74241/
> ---
> 
> (Updated Dec. 7, 2022, 9:23 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-4702
> https://issues.apache.org/jira/browse/ATLAS-4702
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> updated version in pom.xml files from 2.3.0 to 2.4.0-SNAPSHOT
> 
> 
> Diffs
> -
> 
>   addons/falcon-bridge-shim/pom.xml 490d812a0 
>   addons/falcon-bridge/pom.xml 7072ab06f 
>   addons/hbase-bridge-shim/pom.xml a33f78463 
>   addons/hbase-bridge/pom.xml ee4ed0471 
>   addons/hbase-testing-util/pom.xml 560e4f27e 
>   addons/hdfs-model/pom.xml 43362fadf 
>   addons/hive-bridge-shim/pom.xml 6c2c04aac 
>   addons/hive-bridge/pom.xml 4911f98a0 
>   addons/impala-bridge-shim/pom.xml 7c2cae7f5 
>   addons/impala-bridge/pom.xml b7a08f2ec 
>   addons/impala-hook-api/pom.xml 47c0b7003 
>   addons/kafka-bridge/pom.xml fa54a56ae 
>   addons/sqoop-bridge-shim/pom.xml 3e1e7c8ba 
>   addons/sqoop-bridge/pom.xml 45a27e2f2 
>   addons/storm-bridge-shim/pom.xml 098873c3f 
>   addons/storm-bridge/pom.xml 01450e795 
>   atlas-examples/pom.xml e5c336411 
>   atlas-examples/sample-app/pom.xml 7375e6433 
>   authorization/pom.xml e580df8f3 
>   build-tools/pom.xml 12f1c87eb 
>   client/client-v1/pom.xml 4d119ed10 
>   client/client-v2/pom.xml 620c95b24 
>   client/common/pom.xml f5099b110 
>   client/pom.xml 3973f1e67 
>   common/pom.xml 25e5a7b87 
>   dashboardv2/pom.xml 42647005c 
>   dashboardv3/pom.xml bb31b3f72 
>   distro/pom.xml ae13b6574 
>   docs/pom.xml 019c3fbea 
>   graphdb/api/pom.xml 8f0c2dae5 
>   graphdb/common/pom.xml dab86a314 
>   graphdb/graphdb-impls/pom.xml eb53375a0 
>   graphdb/janus-hbase2/pom.xml 73e88d9df 
>   graphdb/janus/pom.xml 8ef052639 
>   graphdb/pom.xml dceecf47c 
>   intg/pom.xml 32acf0286 
>   notification/pom.xml 67bae8958 
>   plugin-classloader/pom.xml 8dff9c555 
>   pom.xml 1217ccc34 
>   repository/pom.xml 974c917ad 
>   server-api/pom.xml 9e018874f 
>   test-tools/pom.xml 838b970c9 
>   tools/atlas-index-repair/pom.xml 732d2be3d 
>   tools/classification-updater/pom.xml 9399420d2 
>   webapp/pom.xml 25214a24d 
> 
> 
> Diff: https://reviews.apache.org/r/74241/diff/1/
> 
> 
> Testing
> ---
> 
> verified that build completes successfully
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Review Request 74240: ATLAS-4700: updated documentation for 2.3.0 release

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74240/#review224972
---


Ship it!




Ship It!

- Ashutosh Mestry


On Dec. 7, 2022, 8:08 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74240/
> ---
> 
> (Updated Dec. 7, 2022, 8:08 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Mandar Ambawane, 
> Nikhil Bonte, Nixon Rodrigues, Pinal Shah, Prasad Pawar, Radhika Kundam, 
> Sarath Subramanian, Sharmadha Sainath, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-4700
> https://issues.apache.org/jira/browse/ATLAS-4700
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> updated documentation for 2.3.0 release
> 
> 
> Diffs
> -
> 
>   docs/docz-lib/config/menu.js 37acf46d4 
>   docs/docz-lib/config/versions.js 3ef55f681 
>   docs/src/documents/Downloads/Downloads.md e486a1d0b 
>   docs/src/documents/Overview.md 8614106ab 
>   docs/src/documents/Whats-New/WhatsNew-2.3.md PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/74240/diff/1/
> 
> 
> Testing
> ---
> 
> verified the updated site with 'mvn site:run" in docs directory
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: [VOTE] Release Apache Atlas 2.3.0 rc1

[Ashutosh Mestry]

Thank you Madhan for the release candidate.

+1 for Apache Atlas 2.3.0 release candidate #1

~ ashutosh

On 2022/12/05 18:24:36 Jayendra Parab wrote:
> Thank you Madhan for the release candidate.
> 
> +1 for Apache Atlas 2.3.0 release candidate #1
> 
> Validated the following:
> - Able to download source, signature, md5 and sha512 files and verified
> checksum hash
> - Validated signature of source from release manager (Madhan Neethiraj <
> mad...@apache.org>)
> - Built the source successfully using embedded-hbase-solr profile
> - Started Atlas and ran quickstart script to preload sample types and
> entities
> - Validated relation search, basic and advanced search
> 
> Thanks and regards,
> Jayendra.
> 
> On Mon, Dec 5, 2022 at 1:02 PM Nixon Rodrigues 
> wrote:
> 
> > +1 for the Atlas 2.3.0 RC#1 for release
> >
> > The RC bits build looks good.
> >
> > Thanks Madhan for putting this together for release voting.
> >
> > Thanks and Regards
> > Nixon Rodrigues
> >
> > On Sun, 4 Dec 2022 at 20:58, Nikhil Bonte  wrote:
> >
> > > +1 for the Apache Atlas 2.3.0 release candidate #1.
> > >
> > > Performed Following:
> > > - Able to download source, signature, md5 and sha512 files and verify the
> > > checksum hash
> > >
> > > - Build the source successfully using embedded-hbase-solr profile
> > >
> > >
> > > Thanks
> > > Nikhil P. Bonte
> > >
> > > On Sun, Dec 4, 2022 at 4:17 AM Ramesh Mani  wrote:
> > >
> > > > +1 for the Apache Atlas 2.3.0 release candidate #1.
> > > >
> > > >   Verified Build in CentOS7
> > > >  /apache-maven-3.8.6/bin/mvn clean -DskipTests
> > > > -Dmaven.wagon.http.ssl.insecure=true
> > -Dmaven.wagon.http.ssl.allowall=true
> > > > -Dmaven.wagon.http.ssl.ignore.validity.dates=true install
> > > >  Verified Signature
> > > >  gpg --verify apache-atlas-2.3.0-sources.tar.gz.asc
> > > > gpg: Signature made Mon 28 Nov 2022 09:14:22 PM UTC using
> > RSA
> > > > key ID B01179D4
> > > > gpg: Good signature from "Madhan Neethiraj <
> > > mad...@apache.org
> > > > >"
> > > > gpg: WARNING: This key is not certified with a trusted
> > > > signature!
> > > >  gpg:  There is no indication that the signature
> > > > belongs to the owner.
> > > > Primary key fingerprint: 0524 DD1F 7940 6654 6D61  0744 FED4 67D3 B011
> > > 79D4
> > > >
> > > >  Verified  SHA256 and SHA512
> > > >  openssl dgst -sha256 apache-atlas-2.3.0-sources.tar.gz
> > > > SHA256(apache-atlas-2.3.0-sources.tar.gz)=
> > > > d56b89edd0499f109488b1d8394ec3f4f23609b9ea29a5099c473e84ec7e91c0
> > > >
> > > > openssl dgst -sha512 apache-atlas-2.3.0-sources.tar.gz
> > > >SHA512(apache-atlas-2.3.0-sources.tar.gz)=
> > > >
> > > >
> > >
> > b24c9785f601b4644090ef0b4c16974fc572100b61f04ef2d6c68fdaa45293e902483f3574342a0e70c54f757c509d3d7ec6a85e494599b1018c65d8126d3480
> > > >
> > > > Thanks,
> > > > Ramesh
> > > >
> > > >
> > > > [root@rmani-7216-1 tmp]# gpg --verify
> > > apache-atlas-2.3.0-sources.tar.gz.as
> > > >
> > > > On Mon, Nov 28, 2022 at 1:30 PM Madhan Neethiraj 
> > > > wrote:
> > > >
> > > > > Atlas team,
> > > > >
> > > > >
> > > > >
> > > > > Apache Atlas 2.3.0 release candidate #1 is now available for a vote
> > > > within
> > > > > dev community. Links to the release artifacts are given below. Please
> > > > > review and vote.
> > > > >
> > > > >
> > > > >
> > > > > The vote will be open for at least 72 hours or until necessary votes
> > > are
> > > > > reached.
> > > > >
> > > > >   [ ] +1 approve
> > > > >
> > > > >   [ ] +0 no opinion
> > > > >
> > > > >   [ ] -1 disapprove (and reason why)
> > > > >
> > > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Madhan
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > List of issues addressed in this release:
> > > > >
> > > >
> > >
> > https://issues.apache.org/jira/issues/?jql=project%20%3D%20Atlas%20AND%20fixVersion%20%3D%202.3.0%20ORDER%20BY%20key%20DESC
> > > > >
> > > > >
> > > > >
> > > > > Git tag for the release:
> > > > > https://github.com/apache/atlas/tree/release-2.3.0-rc1
> > > > >
> > > > >
> > > > >
> > > > > Sources for the release:
> > > > >
> > > >
> > >
> > https://dist.apache.org/repos/dist/dev/atlas/2.3.0-rc1/apache-atlas-2.3.0-sources.tar.gz
> > > > >
> > > > >
> > > > >
> > > > > Source release verification:
> > > > >
> > > > >   PGP Signature:
> > > > >
> > > >
> > >
> > https://dist.apache.org/repos/dist/dev/atlas/2.3.0-rc1/apache-atlas-2.3.0-sources.tar.gz.asc
> > > > >
> > > > >   MD5 Hash:
> > > > >
> > > >
> > >
> > https://dist.apache.org/repos/dist/dev/atlas/2.3.0-rc1/apache-atlas-2.3.0-sources.tar.gz.md5
> > > > >
> > > > >
> > > > >
> > > > > Keys to verify the signature of the release artifacts are available
> > at:
> > > > > https://dist.apache.org/repos/dist/dev/atlas/KEYS
> > > > >
> > > > >
> > > >
> > >
> >
> 

Re: Review Request 74212: ATLAS-4714: exclude downloaded/generated files from source package

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74212/#review224912
---


Ship it!




Ship It!

- Ashutosh Mestry


On Nov. 22, 2022, 10:12 p.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74212/
> ---
> 
> (Updated Nov. 22, 2022, 10:12 p.m.)
> 
> 
> Review request for atlas, Ayub Pathan, Ashutosh Mestry, Jayendra Parab, 
> Mandar Ambawane, Pinal Shah, Prasad Pawar, Radhika Kundam, Sarath 
> Subramanian, Sharmadha Sainath, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-4714
> https://issues.apache.org/jira/browse/ATLAS-4714
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Following downloaded/generated files are excluded from source package:
> - distro/elasticsearch/**
> - distro/zk/**
> - dashboardv2/node_modules/**
> - dashboardv3/node_modules/**
> - dev-support/atlas-docker/downloads/*
> 
> 
> Diffs
> -
> 
>   distro/src/main/assemblies/src-package.xml b53487155 
> 
> 
> Diff: https://reviews.apache.org/r/74212/diff/1/
> 
> 
> Testing
> ---
> 
> - Built and deployed Atlas using the generated source package in docker
> - Sanity testing of Atlas UI, Hive/HBase hooks, import utilities for 
> Hive/Hbase/Kafka/Glossaries
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Review Request 74197: ATLAS-4707: updated loader-utils version from 1.4.0 to 1.4.1

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74197/#review224861
---


Ship it!




Ship It!

- Ashutosh Mestry


On Nov. 8, 2022, 6:12 p.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74197/
> ---
> 
> (Updated Nov. 8, 2022, 6:12 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nikhil Bonte, 
> Nixon Rodrigues, Pinal Shah, Prasad Pawar, Radhika Kundam, Sarath 
> Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-4707
> https://issues.apache.org/jira/browse/ATLAS-4707
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Patch from following pull requests:
> - https://github.com/apache/atlas/pull/205
> - https://github.com/apache/atlas/pull/206
> 
> 
> Diffs
> -
> 
>   dashboardv2/public/js/external_lib/atlas-lineage/package-lock.json 
> 890350198 
>   dashboardv3/public/js/external_lib/atlas-lineage/package-lock.json 
> 890350198 
> 
> 
> Diff: https://reviews.apache.org/r/74197/diff/1/
> 
> 
> Testing
> ---
> 
> - sanity testing of Atlas UI
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Review Request 74158: ATLAS-4685: Advanced Search : Not Equals operator doesn't work for String type

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74158/#review224783
---


Ship it!




Ship It!

- Ashutosh Mestry


On Oct. 11, 2022, 3:38 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74158/
> ---
> 
> (Updated Oct. 11, 2022, 3:38 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Deep Singh, and Jayendra Parab.
> 
> 
> Bugs: ATLAS-4685
> https://issues.apache.org/jira/browse/ATLAS-4685
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Issue:** If use "!=" operator in where clause, it is not working as 
> expected and it still returns the row which has the value 'authors'
> 
> **Example:** from hive_table where name != 'authors' select name, owner
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> bc3930250 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 1d3e245c5 
>   
> repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
> 4002a32b1 
>   repository/src/test/java/org/apache/atlas/query/TraversalComposerTest.java 
> 359ac7045 
> 
> 
> Diff: https://reviews.apache.org/r/74158/diff/1/
> 
> 
> Testing
> ---
> 
> Added unit testcases
> Precommit : 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1251/
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 74150: ATLAS-4652: fix to address potential NPE in AtlasAttributeDef.isSoftReferenced()

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74150/#review224755
---


Ship it!




Ship It!

- Ashutosh Mestry


On Oct. 2, 2022, 2:55 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74150/
> ---
> 
> (Updated Oct. 2, 2022, 2:55 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nikhil Bonte, 
> Nixon Rodrigues, Pinal Shah, Radhika Kundam, Sarath Subramanian, Sharmadha 
> Sainath, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-4652
> https://issues.apache.org/jira/browse/ATLAS-4652
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> updated to handle explicit null value for AtlasAttributeDef.options
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java 
> a621fb0b5 
>   intg/src/main/java/org/apache/atlas/utils/AtlasStringUtil.java PRE-CREATION 
>   intg/src/test/java/org/apache/atlas/utils/AtlasStringUtilTest.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
>  1d29bf833 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
>  8e17fd410 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java
>  d6f23d6e2 
> 
> 
> Diff: https://reviews.apache.org/r/74150/diff/2/
> 
> 
> Testing
> ---
> 
> pre-commit tests run: 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1239/
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Review Request 74093: ATLAS-4662 : Upgrade POI to 5.2.2

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74093/#review224638
---


Ship it!




Ship It!

- Ashutosh Mestry


On Aug. 23, 2022, 7:43 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74093/
> ---
> 
> (Updated Aug. 23, 2022, 7:43 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab and Pinal Shah.
> 
> 
> Bugs: ATLAS-4662
> https://issues.apache.org/jira/browse/ATLAS-4662
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> POI and POI-ooxml version is 4.1.1
> 
> 
> Diffs
> -
> 
>   pom.xml 548382446 
> 
> 
> Diff: https://reviews.apache.org/r/74093/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean install
> apache server running fine
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1198/console
> 
> 
> Thanks,
> 
> chaitali
> 
>

Re: Review Request 74079: ATLAS-4653 : Upgrade Hadoop to 3.3.3

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74079/#review224596
---


Ship it!




Ship It!

- Ashutosh Mestry


On Aug. 2, 2022, 11:51 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74079/
> ---
> 
> (Updated Aug. 2, 2022, 11:51 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab and Pinal Shah.
> 
> 
> Bugs: ATLAS-4653
> https://issues.apache.org/jira/browse/ATLAS-4653
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> currently Hadoop is using 3.3.0 and 3.0.3
> This patch also upgrades jackson-databind to 2.13.3 and commons-beanutils to 
> 1.9.4
> 
> 
> Diffs
> -
> 
>   addons/hbase-bridge/pom.xml a6ed51421 
>   addons/hbase-testing-util/pom.xml 982e9c85e 
>   addons/hive-bridge/pom.xml 8c3636e5d 
>   addons/sqoop-bridge/pom.xml 021e93f56 
>   addons/storm-bridge/pom.xml e8106afce 
>   notification/pom.xml aaf11c7a4 
>   pom.xml 1e32e45f3 
>   test-tools/pom.xml 991fa14c9 
>   webapp/pom.xml 37eeaf4bf 
> 
> 
> Diff: https://reviews.apache.org/r/74079/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean -DskipTests package -Pdist,embedded-hbase-solr
> mvn clean install -Drat.skip
> atlas server running successfully
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1175/consoleFull
> 
> 
> Thanks,
> 
> chaitali
> 
>

Re: Review Request 74077: Refactor Atlas webapp module to remove Kafka core dependency

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74077/#review224593
---


Ship it!




Ship It!

- Ashutosh Mestry


On Aug. 2, 2022, 1:21 p.m., Patrik Márton wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74077/
> ---
> 
> (Updated Aug. 2, 2022, 1:21 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Nixon 
> Rodrigues.
> 
> 
> Bugs: ATLAS-4619
> https://issues.apache.org/jira/browse/ATLAS-4619
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Goal is to break the strong coupling between Atlas components and Kafka. 
> These dependencies include using server side libraries of Kafka (this couples 
> the Scala version and other non-public interfaces of Kafka). Any code using 
> server side libraries of Kafka should be refactored.
> The webapp module uses ShutdownAbleThread from the core kafka library. With 
> this commit, it is changed to Thread, as the try-catch-finally block inside 
> the run() method should solve the issue with closing the consumer.
> 
> 
> Diffs
> -
> 
>   
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  49c504f9f 
>   
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java
>  fdfc2560d 
> 
> 
> Diff: https://reviews.apache.org/r/74077/diff/2/
> 
> 
> Testing
> ---
> 
> - Unit tests are green
> - Atlas server is healthy on manually provisoned cluster
> - Tested the functionality with Hive Hook, no ConcurrentModificationException 
> was present in the logs after stopping Atlas server.
> 
> 
> Thanks,
> 
> Patrik Márton
> 
>

Re: Review Request 74074: ATLAS-4650 : Upgrade Apache Commons BeanUtils to 1.9.4

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74074/#review224591
---


Ship it!




Ship It!

- Ashutosh Mestry


On July 28, 2022, 11:06 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74074/
> ---
> 
> (Updated July 28, 2022, 11:06 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab and Pinal Shah.
> 
> 
> Bugs: ATLAS-4650
> https://issues.apache.org/jira/browse/ATLAS-4650
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently Commons BeanUtils version is 1.9.2,1.9.3,1.8.3
> 
> 
> Diffs
> -
> 
>   addons/hbase-bridge/pom.xml a6ed51421 
>   addons/hbase-testing-util/pom.xml 982e9c85e 
>   pom.xml 1e32e45f3 
> 
> 
> Diff: https://reviews.apache.org/r/74074/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean -DskipTests package -Pdist,embedded-hbase-solr
> mvn clean install -Drat.skip
> atlas server running successfully
> 
> 
> Thanks,
> 
> chaitali
> 
>

Re: Review Request 74061: ATLAS-4637: Import/Export Feature for MetricsStat

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74061/#review224586
---



Can you please add unit test for this?


webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
Line 22 (original), 28 (patched)
<https://reviews.apache.org/r/74061/#comment313380>

Can you please restore individual imports.



webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
Line 95 (original), 104 (patched)
<https://reviews.apache.org/r/74061/#comment313381>

Please restore individual imports.


- Ashutosh Mestry


On July 26, 2022, 4:15 p.m., Beakal Lemeneh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74061/
> ---
> 
> (Updated July 26, 2022, 4:15 p.m.)
> 
> 
> Review request for atlas, Disha Talreja, Farhan Khan, Jayendra Parab, Prasad 
> Pawar, Radhika Kundam, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-4637
> https://issues.apache.org/jira/browse/ATLAS-4637
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Add new attribute - importLabel - to MetricsStat. It contains the 
> name/identity of the user who is importing MetricsStat file from one machine 
> to another machine. For some Machine A, importLabel is set to "self" for the 
> already existing MetricsStat that collects the metrics info in Machine A, and 
> has not been imported from another machine.
> 
> 2. Update metricsId to contain both importLabel and collectionTime to 
> continue holding its property of being a unique attribute.
> 
> 3. Update MetricsService to implement REST APIs in AdminResource to realize 
> api calls of:
>   1. Post MetricsStat with a specific importLabel obtained from a another 
> machine.
>   2. Get MetricsStat with a specific importLabel found in some machine 
> with a boolean flag that returns metrics with specific details, or with 
> minimal information.
>   3. Delete all existing MetricsStats entities in a given machine.
>   4. Delete all existing MetricsStats entities in a given machine with a 
> specific importLabel.
>   5. Get all importLabels.
> 
> 6. Update MetricsService to include helper methods of:
>   1. Get a persisted metricsStat by importLabel.
>   2. Get a persisted metricsStat by collectionTime and importLabel.
> 
> 7. UI support enabled.
> 
> 
> Diffs
> -
> 
>   
> addons/models/-Area0/patches/007-base_model_add_importLabel_attribute.json
>  PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetricsStat.java 
> 5a30cad06a8616184e09bd48fe99975b6750adc8 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  9924b2e4ce2c7b2d44b64657b300e1592b2bd394 
>   
> repository/src/main/java/org/apache/atlas/repository/ogm/metrics/AtlasMetricsStatDTO.java
>  6af935ebbb1f3b5e03589d19ea16d51ba675d5db 
>   repository/src/main/java/org/apache/atlas/services/MetricsService.java 
> 9ec2cd2cb249b1f881f01261e7ae7e10f0adbe67 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
> d55ada77edd9d006d6ef31720c427e08984097ba 
> 
> 
> Diff: https://reviews.apache.org/r/74061/diff/2/
> 
> 
> Testing
> ---
> 
> 
> File Attachments
> 
> 
> 0001-MetricsImportExportUI-ApacheMasterBranch.patch
>   
> https://reviews.apache.org/media/uploaded/files/2022/07/26/0eabb428-689a-4d59-bbae-55b76495c131__0001-MetricsImportExportUI-ApacheMasterBranch.patch
> 
> 
> Thanks,
> 
> Beakal Lemeneh
> 
>

Re: Review Request 74070: ATLAS-4647 : Upgrade Spring Security to 5.7.2

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74070/#review224585
---


Ship it!




Ship It!

- Ashutosh Mestry


On July 27, 2022, 6:08 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74070/
> ---
> 
> (Updated July 27, 2022, 6:08 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab and Pinal Shah.
> 
> 
> Bugs: ATLAS-4647
> https://issues.apache.org/jira/browse/ATLAS-4647
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Current version for spring-security is 5.7.1 upgrading it to 5.7.2
> 
> 
> Diffs
> -
> 
>   pom.xml 1e32e45f3 
> 
> 
> Diff: https://reviews.apache.org/r/74070/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean -DskipTests package -Pdist,embedded-hbase-solr
> mvn clean install -Drat.skip
> atlas server running successfully
> 
> 
> Thanks,
> 
> chaitali
> 
>

Re: Review Request 74072: ATLAS-4648 : Upgrade Netty version to 4.1.78.Final

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74072/#review224584
---


Ship it!




Ship It!

- Ashutosh Mestry


On July 27, 2022, 6:25 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74072/
> ---
> 
> (Updated July 27, 2022, 6:25 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab and Pinal Shah.
> 
> 
> Bugs: ATLAS-4648
> https://issues.apache.org/jira/browse/ATLAS-4648
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> current version for Netty to 4.1.77.Final
> 
> 
> Diffs
> -
> 
>   pom.xml 1e32e45f3 
> 
> 
> Diff: https://reviews.apache.org/r/74072/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean -DskipTests package -Pdist,embedded-hbase-solr
> mvn clean install -Drat.skip
> atlas server running successfully
> 
> 
> Thanks,
> 
> chaitali
> 
>

Re: Review Request 74071: ATLAS-4649 : Upgrade Spring Framework to 5.3.21

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74071/#review224583
---


Ship it!




Ship It!

- Ashutosh Mestry


On July 27, 2022, 6:19 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74071/
> ---
> 
> (Updated July 27, 2022, 6:19 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab and Pinal Shah.
> 
> 
> Bugs: ATLAS-4649
> https://issues.apache.org/jira/browse/ATLAS-4649
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> current version for Spring Framework is 5.3.20
> 
> 
> Diffs
> -
> 
>   pom.xml 1e32e45f3 
> 
> 
> Diff: https://reviews.apache.org/r/74071/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean -DskipTests package -Pdist,embedded-hbase-solr
> mvn clean install -Drat.skip
> atlas server running successfully
> 
> 
> Thanks,
> 
> chaitali
> 
>

Re: Review Request 74072: ATLAS-4648 : Upgrade Netty version to 4.1.78.Final

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74072/#review224582
---




pom.xml
Line 751 (original), 751 (patched)
<https://reviews.apache.org/r/74072/#comment313379>

I don't see a version update. Not sure if I am missing anything.


- Ashutosh Mestry


On July 27, 2022, 6:25 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74072/
> ---
> 
> (Updated July 27, 2022, 6:25 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab and Pinal Shah.
> 
> 
> Bugs: ATLAS-4648
> https://issues.apache.org/jira/browse/ATLAS-4648
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> current version for Netty to 4.1.77.Final
> 
> 
> Diffs
> -
> 
>   pom.xml 1e32e45f3 
> 
> 
> Diff: https://reviews.apache.org/r/74072/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean -DskipTests package -Pdist,embedded-hbase-solr
> mvn clean install -Drat.skip
> atlas server running successfully
> 
> 
> Thanks,
> 
> chaitali
> 
>

Re: Review Request 74061: ATLAS-4637: Import/Export Feature for MetricsStat

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74061/#review224568
---



Can you please add unit tests that demonstrate flow of this feature?


webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
Line 95 (original), 104 (patched)
<https://reviews.apache.org/r/74061/#comment313323>

Please replace '*' with individual imports. Updating IntelliJ setting 
avoids this problem.



webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
Lines 573 (patched)
<https://reviews.apache.org/r/74061/#comment313326>

I think endpoint like importStatsByLabel will be easier to understand.



webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
Line 485 (original), 612 (patched)
<https://reviews.apache.org/r/74061/#comment313325>

Please remove commented code.



webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
Lines 619 (patched)
<https://reviews.apache.org/r/74061/#comment313324>

Please remove commented code.


- Ashutosh Mestry


On July 15, 2022, 7:34 p.m., Beakal Lemeneh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74061/
> ---
> 
> (Updated July 15, 2022, 7:34 p.m.)
> 
> 
> Review request for atlas, Disha Talreja, Radhika Kundam, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-4637
> https://issues.apache.org/jira/browse/ATLAS-4637
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Add new attribute - importLabel - to MetricsStat. It contains the 
> name/identity of the user who is importing MetricsStat file from one machine 
> to another machine. For some Machine A, importLabel is set to "self" for the 
> already existing MetricsStat that collects the metrics info in Machine A, and 
> has not been imported from another machine.
> 
> 2. Update metricsId to contain both importLabel and collectionTime to 
> continue holding its property of being a unique attribute.
> 
> 3. Update MetricsService to implement REST APIs in AdminResource to realize 
> api calls of:
>   1. Post MetricsStat with a specific importLabel obtained from a another 
> machine.
>   2. Get MetricsStat with a specific importLabel found in some machine 
> with a boolean flag that returns metrics with specific details, or with 
> minimal information.
>   3. Delete all existing MetricsStats entities in a given machine.
>   4. Delete all existing MetricsStats entities in a given machine with a 
> specific importLabel.
>   5. Get all importLabels.
> 
> 6. Update MetricsService to include helper methods of:
>   1. Get a persisted metricsStat by importLabel.
>   2. Get a persisted metricsStat by collectionTime and importLabel.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetricsStat.java 
> 5a30cad06a8616184e09bd48fe99975b6750adc8 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  9924b2e4ce2c7b2d44b64657b300e1592b2bd394 
>   
> repository/src/main/java/org/apache/atlas/repository/ogm/metrics/AtlasMetricsStatDTO.java
>  6af935ebbb1f3b5e03589d19ea16d51ba675d5db 
>   repository/src/main/java/org/apache/atlas/services/MetricsService.java 
> 9ec2cd2cb249b1f881f01261e7ae7e10f0adbe67 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
> d55ada77edd9d006d6ef31720c427e08984097ba 
> 
> 
> Diff: https://reviews.apache.org/r/74061/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Beakal Lemeneh
> 
>

[jira] [Updated] (ATLAS-4619) Refactor Atlas webapp module to remove Kafka core dependency

[Ashutosh Mestry (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-4619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry updated ATLAS-4619:
---
Component/s:  atlas-core
 (was: atlas-webui)

> Refactor Atlas webapp module to remove Kafka core dependency
> 
>
> Key: ATLAS-4619
> URL: https://issues.apache.org/jira/browse/ATLAS-4619
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Reporter: Patrik Márton
>Priority: Major
>
> Goal is to break the strong coupling between Atlas components and Kafka. 
> These dependencies include using server side libraries of Kafka (this couples 
> the Scala version and other non-public interfaces of Kafka). Any code using 
> server side libraries of Kafka should be refactored.
> Since atlas webapp module uses the ShutdownableThread from core kafka, it 
> should be refactored in a way to eliminate this dependency.
> https://github.com/apache/atlas/blob/master/webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java#L526



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Review Request 74036: ATLAS-4627 :- Atlas - Delete Existing entity in Atlas which are not present in Hive, DB wise through Import-hive command.

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74036/#review224552
---


Ship it!




Ship It!

- Ashutosh Mestry


On June 23, 2022, 6:52 a.m., Paresh Devalia wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74036/
> ---
> 
> (Updated June 23, 2022, 6:52 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Mandar Ambawane, and Pinal Shah.
> 
> 
> Bugs: ATLAS-4627
> https://issues.apache.org/jira/browse/ATLAS-4627
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> PROBLEM STATEMENT
> 
> Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and 
> hive_tez.
> Created tables with 10 record in DB_1 and DB_2 .
> In Atlas UI we checked it has no entity created, so we ran import-hive.sh
> bash import-hive.sh -d DB_1 -t table_1, so this command will only create one 
> entity in atlas.
> bash import-hive.sh -d DB_1, so this command will create all entity in atlas.
> 
> Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as 
> hooks are disable, so we will ran import-hive command to delete from atlas .
> 
> bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command will 
> delete all the table from all DB which are present in Atlas and dropped from 
> Hive.
> As we have mention -d and -t argument in command for DB and table wise 
> deleting, but still it ran for all DB and Tables.
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  28365bc5c 
> 
> 
> Diff: https://reviews.apache.org/r/74036/diff/1/
> 
> 
> Testing
> ---
> 
> Manual testing is done.
> 
> 1) mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
>   Atlas server runnning
> 2) Quick started was passed successfully
> 
> Pre-commit : 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1130/consoleFull
> 
> Hive DB Testing 
> 
> 1) Created a DB_1 and DB_2 in Hive and disable the Atlas hook from hive and 
> hive_tez.
> 2) Created tables with 10 record in DB_1 and DB_2 .
> 3) In Atlas UI we checked it has no entity created, so we ran import-hive.sh
> 4) bash import-hive.sh -d DB_1 -t table_1, so this command will only create 
> one entity in atlas.
> 5) bash import-hive.sh -d DB_1, so this command will create all entity in 
> atlas.
> 6) Drop table_1 from DB_1 and DB_2 in Hive, but will be present in atlas as 
> hooks are disable, so we will ran import-hive command to delete from atlas 
> 7) bash import-hive.sh -d DB_1 -t table_1 -deleteNonExisting, this command 
> will delete only table_1 from DB_1 which are present in Atlas and dropped 
> from Hive.
> 8) bash import-hive.sh -d DB_2 -t table_1 -deleteNonExisting, this command 
> will delete only table_1 table from DB_2 which are present in Atlas and 
> dropped from Hive.
> 9) bash import-hive.sh -d DB_1 -deleteNonExisting, this command will delete 
> all table from DB_1 which are present in Atlas and dropped from Hive.
> 10) bash import-hive.sh -d DB_2 -deleteNonExisting, this command will delete 
> all table from DB_2 which are present in Atlas and dropped from Hive.
> 11) Checked the deleted table on statistics page in Atlas UI.
> 
> 
> Thanks,
> 
> Paresh Devalia
> 
>

Re: Review Request 74028: ATLAS-4624 : Upgrade gson to 2.9.0

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74028/#review224551
---


Ship it!




Ship It!

- Ashutosh Mestry


On June 24, 2022, 8:52 a.m., Vinayak Marraiya wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74028/
> ---
> 
> (Updated June 24, 2022, 8:52 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab and Pinal Shah.
> 
> 
> Bugs: ATLAS-4624
> https://issues.apache.org/jira/browse/ATLAS-4624
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently Atlas is using gson version 2.5 upgrading it to 2.9.0
> 
> 
> Diffs
> -
> 
>   pom.xml c83099f9a 
> 
> 
> Diff: https://reviews.apache.org/r/74028/diff/1/
> 
> 
> Testing
> ---
> 
> 1)mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
>   Atlas server runnning
> 2)Quick started was passed successfully
> 
> Pre-Commit - 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1131/console
> 
> 
> Thanks,
> 
> Vinayak Marraiya
> 
>

Re: [DRAFT] [REPORT] Apache Atlas - July 2022

[Ashutosh Mestry]

+1 

Thanks for taking care of this!

~ ashutosh

On 2022/07/11 05:45:50 Madhan Neethiraj wrote:
> Atlas community,
> 
> Please review the following draft report for this quarter.
> 
> Thanks,
> Madhan
> 
> ## Description:
> The mission of Apache Atlas is the creation and maintenance of software 
> related 
> to a scalable and extensible set of core foundational governance services - 
> enabling enterprises to effectively and efficiently meet their compliance 
> requirements
> 
> ## Issues:
> There are not issues requiring board attention.
> 
> ## Membership Data:
> Apache Atlas was founded 2017-06-20 (5 years ago)
> There are currently 45 committers and 32 PMC members in this project.
> The Committer-to-PMC ratio is roughly 3:2.
> 
> Community changes, past quarter:
> - No new PMC members. Last addition was Ashutosh Mestry on 2019-04-16.
> - No new committers. Last addition was Radhika Kundam on 2022-02-04.
> 
> ## Project Activity:
> - resolved failures with Atlas HA with index-recovery enabled 
> - notification message processing improvements to enable server and hooks to
>   be of different versions
> - Kafka importer utility improvements
> - Impala hook fix to handle large query string
> - Glossary module performance improvements to handle glossaries with large
>   number of categories and terms
> - import zip functionality improvements
> - several UI enhancements/fixes
> - dependent component version updates: Spring 5.3.20, jQuery-UI 1.13.0, netty
>   4.1.77, Spring security 5.7.1, gson 2.9.0
> 
> ## Community Health:
> - dev@atlas.apache.org had a 22% decrease in traffic in the past quarter (680
>   emails compared to 868)
> - u...@atlas.apache.org had a 37% decrease in traffic in the past quarter (12
>   emails compared to 19)
> - 48 issues opened in JIRA, past quarter (-12% change)
> - 24 issues closed in JIRA, past quarter (-20% change)
> - 48 commits in the past quarter (-27% change)
> - 11 code contributors in the past quarter (-15% change)
> - 24 PRs opened on GitHub, past quarter (20% increase)
> - 11 PRs closed on GitHub, past quarter (450% increase)
> 
> ## Most Recent releases:
> 2.2.0 was released on 2021-08-17
> 2.1.0 was released on 2020-07-15
> 0.8.4 was released on 2019-06-21
> 
> 
> 
> 

Re: Review Request 74005: ATLAS-4609 :- Atlas - Upgrade Spring-security to 5.7.1

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74005/#review224490
---


Ship it!




Ship It!

- Ashutosh Mestry


On June 9, 2022, 5:55 p.m., Vinayak Marraiya wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74005/
> ---
> 
> (Updated June 9, 2022, 5:55 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab and Pinal Shah.
> 
> 
> Bugs: ATLAS-4609
> https://issues.apache.org/jira/browse/ATLAS-4609
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently Atlas is using Spring-security version 5.5.1 upgrading it to 5.7.1
> 
> 
> Diffs
> -
> 
>   pom.xml 29d769296 
> 
> 
> Diff: https://reviews.apache.org/r/74005/diff/1/
> 
> 
> Testing
> ---
> 
> 1)mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
>   Atlas server runnning
> 2)Quick started was passed successfully
> 
> Pre-commit :- 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1113/console
> 
> 
> Thanks,
> 
> Vinayak Marraiya
> 
>

Re: Review Request 73991: ATLAS-4605 Prevent NullPointerException when Atlas Vertex is empty because of corrupted data

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73991/#review224462
---


Ship it!




Ship It!

- Ashutosh Mestry


On May 18, 2022, 10:41 a.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73991/
> ---
> 
> (Updated May 18, 2022, 10:41 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, 
> Radhika Kundam, Sarath Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-4605
> https://issues.apache.org/jira/browse/ATLAS-4605
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Because of corrupted data, there may be a chance of having AtlasVertex as an 
> empty, which means AtlasVertex does not have any property present in it.
> This can cause NullPointerException while populating System attributes as 
> well Normal attributes for AtlasEntity object.
> To avoid NullPointerException we need to skip such vertices, so that they 
> cannot be used to populate system attributes and normal attributes.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  fcd8dfe7f 
> 
> 
> Diff: https://reviews.apache.org/r/73991/diff/1/
> 
> 
> Testing
> ---
> 
> PreCommit: 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1098/consoleFull
> 
> Sanity testing is done.
> 
> Test Scenarios covered:
> 
> Hive hook:
> Create table, check column and table relationships
> Create ctas table, check relationships
> Generate shell entities
> Check if added tables are generating correct entries after reversing shell 
> condition. 
> Update table entities
> 
> Hbase hook:
> Create table, check column family and table relationships
> Check if added tables are generating correct entries after reversing hook.
> Update table entities
> 
> General:
> Search for entities in basic and advanced searches
> Save favourite searches
> Checked for entity-delete and entity-purge scenarios
> Applied classifications
> Applied terms
> Create and apply business metadata
> 
> 
> Thanks,
> 
> Mandar Ambawane
> 
>

Re: Review Request 73974: ATLAS-25071 Deferred Actions : Support to add classification name instead of classificationVertexId

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73974/#review224437
---



Good change overall. We had this during our implementation.


repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
Lines 1200 (patched)
<https://reviews.apache.org/r/73974/#comment313238>

What is the reason for not updating existing method? With the new method, 
there is duplication of code.


- Ashutosh Mestry


On May 10, 2022, 12:15 p.m., Paresh Devalia wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73974/
> ---
> 
> (Updated May 10, 2022, 12:15 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Mandar Ambawane, and Pinal Shah.
> 
> 
> Bugs: ATLAS-25071
> https://issues.apache.org/jira/browse/ATLAS-25071
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> In /api/atlas/admin/tasks call , when a tag is propagated , the 
> classificationVertexId is displayed. It becomes tough to debug when there are 
> many pending tasks . classification name can be added in the response in 
> addition to/instead of classificationVertexId.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  f118ae69a 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  9a7f29017 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationTask.java
>  0bad84e49 
> 
> 
> Diff: https://reviews.apache.org/r/73974/diff/1/
> 
> 
> Testing
> ---
> 
> Manual testing is done.
> 
> Pre-commit : 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1095/consoleFull
> 
> Created 1000  hive_tables dependant on parent table.
> Hive_tables name will be sequence_1 to sequence_999
> 
> Below step are tested for add, update and delete classification.
> 
> 1) Adding classification :- 
>   Create a classification with name test_1.
>   Added test_1 classification with checked on "propagate" of 
> sequence_1 hive_table, so this classification will be assign to all child 
> table.
>   So while background process is going on, will check the task 
> status API.
>   curl k -u :  -request GET 
> 'https://:/api/atlas/admin/tasks'
>   so in response will be
>   [
>   {
>   "type": 
> "CLASSIFICATION_PROPAGATION_ADD",
>   "guid": 
> "39f68061-2c5b-4e0c-9ba6-65b3d8117e99",
>   "createdBy": "admin",
>   "createdTime": 1651591949394,
>   "updatedTime": 1651591949394,
>   "parameters": {
>   "relationshipGuid": null,
>   "classificationName": "test_1",
>   "entityGuid": 
> "29a5ac89-a3fb-46ff-bbb1-646d3b0717dd",
>   "classificationVertexId": 
> "122896440"
>   },
>   "attemptCount": 0,
>   "status": "PENDING"
>   }
>   ]
>   
> 2)Updating classification attribute value:-
>   Creating attribute for test_1 classification.
>   Updating classification attribute value on sequence_1 
> hive_table, then in background process it will update all the child table 
> attribute value.
>   So while background process is going on, will check the task 
> status API.
>   curl k -u :  -request GET 
> 'https://:/api/atlas/admin/tasks' 
>   so in response will be
>   [
>   {
>   "type": 
> "CLASSIFICATION_PROPAGATION_ADD",
>   "guid": 
> 

Re: Review Request 73952: ATLAS-4585: (UI)Incorrect hyphenation of words in property description field

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73952/#review224346
---


Ship it!




Ship It!

- Ashutosh Mestry


On April 21, 2022, 10:37 a.m., Farhan Khan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73952/
> ---
> 
> (Updated April 21, 2022, 10:37 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab and Prasad Pawar.
> 
> 
> Bugs: ATLAS-4585
> https://issues.apache.org/jira/browse/ATLAS-4585
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hyphenation of words is incorrect.
> 
> 
> "Abonnement et" should be :
> 
> "abonnement
> et"
> 
> but we get
> 
> "abonnement e"
> "t"
> => Hyphenation happens at the middle of the word "et"
> 
> Note: This happens on Entity details page in the Technical properties section 
> having description fields.
> 
> 
> Diffs
> -
> 
>   dashboardv2/public/css/scss/common.scss 5e7ffef67 
>   dashboardv2/public/css/scss/table.scss 121769b18 
>   dashboardv3/public/css/scss/common.scss c13c47c20 
>   dashboardv3/public/css/scss/table.scss 121769b18 
> 
> 
> Diff: https://reviews.apache.org/r/73952/diff/1/
> 
> 
> Testing
> ---
> 
> Manually tested on both the UI's by adding lots of description text and it is 
> getting displayed in proper format as expected.
> 
> 
> Thanks,
> 
> Farhan Khan
> 
>

Re: Review Request 73836: ATLAS-4551: Add maven jacoco plugin for code coverage

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73836/#review224173
---


Ship it!




Ship It!

- Ashutosh Mestry


On Feb. 9, 2022, 2:58 p.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73836/
> ---
> 
> (Updated Feb. 9, 2022, 2:58 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4551
> https://issues.apache.org/jira/browse/ATLAS-4551
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Jacoco maven plugin generates test coverage report by measuring lines of code 
> exercised during the tests
> 
> *Step*
> mvn clean install
> 
> *Report*
> report will be generated in /target/site/jacoco directory of each module.
> 
> For more Graphical interface, import the generated reports of jacoco to 
> SonarQube
> sonar plugin will just look at the above path for the xml.
>  
> mvn sonar:sonar -Dsonar.host.url=http://localhost:9000 -DskipTests 
> -Dsonar.projectKey=Atlas -Dsonar.projectName=Atlas 
> -Drat.numUnapprovedLicenses=1 
> -Dsonar.login=bc6e210c3f4c1873eee82e31059215a9deeff526
> 
> 
> Diffs
> -
> 
>   pom.xml 2f7b00af3 
> 
> 
> Diff: https://reviews.apache.org/r/73836/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 73731: ATLAS-4495 Glossary section takes longer time to load when Glossaries have large number of Categories and Terms associated with them

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73731/#review224134
---




repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
Lines 1542 (patched)
<https://reviews.apache.org/r/73731/#comment313070>

Rename: populateGlorraryAttributesIfApplicable



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
Lines 1550 (patched)
<https://reviews.apache.org/r/73731/#comment313069>

Consider being defensive here. Doing this will make it explicit about what 
is happening.

if (StringUtils.notEquals(entityName, GLOSSARY_CATEGORY) {
return;
}


- Ashutosh Mestry


On March 3, 2022, 12:44 p.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73731/
> ---
> 
> (Updated March 3, 2022, 12:44 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Radhika Kundam, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4495
> https://issues.apache.org/jira/browse/ATLAS-4495
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> It is observed that, due to existing framework Glossary GET api takes longer 
> time.
> 
> This patch avoids the unnecessary db calls and improve the performance when 
> there is large number of data associated with the Glossary (say 10K terms or 
> categories)
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java 
> f81b53854 
>   repository/src/main/java/org/apache/atlas/repository/ogm/DataAccess.java 
> f902b2afb 
>   
> repository/src/main/java/org/apache/atlas/repository/ogm/glossary/AbstractGlossaryDTO.java
>  bee88c61f 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  9671df994 
> 
> 
> Diff: https://reviews.apache.org/r/73731/diff/3/
> 
> 
> Testing
> ---
> 
> Manual testing is done.
> 
> PreCommit: 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1009/consoleFull
> 
> PreCommit: 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1049/consoleFull
> 
> 
> Thanks,
> 
> Mandar Ambawane
> 
>

Re: Review Request 73731: ATLAS-4495 Glossary section takes longer time to load when Glossaries have large number of Categories and Terms associated with them

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73731/#review224133
---



Can you please add metrics about the improvement you are seeing.


repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
Line 96 (original), 98 (patched)
<https://reviews.apache.org/r/73731/#comment313066>

Rename: atlasGlossaryDTO to glossaryDTO, this will keep it consistent with 
the rest.



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
Lines 1550 (patched)
<https://reviews.apache.org/r/73731/#comment313067>

Consider declaring AtlasGlossaryCategory.class.getSimpleName() as static 
const. Doing  that will avoid fuction call.



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
Lines 1556 (patched)
<https://reviews.apache.org/r/73731/#comment313068>

Consider declaring the string to static field.


- Ashutosh Mestry


On March 3, 2022, 12:44 p.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73731/
> ---
> 
> (Updated March 3, 2022, 12:44 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Radhika Kundam, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4495
> https://issues.apache.org/jira/browse/ATLAS-4495
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> It is observed that, due to existing framework Glossary GET api takes longer 
> time.
> 
> This patch avoids the unnecessary db calls and improve the performance when 
> there is large number of data associated with the Glossary (say 10K terms or 
> categories)
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java 
> f81b53854 
>   repository/src/main/java/org/apache/atlas/repository/ogm/DataAccess.java 
> f902b2afb 
>   
> repository/src/main/java/org/apache/atlas/repository/ogm/glossary/AbstractGlossaryDTO.java
>  bee88c61f 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  9671df994 
> 
> 
> Diff: https://reviews.apache.org/r/73731/diff/3/
> 
> 
> Testing
> ---
> 
> Manual testing is done.
> 
> PreCommit: 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1009/consoleFull
> 
> PreCommit: 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1049/consoleFull
> 
> 
> Thanks,
> 
> Mandar Ambawane
> 
>

[jira] [Commented] (ATLAS-3057) Index Repair Tool: Add JanusGraph-Specific Index Repair Tool

[Ashutosh Mestry (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17494758#comment-17494758
 ] 

Ashutosh Mestry commented on ATLAS-3057:


We have tested this only with HBase + Solr.

Having said that, we have only used JanusGraph APIs. Which means that if 
appropriate configurations are added, HBase + ES should function.

Hope that helps.

> Index Repair Tool: Add JanusGraph-Specific Index Repair Tool
> 
>
> Key: ATLAS-3057
> URL: https://issues.apache.org/jira/browse/ATLAS-3057
> Project: Atlas
>  Issue Type: New Feature
>    Reporter: Ashutosh Mestry
>Assignee: Nikhil P Bonte
>Priority: Major
> Fix For: 2.0.0, trunk
>
> Attachments: ATLAS-3057-Atlas-Index-Repair-tool-for-JanusGraph.patch, 
> ATLAS-3057-Atlas-Index-Repair-tool-for-JanusGraph.patch
>
>
> *Background*
> For Atlas version that uses _HBase_ and _Solr_, occasionally if Solr is down 
> and entities get created, then there is no record of the created entities 
> within Solr. Basic search does not indicate presence of such entities.
> *Solution*
> Index Repair Tool (which was present in branch-0.8) should be implemented for 
> JanusGraph.
> *Implementation Guidance*
>  * Create Java-based implementation rather than Groovy script that needs 
> graphdb-specific shell to be installed. 
>  * Use JanusGraph APIs for restore instead of custom per-vertex logic.
>  * Investigate possibility of using MapReduce for higher throughput.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: Review Request 73855: ATLAS-4556 Add Ozone "ofs" scheme support for external tables created through hive.

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73855/#review224076
---




common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java
Line 90 (original), 90 (patched)
<https://reviews.apache.org/r/73855/#comment313037>

Good!


- Ashutosh Mestry


On Feb. 18, 2022, 12:20 p.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73855/
> ---
> 
> (Updated Feb. 18, 2022, 12:20 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, Sarath 
> Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-4556
> https://issues.apache.org/jira/browse/ATLAS-4556
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> When we provide ozone key as a path for an external table in hive,
> corresponding Ozone entities of type "Ozone_Volume", "Ozone_Bucket", 
> "Ozone_Key" etc. get created.
> 
> Currently in Atlas, code supports the above in "o3fs" ozone scheme.
> Need to provide improved support in "ofs" scheme, same as that of "o3fs".
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java 
> cf8f262b9 
>   common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java 
> f35e9ae57 
> 
> 
> Diff: https://reviews.apache.org/r/73855/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Mandar Ambawane
> 
>

Re: Review Request 73855: ATLAS-4556 Add Ozone "ofs" scheme support for external tables created through hive.

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73855/#review224075
---




common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java
Line 440 (original), 451 (patched)
<https://reviews.apache.org/r/73855/#comment313034>

Consider using StringUtils.EMPTY



common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java
Lines 452 (patched)
<https://reviews.apache.org/r/73855/#comment313035>

ofs -> Extract to private static const field.



common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java
Lines 453 (patched)
<https://reviews.apache.org/r/73855/#comment313036>

Consider usind String.format to construct the string.


- Ashutosh Mestry


On Feb. 18, 2022, 12:20 p.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73855/
> ---
> 
> (Updated Feb. 18, 2022, 12:20 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, Sarath 
> Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-4556
> https://issues.apache.org/jira/browse/ATLAS-4556
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> When we provide ozone key as a path for an external table in hive,
> corresponding Ozone entities of type "Ozone_Volume", "Ozone_Bucket", 
> "Ozone_Key" etc. get created.
> 
> Currently in Atlas, code supports the above in "o3fs" ozone scheme.
> Need to provide improved support in "ofs" scheme, same as that of "o3fs".
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java 
> cf8f262b9 
>   common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java 
> f35e9ae57 
> 
> 
> Diff: https://reviews.apache.org/r/73855/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Mandar Ambawane
> 
>

Re: Review Request 73822: ATLAS-4538 :- Atlas - Upgrade tinker pop to 3.5.2

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73822/#review224008
---


Ship it!




Ship It!

- Ashutosh Mestry


On Jan. 28, 2022, 11:51 a.m., Vinayak Marraiya wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73822/
> ---
> 
> (Updated Jan. 28, 2022, 11:51 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Pinal Shah, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4538
> https://issues.apache.org/jira/browse/ATLAS-4538
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently Atlas is using tinker pop version 3.5.1 upgrading it to 3.5.2
> 
> 
> Diffs
> -
> 
>   pom.xml 270800d41 
>   repository/src/test/java/org/apache/atlas/query/TraversalComposerTest.java 
> 8f760337e 
> 
> 
> Diff: https://reviews.apache.org/r/73822/diff/1/
> 
> 
> Testing
> ---
> 
> 1)mvn clean install
>   Atlas server runnning
> 2)Quick started was passed successfully
> 3)Precommit - 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1028/console
> 4)DSL queries executed from Atlas-UI
> 
> 
> Thanks,
> 
> Vinayak Marraiya
> 
>

Re: Review Request 73798: ATLAS-4529 : DSL Search : OrderBy does not work for ScriptEngineBasedExecutor (gremlin groovy)

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73798/#review223956
---


Ship it!




Ship It!

- Ashutosh Mestry


On Jan. 12, 2022, 8:55 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73798/
> ---
> 
> (Updated Jan. 12, 2022, 8:55 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-4529
> https://issues.apache.org/jira/browse/ATLAS-4529
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> After upgrading janusgraph version to 0.6.0
> When below property is changed to 'false', order by doesn't work
> 
> atlas.dsl.executor.traversal=false
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/query/GremlinClause.java 
> 7014b4a4f 
>   
> repository/src/main/java/org/apache/atlas/util/AtlasGremlin3QueryProvider.java
>  547a43dd7 
>   
> repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
> 52cb68caa 
> 
> 
> Diff: https://reviews.apache.org/r/73798/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 73791: ATLAS-4523 - Build failure on Atlas docs

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73791/#review223944
---


Ship it!




Ship It!

- Ashutosh Mestry


On Jan. 7, 2022, 12:20 p.m., Farhan Khan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73791/
> ---
> 
> (Updated Jan. 7, 2022, 12:20 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab and Prasad Pawar.
> 
> 
> Bugs: ATLAS-4523
> https://issues.apache.org/jira/browse/ATLAS-4523
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> docz has transitive dependencies which require a newer version of babel (at 
> least 7.13.0). Requiring these old versions ends up bringing in conflicting 
> versions of babel which breaks the atlas docs project.
> 
> 
> Diffs
> -
> 
>   docs/package.json f38a965ed 
> 
> 
> Diff: https://reviews.apache.org/r/73791/diff/1/
> 
> 
> Testing
> ---
> 
> Tested on local, build get success after the babel version upgrade.
> 
> 
> Thanks,
> 
> Farhan Khan
> 
>

Re: Is Linux ARM64 supported ?

[Ashutosh Mestry]

Hi Martin

Your fix was merged on Oct-6. It should be part of the next Atlas release. We 
have not decided on the timeframe for the release yet.

About Jenkin job form ARM64: Right not, I am not sure how to proceed with that. 
Hence we have not picked it up yet. I am not sure on the approvals needed for 
the same. If you have any experience with that, we can certainly look at it.

Best regards,

~ ashutosh
Ashutosh Mestry


From: Martin Grigorov 
Date: Wednesday, December 8, 2021 at 4:24 AM
To: dev@atlas.apache.org 
Subject: Re: Is Linux ARM64 supported ?
Hi Mark,


On Wed, Dec 8, 2021 at 1:51 PM Mark Jens  wrote:

> Thank you for your answer, Martin!
>
> Do you recommend using Atlas on Linux ARM64 ? What is your experience ?
>

Honestly, I don't!
The reason is not technical though!
My main concern is the lack of responsiveness of the Atlas team. It took
many months to get some attention.
Last time I've tested the build it was working but without proper testing
on Linux ARM64 regressions may appear anytime! I've proposed adding Jenkins
agent on ARM64 at
https://lists.apache.org/thread/gmjj2ryzg76jwmfdfxtl51n8rtjly0km but guess
what ? No one ever answered :-/

Regards,
Martin


>
> Mark
>
> On Wed, 8 Dec 2021 at 13:25, Martin Grigorov  wrote:
>
> > Hi Mark,
> >
> > I believe the fix is not part yet of any release.
> > Atlas 2.2.0 has been released at 2021-08-17 (
> > https://atlas.apache.org/#/Downloads) and the fix has been merged at Oct
> > 6th (
> >
> >
> https://github.com/apache/atlas/commit/8b48f0183e5ae34da1a7bad33aa75c427d35d063
> > ).
> >
> > Regards,
> > Martin
> >
> >
> > On Wed, Dec 8, 2021 at 1:09 PM Mark Jens  wrote:
> >
> > > Anyone ?
> > >
> > > On Mon, 6 Dec 2021 at 14:35, Mark Jens  wrote:
> > >
> > > > Hello Atlas devs!
> > > >
> > > > I see that https://issues.apache.org/jira/browse/ATLAS-4159 fixes a
> > > build
> > > > issue on Linux ARM64.
> > > > What is not clear to me is in which version the fix is part of ?
> > > > The ticket's "Fix Version" was not updated when the ticket has been
> > > > resolved.
> > > >
> > > > Thank you!
> > > >
> > > > Mark
> > > >
> > >
> >
>

[jira] [Assigned] (ATLAS-4501) Table 'Allow list' with a default deny to load only a subset of tables

[Ashutosh Mestry (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry reassigned ATLAS-4501:
--

Assignee: Radhika Kundam

> Table 'Allow list' with a default deny to load only a subset of tables
> --
>
> Key: ATLAS-4501
> URL: https://issues.apache.org/jira/browse/ATLAS-4501
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core, atlas-intg, hive-integration
>Reporter: Adriano
>Assignee: Radhika Kundam
>Priority: Major
>
> There are some huge environments where the warehouse has a thousand databases 
> and hundred thousand tables with many columns and most of them are dropped, 
> created, updated at a fast pace. In these environments, the Atlas processing 
> time can slow down increasing the backlog as it starts moving slower than the 
> changes in the warehouse and the {{prune.pattern}} e/o {{ignore.pattern}} it 
> is not suitable.
> It will be nice to have the opportunity to have a default deny behaviour for 
> all the tables and then to 'allow' the import of a subset of tables specified 
> in a parameter regex (in order to process only some important tables): 
> basically that works in the opposite way to the {{prune.pattern}} and 
> {{ignore.pattern.}}
> As far as I know, there is a similar feature for S3 and ADLS but not for hive.
> If this is the case, will be nice to get the feature onboarded in your 
> backlog.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: Review Request 73725: ATLAS-4493 : Upgrade netty to 4.1.68+

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73725/#review223767
---


Ship it!




Ship It!

- Ashutosh Mestry


On Nov. 23, 2021, 9:24 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73725/
> ---
> 
> (Updated Nov. 23, 2021, 9:24 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4493
> https://issues.apache.org/jira/browse/ATLAS-4493
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Upgrading netty to 4.1.69
> 
> 
> Diffs
> -
> 
>   pom.xml 86519daba 
> 
> 
> Diff: https://reviews.apache.org/r/73725/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean install
> mvn clean -DskipTests package -Pdist,embedded-hbase-solr
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/983/consoleFull
> 
> 
> Thanks,
> 
> chaitali
> 
>

[jira] [Commented] (ATLAS-4479) ApplicationProperties cause NPE when using Elasticsearch

[Ashutosh Mestry (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444001#comment-17444001
 ] 

Ashutosh Mestry commented on ATLAS-4479:


I have approved the PR.

> ApplicationProperties cause NPE when using Elasticsearch
> 
>
> Key: ATLAS-4479
> URL: https://issues.apache.org/jira/browse/ATLAS-4479
> Project: Atlas
>  Issue Type: Bug
>Reporter: wuzhiguo
>Priority: Major
>
> code in line 367: `LOG.info("Setting " + SOLR_WAIT_SEARCHER_CONF + " = " + 
> getBoolean(SOLR_WAIT_SEARCHER_CONF));` will cause NPE when using 
> elasticsearch, which should not, cause the configuration belongs to solr



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (ATLAS-4474) Cannot append log in correct file when using Docker

[Ashutosh Mestry (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-4474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442109#comment-17442109
 ] 

Ashutosh Mestry commented on ATLAS-4474:


Can you please tell me what have you done so far?

I have was able to get the docker image to work by following the steps 
mentioned in the readme file.

> Cannot append log in correct file when using Docker
> ---
>
> Key: ATLAS-4474
> URL: https://issues.apache.org/jira/browse/ATLAS-4474
> Project: Atlas
>  Issue Type: Bug
>Reporter: wuzhiguo
>Priority: Major
> Attachments: Dockerfile, image-2021-11-10-18-05-53-971.png, 
> image-2021-11-10-18-07-52-717.png, image-2021-11-10-18-08-34-805.png
>
>
> When I'm using docker to build atlas, the log cannot resolve the vm options 
> defined in python script
> Please see the images and my Dockerfile
> The bin file is based on release-2.2.0-rc1
> I'd like to upload my bin file too, but it exceeds maximum size limit 
> (approximate to 440M)
> !image-2021-11-10-18-05-53-971.png!!image-2021-11-10-18-07-52-717.png!
> !image-2021-11-10-18-08-34-805.png!



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (ATLAS-4246) Make Kafka Interface aware of Kafka Schema Registry

[Ashutosh Mestry (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17441268#comment-17441268
 ] 

Ashutosh Mestry commented on ATLAS-4246:


Here's the new CI build: 
[https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/954/]

Will proceed with commit if this succeeds.

> Make Kafka Interface aware of Kafka Schema Registry
> ---
>
> Key: ATLAS-4246
> URL: https://issues.apache.org/jira/browse/ATLAS-4246
> Project: Atlas
>  Issue Type: Improvement
>  Components: kafka-integration
>Affects Versions: 2.1.0, 3.0.0
>Reporter: Aileen Toleikis
>Assignee: Viktor Somogyi-Vass
>Priority: Major
>  Labels: Kafka
> Fix For: 3.0.0, 2.3.0
>
>
> Kafka Community is using Schema Registry more and more heavily but as Atlas 
> is currently unaware of this, this extension helps Atlas make use of the 
> Schemas.
>  
> We have tested this extension and we have production environments where Atlas 
> will not be allowed without schema registry access. We have received feedback 
> that this extension would be sufficient to allow production use.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (ATLAS-4464) Ingest: Improve Rate of Ingest

[Ashutosh Mestry (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-4464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry updated ATLAS-4464:
---
Attachment: (was: 
ATLAS-4464-concurrent-notification-processing-obj-sync-v3.patch)

> Ingest: Improve Rate of Ingest
> --
>
> Key: ATLAS-4464
> URL: https://issues.apache.org/jira/browse/ATLAS-4464
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: trunk
>    Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Fix For: trunk
>
>
> *Background*
> Existing implementation of _NotificationHookConsumer_ has linear complexity 
> for ingestion. This has several impacts:
>  * Authorization policies will take longer to get enforced.
>  * Unpredictable wait times for metadata showing up in Atlas.
> *Solution*
> Implement a mechanism for processing messages such that:
>  * Determine dependencies within incoming messages.
>  * Dependent messages should get processed serially.
>  * Messages without dependencies are processed concurrently.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-4469) Support HBase secured by Kerberos

[Ashutosh Mestry (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437134#comment-17437134
 ] 

Ashutosh Mestry commented on ATLAS-4469:


We already support HBase authenticated by Kerberos. Can you please add some 
details to this ticket? 

> Support HBase secured by Kerberos
> -
>
> Key: ATLAS-4469
> URL: https://issues.apache.org/jira/browse/ATLAS-4469
> Project: Atlas
>  Issue Type: Improvement
>Reporter: seafish
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 73671: ATLAS-4463: Infinite loop at Index Health Monitor (IndexRecoveryService)

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73671/#review223686
---


Ship it!




Ship It!

- Ashutosh Mestry


On Oct. 28, 2021, 9:32 p.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73671/
> ---
> 
> (Updated Oct. 28, 2021, 9:32 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4463
> https://issues.apache.org/jira/browse/ATLAS-4463
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-4463: Fixed Infinite loop at Index Health Monitor (IndexRecoveryService)
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/IndexRecoveryService.java
>  2f1161067 
> 
> 
> Diff: https://reviews.apache.org/r/73671/diff/2/
> 
> 
> Testing
> ---
> 
> Run local UT/ITs and Manually ran jetty run and some IT to see the loop is 
> exiting at the end
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>

Re: Review Request 73670: ATLAS-4351 Maven-jetty throwing warnings when running Integration tests for conflicting jars in classpath

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73670/#review223685
---


Ship it!




Ship It!

- Ashutosh Mestry


On Oct. 28, 2021, 9:14 p.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73670/
> ---
> 
> (Updated Oct. 28, 2021, 9:14 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4351
> https://issues.apache.org/jira/browse/ATLAS-4351
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Reverting: ATLAS-4351
> Maven-jetty throwing warnings when running Integration tests for conflicting 
> jars in classpath
> 
> 
> Diffs
> -
> 
>   addons/falcon-bridge/pom.xml 9d56fd8d3 
>   addons/hbase-bridge/pom.xml 151a62a77 
>   addons/hbase-testing-util/pom.xml 8ca962adc 
>   addons/hive-bridge/pom.xml f62140918 
>   addons/impala-bridge/pom.xml 87650e1da 
>   addons/kafka-bridge/pom.xml 2ac19fd20 
>   addons/sqoop-bridge/pom.xml 03861df84 
>   addons/storm-bridge-shim/pom.xml e1e89ceda 
>   addons/storm-bridge/pom.xml 6bfe15b5b 
>   common/pom.xml ebf8ea5ae 
>   graphdb/janus-hbase2/pom.xml 9ad64062f 
>   graphdb/janus/pom.xml f38c4e9a3 
>   pom.xml 84630e76b 
>   test-tools/pom.xml 59c7a9dbf 
>   webapp/pom.xml 443559c8d 
> 
> 
> Diff: https://reviews.apache.org/r/73670/diff/1/
> 
> 
> Testing
> ---
> 
> As there are Precommit job failures I am revrting the changes. Later we can 
> fix the warning issue separately.
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>

Re: Review Request 73671: ATLAS-4463: Infinite loop at Index Health Monitor (IndexRecoveryService)

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73671/#review223684
---


Fix it, then Ship it!




Fix it, then Ship it!


repository/src/main/java/org/apache/atlas/repository/graph/IndexRecoveryService.java
Lines 185 (patched)
<https://reviews.apache.org/r/73671/#comment312792>

Use:

try {
LOG.info("==> Index Health Monitor: Shutdown: Starting...");

// handle the case where thread was not started at all
// and shutdown called
if (shouldRun.get() == false) {
return;
}

shouldRun.set(false);
}
finally {
LOG.info("<== Index Health Monitor: Shutdown: Done!");
    }


- Ashutosh Mestry


On Oct. 28, 2021, 9:21 p.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73671/
> ---
> 
> (Updated Oct. 28, 2021, 9:21 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4463
> https://issues.apache.org/jira/browse/ATLAS-4463
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-4463: Fixed Infinite loop at Index Health Monitor (IndexRecoveryService)
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/IndexRecoveryService.java
>  2f1161067 
> 
> 
> Diff: https://reviews.apache.org/r/73671/diff/1/
> 
> 
> Testing
> ---
> 
> Run local UT/ITs and Manually ran jetty run and some IT to see the loop is 
> exiting at the end
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>

Re: Review Request 73669: ATLAS-4462: Atlas tests are failing and PreCommit tests are also failing

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73669/#review223683
---




notification/src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java
Lines 123 (patched)
<https://reviews.apache.org/r/73669/#comment312791>

nit: Fix formatting.



notification/src/main/java/org/apache/atlas/util/CommandHandlerUtility.java
Lines 33 (patched)
<https://reviews.apache.org/r/73669/#comment312787>

Change attemptCount to boolean. That way you caller can simply pass true or 
false.
The logic attemptCount != 0 will depend on caller.



notification/src/main/java/org/apache/atlas/util/CommandHandlerUtility.java
Lines 42 (patched)
<https://reviews.apache.org/r/73669/#comment312788>

Refactor promote to private static final field.



notification/src/main/java/org/apache/atlas/util/CommandHandlerUtility.java
Lines 74 (patched)
<https://reviews.apache.org/r/73669/#comment312789>

Change StringUtils.isBlank to StringUtils.isEmpty.



notification/src/main/java/org/apache/atlas/util/CommandHandlerUtility.java
Lines 78 (patched)
<https://reviews.apache.org/r/73669/#comment312790>

Refactor: Promote to private static final String.


- Ashutosh Mestry


On Oct. 28, 2021, 9:19 p.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73669/
> ---
> 
> (Updated Oct. 28, 2021, 9:19 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4462
> https://issues.apache.org/jira/browse/ATLAS-4462
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-4462: Updated the pom to stop jetty before start, Added daemon flag to 
> true to stop jetty at end of post integration, Added retry logic for Kafka 
> start having port bind issue, Fixed Infinite loop for Index Recovery Service
> 
> 
> Diffs
> -
> 
>   addons/falcon-bridge/pom.xml 9d56fd8d3 
>   addons/hbase-bridge/pom.xml 151a62a77 
>   addons/hive-bridge/pom.xml f62140918 
>   addons/impala-bridge/pom.xml 87650e1da 
>   addons/kafka-bridge/pom.xml 2ac19fd20 
>   addons/sqoop-bridge/pom.xml 03861df84 
>   addons/storm-bridge/pom.xml 6bfe15b5b 
>   notification/src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java 
> 19717fb2b 
>   notification/src/main/java/org/apache/atlas/util/CommandHandlerUtility.java 
> PRE-CREATION 
>   pom.xml 84630e76b 
>   webapp/pom.xml 443559c8d 
> 
> 
> Diff: https://reviews.apache.org/r/73669/diff/2/
> 
> 
> Testing
> ---
> 
> Manually tested the Infinite loop issue at Index Recovery Service, Kafka 
> retry logic for kafka start. Along with this I ran the local tests which 
> passed.
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>

[jira] [Updated] (ATLAS-4464) Ingest: Improve Rate of Ingest

[Ashutosh Mestry (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-4464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry updated ATLAS-4464:
---
Attachment: ATLAS-4464-concurrent-notification-processing-obj-sync-v3.patch

> Ingest: Improve Rate of Ingest
> --
>
> Key: ATLAS-4464
> URL: https://issues.apache.org/jira/browse/ATLAS-4464
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: trunk
>    Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Fix For: trunk
>
> Attachments: 
> ATLAS-4464-concurrent-notification-processing-obj-sync-v3.patch
>
>
> *Background*
> Existing implementation of _NotificationHookConsumer_ has linear complexity 
> for ingestion. This has several impacts:
>  * Authorization policies will take longer to get enforced.
>  * Unpredictable wait times for metadata showing up in Atlas.
> *Solution*
> Implement a mechanism for processing messages such that:
>  * Determine dependencies within incoming messages.
>  * Dependent messages should get processed serially.
>  * Messages without dependencies are processed concurrently.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (ATLAS-4464) Ingest: Improve Rate of Ingest

[Ashutosh Mestry (Jira)]

Ashutosh Mestry created ATLAS-4464:
--

 Summary: Ingest: Improve Rate of Ingest
 Key: ATLAS-4464
 URL: https://issues.apache.org/jira/browse/ATLAS-4464
 Project: Atlas
  Issue Type: Improvement
  Components:  atlas-core
Affects Versions: trunk
Reporter: Ashutosh Mestry
Assignee: Ashutosh Mestry
 Fix For: trunk


*Background*

Existing implementation of _NotificationHookConsumer_ has linear complexity for 
ingestion. This has several impacts:
 * Authorization policies will take longer to get enforced.
 * Unpredictable wait times for metadata showing up in Atlas.

*Solution*

Implement a mechanism for processing messages such that:
 * Determine dependencies within incoming messages.
 * Dependent messages should get processed serially.
 * Messages without dependencies are processed concurrently.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 73132: ATLAS-4079 : Atlas-Upgrade jackson to 2.11.0+

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73132/#review223681
---


Ship it!




Ship It!

- Ashutosh Mestry


On Jan. 12, 2021, 9:39 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73132/
> ---
> 
> (Updated Jan. 12, 2021, 9:39 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4079
> https://issues.apache.org/jira/browse/ATLAS-4079
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch updates jackson databind version to 2.11.3
> 
> 
> Diffs
> -
> 
>   pom.xml 7bfd68b94 
> 
> 
> Diff: https://reviews.apache.org/r/73132/diff/1/
> 
> 
> Testing
> ---
> 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/324/console
> mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
> Atlas server running
> 
> 
> Thanks,
> 
> chaitali
> 
>

[jira] [Commented] (ATLAS-4246) Make Kafka Interface aware of Kafka Schema Registry

[Ashutosh Mestry (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17435501#comment-17435501
 ] 

Ashutosh Mestry commented on ATLAS-4246:


[~aileeen] Please give us couple of days, we are in the process of fixing our 
CI build. Once that is GREEN, we will proceed with committing your patch and 
few others.

> Make Kafka Interface aware of Kafka Schema Registry
> ---
>
> Key: ATLAS-4246
> URL: https://issues.apache.org/jira/browse/ATLAS-4246
> Project: Atlas
>  Issue Type: Improvement
>  Components: kafka-integration
>Affects Versions: 2.1.0, 3.0.0
>Reporter: Aileen Toleikis
>Assignee: Viktor Somogyi-Vass
>Priority: Major
>  Labels: Kafka
> Fix For: 3.0.0, 2.3.0
>
>
> Kafka Community is using Schema Registry more and more heavily but as Atlas 
> is currently unaware of this, this extension helps Atlas make use of the 
> Schemas.
>  
> We have tested this extension and we have production environments where Atlas 
> will not be allowed without schema registry access. We have received feedback 
> that this extension would be sufficient to allow production use.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 73669: ATLAS-4462: Atlas tests are failing and PreCommit tests are also failing

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73669/#review223679
---




notification/src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java
Lines 184 (patched)
<https://reviews.apache.org/r/73669/#comment312780>

Instead of killing the process, why not attempt to use next available port.



notification/src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java
Lines 194 (patched)
<https://reviews.apache.org/r/73669/#comment312779>

This won't work on Windows.



notification/src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java
Lines 207 (patched)
<https://reviews.apache.org/r/73669/#comment312781>

Call this in finally



pom.xml
Line 1688 (original), 1680 (patched)
<https://reviews.apache.org/r/73669/#comment312778>

What is the reason for downgrading this?



repository/src/main/java/org/apache/atlas/repository/graph/IndexRecoveryService.java
Line 70 (original), 72 (patched)
<https://reviews.apache.org/r/73669/#comment312782>

If possible, package this in a separate patch.



repository/src/main/java/org/apache/atlas/repository/graph/IndexRecoveryService.java
Lines 185 (patched)
<https://reviews.apache.org/r/73669/#comment312776>

Change: Index Health Monitor: Shutdown: Starting...



repository/src/main/java/org/apache/atlas/repository/graph/IndexRecoveryService.java
Lines 195 (patched)
<https://reviews.apache.org/r/73669/#comment312777>

Change: Index Health Monitor: Shutdown: Done! Add this to finally block.


- Ashutosh Mestry


On Oct. 28, 2021, 5:08 a.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73669/
> ---
> 
> (Updated Oct. 28, 2021, 5:08 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4462
> https://issues.apache.org/jira/browse/ATLAS-4462
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-4462: Updated the pom to stop jetty before start, Added daemon flag to 
> true to stop jetty at end of post integration, Added retry logic for Kafka 
> start having port bind issue, Fixed Infinite loop for Index Recovery Service
> 
> 
> Diffs
> -
> 
>   addons/falcon-bridge/pom.xml 9d56fd8d3 
>   addons/hbase-bridge/pom.xml 151a62a77 
>   addons/hive-bridge/pom.xml f62140918 
>   addons/impala-bridge/pom.xml 87650e1da 
>   addons/kafka-bridge/pom.xml 2ac19fd20 
>   addons/sqoop-bridge/pom.xml 03861df84 
>   addons/storm-bridge/pom.xml 6bfe15b5b 
>   common/pom.xml ebf8ea5ae 
>   graphdb/janus/pom.xml f38c4e9a3 
>   notification/src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java 
> 19717fb2b 
>   pom.xml 84630e76b 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/IndexRecoveryService.java
>  2f1161067 
>   test-tools/pom.xml 59c7a9dbf 
>   webapp/pom.xml 443559c8d 
> 
> 
> Diff: https://reviews.apache.org/r/73669/diff/1/
> 
> 
> Testing
> ---
> 
> Manually tested the Infinite loop issue at Index Recovery Service, Kafka 
> retry logic for kafka start. Along with this I ran the local tests which 
> passed.
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>

Re: Review Request 73639: ATLAS-4446 : Atlas - Upgrade elasticsearch to 7.14.0

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73639/#review223653
---


Ship it!




Ship It!

- Ashutosh Mestry


On Oct. 21, 2021, 2:50 p.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73639/
> ---
> 
> (Updated Oct. 21, 2021, 2:50 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4446
> https://issues.apache.org/jira/browse/ATLAS-4446
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently Elastic search is at 6.8.17
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 078bb2e5c 
>   pom.xml 84630e76b 
> 
> 
> Diff: https://reviews.apache.org/r/73639/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean package -Pdist,berkeley-elasticsearch
> 
> An additional step is required for the binary built using this profile to be 
> used along with the Atlas distribution. Due to licensing requirements, Atlas 
> does not bundle the BerkeleyDB Java Edition in the tarball.
> 
> You can download the Berkeley DB jar file from the URL: 
> http://download.oracle.com/otn/berkeley-db/je-7.5.11.zip and copy the 
> je-7.5.11.jar to the ${atlas_home}/libext directory.
> 
> Tar can be found in 
> atlas/distro/target/apache-atlas-${project.version}-bin.tar.gz
> 
> 
> Thanks,
> 
> chaitali
> 
>

[jira] [Commented] (ATLAS-4246) Make Kafka Interface aware of Kafka Schema Registry

[Ashutosh Mestry (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17432571#comment-17432571
 ] 

Ashutosh Mestry commented on ATLAS-4246:


Your changes in the new patch look good. I will wait for Sarath's 'Ship it' and 
then we will get it committed.

I am running a CI build: 
[https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/919/]

Thanks for all the work!

> Make Kafka Interface aware of Kafka Schema Registry
> ---
>
> Key: ATLAS-4246
> URL: https://issues.apache.org/jira/browse/ATLAS-4246
> Project: Atlas
>  Issue Type: Improvement
>  Components: kafka-integration
>Affects Versions: 2.1.0, 3.0.0
>Reporter: Aileen Toleikis
>Assignee: Viktor Somogyi-Vass
>Priority: Major
>  Labels: Kafka
> Fix For: 3.0.0, 2.3.0
>
>
> Kafka Community is using Schema Registry more and more heavily but as Atlas 
> is currently unaware of this, this extension helps Atlas make use of the 
> Schemas.
>  
> We have tested this extension and we have production environments where Atlas 
> will not be allowed without schema registry access. We have received feedback 
> that this extension would be sufficient to allow production use.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-4458) Commons-Logging Exclusion Causes Startup Problems

[Ashutosh Mestry (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-4458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry updated ATLAS-4458:
---
Attachment: ATLAS-4458-Commons-logging-reference-fix.patch

> Commons-Logging Exclusion Causes Startup Problems
> -
>
> Key: ATLAS-4458
> URL: https://issues.apache.org/jira/browse/ATLAS-4458
> Project: Atlas
>  Issue Type: Bug
>    Reporter: Ashutosh Mestry
>        Assignee: Ashutosh Mestry
>Priority: Major
> Attachments: ATLAS-4458-Commons-logging-reference-fix.patch
>
>
> Earlier commit of ATLAS-4351,  caused _commons-logging*.jar_ to be excluded. 
> This causes startup problem due to missing dependency.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (ATLAS-4458) Commons-Logging Exclusion Causes Startup Problems

[Ashutosh Mestry (Jira)]

Ashutosh Mestry created ATLAS-4458:
--

 Summary: Commons-Logging Exclusion Causes Startup Problems
 Key: ATLAS-4458
 URL: https://issues.apache.org/jira/browse/ATLAS-4458
 Project: Atlas
  Issue Type: Bug
Reporter: Ashutosh Mestry
Assignee: Ashutosh Mestry


Earlier commit of ATLAS-4351,  caused _commons-logging*.jar_ to be excluded. 
This causes startup problem due to missing dependency.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (ATLAS-4456) Atlas fails to start if Solr wait-searcher property is not set

[Ashutosh Mestry (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-4456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry reassigned ATLAS-4456:
--

Assignee: Ashutosh Mestry

> Atlas fails to start if Solr wait-searcher property is not set
> --
>
> Key: ATLAS-4456
> URL: https://issues.apache.org/jira/browse/ATLAS-4456
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.2.0
>Reporter: Robert Yokota
>Assignee: Ashutosh Mestry
>Priority: Major
>
> Atlas fails to start if Solr wait-searcher property is not set.  This is due 
> to the following line 
> [https://github.com/apache/atlas/blob/release-2.2.0-rc1/intg/src/main/java/org/apache/atlas/ApplicationProperties.java#L365]
> which will throw the following exception:
> {code:java}
> Exception in thread "main" org.apache.atlas.AtlasException: Failed to load 
> application properties
> at 
> org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:150)
> at 
> org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:103)
> at org.apache.atlas.Atlas.main(Atlas.java:111)
> Caused by: java.util.NoSuchElementException: 
> 'atlas.graph.index.search.solr.wait-searcher' doesn't map to an existing 
> object
> at 
> org.apache.commons.configuration.AbstractConfiguration.getBoolean(AbstractConfiguration.java:644)
> at 
> org.apache.atlas.ApplicationProperties.setDefaults(ApplicationProperties.java:365)
> at 
> org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:141)
> ... 2 more
>  {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-4449) atlas版本2.0编译报依赖org.restlet错误

[Ashutosh Mestry (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-4449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427229#comment-17427229
 ] 

Ashutosh Mestry commented on ATLAS-4449:


I used this: 
{code:java}
mvn clean install package -Pdist -DskipTests -am{code}
My build goes through without problems.  It uses the public maven repo.

Can you attempt your build with public repo. 

> atlas版本2.0编译报依赖org.restlet错误
> 
>
> Key: ATLAS-4449
> URL: https://issues.apache.org/jira/browse/ATLAS-4449
> Project: Atlas
>  Issue Type: Bug
>  Components: atlas-intg
> Environment: centos7
>Reporter: cjjxfli
>Priority: Major
>
> [ERROR] Failed to execute goal on project atlas-testtools: Could not resolve 
> dependencies for project org.apache.atlas:atlas-testtools:jar:2.3.0-SNAPSHOT: 
> The following artifacts could not be resolved: 
> org.restlet.jee:org.restlet:jar:2.4.3, 
> org.restlet.jee:org.restlet.ext.servlet:jar:2.4.3: Could not find artifact 
> org.restlet.jee:org.restlet:jar:2.4.3 in alimaven 
> (http://maven.aliyun.com/nexus/content/groups/public/) -> [Help 1]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 73641: Repair Composite indexes

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73641/#review223593
---


Ship it!




Ship It!

- Ashutosh Mestry


On Oct. 8, 2021, 1:38 p.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73641/
> ---
> 
> (Updated Oct. 8, 2021, 1:38 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> If by any chance, commposite index are not enabled, this patch fixes the 
> composite index from REGISTERED state to ENABLED state
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  7e3b2f440 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
>  1cc7f8b55 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java
>  d30971afc 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/RepairCompositeIndexPatch.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/73641/diff/1/
> 
> 
> Testing
> ---
> 
> manually tested
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Review Request 73595: ATLAS-4428 : Enhance Atlas hook messages to capture hook version & message source

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73595/#review223578
---


Fix it, then Ship it!




Ship It!


intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java
Lines 55 (patched)
<https://reviews.apache.org/r/73595/#comment312657>

Rename hookVersion to version



intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java
Lines 59 (patched)
<https://reviews.apache.org/r/73595/#comment312654>

Rename: storedHookVersion to storedVersion.



intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java
Lines 85 (patched)
<https://reviews.apache.org/r/73595/#comment312656>

setSource



intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java
Lines 87 (patched)
<https://reviews.apache.org/r/73595/#comment312655>

    Rename: getVersion


- Ashutosh Mestry


On Oct. 6, 2021, 3:13 p.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73595/
> ---
> 
> (Updated Oct. 6, 2021, 3:13 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4428
> https://issues.apache.org/jira/browse/ATLAS-4428
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas hook messages doesn't specify the source information of the messages 
> coming i.e. which hook source - HiveServer2, HiveMetastore, Impala, Spark, 
> HBase, Kafka.
> 
> Each hook message already captures, the msgSourceIP, msgCreatedBy and 
> msgCreationTime info. In addition to these, we should capture the msgSource 
> information as well.
> 
> When messages come from different hook versions, Atlas server should be aware 
> of the version of different hook messages to process the message based on the 
> hook version.
> 
> 
> Diffs
> -
> 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java
>  8c09d3394 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  0ab06f2c5 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 3cc7b3bc6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveMetastoreHookImpl.java
>  6a492c214 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  10ae08f6d 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> 4785960a7 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  779c5cb20 
>   common/src/main/java/org/apache/atlas/repository/Constants.java aea0c1311 
>   
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationBaseMessage.java
>  ff45d5713 
>   
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java
>  5869910cb 
>   intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java 
> PRE-CREATION 
>   intg/src/main/resources/atlas-buildinfo.properties PRE-CREATION 
>   notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
>  c45a1da95 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  3d8d9cc0a 
>   
> notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
>  0c92c300e 
>   notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java 
> 1ae7c278c 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
>  8078a6ceb 
>   
> notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationDeserializerTest.java
>  13eafb6b1 
>   
> notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java
>  d04817024 
> 
> 
> Diff: https://reviews.apache.org/r/73595/diff/7/
> 
> 
> Testing
> ---
> 
> mvn clean install
> mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
> testcases running successfully
> 
> 
> Thanks,
> 
> chaitali
> 
>

[jira] [Commented] (ATLAS-4246) Make Kafka Interface aware of Kafka Schema Registry

[Ashutosh Mestry (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425068#comment-17425068
 ] 

Ashutosh Mestry commented on ATLAS-4246:


[~aileeen] We will proceed with merging this. I will update RB, if we need any 
updates.

[~viktorsomogyi] Thanks for taking care of this.

> Make Kafka Interface aware of Kafka Schema Registry
> ---
>
> Key: ATLAS-4246
> URL: https://issues.apache.org/jira/browse/ATLAS-4246
> Project: Atlas
>  Issue Type: Improvement
>  Components: kafka-integration
>Affects Versions: 2.1.0, 3.0.0
>Reporter: Aileen Toleikis
>Assignee: Viktor Somogyi-Vass
>Priority: Major
>  Labels: Kafka
> Fix For: 3.0.0, 2.3.0
>
>
> Kafka Community is using Schema Registry more and more heavily but as Atlas 
> is currently unaware of this, this extension helps Atlas make use of the 
> Schemas.
>  
> We have tested this extension and we have production environments where Atlas 
> will not be allowed without schema registry access. We have received feedback 
> that this extension would be sufficient to allow production use.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 73595: ATLAS-4428 : Enhance Atlas hook messages to capture hook version & message source

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73595/#review223575
---




intg/pom.xml
Lines 149 (patched)
<https://reviews.apache.org/r/73595/#comment312647>

Remove this and instead add atlas-buildinfo.properties here:

build.epoch=${timestamp}
project.version=${pom.version}
build.version=${pom.version}
vc.revision=${buildNumber}

This will avoid copying as build step and will keep the build version true 
to the project.



notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java
Line 139 (original), 141 (patched)
<https://reviews.apache.org/r/73595/#comment312648>

Add assertEquals verifying contents of MessageSource.


- Ashutosh Mestry


On Oct. 4, 2021, 3:59 p.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73595/
> ---
> 
> (Updated Oct. 4, 2021, 3:59 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4428
> https://issues.apache.org/jira/browse/ATLAS-4428
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas hook messages doesn't specify the source information of the messages 
> coming i.e. which hook source - HiveServer2, HiveMetastore, Impala, Spark, 
> HBase, Kafka.
> 
> Each hook message already captures, the msgSourceIP, msgCreatedBy and 
> msgCreationTime info. In addition to these, we should capture the msgSource 
> information as well.
> 
> When messages come from different hook versions, Atlas server should be aware 
> of the version of different hook messages to process the message based on the 
> hook version.
> 
> 
> Diffs
> -
> 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java
>  8c09d3394 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  0ab06f2c5 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 3cc7b3bc6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveMetastoreHookImpl.java
>  6a492c214 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  10ae08f6d 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> 4785960a7 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  779c5cb20 
>   common/src/main/java/org/apache/atlas/repository/Constants.java aea0c1311 
>   intg/pom.xml 691cdf0a3 
>   
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationBaseMessage.java
>  ff45d5713 
>   
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java
>  5869910cb 
>   intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java 
> PRE-CREATION 
>   notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
>  c45a1da95 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  3d8d9cc0a 
>   
> notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
>  0c92c300e 
>   notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java 
> 1ae7c278c 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
>  8078a6ceb 
>   
> notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationDeserializerTest.java
>  13eafb6b1 
>   
> notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java
>  d04817024 
> 
> 
> Diff: https://reviews.apache.org/r/73595/diff/6/
> 
> 
> Testing
> ---
> 
> mvn clean install
> mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
> testcases running successfully
> 
> 
> Thanks,
> 
> chaitali
> 
>

Re: Any community support for spark-atlas-connector?

[Ashutosh Mestry]

Hi

Thank you for your interest in Atlas.

To answer your questions:

  *   Spark Atlas Connector is not being developed. Right now we don’t have any 
plans to support Spark natively.
  *   About support for Spark 3.0: We don’t have plans for it as of now.

Having said that, we will welcome contributions from the community for 
improving SAC.

Best regards,

~ ashutosh
Ashutosh Mestry<mailto:ames...@cloudera.com> . Staff Software Engineer . 
Cloudera, Inc
...
No hurry, no pause. – Tim Ferriss, Life Hacker, Author


From: Jiaxin Ye 
Date: Monday, October 4, 2021 at 9:59 AM
To: u...@atlas.apache.org , dev@atlas.apache.org 

Subject: Any community support for spark-atlas-connector?
Hi All,

I saw the git repo for the spark-atlas-connector
<https://github.com/hortonworks-spark/spark-atlas-connector> but it hasn't
been updated in the past two years. Is there any plan to support spark in
Atlas natively? Is the spark-atlas-connector
<https://github.com/hortonworks-spark/spark-atlas-connector> able to
support spark 3.0?

Thanks,
Jiaxin

Re: Review Request 73628: ATLAS-4444: Atlas - Upgrade commons-io to 2.11.0

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73628/#review223573
---



https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/874/

- Ashutosh Mestry


On Oct. 5, 2021, 4:02 p.m., Disha Talreja wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73628/
> ---
> 
> (Updated Oct. 5, 2021, 4:02 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry and Sarath Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch upgrades commons-io 2.8 to 2.11.0
> 
> 
> Diffs
> -
> 
>   pom.xml 47768e9fa 
> 
> 
> Diff: https://reviews.apache.org/r/73628/diff/1/
> 
> 
> Testing
> ---
> 
> 1. Tested locally
> 2. Patched a cluster
> 
> 
> Thanks,
> 
> Disha Talreja
> 
>

Re: Review Request 73628: ATLAS-4444: Atlas - Upgrade commons-io to 2.11.0

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73628/#review223572
---


Ship it!




Ship It!

- Ashutosh Mestry


On Oct. 5, 2021, 4:02 p.m., Disha Talreja wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73628/
> ---
> 
> (Updated Oct. 5, 2021, 4:02 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry and Sarath Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch upgrades commons-io 2.8 to 2.11.0
> 
> 
> Diffs
> -
> 
>   pom.xml 47768e9fa 
> 
> 
> Diff: https://reviews.apache.org/r/73628/diff/1/
> 
> 
> Testing
> ---
> 
> 1. Tested locally
> 2. Patched a cluster
> 
> 
> Thanks,
> 
> Disha Talreja
> 
>

Re: Review Request 73286: Initial commit

[Ashutosh Mestry via Review Board]

> On Oct. 4, 2021, 3:42 p.m., Ashutosh Mestry wrote:
> > Can you please rebase this against latest master?
> 
> Aileen Toleikis wrote:
> Hi Ashutosh!
> We rebased our code to the latest master!
> And we also added schema evolution to the code, therefore there are more 
> new codelines in the diff.
> How do we proceed from here?
> Kind regards,
> Aileen

Thanks! I have asked Viktor from our Kafka team to review the changes.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73286/#review223560
---


On Oct. 5, 2021, 8:06 a.m., Aileen Toleikis wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73286/
> ---
> 
> (Updated Oct. 5, 2021, 8:06 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> The Kafka Community is using Schema Registry more and more heavily but as 
> Atlas is currently unaware of this, this extension helps Atlas make use of 
> the Schemas.
> 
> We have tested this extension and we have production environments where Atlas 
> will not be allowed without schema registry access. We have received feedback 
> that this extension would be sufficient to allow production use.
> 
> Changes:
> 
> initial commit
> 
> 
> added schema creation in Atlas
> 
> 
> added Schema creation, Schema linkage and started to work on versions
> 
> 
> avro_field creation is working. However, there is no connection to schemas yet
> 
> 
> smaller refactorings and improved logging
> 
> 
> added field reference in schemas
> 
> 
> deleted second README and added gitignore
> 
> 
> Smaller changes after rebasing on new version
> 
> 
> moved HTTP REST calls to a separate class (SchemaRegistryConnector.java)
> 
> 
> added testCreateSchema() as test case
> 
> 
> added testCreateField() and testUpdateField() as test case
> 
> 
> added testGetSchemas() and testGetSchemaVersions() as test case
> 
> 
> added testUpdateSchema() as test case
> 
> 
> removed unnecessary comments
> 
> 
> Diffs
> -
> 
>   addons/kafka-bridge/pom.xml 2ac19fd20e0c0322b0d104641ddd5a2ef89bf9d0 
>   
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
>  f9548244364a8f79f346411739348d0d53298c99 
>   
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java
>  PRE-CREATION 
>   
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java
>  0f81b4c37a1d8f84c587a914eb460b290ae36987 
>   
> addons/kafka-bridge/src/test/java/org/apache/atlas/kafka/bridge/KafkaBridgeTest.java
>  f86ceb58fd250d80f3ae3c71085c8c0adbc9116b 
>   pom.xml 47768e9fa7d8abcfc81cd148917400ac478a333a 
> 
> 
> Diff: https://reviews.apache.org/r/73286/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Aileen Toleikis
> 
>

Re: Review Request 73286: Initial commit

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73286/#review223560
---



Can you please rebase this against latest master?

- Ashutosh Mestry


On Sept. 3, 2021, 7:42 a.m., Aileen Toleikis wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73286/
> ---
> 
> (Updated Sept. 3, 2021, 7:42 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> The Kafka Community is using Schema Registry more and more heavily but as 
> Atlas is currently unaware of this, this extension helps Atlas make use of 
> the Schemas.
> 
> We have tested this extension and we have production environments where Atlas 
> will not be allowed without schema registry access. We have received feedback 
> that this extension would be sufficient to allow production use.
> 
> Changes:
> 
> initial commit
> 
> 
> added schema creation in Atlas
> 
> 
> added Schema creation, Schema linkage and started to work on versions
> 
> 
> avro_field creation is working. However, there is no connection to schemas yet
> 
> 
> smaller refactorings and improved logging
> 
> 
> added field reference in schemas
> 
> 
> deleted second README and added gitignore
> 
> 
> Smaller changes after rebasing on new version
> 
> 
> moved HTTP REST calls to a separate class (SchemaRegistryConnector.java)
> 
> 
> added testCreateSchema() as test case
> 
> 
> added testCreateField() and testUpdateField() as test case
> 
> 
> added testGetSchemas() and testGetSchemaVersions() as test case
> 
> 
> added testUpdateSchema() as test case
> 
> 
> removed unnecessary comments
> 
> 
> Diffs
> -
> 
>   addons/kafka-bridge/pom.xml 7fe97eb39897e32f5e2a4104cc06948e1ec48b27 
>   
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
>  bf74c67454120e3bdeebab0d7cc428d6bba8c021 
>   
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java
>  PRE-CREATION 
>   
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java
>  0f81b4c37a1d8f84c587a914eb460b290ae36987 
>   
> addons/kafka-bridge/src/test/java/org/apache/atlas/kafka/bridge/KafkaBridgeTest.java
>  f86ceb58fd250d80f3ae3c71085c8c0adbc9116b 
>   pom.xml 092ba09cee93ca703e29d8ae1afaa20559a06227 
> 
> 
> Diff: https://reviews.apache.org/r/73286/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Aileen Toleikis
> 
>

[jira] [Commented] (ATLAS-4440) Upgrade Atlas's Kafka dependency to 2.8

[Ashutosh Mestry (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-4440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17423997#comment-17423997
 ] 

Ashutosh Mestry commented on ATLAS-4440:


+1 for the patch.

[~chaitali] / [~dishatalreja]: Can you please review.

[~sarath]: Please sign-off.

> Upgrade Atlas's Kafka dependency to 2.8
> ---
>
> Key: ATLAS-4440
> URL: https://issues.apache.org/jira/browse/ATLAS-4440
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Reporter: Viktor Somogyi-Vass
>Assignee: Viktor Somogyi-Vass
>Priority: Major
> Attachments: 0001-ATLAS-4440-Update-Kafka-dependency-to-2.8.1.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-4246) Make Kafka Interface aware of Kafka Schema Registry

[Ashutosh Mestry (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17423993#comment-17423993
 ] 

Ashutosh Mestry commented on ATLAS-4246:


[~aileeen] I have assigned this ticket to Viktor. I will follow up with him.

> Make Kafka Interface aware of Kafka Schema Registry
> ---
>
> Key: ATLAS-4246
> URL: https://issues.apache.org/jira/browse/ATLAS-4246
> Project: Atlas
>  Issue Type: Improvement
>  Components: kafka-integration
>Affects Versions: 2.1.0, 3.0.0
>Reporter: Aileen Toleikis
>Assignee: Viktor Somogyi-Vass
>Priority: Major
>  Labels: Kafka
> Fix For: 3.0.0, 2.3.0
>
>
> Kafka Community is using Schema Registry more and more heavily but as Atlas 
> is currently unaware of this, this extension helps Atlas make use of the 
> Schemas.
>  
> We have tested this extension and we have production environments where Atlas 
> will not be allowed without schema registry access. We have received feedback 
> that this extension would be sufficient to allow production use.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (ATLAS-4246) Make Kafka Interface aware of Kafka Schema Registry

[Ashutosh Mestry (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry reassigned ATLAS-4246:
--

Assignee: Viktor Somogyi-Vass

> Make Kafka Interface aware of Kafka Schema Registry
> ---
>
> Key: ATLAS-4246
> URL: https://issues.apache.org/jira/browse/ATLAS-4246
> Project: Atlas
>  Issue Type: Improvement
>  Components: kafka-integration
>Affects Versions: 2.1.0, 3.0.0
>Reporter: Aileen Toleikis
>Assignee: Viktor Somogyi-Vass
>Priority: Major
>  Labels: Kafka
> Fix For: 3.0.0, 2.3.0
>
>
> Kafka Community is using Schema Registry more and more heavily but as Atlas 
> is currently unaware of this, this extension helps Atlas make use of the 
> Schemas.
>  
> We have tested this extension and we have production environments where Atlas 
> will not be allowed without schema registry access. We have received feedback 
> that this extension would be sufficient to allow production use.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 73586: ATLAS-4424: Enhance the Import hive utility to create export zip files and run bulk import

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73586/#review223551
---




addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java
Lines 130 (patched)
<https://reviews.apache.org/r/73586/#comment312629>

Rename: qualifiedNameGuidMap



addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java
Lines 300 (patched)
<https://reviews.apache.org/r/73586/#comment312626>

A get in to map with db as key should simplify this.



addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java
Lines 300 (patched)
<https://reviews.apache.org/r/73586/#comment312627>

A get in to map with db as key should simplify this.



addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java
Lines 415 (patched)
<https://reviews.apache.org/r/73586/#comment312628>

This will make large number of calls to Atlas thereby slowing the operation 
down.



addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java
Lines 529 (patched)
<https://reviews.apache.org/r/73586/#comment312630>

This method converts Table to AtlasEntity and writes to stream. Consider 
separating it in 2 distinct methods. Say 
AtlasEntityWithExtInfo toAtlasEntity(Table table) and write(OutputStream 
stream, AtlasEntityWithExtInfo entity)

The write method will be reused by all other operations.



intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java
Line 56 (original)
<https://reviews.apache.org/r/73586/#comment312620>

This is not necessary.



repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java
Lines 90 (patched)
<https://reviews.apache.org/r/73586/#comment312623>

What is the reason for this?



repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java
Lines 180 (patched)
<https://reviews.apache.org/r/73586/#comment312622>

This is not necessary.



repository/src/test/java/org/apache/atlas/repository/impexp/DataMigrationStatusServiceTest.java
Line 46 (original), 48 (patched)
<https://reviews.apache.org/r/73586/#comment312621>

What is the reason for doing this?


- Ashutosh Mestry


On Sept. 15, 2021, 11 p.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73586/
> ---
> 
> (Updated Sept. 15, 2021, 11 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4424
> https://issues.apache.org/jira/browse/ATLAS-4424
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-4424: Enhanced the Import hive utility to create export zip files and 
> run bulk import
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/src/bin/import-hive.sh fde486f28 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  08471b480 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeV2.java
>  PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
> 2c1870444 
>   
> intg/src/main/java/org/apache/atlas/model/migration/MigrationImportStatus.java
>  3430fdae8 
>   
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationStatusService.java
>  a22c68790 
>   
> repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java
>  d56261f78 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/DataMigrationStatusServiceTest.java
>  f1dc990e8 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
> baa040f4c 
> 
> 
> Diff: https://reviews.apache.org/r/73586/diff/1/
> 
> 
> Testing
> ---
> 
> Manually tested the new zip file creation and bulk import to atlas
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>

[jira] [Commented] (ATLAS-4440) Upgrade Atlas's Kafka dependency to 2.8

[Ashutosh Mestry (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-4440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17423340#comment-17423340
 ] 

Ashutosh Mestry commented on ATLAS-4440:


PC Build: 
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/871/

> Upgrade Atlas's Kafka dependency to 2.8
> ---
>
> Key: ATLAS-4440
> URL: https://issues.apache.org/jira/browse/ATLAS-4440
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Reporter: Viktor Somogyi-Vass
>Assignee: Viktor Somogyi-Vass
>Priority: Major
> Attachments: 0001-ATLAS-4440-Update-Kafka-dependency-to-2.8.1.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 73585: ATLAS-4425: Migration import should be able to import multiple zip files present in a particular path

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73585/#review223550
---



User specifies:
atlas.migration.data.filename=/tmp/atlas-data/golden-?-of-2-atlas.zip

The directory contains:
golden-1-of-2-atlas.zip
golden-2-of-2-atlas.zip


Algo:
- RW permissions for /tmp/atlas-data/archive.
  - If permissions exist, sucessfully imported files are moved to archive 
directory.
  - If not, LOG.WARN is displayed. Implication of this is the subsequent 
startup in migration mode will restart import from the beginning. 
- - Wildcard gets expanded.
- Files are enumerated.
- They are sequentially fed to the importer.
- Successful files are moved to /tmp/atlas-data/archive folder.

- Ashutosh Mestry


On Sept. 23, 2021, 3:52 a.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73585/
> ---
> 
> (Updated Sept. 23, 2021, 3:52 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4425
> https://issues.apache.org/jira/browse/ATLAS-4425
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-442: Added support for Migration import to run on a directory with 
> multip[le zip files
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceDirect.java
>  04342fa52 
>   
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
>  0a2257eae 
>   
> repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java
>  d56261f78 
> 
> 
> Diff: https://reviews.apache.org/r/73585/diff/2/
> 
> 
> Testing
> ---
> 
> Manually tested with multiple zip files at a location which was imported 
> successfully.
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>

Re: Review Request 73585: ATLAS-4425: Migration import should be able to import multiple zip files present in a particular path

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73585/#review223548
---



I have tested this and it works quite well.


repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceDirect.java
Lines 260 (patched)
<https://reviews.apache.org/r/73585/#comment312619>

I think this change is not necessary given that the input will always have 
a typeDefinition.



repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
Line 68 (original), 68 (patched)
<https://reviews.apache.org/r/73585/#comment312613>

Doing this will impact branch-0.8 to 1.0 migration. The old and new 
migration relies on directory vs file existence check.



repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
Lines 93 (patched)
<https://reviews.apache.org/r/73585/#comment312614>

This won't be necessary as well.



repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java
Line 61 (original), 66 (patched)
<https://reviews.apache.org/r/73585/#comment312615>

If the file contains wildcards (? or '*'), expand them using listFiles.



repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java
Line 62 (original), 67 (patched)
<https://reviews.apache.org/r/73585/#comment312618>

Add check to detect archive folder can be created. If not LOG.WARN and 
proceed without archieving.



repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java
Lines 94 (patched)
<https://reviews.apache.org/r/73585/#comment312616>

Modify this to take the user-specified input, expand if necessary. Sort it 
and then return the list of files to be imported.



repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java
Lines 135 (patched)
<https://reviews.apache.org/r/73585/#comment312617>

Be defensive here.

if (this.archiveDir == null) {
return;
}

    Doing this will add clarity.


- Ashutosh Mestry


On Sept. 23, 2021, 3:52 a.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73585/
> ---
> 
> (Updated Sept. 23, 2021, 3:52 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4425
> https://issues.apache.org/jira/browse/ATLAS-4425
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-442: Added support for Migration import to run on a directory with 
> multip[le zip files
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceDirect.java
>  04342fa52 
>   
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
>  0a2257eae 
>   
> repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java
>  d56261f78 
> 
> 
> Diff: https://reviews.apache.org/r/73585/diff/2/
> 
> 
> Testing
> ---
> 
> Manually tested with multiple zip files at a location which was imported 
> successfully.
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>

Re: Review Request 73595: ATLAS-4428 : Enhance Atlas hook messages to capture hook version & message source

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73595/#review223547
---



Why would you not include messageInfo in the message itself? That will make the 
message self-sufficient and prevent the ctor changes.


intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java
Lines 54 (patched)
<https://reviews.apache.org/r/73595/#comment312611>

Refactor: Extract method. Call it: getBuildVersion



intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java
Lines 80 (patched)
<https://reviews.apache.org/r/73595/#comment312612>

Since build version will remain the same for a deployment, it will help to 
store this into a static property.


- Ashutosh Mestry


On Sept. 29, 2021, 7:30 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73595/
> ---
> 
> (Updated Sept. 29, 2021, 7:30 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4428
> https://issues.apache.org/jira/browse/ATLAS-4428
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas hook messages doesn't specify the source information of the messages 
> coming i.e. which hook source - HiveServer2, HiveMetastore, Impala, Spark, 
> HBase, Kafka.
> 
> Each hook message already captures, the msgSourceIP, msgCreatedBy and 
> msgCreationTime info. In addition to these, we should capture the msgSource 
> information as well.
> 
> When messages come from different hook versions, Atlas server should be aware 
> of the version of different hook messages to process the message based on the 
> hook version.
> 
> 
> Diffs
> -
> 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java
>  8c09d3394 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  0ab06f2c5 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 3cc7b3bc6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveMetastoreHookImpl.java
>  6a492c214 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  10ae08f6d 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> 4785960a7 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  779c5cb20 
>   intg/pom.xml 05914 
>   
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationBaseMessage.java
>  ff45d5713 
>   
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java
>  5869910cb 
>   intg/src/main/java/org/apache/atlas/model/notification/MessageSource.java 
> PRE-CREATION 
>   notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
>  c45a1da95 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  3d8d9cc0a 
>   
> notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
>  0c92c300e 
>   notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java 
> 1ae7c278c 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
>  8078a6ceb 
>   
> notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationDeserializerTest.java
>  13eafb6b1 
>   
> notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java
>  d04817024 
> 
> 
> Diff: https://reviews.apache.org/r/73595/diff/3/
> 
> 
> Testing
> ---
> 
> mvn clean install
> mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
> testcases running successfully
> 
> 
> Thanks,
> 
> chaitali
> 
>

[jira] [Resolved] (ATLAS-4358) Mapping for some internal Atlas attributes ( like __patch.type , __timestamp, etc) does not exist in Elasticsearch

[Ashutosh Mestry (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry resolved ATLAS-4358.

Resolution: Fixed

> Mapping for some internal Atlas attributes ( like __patch.type , __timestamp, 
> etc) does not exist in Elasticsearch
> --
>
> Key: ATLAS-4358
> URL: https://issues.apache.org/jira/browse/ATLAS-4358
> Project: Atlas
>  Issue Type: Bug
>Reporter: Anshul Mehta
>    Assignee: Ashutosh Mestry
>Priority: Major
> Attachments: patch-manager-fix.patch
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> *Impact of the problem -*
>  * Atlas pod taking much longer to become active and this time keep 
> increasing as the assets increase. This basically means a downtime on every 
> Atlas release.
>  * Not able to filter via basic search on attributes like {{__timestamp}} , 
> {{__modificationTimestamp}} , {{createdBy}} and {{modifiedBy}} .
> *Issue -*
> So just before creating the mapping in the mixed index (ES index) Atlas 
> creates something called {{propertyKey}} and this propertyKey is used to 
> create the mapping. The code is written in a way that checks if propertyKey 
> for the current property is null or not. If it is null it creates the 
> propertyKey and then adds it to mixed index. If it is not Null it assumes 
> that the property has already been added to the index and so skips adding it.
> Now in our case when Atlas checked the propertyKey it was not null (which 
> should not have been the case) therefore Atlas skipped adding it to the mixed 
> index and so these properties never got added to the mixed index. This 
> basically meant propertyKey for these properties were getting created 
> somewhere else. We looked into the entire codebase but could not find the use 
> of makePropertyKey method ( which is used to create propertyKey) or any other 
> similar method.
> Then I saw certain java patch vertices getting created even before these 
> internal attributes are added to various indices. Though these patches were 
> applied later once all internal attributes were added to all the indices.
> Now, these patch vertices have 9 attributes and we releaized these 9 
> attributes are the only attributes missing from ES. So basically when patch 
> vertices got created and these vertices with their attributes got added to 
> cassandra via janusgraph, janusgraph automatically created propertyKey for 
> all these attributes (the janusgraph's makePropertyKey method is not called 
> during this process anywhere in the Atlas code). And because internal 
> attributes were getting added to indices in another thread at the same time, 
> when code checked for propertyKey, it was not null and so it did not add the 
> property to the mixed index.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (ATLAS-4358) Mapping for some internal Atlas attributes ( like __patch.type , __timestamp, etc) does not exist in Elasticsearch

[Ashutosh Mestry (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry reassigned ATLAS-4358:
--

Assignee: Ashutosh Mestry

> Mapping for some internal Atlas attributes ( like __patch.type , __timestamp, 
> etc) does not exist in Elasticsearch
> --
>
> Key: ATLAS-4358
> URL: https://issues.apache.org/jira/browse/ATLAS-4358
> Project: Atlas
>  Issue Type: Bug
>Reporter: Anshul Mehta
>    Assignee: Ashutosh Mestry
>Priority: Major
> Attachments: patch-manager-fix.patch
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> *Impact of the problem -*
>  * Atlas pod taking much longer to become active and this time keep 
> increasing as the assets increase. This basically means a downtime on every 
> Atlas release.
>  * Not able to filter via basic search on attributes like {{__timestamp}} , 
> {{__modificationTimestamp}} , {{createdBy}} and {{modifiedBy}} .
> *Issue -*
> So just before creating the mapping in the mixed index (ES index) Atlas 
> creates something called {{propertyKey}} and this propertyKey is used to 
> create the mapping. The code is written in a way that checks if propertyKey 
> for the current property is null or not. If it is null it creates the 
> propertyKey and then adds it to mixed index. If it is not Null it assumes 
> that the property has already been added to the index and so skips adding it.
> Now in our case when Atlas checked the propertyKey it was not null (which 
> should not have been the case) therefore Atlas skipped adding it to the mixed 
> index and so these properties never got added to the mixed index. This 
> basically meant propertyKey for these properties were getting created 
> somewhere else. We looked into the entire codebase but could not find the use 
> of makePropertyKey method ( which is used to create propertyKey) or any other 
> similar method.
> Then I saw certain java patch vertices getting created even before these 
> internal attributes are added to various indices. Though these patches were 
> applied later once all internal attributes were added to all the indices.
> Now, these patch vertices have 9 attributes and we releaized these 9 
> attributes are the only attributes missing from ES. So basically when patch 
> vertices got created and these vertices with their attributes got added to 
> cassandra via janusgraph, janusgraph automatically created propertyKey for 
> all these attributes (the janusgraph's makePropertyKey method is not called 
> during this process anywhere in the Atlas code). And because internal 
> attributes were getting added to indices in another thread at the same time, 
> when code checked for propertyKey, it was not null and so it did not add the 
> property to the mixed index.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Review Request 73606: AtlasPatchManager: Avoid Race Condition in HA Mode

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73606/
---

Review request for atlas, Jayendra Parab, Mandar Ambawane, Nikhil Bonte, Nixon 
Rodrigues, and Sarath Subramanian.


Bugs: ATLAS-4358
https://issues.apache.org/jira/browse/ATLAS-4358


Repository: atlas


Description
---

**Problem**
During start-up in HA mode, the _AtlasPatchManager_ tries to read the patch 
data while another instanace of Atlas is trying to create the indexes.

This causes indexes not to be created correctly. When this happens, the patch 
registry ends up doing a a full database scan. With sizable data, this can 
result in long startup times. This is detremental for setup that use 
containerized cluster setup.

**Approach**
Initialize the _PatchContext_ in the _applyAll_ method.


Diffs
-

  
repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java
 e2a38ab52 


Diff: https://reviews.apache.org/r/73606/diff/1/


Testing
---

**PC Build**
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/864/


Thanks,

Ashutosh Mestry

Re: Review Request 73599: Conditionally Disable 'Session Inactivity Timeout'

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73599/
---

(Updated Sept. 22, 2021, 8:35 p.m.)


Review request for atlas, Nikhil Bonte, Nixon Rodrigues, Prasad Pawar, and 
Sarath Subramanian.


Changes
---

Updates include: Minor refactoring for clarity.


Bugs: ATLAS-4435
https://issues.apache.org/jira/browse/ATLAS-4435


Repository: atlas


Description
---

Please see JIRA for details.

Additional info:
* UI pop-up dialog looks at the value passed in the response of 'admin/session' 
REST endpoint. Not passing the configuration value, disables the feature from 
UI (Smart!).


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java fa519ef5a 
  
webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
 b8d21b9bb 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
baa040f4c 
  
webapp/src/main/java/org/apache/atlas/web/security/AtlasAuthenticationSuccessHandler.java
 1b1a80826 


Diff: https://reviews.apache.org/r/73599/diff/3/

Changes: https://reviews.apache.org/r/73599/diff/2-3/


Testing
---

Manual testing:
* Verified against Knox SSO enabled cluster.
* User-name password enabled cluster.


Thanks,

Ashutosh Mestry

Re: Review Request 73599: Conditionally Disable 'Session Inactivity Timeout'

[Ashutosh Mestry via Review Board]

> On Sept. 21, 2021, 7:02 p.m., Sailaja Polavarapu wrote:
> > webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
> > Line 359 (original), 374 (patched)
> > <https://reviews.apache.org/r/73599/diff/1/?file=2253162#file2253162line377>
> >
> > I think it will be intuitive to move logoutHandler != null check to 
> > inner if condition on line #375.

Please see my comment below.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73599/#review223509
-------


On Sept. 22, 2021, 5:42 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73599/
> ---
> 
> (Updated Sept. 22, 2021, 5:42 p.m.)
> 
> 
> Review request for atlas, Nikhil Bonte, Nixon Rodrigues, Prasad Pawar, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4435
> https://issues.apache.org/jira/browse/ATLAS-4435
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Please see JIRA for details.
> 
> Additional info:
> * UI pop-up dialog looks at the value passed in the response of 
> 'admin/session' REST endpoint. Not passing the configuration value, disables 
> the feature from UI (Smart!).
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java fa519ef5a 
>   
> webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
>  b8d21b9bb 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
> baa040f4c 
>   
> webapp/src/main/java/org/apache/atlas/web/security/AtlasAuthenticationSuccessHandler.java
>  1b1a80826 
> 
> 
> Diff: https://reviews.apache.org/r/73599/diff/2/
> 
> 
> Testing
> ---
> 
> Manual testing:
> * Verified against Knox SSO enabled cluster.
> * User-name password enabled cluster.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 73599: Conditionally Disable 'Session Inactivity Timeout'

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73599/
---

(Updated Sept. 22, 2021, 5:42 p.m.)


Review request for atlas, Nikhil Bonte, Nixon Rodrigues, Prasad Pawar, and 
Sarath Subramanian.


Changes
---

Updates include: 
- Addressed review comments.
- Minor refactoring.


Bugs: ATLAS-4435
https://issues.apache.org/jira/browse/ATLAS-4435


Repository: atlas


Description
---

Please see JIRA for details.

Additional info:
* UI pop-up dialog looks at the value passed in the response of 'admin/session' 
REST endpoint. Not passing the configuration value, disables the feature from 
UI (Smart!).


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java fa519ef5a 
  
webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
 b8d21b9bb 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
baa040f4c 
  
webapp/src/main/java/org/apache/atlas/web/security/AtlasAuthenticationSuccessHandler.java
 1b1a80826 


Diff: https://reviews.apache.org/r/73599/diff/2/

Changes: https://reviews.apache.org/r/73599/diff/1-2/


Testing
---

Manual testing:
* Verified against Knox SSO enabled cluster.
* User-name password enabled cluster.


Thanks,

Ashutosh Mestry

Re: Review Request 73599: Conditionally Disable 'Session Inactivity Timeout'

[Ashutosh Mestry via Review Board]

> On Sept. 22, 2021, 4:58 p.m., Sarath Subramanian wrote:
> > webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
> > Line 311 (original), 325 (patched)
> > <https://reviews.apache.org/r/73599/diff/1/?file=2253162#file2253162line328>
> >
> > would 'sessionTimeout' be initialized by then? I see it getting 
> > initialized only at getConfiguration(), so won't it be always take default 
> > ('0'). line 326 will never be reached.

This follows the AuthenticationFilter initialization path. All our secrets 
(which are also part of the config) get initialized here. In short, yes, it 
does get initialized.


> On Sept. 22, 2021, 4:58 p.m., Sarath Subramanian wrote:
> > webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
> > Line 360 (original), 375 (patched)
> > <https://reviews.apache.org/r/73599/diff/1/?file=2253162#file2253162line378>
> >
> > add null check ('logoutHandler') to inner 'if' - closer to where it is 
> > used for better readabality.

Doing this will change the flow. I want to be able to resort to the state where 
this code did not exist. Hence I have made it this way.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73599/#review223517
---


On Sept. 21, 2021, 6:38 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73599/
> ---
> 
> (Updated Sept. 21, 2021, 6:38 p.m.)
> 
> 
> Review request for atlas, Nikhil Bonte, Nixon Rodrigues, Prasad Pawar, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4435
> https://issues.apache.org/jira/browse/ATLAS-4435
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Please see JIRA for details.
> 
> Additional info:
> * UI pop-up dialog looks at the value passed in the response of 
> 'admin/session' REST endpoint. Not passing the configuration value, disables 
> the feature from UI (Smart!).
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java fa519ef5a 
>   
> webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
>  b8d21b9bb 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
> baa040f4c 
>   
> webapp/src/main/java/org/apache/atlas/web/security/AtlasAuthenticationSuccessHandler.java
>  1b1a80826 
> 
> 
> Diff: https://reviews.apache.org/r/73599/diff/1/
> 
> 
> Testing
> ---
> 
> Manual testing:
> * Verified against Knox SSO enabled cluster.
> * User-name password enabled cluster.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 73589: Indexing Monitor and Recovery

[Ashutosh Mestry via Review Board]

> On Sept. 21, 2021, 7:35 p.m., Sarath Subramanian wrote:
> > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
> > Lines 215 (patched)
> > <https://reviews.apache.org/r/73589/diff/1/?file=2252880#file2252880line217>
> >
> > in that case - only if user explicitly sets the config in atlas 
> > application properties to true/false we should change the global config.
> > 
> > It takes a default value here and always updates the global config even 
> > if user haven't set it.
> > 
> > global config is a site-wide setting for Atlas/Janusgraph and should 
> > not be changed unless explicitly specified.

E.g.: User sets the config then removes it. Unless we maintain a state across 
restarts, we won't know.

Another option is to read existing value and match it against incoming and 
update it only if there is a change.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73589/#review223508
-------


On Sept. 21, 2021, 3:40 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73589/
> ---
> 
> (Updated Sept. 21, 2021, 3:40 p.m.)
> 
> 
> Review request for atlas, Nikhil Bonte, Nixon Rodrigues, Radhika Kundam, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4408
> https://issues.apache.org/jira/browse/ATLAS-4408
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This implementation builds up on Radhika's earlier implementation. 
> https://reviews.apache.org/r/73555/
> 
> Indexing
> 
> atlas.graph.index.status.check.frequency: Frequence of checking Solr for 
> health status.
> atlas.graph.index.recovery.start.time: User can set recovery time in UTC. 
> E.g. 
> ```
> atlas.graph.index.recovery.start.time=2014-12-11T02:35:58.440Z
> ```
> 
> **Implementation**
> New: _updateGlobalConfiguration_: Updating global configuration variables.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java aea0c1311 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndexClient.java
>  9960d8909 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  7e3b2f440 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
>  11267c438 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphIndexClient.java
>  ef42dbd3d 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
>  1cc7f8b55 
>   intg/src/main/java/org/apache/atlas/ApplicationProperties.java 682206dad 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  ddfb0089e 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/IndexRecoveryService.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/RecoveryInfoManagementTest.java
>  PRE-CREATION 
>   
> server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java
>  ba8f0886c 
> 
> 
> Diff: https://reviews.apache.org/r/73589/diff/2/
> 
> 
> Testing
> ---
> 
> Additional unit tests.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 73595: ATLAS-4428 : Enhance Atlas hook messages to capture hook version & message source

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73595/#review223510
---




intg/src/main/java/org/apache/atlas/model/notification/HookInfo.java
Lines 51 (patched)
<https://reviews.apache.org/r/73595/#comment312590>

Since this is a model, it is used for moving data around. Populate an 
instance of this via a helper method.



intg/src/main/java/org/apache/atlas/model/notification/HookInfo.java
Lines 54 (patched)
<https://reviews.apache.org/r/73595/#comment312591>

Refactor: Extract to field.


- Ashutosh Mestry


On Sept. 21, 2021, 10:23 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73595/
> ---
> 
> (Updated Sept. 21, 2021, 10:23 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Pinal Shah, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4428
> https://issues.apache.org/jira/browse/ATLAS-4428
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas hook messages doesn't specify the source information of the messages 
> coming i.e. which hook source - HiveServer2, HiveMetastore, Impala, Spark, 
> HBase, Kafka.
> 
> Each hook message already captures, the msgSourceIP, msgCreatedBy and 
> msgCreationTime info. In addition to these, we should capture the msgSource 
> information as well.
> 
> When messages come from different hook versions, Atlas server should be aware 
> of the version of different hook messages to process the message based on the 
> hook version.
> 
> 
> Diffs
> -
> 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java
>  8c09d3394 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  0ab06f2c5 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 3cc7b3bc6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveMetastoreHookImpl.java
>  6a492c214 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  10ae08f6d 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> 4785960a7 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  779c5cb20 
>   intg/pom.xml 05914 
>   
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationBaseMessage.java
>  ff45d5713 
>   
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java
>  5869910cb 
>   intg/src/main/java/org/apache/atlas/model/notification/HookInfo.java 
> PRE-CREATION 
>   notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
>  c45a1da95 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  3d8d9cc0a 
>   
> notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
>  0c92c300e 
>   notification/src/test/java/org/apache/atlas/hook/AtlasHookTest.java 
> 1ae7c278c 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
>  8078a6ceb 
>   
> notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationDeserializerTest.java
>  13eafb6b1 
>   
> notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java
>  d04817024 
> 
> 
> Diff: https://reviews.apache.org/r/73595/diff/2/
> 
> 
> Testing
> ---
> 
> mvn clean install
> mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip
> testcases running successfully
> 
> 
> Thanks,
> 
> chaitali
> 
>

Review Request 73599: Conditionally Disable 'Session Inactivity Timeout'

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73599/
---

Review request for atlas, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.


Bugs: ATLAS-4435
https://issues.apache.org/jira/browse/ATLAS-4435


Repository: atlas


Description
---

Please see JIRA for details.


Diffs
-

  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java fa519ef5a 
  
webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
 b8d21b9bb 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
baa040f4c 
  
webapp/src/main/java/org/apache/atlas/web/security/AtlasAuthenticationSuccessHandler.java
 1b1a80826 


Diff: https://reviews.apache.org/r/73599/diff/1/


Testing
---

Manual testing:
* Verified against Knox SSO enabled cluster.
* User-name password enabled cluster.


Thanks,

Ashutosh Mestry

[jira] [Updated] (ATLAS-4435) Session Inactivity Timeout: Provide Ability to Disable Session Inactivity Timeout Implementation

[Ashutosh Mestry (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-4435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry updated ATLAS-4435:
---
Attachment: ATLAS-4435-session-timeout-disable.patch

> Session Inactivity Timeout: Provide Ability to Disable Session Inactivity 
> Timeout Implementation
> 
>
> Key: ATLAS-4435
> URL: https://issues.apache.org/jira/browse/ATLAS-4435
> Project: Atlas
>  Issue Type: Improvement
>Affects Versions: trunk
>    Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Fix For: trunk
>
> Attachments: ATLAS-4435-session-timeout-disable.patch
>
>
> *Background*
> The ATLAS-4379 implemented 'Session Inactivity Timeout'.
> *Requirement*
> User should have ability to disable this when needed.
> *Solution*
> The property 'atlas.session.timeout.secs' lets user set timeout value. 
> Setting this to -1 will disable the feature.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (ATLAS-4435) Session Inactivity Timeout: Provide Ability to Disable Session Inactivity Timeout Implementation

[Ashutosh Mestry (Jira)]

Ashutosh Mestry created ATLAS-4435:
--

 Summary: Session Inactivity Timeout: Provide Ability to Disable 
Session Inactivity Timeout Implementation
 Key: ATLAS-4435
 URL: https://issues.apache.org/jira/browse/ATLAS-4435
 Project: Atlas
  Issue Type: Improvement
Affects Versions: trunk
Reporter: Ashutosh Mestry
Assignee: Ashutosh Mestry
 Fix For: trunk


*Background*

The ATLAS-4379 implemented 'Session Inactivity Timeout'.

*Requirement*

User should have ability to disable this when needed.

*Solution*

The property 'atlas.session.timeout.secs' lets user set timeout value. Setting 
this to -1 will disable the feature.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 73589: Indexing Monitor and Recovery

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73589/
---

(Updated Sept. 21, 2021, 3:40 p.m.)


Review request for atlas, Nikhil Bonte, Nixon Rodrigues, Radhika Kundam, and 
Sarath Subramanian.


Changes
---

Updates include: 
- Addressed review comments.
- Added description.


Summary (updated)
-

Indexing Monitor and Recovery


Bugs: ATLAS-4408
https://issues.apache.org/jira/browse/ATLAS-4408


Repository: atlas


Description (updated)
---

This implementation builds up on Radhika's earlier implementation. 
https://reviews.apache.org/r/73555/

Indexing

atlas.graph.index.status.check.frequency: Frequence of checking Solr for health 
status.
atlas.graph.index.recovery.start.time: User can set recovery time in UTC. E.g. 
```
atlas.graph.index.recovery.start.time=2014-12-11T02:35:58.440Z
```

**Implementation**
New: _updateGlobalConfiguration_: Updating global configuration variables.


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/repository/Constants.java aea0c1311 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndexClient.java
 9960d8909 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 7e3b2f440 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
 11267c438 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphIndexClient.java
 ef42dbd3d 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
 1cc7f8b55 
  intg/src/main/java/org/apache/atlas/ApplicationProperties.java 682206dad 
  
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
 ddfb0089e 
  
repository/src/main/java/org/apache/atlas/repository/graph/IndexRecoveryService.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/graph/RecoveryInfoManagementTest.java
 PRE-CREATION 
  
server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java
 ba8f0886c 


Diff: https://reviews.apache.org/r/73589/diff/2/

Changes: https://reviews.apache.org/r/73589/diff/1-2/


Testing (updated)
---

Additional unit tests.


Thanks,

Ashutosh Mestry

[jira] [Updated] (ATLAS-4358) Mapping for some internal Atlas attributes ( like __patch.type , __timestamp, etc) does not exist in Elasticsearch

[Ashutosh Mestry (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry updated ATLAS-4358:
---
Attachment: patch-manager-fix.patch

> Mapping for some internal Atlas attributes ( like __patch.type , __timestamp, 
> etc) does not exist in Elasticsearch
> --
>
> Key: ATLAS-4358
> URL: https://issues.apache.org/jira/browse/ATLAS-4358
> Project: Atlas
>  Issue Type: Bug
>Reporter: Anshul Mehta
>Priority: Major
> Attachments: patch-manager-fix.patch
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> *Impact of the problem -*
>  * Atlas pod taking much longer to become active and this time keep 
> increasing as the assets increase. This basically means a downtime on every 
> Atlas release.
>  * Not able to filter via basic search on attributes like {{__timestamp}} , 
> {{__modificationTimestamp}} , {{createdBy}} and {{modifiedBy}} .
> *Issue -*
> So just before creating the mapping in the mixed index (ES index) Atlas 
> creates something called {{propertyKey}} and this propertyKey is used to 
> create the mapping. The code is written in a way that checks if propertyKey 
> for the current property is null or not. If it is null it creates the 
> propertyKey and then adds it to mixed index. If it is not Null it assumes 
> that the property has already been added to the index and so skips adding it.
> Now in our case when Atlas checked the propertyKey it was not null (which 
> should not have been the case) therefore Atlas skipped adding it to the mixed 
> index and so these properties never got added to the mixed index. This 
> basically meant propertyKey for these properties were getting created 
> somewhere else. We looked into the entire codebase but could not find the use 
> of makePropertyKey method ( which is used to create propertyKey) or any other 
> similar method.
> Then I saw certain java patch vertices getting created even before these 
> internal attributes are added to various indices. Though these patches were 
> applied later once all internal attributes were added to all the indices.
> Now, these patch vertices have 9 attributes and we releaized these 9 
> attributes are the only attributes missing from ES. So basically when patch 
> vertices got created and these vertices with their attributes got added to 
> cassandra via janusgraph, janusgraph automatically created propertyKey for 
> all these attributes (the janusgraph's makePropertyKey method is not called 
> during this process anywhere in the Atlas code). And because internal 
> attributes were getting added to indices in another thread at the same time, 
> when code checked for propertyKey, it was not null and so it did not add the 
> property to the mixed index.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-4358) Mapping for some internal Atlas attributes ( like __patch.type , __timestamp, etc) does not exist in Elasticsearch

[Ashutosh Mestry (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17418168#comment-17418168
 ] 

Ashutosh Mestry commented on ATLAS-4358:


[~mehtaanshul] Would it be possible for you to try out the attached patch? It 
integrates your fix and an additional fix. We had a similar problem in one of 
our customer environments. I am not able to duplicate it in my lab setup. The 
attached fix can address the problem.

> Mapping for some internal Atlas attributes ( like __patch.type , __timestamp, 
> etc) does not exist in Elasticsearch
> --
>
> Key: ATLAS-4358
> URL: https://issues.apache.org/jira/browse/ATLAS-4358
> Project: Atlas
>  Issue Type: Bug
>Reporter: Anshul Mehta
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> *Impact of the problem -*
>  * Atlas pod taking much longer to become active and this time keep 
> increasing as the assets increase. This basically means a downtime on every 
> Atlas release.
>  * Not able to filter via basic search on attributes like {{__timestamp}} , 
> {{__modificationTimestamp}} , {{createdBy}} and {{modifiedBy}} .
> *Issue -*
> So just before creating the mapping in the mixed index (ES index) Atlas 
> creates something called {{propertyKey}} and this propertyKey is used to 
> create the mapping. The code is written in a way that checks if propertyKey 
> for the current property is null or not. If it is null it creates the 
> propertyKey and then adds it to mixed index. If it is not Null it assumes 
> that the property has already been added to the index and so skips adding it.
> Now in our case when Atlas checked the propertyKey it was not null (which 
> should not have been the case) therefore Atlas skipped adding it to the mixed 
> index and so these properties never got added to the mixed index. This 
> basically meant propertyKey for these properties were getting created 
> somewhere else. We looked into the entire codebase but could not find the use 
> of makePropertyKey method ( which is used to create propertyKey) or any other 
> similar method.
> Then I saw certain java patch vertices getting created even before these 
> internal attributes are added to various indices. Though these patches were 
> applied later once all internal attributes were added to all the indices.
> Now, these patch vertices have 9 attributes and we releaized these 9 
> attributes are the only attributes missing from ES. So basically when patch 
> vertices got created and these vertices with their attributes got added to 
> cassandra via janusgraph, janusgraph automatically created propertyKey for 
> all these attributes (the janusgraph's makePropertyKey method is not called 
> during this process anywhere in the Atlas code). And because internal 
> attributes were getting added to indices in another thread at the same time, 
> when code checked for propertyKey, it was not null and so it did not add the 
> property to the mixed index.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 73593: ATLAS-4431: Random NPE when retrieving tasks

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73593/#review223503
---


Ship it!




Ship It!

- Ashutosh Mestry


On Sept. 20, 2021, 8:57 p.m., Disha Talreja wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73593/
> ---
> 
> (Updated Sept. 20, 2021, 8:57 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Sarath Subramanian, and Sidharth 
> Mishra.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> System tests fail randomly because retrieving tasks using the "admin/tasks" 
> endpoint sometimes returns HTTP 500. This patch includes a few null checks to 
> avoid NPE.
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java fae8a4fed 
> 
> 
> Diff: https://reviews.apache.org/r/73593/diff/1/
> 
> 
> Testing
> ---
> 
> Local Build Successful.
> 
> 
> Thanks,
> 
> Disha Talreja
> 
>

[jira] [Comment Edited] (ATLAS-4389) Best practice or a way to bring in large number of entities on a regular basis.

[Ashutosh Mestry (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-4389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17417764#comment-17417764
 ] 

Ashutosh Mestry edited comment on ATLAS-4389 at 9/20/21, 5:37 PM:
--

Sorry for the delay in replying.

Background: Existing implementation of ingest has linear complexity. This is 
done to be able to deal with the create/update/delete message types and the 
temporal nature of these operations.

Here are few that I have tried and worked as solutions for some of our 
customers:

*Approach 1*

Pre-requisite: Entity creation is in your control. 

Solution:
 * Create topologically sorted entities. Parent entities are created before the 
child entities.
 * Create lineage entities after parent participating entities are created.
 * Use REST APIs to concurrently create entities of a type. Start new type only 
after all entities of a type are exhausted.

This is the advantage of being able to create entities concurrently as their 
dependents are already created. This approach gives high throughput and 
continues to maintain consistency of data.

This needs some amount of book-keeping. This may not be a lot if you are 
creating Hive entities and follow a consistent pattern for coming up with names 
for _qualifiedName_ unique attribute.

In my test: I was able to run between 25 to 50 concurrent workers all creating 
entity of a type. 

About code paths: Ingest via Kakfa queue, entity creation via REST APIs and 
ingest via Import API all follow same code path.

 


was (Author: ashutoshm):
Sorry for the delay in replying.

Background: Existing implementation of ingest has linear complexity. This is 
done to be able to deal with the create/update/delete message types and the 
temporal nature of these operations.

Here are few that I have tried and worked as solutions for some of our 
customers:

*Approach 1*

Pre-requisite: Entity creation is in your control. 

Solution: 
 * Create topologically sorted entities. Parent entities are created before the 
child entities. 
 * Create lineage entities after parent participating entities are created.
 * Use REST APIs to concurrently create entities of a type. Start new type only 
after all entities of a type are exhausted.

This is the advantage of being able to create entities concurrently as their 
dependents are already created. This approach gives high throughput and 
continues to maintain consistency of data.

This needs some amount of book-keeping. This may not be a lot if you are 
creating Hive entities and follow a consistent pattern for coming up with names 
for _qualifiedName_ unique attribute.

About code paths: Ingest via Kakfa queue, entity creation via REST APIs and 
ingest via Import API all follow same code path.

 

> Best practice or a way to bring in large number of entities on a regular 
> basis.
> ---
>
> Key: ATLAS-4389
> URL: https://issues.apache.org/jira/browse/ATLAS-4389
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Saad
>    Assignee: Ashutosh Mestry
>Priority: Major
>  Labels: documentation, newbie, performance
> Attachments: image-2021-08-05-11-22-29-259.png, 
> image-2021-08-05-11-23-05-440.png
>
>
> Would you be so kind to let us know if there is any best practice or a way to 
> bring in large number of entities on a regular basis.
> *Our use case:*
> We will be bringing in around 12,000  datasets, 12,000 jobs and 70,000 
> columns. We want to do this as part of our deployment pipeline for other 
> upstream projects.
> At every deploy we want to do the following:
>  - Add the jobs, datasets and columns that are not in Atlas
>  - Update the jobs, datasets and columns that are in Atlas
>  - Delete the jobs from Atlas that are deleted from the upstream systems.
> So far we have considered using the bulk API endpoint(/v2/entity/bulk). This 
> has its own issues. We found that if the payload is too big in our case 
> bigger than 300-500 entities this times out. The more deeper the 
> relationships the fewer the entities you can send through the bulk endpoint.
> Inspecting some of the code we feel that both REST and streaming data through 
> Kafka follow the same codepath and finally yield the same performance.
> Further we found that when creating entities the type registry becomes the 
> bottle neck. We discovered this by profiling the jvm. We found that only one 
> core processes the the entities and their relationships.
> *Questions:*
> 1- What is the best practice when bulk loading lots on entities in a 
> reasonable time. We are aiming to load 12k jobs, 12k

[jira] [Commented] (ATLAS-4389) Best practice or a way to bring in large number of entities on a regular basis.

[Ashutosh Mestry (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-4389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17417764#comment-17417764
 ] 

Ashutosh Mestry commented on ATLAS-4389:


Sorry for the delay in replying.

Background: Existing implementation of ingest has linear complexity. This is 
done to be able to deal with the create/update/delete message types and the 
temporal nature of these operations.

Here are few that I have tried and worked as solutions for some of our 
customers:

*Approach 1*

Pre-requisite: Entity creation is in your control. 

Solution: 
 * Create topologically sorted entities. Parent entities are created before the 
child entities. 
 * Create lineage entities after parent participating entities are created.
 * Use REST APIs to concurrently create entities of a type. Start new type only 
after all entities of a type are exhausted.

This is the advantage of being able to create entities concurrently as their 
dependents are already created. This approach gives high throughput and 
continues to maintain consistency of data.

This needs some amount of book-keeping. This may not be a lot if you are 
creating Hive entities and follow a consistent pattern for coming up with names 
for _qualifiedName_ unique attribute.

About code paths: Ingest via Kakfa queue, entity creation via REST APIs and 
ingest via Import API all follow same code path.

 

> Best practice or a way to bring in large number of entities on a regular 
> basis.
> ---
>
> Key: ATLAS-4389
> URL: https://issues.apache.org/jira/browse/ATLAS-4389
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Saad
>    Assignee: Ashutosh Mestry
>Priority: Major
>  Labels: documentation, newbie, performance
> Attachments: image-2021-08-05-11-22-29-259.png, 
> image-2021-08-05-11-23-05-440.png
>
>
> Would you be so kind to let us know if there is any best practice or a way to 
> bring in large number of entities on a regular basis.
> *Our use case:*
> We will be bringing in around 12,000  datasets, 12,000 jobs and 70,000 
> columns. We want to do this as part of our deployment pipeline for other 
> upstream projects.
> At every deploy we want to do the following:
>  - Add the jobs, datasets and columns that are not in Atlas
>  - Update the jobs, datasets and columns that are in Atlas
>  - Delete the jobs from Atlas that are deleted from the upstream systems.
> So far we have considered using the bulk API endpoint(/v2/entity/bulk). This 
> has its own issues. We found that if the payload is too big in our case 
> bigger than 300-500 entities this times out. The more deeper the 
> relationships the fewer the entities you can send through the bulk endpoint.
> Inspecting some of the code we feel that both REST and streaming data through 
> Kafka follow the same codepath and finally yield the same performance.
> Further we found that when creating entities the type registry becomes the 
> bottle neck. We discovered this by profiling the jvm. We found that only one 
> core processes the the entities and their relationships.
> *Questions:*
> 1- What is the best practice when bulk loading lots on entities in a 
> reasonable time. We are aiming to load 12k jobs, 12k datasets and 70k columns 
> in less than 10 mins.?
> 2- Where should we start if we want to scale the API, is there any known way 
> to horizontally scale Atlas?
> Here are some of the stats for the load testing we did,
>  
> !image-2021-08-05-11-23-05-440.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 73584: ATLAS-4422 : Quick Search: Provide sorting

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73584/#review223491
---


Ship it!




Ship It!

- Ashutosh Mestry


On Sept. 15, 2021, 4:03 p.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73584/
> ---
> 
> (Updated Sept. 15, 2021, 4:03 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-4422
> https://issues.apache.org/jira/browse/ATLAS-4422
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Improvement:** 
> 
> This will add a sorting functionality in quick search API and get the result 
> list in Sorted manner.
> 
> **Note:** 
> 
> - Result will be sorted only when 'typeName' has a valid entity type name as 
> a value.
> - Only string attribute type (not TEXT) guarantees sorted results
> 
> **Example:** 
> 
> { "query":"sales", 
> "typeName":"Asset",
> "sortBy":"__typeName",
> "sortOrder":"ASCENDING" }
> 
> 
> Diffs
> -
> 
>   
> intg/src/main/java/org/apache/atlas/model/discovery/QuickSearchParameters.java
>  9e5e9778a 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  5b3b9486b 
>   
> repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
>  5e47295cb 
>   webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java e4c74a9d1 
> 
> 
> Diff: https://reviews.apache.org/r/73584/diff/2/
> 
> 
> Testing
> ---
> 
> added Unit tests
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>