Re: Review Request 72906: ATLAS-3934: Dockerfile: updated default to build from local repo (instead of from github)

2020-09-24 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72906/#review221946
---


Ship it!




Ship It!

- Sarath Subramanian


On Sept. 23, 2020, 7:44 p.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72906/
> ---
> 
> (Updated Sept. 23, 2020, 7:44 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Sarath 
> Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3934
> https://issues.apache.org/jira/browse/ATLAS-3934
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> - added 'src' volume in docker-compose-atlas-build.yml, so that build uses 
> local repo
> - to address slow access to mounted volumes in MacOS, added "delegated" to 
> .m2 and src volumes. Ref: 
> https://docs.docker.com/docker-for-mac/osxfs-caching/
> 
> 
> Diffs
> -
> 
>   dev-support/atlas-docker/Dockerfile.atlas-build 58d127a1e 
>   dev-support/atlas-docker/README.md 21f9b7548 
>   dev-support/atlas-docker/docker-compose.atlas-build.yml a1bc2f7b7 
>   dev-support/atlas-docker/scripts/atlas-build.sh b920034dc 
> 
> 
> Diff: https://reviews.apache.org/r/72906/diff/1/
> 
> 
> Testing
> ---
> 
> - verified build succeeds with local repo (in host)
> - verified build succeeds with repo downloaded in container (from github)
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>



Review Request 72902: ATLAS-3875: Introduce python client for Atlas

2020-09-23 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72902/
---

Review request for atlas, Ashutosh Mestry and Madhan Neethiraj.


Bugs: ATLAS-3875
https://issues.apache.org/jira/browse/ATLAS-3875


Repository: atlas


Description
---

This is a updated patch from https://reviews.apache.org/r/72730/

Few additional changes:
* Formatting changes
* Renamed atlas_client to apache_atlas
* Moved client and model files into intg module
* Updated README files.

-
Added basic structure for Python client
This includes all classes for Entity and TypeDef
Added all endpoints for typedef, Entity, Glossary, Lineage and Relationship
Added Python Sample project to test basic APIs and as a guideline
Added logging and readme file


Diffs
-

  .gitignore 90ca27646 
  
atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/GlossaryExample.java
 e4fc4edbf 
  atlas-examples/sample-app/src/main/python/README.md PRE-CREATION 
  atlas-examples/sample-app/src/main/python/discovery_example.py PRE-CREATION 
  atlas-examples/sample-app/src/main/python/entity_example.py PRE-CREATION 
  atlas-examples/sample-app/src/main/python/glossary_example.py PRE-CREATION 
  atlas-examples/sample-app/src/main/python/lineage_example.py PRE-CREATION 
  atlas-examples/sample-app/src/main/python/request_json/entity_create_db.json 
PRE-CREATION 
  
atlas-examples/sample-app/src/main/python/request_json/entity_create_process.json
 PRE-CREATION 
  
atlas-examples/sample-app/src/main/python/request_json/entity_create_table_canada.json
 PRE-CREATION 
  
atlas-examples/sample-app/src/main/python/request_json/entity_create_table_us.json
 PRE-CREATION 
  atlas-examples/sample-app/src/main/python/request_json/typedef_create.json 
PRE-CREATION 
  atlas-examples/sample-app/src/main/python/sample_client.py PRE-CREATION 
  atlas-examples/sample-app/src/main/python/typedef_example.py PRE-CREATION 
  atlas-examples/sample-app/src/main/python/utils.py PRE-CREATION 
  intg/src/main/python/LICENSE PRE-CREATION 
  intg/src/main/python/README.md PRE-CREATION 
  intg/src/main/python/apache_atlas/__init__.py PRE-CREATION 
  intg/src/main/python/apache_atlas/base_client.py PRE-CREATION 
  intg/src/main/python/apache_atlas/client/__init__.py PRE-CREATION 
  intg/src/main/python/apache_atlas/client/discovery.py PRE-CREATION 
  intg/src/main/python/apache_atlas/client/entity.py PRE-CREATION 
  intg/src/main/python/apache_atlas/client/glossary.py PRE-CREATION 
  intg/src/main/python/apache_atlas/client/lineage.py PRE-CREATION 
  intg/src/main/python/apache_atlas/client/relationship.py PRE-CREATION 
  intg/src/main/python/apache_atlas/client/typedef.py PRE-CREATION 
  intg/src/main/python/apache_atlas/exceptions.py PRE-CREATION 
  intg/src/main/python/apache_atlas/model/__init__.py PRE-CREATION 
  intg/src/main/python/apache_atlas/model/discovery.py PRE-CREATION 
  intg/src/main/python/apache_atlas/model/entity.py PRE-CREATION 
  intg/src/main/python/apache_atlas/model/glossary.py PRE-CREATION 
  intg/src/main/python/apache_atlas/model/lineage.py PRE-CREATION 
  intg/src/main/python/apache_atlas/model/relationship.py PRE-CREATION 
  intg/src/main/python/apache_atlas/model/typedef.py PRE-CREATION 
  intg/src/main/python/apache_atlas/utils.py PRE-CREATION 
  intg/src/main/python/requirements.txt PRE-CREATION 
  intg/src/main/python/setup.py PRE-CREATION 
  intg/src/main/python/tests/__init__.py PRE-CREATION 


Diff: https://reviews.apache.org/r/72902/diff/1/


Testing
---

Local testing done


Thanks,

Sarath Subramanian



Re: Review Request 72865: ATLAS-3934: docker scripts to build and run Apache Atlas in containers

2020-09-18 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72865/#review221901
---




dev-support/atlas-docker/README.md
Lines 76 (patched)
<https://reviews.apache.org/r/72865/#comment310957>

consider including Atlas Home directory and log file location in README


- Sarath Subramanian


On Sept. 11, 2020, 10:55 p.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72865/
> ---
> 
> (Updated Sept. 11, 2020, 10:55 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, keval bhatt, Mehul 
> Parikh, Nikhil Bonte, Nixon Rodrigues, Sarath Subramanian, Sharmadha Sainath, 
> and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3934
> https://issues.apache.org/jira/browse/ATLAS-3934
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> - added docker files, docker-compose files and scripts to build Atlas in a 
> container. The image to build Atlas supports following volumns:
> -- scripts: build scripts in the local directory will be made available in 
> the container
> -- dist:archives from the build will be copied into this volume
> -- .m2: if provided, build uses the give .m2 directory, which can help 
> avoid large downloads
> -- patches: patche files, if present, will be applied on top of the specified 
> github branch
> -- src: if provided, builds from the sources in this directory (instead 
> of from sources from github)
> 
> - added docker file, docker-compose file and scripts to install and run Atlas 
> in a container.
> 
> - note: this patch depends on updates in ATLAS-3939, which is under review
> 
> 
> Diffs
> -
> 
>   dev-support/atlas-docker/.dockerignore PRE-CREATION 
>   dev-support/atlas-docker/Dockerfile.atlas PRE-CREATION 
>   dev-support/atlas-docker/Dockerfile.atlas-base PRE-CREATION 
>   dev-support/atlas-docker/Dockerfile.atlas-build PRE-CREATION 
>   dev-support/atlas-docker/README.md 5296f05ba 
>   dev-support/atlas-docker/dist/.gitignore PRE-CREATION 
>   dev-support/atlas-docker/docker-compose.atlas-base.yml PRE-CREATION 
>   dev-support/atlas-docker/docker-compose.atlas-build.yml PRE-CREATION 
>   dev-support/atlas-docker/docker-compose.atlas.yml PRE-CREATION 
>   dev-support/atlas-docker/patches/.gitignore PRE-CREATION 
>   dev-support/atlas-docker/scripts/atlas-build.sh PRE-CREATION 
>   dev-support/atlas-docker/scripts/atlas.sh PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/72865/diff/1/
> 
> 
> Testing
> ---
> 
> - verified build with the following command:
>docker-compose -f docker-compose.atlas-base.yml -f 
> docker-compose.atlas-build.yml up
> 
> - verified install and running of Atlas with the following command:
>docker-compose -f docker-compose.atlas-base.yml -f 
> docker-compose.atlas.yml up -d
> 
> - Ran quick start successfully
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>



Re: Review Request 72865: ATLAS-3934: docker scripts to build and run Apache Atlas in containers

2020-09-18 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72865/#review221900
---


Ship it!




Validated the docker build process using docker-compose. 

Atlas builds fine and starts in container. Ran quickstart and verified entities 
are created fine.

- Sarath Subramanian


On Sept. 11, 2020, 10:55 p.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72865/
> ---
> 
> (Updated Sept. 11, 2020, 10:55 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, keval bhatt, Mehul 
> Parikh, Nikhil Bonte, Nixon Rodrigues, Sarath Subramanian, Sharmadha Sainath, 
> and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3934
> https://issues.apache.org/jira/browse/ATLAS-3934
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> - added docker files, docker-compose files and scripts to build Atlas in a 
> container. The image to build Atlas supports following volumns:
> -- scripts: build scripts in the local directory will be made available in 
> the container
> -- dist:archives from the build will be copied into this volume
> -- .m2: if provided, build uses the give .m2 directory, which can help 
> avoid large downloads
> -- patches: patche files, if present, will be applied on top of the specified 
> github branch
> -- src: if provided, builds from the sources in this directory (instead 
> of from sources from github)
> 
> - added docker file, docker-compose file and scripts to install and run Atlas 
> in a container.
> 
> - note: this patch depends on updates in ATLAS-3939, which is under review
> 
> 
> Diffs
> -
> 
>   dev-support/atlas-docker/.dockerignore PRE-CREATION 
>   dev-support/atlas-docker/Dockerfile.atlas PRE-CREATION 
>   dev-support/atlas-docker/Dockerfile.atlas-base PRE-CREATION 
>   dev-support/atlas-docker/Dockerfile.atlas-build PRE-CREATION 
>   dev-support/atlas-docker/README.md 5296f05ba 
>   dev-support/atlas-docker/dist/.gitignore PRE-CREATION 
>   dev-support/atlas-docker/docker-compose.atlas-base.yml PRE-CREATION 
>   dev-support/atlas-docker/docker-compose.atlas-build.yml PRE-CREATION 
>   dev-support/atlas-docker/docker-compose.atlas.yml PRE-CREATION 
>   dev-support/atlas-docker/patches/.gitignore PRE-CREATION 
>   dev-support/atlas-docker/scripts/atlas-build.sh PRE-CREATION 
>   dev-support/atlas-docker/scripts/atlas.sh PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/72865/diff/1/
> 
> 
> Testing
> ---
> 
> - verified build with the following command:
>docker-compose -f docker-compose.atlas-base.yml -f 
> docker-compose.atlas-build.yml up
> 
> - verified install and running of Atlas with the following command:
>docker-compose -f docker-compose.atlas-base.yml -f 
> docker-compose.atlas.yml up -d
> 
> - Ran quick start successfully
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>



[jira] [Commented] (ATLAS-3948) Entity Creation: Index Consistency: Java Patch Handler: Provide Option to Disable

2020-09-17 Thread Sarath Subramanian (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197879#comment-17197879
 ] 

Sarath Subramanian commented on ATLAS-3948:
---

thanks for the patch [~amestry] +1

> Entity Creation: Index Consistency: Java Patch Handler: Provide Option to 
> Disable
> -
>
> Key: ATLAS-3948
> URL: https://issues.apache.org/jira/browse/ATLAS-3948
> Project: Atlas
>  Issue Type: Bug
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Attachments: ATLAS-3948-index-consistency-conditional.patch
>
>
> *Background*
> The ATLAS-3907 provided ability to apply index consistency to existing 
> indexes via Java patch handler.
> However, when _atlas.graph.storage.consistency-lock.enabled=false_ this patch 
> still gets applied.
> *Solution*
> Check for the property in Java patch handler and avoid applying it if the 
> property is set to false.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 72880: ATLAS-3947 : Skip authorization for read of _ALL_ENTITY_TYPES and _ALL_CLASSIFICATION_TYPES types

2020-09-17 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72880/#review221887
---


Ship it!




Ship It!

- Sarath Subramanian


On Sept. 17, 2020, 3:28 a.m., Nixon Rodrigues wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72880/
> ---
> 
> (Updated Sept. 17, 2020, 3:28 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3947
> https://issues.apache.org/jira/browse/ATLAS-3947
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Skip authorization for read of _ALL_ENTITY_TYPES type of entity type and 
> _ALL_CLASSIFICATION_TYPES type of classification type.
> This are internally required by for system attributes search.
> 
> This patch Skip authorization for read of _ALL_ENTITY_TYPES entityType and 
> and _ALL_CLASSIFICATION_TYPES classificationType.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java
>  458ec49ea 
> 
> 
> Diff: https://reviews.apache.org/r/72880/diff/1/
> 
> 
> Testing
> ---
> 
> Tested API's with no policies on types for a user. _ALL_ENTITY_TYPES and 
> _ALL_CLASSIFICATION_TYPES are loaded correctly after it.
> 
> 
> Thanks,
> 
> Nixon Rodrigues
> 
>



Re: Review Request 72879: ATLAS-3946 : Filter TypeDefs in Metrics API and show data accordingly

2020-09-17 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72879/#review221886
---


Ship it!




Ship It!

- Sarath Subramanian


On Sept. 17, 2020, 2:26 a.m., Nixon Rodrigues wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72879/
> ---
> 
> (Updated Sept. 17, 2020, 2:26 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3946
> https://issues.apache.org/jira/browse/ATLAS-3946
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Filter TypeDefs in Metrics API and show data accordingly, currently data is 
> shown for all types even if user do not have access to those types.
> 
> This patch filterTypeDefs after authorization and updates the data for 
> required types.
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/services/MetricsService.java 
> 7917f32e5 
>   webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java 
> e74a9e93a 
> 
> 
> Diff: https://reviews.apache.org/r/72879/diff/1/
> 
> 
> Testing
> ---
> 
> Tested Statistics UI from different users with different policies for 
> entityType and classfication.
> 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/25/console
> 
> 
> Thanks,
> 
> Nixon Rodrigues
> 
>



[jira] [Commented] (ATLAS-3941) NotificationHookConsumer: Reduce Retry Pause Interval

2020-09-14 Thread Sarath Subramanian (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195606#comment-17195606
 ] 

Sarath Subramanian commented on ATLAS-3941:
---

Thanks for the patch. +1

> NotificationHookConsumer: Reduce Retry Pause Interval
> -
>
> Key: ATLAS-3941
> URL: https://issues.apache.org/jira/browse/ATLAS-3941
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: trunk, 2.1.0
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Fix For: trunk
>
> Attachments: 
> ATLAS-3941-NotificationHookConsumer-Reduce-wait-time.patch
>
>
> *Background*
> The retry logic introduced earlier, had a long wait time in case a concurrent 
> entity create was detected. This adversely affect ingest speed in the case 
> where there are a lot of errors in the data being ingested.
> *Solution*
> Reduce the wait time.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 72636: NotificationHookConsumer: Concurrent Message Processing

2020-09-02 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72636/#review221774
---


Ship it!




Ship It!

- Sarath Subramanian


On Aug. 3, 2020, 4:18 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72636/
> ---
> 
> (Updated Aug. 3, 2020, 4:18 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3874
> https://issues.apache.org/jira/browse/ATLAS-3874
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> Please see bug description.
> 
> **New Approach**
> The new patch now uses the improvements made by ATLAS-3398.
> 
> Modified *NotificationHookConsumer*: Uses current retry logic. Handles 
> exceptions encountered during commit and displays messages that do not 
> overwhelm the log file.
> Bug fix: *EntityDiscoveryContext*: The *resolvedIdsByUniqAttribs* did not 
> return the correct value if the map contained *AtlasRelationshipObjectId*. 
> The logic in the method *getResolvedEntityVertex* has been tweaked to handle 
> the *AtlasRelationshipObjectId* while maintaining backward compatibility. New 
> method *getAtlasVertexFromResolvedIdsByAttribs* was added to encapsulate the 
> existing processing of subTypes.
> 
> **Earlier Approach** (Obsolete)
> At a high-level: Introduce a notion where the individual consumers are aware 
> of the entities being processed by each other. If there are no entities being 
> processed concurrently, everything proceeds as usual (the way it is before 
> his change). If same entity is being procesed by multiple consumers, then one 
> consumer waits for the other to finish before proceeding.
> 
> Classes:
> New *UniqueKeysExtractor*: Extracts values of unique keys from 
> *AtlasEntitiesWithExtInfo*. It navigates *relationshipAttributes* and 
> *attributes* that has *objectRef* set.
> New *UniquenessChecker*: Maintains a set of unique keys provided by 
> *UniqueKeysExtractor*. It detects the presence of duplicates and waits until 
> duplicates are resolved.
> Modified *NoitficationHookConsumer.createOrUpdate* 
> - Updates *UniquenessChecker* with output from *UniqueKeysExtractor*. Clears 
> the keys at the end of entity creation.
> - Handles JanusGraph's *PermanentLockingException*. See 
> [ATLAS-3801](https://issues.apache.org/jira/browse/ATLAS-3801).
> Modified *NotificationHookConsumer*: Accepts an shared instance of 
> *UniqunessChecker*.
> 
> **Additional Logging**
> Log entry is added when attempt is made to creates entities concurrently.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/EntityGraphDiscoveryContext.java
>  2221ac4f4 
>   
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  3f1ea05e1 
> 
> 
> Diff: https://reviews.apache.org/r/72636/diff/4/
> 
> 
> Testing
> ---
> 
> **Unit tests**
> Tests added to verify new clases.
> 
> *UniquessCheckerTest*
> Performs worst case checking by adding 100s of keys that are duplicates and 
> verifies output of those.
> 
> **Functional tests**
> Used Spark hook to verify. These sequence of commands will create shell 
> entity and first class entity. If resolution for first class entity is 
> available, then no shell entity is created.
> 
> Start Spark shell using:
> ```
> sudo -u hdfs spark-shell
> ```
> 
> Spark sql commands:
> ```
> spark.sql("create table default.t1_1381104676(col1 int)")
> spark.sql("create table default.t2_1381104676(col2 int)")
> spark.sql("select * from t1_1381104676, t2_1381104676 where 
> col1=col2").write.saveAsTable("t3_1381104676")
> ```
> 
> **Volume test**
> Medium-size Kafka dump added.
> 
> *Setup* Add same kafka dump to multiple topics. This will result in 
> contention for almost every message.
> 
> **Pre-commit Build**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2078/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>



Re: Review Request 72636: NotificationHookConsumer: Concurrent Message Processing

2020-08-26 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72636/#review221725
---


Fix it, then Ship it!





repository/src/main/java/org/apache/atlas/repository/store/graph/EntityGraphDiscoveryContext.java
Lines 100 (patched)
<https://reviews.apache.org/r/72636/#comment310778>

consider moving if block (100-102) to check if 'resolvedIdsByUniqAttribs' 
contains objId  before check for AtlasRelatedObjectId (line 99). something 
similar to:

`
public AtlasVertex getResolvedEntityVertex(AtlasObjectId objId) {
  AtlasVertex ret;

  if (resolvedIdsByUniqAttribs.containsKey(objId)) {
 ret = resolvedIdsByUniqAttribs.get(objId);
  } else if (objId instanceof AtlasRelatedObjectId) {
 objId = new AtlasObjectId(objId.getGuid(), objId.getTypeName(), 
objId.getUniqueAttributes());

 ret = getAtlasVertexFromResolvedIdsByAttribs(objId);
  }

  return ret;
}
`


- Sarath Subramanian


On Aug. 3, 2020, 4:18 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72636/
> ---
> 
> (Updated Aug. 3, 2020, 4:18 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3874
> https://issues.apache.org/jira/browse/ATLAS-3874
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> Please see bug description.
> 
> **New Approach**
> The new patch now uses the improvements made by ATLAS-3398.
> 
> Modified *NotificationHookConsumer*: Uses current retry logic. Handles 
> exceptions encountered during commit and displays messages that do not 
> overwhelm the log file.
> Bug fix: *EntityDiscoveryContext*: The *resolvedIdsByUniqAttribs* did not 
> return the correct value if the map contained *AtlasRelationshipObjectId*. 
> The logic in the method *getResolvedEntityVertex* has been tweaked to handle 
> the *AtlasRelationshipObjectId* while maintaining backward compatibility. New 
> method *getAtlasVertexFromResolvedIdsByAttribs* was added to encapsulate the 
> existing processing of subTypes.
> 
> **Earlier Approach** (Obsolete)
> At a high-level: Introduce a notion where the individual consumers are aware 
> of the entities being processed by each other. If there are no entities being 
> processed concurrently, everything proceeds as usual (the way it is before 
> his change). If same entity is being procesed by multiple consumers, then one 
> consumer waits for the other to finish before proceeding.
> 
> Classes:
> New *UniqueKeysExtractor*: Extracts values of unique keys from 
> *AtlasEntitiesWithExtInfo*. It navigates *relationshipAttributes* and 
> *attributes* that has *objectRef* set.
> New *UniquenessChecker*: Maintains a set of unique keys provided by 
> *UniqueKeysExtractor*. It detects the presence of duplicates and waits until 
> duplicates are resolved.
> Modified *NoitficationHookConsumer.createOrUpdate* 
> - Updates *UniquenessChecker* with output from *UniqueKeysExtractor*. Clears 
> the keys at the end of entity creation.
> - Handles JanusGraph's *PermanentLockingException*. See 
> [ATLAS-3801](https://issues.apache.org/jira/browse/ATLAS-3801).
> Modified *NotificationHookConsumer*: Accepts an shared instance of 
> *UniqunessChecker*.
> 
> **Additional Logging**
> Log entry is added when attempt is made to creates entities concurrently.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/EntityGraphDiscoveryContext.java
>  2221ac4f4 
>   
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  3f1ea05e1 
> 
> 
> Diff: https://reviews.apache.org/r/72636/diff/3/
> 
> 
> Testing
> ---
> 
> **Unit tests**
> Tests added to verify new clases.
> 
> *UniquessCheckerTest*
> Performs worst case checking by adding 100s of keys that are duplicates and 
> verifies output of those.
> 
> **Functional tests**
> Used Spark hook to verify. These sequence of commands will create shell 
> entity and first class entity. If resolution for first class entity is 
> available, then no shell entity is created.
> 
> Start Spark shell using:
> ```
> sudo -u hdfs spark-shell
> ```
> 
> Spark sql commands:
> ```
> spark.sql("create table default.t1_1381104676(col1 int)")
> spark.sql("create table default.t2_1381104676

Re: Review Request 72477: ATLAS-3583 Use Audit framework to generate audit entries for TypeDefs CREATE, UPDATE and DELETE

2020-08-24 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72477/#review221701
---


Ship it!




Ship It!

- Sarath Subramanian


On Aug. 14, 2020, 9:34 a.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72477/
> ---
> 
> (Updated Aug. 14, 2020, 9:34 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> Sarath Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3583
> https://issues.apache.org/jira/browse/ATLAS-3583
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-3583 Use Audit framework to generate audit entries for TypeDefs CREATE, 
> UPDATE and DELETE
> 
> 
> Diffs
> -
> 
>   addons/models/-Area0/0010-base_model.json 9777b7a 
>   
> addons/models/-Area0/patches/006-base_model_add_atlas_operation_attributes.json
>  PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java 
> a95cf4e 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java
>  a0dc816 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/TypeDefAuditListener.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java
>  8e7c1b3 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
>  0dc3193 
>   test-tools/src/main/resources/solr/core-template/solrconfig.xml 8ebbeff 
> 
> 
> Diff: https://reviews.apache.org/r/72477/diff/9/
> 
> 
> Testing
> ---
> 
> Basic testing is done.
> 
> Pre-commit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1871/console
> 
> Pre-commit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1885/console
> 
> Pre-commit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1888/console
> 
> Pre-commit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2071/console
> 
> Pre-commit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2082/console
> 
> Pre-commit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2100/console
> 
> 
> Thanks,
> 
> Mandar Ambawane
> 
>



[jira] [Updated] (ATLAS-3898) Authorize read of typedefs

2020-08-14 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3898:
--
Fix Version/s: 3.0.0

> Authorize read of typedefs
> --
>
> Key: ATLAS-3898
> URL: https://issues.apache.org/jira/browse/ATLAS-3898
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.1.0
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
>  Labels: authorization, ranger
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-3898.1.patch
>
>
> Currently in the Atlas-Ranger plugin for types resource READ permission is 
> not available and read access is available by default to all types of any 
> category.
> This patch authorize read for all typedefs. It implicitly add read permission 
> if create,update or delete permission is assigned to user.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3898) Authorize read of typedefs

2020-08-14 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3898:
--
Fix Version/s: 2.2.0

> Authorize read of typedefs
> --
>
> Key: ATLAS-3898
> URL: https://issues.apache.org/jira/browse/ATLAS-3898
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.1.0
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
>  Labels: authorization, ranger
> Fix For: 2.2.0
>
> Attachments: ATLAS-3898.1.patch
>
>
> Currently in the Atlas-Ranger plugin for types resource READ permission is 
> not available and read access is available by default to all types of any 
> category.
> This patch authorize read for all typedefs. It implicitly add read permission 
> if create,update or delete permission is assigned to user.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3898) Authorize read of typedefs

2020-08-14 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3898:
--
Affects Version/s: 2.1.0

> Authorize read of typedefs
> --
>
> Key: ATLAS-3898
> URL: https://issues.apache.org/jira/browse/ATLAS-3898
> Project: Atlas
>  Issue Type: New Feature
>Affects Versions: 2.1.0
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Attachments: ATLAS-3898.1.patch
>
>
> Currently in the Atlas-Ranger plugin for types resource READ permission is 
> not available and read access is available by default to all types of any 
> category.
> This patch authorize read for all typedefs. It implicitly add read permission 
> if create,update or delete permission is assigned to user.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3898) Authorize read of typedefs

2020-08-14 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3898:
--
Labels: authorization ranger  (was: )

> Authorize read of typedefs
> --
>
> Key: ATLAS-3898
> URL: https://issues.apache.org/jira/browse/ATLAS-3898
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.1.0
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
>  Labels: authorization, ranger
> Attachments: ATLAS-3898.1.patch
>
>
> Currently in the Atlas-Ranger plugin for types resource READ permission is 
> not available and read access is available by default to all types of any 
> category.
> This patch authorize read for all typedefs. It implicitly add read permission 
> if create,update or delete permission is assigned to user.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3898) Authorize read of typedefs

2020-08-14 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3898:
--
Component/s:  atlas-core

> Authorize read of typedefs
> --
>
> Key: ATLAS-3898
> URL: https://issues.apache.org/jira/browse/ATLAS-3898
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.1.0
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Attachments: ATLAS-3898.1.patch
>
>
> Currently in the Atlas-Ranger plugin for types resource READ permission is 
> not available and read access is available by default to all types of any 
> category.
> This patch authorize read for all typedefs. It implicitly add read permission 
> if create,update or delete permission is assigned to user.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 72766: ATLAS-3920 : Harmonize joda-time to version 2.10.latest.

2020-08-13 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72766/#review221569
---


Ship it!




Ship It!

- Sarath Subramanian


On Aug. 12, 2020, 9:53 p.m., mayank jain wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72766/
> ---
> 
> (Updated Aug. 12, 2020, 9:53 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3920
> https://issues.apache.org/jira/browse/ATLAS-3920
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas currently uses outdated joda-time versions, which can cause issues. To 
> avoid issue we need to update it to the latest version of joda-time.
> 
> 
> Diffs
> -
> 
>   pom.xml 5e0442a 
> 
> 
> Diff: https://reviews.apache.org/r/72766/diff/1/
> 
> 
> Testing
> ---
> 
> The test cases passed and also tested basic features and they worked.
> 
> 
> Thanks,
> 
> mayank jain
> 
>



[jira] [Commented] (ATLAS-3900) UI: Allow user to select the date range for date attribute in basic search

2020-08-12 Thread Sarath Subramanian (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176583#comment-17176583
 ] 

Sarath Subramanian commented on ATLAS-3900:
---

Thanks for the patch [~kevalbhatt]. +1.

> UI: Allow user to select the date range for date attribute in basic search
> --
>
> Key: ATLAS-3900
> URL: https://issues.apache.org/jira/browse/ATLAS-3900
> Project: Atlas
>  Issue Type: Sub-task
>  Components: atlas-webui
>Affects Versions: 2.1.0
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>Priority: Major
>  Labels: basic-search, datetimepicker
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-3900-1.patch, ATLAS-3900-2.patch, 
> ATLAS-3900-3.patch, ATLAS-3900.patch, Screen Shot 2020-07-21 at 10.59.59 
> PM.png, Screen Shot 2020-07-21 at 11.00.08 PM.png, Screen Shot 2020-07-21 at 
> 11.00.15 PM.png
>
>
> In basic search attribute popup if the user selects the date type attribute 
> then UI should show a few quick search operator and custom range selection. 
> example:
> !Screen Shot 2020-07-21 at 10.59.59 PM.png|width=631,height=282!
>  
> !Screen Shot 2020-07-21 at 11.00.08 PM.png|width=630,height=302!
>  
> !Screen Shot 2020-07-21 at 11.00.15 PM.png|width=630,height=304!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3919) Handling classification propagation as deferred-action

2020-08-12 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3919:
--
Affects Version/s: 2.0.0
   2.1.0

> Handling classification propagation as deferred-action
> --
>
> Key: ATLAS-3919
> URL: https://issues.apache.org/jira/browse/ATLAS-3919
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Jayendra Parab
>Assignee: Jayendra Parab
>Priority: Major
>
> Currently, whenever a user assigns a tag or updates a tag on an entity, it 
> gets propagated to all the entities derived from the tagged entity. This 
> operation takes quite a lot of time to complete (sometimes into minutes) and 
> causes usability issues on the UI and other clients invoking the REST API
> To resolve this issue, tag-propagation needs to be handled as 
> deferred-action, so that time consuming like propagation, audits & 
> notifications can be processed in background threads



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3916) Get metrics according to the user permissions

2020-08-11 Thread Sarath Subramanian (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175783#comment-17175783
 ] 

Sarath Subramanian commented on ATLAS-3916:
---

[~yued], good requirement. The way metrics query works today is we rely on solr 
to provide you with the totalCount of a specific type. 
{code:java}
atlasGraph.indexQuery(VERTEX_INDEX, indexQuery).vertexTotals();
{code}
If metrics query should display the count of only those entities which user has 
read authorization.

We may need to go over individual entities for each type and check if user has 
authorization. 

 

> Get metrics according to the user permissions
> -
>
> Key: ATLAS-3916
> URL: https://issues.apache.org/jira/browse/ATLAS-3916
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Yue Dong
>Priority: Major
> Attachments: Captura-de-pantalla-de-2020-08-11-10-20-06.png
>
>
> I have two user groups: admin who has access to all tables and reader can 
> only see public data and module A tables. So I have configured Atlas to use a 
> simple authorizer with a little variation, which is to hide entities that are 
> not accessible to the user.
> The searches and displaying results work properly.
> The only problem I find is that the metrics. In the elements of search by 
> type, it indicates the number of all the entities of each type in the system. 
> And this is not consistent with the search result of a reader user. 
> !Captura-de-pantalla-de-2020-08-11-10-20-06.png!
>  
> I have verified that these numbers come from the getMetrics method, which is 
> not secured so it does not obtain the numbers according to the users' 
> configuration. Am I missing something? Is there any way to change these 
> numbers?
> Maybe it'd be nice to have something that allows to modify the querys of the 
> metrics based on security and authorization, like 
> AtlasAuthorizer.scrubSearchResults in search methods.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 72698: ATLAS-3875: Introduce sample project for AtlasClient

2020-08-11 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72698/#review221539
---



The sample-app fails to run with the following error:

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:3.0.0:java 
(default-cli) on project sample-app: An exception occured while executing the 
Java class. Metadata service API org.apache.atlas.AtlasClientV2$API_V2@2eff472d 
failed with status 409 (Conflict) Response Body 
({"errorCode":"ATLAS-409-00-002","errorMessage":"Given type sample_Table_DB has 
references"}) -> [Help 1]

can you review.


atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/SampleApp.java
Lines 123 (patched)
<https://reviews.apache.org/r/72698/#comment310594>

the password here is clear text.

public static String[] getBasicAuthenticationInput() {
String username = null;
String password = null;

try {
Console console = System.console();
if (console == null) {
System.err.println("Couldn't get a console object for user 
input");
System.exit(1);
}

username = console.readLine("Enter username for atlas :- ");

char[] pwdChar = console.readPassword("Enter password for atlas 
:- ");
if(pwdChar != null) {
password = new String(pwdChar);
}

} catch (Exception e) {
System.out.print("Error while reading user input");
System.exit(1);
    }
return new String[]{username, password};
}


- Sarath Subramanian


On Aug. 4, 2020, 9:27 a.m., Jyoti Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72698/
> ---
> 
> (Updated Aug. 4, 2020, 9:27 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath 
> Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3875
> https://issues.apache.org/jira/browse/ATLAS-3875
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Using this project users can get an idea as how to integrate with Atlas using 
> AtlasCleint. This helps the user to understand the basic rest functionality 
> of Atlas such as
> 
> - EntityRest
> - TypeDefRest
> - DiscoveryRest
> - LineageRest
> - GlossaryRest
> 
> 
> Diffs
> -
> 
>   atlas-examples/pom.xml PRE-CREATION 
>   atlas-examples/sample-app/README.md PRE-CREATION 
>   atlas-examples/sample-app/pom.xml PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/DiscoveryExample.java
>  PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/EntityExample.java
>  PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/GlossaryExample.java
>  PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/LineageExample.java
>  PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/SampleApp.java
>  PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/SampleAppConstants.java
>  PRE-CREATION 
>   
> atlas-examples/sample-app/src/main/java/org/apache/atlas/examples/sampleapp/TypeDefExample.java
>  PRE-CREATION 
>   atlas-examples/sample-app/src/main/resources/atlas-application.properties 
> PRE-CREATION 
>   pom.xml 5e0442ae5 
> 
> 
> Diff: https://reviews.apache.org/r/72698/diff/7/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jyoti Singh
> 
>



[jira] [Updated] (ATLAS-3913) Swagger documentation for APIs

2020-08-07 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3913:
--
Labels: REST_API swagger  (was: )

> Swagger documentation for APIs
> --
>
> Key: ATLAS-3913
> URL: https://issues.apache.org/jira/browse/ATLAS-3913
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core, atlas-webui
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>  Labels: REST_API, swagger
>
> Current packaging does not package swagger ui into Atlas war.
> This change will pack swagger ui into the war file.
> Also update enunciate version from 2.11.1 to 2.13.1



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3913) Swagger documentation for APIs

2020-08-07 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3913:
--
Affects Version/s: 2.1.0

> Swagger documentation for APIs
> --
>
> Key: ATLAS-3913
> URL: https://issues.apache.org/jira/browse/ATLAS-3913
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core, atlas-webui
>Affects Versions: 2.1.0
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>  Labels: REST_API, swagger
>
> Current packaging does not package swagger ui into Atlas war.
> This change will pack swagger ui into the war file.
> Also update enunciate version from 2.11.1 to 2.13.1



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3913) Swagger documentation for APIs

2020-08-07 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3913:
--
Fix Version/s: 2.2.0
   3.0.0

> Swagger documentation for APIs
> --
>
> Key: ATLAS-3913
> URL: https://issues.apache.org/jira/browse/ATLAS-3913
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core, atlas-webui
>Affects Versions: 2.1.0
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>  Labels: REST_API, swagger
> Fix For: 3.0.0, 2.2.0
>
>
> Current packaging does not package swagger ui into Atlas war.
> This change will pack swagger ui into the war file.
> Also update enunciate version from 2.11.1 to 2.13.1



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3900) UI: Allow user to select the date range for date attribute in basic search

2020-08-07 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3900:
--
Labels: basic-search datetimepicker  (was: )

> UI: Allow user to select the date range for date attribute in basic search
> --
>
> Key: ATLAS-3900
> URL: https://issues.apache.org/jira/browse/ATLAS-3900
> Project: Atlas
>  Issue Type: Sub-task
>  Components: atlas-webui
>Affects Versions: 2.1.0
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>Priority: Major
>  Labels: basic-search, datetimepicker
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-3900-1.patch, ATLAS-3900-2.patch, 
> ATLAS-3900.patch, Screen Shot 2020-07-21 at 10.59.59 PM.png, Screen Shot 
> 2020-07-21 at 11.00.08 PM.png, Screen Shot 2020-07-21 at 11.00.15 PM.png
>
>
> In basic search attribute popup if the user selects the date type attribute 
> then UI should show a few quick search operator and custom range selection. 
> example:
> !Screen Shot 2020-07-21 at 10.59.59 PM.png|width=631,height=282!
>  
> !Screen Shot 2020-07-21 at 11.00.08 PM.png|width=630,height=302!
>  
> !Screen Shot 2020-07-21 at 11.00.15 PM.png|width=630,height=304!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3900) UI: Allow user to select the date range for date attribute in basic search

2020-08-07 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3900:
--
Affects Version/s: 2.1.0

> UI: Allow user to select the date range for date attribute in basic search
> --
>
> Key: ATLAS-3900
> URL: https://issues.apache.org/jira/browse/ATLAS-3900
> Project: Atlas
>  Issue Type: Sub-task
>Affects Versions: 2.1.0
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-3900-1.patch, ATLAS-3900-2.patch, 
> ATLAS-3900.patch, Screen Shot 2020-07-21 at 10.59.59 PM.png, Screen Shot 
> 2020-07-21 at 11.00.08 PM.png, Screen Shot 2020-07-21 at 11.00.15 PM.png
>
>
> In basic search attribute popup if the user selects the date type attribute 
> then UI should show a few quick search operator and custom range selection. 
> example:
> !Screen Shot 2020-07-21 at 10.59.59 PM.png|width=631,height=282!
>  
> !Screen Shot 2020-07-21 at 11.00.08 PM.png|width=630,height=302!
>  
> !Screen Shot 2020-07-21 at 11.00.15 PM.png|width=630,height=304!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3900) UI: Allow user to select the date range for date attribute in basic search

2020-08-07 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3900:
--
Component/s: atlas-webui

> UI: Allow user to select the date range for date attribute in basic search
> --
>
> Key: ATLAS-3900
> URL: https://issues.apache.org/jira/browse/ATLAS-3900
> Project: Atlas
>  Issue Type: Sub-task
>  Components: atlas-webui
>Affects Versions: 2.1.0
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-3900-1.patch, ATLAS-3900-2.patch, 
> ATLAS-3900.patch, Screen Shot 2020-07-21 at 10.59.59 PM.png, Screen Shot 
> 2020-07-21 at 11.00.08 PM.png, Screen Shot 2020-07-21 at 11.00.15 PM.png
>
>
> In basic search attribute popup if the user selects the date type attribute 
> then UI should show a few quick search operator and custom range selection. 
> example:
> !Screen Shot 2020-07-21 at 10.59.59 PM.png|width=631,height=282!
>  
> !Screen Shot 2020-07-21 at 11.00.08 PM.png|width=630,height=302!
>  
> !Screen Shot 2020-07-21 at 11.00.15 PM.png|width=630,height=304!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 72746: ATLAS-3912: moved classes from intg to repository or common module

2020-08-06 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72746/#review221492
---


Ship it!




Ship It!

- Sarath Subramanian


On Aug. 6, 2020, 4:57 p.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72746/
> ---
> 
> (Updated Aug. 6, 2020, 4:57 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Nikhil Bonte, Nixon Rodrigues, 
> Pinal Shah, Sarath Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3912
> https://issues.apache.org/jira/browse/ATLAS-3912
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> moved classes from intg to repository or common module
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/entitytransform/Action.java  
>   
> intg/src/main/java/org/apache/atlas/entitytransform/AtlasEntityTransformer.java
>   
>   intg/src/main/java/org/apache/atlas/entitytransform/BaseEntityHandler.java  
>   intg/src/main/java/org/apache/atlas/entitytransform/Condition.java  
>   intg/src/main/java/org/apache/atlas/entitytransform/EntityAttribute.java  
>   
> intg/src/main/java/org/apache/atlas/entitytransform/HdfsPathEntityHandler.java
>   
>   
> intg/src/main/java/org/apache/atlas/entitytransform/HiveColumnEntityHandler.java
>   
>   
> intg/src/main/java/org/apache/atlas/entitytransform/HiveDatabaseEntityHandler.java
>   
>   
> intg/src/main/java/org/apache/atlas/entitytransform/HiveStorageDescriptorEntityHandler.java
>   
>   
> intg/src/main/java/org/apache/atlas/entitytransform/HiveTableEntityHandler.java
>   
>   
> intg/src/main/java/org/apache/atlas/entitytransform/TransformationConstants.java
>   
>   intg/src/main/java/org/apache/atlas/entitytransform/TransformerContext.java 
>  
>   intg/src/main/java/org/apache/atlas/listener/ChangedTypeDefs.java  
>   intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java  
>   intg/src/main/java/org/apache/atlas/listener/TypeDefChangeListener.java  
>   intg/src/main/java/org/apache/atlas/pc/StatusReporter.java  
>   intg/src/main/java/org/apache/atlas/pc/WorkItemBuilder.java  
>   intg/src/main/java/org/apache/atlas/pc/WorkItemConsumer.java  
>   intg/src/main/java/org/apache/atlas/pc/WorkItemManager.java  
>   intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java  
>   intg/src/main/java/org/apache/atlas/store/DeleteType.java de5e5e435 
>   
> intg/src/test/java/org/apache/atlas/entitytransform/TransformationHandlerTest.java
>   
>   intg/src/test/java/org/apache/atlas/pc/StatusReporterTest.java  
>   intg/src/test/java/org/apache/atlas/pc/WorkItemConsumerTest.java  
>   intg/src/test/java/org/apache/atlas/pc/WorkItemConsumerWithResultsTest.java 
>  
>   intg/src/test/java/org/apache/atlas/pc/WorkItemManagerTest.java  
>   intg/src/test/java/org/apache/atlas/pc/WorkItemManagerWithResultsTest.java  
>   intg/src/test/resources/json/entity1.json  
>   intg/src/test/resources/json/transform1.json  
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerDelegate.java
>  e414a8c39 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  06341a6d4 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
>  9f8c4c120 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/SoftReferenceTest.java
>  2ffc972ab 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreHardDeleteV2Test.java
>  623235f4b 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreSoftDeleteV2Test.java
>  b169988d0 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2Test.java
>  24683a504 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/InverseReferenceUpdateHardDeleteV2Test.java
>  ed72d5f0b 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/InverseReferenceUpdateSoftDeleteV2Test.java
>  92d1ddcc2 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/InverseReferenceUpdateV2Test.java
>  6364fd44b 
>   server-api/src/main/java/org/apache/atlas/RequestContext.java c5cbf0546 
>   webapp/src/main/java/org/apache/atlas/web/filters/AuditFilter.java 
> 7e6698158 
>   
> webapp/src/test/java/org/apache/atlas/web/adapters/TestEntityRESTDelete.java 
> 6e604f536 
> 
> 
> Diff: https://reviews.apache.org/r/72746/diff/1/
> 
> 
> Testing
> ---
> 
> Pre-commit tests run: 
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2090
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>



Re: Review Request 72713: ATLAS-3892 : Basic search enhanced Feature - Search history

2020-07-30 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72713/#review221433
---




repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
Lines 52 (patched)
<https://reviews.apache.org/r/72713/#comment310439>

nit: unused import. revert



repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
Lines 750 (patched)
<https://reviews.apache.org/r/72713/#comment310440>

escapeIndexQueryValue => rangeStartIndexQueryValue
escapeIndexQueryValue2 => rangeEndIndexQueryValue



repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Lines 902 (patched)
<https://reviews.apache.org/r/72713/#comment310437>

nit: remove new line



repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Line 906 (original), 945 (patched)
<https://reviews.apache.org/r/72713/#comment310438>

nit: extra indent added from line 945-1005. revert


- Sarath Subramanian


On July 30, 2020, 7:34 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72713/
> ---
> 
> (Updated July 30, 2020, 7:34 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3892
> https://issues.apache.org/jira/browse/ATLAS-3892
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1.This patch implements the search history feature where from UI we will 
> receive below filters.
> 2.In backend we have introduced a new operator "TIME_RANGE" where all the 
> below 7 attribute values will have TIME_RANGE operator.
> 3.There will be a slight change in the JSON where previously we were passing 
> the Attribute value in Search parameters from the UI but for below filters   
> we will not pass any attributeValue except for "custom range" filter we 
> provide attribute value 
>   for eg ;"attributeValue":"159354180,159544260" .
> 
> New 7 filters :
> 
> 1.last 7 days.
> 2.Last month.
> 3.Last 30 days.
> 4.Today.
> 5.Yesterday.
> 6.Custom Range.
> 7.This Month.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java a71787bc0 
>   intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 
> fcc4494f2 
>   repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
> c9a605355 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
> 5069d78c8 
>   
> repository/src/test/java/org/apache/atlas/discovery/EntitySearchProcessorTest.java
>  8e42d1700 
> 
> 
> Diff: https://reviews.apache.org/r/72713/diff/5/
> 
> 
> Testing
> ---
> 
> Tested with below json: 
> eg : 
> {
>   "excludeDeletedEntities": true,
>   "includeSubClassifications": true,
>   "includeSubTypes": true,
>   "includeClassificationAttributes": true,
>   "entityFilters": {
> "condition": "AND",
> "criterion": [
>   {
> "attributeName": "__timestamp",
>   "operator": "TIME_RANGE",
>   "attributeValue": "YESTERDAY"
>   }
> ]
>   },
>   "tagFilters": null,
>   "attributes": [
> "__timestamp"
>   ],
>   "limit": 25,
>   "offset": 0,
>   "typeName": "hdfs_path",
>   "classification": null,
>   "termName": null
> }
> eg: Only For custom range filter we provide attribute value.
> {
>   "excludeDeletedEntities": true,
>   "includeSubClassifications": true,
>   "includeSubTypes": true,
>   "includeClassificationAttributes": true,
>   "entityFilters": {
> "condition": "AND",
> "criterion": [
>   {
> "attributeName": "__timestamp",
>   "operator": "TIME_RANGE",
>   "attributeValue": "1595956315473,159596100"
>   }
> ]
>   },
>   "tagFilters": null,
>   "attributes": [
> "__timestamp"
>   ],
>   "limit": 25,
>   "offset": 0,
>   "typeName": "Table",
>   "classification": null,
>   "termName": null
> }
> 
> * Added Test cases to check new processDateRange method functionality
> 
> 
> Thanks,
> 
> chaitali
> 
>



Re: Review Request 72695: Optional configuration to support locks on JanusGraph to ensure data consitency.

2020-07-30 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72695/#review221432
---


Ship it!




Ship It!

- Sarath Subramanian


On July 30, 2020, 3:23 p.m., Damian Warszawski wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72695/
> ---
> 
> (Updated July 30, 2020, 3:23 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Bolke de Bruin, madhan, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Optional configuration to support locks on JanusGraph to ensure data 
> consitency.
> 
> JanusGraph is eventually consistent by default which is efficient but results 
> in duplicates when race condition occurs.
> 
> 
> Reference to jira 
> https://issues.apache.org/jira/projects/ATLAS/issues/ATLAS-3398
> 
> 
> Diffs
> -
> 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
>  6ef9cb76c 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 
>   
> repository/src/test/java/org/apache/atlas/discovery/FreeTextSearchProcessorTest.java
>  464b281fc 
>   test-tools/src/main/resources/solr/core-template/solrconfig.xml 39cc6ab45 
> 
> 
> Diff: https://reviews.apache.org/r/72695/diff/4/
> 
> 
> Testing
> ---
> 
> Not possible to reproduce the error on local machine. Enable locking on our 
> dev env and have not introduce any regression.
> 
> 
> Thanks,
> 
> Damian Warszawski
> 
>



Re: Review Request 72666: Notification: Solution to Memory Build-up

2020-07-30 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72666/#review221431
---


Fix it, then Ship it!




good memory improvement fix!


repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
Line 411 (original), 407 (patched)
<https://reviews.apache.org/r/72666/#comment310435>

nit: revert whitespace only changes in lines:
407-411, 417, 424-425, 470-472, 486-489, 493


- Sarath Subramanian


On July 30, 2020, 3:58 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72666/
> ---
> 
> (Updated July 30, 2020, 3:58 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3878
> https://issues.apache.org/jira/browse/ATLAS-3878
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> See JIRA for details.
> 
> *Analysis* Using memory profiling tools, it was observed that large number of 
> notification objects were created. These stayed in memory and later were 
> promoted to higher generation, thereby taking even longer to be collected.
> 
> **Approach**
> Using the fixed-buffer approach to address the problem of creating large 
> number of small objects.
> 
> New *FixedBufferList* This is an encapsulation over *ArrayList*. During 
> initial allocation, list is populated with default values. Features:
> - Setting of values to these pre-allocated objects is achieved by first doing 
> a *get* on the element and then assigning values to it.
> - *toList* fetches the sub-list from the encapsulating list. This uses the 
> state within the class to fetch the right length for the returning array.
> 
> New *NamedFixedBufferList* Maintains a per-thread *FixedBufferList*. This is 
> necessary since the list is now part class's state.
> Modified *EntityAuditListenerV2* Uses the new classes.
> Modifed *EntityNotificationListener* Uses the new classes.
> 
> **Verification**
> - Using the test setup, the memory usage was observed over a period of 24 
> hrs. 
> - Memory usage and object allocation was obvserved using memory profiler.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 
>   intg/src/main/java/org/apache/atlas/model/Clearable.java PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java 
> 63116d467 
>   intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java PRE-CREATION 
>   intg/src/test/java/org/apache/atlas/utils/FixedBufferListTest.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
>  79527acfa 
> 
> 
> Diff: https://reviews.apache.org/r/72666/diff/11/
> 
> 
> Testing
> ---
> 
> **Unit testing**
> Unit tests added for the new classes.
> 
> **Volume testing**
> Setup:
> - Node: Threads 40, Core: 40, Allocated Memory: 12 GB
> - Multiple Kafka queues ingesting data.
> - Bulk entity creation using custom script ingesting 100M entities.
> 
> Memory usage stayed between 0 and 5% during the 24 hr period.
> 
> With:
> - Workers: 64
> - Batch size: 50 (fewer elements in batch improve commit time and audit write 
> time).
> - Throughput: ~1.2 M entities per hour. Without out of memory error.
> 
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2035/
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2067/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>



Re: Review Request 72567: ATLAS-3782 : Support NOT_CONTAINS operator in basic search

2020-07-23 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72567/#review221344
---


Ship it!




Ship It!

- Sarath Subramanian


On June 30, 2020, 6:09 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72567/
> ---
> 
> (Updated June 30, 2020, 6:09 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3782
> https://issues.apache.org/jira/browse/ATLAS-3782
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Issue:**
> The operator 'SearchParameters.Operator.NOT_CONTAINS' is defined and 
> implemented in SearchProcessors. It would allow a search of entities that do 
> not contain in given string in a specified attribute (eg exclude entities 
> from search that contain 'temp' in the qualified name). 
> 
> **WorkAround:**
> JanusGraph doesn't allow NOT_CONTAINS operator. So we will handle this in 
> inmemory
> BasicSearch generates query via three modes
> 1. Index query -> NOT_CONTAINS will not be supported
> 2. InMemeory Predicates -> NOT_CONTAINS will be supported, Already handled 
> #123 SearchProcessor
> 3. Graph query -> NOT_CONTAINS will not be supported
> 
> As in index and graph query wiil not support not_Contains operator, We need 
> to apply filter(inMemoryPredicate) after either index/graph query.
> 
> To support above, I have modified ClassificationSearchProcessor
> + For both cases index as well as graph, added typeNamePredicate and 
> attributePredicate
> + Added these predicate after query
> - Removed gremlinQuery block
> 
> **Operator Value:**
> It can be either "not_contains" or "NOT_CONTAINS"
> 
> **Note:**
> As part of this jira, not_contains is also added in quick search.
> 
> 
> Diffs
> -
> 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasSolrQueryBuilder.java
>  6c06a3cbe 
>   
> repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java
>  9c72cd4a2 
>   repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
> c9a605355 
>   repository/src/test/java/org/apache/atlas/BasicTestSetup.java 8b98b3990 
>   
> repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/discovery/BasicSearchClassificationTest.java
>  9b16e919d 
>   
> repository/src/test/java/org/apache/atlas/discovery/EntitySearchProcessorTest.java
>  b7ce97845 
> 
> 
> Diff: https://reviews.apache.org/r/72567/diff/4/
> 
> 
> Testing
> ---
> 
> Added testcases
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1999
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>



Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

2020-07-23 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221343
---




repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
Line 607 (original), 616 (patched)
<https://reviews.apache.org/r/72646/#comment310237>

what if out/in edges size is a lot, do you bring everything into memory 
just to get size? 

Do you need entire edges in the list or maybe we can maintain a counter and 
increment?



repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
Lines 682 (patched)
<https://reviews.apache.org/r/72646/#comment310236>

Constants.GUID_PROPERTY_KEY).value() can be null (mostly not, but some bad 
vertices). Consider checking for null value of guid.

Also, since you already have vertex, do we need additional 
'endVerticesGuid' list? Can we extract 'guid' from Vertex 'v' and add it to 
'resultList' is same loop. 

resultList.add(entityRetriever.toAtlasEntityHeader(endVertexGuid, 
attributes));

(or)

Maybe map Vertex to AtlasVertex and directly call:

resultList.add(entityRetriever.toAtlasEntityHeader(atlasVertex, 
attributes));


- Sarath Subramanian


On July 6, 2020, 2:14 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> ---
> 
> (Updated July 6, 2020, 2:14 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
> https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in 
> the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the 
> offset/limit.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related 
> entities**  .
> Average time taken for the Api to search relationship entities having **5000 
> end vertices** with limit **500**:
> Before: 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  4b9564295 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
> 5069d78c8 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/2/
> 
> 
> Testing
> ---
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 
> (Failed in Impala build)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>



Re: Review Request 72703: Import Service: UpdateVertexGuid Now Makes Updates to AtlasEntityWithExtInfo

2020-07-23 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72703/#review221341
---


Ship it!




Ship It!

- Sarath Subramanian


On July 23, 2020, 11:29 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72703/
> ---
> 
> (Updated July 23, 2020, 11:29 a.m.)
> 
> 
> Review request for atlas, Nikhil Bonte, Nixon Rodrigues, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-3902
> https://issues.apache.org/jira/browse/ATLAS-3902
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> (Modified) *RegularImport.updateVertexGuid* Updated method to handle 
> *AtlasEntityWithExtInfo*.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/RegularImport.java
>  3f7e86167 
> 
> 
> Diff: https://reviews.apache.org/r/72703/diff/1/
> 
> 
> Testing
> ---
> 
> **Functional**
> - On a cluster with Atlas, perform import using REST calls with 
> *stocks-1.zip* and *stock-2.zip*.
> 
> ```
> curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H 
> "Cache-Control: no-cache" -F data=@./stocks-1.zip 
> "http://localhost:21000/api/atlas/admin/import;
> 
> ```
> 
> 
> File Attachments
> 
> 
> Stocks-1
>   
> https://reviews.apache.org/media/uploaded/files/2020/07/23/ae57756a-71dc-4cf4-8ae7-70270c14ae08__stocks-1.zip
> Stocks-2
>   
> https://reviews.apache.org/media/uploaded/files/2020/07/23/b0813b02-e44f-40a8-b52c-2dd620d067d9__stocks-2.zip
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>



Re: Review Request 72666: Notification: Solution to Memory Build-up

2020-07-23 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72666/#review221337
---




repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
Lines 409 (patched)
<https://reviews.apache.org/r/72666/#comment310231>

should we reset/clear these values for EntityAuditEventV2? It might have 
remnant values when reused again?


- Sarath Subramanian


On July 23, 2020, 11:48 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72666/
> ---
> 
> (Updated July 23, 2020, 11:48 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3878
> https://issues.apache.org/jira/browse/ATLAS-3878
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> See JIRA for details.
> 
> *Analysis* Using memory profiling tools, it was observed that large number of 
> notification objects were created. These stayed in memory and later were 
> promoted to higher generation, thereby taking even longer to be collected.
> 
> **Approach**
> Using the fixed-buffer approach to address the problem of creating large 
> number of small objects.
> 
> New *FixedBufferList* This is an encapsulation over *ArrayList*. During 
> initial allocation, list is populated with default values. Features:
> - Setting of values to these pre-allocated objects is achieved by first doing 
> a *get* on the element and then assigning values to it.
> - *toList* fetches the sub-list from the encapsulating list. This uses the 
> state within the class to fetch the right length for the returning array.
> 
> New *NamedFixedBufferList* Maintains a per-thread *FixedBufferList*. This is 
> necessary since the list is now part class's state.
> Modified *EntityAuditListenerV2* Uses the new classes.
> Modifed *EntityNotificationListener* Uses the new classes.
> 
> **Verification**
> - Using the test setup, the memory usage was observed over a period of 24 
> hrs. 
> - Memory usage and object allocation was obvserved using memory profiler.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 
>   intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/utils/FixedBufferListAccessor.java 
> PRE-CREATION 
>   intg/src/test/java/org/apache/atlas/utils/FixedBufferListAccessorTest.java 
> PRE-CREATION 
>   intg/src/test/java/org/apache/atlas/utils/FixedBufferListTest.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
>  79527acfa 
>   
> webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java
>  a677b315c 
> 
> 
> Diff: https://reviews.apache.org/r/72666/diff/7/
> 
> 
> Testing
> ---
> 
> **Unit testing**
> Unit tests added for the new classes.
> 
> **Volume testing**
> Setup:
> - Node: Threads 40, Core: 40, Allocated Memory: 12 GB
> - Multiple Kafka queues ingesting data.
> - Bulk entity creation using custom script ingesting 100M entities.
> 
> Memory usage stayed between 0 and 5% during the 24 hr period.
> 
> With:
> - Workers: 64
> - Batch size: 50 (fewer elements in batch improve commit time and audit write 
> time).
> - Throughput: ~1.2 M entities per hour. Without out of memory error.
> 
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2035/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>



Re: Review Request 72666: Notification: Solution to Memory Build-up

2020-07-23 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72666/#review221322
---




intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java
Lines 33 (patched)
<https://reviews.apache.org/r/72666/#comment310204>

ArrayList => List



intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java
Lines 59 (patched)
<https://reviews.apache.org/r/72666/#comment310206>

request => initializeBuffer()



intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java
Lines 105 (patched)
<https://reviews.apache.org/r/72666/#comment310205>

getActualTypeArguments() might return empty array. consider checking length



repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
Lines 97 (patched)
<https://reviews.apache.org/r/72666/#comment310219>

do we need to create a new instance and pass to the constructor? The 
constructor just needs the class.

consider updating constructor of FixedBufferListAccessor to take class:

this.fixedBufferListAccessor = new 
FixedBufferListAccessor<>(EntityAuditEventV2FixedList.class);

same for line EntityNotificationListenerV2.java line#94


- Sarath Subramanian


On July 22, 2020, 9:16 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72666/
> ---
> 
> (Updated July 22, 2020, 9:16 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3878
> https://issues.apache.org/jira/browse/ATLAS-3878
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> See JIRA for details.
> 
> *Analysis* Using memory profiling tools, it was observed that large number of 
> notification objects were created. These stayed in memory and later were 
> promoted to higher generation, thereby taking even longer to be collected.
> 
> **Approach**
> Using the fixed-buffer approach to address the problem of creating large 
> number of small objects.
> 
> New *FixedBufferList* This is an encapsulation over *ArrayList*. During 
> initial allocation, list is populated with default values. Features:
> - Setting of values to these pre-allocated objects is achieved by first doing 
> a *get* on the element and then assigning values to it.
> - *toList* fetches the sub-list from the encapsulating list. This uses the 
> state within the class to fetch the right length for the returning array.
> 
> New *NamedFixedBufferList* Maintains a per-thread *FixedBufferList*. This is 
> necessary since the list is now part class's state.
> Modified *EntityAuditListenerV2* Uses the new classes.
> Modifed *EntityNotificationListener* Uses the new classes.
> 
> **Verification**
> - Using the test setup, the memory usage was observed over a period of 24 
> hrs. 
> - Memory usage and object allocation was obvserved using memory profiler.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 
>   intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/utils/FixedBufferListAccessor.java 
> PRE-CREATION 
>   intg/src/test/java/org/apache/atlas/utils/FixedBufferListAccessorTest.java 
> PRE-CREATION 
>   intg/src/test/java/org/apache/atlas/utils/FixedBufferListTest.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
>  79527acfa 
>   
> webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java
>  a677b315c 
> 
> 
> Diff: https://reviews.apache.org/r/72666/diff/6/
> 
> 
> Testing
> ---
> 
> **Unit testing**
> Unit tests added for the new classes.
> 
> **Volume testing**
> Setup:
> - Node: Threads 40, Core: 40, Allocated Memory: 12 GB
> - Multiple Kafka queues ingesting data.
> - Bulk entity creation using custom script ingesting 100M entities.
> 
> Memory usage stayed between 0 and 5% during the 24 hr period.
> 
> With:
> - Workers: 64
> - Batch size: 50 (fewer elements in batch improve commit time and audit write 
> time).
> - Throughput: ~1.2 M entities per hour. Without out of memory error.
> 
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2035/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>



Re: Review Request 72698: ATLAS-3875: Introduce sample project for AtlasClient

2020-07-22 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72698/#review221306
---




atlas-examples/src/main/java/org/apache/atlas/AtlasClientBaseExample.java
Lines 20 (patched)
<https://reviews.apache.org/r/72698/#comment310188>

nit: removed unused imports line 20,21



atlas-examples/src/main/java/org/apache/atlas/AtlasClientBaseExample.java
Lines 134 (patched)
<https://reviews.apache.org/r/72698/#comment310189>

'url' is never used. consider removing.



atlas-examples/src/main/java/org/apache/atlas/TypeDefExample.java
Lines 58 (patched)
<https://reviews.apache.org/r/72698/#comment310190>

isCompositeSourceConstraint => isOwnedRefConstraint



atlas-examples/src/main/java/org/apache/atlas/TypeDefExample.java
Lines 59 (patched)
<https://reviews.apache.org/r/72698/#comment310191>

isCompositeTargetConstraint => isInverseRefConstraint


- Sarath Subramanian


On July 21, 2020, 9:24 a.m., Jyoti Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72698/
> ---
> 
> (Updated July 21, 2020, 9:24 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath 
> Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3875
> https://issues.apache.org/jira/browse/ATLAS-3875
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Using this project users can get an idea as how to integrate with Atlas using 
> AtlasCleint. This helps the user to understand the basic rest functionality 
> of Atlas such as
> 
> - EntityRest
> - TypeDefRest
> - DiscoveryRest
> - LineageRest
> - GlossaryRest
> 
> 
> Diffs
> -
> 
>   atlas-examples/pom.xml PRE-CREATION 
>   atlas-examples/src/main/java/org/apache/atlas/AtlasClientBaseExample.java 
> PRE-CREATION 
>   atlas-examples/src/main/java/org/apache/atlas/DiscoveryExample.java 
> PRE-CREATION 
>   atlas-examples/src/main/java/org/apache/atlas/EntityRestBaseExample.java 
> PRE-CREATION 
>   atlas-examples/src/main/java/org/apache/atlas/GlossaryExample.java 
> PRE-CREATION 
>   atlas-examples/src/main/java/org/apache/atlas/LineageExample.java 
> PRE-CREATION 
>   atlas-examples/src/main/java/org/apache/atlas/TypeDefExample.java 
> PRE-CREATION 
>   pom.xml 5e0442ae5 
> 
> 
> Diff: https://reviews.apache.org/r/72698/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jyoti Singh
> 
>



[jira] [Updated] (ATLAS-3875) Adding missing APIs in AtlasClient with test cases

2020-07-16 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3875:
--
Affects Version/s: 2.1.0

> Adding missing APIs in AtlasClient with test cases
> --
>
> Key: ATLAS-3875
> URL: https://issues.apache.org/jira/browse/ATLAS-3875
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Jyoti Singh
>Assignee: Jyoti Singh
>Priority: Major
>  Labels: api, client
> Attachments: ATLAS-3875-1.patch
>
>
> There are many new APIs added to Atlas Project but the corresponding  APIs 
> are missing from AtlasClientv2. The aim of this task is to complete the gap 
> amongst existing APIs and their endpoints in Atls client. This will also 
> include adding test cases via integration testing.
> There are functions from AtlasClient for the following REST endpoints
>  * TypeRest
>  * EntityRest
>  * LineageRest
>  * DiscoveryRest
>  * GlossaryRest
>  * RelationshipRest



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 72638: ATLAS-3875: Enhancing AtlasClient by adding missing API endpoints

2020-07-16 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72638/#review221235
---




client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 117 (patched)
<https://reviews.apache.org/r/72638/#comment310069>

SAVED_URI => SAVED_SEARCH_URI



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 118 (patched)
<https://reviews.apache.org/r/72638/#comment310070>

QUICK_URI => QUICK_SEARCH_URI



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 132 (patched)
<https://reviews.apache.org/r/72638/#comment310071>

nit: typo in 'relateed'



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 244 (patched)
<https://reviews.apache.org/r/72638/#comment310072>

nit: remove additional newline



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 250 (patched)
<https://reviews.apache.org/r/72638/#comment310073>

getBusinessMetadataDefGuid => getBusinessMetadataDefByGuid



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 525 (patched)
<https://reviews.apache.org/r/72638/#comment310074>

removeBusinessAttributes => removeBusinessAttributesByName



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 543 (patched)
<https://reviews.apache.org/r/72638/#comment310075>

setLabels => setLabelsByUniqueAttributes
addLabels => addLabelsByUniqueAttributes
removeLabels => removeLabelsByUniqueAttributes



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 570 (patched)
<https://reviews.apache.org/r/72638/#comment310076>

getLineageInfo => getLineageInfoByUniqueAttributes



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 683 (patched)
<https://reviews.apache.org/r/72638/#comment310077>

nit: additional space after 'void'



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 687 (patched)
<https://reviews.apache.org/r/72638/#comment310078>

nit: additional space after 'AtlasSearchResult'



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 693 (patched)
<https://reviews.apache.org/r/72638/#comment310079>

nit: additional space after 'AtlasSearchResult'



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 937 (patched)
<https://reviews.apache.org/r/72638/#comment310080>

nit: fix alignment line 937-940



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 1073 (patched)
<https://reviews.apache.org/r/72638/#comment310082>

DELETE_TYPE_BY_NAME => DELETE_TYPE_DEF_BY_NAME



client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
Lines 1095 (patched)
<https://reviews.apache.org/r/72638/#comment310083>

nit: fix alignment from line 1095-1114



client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java
Lines 685 (patched)
<https://reviews.apache.org/r/72638/#comment310068>

this method is not used anywhere. review and revert  if not needed.



webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java
Lines 579 (patched)
<https://reviews.apache.org/r/72638/#comment310084>

nit: remove additional new line



webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java
Lines 608 (patched)
<https://reviews.apache.org/r/72638/#comment310085>

nit: remove additional new line


- Sarath Subramanian


On July 15, 2020, 8:17 p.m., Jyoti Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72638/
> -------
> 
> (Updated July 15, 2020, 8:17 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath 
> Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3875
> https://issues.apache.org/jira/browse/ATLAS-3875
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> There are many missing API endpoints in AtlasClientV2. This solution is 
> adding functions corresponding to those missing APIs for the following Rest 
> endpoints.
> 
> 1. TypeRest
> 2. EntityRest
> 3. LineageRest
> 4. DiscoveryRest
> 5. GlossaryRest
> 6. RelationshipRest
> 
> This will enable users to test and integrate with APIs more effectively via 
> AtlasCient.
> 
> 
> Diffs
> -
> 
>   client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 
> 7b6d1d0f3 
>   client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java e3d2ebc34 
>   webapp/pom.xml 

[jira] [Updated] (ATLAS-3885) import-hive.sh: Hive entities with Ozone location created by import-hive.sh creates hdfs_path entity instead of ozone in Atlas

2020-07-13 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3885:
--
Labels: import ozone  (was: ozone)

> import-hive.sh: Hive entities with Ozone location created by import-hive.sh 
> creates hdfs_path entity instead of ozone in Atlas
> --
>
> Key: ATLAS-3885
> URL: https://issues.apache.org/jira/browse/ATLAS-3885
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>  Labels: import, ozone
> Fix For: trunk, 2.1.0
>
> Attachments: ATLAS-3885-import-hive-Hive-entities-with-Ozone-v0.patch
>
>
> import-hive.sh when attempts to import hive entities with Ozone location 
> expected is that ozone_key should get created as path entity, but script 
> creates hdfs_path entity instead of ozone_key in Atlas.
>  
> Steps to reproduce
>  # Disable HMS Hook and Hive Hook
>  # Create external table with ozone location
>  # Run import-hive.sh
> The location entity is created as an hdfs_path entity instead of ozone_key.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3885) import-hive.sh: Hive entities with Ozone location created by import-hive.sh creates hdfs_path entity instead of ozone in Atlas

2020-07-13 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3885:
--
Fix Version/s: 2.1.0
   trunk

> import-hive.sh: Hive entities with Ozone location created by import-hive.sh 
> creates hdfs_path entity instead of ozone in Atlas
> --
>
> Key: ATLAS-3885
> URL: https://issues.apache.org/jira/browse/ATLAS-3885
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>  Labels: ozone
> Fix For: trunk, 2.1.0
>
> Attachments: ATLAS-3885-import-hive-Hive-entities-with-Ozone-v0.patch
>
>
> import-hive.sh when attempts to import hive entities with Ozone location 
> expected is that ozone_key should get created as path entity, but script 
> creates hdfs_path entity instead of ozone_key in Atlas.
>  
> Steps to reproduce
>  # Disable HMS Hook and Hive Hook
>  # Create external table with ozone location
>  # Run import-hive.sh
> The location entity is created as an hdfs_path entity instead of ozone_key.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: [VOTE] Release Apache Atlas version 2.1.0 - rc3

2020-07-13 Thread Sarath Subramanian
+1 for Apache Atlas 2.1 rc3

Validated the following:
- Validated MD5 checksum
- Validated sha512 hash
- Verified PGP signature
- No binaries in extracted source
- Built successfully using embedded-hbase and solr profile
- Ran quick start and tested basic sanity - saved search, basic/advanced
search, Glossary, tag propagation, Admin purge, system attribute search



On Thu, Jul 9, 2020 at 9:59 PM Madhan Neethiraj  wrote:

> Atlas team,
>
> Apache Atlas 2.1.0 rc3, with following fixes since rc2, is now available
> for vote within dev community.
>   ATLAS-3877: Purged entity audit throws 404 Entity not found error
>   ATLAS-3882: Classic UI: Loader not disappearing while assigning category
> to term
>
> Links to the release artifacts are given below. Please review and vote.
>
> The vote will be open for at least 72 hours or until necessary votes are
> reached.
>   [ ] +1 Approve
>   [ ] +0 No opinion
>   [ ] -1 Disapprove (and reason why)
>
> Thanks,
> Madhan
>
>
> List of all issues addressed in this release:
> https://issues.apache.org/jira/issues/?jql=project=ATLAS AND
> status=Resolved AND fixVersion=2.1.0 ORDER BY key DESC
>
> Git tag for the release:
> https://github.com/apache/atlas/tree/release-2.1.0-rc3
> Sources for the release:
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc3/apache-atlas-2.1.0-sources.tar.gz
>
> Source release verification:
>   PGP Signature:
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc3/apache-atlas-2.1.0-sources.tar.gz.asc
>   SHA512 Hash:
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc3/apache-atlas-2.1.0-sources.tar.gz.sha512
>   MD5 Hash:
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc3/apache-atlas-2.1.0-sources.tar.gz.md5
>
> Keys to verify the signature of the release artifacts are available at:
> https://dist.apache.org/repos/dist/release/atlas/KEYS
>
> New features/enhancements:
>   - Quick Search: provides a simpler search experience with type-ahead
> suggestions
>   - Business Metadata: enables augmenting entity-types with additional
> attributes, search entities using these attributes
>   - Labels: ability to add/remove labels on entities, and search entities
> using labels
>   - Custom Attributes: ability to add entity instance specific custom
> attributes i.e. attributes not defined in entity-def or business metadata
>   - Entity Purge: added REST APIs to purge deleted entities
>
> Enhancements:
>   - Search: ability to find entities by more than one classification
>   - Performance: improvements in lineage retrieval and
> classification-propagation
>   - Notification: ability to process notifications from multiple Kafka
> topics
>   - Hive Hook: tracks process-executions via hive_process_execution
> entities
>   - Hive Hook: captures DDL operations via hive_db_ddl and hive_table_ddl
> entities
>   - Notification: introduced shell entities to record references to
> non-existing entities in notifications
>   - Spark: added model to capture Spark entities, processes and
> relationships
>   - AWS S3: introduced updated model to capture AWS S3 entities and
> relationships
>   - ADLS-Gen2: introduced model to capture Azure Data Lake Storage Gen2
> entities and relationships
>   - Dependencies: JanusGraph 0.5.1, Tinkerpop 3.4.6, Spring Framework
> 4.3.20
>   - Authorization: updated to cover new features, like: business metadata,
> labels, purge
>   - UI: multiple UI improvements, including a beta UI
>
>
>


Re: Review Request 72658: ATLAS-3877: Audit API returns the audit information for an unauthorised entity

2020-07-08 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72658/#review221157
---


Ship it!




Ship It!

- Sarath Subramanian


On July 8, 2020, 12:31 a.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72658/
> ---
> 
> (Updated July 8, 2020, 12:31 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-3877
> https://issues.apache.org/jira/browse/ATLAS-3877
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-3877: Audit Fixed the authorization for audit both in case of non 
> purged entity to populate classifcation and purged entity by getting the 
> header from HBase Audit
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  717310daf 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
>  bf1629cb3 
>   webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 88de8b679 
> 
> 
> Diff: https://reviews.apache.org/r/72658/diff/2/
> 
> 
> Testing
> ---
> 
> #Deleted Entity
> //Allow
> curl --location --request GET 
> 'http://sid-ycloud-atlas1-2.sid-ycloud-atlas1.root.hwx.site:31000/api/atlas/v2/entity/196bf3a9-07c6-4152-82ad-10176c5df5b9/audit?count=1=ENTITY_DELETE'
>  \
> -u admin:admin123
> 
> Response:
> [{"entityId":"196bf3a9-07c6-4152-82ad-10176c5df5b9","timestamp":1594192518488,"user":"admin","action":"ENTITY_DELETE","details":"Deleted
>  
> entity","eventKey":"196bf3a9-07c6-4152-82ad-10176c5df5b9:1594192518488:1:1594192519277","entity":null,"type":null}]
> 
> //Deny
> curl --location --request GET 
> 'http://sid-ycloud-atlas1-2.sid-ycloud-atlas1.root.hwx.site:31000/api/atlas/v2/entity/196bf3a9-07c6-4152-82ad-10176c5df5b9/audit?count=1=ENTITY_DELETE'
>  \
> -u sid:sid
> 
> Response:
> {"errorCode":"ATLAS-403-00-001","errorMessage":"sid is not authorized to 
> perform read entity: guid=196bf3a9-07c6-4152-82ad-10176c5df5b9"}
> 
> 
> #Purged Entity
> //Allow
> curl --location --request GET 
> 'http://sid-ycloud-atlas1-2.sid-ycloud-atlas1.root.hwx.site:31000/api/atlas/v2/entity/def375bd-9bad-42f3-9fa7-9edc1c7d8d96/audit?count=1=ENTITY_PURGE'
>  \
> -u admin:admin123
> 
> Response:
> [{"entityId":"def375bd-9bad-42f3-9fa7-9edc1c7d8d96","timestamp":1594191071761,"user":"admin","action":"ENTITY_PURGE","details":"Purged:
>  
> {\"typeName\":\"hive_table\",\"attributes\":{\"owner\":\"admin\",\"createTime\":1594177077000,\"qualifiedName\":\"default.test_july_7_2020_ctas_10@cm\",\"name\":\"test_july_7_2020_ctas_10\"},\"guid\":\"def375bd-9bad-42f3-9fa7-9edc1c7d8d96\",\"isIncomplete\":false,\"provenanceType\":0,\"status\":\"DELETED\",\"version\":0,\"classifications\":[{\"typeName\":\"NOT_IMP\",\"entityGuid\":\"5f52ee87-85c8-46ac-ad38-e0e5be935bb2\",\"entityStatus\":\"ACTIVE\",\"propagate\":true,\"validityPeriods\":[],\"removePropagationsOnEntityDelete\":false},{\"typeName\":\"PARENT\",\"entityGuid\":\"73cc5730-494b-4313-8747-9ccf8a1b0227\",\"entityStatus\":\"ACTIVE\",\"propagate\":true,\"validityPeriods\":[],\"removePropagationsOnEntityDelete\":false}],\"meanings\":[],\"proxy\":false}","eventKey":"def375bd-9bad-42f3-9fa7-9edc1c7d8d96:1594191071761:3:1594191103634","entity":nu
 ll,"type":null}]
> 
> //Deny
> curl --location --request GET 
> 'http://sid-ycloud-atlas1-2.sid-ycloud-atlas1.root.hwx.site:31000/api/atlas/v2/entity/def375bd-9bad-42f3-9fa7-9edc1c7d8d96/audit?count=1=ENTITY_PURGE'
>  \
> -u sid:sid
> 
> Response:
> {"errorCode":"ATLAS-403-00-001","errorMessage":"sid is not authorized to 
> perform read entity audit: guid=def375bd-9bad-42f3-9fa7-9edc1c7d8d96"}
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>



Re: Review Request 72615: ATLAS-3855 Bulk entity tag association and bulk api enhancement

2020-07-08 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72615/#review221151
---


Ship it!




Ship It!

- Sarath Subramanian


On July 7, 2020, 11:45 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72615/
> ---
> 
> (Updated July 7, 2020, 11:45 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Nikhil Bonte, Nixon Rodrigues, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3855
> https://issues.apache.org/jira/browse/ATLAS-3855
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> API : /api/atlas/v2/entity/bulk/classification & v2/entity/bulk : if some 
> guids belong to entities on which user is unauthorized other guids belong to 
> entities on which user is authorized This APIs fail with 403 error without 
> returning the authorized entities.
> 
> 1.Unauthorized guids are filtered with this patch for both the APIs.
> 2.Added ignoreUnauthorisedGuids flag for /bulk/classification API as it 
> doesn't return any object
> 2.Also added unauthorized guids in return object of /bulk in response for 
> user's reference.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
>  bf1629cb3 
>   server-api/src/main/java/org/apache/atlas/RequestContext.java 282a66f1d 
>   webapp/src/main/java/org/apache/atlas/web/filters/AuditFilter.java 
> c663b00c9 
> 
> 
> Diff: https://reviews.apache.org/r/72615/diff/9/
> 
> 
> Testing
> ---
> 
> Tested with ranger policies:
> For /bulk api following policy were applied:
> 1.Added hive_column entity as entity type.
> 2.Included admin as user and prohibited access for read entity all this under 
> deny policy.
> 
> For /bulk/classification following policy were applied:
> 1.Added hive_column entity as entity type.
> 2.Gave all the access to admin but kwpt hive_column entity type in excluded 
> state.
> 
> 
> Thanks,
> 
> chaitali
> 
>



[jira] [Updated] (ATLAS-3875) Adding missing APIs in AtlasClient with test cases

2020-07-01 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3875:
--
Labels: api client  (was: )

> Adding missing APIs in AtlasClient with test cases
> --
>
> Key: ATLAS-3875
> URL: https://issues.apache.org/jira/browse/ATLAS-3875
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Jyoti Singh
>Assignee: Jyoti Singh
>Priority: Major
>  Labels: api, client
>
> There are many new APIs added to Atlas Project but the corresponding  APIs 
> are missing from AtlasClientv2. The aim of this task is to complete the gap 
> amongst existing APIs and their endpoints in Atls client. This will also 
> include adding test cases via integration testing.
> There are functions from AtlasClient for the following REST endpoints
>  * TypeRest
>  * EntityRest
>  * LineageRest
>  * DiscoveryRest
>  * GlossaryRest
>  * RelationshipRest



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3875) Adding missing APIs in AtlasClient with test cases

2020-07-01 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3875:
--
Component/s:  atlas-core

> Adding missing APIs in AtlasClient with test cases
> --
>
> Key: ATLAS-3875
> URL: https://issues.apache.org/jira/browse/ATLAS-3875
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Jyoti Singh
>Assignee: Jyoti Singh
>Priority: Major
>
> There are many new APIs added to Atlas Project but the corresponding  APIs 
> are missing from AtlasClientv2. The aim of this task is to complete the gap 
> amongst existing APIs and their endpoints in Atls client. This will also 
> include adding test cases via integration testing.
> There are functions from AtlasClient for the following REST endpoints
>  * TypeRest
>  * EntityRest
>  * LineageRest
>  * DiscoveryRest
>  * GlossaryRest
>  * RelationshipRest



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3869) Upgrade httpclient from 4.5.3 to 4.5.4

2020-07-01 Thread Sarath Subramanian (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149715#comment-17149715
 ] 

Sarath Subramanian commented on ATLAS-3869:
---

[~nixon], consider merging this to branch-2.0 as well.

> Upgrade httpclient from 4.5.3 to 4.5.4
> --
>
> Key: ATLAS-3869
> URL: https://issues.apache.org/jira/browse/ATLAS-3869
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: ATLAS-3869-Upgrade-httpclient-from-4.5.3-to-4.5.4.patch
>
>
> Found below exception where data was not getting added in SOLR from 
> janusgraph.
>  
> {noformat}
> Caused by: java.lang.ClassCastException: [B cannot be cast to java.lang.String
>   at 
> org.apache.http.conn.ssl.DefaultHostnameVerifier.getSubjectAltNames(DefaultHostnameVerifier.java:309)
>   at 
> org.apache.http.conn.ssl.DefaultHostnameVerifier.verify(DefaultHostnameVerifier.java:112)
>   at 
> org.apache.http.conn.ssl.DefaultHostnameVerifier.verify(DefaultHostnameVerifier.java:99)
>   at 
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:463)
>   at 
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:397)
>   at 
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355)
>   at 
> org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
>   at 
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359)
>   at 
> org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
>   at 
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
>   at 
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
>   at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
>   at 
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
>   at 
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
>   at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
>   at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
>   at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:526)
>   at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
>   at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
>   at 
> org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
>   ... 7 more
> 2020-06-29 07:31:25,893 WARN  - [SIGTERM handler:] ~ Unable to close 
> transaction standardjanusgraphtx[0x18d7a16a] (StandardJanusGraph:230)
> java.lang.IllegalArgumentException: The transaction has already been closed
>   at 
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:142)
>   at 
> org.janusgraph.graphdb.transaction.StandardJanusGraphTx.rollback(StandardJanusGraphTx.java:1401)
>   at 
> org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsTransaction$1.doRollback(JanusGraphBlueprintsTransaction.java:190)
>   at 
> org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.rollback(AbstractTransaction.java:114)
>   at 
> org.apache.tinkerpop.gremlin.structure.Transaction$CLOSE_BEHAVIOR$2.accept(Transaction.java:182)
>   at 
> org.apache.tinkerpop.gremlin.structure.Transaction$CLOSE_BEHAVIOR$2.accept(Transaction.java:179)
>   at 
> org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.close(JanusGraphBlueprintsGraph.java:327)
>   at 
> org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsTransaction$1.doClose(JanusGraphBlueprintsTransaction.java:205)
>   at 
> org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.close(AbstractTransaction.java:139)
>   at 
> org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsTransaction.close(JanusGraphBlueprintsTransaction.java:215)
>   at 
> org.janusgraph.graphdb.database.StandardJanusGraph.closeInternal(StandardJanusGraph.java:226)
>   at 
> org.janusgraph.graphdb.database.StandardJanusGraph.access$500(StandardJanusGraph.java:91)
>   at 
> org.janusgr

[jira] [Updated] (ATLAS-3875) Adding missing APIs in AtlasClient with test cases

2020-07-01 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3875:
--
Affects Version/s: 2.0.0

> Adding missing APIs in AtlasClient with test cases
> --
>
> Key: ATLAS-3875
> URL: https://issues.apache.org/jira/browse/ATLAS-3875
> Project: Atlas
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Jyoti Singh
>Assignee: Jyoti Singh
>Priority: Major
>
> There are many new APIs added to Atlas Project but the corresponding  APIs 
> are missing from AtlasClientv2. The aim of this task is to complete the gap 
> amongst existing APIs and their endpoints in Atls client. This will also 
> include adding test cases via integration testing.
> There are functions from AtlasClient for the following REST endpoints
>  * TypeRest
>  * EntityRest
>  * LineageRest
>  * DiscoveryRest
>  * GlossaryRest
>  * RelationshipRest



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (ATLAS-3871) Add unit tests to path extractor utility for s3, s3 v2, adls path entities

2020-07-01 Thread Sarath Subramanian (Jira)


[ 
https://issues.apache.org/jira/browse/ATLAS-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149714#comment-17149714
 ] 

Sarath Subramanian commented on ATLAS-3871:
---

please merge this in branch-2.0 as well.

> Add unit tests to path extractor utility for s3, s3 v2, adls path entities
> --
>
> Key: ATLAS-3871
> URL: https://issues.apache.org/jira/browse/ATLAS-3871
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
>
> AtlasPathExtractorUtilTest added in ATLAS-3836 which covers unit tests for 
> ozone & hdfs path entites.
>  
> This Jira will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & 
> adls path entities.
> Would add more tests for ozone as well.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 72627: ATLAS-3867 : Relationship search API should have a provision to fetch custom attributes in search results

2020-06-30 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72627/#review221106
---


Ship it!




Ship It!

- Sarath Subramanian


On June 30, 2020, 10:26 p.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72627/
> ---
> 
> (Updated June 30, 2020, 10:26 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3867
> https://issues.apache.org/jira/browse/ATLAS-3867
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Issue:** Attributes in search result of Relationship api are minimal.
> 
> **WorkAround:** Relationship api request should have provision to specify 
> attributes to be present in search result.
> 
> **Example Request:** 
> /v2/search/relationship?guid=ac9e04cc-f927-4334-af08-c83bc3733f5b=columns=name=ASCENDING=dcProfiledData
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java
>  e64c31522 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  4169e14ef 
>   webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 076284ec4 
> 
> 
> Diff: https://reviews.apache.org/r/72627/diff/2/
> 
> 
> Testing
> ---
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1994
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>



Re: Review Request 72631: ATLAS-3871 Add unit tests to path extractor utility for s3, s3 v2, adls path entities

2020-06-30 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72631/#review221105
---


Ship it!




Ship It!

- Sarath Subramanian


On June 30, 2020, 9:29 p.m., Nikhil Bonte wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72631/
> ---
> 
> (Updated June 30, 2020, 9:29 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
> Nixon Rodrigues, Sarath Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3871
> https://issues.apache.org/jira/browse/ATLAS-3871
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> AtlasPathExtractorUtilTest added in ATLAS-3836 which covers unit tests for 
> ozone & hdfs path entites.
>  
> This will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & adls 
> path entities.
> 
> Would add more tests for ozone as well.
> 
> 
> Diffs
> -
> 
>   common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java 
> 664bfb72b 
> 
> 
> Diff: https://reviews.apache.org/r/72631/diff/2/
> 
> 
> Testing
> ---
> 
> PC build 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2001/console 
> (Finished: SUCCESS)
> 
> 
> Thanks,
> 
> Nikhil Bonte
> 
>



Re: Review Request 72625: ATLAS-3866 : Relationship search API for hive storage desc throws error code 500

2020-06-30 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72625/#review221104
---


Ship it!




Ship It!

- Sarath Subramanian


On June 29, 2020, 3:49 a.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72625/
> ---
> 
> (Updated June 29, 2020, 3:49 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3866
> https://issues.apache.org/jira/browse/ATLAS-3866
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Issue:** In relationship api v2/relationship, for hive_storagedesc it 
> throws exception
> 
> **Reason:** it is because, sort attribute is assigned 'name' as default, and 
> 'name' attribute is not in the defination of hive_storagedesc.
> 
> **Workaround:** Validate if attribute is present in relationship end def, if 
> not ignore sorting
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  dd4d1b441 
> 
> 
> Diff: https://reviews.apache.org/r/72625/diff/2/
> 
> 
> Testing
> ---
> 
> Manually tested
> Precommit : 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1992/console
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>



[jira] [Updated] (ATLAS-3870) UI: hive_table entity table not updating queryparams based on filter selected

2020-06-30 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3870:
--
Component/s: atlas-webui

> UI: hive_table entity table not updating queryparams based on filter selected
> -
>
> Key: ATLAS-3870
> URL: https://issues.apache.org/jira/browse/ATLAS-3870
> Project: Atlas
>  Issue Type: Improvement
>  Components: atlas-webui
>Affects Versions: 2.0.0
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>Priority: Major
> Fix For: 2.1.0, 3.0.0
>
> Attachments: ATLAS-3870.patch, Screenshot 2020-06-30 at 2.33.11 
> PM.png, Screenshot 2020-06-30 at 2.34.06 PM.png
>
>
> * Go to hive_db entity detail page
>  * Click on Table tab
>  * toggle "Show historical entities" checkbox and check the network for the 
> API
> API always send the same value for
> {code:java}
> excludeDeletedEntities: true
> {code}
> !Screenshot 2020-06-30 at 2.33.11 PM.png|width=374,height=219!
> !Screenshot 2020-06-30 at 2.34.06 PM.png|width=397,height=142!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3870) UI: hive_table entity table not updating queryparams based on filter selected

2020-06-30 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3870:
--
Labels: ui  (was: )

> UI: hive_table entity table not updating queryparams based on filter selected
> -
>
> Key: ATLAS-3870
> URL: https://issues.apache.org/jira/browse/ATLAS-3870
> Project: Atlas
>  Issue Type: Improvement
>  Components: atlas-webui
>Affects Versions: 2.0.0
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>Priority: Major
>  Labels: ui
> Fix For: 2.1.0, 3.0.0
>
> Attachments: ATLAS-3870.patch, Screenshot 2020-06-30 at 2.33.11 
> PM.png, Screenshot 2020-06-30 at 2.34.06 PM.png
>
>
> * Go to hive_db entity detail page
>  * Click on Table tab
>  * toggle "Show historical entities" checkbox and check the network for the 
> API
> API always send the same value for
> {code:java}
> excludeDeletedEntities: true
> {code}
> !Screenshot 2020-06-30 at 2.33.11 PM.png|width=374,height=219!
> !Screenshot 2020-06-30 at 2.34.06 PM.png|width=397,height=142!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Resolved] (ATLAS-3836) Add Apache Ozone support in hive hook

2020-06-30 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian resolved ATLAS-3836.
---
Resolution: Fixed

> Add Apache Ozone support in hive hook
> -
>
> Key: ATLAS-3836
> URL: https://issues.apache.org/jira/browse/ATLAS-3836
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.0.0
>    Reporter: Sarath Subramanian
>Assignee: Nikhil Bonte
>Priority: Major
>  Labels: hive-hooks, ozone
> Fix For: 2.1.0
>
> Attachments: Hive_table_lineage.png, 
> Hive_table_lineage_load_in_path.png, Ozone_bucket.png, Ozone_key.png, 
> Ozone_volume.png
>
>
> Apache Ozone is the new object store for Hadoop - 
> [https://hadoop.apache.org/ozone/]
> Apache Atlas needs to add entity types to support creation of Ozone entities. 
> Hive hook should also be updated to create lineage between ozone entities and 
> hive tables (for EXTERNAL TABLE)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3836) Add Apache Ozone support in hive hook

2020-06-30 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3836:
--
Labels: hive-hooks ozone  (was: )

> Add Apache Ozone support in hive hook
> -
>
> Key: ATLAS-3836
> URL: https://issues.apache.org/jira/browse/ATLAS-3836
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.0.0
>    Reporter: Sarath Subramanian
>Assignee: Nikhil Bonte
>Priority: Major
>  Labels: hive-hooks, ozone
> Fix For: 2.1.0
>
> Attachments: Hive_table_lineage.png, 
> Hive_table_lineage_load_in_path.png, Ozone_bucket.png, Ozone_key.png, 
> Ozone_volume.png
>
>
> Apache Ozone is the new object store for Hadoop - 
> [https://hadoop.apache.org/ozone/]
> Apache Atlas needs to add entity types to support creation of Ozone entities. 
> Hive hook should also be updated to create lineage between ozone entities and 
> hive tables (for EXTERNAL TABLE)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 72630: ATLAS-3868: [Regression] removing a term-association doesn't remove classifications propagated from the term

2020-06-29 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72630/
---

(Updated June 29, 2020, 10:52 p.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
Nikhil Bonte, Nixon Rodrigues, Pinal Shah, and Sidharth Mishra.


Changes
---

moved logic to evaluate removeTagPropagation logic inside hard/soft delete 
handler.


Bugs: ATLAS-3868
https://issues.apache.org/jira/browse/ATLAS-3868


Repository: atlas


Description
---

Issue:
==
removing a term-association doesn't remove classifications propagated from the 
term

Cause:
==
Regression caused by https://issues.apache.org/jira/browse/ATLAS-3863

Solution:
=
Re-evaluate tag propagation only when relationship (edge) is force deleted or 
for internal types.


Diffs (updated)
-

  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/HardDeleteHandlerV1.java
 72dd632f4 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/SoftDeleteHandlerV1.java
 59e7cf864 


Diff: https://reviews.apache.org/r/72630/diff/2/

Changes: https://reviews.apache.org/r/72630/diff/1-2/


Testing
---

Manually validated - term dissassociation removes all propagated classifications

Precommit:
==
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1996/


Thanks,

Sarath Subramanian



[jira] [Updated] (ATLAS-3868) [Regression] removing a term-association doesn't remove classifications propagated from the term

2020-06-29 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3868:
--
Labels: glossary tagpropagation terms  (was: glossary tagpropagation)

> [Regression] removing a term-association doesn't remove classifications 
> propagated from the term
> 
>
> Key: ATLAS-3868
> URL: https://issues.apache.org/jira/browse/ATLAS-3868
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0
>    Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
>  Labels: glossary, tagpropagation, terms
> Fix For: 2.1.0
>
>
> # Create a business term – *term1*
>  # Associate classification *PII* to *term1*
>  # Associate *term1* to an entity
>  # Entity will now have PII classification propagated from the term
>  # Now remove *term1* association with the entity
>  # Entity still shows *PII* classification, propagated from the term => this 
> is a regression
>  
> This regression is caused by ATLAS-3863



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3868) [Regression] removing a term-association doesn't remove classifications propagated from the term

2020-06-29 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3868:
--
Labels: glossary tagpropagation  (was: )

> [Regression] removing a term-association doesn't remove classifications 
> propagated from the term
> 
>
> Key: ATLAS-3868
> URL: https://issues.apache.org/jira/browse/ATLAS-3868
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0
>    Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
>  Labels: glossary, tagpropagation
> Fix For: 2.1.0
>
>
> # Create a business term – *term1*
>  # Associate classification *PII* to *term1*
>  # Associate *term1* to an entity
>  # Entity will now have PII classification propagated from the term
>  # Now remove *term1* association with the entity
>  # Entity still shows *PII* classification, propagated from the term => this 
> is a regression
>  
> This regression is caused by ATLAS-3863



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Review Request 72630: ATLAS-3868: [Regression] removing a term-association doesn't remove classifications propagated from the term

2020-06-29 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72630/
---

Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
Nikhil Bonte, Nixon Rodrigues, Pinal Shah, and Sidharth Mishra.


Bugs: ATLAS-3868
https://issues.apache.org/jira/browse/ATLAS-3868


Repository: atlas


Description
---

Issue:
==
removing a term-association doesn't remove classifications propagated from the 
term

Cause:
==
Regression caused by https://issues.apache.org/jira/browse/ATLAS-3863

Solution:
=
Re-evaluate tag propagation only when relationship (edge) is force deleted or 
for internal types.


Diffs
-

  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
 717310daf 


Diff: https://reviews.apache.org/r/72630/diff/1/


Testing
---

Manually validated - term dissassociation removes all propagated classifications

Precommit:
==
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1996/


Thanks,

Sarath Subramanian



[jira] [Updated] (ATLAS-3868) [Regression] removing a term-association doesn't remove classifications propagated from the term

2020-06-29 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3868:
--
Description: 
# Create a business term – *term1*
 # Associate classification *PII* to *term1*
 # Associate *term1* to an entity
 # Entity will now have PII classification propagated from the term
 # Now remove *term1* association with the entity
 # Entity still shows *PII* classification, propagated from the term => this is 
a regression

 

This regression is caused by ATLAS-3863

  was:
# Create a business term – *term1*
 # Associate classification *PII* to *term1*
 # Associate *term1* to an entity
 # Entity will now have PII classification propagated from the term
 # Now remove *term1* association with the entity
 # Entity still shows *PII* classification, propagated from the term => this is 
a regression

 

This regression is caused by ATLAS-3863


> [Regression] removing a term-association doesn't remove classifications 
> propagated from the term
> 
>
> Key: ATLAS-3868
> URL: https://issues.apache.org/jira/browse/ATLAS-3868
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0
>    Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 2.1.0
>
>
> # Create a business term – *term1*
>  # Associate classification *PII* to *term1*
>  # Associate *term1* to an entity
>  # Entity will now have PII classification propagated from the term
>  # Now remove *term1* association with the entity
>  # Entity still shows *PII* classification, propagated from the term => this 
> is a regression
>  
> This regression is caused by ATLAS-3863



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (ATLAS-3868) [Regression] removing a term-association doesn't remove classifications propagated from the term

2020-06-29 Thread Sarath Subramanian (Jira)
Sarath Subramanian created ATLAS-3868:
-

 Summary: [Regression] removing a term-association doesn't remove 
classifications propagated from the term
 Key: ATLAS-3868
 URL: https://issues.apache.org/jira/browse/ATLAS-3868
 Project: Atlas
  Issue Type: Improvement
  Components:  atlas-core
Affects Versions: 2.0.0
Reporter: Sarath Subramanian
Assignee: Sarath Subramanian
 Fix For: 2.1.0


# Create a business term – *term1*
 # Associate classification *PII* to *term1*
 # Associate *term1* to an entity
 # Entity will now have PII classification propagated from the term
 # Now remove *term1* association with the entity
 # Entity still shows *PII* classification, propagated from the term => this is 
a regression

 

This regression is caused by ATLAS-3863



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3838) Support multiple tag/classification in basic/quick search API

2020-06-29 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3838:
--
Fix Version/s: (was: 2.1.0)

> Support multiple tag/classification in basic/quick search API
> -
>
> Key: ATLAS-3838
> URL: https://issues.apache.org/jira/browse/ATLAS-3838
> Project: Atlas
>  Issue Type: New Feature
>Affects Versions: 2.0.0
>Reporter: Pinal
>Assignee: Pinal
>Priority: Major
>  Labels: BasicSearch
> Fix For: 3.0.0
>
>
> it will allow user to search with multiple tags and the tag attribute filters 
> (attribute should System attributes or attribute which common for all the 
> mentioned tags)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 72625: ATLAS-3866 : Relationship search API for hive storage desc throws error code 500

2020-06-29 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72625/#review221085
---




repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
Lines 607 (patched)
<https://reviews.apache.org/r/72625/#comment309892>

Do we really need to compare edge end points?

can we do:

once you get relationshipEdge, you can get the typeName of the other vertex 
(not entityVertex - comparing vertexId). 

From this typeName - for e.g. 'hive_storagedesc' you can get entityType

relationshipEndEntityType = 
typeRegistry.getEntityTypeByName(otherEntityTypeName);


- Sarath Subramanian


On June 28, 2020, 10:06 p.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72625/
> ---
> 
> (Updated June 28, 2020, 10:06 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3866
> https://issues.apache.org/jira/browse/ATLAS-3866
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Issue:** In relationship api v2/relationship, for hive_storagedesc it 
> throws exception
> 
> **Reason:** it is because, sort attribute is assigned 'name' as default, and 
> 'name' attribute is not in the defination of hive_storagedesc.
> 
> **Workaround:** Validate if attribute is present in relationship end def, if 
> not ignore sorting
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  dd4d1b441 
> 
> 
> Diff: https://reviews.apache.org/r/72625/diff/1/
> 
> 
> Testing
> ---
> 
> Manually tested
> Precommit :
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>



[jira] [Updated] (ATLAS-3849) UI: Create icons for new entity types for Apache ozone

2020-06-26 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3849:
--
Labels: ui  (was: )

> UI: Create icons for new entity types for Apache ozone
> --
>
> Key: ATLAS-3849
> URL: https://issues.apache.org/jira/browse/ATLAS-3849
> Project: Atlas
>  Issue Type: New Feature
>  Components: atlas-webui
>Affects Versions: 2.0.0
>Reporter: Nikhil Bonte
>Assignee: Keval Bhatt
>Priority: Major
>  Labels: ui
> Fix For: 2.1.0, 3.0.0
>
> Attachments: ATLAS-3849.patch
>
>
> New typeDefs:
>  * ozone_volume
>  * ozone_bucket
>  * ozone_key



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3855) Bulk entity tag association and bulk api enhancement

2020-06-25 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3855:
--
Affects Version/s: (was: 3.0.0)
   2.0.0

> Bulk entity tag association and bulk api enhancement
> 
>
> Key: ATLAS-3855
> URL: https://issues.apache.org/jira/browse/ATLAS-3855
> Project: Atlas
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: chaitali borole
>Assignee: chaitali borole
>Priority: Major
>
> API : /api/atlas/v2/entity/bulk/classification
> Body :
> {code:java}
> {"classification":{"typeName":"PII","attributes":{},"propagate":true,"removePropagationsOnEntityDelete":false,"validityPeriods":[]},"entityGuids":["guid1","guid2"]}
>  {code}
> when user doesn't have authorisation on either of guid1 or guid2 , the bulk 
> entity call fails with 403. 
> bulk api _v2/entity/bulk_ to retrieve a list of entities identified by its 
> guids.
> This bulk api fails with 403 if some guids belong to entities on which user 
> is *unauthorized* and other guids belong to entities on which user is 
> *authorized*.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3855) Bulk entity tag association and bulk api enhancement

2020-06-25 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3855:
--
Component/s:  atlas-core

> Bulk entity tag association and bulk api enhancement
> 
>
> Key: ATLAS-3855
> URL: https://issues.apache.org/jira/browse/ATLAS-3855
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: chaitali borole
>Assignee: chaitali borole
>Priority: Major
> Fix For: 2.1.0
>
>
> API : /api/atlas/v2/entity/bulk/classification
> Body :
> {code:java}
> {"classification":{"typeName":"PII","attributes":{},"propagate":true,"removePropagationsOnEntityDelete":false,"validityPeriods":[]},"entityGuids":["guid1","guid2"]}
>  {code}
> when user doesn't have authorisation on either of guid1 or guid2 , the bulk 
> entity call fails with 403. 
> bulk api _v2/entity/bulk_ to retrieve a list of entities identified by its 
> guids.
> This bulk api fails with 403 if some guids belong to entities on which user 
> is *unauthorized* and other guids belong to entities on which user is 
> *authorized*.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3855) Bulk entity tag association and bulk api enhancement

2020-06-25 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3855:
--
Fix Version/s: 2.1.0

> Bulk entity tag association and bulk api enhancement
> 
>
> Key: ATLAS-3855
> URL: https://issues.apache.org/jira/browse/ATLAS-3855
> Project: Atlas
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: chaitali borole
>Assignee: chaitali borole
>Priority: Major
> Fix For: 2.1.0
>
>
> API : /api/atlas/v2/entity/bulk/classification
> Body :
> {code:java}
> {"classification":{"typeName":"PII","attributes":{},"propagate":true,"removePropagationsOnEntityDelete":false,"validityPeriods":[]},"entityGuids":["guid1","guid2"]}
>  {code}
> when user doesn't have authorisation on either of guid1 or guid2 , the bulk 
> entity call fails with 403. 
> bulk api _v2/entity/bulk_ to retrieve a list of entities identified by its 
> guids.
> This bulk api fails with 403 if some guids belong to entities on which user 
> is *unauthorized* and other guids belong to entities on which user is 
> *authorized*.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3855) Bulk entity tag association and bulk api enhancement

2020-06-25 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3855:
--
Labels: api-change  (was: )

> Bulk entity tag association and bulk api enhancement
> 
>
> Key: ATLAS-3855
> URL: https://issues.apache.org/jira/browse/ATLAS-3855
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: chaitali borole
>Assignee: chaitali borole
>Priority: Major
>  Labels: api-change
> Fix For: 2.1.0
>
>
> API : /api/atlas/v2/entity/bulk/classification
> Body :
> {code:java}
> {"classification":{"typeName":"PII","attributes":{},"propagate":true,"removePropagationsOnEntityDelete":false,"validityPeriods":[]},"entityGuids":["guid1","guid2"]}
>  {code}
> when user doesn't have authorisation on either of guid1 or guid2 , the bulk 
> entity call fails with 403. 
> bulk api _v2/entity/bulk_ to retrieve a list of entities identified by its 
> guids.
> This bulk api fails with 403 if some guids belong to entities on which user 
> is *unauthorized* and other guids belong to entities on which user is 
> *authorized*.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Resolved] (ATLAS-3856) Create new entity typeDefs for Apache Ozone

2020-06-25 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian resolved ATLAS-3856.
---
Resolution: Fixed

> Create new entity typeDefs for Apache Ozone
> ---
>
> Key: ATLAS-3856
> URL: https://issues.apache.org/jira/browse/ATLAS-3856
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: 
> ATLAS-3856-Create-new-entity-typeDefs-for-Apache-Ozone-v0.patch
>
>
> Add following ozone typeDefs to Atlas models
>  * Ozone volume
>  * Ozone bucket
>  * Ozone key



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3856) Create new entity typeDefs for Apache Ozone

2020-06-25 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3856:
--
Fix Version/s: 2.1.0

> Create new entity typeDefs for Apache Ozone
> ---
>
> Key: ATLAS-3856
> URL: https://issues.apache.org/jira/browse/ATLAS-3856
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Nikhil Bonte
>Assignee: Nikhil Bonte
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: 
> ATLAS-3856-Create-new-entity-typeDefs-for-Apache-Ozone-v0.patch
>
>
> Add following ozone typeDefs to Atlas models
>  * Ozone volume
>  * Ozone bucket
>  * Ozone key



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 72614: ATLAS-3856 Create new entity types for Apache Ozone

2020-06-25 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72614/#review221069
---


Ship it!




Ship It!

- Sarath Subramanian


On June 24, 2020, 8:07 a.m., Nikhil Bonte wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72614/
> ---
> 
> (Updated June 24, 2020, 8:07 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
> Rodrigues, Sarath Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3856
> https://issues.apache.org/jira/browse/ATLAS-3856
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Add following ozone typeDefs to Atlas models
> 
> entityDefs:
> ozone_volume
> ozone_bucket
> ozone_key
> 
> enumDefs:
> ozone_storage_type
> ozone_replication_type
> 
> structDefs:
> ozone_key_location
> 
> relationshipDefs:
> ozone_volume_buckets
> ozone_bucket_keys
> 
> 
> Diffs
> -
> 
>   addons/models/3000-Cloud/3050-ozone-typedefs.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/72614/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Nikhil Bonte
> 
>



Re: Review Request 72619: ATLAS-3863: Avoid re-evaluating tag propagation when deleting relationship (edge)

2020-06-24 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72619/
---

(Updated June 24, 2020, 5:33 p.m.)


Review request for atlas, Ashutosh Mestry, chaitali, Jayendra Parab, Madhan 
Neethiraj, Mandar Ambawane, mayank jain, Nikhil Bonte, Nixon Rodrigues, Pinal 
Shah, and Sidharth Mishra.


Bugs: ATLAS-3863
https://issues.apache.org/jira/browse/ATLAS-3863


Repository: atlas


Description
---

Tag propagation is re-evaluated every time when a relationship (edge) is 
deleted.

Removal of propagated classifications is controlled at the assignment time to 
an entity:


Remove propagation on entity delete - TRUE : 
Propagated classifications (to all downstream entities) will be removed when 
the source entity where the classification is originally assigned is deleted.
 

Remove propagation on entity delete - FALSE : 
Propagated classifications (to all downstream entities) will not be removed 
when the  source entity where the classification is originally assigned 
is deleted.
 

There is no need to revaluate tag propagation during relationship (edge) 
deletion since removal of tag propagation depends on the above config.


Diffs (updated)
-

  repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
7b7ec6567 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
 3f8503ac3 


Diff: https://reviews.apache.org/r/72619/diff/2/

Changes: https://reviews.apache.org/r/72619/diff/1-2/


Testing
---

Manually validated:

Deleted relationship, Deleted downstream entities and verified that propagated 
tags are not affected and revaluation of tag propagation is not computed.

Precommit - 
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1978/console


`Significant performance improvement observed:`

Deleting a hive_table entity (with 44 columns and 16 propagated classifications)

Before patch : 279 secs (4 mins, 39 secs)
After patch  : 46.92 secs

83% performance improvement seen.


Thanks,

Sarath Subramanian



[jira] [Updated] (ATLAS-3863) Avoid re-evaluating tag propagation when deleting relationship (edge)

2020-06-24 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3863:
--
Description: 
Tag propagation is re-evaluated every time when a relationship (edge) is 
deleted.

Removal of propagated classifications is controlled at the assignment time to 
an entity:

!image-2020-06-24-12-51-38-941.png|width=716,height=272!

 
 * {color:#172b4d}Remove propagation on entity delete - *TRUE :* {color}
 ** _Propagated classifications (to all downstream entities) will be removed 
when the source entity where the classification is originally assigned is 
deleted._

 
 * Remove propagation on entity delete - *FALSE :* 
 ** _Propagated classifications (to all downstream entities) will *not* be 
removed when the          source entity where the classification is originally 
assigned is deleted._

 

There is no need to revaluate tag propagation during relationship (edge) 
deletion since removal of tag propagation depends on the above config

 

*Significant performance improvement observed: 83% performance improvement 
seen.*

Deleting a hive_table entity (with 44 columns and 16 propagated classifications)

{{Before patch : 279 secs (4 mins, 39 secs) }}

{{After patch  : 46.92 secs}}

 
 

  was:
Tag propagation is re-evaluated every time when a relationship (edge) is 
deleted.

Removal of propagated classifications is controlled at the assignment time to 
an entity:

!image-2020-06-24-12-51-38-941.png|width=716,height=272!

 
 * {color:#172b4d}Remove propagation on entity delete - *TRUE :* {color}
 ** _Propagated classifications (to all downstream entities) will be removed 
when the source entity where the classification is originally assigned is 
deleted._

 
 * Remove propagation on entity delete - *FALSE :* 
 ** _Propagated classifications (to all downstream entities) will *not* be 
removed when the          source entity where the classification is originally 
assigned is deleted._

 

There is no need to revaluate tag propagation during relationship (edge) 
deletion since removal of tag propagation depends on the above config.


> Avoid re-evaluating tag propagation when deleting relationship (edge)
> -
>
> Key: ATLAS-3863
> URL: https://issues.apache.org/jira/browse/ATLAS-3863
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0
>    Reporter: Sarath Subramanian
>    Assignee: Sarath Subramanian
>Priority: Major
>  Labels: classification, tagpropagation
> Fix For: 2.1.0
>
> Attachments: ATLAS-3863.001.patch, image-2020-06-24-12-51-38-941.png
>
>
> Tag propagation is re-evaluated every time when a relationship (edge) is 
> deleted.
> Removal of propagated classifications is controlled at the assignment time to 
> an entity:
> !image-2020-06-24-12-51-38-941.png|width=716,height=272!
>  
>  * {color:#172b4d}Remove propagation on entity delete - *TRUE :* {color}
>  ** _Propagated classifications (to all downstream entities) will be removed 
> when the source entity where the classification is originally assigned is 
> deleted._
>  
>  * Remove propagation on entity delete - *FALSE :* 
>  ** _Propagated classifications (to all downstream entities) will *not* be 
> removed when the          source entity where the classification is 
> originally assigned is deleted._
>  
> There is no need to revaluate tag propagation during relationship (edge) 
> deletion since removal of tag propagation depends on the above config
>  
> *Significant performance improvement observed: 83% performance improvement 
> seen.*
> Deleting a hive_table entity (with 44 columns and 16 propagated 
> classifications)
> {{Before patch : 279 secs (4 mins, 39 secs) }}
> {{After patch  : 46.92 secs}}
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Review Request 72619: ATLAS-3863: Avoid re-evaluating tag propagation when deleting relationship (edge)

2020-06-24 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72619/
---

Review request for atlas, Ashutosh Mestry, chaitali, Jayendra Parab, Madhan 
Neethiraj, Mandar Ambawane, mayank jain, Nikhil Bonte, Nixon Rodrigues, Pinal 
Shah, and Sidharth Mishra.


Bugs: ATLAS-3863
https://issues.apache.org/jira/browse/ATLAS-3863


Repository: atlas


Description
---

Tag propagation is re-evaluated every time when a relationship (edge) is 
deleted.

Removal of propagated classifications is controlled at the assignment time to 
an entity:


Remove propagation on entity delete - TRUE : 
Propagated classifications (to all downstream entities) will be removed when 
the source entity where the classification is originally assigned is deleted.
 

Remove propagation on entity delete - FALSE : 
Propagated classifications (to all downstream entities) will not be removed 
when the  source entity where the classification is originally assigned 
is deleted.
 

There is no need to revaluate tag propagation during relationship (edge) 
deletion since removal of tag propagation depends on the above config.


Diffs
-

  repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
7b7ec6567 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
 3f8503ac3 


Diff: https://reviews.apache.org/r/72619/diff/1/


Testing
---

Manually validated:

Deleted relationship, Deleted downstream entities and verified that propagated 
tags are not affected and revaluation of tag propagation is not computed.

Precommit - 
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1978/console


Thanks,

Sarath Subramanian



[jira] [Updated] (ATLAS-3863) Avoid re-evaluating tag propagation when deleting relationship (edge)

2020-06-24 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3863:
--
Attachment: ATLAS-3863.001.patch

> Avoid re-evaluating tag propagation when deleting relationship (edge)
> -
>
> Key: ATLAS-3863
> URL: https://issues.apache.org/jira/browse/ATLAS-3863
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0
>    Reporter: Sarath Subramanian
>    Assignee: Sarath Subramanian
>Priority: Major
>  Labels: classification, tagpropagation
> Fix For: 2.1.0
>
> Attachments: ATLAS-3863.001.patch, image-2020-06-24-12-51-38-941.png
>
>
> Tag propagation is re-evaluated every time when a relationship (edge) is 
> deleted.
> Removal of propagated classifications is controlled at the assignment time to 
> an entity:
> !image-2020-06-24-12-51-38-941.png|width=716,height=272!
>  
>  * {color:#172b4d}Remove propagation on entity delete - *TRUE :* {color}
>  ** _Propagated classifications (to all downstream entities) will be removed 
> when the source entity where the classification is originally assigned is 
> deleted._
>  
>  * Remove propagation on entity delete - *FALSE :* 
>  ** _Propagated classifications (to all downstream entities) will *not* be 
> removed when the          source entity where the classification is 
> originally assigned is deleted._
>  
> There is no need to revaluate tag propagation during relationship (edge) 
> deletion since removal of tag propagation depends on the above config.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3863) Avoid re-evaluating tag propagation when deleting relationship (edge)

2020-06-24 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3863:
--
Labels: classification tagpropagation  (was: )

> Avoid re-evaluating tag propagation when deleting relationship (edge)
> -
>
> Key: ATLAS-3863
> URL: https://issues.apache.org/jira/browse/ATLAS-3863
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0
>    Reporter: Sarath Subramanian
>    Assignee: Sarath Subramanian
>Priority: Major
>  Labels: classification, tagpropagation
> Fix For: 2.1.0
>
> Attachments: image-2020-06-24-12-51-38-941.png
>
>
> Tag propagation is re-evaluated every time when a relationship (edge) is 
> deleted.
> Removal of propagated classifications is controlled at the assignment time to 
> an entity:
> !image-2020-06-24-12-51-38-941.png|width=716,height=272!
>  
>  * {color:#172b4d}Remove propagation on entity delete - *TRUE :* {color}
>  ** _Propagated classifications (to all downstream entities) will be removed 
> when the source entity where the classification is originally assigned is 
> deleted._
>  
>  * Remove propagation on entity delete - *FALSE :* 
>  ** _Propagated classifications (to all downstream entities) will *not* be 
> removed when the          source entity where the classification is 
> originally assigned is deleted._
>  
> There is no need to revaluate tag propagation during relationship (edge) 
> deletion since removal of tag propagation depends on the above config.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3863) Avoid re-evaluating tag propagation when deleting relationship (edge)

2020-06-24 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3863:
--
Description: 
Tag propagation is re-evaluated every time when a relationship (edge) is 
deleted.

Removal of propagated classifications is controlled at the assignment time to 
an entity:

!image-2020-06-24-12-51-38-941.png|width=716,height=272!

 
 * {color:#172b4d}Remove propagation on entity delete - *TRUE :* {color}
 ** _Propagated classifications (to all downstream entities) will be removed 
when the source entity where the classification is originally assigned is 
deleted._

 
 * Remove propagation on entity delete - *FALSE :* 
 ** _Propagated classifications (to all downstream entities) will *not* be 
removed when the          source entity where the classification is originally 
assigned is deleted._

 

There is no need to revaluate tag propagation during relationship (edge) 
deletion since removal of tag propagation depends on the above config.

  was:
Tag propagation is re-evaluated every time when a relationship (edge) is 
deleted.

 

Removal of propagated classifications is controlled at the assignment time to 
an entity:

!image-2020-06-24-12-51-38-941.png|width=716,height=272!

 
 * Remove propagation on entity delete - *TRUE :* 
_Propagated classifications (to all downstream entities) will be removed when 
the source entity where the classification is originally assigned is deleted._

 
 * Remove propagation on entity delete - *FALSE :* 

           _Propagated classifications (to all downstream entities) will *not* 
be removed when the          source entity where the classification is 
originally assigned is deleted._

 

There is no need to revaluate tag propagation during relationship deletion 
since removal of tag propagation depends on the above config.


> Avoid re-evaluating tag propagation when deleting relationship (edge)
> -
>
> Key: ATLAS-3863
> URL: https://issues.apache.org/jira/browse/ATLAS-3863
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0
>    Reporter: Sarath Subramanian
>    Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: image-2020-06-24-12-51-38-941.png
>
>
> Tag propagation is re-evaluated every time when a relationship (edge) is 
> deleted.
> Removal of propagated classifications is controlled at the assignment time to 
> an entity:
> !image-2020-06-24-12-51-38-941.png|width=716,height=272!
>  
>  * {color:#172b4d}Remove propagation on entity delete - *TRUE :* {color}
>  ** _Propagated classifications (to all downstream entities) will be removed 
> when the source entity where the classification is originally assigned is 
> deleted._
>  
>  * Remove propagation on entity delete - *FALSE :* 
>  ** _Propagated classifications (to all downstream entities) will *not* be 
> removed when the          source entity where the classification is 
> originally assigned is deleted._
>  
> There is no need to revaluate tag propagation during relationship (edge) 
> deletion since removal of tag propagation depends on the above config.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (ATLAS-3863) Avoid re-evaluating tag propagation when deleting relationship (edge)

2020-06-24 Thread Sarath Subramanian (Jira)
Sarath Subramanian created ATLAS-3863:
-

 Summary: Avoid re-evaluating tag propagation when deleting 
relationship (edge)
 Key: ATLAS-3863
 URL: https://issues.apache.org/jira/browse/ATLAS-3863
 Project: Atlas
  Issue Type: Improvement
  Components:  atlas-core
Affects Versions: 2.0.0
Reporter: Sarath Subramanian
Assignee: Sarath Subramanian
 Fix For: 2.1.0
 Attachments: image-2020-06-24-12-51-38-941.png

Tag propagation is re-evaluated every time when a relationship (edge) is 
deleted.

 

Removal of propagated classifications is controlled at the assignment time to 
an entity:

!image-2020-06-24-12-51-38-941.png|width=716,height=272!

 
 * Remove propagation on entity delete - *TRUE :* 
_Propagated classifications (to all downstream entities) will be removed when 
the source entity where the classification is originally assigned is deleted._

 
 * Remove propagation on entity delete - *FALSE :* 

           _Propagated classifications (to all downstream entities) will *not* 
be removed when the          source entity where the classification is 
originally assigned is deleted._

 

There is no need to revaluate tag propagation during relationship deletion 
since removal of tag propagation depends on the above config.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3652) Quick Search: API requirement for GET request on multiple entity types

2020-06-24 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3652:
--
Labels: quicksearch  (was: )

> Quick Search: API requirement for GET request on multiple entity types
> --
>
> Key: ATLAS-3652
> URL: https://issues.apache.org/jira/browse/ATLAS-3652
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Pinal
>Assignee: Pinal
>Priority: Major
>  Labels: quicksearch
> Fix For: 2.1.0
>
>
> Need an API that can be used to GET information of multiple selected entity 
> types. Currently it seems to support GET request for
>  # One type (typeName=someType)\{by specifying typeName in the call},
> OR 
>  # All types (typeName= )\{by leaving out typeName empty}.
>  
> Need an API that works for multiple selected types while also returning the 
> aggregationMetrics in the responseJSON as opposed to the POST call(where the 
> aggregationMetrics is left out as empty).
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3652) Quick Search: API requirement for GET request on multiple entity types

2020-06-24 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3652:
--
Fix Version/s: 2.1.0

> Quick Search: API requirement for GET request on multiple entity types
> --
>
> Key: ATLAS-3652
> URL: https://issues.apache.org/jira/browse/ATLAS-3652
> Project: Atlas
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Pinal
>Assignee: Pinal
>Priority: Major
> Fix For: 2.1.0
>
>
> Need an API that can be used to GET information of multiple selected entity 
> types. Currently it seems to support GET request for
>  # One type (typeName=someType)\{by specifying typeName in the call},
> OR 
>  # All types (typeName= )\{by leaving out typeName empty}.
>  
> Need an API that works for multiple selected types while also returning the 
> aggregationMetrics in the responseJSON as opposed to the POST call(where the 
> aggregationMetrics is left out as empty).
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3652) Quick Search: API requirement for GET request on multiple entity types

2020-06-24 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3652:
--
Affects Version/s: 2.0.0

> Quick Search: API requirement for GET request on multiple entity types
> --
>
> Key: ATLAS-3652
> URL: https://issues.apache.org/jira/browse/ATLAS-3652
> Project: Atlas
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Pinal
>Assignee: Pinal
>Priority: Major
>
> Need an API that can be used to GET information of multiple selected entity 
> types. Currently it seems to support GET request for
>  # One type (typeName=someType)\{by specifying typeName in the call},
> OR 
>  # All types (typeName= )\{by leaving out typeName empty}.
>  
> Need an API that works for multiple selected types while also returning the 
> aggregationMetrics in the responseJSON as opposed to the POST call(where the 
> aggregationMetrics is left out as empty).
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3652) Quick Search: API requirement for GET request on multiple entity types

2020-06-24 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3652:
--
Component/s:  atlas-core

> Quick Search: API requirement for GET request on multiple entity types
> --
>
> Key: ATLAS-3652
> URL: https://issues.apache.org/jira/browse/ATLAS-3652
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Pinal
>Assignee: Pinal
>Priority: Major
> Fix For: 2.1.0
>
>
> Need an API that can be used to GET information of multiple selected entity 
> types. Currently it seems to support GET request for
>  # One type (typeName=someType)\{by specifying typeName in the call},
> OR 
>  # All types (typeName= )\{by leaving out typeName empty}.
>  
> Need an API that works for multiple selected types while also returning the 
> aggregationMetrics in the responseJSON as opposed to the POST call(where the 
> aggregationMetrics is left out as empty).
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 72614: ATLAS-3856 Create new entity types for Apache Ozone

2020-06-23 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72614/#review221057
---




addons/models/3000-Cloud/3050-ozone-typedefs.json
Lines 6 (patched)
<https://reviews.apache.org/r/72614/#comment309852>

"Atlas Type representing the locations for key" => "location information 
about Ozone key"



addons/models/3000-Cloud/3050-ozone-typedefs.json
Lines 15 (patched)
<https://reviews.apache.org/r/72614/#comment309853>

"isOptional" is set to false for all attributes of "keyLocations". Can 
there be cases where you don't have all the information and only some.



addons/models/3000-Cloud/3050-ozone-typedefs.json
Lines 22 (patched)
<https://reviews.apache.org/r/72614/#comment309850>

consider setting "isIndexable" to true for all string, int and date type 
attributes. This will enable searching on this attributes using basic search.



addons/models/3000-Cloud/3050-ozone-typedefs.json
Lines 28 (patched)
<https://reviews.apache.org/r/72614/#comment309851>

does "length" type needs to be of string type?



addons/models/3000-Cloud/3050-ozone-typedefs.json
Lines 49 (patched)
<https://reviews.apache.org/r/72614/#comment309854>

ozone => Ozone


- Sarath Subramanian


On June 22, 2020, 10:45 p.m., Nikhil Bonte wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72614/
> ---
> 
> (Updated June 22, 2020, 10:45 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
> Rodrigues, Sarath Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3856
> https://issues.apache.org/jira/browse/ATLAS-3856
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Add following ozone typeDefs to Atlas models
> 
> Ozone volume
> Ozone bucket
> Ozone key
> 
> 
> Diffs
> -
> 
>   addons/models/3000-Cloud/3050-ozone-typedefs.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/72614/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Nikhil Bonte
> 
>



[jira] [Updated] (ATLAS-3845) Audit API returns the audit information for an unauthorised entity

2020-06-22 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3845:
--
Labels: audit  (was: )

> Audit API  returns the audit information for an unauthorised entity
> ---
>
> Key: ATLAS-3845
> URL: https://issues.apache.org/jira/browse/ATLAS-3845
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: chaitali borole
>Assignee: chaitali borole
>Priority: Major
>  Labels: audit
> Fix For: 2.1.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3845) Audit API returns the audit information for an unauthorised entity

2020-06-22 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3845:
--
Affects Version/s: (was: 3.0.0)
   2.0.0

> Audit API  returns the audit information for an unauthorised entity
> ---
>
> Key: ATLAS-3845
> URL: https://issues.apache.org/jira/browse/ATLAS-3845
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: chaitali borole
>Assignee: chaitali borole
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3845) Audit API returns the audit information for an unauthorised entity

2020-06-22 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3845:
--
Component/s:  atlas-core

> Audit API  returns the audit information for an unauthorised entity
> ---
>
> Key: ATLAS-3845
> URL: https://issues.apache.org/jira/browse/ATLAS-3845
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: chaitali borole
>Assignee: chaitali borole
>Priority: Major
> Fix For: 2.1.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3845) Audit API returns the audit information for an unauthorised entity

2020-06-22 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3845:
--
Fix Version/s: 2.1.0

> Audit API  returns the audit information for an unauthorised entity
> ---
>
> Key: ATLAS-3845
> URL: https://issues.apache.org/jira/browse/ATLAS-3845
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: chaitali borole
>Assignee: chaitali borole
>Priority: Major
> Fix For: 2.1.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Resolved] (ATLAS-3854) Upgrade Spring Security version to 4.2.16

2020-06-22 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian resolved ATLAS-3854.
---
Resolution: Fixed

> Upgrade Spring Security version to 4.2.16
> -
>
> Key: ATLAS-3854
> URL: https://issues.apache.org/jira/browse/ATLAS-3854
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Mandar Ambawane
>Assignee: Mandar Ambawane
>Priority: Major
> Fix For: 2.1.0
>
>
> Spring Security versions 5.3.x prior to 5.3.2, 5.2.x prior to 5.2.4, 5.1.x 
> prior to 5.1.10, 5.0.x prior to 5.0.16 and 4.2.x prior to 4.2.16 use a fixed 
> null initialization vector with CBC Mode in the implementation of the 
> queryable text encryptor. A malicious user with access to the data that has 
> been encrypted using such an encryptor may be able to derive the unencrypted 
> values using a dictionary attack.
>  To resolve this need to upgrade Spring security to 4.2.16



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3854) Upgrade Spring Security version to 4.2.16

2020-06-22 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3854:
--
Fix Version/s: 2.1.0

> Upgrade Spring Security version to 4.2.16
> -
>
> Key: ATLAS-3854
> URL: https://issues.apache.org/jira/browse/ATLAS-3854
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Mandar Ambawane
>Assignee: Mandar Ambawane
>Priority: Major
> Fix For: 2.1.0
>
>
> Spring Security versions 5.3.x prior to 5.3.2, 5.2.x prior to 5.2.4, 5.1.x 
> prior to 5.1.10, 5.0.x prior to 5.0.16 and 4.2.x prior to 4.2.16 use a fixed 
> null initialization vector with CBC Mode in the implementation of the 
> queryable text encryptor. A malicious user with access to the data that has 
> been encrypted using such an encryptor may be able to derive the unencrypted 
> values using a dictionary attack.
>  To resolve this need to upgrade Spring security to 4.2.16



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3854) Upgrade Spring Security version to 4.2.16

2020-06-22 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3854:
--
Component/s:  atlas-core

> Upgrade Spring Security version to 4.2.16
> -
>
> Key: ATLAS-3854
> URL: https://issues.apache.org/jira/browse/ATLAS-3854
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Mandar Ambawane
>Assignee: Mandar Ambawane
>Priority: Major
> Fix For: 2.1.0
>
>
> Spring Security versions 5.3.x prior to 5.3.2, 5.2.x prior to 5.2.4, 5.1.x 
> prior to 5.1.10, 5.0.x prior to 5.0.16 and 4.2.x prior to 4.2.16 use a fixed 
> null initialization vector with CBC Mode in the implementation of the 
> queryable text encryptor. A malicious user with access to the data that has 
> been encrypted using such an encryptor may be able to derive the unencrypted 
> values using a dictionary attack.
>  To resolve this need to upgrade Spring security to 4.2.16



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (ATLAS-3854) Upgrade Spring Security version to 4.2.16

2020-06-22 Thread Sarath Subramanian (Jira)


 [ 
https://issues.apache.org/jira/browse/ATLAS-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3854:
--
Affects Version/s: 2.0.0

> Upgrade Spring Security version to 4.2.16
> -
>
> Key: ATLAS-3854
> URL: https://issues.apache.org/jira/browse/ATLAS-3854
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Mandar Ambawane
>Assignee: Mandar Ambawane
>Priority: Major
>
> Spring Security versions 5.3.x prior to 5.3.2, 5.2.x prior to 5.2.4, 5.1.x 
> prior to 5.1.10, 5.0.x prior to 5.0.16 and 4.2.x prior to 4.2.16 use a fixed 
> null initialization vector with CBC Mode in the implementation of the 
> queryable text encryptor. A malicious user with access to the data that has 
> been encrypted using such an encryptor may be able to derive the unencrypted 
> values using a dictionary attack.
>  To resolve this need to upgrade Spring security to 4.2.16



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


Re: Review Request 72601: ATLAS-3845 : Audit API returns the audit information for an unauthorised entity

2020-06-22 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72601/#review221045
---


Ship it!




Ship It!

- Sarath Subramanian


On June 22, 2020, 7:49 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72601/
> ---
> 
> (Updated June 22, 2020, 7:49 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3845
> https://issues.apache.org/jira/browse/ATLAS-3845
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Audit api was returning all entities details even after blocking the rights 
> for read entity in Ranger policy-This patch handles the issue by adding 
> verify access check.
> 
> 
> Diffs
> -
> 
>   webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java b1055605f 
> 
> 
> Diff: https://reviews.apache.org/r/72601/diff/2/
> 
> 
> Testing
> ---
> 
> Tested by adding deny policy for read/create entity in Ranger
> 
> 
> Thanks,
> 
> chaitali
> 
>



Re: Review Request 72612: ATLAS-3854: Upgrade Spring Security version to 4.2.16

2020-06-22 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72612/#review221044
---


Ship it!




Ship It!

- Sarath Subramanian


On June 22, 2020, 7:53 a.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72612/
> ---
> 
> (Updated June 22, 2020, 7:53 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3854
> https://issues.apache.org/jira/browse/ATLAS-3854
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Spring Security versions 5.3.x prior to 5.3.2, 5.2.x prior to 5.2.4, 5.1.x 
> prior to 5.1.10, 5.0.x prior to 5.0.16 and 4.2.x prior to 4.2.16 use a fixed 
> null initialization vector with CBC Mode in the implementation of the 
> queryable text encryptor. A malicious user with access to the data that has 
> been encrypted using such an encryptor may be able to derive the unencrypted 
> values using a dictionary attack.
> 
> To resolve this need to upgrade Spring security to 4.2.16
> 
> 
> Diffs
> -
> 
>   pom.xml 8d02a6f 
> 
> 
> Diff: https://reviews.apache.org/r/72612/diff/1/
> 
> 
> Testing
> ---
> 
> PreCommit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1970/console
> 
> 
> Thanks,
> 
> Mandar Ambawane
> 
>



Re: Review Request 72597: ATLAS-3844 :- Ignore relationship attribute while fetching entities in Import Hive bridge.

2020-06-18 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72597/#review221032
---


Ship it!




Ship It!

- Sarath Subramanian


On June 17, 2020, 5:03 a.m., Nixon Rodrigues wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72597/
> ---
> 
> (Updated June 17, 2020, 5:03 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3844
> https://issues.apache.org/jira/browse/ATLAS-3844
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently in Import Hive bridge, the hive_db and hive_table entitites are 
> fetched along relationshipAttributes and related Attributes and this 
> attributes are cleared thereafter.
> Instead fetching relationshipAttributes and related Attributes in payload, it 
> is will good to ignoreRelationship to reduce the size of payload.
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  eb691dc18 
> 
> 
> Diff: https://reviews.apache.org/r/72597/diff/3/
> 
> 
> Testing
> ---
> 
> Created database / table by disabling the atlas-hive hook and executed 
> import-hive.sh due import database and its tables.
> 
> Tested following cases.
> 
> 1. DB exists in atlas with some tables
> 2. DB exists in atlas with no tables
> 3. No DB exists in atlas.
> 
> 
> Thanks,
> 
> Nixon Rodrigues
> 
>



Re: Review Request 72601: ATLAS-3845 : Audit API returns the audit information for an unauthorised entity

2020-06-18 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72601/#review221031
---




webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java
Line 808 (original), 811 (patched)
<https://reviews.apache.org/r/72601/#comment309813>

consider replacing 811 and 812 with:

entitiesStore.getHeaderById(guid);

this internally checks for authorization.


- Sarath Subramanian


On June 17, 2020, 7:48 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72601/
> ---
> 
> (Updated June 17, 2020, 7:48 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3845
> https://issues.apache.org/jira/browse/ATLAS-3845
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Audit api was returning all entities details even after blocking the rights 
> for read entity in Ranger policy-This patch handles the issue by adding 
> verify access check.
> 
> 
> Diffs
> -
> 
>   webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java b1055605f 
> 
> 
> Diff: https://reviews.apache.org/r/72601/diff/1/
> 
> 
> Testing
> ---
> 
> Tested by adding deny policy for read/create entity in Ranger
> 
> 
> Thanks,
> 
> chaitali
> 
>



Re: Review Request 72566: ATLAS-1798 : Fix Findbugs problems in repository module

2020-06-16 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72566/#review221018
---




repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java
Line 265 (original)
<https://reviews.apache.org/r/72566/#comment309794>

why is this logic removed? please revert if not intended.

how will sortBy and sortOrder be normalized?



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasStructDefStoreV2.java
Line 629 (original)
<https://reviews.apache.org/r/72566/#comment309791>

why reverse this logic?

if isUnique is null or false - it should be LIST.

Revert if not intended.



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
Line 2249 (original), 2235 (patched)
<https://reviews.apache.org/r/72566/#comment309792>

these checks are confusing, revert to old one.



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
Line 2273 (original), 2260 (patched)
<https://reviews.apache.org/r/72566/#comment309793>

these checks are confusing, revert to old one.



repository/src/main/java/org/apache/atlas/util/FileUtils.java
Lines 36 (patched)
<https://reviews.apache.org/r/72566/#comment309795>

nit: unused import. consider removing it.


- Sarath Subramanian


On June 10, 2020, 7:56 p.m., mayank jain wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72566/
> ---
> 
> (Updated June 10, 2020, 7:56 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-1798
> https://issues.apache.org/jira/browse/ATLAS-1798
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently Findbugs complaints about some problems (see attachment) in the 
> repository module. They should be fixed to get the code more reliable.
> 
> 
> Diffs
> -
> 
>   build-tools/src/main/resources/findbugs-exclude.xml da6c58d 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
> 57e454a 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  dd4d1b4 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java
>  56956e6 
>   
> repository/src/main/java/org/apache/atlas/discovery/SearchAggregatorImpl.java 
> e8f7dbc 
>   repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
> 804c694 
>   repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java 
> d630f66 
>   repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java 
> 2c84ec7 
>   repository/src/main/java/org/apache/atlas/glossary/GlossaryUtils.java 
> 2a2cebb 
>   repository/src/main/java/org/apache/atlas/query/AtlasDSL.java b8a744b 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 801e898 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java 
> 142b9ca 
>   repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java 
> 6b33edb 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java
>  1aac375 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java
>  69d373d 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
>  79527ac 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
>  9fca744 
>   
> repository/src/main/java/org/apache/atlas/repository/converters/AtlasArrayFormatConverter.java
>  c335f0a 
>   
> repository/src/main/java/org/apache/atlas/repository/converters/AtlasInstanceConverter.java
>  6fc0c65 
>   
> repository/src/main/java/org/apache/atlas/repository/converters/AtlasMapFormatConverter.java
>  0eacd8e 
>   
> repository/src/main/java/org/apache/atlas/repository/converters/AtlasStructFormatConverter.java
>  ae92b8b 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/FullTextMapperV2.java
>  497a877 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  4a09b08 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 7b7ec65 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/AtlasServerService.java
>  542106f 
>   
> repository/src/main/jav

Re: Review Request 72593: ATLAS-3841 Response Headers: Code refactoring

2020-06-16 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72593/#review221017
---


Fix it, then Ship it!





webapp/src/main/java/org/apache/atlas/web/filters/AtlasHeaderFilter.java
Lines 40 (patched)
<https://reviews.apache.org/r/72593/#comment309790>

consider refactoring to:

```
private boolean isFileExtensionExcluded(String fileExtension) {
  boolean ret = false;

  if (ArrayUtils.isNotEmpty(excludeFileFormats)) {
ret = 
Arrays.asList(excludeFileFormats).contains(fileExtension.toLowerCase());
  }
  
  return ret;
}
```



webapp/src/main/java/org/apache/atlas/web/filters/AtlasHeaderFilter.java
Lines 51 (patched)
<https://reviews.apache.org/r/72593/#comment309789>

consider using the following to get file extension:
```
String fileExtension = FilenameUtils.getExtension(path);

if (!isFileExtensionExcluded(fileExtension)) {
  setHeaders((HttpServletResponse) response);
}
```


- Sarath Subramanian


On June 16, 2020, 7:37 a.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72593/
> ---
> 
> (Updated June 16, 2020, 7:37 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3841
> https://issues.apache.org/jira/browse/ATLAS-3841
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently headers setting code is scattered at different places.
> In this patch, code refactoring is done to set response headers from 
> AtlasHeaderFilter.java
> 
> Also this patch provides flexibility to select file formats to be excluded 
> from applying response headers.
> 
> 
> Diffs
> -
> 
>   distro/src/conf/atlas-application.properties e06e74a 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca 
>   
> webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
>  d9b1c82 
>   
> webapp/src/main/java/org/apache/atlas/web/filters/AtlasDelegatingAuthenticationEntryPoint.java
>  c629a7e 
>   webapp/src/main/java/org/apache/atlas/web/filters/AtlasHeaderFilter.java 
> fa7218c 
>   
> webapp/src/main/java/org/apache/atlas/web/filters/AtlasKnoxSSOAuthenticationFilter.java
>  1944a9f 
>   webapp/src/main/java/org/apache/atlas/web/filters/HeadersUtil.java 1f8845d 
>   webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java 
> e74a9e9 
>   webapp/src/main/webapp/WEB-INF/web.xml 2595a15 
> 
> 
> Diff: https://reviews.apache.org/r/72593/diff/1/
> 
> 
> Testing
> ---
> 
> PreCommit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1964/console
> 
> 
> Thanks,
> 
> Mandar Ambawane
> 
>



Re: Review Request 72595: ATLAS-3833 : Packaging for atlas index repair tool

2020-06-16 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72595/#review221015
---


Ship it!




Ship It!

- Sarath Subramanian


On June 16, 2020, 6:44 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72595/
> ---
> 
> (Updated June 16, 2020, 6:44 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3833
> https://issues.apache.org/jira/browse/ATLAS-3833
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Add packaging for apache atlas build for tools/atlas-index-repair
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 7159b16cf 
>   distro/src/main/assemblies/atlas-repair-index-package.xml PRE-CREATION 
>   pom.xml 8d02a6f26 
>   tools/atlas-index-repair/README PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/72595/diff/3/
> 
> 
> Testing
> ---
> 
> verified tools directory after packaging atlas
> 
> 
> Thanks,
> 
> chaitali
> 
>



Re: Review Request 72581: Upgrade maven-deploy-plugin version from 2.7 to 2.8.2 to enable

2020-06-11 Thread Sarath Subramanian

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72581/#review220996
---


Ship it!




Ship It!

- Sarath Subramanian


On June 10, 2020, 11:39 p.m., Lei Xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72581/
> ---
> 
> (Updated June 10, 2020, 11:39 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-3834
> https://issues.apache.org/jira/browse/ATLAS-3834
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Upgrade maven-deploy-plugin version from 2.7 to 2.8.2 to enable 
> 
> 
> Diffs
> -
> 
>   pom.xml 8d02a6f26 
> 
> 
> Diff: https://reviews.apache.org/r/72581/diff/1/
> 
> 
> Testing
> ---
> 
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] Apache Atlas Server Build Tools 1.0  SUCCESS [  0.560 
> s]
> [INFO] apache-atlas 3.0.0-SNAPSHOT  SUCCESS [  2.058 
> s]
> [INFO] Apache Atlas Test Utility Tools 3.0.0-SNAPSHOT . SUCCESS [  6.603 
> s]
> [INFO] Apache Atlas Integration 3.0.0-SNAPSHOT  SUCCESS [  3.921 
> s]
> [INFO] Apache Atlas Common 3.0.0-SNAPSHOT . SUCCESS [  1.279 
> s]
> [INFO] Apache Atlas Client 3.0.0-SNAPSHOT . SUCCESS [  0.162 
> s]
> [INFO] atlas-client-common 3.0.0-SNAPSHOT . SUCCESS [  1.159 
> s]
> [INFO] atlas-client-v1 3.0.0-SNAPSHOT . SUCCESS [  0.950 
> s]
> [INFO] Apache Atlas Server API 3.0.0-SNAPSHOT . SUCCESS [  0.939 
> s]
> [INFO] Apache Atlas Notification 3.0.0-SNAPSHOT ... SUCCESS [  1.769 
> s]
> [INFO] atlas-client-v2 3.0.0-SNAPSHOT . SUCCESS [  0.729 
> s]
> [INFO] Apache Atlas Graph Database Projects 3.0.0-SNAPSHOT  SUCCESS [  0.081 
> s]
> [INFO] Apache Atlas Graph Database API 3.0.0-SNAPSHOT . SUCCESS [  0.880 
> s]
> [INFO] Graph Database Common Code 3.0.0-SNAPSHOT .. SUCCESS [  0.861 
> s]
> [INFO] Apache Atlas JanusGraph-HBase2 Module 3.0.0-SNAPSHOT SUCCESS [  1.750 
> s]
> [INFO] Apache Atlas JanusGraph DB Impl 3.0.0-SNAPSHOT . SUCCESS [  4.716 
> s]
> [INFO] Apache Atlas Graph Database Implementation Dependencies 3.0.0-SNAPSHOT 
> SUCCESS [  1.521 s]
> [INFO] Apache Atlas Authorization 3.0.0-SNAPSHOT .. SUCCESS [  0.822 
> s]
> [INFO] Apache Atlas Repository 3.0.0-SNAPSHOT . SUCCESS [ 10.036 
> s]
> [INFO] Apache Atlas UI 3.0.0-SNAPSHOT . SUCCESS [02:02 
> min]
> [INFO] Apache Atlas New UI 3.0.0-SNAPSHOT . SUCCESS [ 35.906 
> s]
> [INFO] Apache Atlas Web Application 3.0.0-SNAPSHOT  SUCCESS [ 19.064 
> s]
> [INFO] Apache Atlas Documentation 3.0.0-SNAPSHOT .. SUCCESS [  1.001 
> s]
> [INFO] Apache Atlas FileSystem Model 3.0.0-SNAPSHOT ... SUCCESS [  2.191 
> s]
> [INFO] Apache Atlas Plugin Classloader 3.0.0-SNAPSHOT . SUCCESS [  1.002 
> s]
> [INFO] Apache Atlas Hive Bridge Shim 3.0.0-SNAPSHOT ... SUCCESS [  2.679 
> s]
> [INFO] Apache Atlas Hive Bridge 3.0.0-SNAPSHOT  SUCCESS [  6.436 
> s]
> [INFO] Apache Atlas Falcon Bridge Shim 3.0.0-SNAPSHOT . SUCCESS [  0.838 
> s]
> [INFO] Apache Atlas Falcon Bridge 3.0.0-SNAPSHOT .. SUCCESS [  0.860 
> s]
> [INFO] Apache Atlas Sqoop Bridge Shim 3.0.0-SNAPSHOT .. SUCCESS [  0.099 
> s]
> [INFO] Apache Atlas Sqoop Bridge 3.0.0-SNAPSHOT ... SUCCESS [  5.880 
> s]
> [INFO] Apache Atlas Storm Bridge Shim 3.0.0-SNAPSHOT .. SUCCESS [  0.271 
> s]
> [INFO] Apache Atlas Storm Bridge 3.0.0-SNAPSHOT ... SUCCESS [  1.446 
> s]
> [INFO] Apache Atlas Hbase Bridge Shim 3.0.0-SNAPSHOT .. SUCCESS [  1.990 
> s]
> [INFO] Apache Atlas Hbase Bridge 3.0.0-SNAPSHOT ... SUCCESS [ 10.128 
> s]
> [INFO] Apache HBase - Testing Util 3.0.0-SNAPSHOT . SUCCESS [  4.317 
> s]
> [INFO] Apache Atlas Kafka Bridge 3.0.0-SNAPSHOT ... SUCCESS [  1.532 
> s]
> [INFO] Apache Atlas classification updater 3.0.0-SNAPSHOT . SUCCESS [  0.699 
> s]
> [INFO] Apache Atlas Impala Hook API 3.0.0-SNAPSHOT  SUCCESS [  0.138 
> s]
> [INFO] Apache Atlas Impala Bridge Shim 3.0.0-SNAPSHOT . SUCCESS [  0.172 
> s]
> [INFO] Apache Atlas Impala Bridge 3.0.0-SNAPSHOT .. SUCCESS [  2.885 
> s]
> [INFO] Apache Atlas Distribution 3.0.0-SNAPSHOT ... SUCCESS [  0.757 
> s]
> [INFO] 
> -

  1   2   3   4   5   6   7   8   9   10   >