[jira] [Updated] (ATLAS-3133) Capture metadata for different executions of the same process in Atlas

[Srikanth Venkat (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Srikanth Venkat updated ATLAS-3133:
---
Description: 
Background: The current process metadata model within Atlas does not track 
multiple instances of execution of the same process. For example if we run same 
DDL (e.g. insert into table A select * from table B, C) multiple times Atlas 
does not capture the multiple instances of execution of the same logical 
process. 

User story: As a compliance admin or data steward, I need to be able to track 
multiple executions of the same process or pipeline that were done at different 
times and relate them to the logical process so that I can understand 
traceability and how different instances of my data pipelining operations 
performed over time.

Acceptance Criteria:

1) Every new instance of process execution is captured with the appropriate 
metadata for the process along with context (who, what, when)

2) One can navigate between process and process execution in Atlas UI and 
explore relevant metadata of each entity

3) Process nodes in lineage view display high level information about number of 
process executions associated with a particular process node.

 

 

 

 

  was:
Background: The current Atlas process metadata model within Atlas does not 
track multiple instances of execution of the same process. For example if we 
run same DDL (e.g. insert into table A select * from table B, C) multiple times 
Atlas does not capture the multiple instances of execution of the same logical 
process. 

User story: As a compliance admin or data steward, I need to be able to track 
multiple executions of the same process or pipeline that were done at different 
times and relate them to the logical process so that I can provide traceability 
and understand how different instances of my data pipelining operations 
performed over time.

Acceptance Criteria:

1) Every new instance of process execution is captured with the appropriate 
metadata for the process along with context (who, what, when)

2) One can navigate between process and process execution in Atlas UI and 
explore relevant metadata

3) Process nodes in lineage display high level information about number of 
process executions associated with a particular process node.

 

 

 

 


> Capture metadata for different executions of the same process in Atlas
> --
>
> Key: ATLAS-3133
> URL: https://issues.apache.org/jira/browse/ATLAS-3133
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core, atlas-webui
>Reporter: Srikanth Venkat
>Assignee: Aadarsh Jajodia
>Priority: Critical
> Attachments: 
> 0001-ATLAS-3133-Adding-support-for-Process-Executions-in-.patch, Adding 
> Process Executions In Atlas(Community).pdf
>
>
> Background: The current process metadata model within Atlas does not track 
> multiple instances of execution of the same process. For example if we run 
> same DDL (e.g. insert into table A select * from table B, C) multiple times 
> Atlas does not capture the multiple instances of execution of the same 
> logical process. 
> User story: As a compliance admin or data steward, I need to be able to track 
> multiple executions of the same process or pipeline that were done at 
> different times and relate them to the logical process so that I can 
> understand traceability and how different instances of my data pipelining 
> operations performed over time.
> Acceptance Criteria:
> 1) Every new instance of process execution is captured with the appropriate 
> metadata for the process along with context (who, what, when)
> 2) One can navigate between process and process execution in Atlas UI and 
> explore relevant metadata of each entity
> 3) Process nodes in lineage view display high level information about number 
> of process executions associated with a particular process node.
>  
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 70574: Import Transforms: Fix QualifiedName Transformation

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70574/#review214969
---




intg/src/main/java/org/apache/atlas/entitytransform/HiveTableEntityHandler.java
Lines 133 (patched)


use of .replace() will not work db/tbl/clu are substrings of other - like:

 - dbName:  default
 - tableName:   test1
 - clusterName: test (or default or t or ..)


- Madhan Neethiraj


On April 30, 2019, 11:09 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70574/
> ---
> 
> (Updated April 30, 2019, 11:09 p.m.)
> 
> 
> Review request for atlas, Kapildeo Nayak, Madhan Neethiraj, Nikhil Bonte, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3182
> https://issues.apache.org/jira/browse/ATLAS-3182
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Simple string manipulation.
> 
> **Transforms**
> 
> import-options.json
> ```
> {
> "options": {
> "transformers": "[{\"conditions\":{\"hive_db.clusterName\":\"EQUALS: 
> mycluster0\"},\"action\":{\"hive_db.clusterName\":\"SET: 
> mycluster1\"}},{\"conditions\":{\"hive_db.name\":\"EQUALS: 
> test_partition_bootstrap\"},\"action\":{\"hive_db.name\":\"SET: 
> test_partition_bootstrap_target\"}}]"
> }
> }
> ```
> **Import Zip**
> (Attached)
> 
> **CURL**
> 
> curl -X POST -u admin:admin123 -H "Content-Type: multipart/form-data" -H 
> "Cache-Control: no-cache" -F data=@./test_partition_bootstrap.zip -F 
> request=@./import-options.json http://localhost:21000/api/atlas/admin/import
> 
> 
> Diffs
> -
> 
>   
> intg/src/main/java/org/apache/atlas/entitytransform/HiveTableEntityHandler.java
>  9eb44d782 
>   
> intg/src/test/java/org/apache/atlas/entitytransform/TransformationHandlerTest.java
>  d6b0ede2b 
>   intg/src/test/resources/json/entity1.json PRE-CREATION 
>   intg/src/test/resources/json/transform1.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/70574/diff/1/
> 
> 
> Testing
> ---
> 
> **Unit tests**
> Additional test.
> 
> **Functional test**
> Existing Import flow using test data.
> 
> 
> File Attachments
> 
> 
> Zip to Import
>   
> https://reviews.apache.org/media/uploaded/files/2019/04/30/44469b66-884c-4bde-8200-e41306a19b3b__test_partition_bootstrap.zip
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

[jira] [Updated] (ATLAS-3133) Capture metadata for different executions of the same process in Atlas

[Aadarsh Jajodia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aadarsh Jajodia updated ATLAS-3133:
---
Attachment: Adding Process Executions In Atlas(Community).pdf

> Capture metadata for different executions of the same process in Atlas
> --
>
> Key: ATLAS-3133
> URL: https://issues.apache.org/jira/browse/ATLAS-3133
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core, atlas-webui
>Reporter: Srikanth Venkat
>Assignee: Aadarsh Jajodia
>Priority: Critical
> Attachments: 
> 0001-ATLAS-3133-Adding-support-for-Process-Executions-in-.patch, Adding 
> Process Executions In Atlas(Community).pdf
>
>
> Background: The current Atlas process metadata model within Atlas does not 
> track multiple instances of execution of the same process. For example if we 
> run same DDL (e.g. insert into table A select * from table B, C) multiple 
> times Atlas does not capture the multiple instances of execution of the same 
> logical process. 
> User story: As a compliance admin or data steward, I need to be able to track 
> multiple executions of the same process or pipeline that were done at 
> different times and relate them to the logical process so that I can provide 
> traceability and understand how different instances of my data pipelining 
> operations performed over time.
> Acceptance Criteria:
> 1) Every new instance of process execution is captured with the appropriate 
> metadata for the process along with context (who, what, when)
> 2) One can navigate between process and process execution in Atlas UI and 
> explore relevant metadata
> 3) Process nodes in lineage display high level information about number of 
> process executions associated with a particular process node.
>  
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

contribute to Atlas project

[Carol Drummond]

Would you please add me to the list of potential contributors to the Apache
project? My Jira ID is "cdrummond".
Thank you!
Carol

Review Request 70574: Import Transforms: Fix QualifiedName Transformation

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70574/
---

Review request for atlas, Kapildeo Nayak, Madhan Neethiraj, Nikhil Bonte, and 
Sarath Subramanian.


Bugs: ATLAS-3182
https://issues.apache.org/jira/browse/ATLAS-3182


Repository: atlas


Description
---

Simple string manipulation.


Diffs
-

  
intg/src/main/java/org/apache/atlas/entitytransform/HiveTableEntityHandler.java 
9eb44d782 
  
intg/src/test/java/org/apache/atlas/entitytransform/TransformationHandlerTest.java
 d6b0ede2b 
  intg/src/test/resources/json/entity1.json PRE-CREATION 
  intg/src/test/resources/json/transform1.json PRE-CREATION 


Diff: https://reviews.apache.org/r/70574/diff/1/


Testing
---

**Unit tests**
Additional test.

**Functional test**
Existing Import flow using test data.


Thanks,

Ashutosh Mestry

[jira] [Created] (ATLAS-3182) Import Transforms: Hive Table Qualified Name is Modified Incorrectly

[Ashutosh Mestry (JIRA)]

Ashutosh Mestry created ATLAS-3182:
--

 Summary: Import Transforms: Hive Table Qualified Name is Modified 
Incorrectly
 Key: ATLAS-3182
 URL: https://issues.apache.org/jira/browse/ATLAS-3182
 Project: Atlas
  Issue Type: Bug
Reporter: Ashutosh Mestry
Assignee: Ashutosh Mestry


*Background*

Existing transformation framework that is used in _ImportService_ does not 
correctly replace _qualifiedName_ for _hive_table_ entities in cases where 
entity _name_ and _qualifiedName_ are different.

*Steps to Duplicate*
 # Create _hive_db_, say _db1_.
 # Create _hive_table_ entity such that it has name = _testtable_ and 
_qualifiedName = db1.testtable-123@cl1_.
 # Use the transforms handler to convert _hive_db_ (_db1_) name to dbX and 
_clusterName = clX_.

*Expected Results*

The resulting _hive_db_ should have _name_ and _qualifiedName_ changed to _dbX._

The _hive_table_ should have its name = _testtable_ and _qualifiedName = 
dbX.testtable-123@clX._

*Actual Results*

The _hive_db_ entity is correctly transformed.

The _hive_table_ entity's qualifiedName gets changed to _dbX.testtable@clX._



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ATLAS-3181) The URL produced by the swagger interactive API is missing "/api/atlas/"

[Carol Drummond (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830739#comment-16830739
 ] 

Carol Drummond commented on ATLAS-3181:
---

The fix is to augment the enunciate.xml file with the "basePath" parameter as 
follows:

{{}}

> The URL produced by the swagger interactive API is missing "/api/atlas/"
> 
>
> Key: ATLAS-3181
> URL: https://issues.apache.org/jira/browse/ATLAS-3181
> Project: Atlas
>  Issue Type: Bug
>  Components: atlas-webui
>Affects Versions: 1.0.0
>Reporter: Carol Drummond
>Priority: Minor
>
> Swagger isn't building the API url correctly: it's missing the "api/atlas" 
> parts of the URL:
>  
> Here's what it builds:
> [https://localhost:21000/v2/entity/guid/4b2aa848-fb3a-401a-9e05-29a08ab2c846/classifications]
> {quote}curl -X GET --header 'Accept: application/json' 
> '[https://localhost:21000/v2/entity/guid/4b2aa848-fb3a-401a-9e05-29a08ab2c846/classifications]'{quote}
>  
> Here's what works (added "/api/atlas"):
>  
> [http://localhost:21000|http://localhost:21000/]*/api/atlas/*v2/entity/guid/4b2aa848-fb3a-401a-9e05-29a08ab2c846/classifications
>  
> {quote}curl -X GET 
> '[http://localhost:21000/api/atlas/v2/entity/guid/4b2aa848-fb3a-401a-9e05-29a08ab2c846/classifications]'
>  -u user:password{quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (ATLAS-3181) The URL produced by the swagger interactive API is missing "/api/atlas/"

[Carol Drummond (JIRA)]

Carol Drummond created ATLAS-3181:
-

 Summary: The URL produced by the swagger interactive API is 
missing "/api/atlas/"
 Key: ATLAS-3181
 URL: https://issues.apache.org/jira/browse/ATLAS-3181
 Project: Atlas
  Issue Type: Bug
  Components: atlas-webui
Affects Versions: 1.0.0
Reporter: Carol Drummond


Swagger isn't building the API url correctly: it's missing the "api/atlas" 
parts of the URL:
 
Here's what it builds:

[https://localhost:21000/v2/entity/guid/4b2aa848-fb3a-401a-9e05-29a08ab2c846/classifications]


{quote}curl -X GET --header 'Accept: application/json' 
'[https://localhost:21000/v2/entity/guid/4b2aa848-fb3a-401a-9e05-29a08ab2c846/classifications]'{quote}
 
Here's what works (added "/api/atlas"):
 
[http://localhost:21000|http://localhost:21000/]*/api/atlas/*v2/entity/guid/4b2aa848-fb3a-401a-9e05-29a08ab2c846/classifications
 
{quote}curl -X GET 
'[http://localhost:21000/api/atlas/v2/entity/guid/4b2aa848-fb3a-401a-9e05-29a08ab2c846/classifications]'
 -u user:password{quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[VOTE] Release Apache Atlas version 2.0.0 - rc1

[Sarath Subramanian]

Atlas team,

Apache Atlas 2.0.0 Release Candidate #1 is now available for a vote within
dev community.

Following commits went into branch-2.0 since the last release candidate
(rc0):

 - ATLAS-3168: Fix Intermittent UT failure:
KafkaNotificationTest.initNotificationService()
 - ATLAS-3148: Implement Hive Metastore hook for Atlas
 - ATLAS-3167: Exception while starting Atlas with Cassandra
 - ATLAS-3160: Remove unthrown exception specified in AtlasRelationshipDef
constructors
 - ATLAS-3157: Add Integration tests for Hive metastore hook
 - ATLAS-3174: UI - Exclude process or deleted entity from search drop-down
if respective options are chosen
 - ATLAS-3155: UI - IE-11 - Name entry not clickable
 - ATLAS-3168: PatchFix: Unit test fixes and optimization.
 - ATLAS-3168: PatchFix: Support for HA Mode
 - ATLAS-3129: Fix SSL Truststore reloader leak from AtlasBaseClient
 - ATLAS-3159: UI - Add Reset button for Lienage
 - ATLAS-3154: UI - Lineage PNG export not working on Firefox
 - ATLAS-3158: UI - Render schema tab using relationshipAttributes (Columns)
 - ATLAS-3145: UI - Entity showing Active even if the relationshipStatus is
Deleted.

Links to the release artifacts are given below. Please review and vote.

The vote will be open for at least 72 hours or until necessary votes are
reached.
  [ ] +1 approve
  [ ] +0 no opinion
  [ ] -1 disapprove (and reason why)


Thanks,
Sarath

List of issues addressed in this release:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20ATLAS%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%202.0.0%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
Git tag for the release:
https://github.com/apache/atlas/tree/release-2.0.0-rc1
Sources for the release:
https://dist.apache.org/repos/dist/dev/atlas/2.0.0-rc1/apache-atlas-2.0.0-sources.tar.gz

Source release verification:
PGP Signature:
https://dist.apache.org/repos/dist/dev/atlas/2.0.0-rc1/apache-atlas-2.0.0-sources.tar.gz.asc
SHA512 Hash:
https://dist.apache.org/repos/dist/dev/atlas/2.0.0-rc1/apache-atlas-2.0.0-sources.tar.gz.sha512
MD5 Hash:
https://dist.apache.org/repos/dist/dev/atlas/2.0.0-rc1/apache-atlas-2.0.0-sources.tar.gz.md5

Keys to verify the signature of the release artifacts are available at:
https://dist.apache.org/repos/dist/release/atlas/KEYS