Re: Review Request 65779: Data Migration: Moving Data to New Version of Atlas

[Apoorv Naik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65779/#review198242
---




common/src/main/java/org/apache/atlas/service/Services.java
Line 52 (original), 61 (patched)


Wouldn't it be easier to add another state called MIGRATION in ServiceState 
enum and the Service.start() would be a no-op when Atlas is in migration state.



repository/src/main/java/org/apache/atlas/repository/impexp/DataMigrationService.java
Lines 89 (patched)


AtlasException is old class, use AtlasBaseException instead.



repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
Lines 53 (patched)


Constructor injection over field injection. Will be easier to test



webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
Lines 419 (patched)


Might be a better idea to send a java object instead of String.


- Apoorv Naik


On Feb. 23, 2018, 10:38 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65779/
> ---
> 
> (Updated Feb. 23, 2018, 10:38 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Ruchi Solani.
> 
> 
> Bugs: ATLAS-2460
> https://issues.apache.org/jira/browse/ATLAS-2460
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> This implementation deals with the 'import into' part of the data migration 
> process. 
> 
> It assumes:
> - Export from older cluster is done.
> - Generated file has been moved to newer cluster.
> 
> **Implementation**
> 
> During _Atlas_ server startup, the configuration parameter is checked, if 
> that parameter exists, all services except _DataMigrationService_ is started. 
> Migration is started. Atlas server is available in _MIGRATION_ mode. It 
> processes REST calls made only to the _AdminResource_.
> 
> Here's are the udpates:
> - New configuration parameter has been added:
> _atlas.migration.mode.filename=_
>   This configuration parameter is set by Ambari as part of its migration 
> orchestration. 
> - _DataMigrationService_: New service that performs async migration as soon 
> as Atlas server starts up.
> - _ImportProgressTracker_: Added to be able to query progress of import.
> - _ImportService_: Modified to send updates to _ImportProgressTracker_.
> - _AdminResource.getStatus()_ Now supplies additional status about migration.
> - _ServiceState_ Modified to carry additional status _MIGRATION_. This status 
> is set by looking at the configuration parameter above.
> - _AtlasEntityStoreV1_ modified to block sending out notifications during 
> migration.
> - _Services_ modified for special handling of _DataMigrationService_.
> 
> 
> **CURL**
> Check status using:
> ```
> curl -X GET -u admin:admin -H "Content-Type: application/json" -H 
> "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
> ```
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3 
>   common/src/main/java/org/apache/atlas/service/Services.java 1267dc92 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 
> 1ea961d8 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportResult.java 
> b97cbb3f 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/DataMigrationService.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportProgressTracker.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
>  c976c593 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java
>  bf417dd0 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/BulkImporterImpl.java
>  af310954 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java
>  0ba06b94 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java
>  07524d08 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java
>  f4b70531 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java
>  8de978c8 
>   webapp/src/main/java/org/apache/atlas/Atlas.java 7cf6e3ea 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
> 86485fba 
>   webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 
> 3fe8d18c 
>   

Re: Review Request 65790: Data Migration Utility: Export

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65790/
---

(Updated Feb. 24, 2018, 12:20 a.m.)


Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Ruchi Solani.


Changes
---

Updates include: Migration utility zip.


Bugs: ATLAS-2461
https://issues.apache.org/jira/browse/ATLAS-2461


Repository: atlas


Description
---

**Background**
The data migration utility allows for exporting data from an Atlas instance 
without the server running. This is important as it will prevent Atlas from 
processing any requests.

**Approach**
The migration is a new utility that will perform export of data using a 
pre-defined export request file.

The approach used by this application:
- Create an application context (_migrationContext.xml_). This context, 
prevents instantiation of number of classes, most notably _webapp_, 
_notifications_, _listeners_.
- Create _ImportService_ using Spring's framework classes.

Here are the pieces:
- _MigrationApp_: Contains _main_. It is the entry point for the app.
- _Interactors_: Contains plumbing needed to use the new _migrationContext.xml_ 
and create _applicationContext_ for use.
- Dengenerated classes _EmptyNotification_, _EmptyNotificationChangeListener_. 
These are necessary to launch the application.
- _atlas-migration-setup.sh_ Installs the JAR at 
_/usr/hdp/current/atlas-server/server/webapp/atlas/WEB-INF/lib_
- _atlas_migration.py_ Creates environment for the app to execute. It launches 
the application.
- _export-request.json_ This can be modified for environments that do not have 
latest improvements to _Export_.
- _atlas-migration-shutdown.sh_ Removes the installed migration app and 
restarts Atlas.

**Build**
The project maven's assembly building plugin. The build will create a ZIP file 
with the necessary files.

_mvn install package_ generates a ZIP file in _target_ directory.

**Usage**
- Copy the ZIP file to the server location.
- Use _unzip atlas-migration-kit-0.8.3-SNAPSHOT-bin.zip_
- Contents will be unzipped to _atlas-migration-kit-0.8.3-SNAPSHOT_ directory.
- cd _atlas-migration-kit-0.8.3-SNAPSHOT_
- _bash ./atlas-migration-setup.sh_
- _python /usr/hdp/current/atlas-server/bin/atlas_migration.py export 
export-2.zip_
- Watch the progress using _tail -f /var/log/atlas/application.log_
- When done use: _bash ./atlas-migration-shutdown.sh_
- Move the generated ZIP file to the new clusters server location.


Diffs
-

  
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
 655252cb 
  
repository/src/main/java/org/apache/atlas/repository/userprofile/UserProfileService.java
 a428b92e 
  tools/atlas-migration-utility/pom.xml PRE-CREATION 
  tools/atlas-migration-utility/src/assembly/bin.xml PRE-CREATION 
  
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/EmptyNotification.java
 PRE-CREATION 
  
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/EmptyNotificationChangeListener.java
 PRE-CREATION 
  
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/Interactors.java
 PRE-CREATION 
  
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/MigrationApp.java
 PRE-CREATION 
  tools/atlas-migration-utility/src/main/resources/README PRE-CREATION 
  tools/atlas-migration-utility/src/main/resources/atlas-migration-setup.sh 
PRE-CREATION 
  tools/atlas-migration-utility/src/main/resources/atlas-migration-shutdown.sh 
PRE-CREATION 
  tools/atlas-migration-utility/src/main/resources/atlas_migration.py 
PRE-CREATION 
  tools/atlas-migration-utility/src/main/resources/export-request.json 
PRE-CREATION 
  tools/atlas-migration-utility/src/main/resources/migrationContext.xml 
PRE-CREATION 


Diff: https://reviews.apache.org/r/65790/diff/1/


Testing
---

**Unit tests**
None.

**Functional tests**
Exports from existing clusters.


File Attachments (updated)


Migration Utility
  
https://reviews.apache.org/media/uploaded/files/2018/02/24/e8090ed0-13b6-4253-a59c-d3f2098943af__atlas-migration-kit-0.8.3-SNAPSHOT-bin.zip


Thanks,

Ashutosh Mestry

Review Request 65790: Data Migration Utility: Export

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65790/
---

Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Ruchi Solani.


Bugs: ATLAS-2461
https://issues.apache.org/jira/browse/ATLAS-2461


Repository: atlas


Description
---

**Background**
The data migration utility allows for exporting data from an Atlas instance 
without the server running. This is important as it will prevent Atlas from 
processing any requests.

**Approach**
The migration is a new utility that will perform export of data using a 
pre-defined export request file.

The approach used by this application:
- Create an application context (_migrationContext.xml_). This context, 
prevents instantiation of number of classes, most notably _webapp_, 
_notifications_, _listeners_.
- Create _ImportService_ using Spring's framework classes.

Here are the pieces:
- _MigrationApp_: Contains _main_. It is the entry point for the app.
- _Interactors_: Contains plumbing needed to use the new _migrationContext.xml_ 
and create _applicationContext_ for use.
- Dengenerated classes _EmptyNotification_, _EmptyNotificationChangeListener_. 
These are necessary to launch the application.
- _atlas-migration-setup.sh_ Installs the JAR at 
_/usr/hdp/current/atlas-server/server/webapp/atlas/WEB-INF/lib_
- _atlas_migration.py_ Creates environment for the app to execute. It launches 
the application.
- _export-request.json_ This can be modified for environments that do not have 
latest improvements to _Export_.
- _atlas-migration-shutdown.sh_ Removes the installed migration app and 
restarts Atlas.

**Build**
The project maven's assembly building plugin. The build will create a ZIP file 
with the necessary files.

_mvn install package_ generates a ZIP file in _target_ directory.

**Usage**
- Copy the ZIP file to the server location.
- Use _unzip atlas-migration-kit-0.8.3-SNAPSHOT-bin.zip_
- Contents will be unzipped to _atlas-migration-kit-0.8.3-SNAPSHOT_ directory.
- cd _atlas-migration-kit-0.8.3-SNAPSHOT_
- _bash ./atlas-migration-setup.sh_
- _python /usr/hdp/current/atlas-server/bin/atlas_migration.py export 
export-2.zip_
- Watch the progress using _tail -f /var/log/atlas/application.log_
- When done use: _bash ./atlas-migration-shutdown.sh_
- Move the generated ZIP file to the new clusters server location.


Diffs
-

  
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
 655252cb 
  
repository/src/main/java/org/apache/atlas/repository/userprofile/UserProfileService.java
 a428b92e 
  tools/atlas-migration-utility/pom.xml PRE-CREATION 
  tools/atlas-migration-utility/src/assembly/bin.xml PRE-CREATION 
  
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/EmptyNotification.java
 PRE-CREATION 
  
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/EmptyNotificationChangeListener.java
 PRE-CREATION 
  
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/Interactors.java
 PRE-CREATION 
  
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/MigrationApp.java
 PRE-CREATION 
  tools/atlas-migration-utility/src/main/resources/README PRE-CREATION 
  tools/atlas-migration-utility/src/main/resources/atlas-migration-setup.sh 
PRE-CREATION 
  tools/atlas-migration-utility/src/main/resources/atlas-migration-shutdown.sh 
PRE-CREATION 
  tools/atlas-migration-utility/src/main/resources/atlas_migration.py 
PRE-CREATION 
  tools/atlas-migration-utility/src/main/resources/export-request.json 
PRE-CREATION 
  tools/atlas-migration-utility/src/main/resources/migrationContext.xml 
PRE-CREATION 


Diff: https://reviews.apache.org/r/65790/diff/1/


Testing
---

**Unit tests**
None.

**Functional tests**
Exports from existing clusters.


Thanks,

Ashutosh Mestry

[jira] [Updated] (ATLAS-2461) Data Migration: Create Migration Utility to Export Contents from Atlas Server

[Ashutosh Mestry (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry updated ATLAS-2461:
---
Attachment: ATLAS-2461-migration-utility-branch-0.8.patch

> Data Migration: Create Migration Utility to Export Contents from Atlas Server
> -
>
> Key: ATLAS-2461
> URL: https://issues.apache.org/jira/browse/ATLAS-2461
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 0.8.2
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Fix For: 0.8.3
>
> Attachments: ATLAS-2461-migration-utility-branch-0.8.patch
>
>
> *Background*
> As part of data migration, earlier versions of Atlas (prior to v 1.x) should 
> be export contents of their database to a ZIP file, that would later be 
> imported into a cluster running new version.
> During this process, the server should not accept any requests or process 
> notifications.
> This process should work on an existing cluster without making it necessary 
> to upgrade.
> *Approach*
> Create a command line utility that would perform export using core components 
> from Atlas. This would allow Atlas to be shutdown, so that it does not 
> process any incoming requests or hook messages.
> This utility should allow for using external configuration JSON so that it 
> can be used on Atlas setup that does not have latest version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (ATLAS-2461) Data Migration: Create Migration Utility to Export Contents from Atlas Server

[Ashutosh Mestry (JIRA)]

Ashutosh Mestry created ATLAS-2461:
--

 Summary: Data Migration: Create Migration Utility to Export 
Contents from Atlas Server
 Key: ATLAS-2461
 URL: https://issues.apache.org/jira/browse/ATLAS-2461
 Project: Atlas
  Issue Type: Improvement
  Components:  atlas-core
Affects Versions: 0.8.2
Reporter: Ashutosh Mestry
Assignee: Ashutosh Mestry
 Fix For: 0.8.3


*Background*

As part of data migration, earlier versions of Atlas (prior to v 1.x) should be 
export contents of their database to a ZIP file, that would later be imported 
into a cluster running new version.

During this process, the server should not accept any requests or process 
notifications.

This process should work on an existing cluster without making it necessary to 
upgrade.

*Approach*

Create a command line utility that would perform export using core components 
from Atlas. This would allow Atlas to be shutdown, so that it does not process 
any incoming requests or hook messages.

This utility should allow for using external configuration JSON so that it can 
be used on Atlas setup that does not have latest version.


**



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (ATLAS-2461) Data Migration: Create Migration Utility to Export Contents from Atlas Server

[Ashutosh Mestry (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry updated ATLAS-2461:
---
Description: 
*Background*

As part of data migration, earlier versions of Atlas (prior to v 1.x) should be 
export contents of their database to a ZIP file, that would later be imported 
into a cluster running new version.

During this process, the server should not accept any requests or process 
notifications.

This process should work on an existing cluster without making it necessary to 
upgrade.

*Approach*

Create a command line utility that would perform export using core components 
from Atlas. This would allow Atlas to be shutdown, so that it does not process 
any incoming requests or hook messages.

This utility should allow for using external configuration JSON so that it can 
be used on Atlas setup that does not have latest version.

  was:
*Background*

As part of data migration, earlier versions of Atlas (prior to v 1.x) should be 
export contents of their database to a ZIP file, that would later be imported 
into a cluster running new version.

During this process, the server should not accept any requests or process 
notifications.

This process should work on an existing cluster without making it necessary to 
upgrade.

*Approach*

Create a command line utility that would perform export using core components 
from Atlas. This would allow Atlas to be shutdown, so that it does not process 
any incoming requests or hook messages.

This utility should allow for using external configuration JSON so that it can 
be used on Atlas setup that does not have latest version.


**


> Data Migration: Create Migration Utility to Export Contents from Atlas Server
> -
>
> Key: ATLAS-2461
> URL: https://issues.apache.org/jira/browse/ATLAS-2461
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 0.8.2
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Fix For: 0.8.3
>
>
> *Background*
> As part of data migration, earlier versions of Atlas (prior to v 1.x) should 
> be export contents of their database to a ZIP file, that would later be 
> imported into a cluster running new version.
> During this process, the server should not accept any requests or process 
> notifications.
> This process should work on an existing cluster without making it necessary 
> to upgrade.
> *Approach*
> Create a command line utility that would perform export using core components 
> from Atlas. This would allow Atlas to be shutdown, so that it does not 
> process any incoming requests or hook messages.
> This utility should allow for using external configuration JSON so that it 
> can be used on Atlas setup that does not have latest version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ATLAS-2456) Implement tag propagation using relationships

[Madhan Neethiraj (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-2456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375114#comment-16375114
 ] 

Madhan Neethiraj commented on ATLAS-2456:
-

[~sarath.ku...@gmail.com] - added another minor fix in handling of string 
conversion to EntityAuditAction V1/V2. Please review  [^ATLAS-2456-7.patch] .

> Implement tag propagation using relationships
> -
>
> Key: ATLAS-2456
> URL: https://issues.apache.org/jira/browse/ATLAS-2456
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 1.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: ATLAS-2456-6.patch, ATLAS-2456-7.patch, TAG PROPAGATION 
> IN ATLAS v.1.pdf, tag_propagation_rest_api
>
>
> {color:#172b4d}Scalable way to quickly and efficiently propagate tags for 
> efficient searches and tag based security. Likewise tags for derivative 
> dataset should be inherited from the parent. For example, if an entity is 
> tagged "PII" then resulting entity created from a CTAS operation should also 
> be tagged "secret" to maintain the classification of the parent. In the case 
> where 2 or more datasets are aggregated the derivative dataset should be a 
> union of all parent tags.{color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (ATLAS-2456) Implement tag propagation using relationships

[Madhan Neethiraj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-2456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-2456:

Attachment: ATLAS-2456-7.patch

> Implement tag propagation using relationships
> -
>
> Key: ATLAS-2456
> URL: https://issues.apache.org/jira/browse/ATLAS-2456
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 1.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: ATLAS-2456-6.patch, ATLAS-2456-7.patch, TAG PROPAGATION 
> IN ATLAS v.1.pdf, tag_propagation_rest_api
>
>
> {color:#172b4d}Scalable way to quickly and efficiently propagate tags for 
> efficient searches and tag based security. Likewise tags for derivative 
> dataset should be inherited from the parent. For example, if an entity is 
> tagged "PII" then resulting entity created from a CTAS operation should also 
> be tagged "secret" to maintain the classification of the parent. In the case 
> where 2 or more datasets are aggregated the derivative dataset should be a 
> union of all parent tags.{color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 65779: Data Migration: Moving Data to New Version of Atlas

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65779/
---

Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Ruchi Solani.


Bugs: ATLAS-2460
https://issues.apache.org/jira/browse/ATLAS-2460


Repository: atlas


Description
---

**Background**
This implementation deals with the 'import into' part of the data migration 
process. 

It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.

**Implementation**

During _Atlas_ server startup, the configuration parameter is checked, if that 
parameter exists, all services except _DataMigrationService_ is started. 
Migration is started. Atlas server is available in _MIGRATION_ mode. It 
processes REST calls made only to the _AdminResource_.

Here's are the udpates:
- New configuration parameter has been added:
_atlas.migration.mode.filename=_
  This configuration parameter is set by Ambari as part of its migration 
orchestration. 
- _DataMigrationService_: New service that performs async migration as soon as 
Atlas server starts up.
- _ImportProgressTracker_: Added to be able to query progress of import.
- _ImportService_: Modified to send updates to _ImportProgressTracker_.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status 
is set by looking at the configuration parameter above.
- _AtlasEntityStoreV1_ modified to block sending out notifications during 
migration.
- _Services_ modified for special handling of _DataMigrationService_.


**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H 
"Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```


Diffs
-

  common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3 
  common/src/main/java/org/apache/atlas/service/Services.java 1267dc92 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 
1ea961d8 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportResult.java 
b97cbb3f 
  
repository/src/main/java/org/apache/atlas/repository/impexp/DataMigrationService.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ImportProgressTracker.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 
c976c593 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java
 bf417dd0 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/BulkImporterImpl.java
 af310954 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java
 0ba06b94 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java
 07524d08 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java
 f4b70531 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java
 8de978c8 
  webapp/src/main/java/org/apache/atlas/Atlas.java 7cf6e3ea 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
86485fba 
  webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c 
  webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 
fd96fc3d 


Diff: https://reviews.apache.org/r/65779/diff/1/


Testing
---

**Unit tests**
None added.

**Functional tests**
Steps to test:
- Place the exported file say _export-1.zip_ at _/root_
- Add to _Atlas_ Ambari's customer property:
_atlas.migration.mode.filename=/root/export-1.zip_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f 
/var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the 
import.


Thanks,

Ashutosh Mestry

[jira] [Updated] (ATLAS-2460) Data Migration: Moving Data from Earlier Versions of Atlas to Most Recent

[Ashutosh Mestry (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-2460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry updated ATLAS-2460:
---
Description: 
*Background*

The most recent version of Atlas uses _JanusGraph_ as database. Earlier 
versions used to use _Titan v0.5.4_. The formats used for storing data have 
changed.

The current version of Atlas also implement features within Atlas entities. 
This makes the storages structures differ.

A data migration approach thus becomes necessary to address the format 
incompatibilities.

*Approach*

Earlier version of Atlas could use _Export_ process to extract data out of the 
_Titan_ database. This ZIP file would then be moved to a cluster with newer 
version of Atlas installed. An _Import_ process on the new cluster would update 
the new cluster with data migrated to the new format.

It should be possible for _Data Migration_ to be initiated from Ambari, so that 
it becomes part of the Ambari's upgrade process.

It should also be possible to see the status of the migration as it progresses. 
During the process, no hook messages should be processed or any notifications 
sent out.

  was:
*Background*

The most recent version of Atlas uses _JanusGraph_ as database. Earlier 
versions used to use _Titan v0.5.4_. The formats used for storing data have 
changed.

The current version of Atlas also implement features within Atlas entities. 
This makes the storages structures differ.

A data migration approach thus becomes necessary to address the format 
incompatibilities.

*Approach*

Earlier version of Atlas could use _Export_ process to extract data out of the 
_Titan_ database. This ZIP file would then be moved to a cluster with newer 
version of Atlas installed. An _Import_ process on the new cluster would update 
the new cluster with data migrated to the new format.

It should be possible for _Data Migration_ to be initiated from Ambari, so that 
it becomes part of the Ambari's upgrade process.

 


> Data Migration: Moving Data from Earlier Versions of Atlas to Most Recent
> -
>
> Key: ATLAS-2460
> URL: https://issues.apache.org/jira/browse/ATLAS-2460
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: trunk
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Attachments: ATLAS-2460-Data-Migration.patch
>
>
> *Background*
> The most recent version of Atlas uses _JanusGraph_ as database. Earlier 
> versions used to use _Titan v0.5.4_. The formats used for storing data have 
> changed.
> The current version of Atlas also implement features within Atlas entities. 
> This makes the storages structures differ.
> A data migration approach thus becomes necessary to address the format 
> incompatibilities.
> *Approach*
> Earlier version of Atlas could use _Export_ process to extract data out of 
> the _Titan_ database. This ZIP file would then be moved to a cluster with 
> newer version of Atlas installed. An _Import_ process on the new cluster 
> would update the new cluster with data migrated to the new format.
> It should be possible for _Data Migration_ to be initiated from Ambari, so 
> that it becomes part of the Ambari's upgrade process.
> It should also be possible to see the status of the migration as it 
> progresses. During the process, no hook messages should be processed or any 
> notifications sent out.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (ATLAS-2460) Data Migration: Moving Data from Earlier Versions of Atlas to Most Recent

[Ashutosh Mestry (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-2460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry updated ATLAS-2460:
---
Description: 
*Background*

The most recent version of Atlas uses _JanusGraph_ as database. Earlier 
versions used to use _Titan v0.5.4_. The formats used for storing data have 
changed.

The current version of Atlas also implement features within Atlas entities. 
This makes the storages structures differ.

A data migration approach thus becomes necessary to address the format 
incompatibilities.

*Approach*

Earlier version of Atlas could use _Export_ process to extract data out of the 
_Titan_ database. This ZIP file would then be moved to a cluster with newer 
version of Atlas installed. An _Import_ process on the new cluster would update 
the new cluster with data migrated to the new format.

It should be possible for _Data Migration_ to be initiated from Ambari, so that 
it becomes part of the Ambari's upgrade process.

 

  was:
*Background*

The most recent version of Atlas uses _JanusGraph_ as database. Earlier 
versions used to use _Titan v0.5.4_. The formats used for storing data have 
changed.

The current version of Atlas also implement features within Atlas entities. 
This makes the storages structures differ.

A data migration approach thus becomes necessary to address the format 
incompatibilities.

*Approach***

Earlier version of Atlas could use _Export_ process to extract data out of the 
_Titan_ database. This ZIP file would then be moved to a cluster with newer 
version of Atlas installed. An _Import_ process on the new cluster would update 
the new cluster with data migrated to the new format.

It should be possible for _Data Migration_ to be initiated from Ambari, so that 
it becomes part of the Ambari's upgrade process.

 


> Data Migration: Moving Data from Earlier Versions of Atlas to Most Recent
> -
>
> Key: ATLAS-2460
> URL: https://issues.apache.org/jira/browse/ATLAS-2460
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: trunk
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Attachments: ATLAS-2460-Data-Migration.patch
>
>
> *Background*
> The most recent version of Atlas uses _JanusGraph_ as database. Earlier 
> versions used to use _Titan v0.5.4_. The formats used for storing data have 
> changed.
> The current version of Atlas also implement features within Atlas entities. 
> This makes the storages structures differ.
> A data migration approach thus becomes necessary to address the format 
> incompatibilities.
> *Approach*
> Earlier version of Atlas could use _Export_ process to extract data out of 
> the _Titan_ database. This ZIP file would then be moved to a cluster with 
> newer version of Atlas installed. An _Import_ process on the new cluster 
> would update the new cluster with data migrated to the new format.
> It should be possible for _Data Migration_ to be initiated from Ambari, so 
> that it becomes part of the Ambari's upgrade process.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (ATLAS-2460) Data Migration: Moving Data from Earlier Versions of Atlas to Most Recent

[Ashutosh Mestry (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-2460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry updated ATLAS-2460:
---
Summary: Data Migration: Moving Data from Earlier Versions of Atlas to Most 
Recent  (was: Data Migration: Moving Data from Earlier Versions of Atlas to be 
Migrated to Most Recent)

> Data Migration: Moving Data from Earlier Versions of Atlas to Most Recent
> -
>
> Key: ATLAS-2460
> URL: https://issues.apache.org/jira/browse/ATLAS-2460
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: trunk
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Attachments: ATLAS-2460-Data-Migration.patch
>
>
> *Background*
> The most recent version of Atlas uses _JanusGraph_ as database. Earlier 
> versions used to use _Titan v0.5.4_. The formats used for storing data have 
> changed.
> The current version of Atlas also implement features within Atlas entities. 
> This makes the storages structures differ.
> A data migration approach thus becomes necessary to address the format 
> incompatibilities.
> *Approach***
> Earlier version of Atlas could use _Export_ process to extract data out of 
> the _Titan_ database. This ZIP file would then be moved to a cluster with 
> newer version of Atlas installed. An _Import_ process on the new cluster 
> would update the new cluster with data migrated to the new format.
> It should be possible for _Data Migration_ to be initiated from Ambari, so 
> that it becomes part of the Ambari's upgrade process.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (ATLAS-2460) Data Migration: Moving Data from Earlier Versions of Atlas to be Migrated to Most Recent

[Ashutosh Mestry (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-2460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry updated ATLAS-2460:
---
Attachment: ATLAS-2460-Data-Migration.patch

> Data Migration: Moving Data from Earlier Versions of Atlas to be Migrated to 
> Most Recent
> 
>
> Key: ATLAS-2460
> URL: https://issues.apache.org/jira/browse/ATLAS-2460
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: trunk
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Attachments: ATLAS-2460-Data-Migration.patch
>
>
> *Background*
> The most recent version of Atlas uses _JanusGraph_ as database. Earlier 
> versions used to use _Titan v0.5.4_. The formats used for storing data have 
> changed.
> The current version of Atlas also implement features within Atlas entities. 
> This makes the storages structures differ.
> A data migration approach thus becomes necessary to address the format 
> incompatibilities.
> *Approach***
> Earlier version of Atlas could use _Export_ process to extract data out of 
> the _Titan_ database. This ZIP file would then be moved to a cluster with 
> newer version of Atlas installed. An _Import_ process on the new cluster 
> would update the new cluster with data migrated to the new format.
> It should be possible for _Data Migration_ to be initiated from Ambari, so 
> that it becomes part of the Ambari's upgrade process.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (ATLAS-2460) Data Migration: Moving Data from Earlier Versions of Atlas to be Migrated to Most Recent

[Ashutosh Mestry (JIRA)]

Ashutosh Mestry created ATLAS-2460:
--

 Summary: Data Migration: Moving Data from Earlier Versions of 
Atlas to be Migrated to Most Recent
 Key: ATLAS-2460
 URL: https://issues.apache.org/jira/browse/ATLAS-2460
 Project: Atlas
  Issue Type: Improvement
  Components:  atlas-core
Affects Versions: trunk
Reporter: Ashutosh Mestry
Assignee: Ashutosh Mestry


*Background*

The most recent version of Atlas uses _JanusGraph_ as database. Earlier 
versions used to use _Titan v0.5.4_. The formats used for storing data have 
changed.

The current version of Atlas also implement features within Atlas entities. 
This makes the storages structures differ.

A data migration approach thus becomes necessary to address the format 
incompatibilities.

*Approach***

Earlier version of Atlas could use _Export_ process to extract data out of the 
_Titan_ database. This ZIP file would then be moved to a cluster with newer 
version of Atlas installed. An _Import_ process on the new cluster would update 
the new cluster with data migrated to the new format.

It should be possible for _Data Migration_ to be initiated from Ambari, so that 
it becomes part of the Ambari's upgrade process.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Add me as contributor ...

[Madhan Neethiraj]

Bosco,

Thanks for your interest in contributing to Apache Atlas. You have been added 
as a contributor. Welcome to Apache Atlas community!

Regards,
Madhan




On 2/23/18, 11:06 AM, "Don Bosco Durai"  wrote:

Can one of the committer please add me as contributor to the Atlas project?

 

Thanks

 

Bosco

 

Add me as contributor ...

[Don Bosco Durai]

Can one of the committer please add me as contributor to the Atlas project?

 

Thanks

 

Bosco

 

Re: Please add me as a contributor

[Madhan Neethiraj]

Pierre,

Thanks for your interest in contributing to Apache Atlas. You have been added 
as a contributor. Welcome to Apache Atlas community!

Regards,
Madhan




On 2/23/18, 5:40 AM, "Pierre Padovani"  
wrote:

I'd like to be added as a contributor please.

My GitHub username is: ppadovani

Thanks!

Pierre

[jira] [Commented] (ATLAS-2456) Implement tag propagation using relationships

[Madhan Neethiraj (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-2456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374660#comment-16374660
 ] 

Madhan Neethiraj commented on ATLAS-2456:
-

Committed following additional patches to master: 
 - http://git-wip-us.apache.org/repos/asf/atlas/commit/5e496e5e
 - http://git-wip-us.apache.org/repos/asf/atlas/commit/bf4bf7fe

> Implement tag propagation using relationships
> -
>
> Key: ATLAS-2456
> URL: https://issues.apache.org/jira/browse/ATLAS-2456
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 1.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: ATLAS-2456-6.patch, TAG PROPAGATION IN ATLAS v.1.pdf, 
> tag_propagation_rest_api
>
>
> {color:#172b4d}Scalable way to quickly and efficiently propagate tags for 
> efficient searches and tag based security. Likewise tags for derivative 
> dataset should be inherited from the parent. For example, if an entity is 
> tagged "PII" then resulting entity created from a CTAS operation should also 
> be tagged "secret" to maintain the classification of the parent. In the case 
> where 2 or more datasets are aggregated the derivative dataset should be a 
> union of all parent tags.{color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (ATLAS-2459) Authorization enhancements to support instance level access controls

[Nixon Rodrigues (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-2459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nixon Rodrigues updated ATLAS-2459:
---
Attachment: ATLAS-2459.patch

> Authorization enhancements to support instance level access controls
> 
>
> Key: ATLAS-2459
> URL: https://issues.apache.org/jira/browse/ATLAS-2459
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Reporter: Srikanth Venkat
>Assignee: Nixon Rodrigues
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: ATLAS-2459.patch
>
>
> Current authorization model in Atlas supports policies at the level of 
> operations - like which users/groups can create/update/delete of 
> types/classifications/entities.
> This model needs to be enhanced/replaced to support more finer level of 
> control - for example:
>  - control operations on entity instances based on 
> entity-type/entity-classification/entity-id
>  -- users/groups allowed to create/update/delete entities, add/update/remove 
> classifications on entities, what classifications users are allowed to 
> add/update/remove, etc
>  - control operations on types based on type-name/type-category
>  -- users/groups allowed to create/update/delete classifications, 
> relationships, etc
>  - control access to admin operations - like import/export



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Please add me as a contributor

[Pierre Padovani]

I'd like to be added as a contributor please.

My GitHub username is: ppadovani

Thanks!

Pierre

Review Request 65770: ATLAS-2459:-Authorization enhancements to support instance level access controls

[Nixon Rodrigues]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65770/
---

Review request for atlas, Apoorv Naik, Ashutosh Mestry, and Madhan Neethiraj.


Summary (updated)
-

ATLAS-2459:-Authorization enhancements to support instance level access controls


Bugs: ATLAS-2459
https://issues.apache.org/jira/browse/ATLAS-2459


Repository: atlas


Description (updated)
---

The patch contains changes for Fine grained authorization at Instance level.
Earlier implementation of authorization at API level is removed.

https://issues.apache.org/jira/browse/ATLAS-2459


Diffs (updated)
-

  authorization/pom.xml f210a2f8 
  
authorization/src/main/java/org/apache/atlas/authorize/AtlasAccessRequest.java 
07cb2b07 
  authorization/src/main/java/org/apache/atlas/authorize/AtlasActionTypes.java 
c5969db1 
  authorization/src/main/java/org/apache/atlas/authorize/AtlasAuthorizer.java 
d64c6923 
  
authorization/src/main/java/org/apache/atlas/authorize/AtlasEntityAccessRequest.java
 PRE-CREATION 
  
authorization/src/main/java/org/apache/atlas/authorize/AtlasEntityAttribute.java
 PRE-CREATION 
  authorization/src/main/java/org/apache/atlas/authorize/AtlasPrivilege.java 
PRE-CREATION 
  
authorization/src/main/java/org/apache/atlas/authorize/AtlasResourceTypes.java 
7e2808cb 
  
authorization/src/main/java/org/apache/atlas/authorize/simple/AtlasAuthorizationUtils.java
 5bc19414 
  
authorization/src/main/java/org/apache/atlas/authorize/simple/AtlasSimpleAuthzPolicy.java
 PRE-CREATION 
  
authorization/src/main/java/org/apache/atlas/authorize/simple/FileReaderUtil.java
 88bf56b1 
  authorization/src/main/java/org/apache/atlas/authorize/simple/PolicyDef.java 
6b2b8b30 
  
authorization/src/main/java/org/apache/atlas/authorize/simple/PolicyParser.java 
f61bbf76 
  authorization/src/main/java/org/apache/atlas/authorize/simple/PolicyUtil.java 
9c08e405 
  
authorization/src/main/java/org/apache/atlas/authorize/simple/SimpleAtlasAuthorizer.java
 2eb0cd50 
  
authorization/src/test/java/org/apache/atlas/authorize/simple/AtlasAuthorizationUtilsTest.java
 adebb627 
  
authorization/src/test/java/org/apache/atlas/authorize/simple/PolicyParserTest.java
 3b7869aa 
  
authorization/src/test/java/org/apache/atlas/authorize/simple/PolicyUtilTest.java
 1cefbcdf 
  
authorization/src/test/java/org/apache/atlas/authorize/simple/SimpleAtlasAuthorizerTest.java
 b36c9c75 
  distro/src/conf/atlas-simple-authz-policy.json PRE-CREATION 
  intg/src/main/java/org/apache/atlas/AtlasErrorCode.java ff09e6c9 
  intg/src/main/java/org/apache/atlas/model/instance/AtlasEntityHeader.java 
340cd05c 
  intg/src/main/java/org/apache/atlas/utils/AtlasJson.java 4f7b716e 
  repository/pom.xml 87fe7fde 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasClassificationDefStoreV1.java
 8214cea6 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityDefStoreV1.java
 1d784ef8 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java
 ca0eeeb6 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasStructDefStoreV1.java
 83a6d1d6 
  
webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthorizationFilter.java 
8a29bb3f 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
86485fba 
  webapp/src/main/java/org/apache/atlas/web/rest/TypesREST.java 7177ac74 


Diff: https://reviews.apache.org/r/65770/diff/1/


Testing (updated)
---

maven build is gone fine.
Testing of endpoints is in progress.


Thanks,

Nixon Rodrigues

[jira] [Updated] (ATLAS-2459) Authorization enhancements to support instance level access controls

[Nixon Rodrigues (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-2459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nixon Rodrigues updated ATLAS-2459:
---
Fix Version/s: 1.0.0

> Authorization enhancements to support instance level access controls
> 
>
> Key: ATLAS-2459
> URL: https://issues.apache.org/jira/browse/ATLAS-2459
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Reporter: Srikanth Venkat
>Assignee: Nixon Rodrigues
>Priority: Major
> Fix For: 1.0.0
>
>
> Current authorization model in Atlas supports policies at the level of 
> operations - like which users/groups can create/update/delete of 
> types/classifications/entities.
> This model needs to be enhanced/replaced to support more finer level of 
> control - for example:
>  - control operations on entity instances based on 
> entity-type/entity-classification/entity-id
>  -- users/groups allowed to create/update/delete entities, add/update/remove 
> classifications on entities, what classifications users are allowed to 
> add/update/remove, etc
>  - control operations on types based on type-name/type-category
>  -- users/groups allowed to create/update/delete classifications, 
> relationships, etc
>  - control access to admin operations - like import/export



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ATLAS-2270) Supported combinations of persistent store and index backend

[David Radley (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-2270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374297#comment-16374297
 ] 

David Radley commented on ATLAS-2270:
-

Hi Pierre,
Great; you ask to be a contributer on the dev list - this allows you to 
assign a Jira to yourself.

Then you create a patch in git and attach it to the Jira and put it into 
the review board for review. There may be review comments or a "ship it". 
After that a committer (probably me in this case)  should take your patch 
and push it to master. 
 
I was new to git and have created a wiki page with the git commands I use 
https://cwiki.apache.org/confluence/display/ATLAS/Using+Git+with+Atlas. 
You may find this useful if you have not used git too much.
There is a also a wiki page for committers 
https://cwiki.apache.org/confluence/display/ATLAS/Developer+Resources you 
may be interested in . 

I suggest that the Jira should be scoped just to the fix you want to put 
in. You could change the words of 2270 or create a new one, 
   all the best, David. 




From:   "Pierre Padovani (JIRA)" 
To: david...@apache.org
Date:   22/02/2018 16:44
Subject:[jira] [Commented] (ATLAS-2270) Supported combinations of 
persistent store and index backend




[ 
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_ATLAS-2D2270-3Fpage-3Dcom.atlassian.jira.plugin.system.issuetabpanels-3Acomment-2Dtabpanel-26focusedCommentId-3D16373048-23comment-2D16373048=DwICaQ=jf_iaSHvJObTbx-siA1ZOg=QhpUQPr5YlG95aAgCvZGStEXHg4hBbSYQ9JkRqR_svY=XQ5cRTBSZfDIanJGha8zM6Dkgnw0IjOYn8x5TlzRFBc=CvAzKNiTZy_yOqiqGZadB1tR74aieY28pWb2P8q3ucI=
 
] 

Pierre Padovani commented on ATLAS-2270:


[~davidrad] I'd be happy to be a contributor and get this in. How does one 
go about becoming one? Are there a set of guidelines somewhere specific to 
this project?

We've been running the Cassandra + ES 5.x flavor of Atlas as both a self 
contained docker container for dev purposes, as well as a full blown HA 
cluster for at least the last two months. So I think it looks like a 
pretty stable setup.

https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_ATLAS-2D2270=DwICaQ=jf_iaSHvJObTbx-siA1ZOg=QhpUQPr5YlG95aAgCvZGStEXHg4hBbSYQ9JkRqR_svY=XQ5cRTBSZfDIanJGha8zM6Dkgnw0IjOYn8x5TlzRFBc=r6UWXjJszzenyZwhDWtpds09zR_aEaPTnA245uXb-vU=

indexing backend Atlas 1.0.0 (master) should support. This includes 
building/running Atlas as a standalone package and running UTs/ITs as part 
of the Atlas build. 
databases that will be supported in master/1.0.0. This JIRA deliberately 
ignores titan1 and janusgraph 0.1.1 as the former should be 
deprecated/removed and the other is a transient state as we get to 
janusgraph 0.2.0. 
combinations of persistent store and indexer. It is suggested that this 
set is kept unchanged:
additional combinations. Cassandra is included in this discussion pending 
response to ATLAS-2259.
continued and the remaining 4 combinations, marked with '?', should be 
considered. There seems to be evidence of people using all 4 of these 
combinations, although not necessarily with Atlas.
possible to build Atlas as a standalone package with any of the 
combinations - i.e. that they are mutually exclusive and do not interfere 
with one another. They currently interfere which makes it impossible to 
build Atlas with -Pdist,berkeley-elasticsearch because the 'dist' profile 
will exclude jars that are needed by the berkeley-elasticsearch profile - 
which leads to class not found exceptions when the Atlas server is 
started. The solution to this could be very simple, or slightly more 
sophisticated, depending on how many of the combinations we choose to 
support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)




Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


> Supported combinations of persistent store and index backend
> 
>
> Key: ATLAS-2270
> URL: https://issues.apache.org/jira/browse/ATLAS-2270
> Project: Atlas
>  Issue Type: Bug
>Reporter: Graham Wallis
>Priority: Major
>
> We need to discuss and decide which combinations of persistent store and 
> indexing backend Atlas 1.0.0 (master) should support. This includes 
> building/running Atlas as a standalone package and running UTs/ITs as part of 
> the Atlas build. 
> This JIRA focusses on titan0 and janusgraph 0.2.0, as they are the graph 
> databases that will be supported in master/1.0.0. This JIRA deliberately 
> ignores titan1 and janusgraph 0.1.1 as the former should be 
> deprecated/removed and the other is a transient state as we get to janusgraph 
> 

Re: [jira] [Commented] (ATLAS-2270) Supported combinations of persistent store and index backend

[David Radley]

Hi Pierre,
Great; you ask to be a contributer on the dev list - this allows you to 
assign a Jira to yourself.

Then you create a patch in git and attach it to the Jira and put it into 
the review board for review. There may be review comments or a "ship it". 
After that a committer (probably me in this case)  should take your patch 
and push it to master. 
 
I was new to git and have created a wiki page with the git commands I use 
https://cwiki.apache.org/confluence/display/ATLAS/Using+Git+with+Atlas. 
You may find this useful if you have not used git too much.
There is a also a wiki page for committers 
https://cwiki.apache.org/confluence/display/ATLAS/Developer+Resources you 
may be interested in . 

I suggest that the Jira should be scoped just to the fix you want to put 
in. You could change the words of 2270 or create a new one, 
   all the best, David. 




From:   "Pierre Padovani (JIRA)" 
To: david...@apache.org
Date:   22/02/2018 16:44
Subject:[jira] [Commented] (ATLAS-2270) Supported combinations of 
persistent store and index backend




[ 
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_ATLAS-2D2270-3Fpage-3Dcom.atlassian.jira.plugin.system.issuetabpanels-3Acomment-2Dtabpanel-26focusedCommentId-3D16373048-23comment-2D16373048=DwICaQ=jf_iaSHvJObTbx-siA1ZOg=QhpUQPr5YlG95aAgCvZGStEXHg4hBbSYQ9JkRqR_svY=XQ5cRTBSZfDIanJGha8zM6Dkgnw0IjOYn8x5TlzRFBc=CvAzKNiTZy_yOqiqGZadB1tR74aieY28pWb2P8q3ucI=
 
] 

Pierre Padovani commented on ATLAS-2270:


[~davidrad] I'd be happy to be a contributor and get this in. How does one 
go about becoming one? Are there a set of guidelines somewhere specific to 
this project?

We've been running the Cassandra + ES 5.x flavor of Atlas as both a self 
contained docker container for dev purposes, as well as a full blown HA 
cluster for at least the last two months. So I think it looks like a 
pretty stable setup.

> Supported combinations of persistent store and index backend
> 
>
> Key: ATLAS-2270
> URL: 
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_ATLAS-2D2270=DwICaQ=jf_iaSHvJObTbx-siA1ZOg=QhpUQPr5YlG95aAgCvZGStEXHg4hBbSYQ9JkRqR_svY=XQ5cRTBSZfDIanJGha8zM6Dkgnw0IjOYn8x5TlzRFBc=r6UWXjJszzenyZwhDWtpds09zR_aEaPTnA245uXb-vU=

> Project: Atlas
>  Issue Type: Bug
>Reporter: Graham Wallis
>Priority: Major
>
> We need to discuss and decide which combinations of persistent store and 
indexing backend Atlas 1.0.0 (master) should support. This includes 
building/running Atlas as a standalone package and running UTs/ITs as part 
of the Atlas build. 
> This JIRA focusses on titan0 and janusgraph 0.2.0, as they are the graph 
databases that will be supported in master/1.0.0. This JIRA deliberately 
ignores titan1 and janusgraph 0.1.1 as the former should be 
deprecated/removed and the other is a transient state as we get to 
janusgraph 0.2.0. 
> With titan0 as the graph provider, Atlas has supported the following 
combinations of persistent store and indexer. It is suggested that this 
set is kept unchanged:
> {{
> titan0  solr  es
> 
> berkeley   0  1
> hbase   1  0
> cassandra0  0
> }}
> With janusgraph (0.2.0) as the graph provider, Atlas *could* support 
additional combinations. Cassandra is included in this discussion pending 
response to ATLAS-2259.
> {{
> janus 0.2.0  solr  es
> 
> berkeley   ?  1
> hbase   1  ?
> cassandra?  ?
> }}
> It is suggested that the combinations marked with '1' should be 
continued and the remaining 4 combinations, marked with '?', should be 
considered. There seems to be evidence of people using all 4 of these 
combinations, although not necessarily with Atlas.
> Depending on the decision made above, we need to ensure that it is 
possible to build Atlas as a standalone package with any of the 
combinations - i.e. that they are mutually exclusive and do not interfere 
with one another. They currently interfere which makes it impossible to 
build Atlas with -Pdist,berkeley-elasticsearch because the 'dist' profile 
will exclude jars that are needed by the berkeley-elasticsearch profile - 
which leads to class not found exceptions when the Atlas server is 
started. The solution to this could be very simple, or slightly more 
sophisticated, depending on how many of the combinations we choose to 
support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)




Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 

[jira] [Commented] (ATLAS-2456) Implement tag propagation using relationships

[Nixon Rodrigues (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-2456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374074#comment-16374074
 ] 

Nixon Rodrigues commented on ATLAS-2456:


+1 for ATLAS-2456-6.patch, the changes LGTM.

Thanks [~madhan.neethiraj] for the patch.

> Implement tag propagation using relationships
> -
>
> Key: ATLAS-2456
> URL: https://issues.apache.org/jira/browse/ATLAS-2456
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 1.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: ATLAS-2456-6.patch, TAG PROPAGATION IN ATLAS v.1.pdf, 
> tag_propagation_rest_api
>
>
> {color:#172b4d}Scalable way to quickly and efficiently propagate tags for 
> efficient searches and tag based security. Likewise tags for derivative 
> dataset should be inherited from the parent. For example, if an entity is 
> tagged "PII" then resulting entity created from a CTAS operation should also 
> be tagged "secret" to maintain the classification of the parent. In the case 
> where 2 or more datasets are aggregated the derivative dataset should be a 
> union of all parent tags.{color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ATLAS-2456) Implement tag propagation using relationships

[Madhan Neethiraj (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-2456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374070#comment-16374070
 ] 

Madhan Neethiraj commented on ATLAS-2456:
-

[~sarath.ku...@gmail.com] - added a fix about entity-notification 
deserialization to support V2 notification message. Please review.

> Implement tag propagation using relationships
> -
>
> Key: ATLAS-2456
> URL: https://issues.apache.org/jira/browse/ATLAS-2456
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 1.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: ATLAS-2456-6.patch, TAG PROPAGATION IN ATLAS v.1.pdf, 
> tag_propagation_rest_api
>
>
> {color:#172b4d}Scalable way to quickly and efficiently propagate tags for 
> efficient searches and tag based security. Likewise tags for derivative 
> dataset should be inherited from the parent. For example, if an entity is 
> tagged "PII" then resulting entity created from a CTAS operation should also 
> be tagged "secret" to maintain the classification of the parent. In the case 
> where 2 or more datasets are aggregated the derivative dataset should be a 
> union of all parent tags.{color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (ATLAS-2456) Implement tag propagation using relationships

[Madhan Neethiraj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-2456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-2456:

Attachment: ATLAS-2456-6.patch

> Implement tag propagation using relationships
> -
>
> Key: ATLAS-2456
> URL: https://issues.apache.org/jira/browse/ATLAS-2456
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 1.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: ATLAS-2456-6.patch, TAG PROPAGATION IN ATLAS v.1.pdf, 
> tag_propagation_rest_api
>
>
> {color:#172b4d}Scalable way to quickly and efficiently propagate tags for 
> efficient searches and tag based security. Likewise tags for derivative 
> dataset should be inherited from the parent. For example, if an entity is 
> tagged "PII" then resulting entity created from a CTAS operation should also 
> be tagged "secret" to maintain the classification of the parent. In the case 
> where 2 or more datasets are aggregated the derivative dataset should be a 
> union of all parent tags.{color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)