AtlasClientV2 and Spring boot

[Pierre Padovani]

Has anyone managed to find a way to get the Atlas client to function within
a spring boot container that is running Jersey 2? I.e. Has someone gotten
it shaded in a way that works?

I've tried two approaches:

1) Updated the atlas client code to Jersey 2. The issue here is that for
large payloads you will randomly get a low level socket reset exception on
POST. We have not been able to track down exactly what the issue is.

2) Shade the Atlas client and relocate the jersey 1 code. Most of the calls
seem to work with this approach, except we cannot seem to create
relationships. In our case, it would be the old style of updating both
sides with a reference to each entity. The same test case passes if we use
our version of the client from #1 above.

Any ideas would be greatly appreciated.

Thanks!

Pierre Padovani

Re: Please Add me as contributor

[Madhan Neethiraj]

Shrinivas,

 

You have been added as a contributor in Apache Atlas. Welcome to Apache Atlas 
community!

 

Regards,

Madhan

 

 

 

From: Madhan Neethiraj 
Date: Monday, February 26, 2018 at 2:20 PM
To: "dev@atlas.apache.org" 
Subject: Re: Please Add me as contributor

 

Shrinivas,

 

Thanks for your interest in contributing to Apache Atlas. Can you please send 
your email-address/id registered in JIRA? 

 

Regards,

Madhan

 

 

 

From: Shrinivas Kane 
Reply-To: "dev@atlas.apache.org" 
Date: Monday, February 26, 2018 at 1:22 PM
To: "dev@atlas.apache.org" 
Subject: Please Add me as contributor

 




Regards,

Shrinivas Kane
Software Developer, IBM InformationServer - Metadata Infrastructure
IBM Analytics, Platform

 

Phone: +1 978-899-4212 | Tie-Line: 276-4212
E-mail: ka...@us.ibm.com

550 King St
Littleton, MA 01460-1250
United States

 

Re: Review Request 65779: Data Migration: Moving Data to New Version of Atlas

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65779/
---

(Updated Feb. 27, 2018, 7:57 p.m.)


Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Ruchi Solani.


Changes
---

Updates include:
- Re-enabled disabled unit tests from _ImportService_.
- Fixed indvertently introduced bug related to relationship mapping.


Bugs: ATLAS-2460
https://issues.apache.org/jira/browse/ATLAS-2460


Repository: atlas


Description
---

**Background**
This implementation deals with the 'import into' part of the data migration 
process. 

It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.

**Implementation**

During _Atlas_ server startup, the configuration parameter is checked, if that 
parameter exists, all services except _DataMigrationService_ is started. 
Migration is started. Atlas server is available in _MIGRATION_ mode. It 
processes REST calls made only to the _AdminResource_.

Here's are the udpates:
- New configuration parameter has been added:
_atlas.migration.mode.filename=_
  This configuration parameter is set by Ambari as part of its migration 
orchestration. 
- _DataMigrationService_: New service that performs async migration as soon as 
Atlas server starts up.
- _ImportProgressTracker_: Added to be able to query progress of import.
- _ImportService_: Modified to send updates to _ImportProgressTracker_.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status 
is set by looking at the configuration parameter above.
- _AtlasEntityStoreV1_ modified to block sending out notifications during 
migration.
- _Services_ modified for special handling of _DataMigrationService_.


**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H 
"Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3 
  common/src/main/java/org/apache/atlas/service/Services.java 1267dc92 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 
1ea961d8 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportResult.java 
b97cbb3f 
  pom.xml 50476262 
  repository/pom.xml b1d6b1f9 
  
repository/src/main/java/org/apache/atlas/repository/impexp/DataMigrationService.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/impexp/DirectorySource.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ImportProgressTracker.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 
c976c593 
  repository/src/main/java/org/apache/atlas/repository/impexp/ImportSource.java 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 
ecfbd242 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java
 b5461d4b 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/BulkImporterImpl.java
 af310954 
  
repository/src/test/java/org/apache/atlas/repository/impexp/DirectorySourceTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java
 6fb6b454 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
 080a96ac 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java
 d8e916d7 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java
 8257faa1 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java
 ac35860d 
  repository/src/test/resources/stocks-2.zip PRE-CREATION 
  
repository/src/test/resources/stocks-2/4e13b36b-9c54-4616-9001-1058221165d0.json
 PRE-CREATION 
  
repository/src/test/resources/stocks-2/5cfc2540-9947-40e0-8905-367e07481774.json
 PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION 
  webapp/src/main/java/org/apache/atlas/Atlas.java 7cf6e3ea 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
1b3f2c86 
  webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c 
  webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 
aab2bb8f 


Diff: https://reviews.apache.org/r/65779/diff/6/

Changes: https://reviews.apache.org/r/65779/diff/5-6/


Testing
---

**Unit tests**
Unit tests for _DirectorySource_ added.

**Functional tests**
Steps to test:
- Place the exported file say _export-1.zip_ at _/root_
- Add to 

Re: Review Request 65813: ATLAS-2462 Sqoop import for all tables throws NPE for no table provided in command

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65813/#review198340
---




addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java
Lines 205 (patched)


This would result in invalid process entity, with:
 - empty 'outputs' (line #206)
 - empty 'inputs' (line #208)

If Sqoop hook is not provided with details of the Hive table involved, I 
think the hook should create lineage between SqoopDataStore & HiveDatabase.



addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java
Lines 236 (patched)


add 'clusterName'.


- Madhan Neethiraj


On Feb. 27, 2018, 11:24 a.m., Ruchi Solani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65813/
> ---
> 
> (Updated Feb. 27, 2018, 11:24 a.m.)
> 
> 
> Review request for atlas and Madhan Neethiraj.
> 
> 
> Bugs: ATLAS-2462
> https://issues.apache.org/jira/browse/ATLAS-2462
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> When tried importing a specific table with below command there is no error:
> 
> sqoop import --connect jdbc:mysql://localhost/test --table testtable221 
> --hive-import --hive-database sqoopimport023 --hive-table hivetesttable221 
> --username root -P -m 1 --fetch-size 1
> 
> When tried to import all tables from a specific database using below command, 
> got NP error:
> 
> sqoop import-all-tables --connect jdbc:mysql://localhost/test --hive-import 
> --hive-database sqoopimport022 --username root -P -m 1 --fetch-size 1
> 
> 
> Diffs
> -
> 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> 06a0e9c 
>   intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 2f870dc 
> 
> 
> Diff: https://reviews.apache.org/r/65813/diff/2/
> 
> 
> Testing
> ---
> 
> sanity testing done.
> 
> 
> Thanks,
> 
> Ruchi Solani
> 
>

[jira] [Updated] (ATLAS-2456) Implement tag propagation using relationships

[Madhan Neethiraj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-2456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-2456:

Issue Type: Sub-task  (was: New Feature)
Parent: ATLAS-1821

> Implement tag propagation using relationships
> -
>
> Key: ATLAS-2456
> URL: https://issues.apache.org/jira/browse/ATLAS-2456
> Project: Atlas
>  Issue Type: Sub-task
>  Components:  atlas-core
>Affects Versions: 1.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: ATLAS-2456-6.patch, ATLAS-2456-7.patch, TAG PROPAGATION 
> IN ATLAS v.1.pdf, tag_propagation_rest_api
>
>
> {color:#172b4d}Scalable way to quickly and efficiently propagate tags for 
> efficient searches and tag based security. Likewise tags for derivative 
> dataset should be inherited from the parent. For example, if an entity is 
> tagged "PII" then resulting entity created from a CTAS operation should also 
> be tagged "secret" to maintain the classification of the parent. In the case 
> where 2 or more datasets are aggregated the derivative dataset should be a 
> union of all parent tags.{color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (ATLAS-2456) Implement tag propagation using relationships

[Madhan Neethiraj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-2456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-2456:

Reporter: Srikanth Venkat  (was: Sarath Subramanian)

> Implement tag propagation using relationships
> -
>
> Key: ATLAS-2456
> URL: https://issues.apache.org/jira/browse/ATLAS-2456
> Project: Atlas
>  Issue Type: Sub-task
>  Components:  atlas-core
>Affects Versions: 1.0.0
>Reporter: Srikanth Venkat
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: ATLAS-2456-6.patch, ATLAS-2456-7.patch, TAG PROPAGATION 
> IN ATLAS v.1.pdf, tag_propagation_rest_api
>
>
> {color:#172b4d}Scalable way to quickly and efficiently propagate tags for 
> efficient searches and tag based security. Likewise tags for derivative 
> dataset should be inherited from the parent. For example, if an entity is 
> tagged "PII" then resulting entity created from a CTAS operation should also 
> be tagged "secret" to maintain the classification of the parent. In the case 
> where 2 or more datasets are aggregated the derivative dataset should be a 
> union of all parent tags.{color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: please add me as a contributor

[Madhan Neethiraj]

Ruxandra Rosu,

Thanks for your interest in contributing to Apache Atlas. You have been added 
as a contributor. Welcome to Apache Atlas community!

Regards,
Madhan




On 2/27/18, 8:49 AM, "Rosu, R.G. (Ruxandra - Gabriela)" 
 wrote:

Hello All,

Please add me as a contributor to Apache Atlas project. My username and 
email are ruxandra.rosu and 
ruxandra-gabriela.r...@ing.com.
I would like to be able to assign jiras to myself and others and be able to 
attach patches to jiras.

Thank you,
Ruxandra Rosu


---
ATTENTION:
The information in this electronic mail message is private and
confidential, and only intended for the addressee. 

The information in this e-mail does not constitute 
Investment Advice as defined in Art 4 of the Markets
in Financial Instruments Directive 2004/39/EC. The Markets
in Financial Instruments Directive 2004/39/EC is not applicable to 
Life insurance and Private pensions products.

Should you receive this message by mistake, you are hereby 
notified that any disclosure, reproduction, distribution or use of this
message is strictly prohibited. Please inform the sender by
reply transmission and delete the message without copying or
opening it.

Messages and attachments are scanned for all viruses known.
If this message contains password-protected attachments, the
files have NOT been scanned for viruses by the ING mail domain.
Always scan attachments before opening them.

---

please add me as a contributor

[Rosu, R.G. (Ruxandra - Gabriela)]

Hello All,

Please add me as a contributor to Apache Atlas project. My username and email 
are ruxandra.rosu and 
ruxandra-gabriela.r...@ing.com.
I would like to be able to assign jiras to myself and others and be able to 
attach patches to jiras.

Thank you,
Ruxandra Rosu

---
ATTENTION:
The information in this electronic mail message is private and
confidential, and only intended for the addressee.

The information in this e-mail does not constitute
Investment Advice as defined in Art 4 of the Markets
in Financial Instruments Directive 2004/39/EC. The Markets
in Financial Instruments Directive 2004/39/EC is not applicable to
Life insurance and Private pensions products.

Should you receive this message by mistake, you are hereby
notified that any disclosure, reproduction, distribution or use of this
message is strictly prohibited. Please inform the sender by
reply transmission and delete the message without copying or
opening it.

Messages and attachments are scanned for all viruses known.
If this message contains password-protected attachments, the
files have NOT been scanned for viruses by the ING mail domain.
Always scan attachments before opening them.
---

[jira] [Assigned] (ATLAS-2464) Information View OMAS Listener

[Maryna Strelchuk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-2464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maryna Strelchuk reassigned ATLAS-2464:
---

Assignee: Maryna Strelchuk

> Information View OMAS Listener
> --
>
> Key: ATLAS-2464
> URL: https://issues.apache.org/jira/browse/ATLAS-2464
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Maryna Strelchuk
>Assignee: Maryna Strelchuk
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (ATLAS-2464) Information View OMAS Listener

[Maryna Strelchuk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-2464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maryna Strelchuk reassigned ATLAS-2464:
---

Assignee: (was: Maryna Strelchuk)

> Information View OMAS Listener
> --
>
> Key: ATLAS-2464
> URL: https://issues.apache.org/jira/browse/ATLAS-2464
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Maryna Strelchuk
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (ATLAS-2464) Information View OMAS Listener

[Maryna Strelchuk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-2464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maryna Strelchuk reassigned ATLAS-2464:
---

Assignee: Maryna Strelchuk

> Information View OMAS Listener
> --
>
> Key: ATLAS-2464
> URL: https://issues.apache.org/jira/browse/ATLAS-2464
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Maryna Strelchuk
>Assignee: Maryna Strelchuk
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (ATLAS-2464) Information View OMAS Listener

[Maryna Strelchuk (JIRA)]

Maryna Strelchuk created ATLAS-2464:
---

 Summary: Information View OMAS Listener
 Key: ATLAS-2464
 URL: https://issues.apache.org/jira/browse/ATLAS-2464
 Project: Atlas
  Issue Type: Sub-task
Reporter: Maryna Strelchuk






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (ATLAS-1711) Information View OMAS

[Maryna Strelchuk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maryna Strelchuk reassigned ATLAS-1711:
---

Assignee: Maryna Strelchuk

> Information View OMAS
> -
>
> Key: ATLAS-1711
> URL: https://issues.apache.org/jira/browse/ATLAS-1711
> Project: Atlas
>  Issue Type: New Feature
>Reporter: Maryna Strelchuk
>Assignee: Maryna Strelchuk
>Priority: Major
>  Labels: VirtualDataConnector
>
> Information View OMAS manages the creation and maintenance of View metadata. 
> and is called by the Virtualizer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (ATLAS-2463) Documentation for Atlas authorization at instance level

[Nixon Rodrigues (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-2463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nixon Rodrigues updated ATLAS-2463:
---
Attachment: ATLAS-2463.patch

> Documentation for Atlas authorization at instance level
> ---
>
> Key: ATLAS-2463
> URL: https://issues.apache.org/jira/browse/ATLAS-2463
> Project: Atlas
>  Issue Type: Bug
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: ATLAS-2463.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (ATLAS-2463) Documentation for Atlas authorization at instance level

[Nixon Rodrigues (JIRA)]

Nixon Rodrigues created ATLAS-2463:
--

 Summary: Documentation for Atlas authorization at instance level
 Key: ATLAS-2463
 URL: https://issues.apache.org/jira/browse/ATLAS-2463
 Project: Atlas
  Issue Type: Bug
Reporter: Nixon Rodrigues
Assignee: Nixon Rodrigues
 Fix For: 1.0.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65813: ATLAS-2462 Sqoop import for all tables throws NPE for no table provided in command

[Ruchi Solani]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65813/
---

(Updated Feb. 27, 2018, 11:24 a.m.)


Review request for atlas and Madhan Neethiraj.


Bugs: ATLAS-2462
https://issues.apache.org/jira/browse/ATLAS-2462


Repository: atlas


Description
---

When tried importing a specific table with below command there is no error:

sqoop import --connect jdbc:mysql://localhost/test --table testtable221 
--hive-import --hive-database sqoopimport023 --hive-table hivetesttable221 
--username root -P -m 1 --fetch-size 1

When tried to import all tables from a specific database using below command, 
got NP error:

sqoop import-all-tables --connect jdbc:mysql://localhost/test --hive-import 
--hive-database sqoopimport022 --username root -P -m 1 --fetch-size 1


Diffs (updated)
-

  addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
06a0e9c 
  intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 2f870dc 


Diff: https://reviews.apache.org/r/65813/diff/2/

Changes: https://reviews.apache.org/r/65813/diff/1-2/


Testing
---

sanity testing done.


Thanks,

Ruchi Solani

Jenkins build is back to normal : Atlas-master-UnitTests #630

[Apache Jenkins Server]

See