date:20170524

Re: [VOTE] Graduate Apache Atlas Project from Incubator

2017-05-24 Thread Hemanth Yamijala

+1. 

From: Suma Shivaprasad 
Sent: Thursday, May 25, 2017 4:53 AM
To: dev@atlas.incubator.apache.org
Subject: [VOTE] Graduate Apache Atlas Project from Incubator

Dear Atlas Community Members,

Following the discussion in Atlas community on graduating Apache Atlas
Project from Incubation to a Top Level Project, starting this
voting thread within the Atlas community. I appreciate mentors and
community members in making this happen and thanks for their
support. Apache Atlas Project has come a long way since entering into
incubation during May 2015 and the project summary shows the
details on this.

Please vote on the Apache Atlas Project resolution that is found in
bottom of this email.

[ ] +1 Graduate Apache Atlas from the Incubator.
[ ] +0 No opinion
[ ] -1 Don't graduate Apache Atlas from the Incubator ( please provide
the reason)

This VOTE will be opened for next 72 hours.

If this VOTE succeeds, a similar VOTE will be done in general@incubator
mailing list. If that succeeds, a resolution will be
included in the next Apache Board Meeting.

Here is my Vote +1 (binding)

Thanks all for your participation
Suma

Project Summary:

=
http://incubator.apache.org/projects/atlas.html

Project website:
=

http://atlas.incubator.apache.org

Project Documentation:
===

http://atlas.incubator.apache.org/index.html
Architecture - http://atlas.incubator.apache.org/Architecture.html
Installation guide - http://atlas.incubator.apache.org/InstallationSteps.
html
QuickStart Guide- http://atlas.incubator.apache.org/QuickStart.html
REST API - http://atlas.incubator.apache.org/api/v2/index.html
Atlas Release Guide - https://cwiki.apache.org/confluence/display/ATLAS/
Developer+Resources

Project maturity Assessment:
===

https://cwiki.apache.org/confluence/display/ATLAS/
Apache+Atlas+Project+Maturity+Model

Proposed PMC size: 38

Total number of committers  : 38 members
Total number of contributors : 62 members

PMC affiliation (* indicated chair)

* Hortonworks (13)
   IBM (3)
   Aetna(4)
   Freestone (3)
   Merck (2)
   Oracle(2)
   Target (2)
   Schlumberger(2)
   Dataguise(1)
   Inmobi(1)
   JPMC(1)
   MPR(1)
   SAP(1)
   SparklineData(1)
   Others(1)

1660 commits on master
80 contributors across all branches
Dev list averaged ~500-800 msgs/month in 2017
User list was created recently in April 2017. Hence not considering stats
for this.
1827 issues created

Committer¹s affiliation:
===
* Hortonworks (13)
   IBM (3)
   Aetna(4)
   Freestone (3)
   Merck (2)
   Oracle(2)
   Target (2)
   Schlumberger(2)
   Dataguise(1)
   Inmobi(1)
   JPMC(1)
   MPR(1)
   SAP(1)
   SparklineData(1)
   Others(1)

Apache Atlas Top Level Project Resolution:

Establish the Apache Atlas Project

WHEREAS, the Board of Directors deems it to be in the best interests of
the Foundation and consistent with the Foundation¹s purpose to establish a
Project Management Committee charged with the creation and maintenance of
open-source software, for distribution at no charge to the public, related
to a data management platform That provides real-time, consistent access
to data-intensive applications throughout widely distributed cloud
architectures.

NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee
(PMC), to be known as the "Apache Atlas Project", be and hereby is
established pursuant to Bylaws of the Foundation; and be it further
RESOLVED,that the Apache Atlas Project be and hereby is responsible for
the creation and maintenance of software related to a data management
platform that provides real-time, consistent access to data-intensive
applications throughout widely distributed cloud architectures.

RESOLVED, that the office of "Vice President, Apache Atlas" be and
hereby is created, the person holding such office to serve at the
direction of the Board of Directors as the chair of the Apache Atlas
Project, and to have primary responsibility for management of the projects
within the scope of responsibility of the Apache Atlas Project; and be it
Further.

>RESOLVED,that the persons listed immediately below be and hereby are
>appointed to serve as the initial members of the Apache Atlas Project:

Aaron Dossett adoss...@target.com 
Andrew Ahn a...@apache.org 
Anil Varma ava...@slb.com 
Arun C. Murthy acmur...@apache.org 
Apoorv Naik apoorvn...@apache.org 
Ayub Pathan ayubpat...@apache.org 
Barbara Stortz bsto...@sap.com 
Chris Hyzer chy...@aetna.com 
Daniel Markwat dmark...@aetna.com 
Darshan Kumar darshanku...@apache.org 
Dave Kantor dkan...@apache.org 
David

Re: [VOTE] Graduate Apache Atlas Project from Incubator

2017-05-24 Thread Jean-Baptiste Onofré


+1

Regards
JB

On 05/25/2017 01:23 AM, Suma Shivaprasad wrote:

Dear Atlas Community Members,

Following the discussion in Atlas community on graduating Apache Atlas
Project from Incubation to a Top Level Project, starting this
voting thread within the Atlas community. I appreciate mentors and
community members in making this happen and thanks for their
support. Apache Atlas Project has come a long way since entering into
incubation during May 2015 and the project summary shows the
details on this.

Please vote on the Apache Atlas Project resolution that is found in
bottom of this email.

[ ] +1 Graduate Apache Atlas from the Incubator.
[ ] +0 No opinion
[ ] -1 Don't graduate Apache Atlas from the Incubator ( please provide
the reason)

This VOTE will be opened for next 72 hours.

If this VOTE succeeds, a similar VOTE will be done in general@incubator
mailing list. If that succeeds, a resolution will be
included in the next Apache Board Meeting.

Here is my Vote +1 (binding)

Thanks all for your participation
Suma


Project Summary:

=
http://incubator.apache.org/projects/atlas.html

Project website:
=

http://atlas.incubator.apache.org

Project Documentation:
===

http://atlas.incubator.apache.org/index.html
Architecture - http://atlas.incubator.apache.org/Architecture.html
Installation guide - http://atlas.incubator.apache.org/InstallationSteps.
html
QuickStart Guide- http://atlas.incubator.apache.org/QuickStart.html
REST API - http://atlas.incubator.apache.org/api/v2/index.html
Atlas Release Guide - https://cwiki.apache.org/confluence/display/ATLAS/
Developer+Resources

Project maturity Assessment:
===

https://cwiki.apache.org/confluence/display/ATLAS/
Apache+Atlas+Project+Maturity+Model

Proposed PMC size: 38

Total number of committers  : 38 members
Total number of contributors : 62 members

PMC affiliation (* indicated chair)

* Hortonworks (13)
IBM (3)
Aetna(4)
Freestone (3)
Merck (2)
Oracle(2)
Target (2)
Schlumberger(2)
Dataguise(1)
Inmobi(1)
JPMC(1)
MPR(1)
SAP(1)
SparklineData(1)
Others(1)

1660 commits on master
80 contributors across all branches
Dev list averaged ~500-800 msgs/month in 2017
User list was created recently in April 2017. Hence not considering stats
for this.
1827 issues created

Committer¹s affiliation:
===
* Hortonworks (13)
IBM (3)
Aetna(4)
Freestone (3)
Merck (2)
Oracle(2)
Target (2)
Schlumberger(2)
Dataguise(1)
Inmobi(1)
JPMC(1)
MPR(1)
SAP(1)
SparklineData(1)
Others(1)


Apache Atlas Top Level Project Resolution:


Establish the Apache Atlas Project

WHEREAS, the Board of Directors deems it to be in the best interests of
the Foundation and consistent with the Foundation¹s purpose to establish a
Project Management Committee charged with the creation and maintenance of
open-source software, for distribution at no charge to the public, related
to a data management platform That provides real-time, consistent access
to data-intensive applications throughout widely distributed cloud
architectures.

NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee
(PMC), to be known as the "Apache Atlas Project", be and hereby is
established pursuant to Bylaws of the Foundation; and be it further
RESOLVED,that the Apache Atlas Project be and hereby is responsible for
the creation and maintenance of software related to a data management
platform that provides real-time, consistent access to data-intensive
applications throughout widely distributed cloud architectures.

RESOLVED, that the office of "Vice President, Apache Atlas" be and
hereby is created, the person holding such office to serve at the
direction of the Board of Directors as the chair of the Apache Atlas
Project, and to have primary responsibility for management of the projects
within the scope of responsibility of the Apache Atlas Project; and be it
Further.


RESOLVED,that the persons listed immediately below be and hereby are
appointed to serve as the initial members of the Apache Atlas Project:


Aaron Dossett adoss...@target.com 
Andrew Ahn a...@apache.org 
Anil Varma ava...@slb.com 
Arun C. Murthy acmur...@apache.org 
Apoorv Naik apoorvn...@apache.org 
Ayub Pathan ayubpat...@apache.org 
Barbara Stortz bsto...@sap.com 
Chris Hyzer chy...@aetna.com 
Daniel Markwat dmark...@aetna.com 
Darshan Kumar darshanku...@apache.org 
Dave Kantor dkan...@apache.org 
David Kaspar dkas...@merck.com 
Dennis Fusaro dfus...@aetna.com 
Greg Senia gse...@aetna.com

Re: Review Request 56959: Import API: Added Support for Transforming Entities During Import

2017-05-24 Thread Madhan Neethiraj


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/#review176041
---




intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java
Line 50 (original), 55 (patched)


AtlasBaseException seems to be not thrown. Please review and remove it if 
not thrown.



intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java
Lines 92 (patched)


Given an instance of AtlasImportRequest will be serialized/deserialized 
during REST calls, values in 'options' map will not be typed - as this line 
expects the value of type AtlasImportTransforms. Please review and update to 
handle this.

I would suggest to keep the classes used in REST APIs to only have simple 
get/set. Adding any further methods/expectations might be troublesome.



intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java
Lines 100 (patched)


Consider removing this method; instead use AtlasType.fromJson().



intg/src/main/java/org/apache/atlas/model/impexp/ImportTransformerEnum.java
Lines 45 (patched)


The enum implementation looks too complex to pass around instances via REST 
API. Consider keeping the classes/enums passed via REST simple i.e. strictly 
data-only. Any functionality/constraits should be handled in the application.


- Madhan Neethiraj


On May 25, 2017, 5:16 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56959/
> ---
> 
> (Updated May 25, 2017, 5:16 a.m.)
> 
> 
> Review request for atlas and Madhan Neethiraj.
> 
> 
> Bugs: ATLAS-1825
> https://issues.apache.org/jira/browse/ATLAS-1825
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Please refer to 
> [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios 
> and solution approach.
> 
> Impact
> ==
> * New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This 
> holds a map of entity-type to attribute and the tranforms to be applied.
> * New class _AtlasImportTransformBuiler_ reads the requests specified in the 
> query string of the import API and creates a _AtlasImportTransfor_ structure.
> * _AtlasImportTransformBuiler_ has new classes that support transformations. 
> They are:
> * _replace_ Implements find and replace on attribute values.
> * _lowercase_ Converts attribute value to lower case. 
> * Additional classes that unit test these classes have been added.
> 
> Implementation Approach
> ===
> * _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the 
> transform every entity that is fetched from the source.
> * _Import API_ now accepts multi-part input. This allows for elegantly 
> specifying import parameters as well as the binary input from the file.
> * Please take note of _ImportTransformerEnum_. This is first case where Enum 
> has state variables.
> 
> CURL
> 
> 
> *Setup*
> Create file with these contents call it _importTransform.json_:
> ```javascript
> {
> "options": {},
> "importTransform": {
> "transforms": {
> "hive_table": {
> "qualifiedName": [
> "lowercase",
> "replace:@cl1:@cl2"
> ]
> }
> }
> }
> }
> ```
> 
> ```
> curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H 
> "Cache-Control: no-cache" -F request=@../docs/importTransform.json -F 
> data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import;
> ```
> 
> Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is 
> not required anymore.
> 
> Notice the output from the API, it has the _AtlasRequest_ that was passed as 
> input.
> 
> Documentation
> =
> Need to update TWIKI documents.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
> 7530196d 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransforms.java 
> PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/impexp/ImportTransformerEnum.java 
> PRE-CREATION 
>   
> intg/src/test/java/org/apache/atlas/model/impexp/ImportTransformerEnumTest.java
>  PRE-CREATION 
>   pom.xml a93b8ad9 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportEntityTransformer.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
>  9ec15e00 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 
>

Re: [VOTE] Graduate Apache Atlas Project from Incubator

2017-05-24 Thread Nixon Rodrigues

+1 to Graduate Apache Atlas project from incubator to main Apache project.

Salute to all the PMC, Committers & Contributors for their hard work.

Congrats to Atlas community as whole.

Nixon

On Thu, May 25, 2017 at 10:39 AM, Vimal Sharma 
wrote:

> +1 (binding)
> Huge cheers to the Atlas community!
>
> -Vimal
>
>
>
>
> On 5/25/17, 10:12 AM, "Chris Douglas"  wrote:
>
> >+1 (binding) -C
> >
> >On Wed, May 24, 2017 at 4:23 PM, Suma Shivaprasad
> > wrote:
> >> Dear Atlas Community Members,
> >>
> >> Following the discussion in Atlas community on graduating Apache Atlas
> >> Project from Incubation to a Top Level Project, starting this
> >> voting thread within the Atlas community. I appreciate mentors and
> >> community members in making this happen and thanks for their
> >> support. Apache Atlas Project has come a long way since entering into
> >> incubation during May 2015 and the project summary shows the
> >> details on this.
> >>
> >> Please vote on the Apache Atlas Project resolution that is found in
> >> bottom of this email.
> >>
> >> [ ] +1 Graduate Apache Atlas from the Incubator.
> >> [ ] +0 No opinion
> >> [ ] -1 Don't graduate Apache Atlas from the Incubator ( please provide
> >> the reason)
> >>
> >> This VOTE will be opened for next 72 hours.
> >>
> >> If this VOTE succeeds, a similar VOTE will be done in general@incubator
> >> mailing list. If that succeeds, a resolution will be
> >> included in the next Apache Board Meeting.
> >>
> >> Here is my Vote +1 (binding)
> >>
> >> Thanks all for your participation
> >> Suma
> >>
> >>
> >> Project Summary:
> >>
> >> =
> >> http://incubator.apache.org/projects/atlas.html
> >>
> >> Project website:
> >> =
> >>
> >> http://atlas.incubator.apache.org
> >>
> >> Project Documentation:
> >> ===
> >>
> >> http://atlas.incubator.apache.org/index.html
> >> Architecture - http://atlas.incubator.apache.org/Architecture.html
> >> Installation guide - http://atlas.incubator.apache.
> org/InstallationSteps.
> >> html
> >> QuickStart Guide- http://atlas.incubator.apache.org/QuickStart.html
> >> REST API - http://atlas.incubator.apache.org/api/v2/index.html
> >> Atlas Release Guide - https://cwiki.apache.org/
> confluence/display/ATLAS/
> >> Developer+Resources
> >>
> >> Project maturity Assessment:
> >> ===
> >>
> >> https://cwiki.apache.org/confluence/display/ATLAS/
> >> Apache+Atlas+Project+Maturity+Model
> >>
> >> Proposed PMC size: 38
> >>
> >> Total number of committers  : 38 members
> >> Total number of contributors : 62 members
> >>
> >> PMC affiliation (* indicated chair)
> >>
> >> * Hortonworks (13)
> >>IBM (3)
> >>Aetna(4)
> >>Freestone (3)
> >>Merck (2)
> >>Oracle(2)
> >>Target (2)
> >>Schlumberger(2)
> >>Dataguise(1)
> >>Inmobi(1)
> >>JPMC(1)
> >>MPR(1)
> >>SAP(1)
> >>SparklineData(1)
> >>Others(1)
> >>
> >> 1660 commits on master
> >> 80 contributors across all branches
> >> Dev list averaged ~500-800 msgs/month in 2017
> >> User list was created recently in April 2017. Hence not considering
> stats
> >> for this.
> >> 1827 issues created
> >>
> >> Committer¹s affiliation:
> >> ===
> >> * Hortonworks (13)
> >>IBM (3)
> >>Aetna(4)
> >>Freestone (3)
> >>Merck (2)
> >>Oracle(2)
> >>Target (2)
> >>Schlumberger(2)
> >>Dataguise(1)
> >>Inmobi(1)
> >>JPMC(1)
> >>MPR(1)
> >>SAP(1)
> >>SparklineData(1)
> >>Others(1)
> >>
> >>
> >> Apache Atlas Top Level Project Resolution:
> >> 
> >>
> >> Establish the Apache Atlas Project
> >>
> >> WHEREAS, the Board of Directors deems it to be in the best interests of
> >> the Foundation and consistent with the Foundation¹s purpose to
> establish a
> >> Project Management Committee charged with the creation and maintenance
> of
> >> open-source software, for distribution at no charge to the public,
> related
> >> to a data management platform That provides real-time, consistent access
> >> to data-intensive applications throughout widely distributed cloud
> >> architectures.
> >>
> >> NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee
> >> (PMC), to be known as the "Apache Atlas Project", be and hereby is
> >> established pursuant to Bylaws of the Foundation; and be it further
> >> RESOLVED,that the Apache Atlas Project be and hereby is responsible for
> >> the creation and maintenance of software related to a data management
> >> platform that provides real-time, consistent access to data-intensive
> >> applications throughout widely distributed cloud architectures.
> >>
> >> RESOLVED, that the office of "Vice President, Apache Atlas" be and
> >> hereby is created, the person holding such office to serve at the
> >> direction of the Board of Directors as the chair of the Apache Atlas
> >> Project, and to have

Re: Review Request 56959: Import API: Added Support for Transforming Entities During Import

2017-05-24 Thread Ashutosh Mestry


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/
---

(Updated May 25, 2017, 5:16 a.m.)


Review request for atlas and Madhan Neethiraj.


Changes
---

Fixed problem with enum used as factory.


Bugs: ATLAS-1825
https://issues.apache.org/jira/browse/ATLAS-1825


Repository: atlas


Description
---

Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) 
for scenarios and solution approach.

Impact
==
* New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This 
holds a map of entity-type to attribute and the tranforms to be applied.
* New class _AtlasImportTransformBuiler_ reads the requests specified in the 
query string of the import API and creates a _AtlasImportTransfor_ structure.
* _AtlasImportTransformBuiler_ has new classes that support transformations. 
They are:
* _replace_ Implements find and replace on attribute values.
* _lowercase_ Converts attribute value to lower case. 
* Additional classes that unit test these classes have been added.

Implementation Approach
===
* _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the 
transform every entity that is fetched from the source.
* _Import API_ now accepts multi-part input. This allows for elegantly 
specifying import parameters as well as the binary input from the file.
* Please take note of _ImportTransformerEnum_. This is first case where Enum 
has state variables.

CURL


*Setup*
Create file with these contents call it _importTransform.json_:
```javascript
{
"options": {},
"importTransform": {
"transforms": {
"hive_table": {
"qualifiedName": [
"lowercase",
"replace:@cl1:@cl2"
]
}
}
}
}
```

```
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H 
"Cache-Control: no-cache" -F request=@../docs/importTransform.json -F 
data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import;
```

Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is not 
required anymore.

Notice the output from the API, it has the _AtlasRequest_ that was passed as 
input.

Documentation
=
Need to update TWIKI documents.


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
7530196d 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransforms.java 
PRE-CREATION 
  intg/src/main/java/org/apache/atlas/model/impexp/ImportTransformerEnum.java 
PRE-CREATION 
  
intg/src/test/java/org/apache/atlas/model/impexp/ImportTransformerEnumTest.java 
PRE-CREATION 
  pom.xml a93b8ad9 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ImportEntityTransformer.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 
9ec15e00 
  repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 
87c9f0eb 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ImportEntityTransformerTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
 08e9ee81 
  
repository/src/test/java/org/apache/atlas/repository/impexp/TransformEvaluatorTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/services/EntityDiscoveryServiceTest.java
 d4900838 
  webapp/pom.xml 4132912d 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
01f9a860 


Diff: https://reviews.apache.org/r/56959/diff/6/

Changes: https://reviews.apache.org/r/56959/diff/5-6/


Testing
---

**Unit Test**
* Added tests for *AtlasImportTransformBuilder* and contained classes.
* Updated *ZipSourceTest*
* Added tests to *ImportServiceTest* to exercise new option.

**Functional Test**
* Small size data export & import.
* Medium size data export & import.

**Performance Tests**
Not exhaustive. Simple stop watch tests to compare against baseline.


Thanks,

Ashutosh Mestry

Re: [VOTE] Graduate Apache Atlas Project from Incubator

2017-05-24 Thread Chris Douglas

+1 (binding) -C

On Wed, May 24, 2017 at 4:23 PM, Suma Shivaprasad
 wrote:
> Dear Atlas Community Members,
>
> Following the discussion in Atlas community on graduating Apache Atlas
> Project from Incubation to a Top Level Project, starting this
> voting thread within the Atlas community. I appreciate mentors and
> community members in making this happen and thanks for their
> support. Apache Atlas Project has come a long way since entering into
> incubation during May 2015 and the project summary shows the
> details on this.
>
> Please vote on the Apache Atlas Project resolution that is found in
> bottom of this email.
>
> [ ] +1 Graduate Apache Atlas from the Incubator.
> [ ] +0 No opinion
> [ ] -1 Don't graduate Apache Atlas from the Incubator ( please provide
> the reason)
>
> This VOTE will be opened for next 72 hours.
>
> If this VOTE succeeds, a similar VOTE will be done in general@incubator
> mailing list. If that succeeds, a resolution will be
> included in the next Apache Board Meeting.
>
> Here is my Vote +1 (binding)
>
> Thanks all for your participation
> Suma
>
>
> Project Summary:
>
> =
> http://incubator.apache.org/projects/atlas.html
>
> Project website:
> =
>
> http://atlas.incubator.apache.org
>
> Project Documentation:
> ===
>
> http://atlas.incubator.apache.org/index.html
> Architecture - http://atlas.incubator.apache.org/Architecture.html
> Installation guide - http://atlas.incubator.apache.org/InstallationSteps.
> html
> QuickStart Guide- http://atlas.incubator.apache.org/QuickStart.html
> REST API - http://atlas.incubator.apache.org/api/v2/index.html
> Atlas Release Guide - https://cwiki.apache.org/confluence/display/ATLAS/
> Developer+Resources
>
> Project maturity Assessment:
> ===
>
> https://cwiki.apache.org/confluence/display/ATLAS/
> Apache+Atlas+Project+Maturity+Model
>
> Proposed PMC size: 38
>
> Total number of committers  : 38 members
> Total number of contributors : 62 members
>
> PMC affiliation (* indicated chair)
>
> * Hortonworks (13)
>IBM (3)
>Aetna(4)
>Freestone (3)
>Merck (2)
>Oracle(2)
>Target (2)
>Schlumberger(2)
>Dataguise(1)
>Inmobi(1)
>JPMC(1)
>MPR(1)
>SAP(1)
>SparklineData(1)
>Others(1)
>
> 1660 commits on master
> 80 contributors across all branches
> Dev list averaged ~500-800 msgs/month in 2017
> User list was created recently in April 2017. Hence not considering stats
> for this.
> 1827 issues created
>
> Committer¹s affiliation:
> ===
> * Hortonworks (13)
>IBM (3)
>Aetna(4)
>Freestone (3)
>Merck (2)
>Oracle(2)
>Target (2)
>Schlumberger(2)
>Dataguise(1)
>Inmobi(1)
>JPMC(1)
>MPR(1)
>SAP(1)
>SparklineData(1)
>Others(1)
>
>
> Apache Atlas Top Level Project Resolution:
> 
>
> Establish the Apache Atlas Project
>
> WHEREAS, the Board of Directors deems it to be in the best interests of
> the Foundation and consistent with the Foundation¹s purpose to establish a
> Project Management Committee charged with the creation and maintenance of
> open-source software, for distribution at no charge to the public, related
> to a data management platform That provides real-time, consistent access
> to data-intensive applications throughout widely distributed cloud
> architectures.
>
> NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee
> (PMC), to be known as the "Apache Atlas Project", be and hereby is
> established pursuant to Bylaws of the Foundation; and be it further
> RESOLVED,that the Apache Atlas Project be and hereby is responsible for
> the creation and maintenance of software related to a data management
> platform that provides real-time, consistent access to data-intensive
> applications throughout widely distributed cloud architectures.
>
> RESOLVED, that the office of "Vice President, Apache Atlas" be and
> hereby is created, the person holding such office to serve at the
> direction of the Board of Directors as the chair of the Apache Atlas
> Project, and to have primary responsibility for management of the projects
> within the scope of responsibility of the Apache Atlas Project; and be it
> Further.
>
>>RESOLVED,that the persons listed immediately below be and hereby are
>>appointed to serve as the initial members of the Apache Atlas Project:
>
> Aaron Dossett adoss...@target.com 
> Andrew Ahn a...@apache.org 
> Anil Varma ava...@slb.com 
> Arun C. Murthy acmur...@apache.org 
> Apoorv Naik apoorvn...@apache.org 
> Ayub Pathan ayubpat...@apache.org 
> Barbara Stortz bsto...@sap.com 
> Chris Hyzer chy...@aetna.com 
> Daniel Markwat dmark...@aetna.com 
>

Re: Review Request 56959: Import API: Added Support for Transforming Entities During Import

2017-05-24 Thread Apoorv Naik


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/#review176039
---




intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java
Lines 45 (patched)


Is this used anymore ?



intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java
Lines 83 (patched)


Same here


- Apoorv Naik


On May 24, 2017, 11:15 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56959/
> ---
> 
> (Updated May 24, 2017, 11:15 p.m.)
> 
> 
> Review request for atlas and Madhan Neethiraj.
> 
> 
> Bugs: ATLAS-1825
> https://issues.apache.org/jira/browse/ATLAS-1825
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Please refer to 
> [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios 
> and solution approach.
> 
> Impact
> ==
> * New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This 
> holds a map of entity-type to attribute and the tranforms to be applied.
> * New class _AtlasImportTransformBuiler_ reads the requests specified in the 
> query string of the import API and creates a _AtlasImportTransfor_ structure.
> * _AtlasImportTransformBuiler_ has new classes that support transformations. 
> They are:
> * _replace_ Implements find and replace on attribute values.
> * _lowercase_ Converts attribute value to lower case. 
> * Additional classes that unit test these classes have been added.
> 
> Implementation Approach
> ===
> * _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the 
> transform every entity that is fetched from the source.
> * _Import API_ now accepts multi-part input. This allows for elegantly 
> specifying import parameters as well as the binary input from the file.
> * Please take note of _ImportTransformerEnum_. This is first case where Enum 
> has state variables.
> 
> CURL
> 
> 
> *Setup*
> Create file with these contents call it _importTransform.json_:
> ```javascript
> {
> "options": {},
> "importTransform": {
> "transforms": {
> "hive_table": {
> "qualifiedName": [
> "lowercase",
> "replace:@cl1:@cl2"
> ]
> }
> }
> }
> }
> ```
> 
> ```
> curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H 
> "Cache-Control: no-cache" -F request=@../docs/importTransform.json -F 
> data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import;
> ```
> 
> Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is 
> not required anymore.
> 
> Notice the output from the API, it has the _AtlasRequest_ that was passed as 
> input.
> 
> Documentation
> =
> Need to update TWIKI documents.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
> 7530196d 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransforms.java 
> PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/impexp/ImportTransformerEnum.java 
> PRE-CREATION 
>   
> intg/src/test/java/org/apache/atlas/model/impexp/ImportTransformerEnumTest.java
>  PRE-CREATION 
>   pom.xml a93b8ad9 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportEntityTransformer.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
>  9ec15e00 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 
> 87c9f0eb 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportEntityTransformerTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
>  08e9ee81 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/TransformEvaluatorTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/services/EntityDiscoveryServiceTest.java
>  d4900838 
>   webapp/pom.xml 4132912d 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
> 01f9a860 
> 
> 
> Diff: https://reviews.apache.org/r/56959/diff/5/
> 
> 
> Testing
> ---
> 
> **Unit Test**
> * Added tests for *AtlasImportTransformBuilder* and contained classes.
> * Updated *ZipSourceTest*
> * Added tests to *ImportServiceTest* to exercise new option.
> 
> **Functional Test**
> * Small size data export & import.
> * Medium size data export & import.
> 
> **Performance Tests**
> Not exhaustive. Simple stop watch tests to compare against baseline.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

[jira] [Assigned] (ATLAS-1811) UI : Audit Events are not shown in sorted order based on time sometimes.

2017-05-24 Thread Keval Bhatt (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Keval Bhatt reassigned ATLAS-1811:
--

Assignee: Kalyani Kashikar

> UI : Audit Events are not shown in sorted order based on time sometimes.
> 
>
> Key: ATLAS-1811
> URL: https://issues.apache.org/jira/browse/ATLAS-1811
> Project: Atlas
>  Issue Type: Bug
>  Components: atlas-webui
>Affects Versions: trunk, 0.9-incubating
>Reporter: Sharmadha Sainath
>Assignee: Kalyani Kashikar
> Attachments: AuditDetailsUnordered.png
>
>
> Created an entity and made did many tag association , dis-association and tag 
> update to that entity. All audit details are captured but not displayed in 
> sorted order based on timestamp value present in audit response. A few 
> entries are missing order.
> Attached the screenshot .



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Review Request 59568: ATLAS-1827: HiveHookIT failures due to missing files in target folder

2017-05-24 Thread Yussuf Shaikh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59568/
---

Review request for atlas.


Bugs: ATLAS-1827
https://issues.apache.org/jira/browse/ATLAS-1827


Repository: atlas


Description
---

Tests in HiveHookIT fail due to missing files in target folder. Changes made to 
use explicit path in target for test files.


Diffs
-

  addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 
3e20cf0 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
979e729 


Diff: https://reviews.apache.org/r/59568/diff/1/


Testing
---

Below tests passing.
org.apache.atlas.hive.hook.HiveHookIT.testLoadDFSPathPartitioned
org.apache.atlas.hive.hook.HiveHookIT.testLoadLocalPath
org.apache.atlas.hive.hook.HiveHookIT.testLoadLocalPathIntoPartition


Thanks,

Yussuf Shaikh

Re: [VOTE] Graduate Apache Atlas Project from Incubator

2017-05-24 Thread Srikanth Venkat

+1 (non-binding)
Huge thanks to Atlas community for your wonderful support and collaboration 
over the past years!

Srikanth Venkat | Senior Director, Product Management | 
Hortonworks Inc 

On 5/24/17, 4:23 PM, "Suma Shivaprasad"  wrote:

Dear Atlas Community Members,

Following the discussion in Atlas community on graduating Apache Atlas
Project from Incubation to a Top Level Project, starting this
voting thread within the Atlas community. I appreciate mentors and
community members in making this happen and thanks for their
support. Apache Atlas Project has come a long way since entering into
incubation during May 2015 and the project summary shows the
details on this.

Please vote on the Apache Atlas Project resolution that is found in
bottom of this email.

[ ] +1 Graduate Apache Atlas from the Incubator.
[ ] +0 No opinion
[ ] -1 Don't graduate Apache Atlas from the Incubator ( please provide
the reason)

This VOTE will be opened for next 72 hours.

If this VOTE succeeds, a similar VOTE will be done in general@incubator
mailing list. If that succeeds, a resolution will be
included in the next Apache Board Meeting.

Here is my Vote +1 (binding)

Thanks all for your participation
Suma

Project Summary:

=
http://incubator.apache.org/projects/atlas.html

Project website:
=

http://atlas.incubator.apache.org

Project Documentation:
===

http://atlas.incubator.apache.org/index.html
Architecture - http://atlas.incubator.apache.org/Architecture.html
Installation guide - http://atlas.incubator.apache.org/InstallationSteps.
html
QuickStart Guide- http://atlas.incubator.apache.org/QuickStart.html
REST API - http://atlas.incubator.apache.org/api/v2/index.html
Atlas Release Guide - https://cwiki.apache.org/confluence/display/ATLAS/
Developer+Resources

Project maturity Assessment:
===

https://cwiki.apache.org/confluence/display/ATLAS/
Apache+Atlas+Project+Maturity+Model

Proposed PMC size: 38

Total number of committers  : 38 members
Total number of contributors : 62 members

PMC affiliation (* indicated chair)

* Hortonworks (13)
   IBM (3)
   Aetna(4)
   Freestone (3)
   Merck (2)
   Oracle(2)
   Target (2)
   Schlumberger(2)
   Dataguise(1)
   Inmobi(1)
   JPMC(1)
   MPR(1)
   SAP(1)
   SparklineData(1)
   Others(1)

1660 commits on master
80 contributors across all branches
Dev list averaged ~500-800 msgs/month in 2017
User list was created recently in April 2017. Hence not considering stats
for this.
1827 issues created

Committer¹s affiliation:
===
* Hortonworks (13)
   IBM (3)
   Aetna(4)
   Freestone (3)
   Merck (2)
   Oracle(2)
   Target (2)
   Schlumberger(2)
   Dataguise(1)
   Inmobi(1)
   JPMC(1)
   MPR(1)
   SAP(1)
   SparklineData(1)
   Others(1)

Apache Atlas Top Level Project Resolution:

Establish the Apache Atlas Project

WHEREAS, the Board of Directors deems it to be in the best interests of
the Foundation and consistent with the Foundation¹s purpose to establish a
Project Management Committee charged with the creation and maintenance of
open-source software, for distribution at no charge to the public, related
to a data management platform That provides real-time, consistent access
to data-intensive applications throughout widely distributed cloud
architectures.

NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee
(PMC), to be known as the "Apache Atlas Project", be and hereby is
established pursuant to Bylaws of the Foundation; and be it further
RESOLVED,that the Apache Atlas Project be and hereby is responsible for
the creation and maintenance of software related to a data management
platform that provides real-time, consistent access to data-intensive
applications throughout widely distributed cloud architectures.

RESOLVED, that the office of "Vice President, Apache Atlas" be and
hereby is created, the person holding such office to serve at the
direction of the Board of Directors as the chair of the Apache Atlas
Project, and to have primary responsibility for management of the projects
within the scope of responsibility of the Apache Atlas Project; and be it
Further.

>RESOLVED,that the persons listed immediately below be and hereby are
>appointed to serve as the initial members of the Apache Atlas Project:

Aaron Dossett adoss...@target.com 
Andrew

[jira] [Updated] (ATLAS-1825) Import API: Ability to Apply Transforms to Attributes During Import

2017-05-24 Thread Ashutosh Mestry (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry updated ATLAS-1825:
---
Attachment: ATLAS-1825-Import-transform.2.patch

> Import API: Ability to Apply Transforms to Attributes During Import
> ---
>
> Key: ATLAS-1825
> URL: https://issues.apache.org/jira/browse/ATLAS-1825
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: trunk
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
> Fix For: trunk
>
> Attachments: ATLAS-1825-Import-transform.2.patch
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> h3. Scenarios
> *Case 1*
> In a situation where, data exported from a cluster is imported to a different 
> cluster, it is necessary to make changes to attributes of entities such that 
> their values represent their environment
> Example: Entities exported from cluster called _'cl1'_ will have _@cl1_ in 
> their qualifiedName attribute. When these entities are imported to a cluster 
> called _'cl2'_ the qualifiedName attribute should read _@cl2_ to correctly 
> reflect their new environment.
> *Case 2*
> Exported data contains an attribute that has value that does not conform to 
> the conventions followed within the cluster. This can be corrected by 
> exporting the data and reimporting it with an applied transform.
> Example: Entities were created with an attribute value as upper case, this 
> needs to be lower case to conform to a convention.
> h3. Solution
> As part of Import API, it will help to be able to specify in a generic manner 
> the transforms that could be applied to an entity type.
> The implementation should be extensible so that new transforms could be added 
> in future.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[VOTE] Graduate Apache Atlas Project from Incubator

2017-05-24 Thread Suma Shivaprasad

Dear Atlas Community Members,

Following the discussion in Atlas community on graduating Apache Atlas
Project from Incubation to a Top Level Project, starting this
voting thread within the Atlas community. I appreciate mentors and
community members in making this happen and thanks for their
support. Apache Atlas Project has come a long way since entering into
incubation during May 2015 and the project summary shows the
details on this.

Please vote on the Apache Atlas Project resolution that is found in
bottom of this email.

[ ] +1 Graduate Apache Atlas from the Incubator.
[ ] +0 No opinion
[ ] -1 Don't graduate Apache Atlas from the Incubator ( please provide
the reason)

This VOTE will be opened for next 72 hours.

If this VOTE succeeds, a similar VOTE will be done in general@incubator
mailing list. If that succeeds, a resolution will be
included in the next Apache Board Meeting.

Here is my Vote +1 (binding)

Thanks all for your participation
Suma


Project Summary:

=
http://incubator.apache.org/projects/atlas.html

Project website:
=

http://atlas.incubator.apache.org

Project Documentation:
===

http://atlas.incubator.apache.org/index.html
Architecture - http://atlas.incubator.apache.org/Architecture.html
Installation guide - http://atlas.incubator.apache.org/InstallationSteps.
html
QuickStart Guide- http://atlas.incubator.apache.org/QuickStart.html
REST API - http://atlas.incubator.apache.org/api/v2/index.html
Atlas Release Guide - https://cwiki.apache.org/confluence/display/ATLAS/
Developer+Resources

Project maturity Assessment:
===

https://cwiki.apache.org/confluence/display/ATLAS/
Apache+Atlas+Project+Maturity+Model

Proposed PMC size: 38

Total number of committers  : 38 members
Total number of contributors : 62 members

PMC affiliation (* indicated chair)

* Hortonworks (13)
   IBM (3)
   Aetna(4)
   Freestone (3)
   Merck (2)
   Oracle(2)
   Target (2)
   Schlumberger(2)
   Dataguise(1)
   Inmobi(1)
   JPMC(1)
   MPR(1)
   SAP(1)
   SparklineData(1)
   Others(1)

1660 commits on master
80 contributors across all branches
Dev list averaged ~500-800 msgs/month in 2017
User list was created recently in April 2017. Hence not considering stats
for this.
1827 issues created

Committer¹s affiliation:
===
* Hortonworks (13)
   IBM (3)
   Aetna(4)
   Freestone (3)
   Merck (2)
   Oracle(2)
   Target (2)
   Schlumberger(2)
   Dataguise(1)
   Inmobi(1)
   JPMC(1)
   MPR(1)
   SAP(1)
   SparklineData(1)
   Others(1)


Apache Atlas Top Level Project Resolution:


Establish the Apache Atlas Project

WHEREAS, the Board of Directors deems it to be in the best interests of
the Foundation and consistent with the Foundation¹s purpose to establish a
Project Management Committee charged with the creation and maintenance of
open-source software, for distribution at no charge to the public, related
to a data management platform That provides real-time, consistent access
to data-intensive applications throughout widely distributed cloud
architectures.

NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee
(PMC), to be known as the "Apache Atlas Project", be and hereby is
established pursuant to Bylaws of the Foundation; and be it further
RESOLVED,that the Apache Atlas Project be and hereby is responsible for
the creation and maintenance of software related to a data management
platform that provides real-time, consistent access to data-intensive
applications throughout widely distributed cloud architectures.

RESOLVED, that the office of "Vice President, Apache Atlas" be and
hereby is created, the person holding such office to serve at the
direction of the Board of Directors as the chair of the Apache Atlas
Project, and to have primary responsibility for management of the projects
within the scope of responsibility of the Apache Atlas Project; and be it
Further.

>RESOLVED,that the persons listed immediately below be and hereby are
>appointed to serve as the initial members of the Apache Atlas Project:

Aaron Dossett adoss...@target.com 
Andrew Ahn a...@apache.org 
Anil Varma ava...@slb.com 
Arun C. Murthy acmur...@apache.org 
Apoorv Naik apoorvn...@apache.org 
Ayub Pathan ayubpat...@apache.org 
Barbara Stortz bsto...@sap.com 
Chris Hyzer chy...@aetna.com 
Daniel Markwat dmark...@aetna.com 
Darshan Kumar darshanku...@apache.org 
Dave Kantor dkan...@apache.org 
David Kaspar dkas...@merck.com 
Dennis Fusaro dfus...@aetna.com 
Greg Senia gse...@aetna.com 
Harish Butani rhbut...@apache.org 
Hemanth

Re: Review Request 56959: Import API: Added Support for Transforming Entities During Import

2017-05-24 Thread Ashutosh Mestry


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/
---

(Updated May 24, 2017, 11:15 p.m.)


Review request for atlas and Madhan Neethiraj.


Changes
---

Additional text.


Bugs: ATLAS-1825
https://issues.apache.org/jira/browse/ATLAS-1825


Repository: atlas


Description (updated)
---

Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) 
for scenarios and solution approach.

Impact
==
* New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This 
holds a map of entity-type to attribute and the tranforms to be applied.
* New class _AtlasImportTransformBuiler_ reads the requests specified in the 
query string of the import API and creates a _AtlasImportTransfor_ structure.
* _AtlasImportTransformBuiler_ has new classes that support transformations. 
They are:
* _replace_ Implements find and replace on attribute values.
* _lowercase_ Converts attribute value to lower case. 
* Additional classes that unit test these classes have been added.

Implementation Approach
===
* _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the 
transform every entity that is fetched from the source.
* _Import API_ now accepts multi-part input. This allows for elegantly 
specifying import parameters as well as the binary input from the file.
* Please take note of _ImportTransformerEnum_. This is first case where Enum 
has state variables.

CURL


*Setup*
Create file with these contents call it _importTransform.json_:
```javascript
{
"options": {},
"importTransform": {
"transforms": {
"hive_table": {
"qualifiedName": [
"lowercase",
"replace:@cl1:@cl2"
]
}
}
}
}
```

```
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H 
"Cache-Control: no-cache" -F request=@../docs/importTransform.json -F 
data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import;
```

Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is not 
required anymore.

Notice the output from the API, it has the _AtlasRequest_ that was passed as 
input.

Documentation
=
Need to update TWIKI documents.


Diffs
-

  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
7530196d 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransforms.java 
PRE-CREATION 
  intg/src/main/java/org/apache/atlas/model/impexp/ImportTransformerEnum.java 
PRE-CREATION 
  
intg/src/test/java/org/apache/atlas/model/impexp/ImportTransformerEnumTest.java 
PRE-CREATION 
  pom.xml a93b8ad9 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ImportEntityTransformer.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 
9ec15e00 
  repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 
87c9f0eb 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ImportEntityTransformerTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
 08e9ee81 
  
repository/src/test/java/org/apache/atlas/repository/impexp/TransformEvaluatorTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/services/EntityDiscoveryServiceTest.java
 d4900838 
  webapp/pom.xml 4132912d 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
01f9a860 


Diff: https://reviews.apache.org/r/56959/diff/4/


Testing
---

**Unit Test**
* Added tests for *AtlasImportTransformBuilder* and contained classes.
* Updated *ZipSourceTest*
* Added tests to *ImportServiceTest* to exercise new option.

**Functional Test**
* Small size data export & import.
* Medium size data export & import.

**Performance Tests**
Not exhaustive. Simple stop watch tests to compare against baseline.


Thanks,

Ashutosh Mestry

Re: Review Request 56959: Import API: Added Support for Transforming Entities During Import

2017-05-24 Thread Ashutosh Mestry


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/
---

(Updated May 24, 2017, 11:06 p.m.)


Review request for atlas and Madhan Neethiraj.


Changes
---

Refactoring:
* To support muti-part input to Import API.
* Overall to reduce code.


Bugs: ATLAS-1825
https://issues.apache.org/jira/browse/ATLAS-1825


Repository: atlas


Description (updated)
---

Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) 
for scenarios and solution approach.

Impact
==
* New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This 
holds a map of entity-type to attribute and the tranforms to be applied.
* New class _AtlasImportTransformBuiler_ reads the requests specified in the 
query string of the import API and creates a _AtlasImportTransfor_ structure.
* _AtlasImportTransformBuiler_ has new classes that support transformations. 
They are:
* _replace_ Implements find and replace on attribute values.
* _lowercase_ Converts attribute value to lower case. 
* Additional classes that unit test these classes have been added.

Implementation Approach
===
* _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the 
transform every entity that is fetched from the source.
* _Import API_ now accepts multi-part input. This allows for elegantly 
specifying import parameters as well as the binary input from the file.

CURL


*Setup*
Create file with these contents call it _importTransform.json_:
```javascript
{
"options": {},
"importTransform": {
"transforms": {
"hive_table": {
"qualifiedName": [
"lowercase",
"replace:@cl1:@cl2"
]
}
}
}
}
```

```
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H 
"Cache-Control: no-cache" -F request=@../docs/importTransform.json -F 
data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import;
```

Note: _-g_ needs to be added to CURL calls to prevent gobble check.

Documentation
=
Need to update TWIKI documents.


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
7530196d 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransforms.java 
PRE-CREATION 
  intg/src/main/java/org/apache/atlas/model/impexp/ImportTransformerEnum.java 
PRE-CREATION 
  
intg/src/test/java/org/apache/atlas/model/impexp/ImportTransformerEnumTest.java 
PRE-CREATION 
  pom.xml a93b8ad9 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ImportEntityTransformer.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 
9ec15e00 
  repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 
87c9f0eb 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ImportEntityTransformerTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
 08e9ee81 
  
repository/src/test/java/org/apache/atlas/repository/impexp/TransformEvaluatorTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/services/EntityDiscoveryServiceTest.java
 d4900838 
  webapp/pom.xml 4132912d 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
01f9a860 


Diff: https://reviews.apache.org/r/56959/diff/4/

Changes: https://reviews.apache.org/r/56959/diff/3-4/


Testing
---

**Unit Test**
* Added tests for *AtlasImportTransformBuilder* and contained classes.
* Updated *ZipSourceTest*
* Added tests to *ImportServiceTest* to exercise new option.

**Functional Test**
* Small size data export & import.
* Medium size data export & import.

**Performance Tests**
Not exhaustive. Simple stop watch tests to compare against baseline.


Thanks,

Ashutosh Mestry

Re: Review Request 56959: Import API: Added Support for Transforming Entities During Import

2017-05-24 Thread Ashutosh Mestry



> On May 24, 2017, 1:06 a.m., Apoorv Naik wrote:
> > This should help dealing with multi-part data 
> > 
> > ```xml
> > 
> >   com.sun.jersey.contribs
> >   jersey-multipart
> >   1.8
> > 
> > ```
> > 
> > A sample of java REST implementation
> > ```java
> > @POST
> > @Path("multipart")
> > @Consumes(MediaType.MULTIPART_FORM_DATA)
> > @Produces(Servlets.JSON_MEDIA_TYPE)
> > public String processMultipart(@FormDataParam("meta") String jsonMeta, 
> > @FormDataParam("data") InputStream inputStream) {
> > return null;
> > }
> > ```

Thanks a lot! This is really useful.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/#review175870
---


On May 23, 2017, 10:11 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56959/
> ---
> 
> (Updated May 23, 2017, 10:11 p.m.)
> 
> 
> Review request for atlas and Madhan Neethiraj.
> 
> 
> Bugs: ATLAS-1825
> https://issues.apache.org/jira/browse/ATLAS-1825
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Please refer to 
> [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios 
> and solution approach.
> 
> Impact
> ==
> * New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This 
> holds a map of entity-type to attribute and the tranforms to be applied.
> * New class _AtlasImportTransformBuiler_ reads the requests specified in the 
> query string of the import API and creates a _AtlasImportTransfor_ structure.
> * _AtlasImportTransformBuiler_ has new classes that support transformations. 
> They are:
> * _replace_ Implements find and replace on attribute values.
> * _lowercase_ Converts attribute value to lower case. 
> * Additional classes that unit test these classes have been added.
> 
> Implementation Approach
> ===
> * _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the 
> transform every entity that is fetched from the source.
> 
> CURL
> 
> ```
> curl -g -X POST -u admin:admin -H "Content-Type: application/octet-stream" -H 
> "Cache-Control: no-cache" --data-binary @../docs/DefaultDB-Stocks-2.zip 
> "http://localhost:21000/api/atlas/admin/import?tr={'transforms':{'hive_db':{'qualifiedName':['lowercase','replace:@cl1:@cl2']}}}"
> ```
> ```
> curl -g -X POST -u admin:admin -H "Content-Type: application/octet-stream" -H 
> "Cache-Control: no-cache" --data-binary @../docs/DefaultDB-Stocks-2.zip 
> "http://localhost:21000/api/atlas/admin/import?tr={'transforms':{'hive_table':{'qualifiedName':['replace:@cl1:@cl2']}}}"
> ```
> 
> Note: _-g_ needs to be added to CURL calls to prevent gobble check.
> 
> Documentation
> =
> Need to update TWIKI documents.
> 
> 
> Diffs
> -
> 
>   intg/pom.xml 7f3ab12b 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
> 7530196d 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransform.java 
> PRE-CREATION 
>   
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransformBuilder.java
>  PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/impexp/ImportTransform.java 
> PRE-CREATION 
>   
> intg/src/test/java/org/apache/atlas/model/impexp/AtlasImportTransformBuilderTest.java
>  PRE-CREATION 
>   
> intg/src/test/java/org/apache/atlas/model/impexp/ContainsReplaceTransformTest.java
>  PRE-CREATION 
>   
> intg/src/test/java/org/apache/atlas/model/impexp/TransformEvaluatorTest.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
>  9ec15e00 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 
> 87c9f0eb 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
>  91966f8f 
> 
> 
> Diff: https://reviews.apache.org/r/56959/diff/3/
> 
> 
> Testing
> ---
> 
> **Unit Test**
> * Added tests for *AtlasImportTransformBuilder* and contained classes.
> * Updated *ZipSourceTest*
> * Added tests to *ImportServiceTest* to exercise new option.
> 
> **Functional Test**
> * Small size data export & import.
> * Medium size data export & import.
> 
> **Performance Tests**
> Not exhaustive. Simple stop watch tests to compare against baseline.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Build failed in Jenkins: apache-atlas-nightly #751

2017-05-24 Thread Apache Jenkins Server

See 


Changes:

[apoorvnaik] ATLAS-1198: Spring Framework (v4 with Spring security) over Guice

--
[...truncated 327.07 KB...]
[WARNING] Note: Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ 
atlas-repository ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 54 source files to 

[WARNING] 
:
 Some input files use or override a deprecated API.
[WARNING] 
:
 Recompile with -Xlint:deprecation for details.
[WARNING] 
:
 Some input files use unchecked or unsafe operations.
[WARNING] 
:
 Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ atlas-repository 
---
[INFO] Surefire report directory: 

[INFO] Using configured provider org.apache.maven.surefire.testng.TestNGProvider
[WARNING] file.encoding cannot be set as system property, use 
-Dfile.encoding=... instead

---
 T E S T S
---
Running org.apache.atlas.RepositoryServiceLoadingTest
Running 
org.apache.atlas.repository.store.graph.v1.InverseReferenceUpdateSoftDeleteV1Test
Running org.apache.atlas.repository.store.graph.v1.AtlasDeleteHandlerV1Test
Running org.apache.atlas.TestModules
Running org.apache.atlas.repository.store.graph.AtlasEntityDefStoreV1Test
Running org.apache.atlas.util.CompiledQueryCacheKeyTest
Running org.apache.atlas.discovery.GraphBackedDiscoveryServiceTest
Running 
org.apache.atlas.repository.store.graph.v1.InverseReferenceUpdateHardDeleteV1Test
Running org.apache.atlas.discovery.DataSetLineageServiceTest
Running org.apache.atlas.repository.impexp.ImportServiceReportingTest
Running org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStoreTest
Running org.apache.atlas.service.DefaultMetadataServiceTest
Running org.apache.atlas.repository.impexp.ZipSinkTest
Running org.apache.atlas.service.StoreBackedTypeCacheMetadataServiceTest
Running org.apache.atlas.repository.graph.GraphBackedMetadataRepositoryTest
Running org.apache.atlas.repository.graph.ReverseReferenceUpdateHardDeleteTest
Running org.apache.atlas.repository.store.graph.v1.HardDeleteHandlerV1Test
Running org.apache.atlas.repository.impexp.ZipSourceTest
Running org.apache.atlas.repository.audit.HBaseBasedAuditRepositoryHATest
Running org.apache.atlas.repository.impexp.ImportServiceTest
Running org.apache.atlas.repository.store.graph.v1.SoftDeleteHandlerV1Test
Running org.apache.atlas.repository.audit.HBaseBasedAuditRepositoryTest
Running org.apache.atlas.repository.impexp.ExportServiceTest
Running org.apache.atlas.repository.store.graph.v1.InverseReferenceUpdateV1Test
Running org.apache.atlas.repository.audit.InMemoryAuditRepositoryTest
Running org.apache.atlas.repository.graph.GraphHelperTest
Running org.apache.atlas.repository.graph.ReverseReferenceUpdateSoftDeleteTest
Running org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1Test
Running org.apache.atlas.repository.graph.Gremlin3QueryOptimizerTest
Running org.apache.atlas.repository.graph.TestIntSequence
Running org.apache.atlas.repository.graph.GraphBackedRepositorySoftDeleteTest
Running org.apache.atlas.repository.graph.GraphBackedRepositoryHardDeleteTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.076 sec - in 
org.apache.atlas.util.CompiledQueryCacheKeyTest
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.544 sec - in 
org.apache.atlas.repository.store.graph.v1.AtlasDeleteHandlerV1Test
Running org.apache.atlas.repository.graph.AbstractGremlinQueryOptimizerTest
Running org.apache.atlas.repository.graph.GraphBackedSearchIndexerMockTest
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.485 sec - in 
org.apache.atlas.TestModules
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.305 sec - in 
org.apache.atlas.repository.impexp.ImportServiceReportingTest
Running org.apache.atlas.repository.graph.GraphHelperMockTest
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.429 sec - in

Re: Review Request 59533: Test Sandbox fix

2017-05-24 Thread Sarath Subramanian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59533/#review175961
---




graphdb/common/src/test/java/org/apache/atlas/graph/GraphSandboxUtil.java
Line 40 (original), 41 (patched)


can you consider appending UUID with storage/index directory. It should be 
unique across forked process.


- Sarath Subramanian


On May 24, 2017, 10:30 a.m., Apoorv Naik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59533/
> ---
> 
> (Updated May 24, 2017, 10:30 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Sarath Subramanian, and Suma 
> Shivaprasad.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Due to race condition, a couple of test still end up using the same directory 
> for DB/index storage. 
> 
> This change checks for the existence of the folder and generates the name 
> again in case of any discrepancy.
> 
> 
> Diffs
> -
> 
>   graphdb/common/src/test/java/org/apache/atlas/graph/GraphSandboxUtil.java 
> 44ad4fc0 
> 
> 
> Diff: https://reviews.apache.org/r/59533/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean package executes successfully
> mvn clean install executes successfully with one consitent failure related to 
> notification processing.
> 
> 
> Thanks,
> 
> Apoorv Naik
> 
>

Review Request 59533: Test Sandbox fix

2017-05-24 Thread Apoorv Naik


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59533/
---

Review request for atlas, Madhan Neethiraj, Sarath Subramanian, and Suma 
Shivaprasad.


Repository: atlas


Description
---

Due to race condition, a couple of test still end up using the same directory 
for DB/index storage. 

This change checks for the existence of the folder and generates the name again 
in case of any discrepancy.


Diffs
-

  graphdb/common/src/test/java/org/apache/atlas/graph/GraphSandboxUtil.java 
44ad4fc0 


Diff: https://reviews.apache.org/r/59533/diff/1/


Testing
---

mvn clean package executes successfully
mvn clean install executes successfully with one consitent failure related to 
notification processing.


Thanks,

Apoorv Naik

[jira] [Commented] (ATLAS-1806) Error while redirecting it to Tag tab after doing advance search and creating entity

2017-05-24 Thread Kalyani Kashikar (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022725#comment-16022725
 ] 

Kalyani Kashikar commented on ATLAS-1806:
-

This issue has been fixed in ATLAS-1812.

> Error while redirecting it to Tag tab after doing advance search and creating 
> entity
> 
>
> Key: ATLAS-1806
> URL: https://issues.apache.org/jira/browse/ATLAS-1806
> Project: Atlas
>  Issue Type: Bug
>Reporter: Nixon Rodrigues
>Assignee: Kalyani Kashikar
> Attachments: SS11.png
>
>
>  Steps to reproduce:-
> 1.Do an Advance Search
> 2.Then create an entity
> 3.Click on search tab
> 4.Now click on Tags Tab
> Error:- It would redirect back to Search Tab which is incorrect



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (ATLAS-1806) Error while redirecting it to Tag tab after doing advance search and creating entity

2017-05-24 Thread Kalyani Kashikar (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kalyani Kashikar resolved ATLAS-1806.
-
Resolution: Fixed

> Error while redirecting it to Tag tab after doing advance search and creating 
> entity
> 
>
> Key: ATLAS-1806
> URL: https://issues.apache.org/jira/browse/ATLAS-1806
> Project: Atlas
>  Issue Type: Bug
>Reporter: Nixon Rodrigues
>Assignee: Kalyani Kashikar
> Attachments: SS11.png
>
>
>  Steps to reproduce:-
> 1.Do an Advance Search
> 2.Then create an entity
> 3.Click on search tab
> 4.Now click on Tags Tab
> Error:- It would redirect back to Search Tab which is incorrect



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (ATLAS-1806) Error while redirecting it to Tag tab after doing advance search and creating entity

2017-05-24 Thread Kalyani Kashikar (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kalyani Kashikar reassigned ATLAS-1806:
---

Assignee: Kalyani Kashikar

> Error while redirecting it to Tag tab after doing advance search and creating 
> entity
> 
>
> Key: ATLAS-1806
> URL: https://issues.apache.org/jira/browse/ATLAS-1806
> Project: Atlas
>  Issue Type: Bug
>Reporter: Nixon Rodrigues
>Assignee: Kalyani Kashikar
> Attachments: SS11.png
>
>
>  Steps to reproduce:-
> 1.Do an Advance Search
> 2.Then create an entity
> 3.Click on search tab
> 4.Now click on Tags Tab
> Error:- It would redirect back to Search Tab which is incorrect



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (ATLAS-1827) HiveHookIT failures due to missing files in target folder

2017-05-24 Thread Yussuf Shaikh (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yussuf Shaikh updated ATLAS-1827:
-
Attachment: ATLAS-1827.patch

> HiveHookIT failures due to missing files in target folder
> -
>
> Key: ATLAS-1827
> URL: https://issues.apache.org/jira/browse/ATLAS-1827
> Project: Atlas
>  Issue Type: Bug
>  Components: atlas-intg
>Affects Versions: 0.8-incubating
> Environment: # cat /etc/lsb-release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=14.04
> DISTRIB_CODENAME=trusty
> DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS"
> # uname -a
> Linux 3b9700711ca1 3.19.0-37-generic #42-Ubuntu SMP Fri Nov 20 18:22:05 UTC 
> 2015 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Yussuf Shaikh
> Attachments: ATLAS-1827.patch
>
>
> Below tests fail because Hive cannot read the test files from 
> addons/hive-bridge/target folder.
> org.apache.atlas.hive.hook.HiveHookIT.testLoadDFSPathPartitioned
> org.apache.atlas.hive.hook.HiveHookIT.testLoadLocalPath
> org.apache.atlas.hive.hook.HiveHookIT.testLoadLocalPathIntoPartition
> Stack trace for testLoadLocalPath:
> 2017-05-24 06:37:37,150 ERROR - [main:] ~ FAILED: SemanticException Line 1:23 
> Invalid path 
> ''file:///var/lib/jenkins/workspace/atlas-test/addons/hive-bridge/./target/load-data-7KV7AalW6D'':
>  No files matching path 
> file:/var/lib/jenkins/workspace/atlas-test/addons/hive-bridge/target/load-data-7KV7AalW6D
> org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:23 Invalid path 
> ''file:///var/lib/jenkins/workspace/atlas-test/addons/hive-bridge/./target/load-data-7KV7AalW6D'':
>  No files matching path 
> file:/var/lib/jenkins/workspace/atlas-test/addons/hive-bridge/target/load-data-7KV7AalW6D
>   at 
> org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.applyConstraints(LoadSemanticAnalyzer.java:139)
>   at 
> org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:230)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:424)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
>   at 
> org.apache.atlas.hive.HiveITBase.runCommandWithDelay(HiveITBase.java:119)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (ATLAS-1827) HiveHookIT failures due to missing files in target folder

2017-05-24 Thread Yussuf Shaikh (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022586#comment-16022586
 ] 

Yussuf Shaikh commented on ATLAS-1827:
--

The root cause of the issue seems to be at the file creation in 
HiveHookIT.file() where . indicates that the file is created in target folder 
of the parent project and not in hive-bridge project.
Correct way of implementing it is to give explicit path to the target folder 
using System.getProperty("projectBaseDir").

> HiveHookIT failures due to missing files in target folder
> -
>
> Key: ATLAS-1827
> URL: https://issues.apache.org/jira/browse/ATLAS-1827
> Project: Atlas
>  Issue Type: Bug
>  Components: atlas-intg
>Affects Versions: 0.8-incubating
> Environment: # cat /etc/lsb-release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=14.04
> DISTRIB_CODENAME=trusty
> DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS"
> # uname -a
> Linux 3b9700711ca1 3.19.0-37-generic #42-Ubuntu SMP Fri Nov 20 18:22:05 UTC 
> 2015 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Yussuf Shaikh
>
> Below tests fail because Hive cannot read the test files from 
> addons/hive-bridge/target folder.
> org.apache.atlas.hive.hook.HiveHookIT.testLoadDFSPathPartitioned
> org.apache.atlas.hive.hook.HiveHookIT.testLoadLocalPath
> org.apache.atlas.hive.hook.HiveHookIT.testLoadLocalPathIntoPartition
> Stack trace for testLoadLocalPath:
> 2017-05-24 06:37:37,150 ERROR - [main:] ~ FAILED: SemanticException Line 1:23 
> Invalid path 
> ''file:///var/lib/jenkins/workspace/atlas-test/addons/hive-bridge/./target/load-data-7KV7AalW6D'':
>  No files matching path 
> file:/var/lib/jenkins/workspace/atlas-test/addons/hive-bridge/target/load-data-7KV7AalW6D
> org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:23 Invalid path 
> ''file:///var/lib/jenkins/workspace/atlas-test/addons/hive-bridge/./target/load-data-7KV7AalW6D'':
>  No files matching path 
> file:/var/lib/jenkins/workspace/atlas-test/addons/hive-bridge/target/load-data-7KV7AalW6D
>   at 
> org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.applyConstraints(LoadSemanticAnalyzer.java:139)
>   at 
> org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:230)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:424)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
>   at 
> org.apache.atlas.hive.HiveITBase.runCommandWithDelay(HiveITBase.java:119)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (ATLAS-1827) HiveHookIT failures due to missing files in target folder

2017-05-24 Thread Yussuf Shaikh (JIRA)

Yussuf Shaikh created ATLAS-1827:


 Summary: HiveHookIT failures due to missing files in target folder
 Key: ATLAS-1827
 URL: https://issues.apache.org/jira/browse/ATLAS-1827
 Project: Atlas
  Issue Type: Bug
  Components: atlas-intg
Affects Versions: 0.8-incubating
 Environment: # cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS"
# uname -a
Linux 3b9700711ca1 3.19.0-37-generic #42-Ubuntu SMP Fri Nov 20 18:22:05 UTC 
2015 x86_64 x86_64 x86_64 GNU/Linux
Reporter: Yussuf Shaikh


Below tests fail because Hive cannot read the test files from 
addons/hive-bridge/target folder.
org.apache.atlas.hive.hook.HiveHookIT.testLoadDFSPathPartitioned
org.apache.atlas.hive.hook.HiveHookIT.testLoadLocalPath
org.apache.atlas.hive.hook.HiveHookIT.testLoadLocalPathIntoPartition

Stack trace for testLoadLocalPath:

2017-05-24 06:37:37,150 ERROR - [main:] ~ FAILED: SemanticException Line 1:23 
Invalid path 
''file:///var/lib/jenkins/workspace/atlas-test/addons/hive-bridge/./target/load-data-7KV7AalW6D'':
 No files matching path 
file:/var/lib/jenkins/workspace/atlas-test/addons/hive-bridge/target/load-data-7KV7AalW6D
org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:23 Invalid path 
''file:///var/lib/jenkins/workspace/atlas-test/addons/hive-bridge/./target/load-data-7KV7AalW6D'':
 No files matching path 
file:/var/lib/jenkins/workspace/atlas-test/addons/hive-bridge/target/load-data-7KV7AalW6D
at 
org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.applyConstraints(LoadSemanticAnalyzer.java:139)
at 
org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:230)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:424)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
at 
org.apache.atlas.hive.HiveITBase.runCommandWithDelay(HiveITBase.java:119)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 59494: ATLAS-1804 Allow PAM for authentication

2017-05-24 Thread Nixon Rodrigues


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59494/#review175896
---




webapp/src/main/java/org/apache/atlas/web/security/AtlasPamAuthenticationProvider.java
Lines 60 (patched)


loginModuleName & controlFlag are constant values, make no sense 
initializing for every user login call, they qualify to be static member 
variable.



webapp/src/main/java/org/apache/atlas/web/security/AtlasPamAuthenticationProvider.java
Lines 62 (patched)


The"atlas.authentication.method.pam" is application based property and it 
is getting called for every getPamAuthentication call,

Should be called constructor or init method or method with @postconstruct



webapp/src/main/java/org/apache/atlas/web/security/AtlasPamAuthenticationProvider.java
Lines 69 (patched)


can add opening and closing brackets {} for better readability.



webapp/src/main/java/org/apache/atlas/web/security/AtlasPamAuthenticationProvider.java
Lines 75 (patched)


From line # 60 to line 81 code can be initialized once in init method, 
please check



webapp/src/main/java/org/apache/atlas/web/security/AtlasPamAuthenticationProvider.java
Lines 105 (patched)


From where is the groups for user come from for PAM authentication ?  

I tested PAM based authentication and got DATA_SCIENTIST always.

For LDAP/AD if spring authentication does not returns user group then  
groups are retrived from UNIX/Hadoop UGI using 
getAuthenticationWithGrantedAuthorityFromUGI, check 
AtlasADAuthenticationProvider for more details.



webapp/src/main/java/org/apache/atlas/web/security/UserAuthorityGranter.java
Lines 29 (patched)


Remove this comment


- Nixon Rodrigues


On May 23, 2017, 6:20 p.m., Shi Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59494/
> ---
> 
> (Updated May 23, 2017, 6:20 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik and Nixon Rodrigues.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas currently support File, Kerberos and Ldap authentication. An 
> improvement feature will be adding PAM as another authentication type.
> 
> 
> Diffs
> -
> 
>   webapp/pom.xml 045ccdb 
>   
> webapp/src/main/java/org/apache/atlas/web/security/AtlasAuthenticationProvider.java
>  953d737 
>   
> webapp/src/main/java/org/apache/atlas/web/security/AtlasPamAuthenticationProvider.java
>  PRE-CREATION 
>   webapp/src/main/java/org/apache/atlas/web/security/PamLoginModule.java 
> PRE-CREATION 
>   webapp/src/main/java/org/apache/atlas/web/security/PamPrincipal.java 
> PRE-CREATION 
>   
> webapp/src/main/java/org/apache/atlas/web/security/UserAuthorityGranter.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/59494/diff/1/
> 
> 
> Testing
> ---
> 
> create a pam file atlas-login under /etc/pam.d/
> 
> in the file using unix login module for testing.
> 
> content could be something like:
> authsufficientpam_unix.so
> account sufficientpam_unix.so
> 
> create a unix user with password.
> login to atlas ui with the unix user will success.
> 
> 
> Thanks,
> 
> Shi Wang
> 
>

[jira] [Created] (ATLAS-1826) UI Create Entity : Type doesn't list the inverseRef constrained attribute options when name length is less than 3.

2017-05-24 Thread Sharmadha Sainath (JIRA)

Sharmadha Sainath created ATLAS-1826:


 Summary: UI Create Entity : Type doesn't list the inverseRef 
constrained attribute options when name length is less than 3.
 Key: ATLAS-1826
 URL: https://issues.apache.org/jira/browse/ATLAS-1826
 Project: Atlas
  Issue Type: Bug
  Components: atlas-webui
Affects Versions: 0.9-incubating
Reporter: Sharmadha Sainath


1. Created a hbase_table h1.
2. Tried to create a hbase_column_family with hbase table as h1. Since "h1" has 
only 2 characters , the drop down didn't display "h1" but displayed "Please 
enter 1 or more characters".
3. When hbase_table name length is more than 2 , the drop down lists the name.

Same happens when adding hbase_column_family with name "hf" to hbase_column.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] Graduate Apache Atlas Project from Incubator

Re: [VOTE] Graduate Apache Atlas Project from Incubator

Re: Review Request 56959: Import API: Added Support for Transforming Entities During Import

Re: [VOTE] Graduate Apache Atlas Project from Incubator

Re: Review Request 56959: Import API: Added Support for Transforming Entities During Import

Re: [VOTE] Graduate Apache Atlas Project from Incubator

Re: Review Request 56959: Import API: Added Support for Transforming Entities During Import

[jira] [Assigned] (ATLAS-1811) UI : Audit Events are not shown in sorted order based on time sometimes.

Review Request 59568: ATLAS-1827: HiveHookIT failures due to missing files in target folder

Re: [VOTE] Graduate Apache Atlas Project from Incubator

[jira] [Updated] (ATLAS-1825) Import API: Ability to Apply Transforms to Attributes During Import

[VOTE] Graduate Apache Atlas Project from Incubator

Re: Review Request 56959: Import API: Added Support for Transforming Entities During Import

Re: Review Request 56959: Import API: Added Support for Transforming Entities During Import

Re: Review Request 56959: Import API: Added Support for Transforming Entities During Import

Build failed in Jenkins: apache-atlas-nightly #751

Re: Review Request 59533: Test Sandbox fix

Review Request 59533: Test Sandbox fix

[jira] [Commented] (ATLAS-1806) Error while redirecting it to Tag tab after doing advance search and creating entity

[jira] [Resolved] (ATLAS-1806) Error while redirecting it to Tag tab after doing advance search and creating entity

[jira] [Assigned] (ATLAS-1806) Error while redirecting it to Tag tab after doing advance search and creating entity

[jira] [Updated] (ATLAS-1827) HiveHookIT failures due to missing files in target folder

[jira] [Commented] (ATLAS-1827) HiveHookIT failures due to missing files in target folder

[jira] [Created] (ATLAS-1827) HiveHookIT failures due to missing files in target folder

Re: Review Request 59494: ATLAS-1804 Allow PAM for authentication

[jira] [Created] (ATLAS-1826) UI Create Entity : Type doesn't list the inverseRef constrained attribute options when name length is less than 3.

26 matches

Site Navigation

Mail list logo

Footer information