Re: Consensus for migration of git repositories to gitbox

2019-01-11 Thread David Radley
Hi Madhan,
That is a good idea - I think a couple of Atlas changes are going in or 
are in progress make the Atlas connector work, around deletes and saving 
reference copies, then we could branch,
many thanks , David.



From:   Madhan Neethiraj 
To: David Radley 
Cc: "dev@atlas.apache.org" 
Date:   11/01/2019 17:49
Subject:Re: Consensus for migration of git repositories to gitbox



David,
 
ATLAS-2987 (Update component versions of Atlas to use Hadoop3, HBase2 and 
Solr7) is the only big change I am aware of in the pipeline i.e. under 
review.
 
To minimize potential impacts due to commits that go into master, I would 
suggest you to consider using branch-1.0 for your demo. Only bugfixes 
should go into maint branches (like branch-1.0), hence it should be safer.
 
Thanks,
Madhan
 
 
 
From: David Radley 
Date: Thursday, January 10, 2019 at 2:44 AM
To: Madhan Neethiraj 
Cc: "dev@atlas.apache.org" 
Subject: Re: Consensus for migration of git repositories to gitbox
 
Hi Madhan, 
+1 I like the idea of being pro active with this. 
  
We are looking to demo master Atlas code at a conference in the following 
week, so we need to have a working Atlas master build, so it would be good 
to make this change asap. 

On a related subject, would it be possible for the community to not make 
any big changes (types, software level changes or API changes) coming up 
to the week of the 16th of February - so the Atlas / Egeria demo continues 
to build. If there are big changes you need in, could I request that the 
community be informed and given a change to assess the impact during this 
period and potentially delay pervasive changes, 
many thanks , David. 


From:Madhan Neethiraj  
To:"dev@atlas.apache.org"  
Date:10/01/2019 07:37 
Subject:Re: Consensus for migration of git repositories to gitbox 




+1

Thanks Apoorv for volunteering. Please keep the community updated of the 
progress and actions to be done by each to access the migrated repo.

Thanks,
Madhan




On 1/9/19, 11:01 PM, "Ashutosh Mestry"  wrote:

   +1 for migrating Atlas repositories.
 
   Thanks Apoorv!
 
   ~ ashutosh
 
   On 1/9/19, 10:58 PM, "Nixon Rodrigues" 
 wrote:
 
   Thanks Apoorv for volunteering this effort
 
   +1 for migrating Atlas repositories to gitbox server.
 
   Regards,
   Nixon
 
 
 
   On Thu, Jan 10, 2019 at 11:50 AM Apoorv Naik 
 wrote:
 
   > Hi Atlas devs,
   >
   > There's an ASF initiative to migrate the git-wip-us-apache.org
   > repositories
   > to the gitbox server before Feb 7, after which it'll be mass 
migrated
   > without a warning. I volunteer to take this migration effort to 
avoid any
   > disruption to commit activity.
   >
   > PMCs and committers, please share your +1 or -1 vote for this 
move (this is
   > the first step in starting the migration)
   >
   > *Apoorv Naik*
   >
 
 
 





Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU



Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


[jira] [Commented] (ATLAS-1773) Create the OMRS Connector for Atlas

2019-01-10 Thread David Radley (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739509#comment-16739509
 ] 

David Radley commented on ATLAS-1773:
-

[~grahamwallis] I wonder if you could add the link to this Jira to the 
associated RBT(s) for the latest patches. I assume you are looking for the RBTs 
to be reviewed

> Create the OMRS Connector for Atlas
> ---
>
> Key: ATLAS-1773
> URL: https://issues.apache.org/jira/browse/ATLAS-1773
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 1.0.0
>Reporter: Mandy Chessell
>Assignee: Graham Wallis
>Priority: Major
> Attachments: ATLAS-1773-2018-12-21.patch, 
> ATLAS-2810-2019-01-10.patch, ATLAS-2939-2018-12-21.patch, 
> ATLAS-2985-2019-01-10.patch
>
>
> This JIRA provides the definition of the OMRS Connector API and an 
> implementation of this API for a local Apache Atlas metadata repository and 
> for the OMRS REST API.
> The OMRS Connector has 3 API groups
> * The types API - this is the metadata API for a metadata repository
> * The entity and relationships APIs that provide the type-agnostic interfaces 
> that can access any type - even those added dynamically
> * The fine-grained type-safe APIs that are generated from the addons models 
> in the build.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Consensus for migration of git repositories to gitbox

2019-01-10 Thread David Radley
Hi Madhan, 
+1 I like the idea of being pro active with this.
 
We are looking to demo master Atlas code at a conference in the following 
week, so we need to have a working Atlas master build, so it would be good 
to make this change asap. 

On a related subject, would it be possible for the community to not make 
any big changes (types, software level changes or API changes) coming up 
to the week of the 16th of February - so the Atlas / Egeria demo continues 
to build. If there are big changes you need in, could I request that the 
community be informed and given a change to assess the impact during this 
period and potentially delay pervasive changes, 
many thanks , David. 


From:   Madhan Neethiraj 
To: "dev@atlas.apache.org" 
Date:   10/01/2019 07:37
Subject:Re: Consensus for migration of git repositories to gitbox



+1

Thanks Apoorv for volunteering. Please keep the community updated of the 
progress and actions to be done by each to access the migrated repo.

Thanks,
Madhan




On 1/9/19, 11:01 PM, "Ashutosh Mestry"  wrote:

+1 for migrating Atlas repositories.
 
Thanks Apoorv!
 
~ ashutosh
 
On 1/9/19, 10:58 PM, "Nixon Rodrigues" 
 wrote:
 
Thanks Apoorv for volunteering this effort
 
+1 for migrating Atlas repositories to gitbox server.
 
Regards,
Nixon
 
 
 
On Thu, Jan 10, 2019 at 11:50 AM Apoorv Naik 
 wrote:
 
> Hi Atlas devs,
>
> There's an ASF initiative to migrate the git-wip-us-apache.org
> repositories
> to the gitbox server before Feb 7, after which it'll be mass 
migrated
> without a warning. I volunteer to take this migration effort to 
avoid any
> disruption to commit activity.
>
> PMCs and committers, please share your +1 or -1 vote for this 
move (this is
> the first step in starting the migration)
>
> *Apoorv Naik*
>
 
 
 





Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


Re: [DRAFT] Board report for Apache Atlas: December 2018

2018-12-18 Thread David Radley
Hi Madhan,
That is a shame. I hope we can work together so by the next report, the 
Egeria connector for Atlas will be in the Atlas code, so we will have an 
even bigger good news story, 
 all the best, David. 



From:   Madhan Neethiraj 
To: "dev@atlas.apache.org" 
Date:   13/12/2018 22:42
Subject:Re: [DRAFT] Board report for Apache Atlas: December 2018



David,

Thanks for reviewing the report and suggestion to reference Egeria. This 
would indeed be a good item to include. However, the report was sent to 
the board yesterday - as it was the last day. I will include Egeria 
reference in the next report. Hope it is alright with you.

Thanks,
Madhan


On 12/13/18, 1:23 AM, "David Radley"  wrote:

Hi Madhan,
Enhancements have gone into Atlas to enable Atlas to be a reference 
implementation for Egeria. The Egeria Atlas connector is out for 
review. 
This is a very positive story for Atlas, that we could include in the 
draft board report,all the best, David. 
 
 
 
From:   Madhan Neethiraj 
To: "dev@atlas.apache.org" 
Date:   11/12/2018 05:41
Subject:[DRAFT] Board report for Apache Atlas: December 2018
 
 
 
Atlas team,
 
 
 
Please review the draft board report below and send your 
feedback/comments.
 
 
 
Thanks,
 
Madhan
 
 
 
 
 
## Description:
 
  Apache Atlas is a scalable and extensible set of core foundational
 
  governance services that enables enterprises to effectively and 
efficiently
 
  meet their compliance requirements within Hadoop and allows 
integration 
with
 
  the complete enterprise data ecosystem
 
 
 
## Issues:
 
  There are no issues requiring board attention at this time.
 
 
 
## Activity:
 
  - released 0.8.3 on 10/31/2018
 
  - released 1.1.0 on 09/17/2018 
 
  - working on 2.0.0 release, to support Hadoop 3, HBase 2, Solr 7, 
Kafka 
2, Hive 3
 
  - updated to support Hadoop trusted-proxy authentication
 
  - updated lineage UI to support entity-type specific icons, 
customizable 
depth, option to hide process entities
 
  - performance related fixes in Hive hook and notification processing
 
  - model enhancements to support soft-ref
 
  - export/import enhancements to create audit entries containing 
summary 
of the operation
 
 
 
## Health report:
 
  - 1 new contributor added in last 3 months: Nikhil Bonte
 
 
 
## PMC changes:
 
  - Currently 33 PMC members
 
  - No new PMC members added in last 3 months
 
  - Last PMC member addition was on 6/21/2017
 
 
 
## Committer base changes:
 
  - Currently 38 committers
 
  - 1 new committer was added in last 3 months: Ramesh Mani
 
  - Last addition to committer role was on 10/15/2018
 
 
 
## Releases:
 
  2.0.0plan to release by 12/31/2018
 
  0.8.3was released on 10/31/2018
 
  1.1.0was released on 09/17/2018
 
  1.0.0was released on 06/02/2018
 
  0.8.2was released on 02/05/2018
 
  1.0.0-alpha  was released on 01/25/2018
 
  0.8.1was released on 08/29/2017
 
  0.8-incubating   was released on 03/16/2017
 
  0.7.1-incubating was released on 01/26/2017
 
  0.7-incubating   was released on 07/09/2016
 
  0.6-incubating   was released on 12/31/2015
 
  0.5-incubating   was released on 07/11/2015
 
 
 
 
 
 
 
 
 
 
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with 
number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 
3AU
 





Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


Re: [DRAFT] Board report for Apache Atlas: December 2018

2018-12-13 Thread David Radley
Hi Madhan,
Enhancements have gone into Atlas to enable Atlas to be a reference 
implementation for Egeria. The Egeria Atlas connector is out for review. 
This is a very positive story for Atlas, that we could include in the 
draft board report,all the best, David. 



From:   Madhan Neethiraj 
To: "dev@atlas.apache.org" 
Date:   11/12/2018 05:41
Subject:[DRAFT] Board report for Apache Atlas: December 2018



Atlas team,

 

Please review the draft board report below and send your 
feedback/comments.

 

Thanks,

Madhan

 

 

## Description:

  Apache Atlas is a scalable and extensible set of core foundational

  governance services that enables enterprises to effectively and 
efficiently

  meet their compliance requirements within Hadoop and allows integration 
with

  the complete enterprise data ecosystem

 

## Issues:

  There are no issues requiring board attention at this time.

 

## Activity:

  - released 0.8.3 on 10/31/2018

  - released 1.1.0 on 09/17/2018 

  - working on 2.0.0 release, to support Hadoop 3, HBase 2, Solr 7, Kafka 
2, Hive 3

  - updated to support Hadoop trusted-proxy authentication

  - updated lineage UI to support entity-type specific icons, customizable 
depth, option to hide process entities

  - performance related fixes in Hive hook and notification processing

  - model enhancements to support soft-ref

  - export/import enhancements to create audit entries containing summary 
of the operation

 

## Health report:

  - 1 new contributor added in last 3 months: Nikhil Bonte

 

## PMC changes:

  - Currently 33 PMC members

  - No new PMC members added in last 3 months

  - Last PMC member addition was on 6/21/2017

 

## Committer base changes:

  - Currently 38 committers

  - 1 new committer was added in last 3 months: Ramesh Mani

  - Last addition to committer role was on 10/15/2018

 

## Releases:

  2.0.0plan to release by 12/31/2018

  0.8.3was released on 10/31/2018

  1.1.0was released on 09/17/2018

  1.0.0was released on 06/02/2018

  0.8.2was released on 02/05/2018

  1.0.0-alpha  was released on 01/25/2018

  0.8.1was released on 08/29/2017

  0.8-incubating   was released on 03/16/2017

  0.7.1-incubating was released on 01/26/2017

  0.7-incubating   was released on 07/09/2016

  0.6-incubating   was released on 12/31/2015

  0.5-incubating   was released on 07/11/2015

 

 

 




Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


Re: Review Request 69511: Atlas OMRS Connector and Event Mapper

2018-12-06 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69511/#review211091
---




open-metadata/README.md
Lines 19 (patched)
<https://reviews.apache.org/r/69511/#comment295989>

I was expecting the instructions on how to configure and run the Atlas 
connector to be in this readme.



open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/eventmapper/AtlasOMRSRepositoryEventMapper.java
Lines 64 (patched)
<https://reviews.apache.org/r/69511/#comment295974>

I am wondering what happens if you run 2 Atlas instances on the same 
machine - I assume we would need to have the ability to specify different topic 
names.



open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasEntityMapper.java
Lines 194 (patched)
<https://reviews.apache.org/r/69511/#comment295987>

remove?



open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasEntityMapper.java
Lines 344 (patched)
<https://reviews.apache.org/r/69511/#comment295988>

remove?



open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/LocalAtlasOMRSMetadataCollection.java
Lines 18 (patched)
<https://reviews.apache.org/r/69511/#comment295975>

License here twice



open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/LocalAtlasOMRSMetadataCollection.java
Lines 474 (patched)
<https://reviews.apache.org/r/69511/#comment295980>

Shouldn't we catch and throw the specific exception types? This pattern is 
used in all the methods. I am wondering why we need this try catch at all, I 
wonder if we should remove all these try catches; it seems to be there to just 
add debug tracing, if we are going to trace shouldn't we be putting out error 
tracing for the error - or is this done elsewhere? Similar to method 
processAtlasClassificationDef.



open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/LocalAtlasOMRSMetadataCollection.java
Lines 11945 (patched)
<https://reviews.apache.org/r/69511/#comment295981>

I am not sure what the CTOR is



open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/LocalAtlasOMRSMetadataCollection.java
Lines 15188 (patched)
<https://reviews.apache.org/r/69511/#comment295977>

is this right it looks like it is processing as a Boolean? Most of the 
other debug steing do not look right in this method.



open-metadata/src/main/java/org/apache/atlas/openmetadata/admin/server/spring/OpenMetadataAdminResource.java
Lines 429 (patched)
<https://reviews.apache.org/r/69511/#comment295982>

Should we remove this ?



open-metadata/src/main/java/org/apache/atlas/openmetadata/admin/server/spring/OpenMetadataAdminResource.java
Lines 454 (patched)
<https://reviews.apache.org/r/69511/#comment295983>

should we remove this?



open-metadata/src/main/java/org/apache/atlas/openmetadata/admin/server/spring/OpenMetadataAdminResource.java
Lines 477 (patched)
<https://reviews.apache.org/r/69511/#comment295984>

remove?



open-metadata/src/main/java/org/apache/atlas/openmetadata/admin/server/spring/OpenMetadataAdminResource.java
Lines 496 (patched)
<https://reviews.apache.org/r/69511/#comment295985>

remove?



open-metadata/src/main/java/org/apache/atlas/openmetadata/admin/server/spring/OpenMetadataAdminResource.java
Lines 516 (patched)
<https://reviews.apache.org/r/69511/#comment295986>

remove?


- David Radley


On Dec. 5, 2018, 3:52 p.m., Graham Wallis wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69511/
> ---
> 
> (Updated Dec. 5, 2018, 3:52 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-1773: Atlas OMRS conenctor and event mapper
> 
> 
> Diffs
> -
> 
>   open-metadata/README.md PRE-CREATION 
>   open-metadata/pom.xml PRE-CREATION 
>   
> open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/eventmapper/AtlasOMRSRepositoryEventMapper.java
>  PRE-CREATION 
>   
> open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/eventmapper/AtlasOMRSRepositoryEventMapperProvider.java
>  PRE-CREATION 
>   
> open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasAttributeDefMapper.java
>  PRE-CREATION 
>   
> open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasAttributeMapper.java
>  PRE-CREATION 
>   
> open-metadata/src/main/java

[jira] [Comment Edited] (ATLAS-2817) Update to JanusGraph 0.3.x

2018-08-23 Thread David Radley (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589863#comment-16589863
 ] 

David Radley edited comment on ATLAS-2817 at 8/23/18 8:01 AM:
--

[~apoorvnaik] I notice that Janus 0.3.0 
https://github.com/JanusGraph/janusgraph/releases
says it supports Apache HBase 1.2.6, 1.3.1, 1.4.4. In Atlas we are using HBase 
1.1. It looks like there is a mismatch. Also Atlas is using 5.5.1 and 
JanusGraph prereqs 5.5.4.

I have not checked that Cassandra ES levels that Atlas uses are compatible with 
this new level of JanusGraph.


was (Author: davidrad):
[~apoorvnaik] I notice that Janus 0.3.0 
https://github.com/JanusGraph/janusgraph/releases
says it supports Apache HBase 1.2.6, 1.3.1, 1.4.4. In Atlas we are using HBase 
1.1. It looks like there is a mismatch.

I have not checked that solr, Cassandra ES levels that Atlas uses are 
compatible with this new level of JanusGraph.

> Update to JanusGraph 0.3.x
> --
>
> Key: ATLAS-2817
> URL: https://issues.apache.org/jira/browse/ATLAS-2817
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Apoorv Naik
>Assignee: Apoorv Naik
>Priority: Major
> Attachments: 0001-Janusgraph-update-to-0.3.0.patch
>
>
> There are couple of index related improvements in this version which helps 
> resolve the slowness in few of our DSL queries. This upgrade would speed up 
> DSL execution.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (ATLAS-2817) Update to JanusGraph 0.3.x

2018-08-23 Thread David Radley (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589863#comment-16589863
 ] 

David Radley commented on ATLAS-2817:
-

[~apoorvnaik] I notice that Janus 0.3.0 
https://github.com/JanusGraph/janusgraph/releases
says it supports Apache HBase 1.2.6, 1.3.1, 1.4.4. In Atlas we are using HBase 
1.1. It looks like there is a mismatch.

I have not checked that solr, Cassandra ES levels that Atlas uses are 
compatible with this new level of JanusGraph.

> Update to JanusGraph 0.3.x
> --
>
> Key: ATLAS-2817
> URL: https://issues.apache.org/jira/browse/ATLAS-2817
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Apoorv Naik
>Assignee: Apoorv Naik
>Priority: Major
> Attachments: 0001-Janusgraph-update-to-0.3.0.patch
>
>
> There are couple of index related improvements in this version which helps 
> resolve the slowness in few of our DSL queries. This upgrade would speed up 
> DSL execution.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Review Request 68412: ATLAS-1773: Atlas OMRS Repository Connector

2018-08-17 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68412/#review207494
---




open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/eventmapper/AtlasOMRSRepositoryEventMapper.java
Lines 346 (patched)
<https://reviews.apache.org/r/68412/#comment290894>

What is the implication of leaving htis TODO in the code? I suggest at 
least there should be a Jira numebr here to track ths issue.



open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/eventmapper/AtlasOMRSRepositoryEventMapper.java
Lines 376 (patched)
<https://reviews.apache.org/r/68412/#comment290896>

siple => simple



open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/eventmapper/AtlasOMRSRepositoryEventMapper.java
Lines 425 (patched)
<https://reviews.apache.org/r/68412/#comment290897>

I think this s a duplicate line of the one above



open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/eventmapper/AtlasOMRSRepositoryEventMapper.java
Lines 434 (patched)
<https://reviews.apache.org/r/68412/#comment290898>

typo doesn#t



open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasClassificationDefMapper.java
Lines 244 (patched)
<https://reviews.apache.org/r/68412/#comment290899>

typo fial


- David Radley


On Aug. 17, 2018, 2:38 p.m., Graham Wallis wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68412/
> ---
> 
> (Updated Aug. 17, 2018, 2:38 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-1773: Atlas OMRS Repository Connector
> 
> 
> Diffs
> -
> 
>   open-metadata/pom.xml PRE-CREATION 
>   
> open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/eventmapper/AtlasOMRSRepositoryEventMapper.java
>  PRE-CREATION 
>   
> open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/eventmapper/AtlasOMRSRepositoryEventMapperProvider.java
>  PRE-CREATION 
>   
> open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasAttributeDefMapper.java
>  PRE-CREATION 
>   
> open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasAttributeMapper.java
>  PRE-CREATION 
>   
> open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasBaseTypeDefMapper.java
>  PRE-CREATION 
>   
> open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasClassificationDefMapper.java
>  PRE-CREATION 
>   
> open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasEntityDefMapper.java
>  PRE-CREATION 
>   
> open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasEntityMapper.java
>  PRE-CREATION 
>   
> open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasRelationshipDefMapper.java
>  PRE-CREATION 
>   
> open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasRelationshipMapper.java
>  PRE-CREATION 
>   
> open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasStoresProxy.java
>  PRE-CREATION 
>   
> open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasStoresProxyImpl.java
>  PRE-CREATION 
>   
> open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/Comparator.java
>  PRE-CREATION 
>   
> open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/DSLQueryHelper.java
>  PRE-CREATION 
>   
> open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/EntityDefMapper.java
>  PRE-CREATION 
>   
> open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/FamousFive.java
>  PRE-CREATION 
>   
> open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/ISpringBridge.java
>  PRE-CREATION 
>   
> open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/LocalAtlasOMRSErrorCode.java
>  PRE-CREATION 
>   
> open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/LocalAtlasOMRSMetadataCollection.java
>  PRE-CREATION 
>   
> open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/LocalAtlasOMRSReposito

Re: Review Request 68188: ATLAS-2808: Enable Atlas to support entity proxies

2018-08-17 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68188/#review207487
---


Ship it!




Ship It!

- David Radley


On Aug. 17, 2018, 12:03 p.m., Graham Wallis wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68188/
> ---
> 
> (Updated Aug. 17, 2018, 12:03 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS_2808: Support for entity proxies
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java a892551f7 
>   intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java 
> dc2be412b 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 9c4b9f232 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  2153a5a9e 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  52c374564 
> 
> 
> Diff: https://reviews.apache.org/r/68188/diff/3/
> 
> 
> Testing
> ---
> 
> Tested with saveRelationshipReferenceCopy between entities that do not exist 
> in the Atlas repository.
> 
> 
> Thanks,
> 
> Graham Wallis
> 
>



Re: Review Request 68287: ATLAS-2807: Re-evaluate classification propagation during entity delete

2018-08-10 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68287/#review207066
---




repository/src/main/java/org/apache/atlas/repository/store/graph/v1/HardDeleteHandlerV1.java
Line 56 (original)
<https://reviews.apache.org/r/68287/#comment290253>

I assume this is implementing the reporting text "Hard Delete:
In cases of hard delete, the source entity (f1) is deleted from atlas, but 
its classification vertex
(PII) continue to exist and will continue propagating to process1 and t1." 

I am not sure about this - as if a source entity is PII and it is deleted - 
I would think any downstream entities should not be PII.


- David Radley


On Aug. 10, 2018, 2:36 a.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68287/
> ---
> 
> (Updated Aug. 10, 2018, 2:36 a.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and 
> Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-2807
> https://issues.apache.org/jira/browse/ATLAS-2807
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> The current behavior - when we delete an entity f1, all tags associate to f1 
> which got propagated
> 
> 
> to downstream entities are removed – PII tag propagated to process1 and t1 is 
> removed
> 
> 
> Proposed Change
> 
> 
> Soft Delete:
> 
> 
> When entity f1 is deleted, retain the propagated classification edges, so the 
> downstream entities
> 
> 
> – process1 and t1 continue to have PII classification associated to them
> 
> 
> Hard Delete:
> 
> 
> In cases of hard delete, the source entity (f1) is deleted from atlas, but 
> its classification vertex
> 
> 
> (PII) continue to exist and will continue propagating to process1 and t1.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 7988d1d81 
>   intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 328b76734 
>   intg/src/main/java/org/apache/atlas/model/instance/AtlasClassification.java 
> 3132379b9 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> d328873f1 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java
>  c6be4061b 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  4a0924bbe 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/HardDeleteHandlerV1.java
>  a95e6898e 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/SoftDeleteHandlerV1.java
>  41e65d4a3 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
>  28a26ab4f 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  f57ce99f7 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  8f4faaff8 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
>  58005c458 
>   
> repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java
>  8ceeedb01 
>   webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java 
> e01cba92d 
>   webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java fd331faa0 
>   webapp/src/test/java/org/apache/atlas/web/adapters/TestEntityREST.java 
> 78bd53ce7 
> 
> 
> Diff: https://reviews.apache.org/r/68287/diff/1/
> 
> 
> Testing
> ---
> 
> Validated basic cases using REST clirnt.
> 
> 
> PreCommit: In progress
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>



Re: Review Request 68188: ATLAS-2808: Enable Atlas to support entity proxies

2018-08-09 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68188/#review207015
---




common/src/main/java/org/apache/atlas/repository/Constants.java
Lines 96 (patched)
<https://reviews.apache.org/r/68188/#comment290179>

I suggest a comment to document what the flag is. In the style of the 
homeId comment.



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
Lines 174 (patched)
<https://reviews.apache.org/r/68188/#comment290180>

I am wondering what your thoughts are on validation in Atlas. Specifically 
for cases like:
- the entity is a proxy but has no homeid. 
- the entity is a proxy but has none manditory attributes set which should 
not be returned
- the entity is a proxy so should not be in search results

As coded the caller is responsible for this sort of validation.


- David Radley


On Aug. 3, 2018, 1:33 p.m., Graham Wallis wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68188/
> ---
> 
> (Updated Aug. 3, 2018, 1:33 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS_2808: Support for entity proxies
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 
> 7988d1d81595eb3d65ecca2480c0e67d17f53573 
>   intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java 
> dc2be412b9ec1b07d7c7befc722810273464cd3f 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> d328873f1f2695954d79e476a40166f48ffda482 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  f57ce99f764a5c779f1cfa4325849c15ac804b2d 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  448d03bc069869db85cd86e14021a3fa59b57e33 
> 
> 
> Diff: https://reviews.apache.org/r/68188/diff/1/
> 
> 
> Testing
> ---
> 
> Tested with saveRelationshipReferenceCopy between entities that do not exist 
> in the Atlas repository.
> 
> 
> Thanks,
> 
> Graham Wallis
> 
>



[jira] [Commented] (ATLAS-2774) Options for hard and soft delete of instances

2018-07-04 Thread David Radley (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532700#comment-16532700
 ] 

David Radley commented on ATLAS-2774:
-

Sounds like a good change. A couple of thoughts:
- we may need to have new return codes if we attempt a purge and it does not 
work.
- I assume a soft delete of an entity soft deletes any associated relationships 
around it. I guess we may need special logic to handle deletion when there is a 
 composition relationship. It would make sense that deleting the container - 
deletes the children entities in this case (recursively) or is not allowed 
until the children are themselves deleted.

> Options for hard and soft delete of instances
> -
>
> Key: ATLAS-2774
> URL: https://issues.apache.org/jira/browse/ATLAS-2774
> Project: Atlas
>  Issue Type: New Feature
>Reporter: Graham Wallis
>Priority: Major
>
> *Overview*
> For Entities and Relationships, the following delete behaviors are desired.
> Atlas can be configured to offer hard delete (only). Alternatively it can be 
> configured to offer soft-delete. Configuration is achieved by setting the 
> delete handler configuration property (see atlas-application.properties) to 
> either the SoftDeleteHandlerV1 or the HardDeleteHandlerV1. The default (for 
> when the property is not set) is to use the SoftDeleteHandlerV1.
> The AtlasEntityStoreV2 deleteById() and deleteByIds() and methods and the 
> AtlasRelationshipStoreV2 deleteById() method are sensitive to the above 
> configuration. When the configuration is soft these methods will perform a 
> soft delete; when it is hard they will perform a hard delete.
> In addition to the above methods, the AtlasEntityStoreV2 and 
> AtlasRelationshipStoreV2 offer a (new) purgeById() method that ALWAYS 
> performs a hard delete. This is true regardless of which delete handler has 
> been configured. When the configuration is hard, the purgeById() methods and 
> deleteById() methods are essentially equivalent. When the configuration is 
> soft, the purgeById() methods continue to provide a hard delete capability.
> The AtlasEntityStoreV2 deleteById(), deleteByIds() and purgeById() methods 
> will delete the specified entities and any relationships to which they are 
> connected. The AtlasRelationshipStoreV2 deleteById() and purgeById() methods 
> will delete the specified relationship. Deletion of a relationship may cause 
> an upate to an entity to which the relationhsip is connected, if it changes 
> the propagation of classifications, for example.
> In a hard delete or purge operation, an affected entity and relationship will 
> be removed from the graph and will not be returned in response to any future 
> queries. This is true regardless of whether the affected entity or 
> relationship has already been soft deleted or not. i.e. it's status could 
> initially be ACTIVE or DELETED. Following the operation it will not exist.
> In a soft delete, the affected entities and relationships (that initially 
> have status ACTIVE) are updated to set the status to DELETED. These instances 
> can still be returned from queries and searches if the appropriate control is 
> selected (to include deleted instances). Without that control selected they 
> will not be returned.
> *Use Case for soft delete*
> Soft delete provides a 'safe' means of deleting instances from the repository 
> whilst preserving an audit trail and (if supported by the repository) 
> enabling a deleted instance to be restored (to ACTIVE status).
> *Use Cases for hard delete*
> One use case for hard delete is to permanently remove soft-deleted items some 
> period of time after they were soft deleted.
>  Another use case for hard delete is to is to correct (clean up after) a 
> mistake in which a potentially large batch of incomplete/corrupted/wrong 
> metadata is loaded and needs to be fully removed.
> *Use Scenario for an OMAS user*
> An OMAS provides a delete function that does NOT offer the user a choice of 
> hard/soft. The OMAS's delete function will invoke the relevant soft delete 
> method provided by the OMRS - deleteEntity or deleteRelationship. These are 
> both soft delete methods.
> The Atlas OMRS Connector's deleteEntity/deleteRelationship methods will check 
> whether Atlas is configured for hard or soft delete. If Atlas is configured 
> for hard delete then soft-delete is not possible, and the method will throw a 
> FunctionNotSupported exception which is caught by the OMAS.
> On catching this exception the OMAS author should automatically escalate the 
> soft delete to a hard delete by calling eith

[jira] [Commented] (ATLAS-2708) AWS S3 data lake typedefs for Atlas

2018-06-11 Thread David Radley (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507807#comment-16507807
 ] 

David Radley commented on ATLAS-2708:
-

[~barbara] I have had a quick look at the json file, it looks like you are 
using the old style of definition. I would recommend that that you use the new 
RelationshipDef style rather than constraints. This is more descriptive of 
relationships - it allows you to specify attributes on the relationship as 
well.  I think the constraints style is only there for so that the Hadoop types 
could continue to work.

> AWS S3 data lake typedefs for Atlas
> ---
>
> Key: ATLAS-2708
> URL: https://issues.apache.org/jira/browse/ATLAS-2708
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Reporter: Barbara Eckman
>Assignee: Barbara Eckman
>Priority: Critical
> Attachments: all_datalake_typedefs.json
>
>
> Currently the base types in Atlas do not include AWS data lake objects. It 
> would be nice to add typedefs for AWS data lake objects (buckets and 
> pseudo-directories) and lineage processes that move the data from another 
> source (e.g., kafka topic) to the data lake.  For example:
>  * AWSS3PseudoDir type represents the pseudo-directory “prefix” of objects in 
> an S3 bucket.  For example, in the case of an object with key 
> “myWork/Development/Projects1.xls”, “myWork/Development” is the 
> pseudo-directory.  It supports:
>  ** Array of avro schemas that are associated with the data in the 
> pseudo-directory (based on Avro schema extensions outlined in ATLAS-2694)
>  ** what type of data it contains, e.g., avro, json, unstructured
>  ** time of creation
>  * AWSS3BucketLifeCycleRule type represents a rule specifying a transition of 
> the data in a bucket to a storageClass after a specific time interval, or 
> expiration.  For example, transition to GLACIER after 60 days, or expire 
> (i.e. be deleted) after 90 days:
>  ** ruleType (e.g., transition or expiration)
>  ** time interval in days before rule is executed  
>  ** storageClass to which the data is transitioned (null if ruleType is 
> expiration)
>  * AWSTag type represents a tag-value pair created by the user and associated 
> with an AWS object.
>  **  tag
>  ** value
>  * AWSCloudWatchMetric type represents a storage or request metric that is 
> monitored by AWS CloudWatch and can be configured for a bucket
>  ** metricName, for example, “AllRequests”, “GetRequests”, 
> TotalRequestLatency, BucketSizeBytes
>  ** scope: null if entire bucket; otherwise, the prefixes/tags that filter or 
> limit the monitoring of the metric.
>  * AWSS3Bucket type represents a bucket in an S3 instance.  It supports:
>  ** Array of AWSS3PseudoDirectories that are associated with objects stored 
> in the bucket 
>  ** AWS region
>  ** IsEncrypted (boolean) 
>  ** encryptionType, e.g., AES-256
>  ** S3AccessPolicy, a JSON object expressing access policies, eg GetObject, 
> PutObject
>  ** time of creation
>  ** Array of AWSS3BucketLifeCycleRules that are associated with the bucket 
>  ** Array of AWSS3CloudWatchMetrics that are associated with the bucket or 
> its tags or prefixes
>  ** Array of AWSTags that are associated with the bucket
>  * Generic dataset2Dataset process to represent movement of data from one 
> dataset to another.  It supports:
>  ** array of transforms performed by the process 
>  ** map of tag/value pairs representing configurationParameters of the process
>  ** inputs and outputs are arrays of dataset objects, e.g., kafka topic and 
> S3 pseudo-directory.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (ATLAS-1773) Create the OMRS Connector for Atlas

2018-06-10 Thread David Radley (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507324#comment-16507324
 ] 

David Radley commented on ATLAS-1773:
-

[~grahamwallis]  I have committed 2745. I then tried to follow the notes. I am 
on the latest master. The patch for ATLAS-1773 applied. 
I then ran 
git am < ATLAS-1773-2018-06-07.patch 
and got:
Applying: ATLAS-1773: Atlas OMRS Connector
✔ ~/atlasreview [master ↑·1|…2] 
09:20 $ git am < ATLAS-2665-Add-OMAG-API-in-Atlas-v7.patch 
Applying: ATLAS-2665 Add OMAG-API in Atlas
error: patch failed: omag-api/pom.xml:41
error: omag-api/pom.xml: patch does not apply
error: patch failed: omag-server/pom.xml:53
error: omag-server/pom.xml: patch does not apply
error: patch failed: omas-assetconsumer/pom.xml:39
error: omas-assetconsumer/pom.xml: patch does not apply
error: patch failed: omas-connectedasset/pom.xml:40
error: omas-connectedasset/pom.xml: patch does not apply
error: patch failed: omrs/pom.xml:39
error: omrs/pom.xml: patch does not apply
Patch failed at 0001 ATLAS-2665 Add OMAG-API in Atlas
The copy of the patch that failed is found in: .git/rebase-apply/patch

I did a 3 way merge choosing the patch content over head to apply the content. 
This was all around pom dependancies it did not like. This built successfully.
I got the log error even though I have made your suggested change (though 
possibly made a mistake). I have changed the omag-server pom to:

omag-server
Open Metadata and Governance (OMAG) Server

Open Metadata and Governance (OMAG) server for running open metadata 
function outside of the Apache Atlas server.


jar




org.springframework.boot
spring-boot-starter
1.5.7.RELEASE


ch.qos.logback
*





org.springframework.boot
spring-boot-starter-web
1.5.7.RELEASE

  

> Create the OMRS Connector for Atlas
> ---
>
> Key: ATLAS-1773
> URL: https://issues.apache.org/jira/browse/ATLAS-1773
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 1.0.0
>Reporter: Mandy Chessell
>Assignee: Graham Wallis
>Priority: Major
> Attachments: ATLAS-1773-2018-06-07.patch, Notes on running the OMRS 
> stack including the Atlas OMRS connector.txt, 
> RELATIONSHIP_DEFS_AS_ASSOCIATIONS.patch
>
>
> This JIRA provides the definition of the OMRS Connector API and an 
> implementation of this API for a local Apache Atlas metadata repository and 
> for the OMRS REST API.
> The OMRS Connector has 3 API groups
> * The types API - this is the metadata API for a metadata repository
> * The entity and relationships APIs that provide the type-agnostic interfaces 
> that can access any type - even those added dynamically
> * The fine-grained type-safe APIs that are generated from the addons models 
> in the build.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Resolved] (ATLAS-2745) Atlas EnumDefStore does not remember default value

2018-06-10 Thread David Radley (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley resolved ATLAS-2745.
-
Resolution: Fixed


commit ce5ffeb710721b78135e8b3c3ebe593bd70d2cdf (HEAD -> master, origin/master, 
origin/HEAD)
Author: Graham Wallis 
Date:   Thu Jun 7 14:25:53 2018 +0100

ATLAS-2745 - AtlasEnumDefStore should remember default value

Signed-off-by: David Radley 

> Atlas EnumDefStore does not remember default value
> --
>
> Key: ATLAS-2745
> URL: https://issues.apache.org/jira/browse/ATLAS-2745
> Project: Atlas
>  Issue Type: Bug
>Reporter: Graham Wallis
>Assignee: Graham Wallis
>Priority: Minor
> Attachments: ATLAS-2745-2018-06-07.patch
>
>
> If an enum def has a default value it is not remembered when the enum is 
> stored.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Review Request 67486: ATLAS-2745 EnumDefStore manages defaultValue

2018-06-08 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67486/#review204487
---


Ship it!




Ship It!

- David Radley


On June 7, 2018, 1:57 p.m., Graham Wallis wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67486/
> ---
> 
> (Updated June 7, 2018, 1:57 p.m.)
> 
> 
> Review request for atlas and Madhan Neethiraj.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-2745 - AtlasEnumDefStore should remember default value
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEnumDefStoreV2.java
>  7d7233fe8c6e9b8449df7798afdc7216e30d538b 
> 
> 
> Diff: https://reviews.apache.org/r/67486/diff/1/
> 
> 
> Testing
> ---
> 
> Bring-up testing loading Open Metadata types including EnumDefs with default 
> values.
> 
> 
> Thanks,
> 
> Graham Wallis
> 
>



Re: [VOTE] Release Apache Atlas version 1.0.0 - rc2

2018-06-01 Thread David Radley
Thanks Madhan for putting out the release candidate 2 for Apache Atlas 1.0


  - Built with embedded-hbase-solr profile successfully with no test 
failures
  - built with clean install with no test failures.
  - Ran quick start and the UI.

+1 for release candidate #2.

all the best, David. 

From:   Sarath Subramanian 
To: dev@atlas.apache.org
Date:   31/05/2018 17:17
Subject:Re: [VOTE] Release Apache Atlas version 1.0.0 - rc2



Thanks Madhan for putting out the release candidate 2 for Apache Atlas 1.0

   - Verified checksum and signature
   - Built with embedded-hbase-solr profile successfully with no test
   failures.
   - Validated new functionalities - glossary, relationships, lineage, tag
   propagation and UI enhancements

+1 for release candidate #2.


Thanks,
Sarath Subramanian


On Wed, May 30, 2018 at 12:31 AM, Madhan Neethiraj 
wrote:

> Atlas team,
>
> Apache Atlas 1.0.0 release candidate #2 is now available for a vote 
within
> dev community.
>
> Following commit went into branch-1.0 since the last release candidate,
> #1, was sent yesterday.
>   ATLAS-2726: updated migration-import to add __name property to
> classification edges
>
> Links to the release artifacts are given below. Please review and vote.
>
> The vote will be open for at least 72 hours or until necessary votes are
> reached.
>   [ ] +1 approve
>   [ ] +0 no opinion
>   [ ] -1 disapprove (and reason why)
>
> Thanks,
> Madhan
>
>
> List of issues addressed in this release: 
https://issues.apache.org/

> jira/issues/?jql=project%20%3D%20Atlas%20AND%20resolution%
> 20!%3D%20Unresolved%20%20AND%20fixVersion%20%3D%201.0.0%
> 20ORDER%20BY%20key%20DESC
>
> Git tag for the release: 
https://github.com/apache/

> atlas/tree/release-1.0.0-rc2
>
> Sources for the release: 
https://dist.apache.org/repos/

> dist/dev/atlas/1.0.0-rc2/apache-atlas-1.0.0-sources.tar.gz
>
> Documentation for the release: 
http://atlas.apache.org/1.0.0-

> rc0/index.html
>
> Source release verification:
>   PGP Signature: 
https://dist.apache.org/repos/dist/dev/atlas/1.0.0-rc2/

> apache-atlas-1.0.0-sources.tar.gz.asc
>   SHA512 Hash: 
https://dist.apache.org/repos/dist/dev/atlas/1.0.0-rc2/

> apache-atlas-1.0.0-sources.tar.gz.sha512
>
> Keys to verify the signature of the release artifacts are available at:
> 
https://dist.apache.org/repos/dist/dev/atlas/KEYS

>
>
>
>



Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


Re: Problem deleting edges with 1.0.0 - pre rc

2018-06-01 Thread David Radley
Hi Pierre,
Just so I am clear, does"Entity A -> Composition Relationship -> Entity B" 
mean that B or A is the container?

If B is the container then - you should be able to delete A. You are 
saying that B can be deleted and leaves its part A. If this is the case 
then I would suggest that the code could be enhanced to either stop this 
sort of deletion or delete it and all of its composed parts. I suggest 
this is not a stopper for v1.0, as you have a circumvention where you 
manually delete each entity. 

If this is an issue - I suggest looking at AtlasEntityStoreV1 deleteById 
to add some checking around composition relationships prior to deletion, 

   all the best, David. 
 




From:   Pierre Padovani 
To: dev@atlas.apache.org
Date:   31/05/2018 17:47
Subject:Re: Problem deleting edges with 1.0.0 - pre rc



Sorry for the spam... One last data point. It looks like the deletion of B
is the issue. I can delete A, but this doesn't delete B.

On Thu, May 31, 2018 at 11:21 AM, Pierre Padovani <
pierre.padov...@civitaslearning.com> wrote:

> The level we are using is a few weeks old, but I didn't see an issue 
that
> fixed this in the interim.
>
> Problem:
>
> Entity A -> Composition Relationship -> Entity B
>
> Delete Entity A and Entity B
>
> I get the following exception trace:
>
> java.lang.IllegalArgumentException: Invalid edge label
> r:DiscoveryPackTables: expected 2 or 3 label components but found 1 at
> 
org.apache.atlas.repository.graph.AtlasEdgeLabel.(AtlasEdgeLabel.java:37)
> at org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.
> getAttributeForEdge(DeleteHandlerV1.java:722) at
> 
org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.deleteVertex(DeleteHandlerV1.java:865)
> at org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.
> deleteTypeVertex(DeleteHandlerV1.java:718) at 
org.apache.atlas.repository.
> store.graph.v1.DeleteHandlerV1.deleteEntities(DeleteHandlerV1.java:140)
> at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.
> deleteVertices(AtlasEntityStoreV1.java:704) at
> 
org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.deleteById(AtlasEntityStoreV1.java:297)
> at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1$$
> FastClassBySpringCGLIB$$80c00649.invoke() at
> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> at 
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.
> invokeJoinpoint(CglibAopProxy.java:738) at org.springframework.aop.
> 
framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
> at org.apache.atlas.GraphTransactionInterceptor.invoke(
> GraphTransactionInterceptor.java:75) at 
org.springframework.aop.framework.
> ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> at org.springframework.aop.framework.CglibAopProxy$
> DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) at
> org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1$$
> EnhancerBySpringCGLIB$$2072786c.deleteById() at
> org.apache.atlas.web.rest.EntityREST.deleteByGuid(EntityREST.java:327) 
at
> sun.reflect.GeneratedMethodAccessor231.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43) at 
java.lang.reflect.Method.invoke(Method.java:498)
> at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(
> JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.
> model.method.dispatch.AbstractResourceMethodDispatch
> 
Provider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> at com.sun.jersey.server.impl.model.method.dispatch.
> 
ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.
> accept(HttpMethodRule.java:302) at com.sun.jersey.server.impl.
> uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.
> accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.
> uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(
> RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.
> 
application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
> at com.sun.jersey.server.impl.application.WebApplicationImpl._
> handleRequest(WebApplicationImpl.java:1473) at 
com.sun.jersey.server.impl.
> 
application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
> at com.sun.jersey.server.impl.application.WebApplicationImpl.
> handleRequest(WebApplicationImpl.java:1409) at
> 
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
> at com.sun.jersey.spi.container.servlet.ServletContainer.
> service(ServletContainer.java:558) at com.sun.jersey.spi.container.
> servlet.ServletContainer.service(ServletContainer.java:733) at
> 

[jira] [Resolved] (ATLAS-2523) Import relationship, preserving existing GUID

2018-05-31 Thread David Radley (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-2523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley resolved ATLAS-2523.
-
Resolution: Fixed

commit 72cc566a7f23bef86c00afb6c7ef136c1d01b424 (HEAD -> master, origin/master, 
origin/HEAD)
Author: Graham Wallis 
Date:   Fri May 25 13:35:16 2018 +0100

ATLAS-2523 Add HomeId and allow GUIDs to be specified on creates

> Import relationship, preserving existing GUID
> -
>
> Key: ATLAS-2523
> URL: https://issues.apache.org/jira/browse/ATLAS-2523
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Graham Wallis
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: ATLAS-2523-2018-05-25.patch
>
>
> To add a relationship defined externally (in another metadata repository), I 
> want to be able to add the relationship and maintain the existing GUID.
> This is possible for an entity by performing an import. It does not seem to 
> be possible for relationships., because the relationship store always 
> generates a new GUID, which overwrites the existing GUID.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Review Request 67332: ATLAS-2709: added model for RDBMS types

2018-05-26 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67332/#review203934
---




addons/models/2000-RDBMS/2010-rdbms_model.json
Lines 7 (patched)
<https://reviews.apache.org/r/67332/#comment286309>

I wonder if you have seen the open metadata types. There is quite a 
sophisticated representation of schemas and databases and the like. See the 
wiki for the details of the types; the ones involving schemas are documented 
here: https://cwiki.apache.org/confluence/display/ATLAS/Area+5+-+Standards

The idea is that these types and the others would be the standard 
representations of RDMS concepts. I am interested to see whether these types 
are rich enough to meet your requirements.


- David Radley


On May 25, 2018, 11:48 p.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67332/
> ---
> 
> (Updated May 25, 2018, 11:48 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-2709
> https://issues.apache.org/jira/browse/ATLAS-2709
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> added model containing RDBMS typedefs - author: Barbara Eckman
> 
> 
> Diffs
> -
> 
>   addons/models/2000-RDBMS/2010-rdbms_model.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/67332/diff/1/
> 
> 
> Testing
> ---
> 
> - verified that Atlas successfully loads RDBMS model file contents
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>



Re: Review Request 67083: ATLAS-2665 - Add OMAG-API in Atlas

2018-05-17 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67083/#review203328
---




omrs/src/main/java/org/apache/atlas/omrs/admin/OMRSConfigurationFactory.java
Line 55 (original), 55 (patched)
<https://reviews.apache.org/r/67083/#comment285453>

This should be plural to be consistent with access-services. there are a 
few otehr instances of this in the patch



omrs/src/main/java/org/apache/atlas/omrs/admin/OMRSConfigurationFactory.java
Line 291 (original), 291 (patched)
<https://reviews.apache.org/r/67083/#comment285454>

this is not what the code does. For this example wouldn't the code would 
produce urls starting with. 


ocalhost:8080/open-metadata/localServerName/open-metadata/repository-service/



omrs/src/main/java/org/apache/atlas/omrs/admin/OMRSConfigurationFactory.java
Line 306 (original), 306 (patched)
<https://reviews.apache.org/r/67083/#comment285455>

should be repository-services to be consistant



omrs/src/main/java/org/apache/atlas/omrs/rest/server/OMRSRepositoryRESTServices.java
Line 202 (original), 201 (patched)
<https://reviews.apache.org/r/67083/#comment285456>

Are we missing the /users/ before the {userId} in the path? The same for 
the rest of this file.



omrs/src/main/java/org/apache/atlas/omrs/rest/server/spring/OMRSRepositoryResource.java
Lines 84 (patched)
<https://reviews.apache.org/r/67083/#comment285457>

plural for consistancy ?



webapp/src/main/webapp/WEB-INF/openMetadataContext.xml
Lines 3 (patched)
<https://reviews.apache.org/r/67083/#comment285452>

Bad end of line character on this and the next 6 lines


- David Radley


On May 17, 2018, 12:10 p.m., Bogdan Sava wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67083/
> ---
> 
> (Updated May 17, 2018, 12:10 p.m.)
> 
> 
> Review request for atlas, Daniela Otelea, David Radley, Graham Wallis, Mandy 
> Chessell, and Ruxandra Rosu.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Create configuration for OMAG API in Atlas
> 
> 
> Diffs
> -
> 
>   om-fwk-ocf/pom.xml 8b1cefd 
>   omag-api/pom.xml 2338286 
>   
> omag-api/src/main/java/org/apache/atlas/omag/admin/OMAGServerAdministration.java
>  dd19018 
>   omag-api/src/main/java/org/apache/atlas/omag/admin/OMAGServerCategory.java 
> PRE-CREATION 
>   
> omag-api/src/main/java/org/apache/atlas/omag/admin/server/OMAGServerAdminRESTServices.java
>  PRE-CREATION 
>   
> omag-api/src/main/java/org/apache/atlas/omag/admin/server/OMAGServerAdminResource.java
>  ebc9f2a 
>   
> omag-api/src/main/java/org/apache/atlas/omag/admin/server/spring/OMAGServerAdminResource.java
>  PRE-CREATION 
>   
> omag-api/src/main/java/org/apache/atlas/omag/configuration/registration/AccessServiceDescription.java
>  1a15215 
>   
> omag-api/src/main/java/org/apache/atlas/omag/configuration/registration/AccessServiceRegistration.java
>  c2950c4 
>   omag-server/pom.xml 527c5c3 
>   
> omag-server/src/main/java/org/apache/atlas/omag/application/OMAGApplication.java
>  88397de 
>   omag-server/src/main/resources/application.properties PRE-CREATION 
>   omas-assetconsumer/pom.xml 29935e7 
>   
> omas-assetconsumer/src/main/java/org/apache/atlas/omas/assetconsumer/client/AssetConsumer.java
>  80a98f5 
>   
> omas-assetconsumer/src/main/java/org/apache/atlas/omas/assetconsumer/publisher/AssetConsumerPublisher.java
>   
>   
> omas-assetconsumer/src/main/java/org/apache/atlas/omas/assetconsumer/server/AssetConsumerRESTServices.java
>  967be5a 
>   
> omas-assetconsumer/src/main/java/org/apache/atlas/omas/assetconsumer/server/spring/AssetConsumerOMASResource.java
>  PRE-CREATION 
>   omas-connectedasset/README.md 2f431df 
>   omas-connectedasset/pom.xml 219c972 
>   
> omas-connectedasset/src/main/java/org/apache/atlas/omas/connectedasset/server/ConnectedAssetRESTServices.java
>  8ae6c37 
>   
> omas-connectedasset/src/main/java/org/apache/atlas/omas/connectedasset/server/spring/ConnectedAssetOMASResource.java
>  PRE-CREATION 
>   omrs/pom.xml 50b9898 
>   
> omrs/src/main/java/org/apache/atlas/omrs/admin/OMRSConfigurationFactory.java 
> 7ae87c6 
>   
> omrs/src/main/java/org/apache/atlas/omrs/rest/server/OMRSRepositoryRESTServices.java
>  5948482 
>   
> omrs/src/main/java/org/apache/atlas/omrs/rest/server/spring/OMRSRepositoryResource.java
>  PRE-CREATION 
>   
> omrs/src/test/java/org/apache/atlas/omrs/archivemanager/opentypes/TestOpenMetadataTypes

Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0

2018-05-16 Thread David Radley


> On May 7, 2018, 11:41 p.m., Ashutosh Mestry wrote:
> > On executing ./bin/atlas_start.py displays these messages:
> > ./target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/conf/atlas-env.sh:
> >  line 65: MANAGE_EMBEDDED_CASSANDRA=${cassandra.embedded}: bad substitution
> > ./target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/conf/atlas-env.sh:
> >  line 68: MANAGE_LOCAL_ELASTICSEARCH=${elasticsearch.managed}: bad 
> > substitution
> > Exception: ('Could not find hbase-site.xml in %s. Please set env var 
> > HBASE_CONF_DIR to the hbase client conf dir', 
> > './target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/hbase/conf')
> > Traceback (most recent call last):
> >   File "./bin/atlas_start.py", line 163, in 
> >   File "./bin/atlas_start.py", line 92, in main
> > Exception: ('Could not find hbase-site.xml in %s. Please set env var 
> > HBASE_CONF_DIR to the hbase client conf dir', 
> > './target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/hbase/conf')
> 
> Pierre Padovani wrote:
> Fixed in the updated patch.
> 
> Pierre Padovani wrote:
> Any updates on this review?

Hi Pierre - there are outstanding issues against this entry - which normally 
indicates there is work for the controbuter to do. These need to be fixed or 
dropped before a subsequent review.


- David


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review202603
---


On May 8, 2018, 3 p.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> ---
> 
> (Updated May 8, 2018, 3 p.m.)
> 
> 
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates 
> documentation.
> 
> Included with this patch is an update to the berkley-elasticsearch profile to 
> automatically download and include elasticsearch as a side application much 
> like solr is. Updates to the start/stop/conf scripts are included as well.
> 
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to 
> /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There 
> are six constants that are incorrectly named with a '.' (dot). This is not 
> supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** 
> the field names can be collectively thought of as an object. In the case of 
> the fields defined in the Constants.java file, 'type' is defined as a string 
> field, and 'type.name' is also defined as a string field. Elasticsearch sees 
> this as an error, since it cannot convert type to an object. The fix included 
> simply changes the field names from using a '.' (dot) to an '_' (underscore). 
> This should NOT affect compatibility with hbase/solr for new installs. For 
> existing installations, a reindex will be required as the field names will 
> have changed.
> 
> **Query**: There is a way we can simplify integration/unit tests for the 
> in-memory graph store by using a maven plugin that will download and run an 
> elasticsearch node. This is nothing more than a maven change, and change to 
> the atlas-application.properties to switch to elasticsearch from solr. I did 
> not implement this, but am curious if this change would be desired. If so, 
> this can be done with a separate ticket.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 16a3aaa15 
>   distro/pom.xml 6431fd86d 
>   distro/src/bin/atlas_config.py 9062da649 
>   distro/src/bin/atlas_start.py 61d69eb21 
>   distro/src/bin/atlas_stop.py 94c3d6d46 
>   distro/src/conf/atlas-env.sh 053cbd500 
>   distro/src/main/assemblies/standalone-package.xml dc88add56 
>   docs/src/site/twiki/Configuration.twiki 63c3fce96 
>   docs/src/site/twiki/HighAvailability.twiki 4270d0974 
>   docs/src/site/twiki/InstallationSteps.twiki dca0618e3 
>   graphdb/janus/pom.xml 016a09c33 
>   pom.xml 3469a393a 
>   webapp/pom.xml 03b84087f 
> 
> 
> Diff: https://reviews.apache.org/r/66064/diff/4/
> 
> 
> Testing
> ---
> 
> We currently use this fix with our Atlas setup in a fork of the Atlas code, 
> and have found no issues with it. Additionally, with the update to the 
> berkley-elasticsearch profile, I have extensively tested that profile to make 
> sure that management of Elasticsearch functioned correctly.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



Re: Review Request 67148: ATLAS-2685: Impose displayName restrictions on Glossary, Term & Category

2018-05-16 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67148/#review203206
---




repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
Lines 69 (patched)
<https://reviews.apache.org/r/67148/#comment285300>

I am wondering why the display name is restricted with these characters - 
does it cause a probem? Where would you use the display name in a dotted 
qualfied name? I assume the display name could be Janaese characters - I am not 
sure it would be appropriate to concatinate dots and a Japanese string.  

If you want to restrict it, presumably you need to remove all punctuation 
content in all languages. 
I suggest you be extra safe for now and only allow upper case case 
letters and blanks - unless you already have nls requirements.

Then extend  as requirements come in. Out of interest I assume that the 
Janus keywords checking could effect what term names can be used.  

Fyi translation is covered in the open metadata model with relationships.


- David Radley


On May 16, 2018, 6:11 a.m., Apoorv Naik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67148/
> ---
> 
> (Updated May 16, 2018, 6:11 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-2685
> https://issues.apache.org/jira/browse/ATLAS-2685
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Glossary & term name can only have letters, spaces, underscores and dashes
> Category can only have letters and spaces
> 
> Added appropriate error codes around this check
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 3a609786a 
>   repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java 
> bc9fe2a2e 
>   repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java 
> f66165050 
> 
> 
> Diff: https://reviews.apache.org/r/67148/diff/2/
> 
> 
> Testing
> ---
> 
> PreCommit: 
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/408/console
> 
> 
> Thanks,
> 
> Apoorv Naik
> 
>



Re: Review Request 67148: ATLAS-2685: Impose displayName restrictions on Glossary, Term & Category

2018-05-16 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67148/#review203205
---




repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
Lines 139 (patched)
<https://reviews.apache.org/r/67148/#comment285299>

I see qualified name as a unique identifier, being derived from 
referenencable. In one of our meeting we talked of using the guid and name 
here. I thought we talked of deriving a path attribute post v1 - using a new 
generic Atlas mechanism.  

Display name might not be unique, as we want to support categories with the 
same name. I see display name is not the hierarchical path. It should be the 
name. 

On the policing I wonder what happens with janus keywords and whether mixed 
case are viewed as different.


- David Radley


On May 16, 2018, 6:11 a.m., Apoorv Naik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67148/
> ---
> 
> (Updated May 16, 2018, 6:11 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-2685
> https://issues.apache.org/jira/browse/ATLAS-2685
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Glossary & term name can only have letters, spaces, underscores and dashes
> Category can only have letters and spaces
> 
> Added appropriate error codes around this check
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 3a609786a 
>   repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java 
> bc9fe2a2e 
>   repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java 
> f66165050 
> 
> 
> Diff: https://reviews.apache.org/r/67148/diff/2/
> 
> 
> Testing
> ---
> 
> PreCommit: 
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/408/console
> 
> 
> Thanks,
> 
> Apoorv Naik
> 
>



Re: Review Request 67083: ATLAS-2665 - Add OMAG-API in Atlas

2018-05-15 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67083/#review203132
---




omag-api/pom.xml
Line 42 (original), 42 (patched)
<https://reviews.apache.org/r/67083/#comment285217>

Atlas (built with embedded hbase) fails to start with this in the 
application.log:

2018-05-15 16:56:33,093 INFO  - [main:] ~ Loading 
atlas-application.properties from 
file:/Users/david/atlasreview/distro/target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/conf/atlas-application.properties
 (ApplicationProperties:98)
2018-05-15 16:56:33,100 INFO  - [main:] ~ Property (set to default) 
atlas.graph.cache.db-cache = true (ApplicationProperties:242)
2018-05-15 16:56:33,100 INFO  - [main:] ~ Property (set to default) 
atlas.graph.cache.db-cache-clean-wait = 20 (ApplicationProperties:242)
2018-05-15 16:56:33,102 INFO  - [main:] ~ Property (set to default) 
atlas.graph.cache.db-cache-size = 0.5 (ApplicationProperties:242)
2018-05-15 16:56:33,102 INFO  - [main:] ~ Property (set to default) 
atlas.graph.cache.tx-cache.size = 15000 (ApplicationProperties:242)
2018-05-15 16:56:33,102 INFO  - [main:] ~ Property (set to default) 
atlas.graph.cache.tx-dirty-size = 120 (ApplicationProperties:242)
2018-05-15 16:56:33,116 INFO  - [main:] ~ 


   Atlas Server (STARTUP)

project.name:   apache-atlas
project.description:Metadata Management and Data Governance 
Platform over Hadoop
build.user: david
build.epoch:1526398417725
project.version:1.0.0-SNAPSHOT
build.version:  1.0.0-SNAPSHOT-r7e70b1e76324305fb1cd50302206c9d135b2be53
vc.revision:7e70b1e76324305fb1cd50302206c9d135b2be53
vc.source.url:  scm:git:git://git.apache.org/atlas.git/atlas-webapp


 (Atlas:215)
2018-05-15 16:56:33,117 INFO  - [main:] ~ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
(Atlas:216)
2018-05-15 16:56:33,117 INFO  - [main:] ~ Server starting with TLS ? false 
on port 21000 (Atlas:217)
2018-05-15 16:56:33,117 INFO  - [main:] ~ <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 
(Atlas:218)
2018-05-15 16:56:33,694 INFO  - [main:] ~ No authentication method 
configured.  Defaulting to simple authentication (LoginProcessor:102)
2018-05-15 16:56:33,806 INFO  - [main:] ~ Logged in user david 
(auth:SIMPLE) (LoginProcessor:77)
2018-05-15 16:56:34,210 INFO  - [main:] ~ Not running setup per 
configuration atlas.server.run.setup.on.start. (SetupSteps$SetupRequired:189)


- David Radley


On May 15, 2018, 1:35 p.m., Bogdan Sava wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67083/
> ---
> 
> (Updated May 15, 2018, 1:35 p.m.)
> 
> 
> Review request for atlas, Daniela Otelea, David Radley, Graham Wallis, Mandy 
> Chessell, and Ruxandra Rosu.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Create configuration for OMAG API in Atlas
> 
> 
> Diffs
> -
> 
>   om-fwk-ocf/pom.xml 8b1cefd 
>   omag-api/pom.xml 2338286 
>   
> omag-api/src/main/java/org/apache/atlas/omag/admin/OMAGServerAdministration.java
>  dd19018 
>   omag-api/src/main/java/org/apache/atlas/omag/admin/OMAGServerCategory.java 
> PRE-CREATION 
>   
> omag-api/src/main/java/org/apache/atlas/omag/admin/server/OMAGServerAdminRESTServices.java
>  PRE-CREATION 
>   
> omag-api/src/main/java/org/apache/atlas/omag/admin/server/OMAGServerAdminResource.java
>  ebc9f2a 
>   
> omag-api/src/main/java/org/apache/atlas/omag/admin/server/spring/OMAGServerAdminResource.java
>  PRE-CREATION 
>   
> omag-api/src/main/java/org/apache/atlas/omag/configuration/registration/AccessServiceDescription.java
>  1a15215 
>   
> omag-api/src/main/java/org/apache/atlas/omag/configuration/registration/AccessServiceRegistration.java
>  c2950c4 
>   omag-server/pom.xml 527c5c3 
>   
> omag-server/src/main/java/org/apache/atlas/omag/application/OMAGApplication.java
>  88397de 
>   omag-server/src/main/resources/application.properties PRE-CREATION 
>   omas-assetconsumer/pom.xml 29935e7 
>   
> omas-assetconsumer/src/main/java/org/apache/atlas/omas/assetconsumer/client/AssetConsumer.java
>  80a98f5 

Re: Review Request 67083: ATLAS-2665 - Add OMAG-API in Atlas

2018-05-15 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67083/#review203130
---




omag-api/src/main/java/org/apache/atlas/omag/admin/OMAGServerAdministration.java
Line 180 (original), 181 (patched)
<https://reviews.apache.org/r/67083/#comment285212>

you have remove the throws clause in the latest patch but the javadoc is 
left in. I suggest leaving in the throws clause. I think they are there to 
indicate these are the exceptions that any validation code we add in the future 
might throw



omag-api/src/main/java/org/apache/atlas/omag/admin/server/OMAGServerAdminRESTServices.java
Lines 1143 (patched)
<https://reviews.apache.org/r/67083/#comment285213>

javadoc of exceptions mismatches the code. It is important this matches as 
it will be the basis of the the swagger. This is the same for many methods



omag-server/src/main/java/org/apache/atlas/omag/application/OMAGApplication.java
Line 28 (original), 27 (patched)
<https://reviews.apache.org/r/67083/#comment285210>

what about omrs?



omas-assetconsumer/src/main/java/org/apache/atlas/omas/assetconsumer/server/AssetConsumerRESTServices.java
Line 230 (original), 211 (patched)
<https://reviews.apache.org/r/67083/#comment285211>

this change have removed the PathVariable and RequestParam annotations, it 
looks like we have lost inforration about which parameters are PathVariables 
and which are RequestParams



omrs/src/main/java/org/apache/atlas/omrs/admin/OMRSConfigurationFactory.java
Line 306 (original), 306 (patched)
<https://reviews.apache.org/r/67083/#comment285214>

shouldn't this url include the open-metadata segment?



webapp/src/main/webapp/WEB-INF/openMetadataContext.xml
Lines 2 (patched)
<https://reviews.apache.org/r/67083/#comment285216>

this licenses has bad characters that are casing git rebase errors when I 
apply the patch.


- David Radley


On May 15, 2018, 1:35 p.m., Bogdan Sava wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67083/
> ---
> 
> (Updated May 15, 2018, 1:35 p.m.)
> 
> 
> Review request for atlas, Daniela Otelea, David Radley, Graham Wallis, Mandy 
> Chessell, and Ruxandra Rosu.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Create configuration for OMAG API in Atlas
> 
> 
> Diffs
> -
> 
>   om-fwk-ocf/pom.xml 8b1cefd 
>   omag-api/pom.xml 2338286 
>   
> omag-api/src/main/java/org/apache/atlas/omag/admin/OMAGServerAdministration.java
>  dd19018 
>   omag-api/src/main/java/org/apache/atlas/omag/admin/OMAGServerCategory.java 
> PRE-CREATION 
>   
> omag-api/src/main/java/org/apache/atlas/omag/admin/server/OMAGServerAdminRESTServices.java
>  PRE-CREATION 
>   
> omag-api/src/main/java/org/apache/atlas/omag/admin/server/OMAGServerAdminResource.java
>  ebc9f2a 
>   
> omag-api/src/main/java/org/apache/atlas/omag/admin/server/spring/OMAGServerAdminResource.java
>  PRE-CREATION 
>   
> omag-api/src/main/java/org/apache/atlas/omag/configuration/registration/AccessServiceDescription.java
>  1a15215 
>   
> omag-api/src/main/java/org/apache/atlas/omag/configuration/registration/AccessServiceRegistration.java
>  c2950c4 
>   omag-server/pom.xml 527c5c3 
>   
> omag-server/src/main/java/org/apache/atlas/omag/application/OMAGApplication.java
>  88397de 
>   omag-server/src/main/resources/application.properties PRE-CREATION 
>   omas-assetconsumer/pom.xml 29935e7 
>   
> omas-assetconsumer/src/main/java/org/apache/atlas/omas/assetconsumer/client/AssetConsumer.java
>  80a98f5 
>   
> omas-assetconsumer/src/main/java/org/apache/atlas/omas/assetconsumer/publisher/AssetConsumerPublisher.java
>   
>   
> omas-assetconsumer/src/main/java/org/apache/atlas/omas/assetconsumer/server/AssetConsumerRESTServices.java
>  967be5a 
>   
> omas-assetconsumer/src/main/java/org/apache/atlas/omas/assetconsumer/server/spring/AssetConsumerOMASResource.java
>  PRE-CREATION 
>   omas-connectedasset/README.md 2f431df 
>   omas-connectedasset/pom.xml 219c972 
>   
> omas-connectedasset/src/main/java/org/apache/atlas/omas/connectedasset/server/ConnectedAssetRESTServices.java
>  8ae6c37 
>   
> omas-connectedasset/src/main/java/org/apache/atlas/omas/connectedasset/server/spring/ConnectedAssetOMASResource.java
>  PRE-CREATION 
>   omrs/pom.xml 50b9898 
>   
> omrs/src/main/java/org/apache/atlas/omrs/admin/OMRSConfigurationFactory.java 
> 7ae87c6 
>   
> omrs/src/main/java/org/apache/atlas/omrs/rest/server/OMRSRepositoryRESTServices.

Re: Review Request 67114: ATLAS-2463 Documentation for Atlas Authorization model and simple authorizer.

2018-05-14 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67114/#review203019
---




docs/src/site/twiki/Authentication-Authorization.twiki
Line 114 (original)
<https://reviews.apache.org/r/67114/#comment285070>

I suggest adding some words around the nature of this authorization. The 
fact that Ranger provides instance level authorization. Atlas itself provides 
authorization mechanisms that allows the user to protect the Atlas metadata. 
this can be done in a simple manner or using Ranger.



docs/src/site/twiki/Authentication-Authorization.twiki
Line 121 (original)
<https://reviews.apache.org/r/67114/#comment285071>

missing 



docs/src/site/twiki/Authentication-Authorization.twiki
Line 143 (original)
<https://reviews.apache.org/r/67114/#comment285072>

of => for



docs/src/site/twiki/Authentication-Authorization.twiki
Line 164 (original)
<https://reviews.apache.org/r/67114/#comment285073>

Can this run without Ambari?



docs/src/site/twiki/Authentication-Authorization.twiki
Line 166 (original)
<https://reviews.apache.org/r/67114/#comment285076>

I cannot see the link on the Ranger page to its documentation



docs/src/site/twiki/Configure-simple-authorizor.twiki
Lines 35 (patched)
<https://reviews.apache.org/r/67114/#comment285074>

white space issue



docs/src/site/twiki/Configure-simple-authorizor.twiki
Lines 68 (patched)
<https://reviews.apache.org/r/67114/#comment285075>

white space


- David Radley


On May 14, 2018, 1:44 p.m., Nixon Rodrigues wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67114/
> ---
> 
> (Updated May 14, 2018, 1:44 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-2463
> https://issues.apache.org/jira/browse/ATLAS-2463
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch adds documentation to Atlas authorization at entity level for 
> simple authorizer.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/Atlas-Authorization-Model.twiki PRE-CREATION 
>   docs/src/site/twiki/Authentication-Authorization.twiki 9832a92e8 
>   docs/src/site/twiki/Configure-simple-authorizor.twiki PRE-CREATION 
>   docs/src/site/twiki/index.twiki 258dfbb7f 
> 
> 
> Diff: https://reviews.apache.org/r/67114/diff/1/
> 
> 
> Testing
> ---
> 
> Tested the documentation after running mvn site
> 
> 
> Thanks,
> 
> Nixon Rodrigues
> 
>



[jira] [Assigned] (ATLAS-2665) Add OMAG-API in Atlas

2018-05-11 Thread David Radley (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley reassigned ATLAS-2665:
---

Assignee: Bogdan Sava

> Add OMAG-API in Atlas
> -
>
> Key: ATLAS-2665
> URL: https://issues.apache.org/jira/browse/ATLAS-2665
> Project: Atlas
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Bogdan Sava
>Assignee: Bogdan Sava
>Priority: Major
> Attachments: 0001-ATLAS-2665-Add-OMAG-API-in-Atlas.patch
>
>
> Create configuration for OMAG API in Atlas using Spring MVC Dispatcher 
> servlet.
> Change base URL for the API to "/open-metadata/access-services" 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Review Request 67089: ATLAS-2674 Unable to set an array value for an attribute in the open metadata

2018-05-11 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67089/
---

Review request for atlas, Graham Wallis and MandyDW MandyDW.


Bugs: ATLAS2674
https://issues.apache.org/jira/browse/ATLAS2674


Repository: atlas


Description
---

Unable to set an array value for an attribute in the open metadata


Diffs
-

  
omrs/src/main/java/org/apache/atlas/omrs/metadatacollection/properties/instances/ArrayPropertyValue.java
 154eb16c9 
  
omrs/src/test/java/org/apache/atlas/omrs/metadatacollection/properties/instances/TestArrayPropertyValue.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/67089/diff/1/


Testing
---

Ran junits- that failed without the fix.


Thanks,

David Radley



[jira] [Created] (ATLAS-2674) Unable to set an array value for an attribute in the open metadata

2018-05-11 Thread David Radley (JIRA)
David Radley created ATLAS-2674:
---

 Summary: Unable to set an array value for an attribute in the open 
metadata
 Key: ATLAS-2674
 URL: https://issues.apache.org/jira/browse/ATLAS-2674
 Project: Atlas
  Issue Type: Improvement
  Components:  atlas-core
Reporter: David Radley
Assignee: David Radley


Unable to set an array value for an attribute in the open metadata



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (ATLAS-2665) Add OMAG-API in Atlas

2018-05-11 Thread David Radley (JIRA)

[ 
https://issues.apache.org/jira/browse/ATLAS-2665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471885#comment-16471885
 ] 

David Radley commented on ATLAS-2665:
-

Hi Bogdan - please can you combine the patches so they can be git pushed as one 
commit. 

> Add OMAG-API in Atlas
> -
>
> Key: ATLAS-2665
> URL: https://issues.apache.org/jira/browse/ATLAS-2665
> Project: Atlas
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Bogdan Sava
>Priority: Major
> Attachments: 0001-ATLAS-2665-Add-OMAG-API-in-Atlas.patch, 
> 0002-ATLAS-2665-Added-apache-licence-to-properties-file.patch, 
> 0003-ATLAS-2665-comments-fixed.patch
>
>
> Create configuration for OMAG API in Atlas using Spring MVC Dispatcher 
> servlet.
> Change base URL for the API to "/open-metadata/access-services" 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Review Request 67060: ATLAS-2668: Add OMAG Server to distribution

2018-05-11 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67060/#review202921
---




ATLAS-2668a.patch
Lines 1 (patched)
<https://reviews.apache.org/r/67060/#comment284979>

looks like a patch has got into the diff


- David Radley


On May 11, 2018, 11:18 a.m., Nigel Jones wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67060/
> ---
> 
> (Updated May 11, 2018, 11:18 a.m.)
> 
> 
> Review request for atlas, David Radley, Graham Wallis, Madhan Neethiraj, and 
> Mandy Chessell.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Added OMAG Server to distribution with an easy to launch jar
> (See JIRA for more information)
> 
> 
> Diffs
> -
> 
>   ATLAS-2668a.patch PRE-CREATION 
>   distro/pom.xml 6431fd86d 
>   distro/src/main/assemblies/omag-server.xml PRE-CREATION 
>   omag-server/README.md PRE-CREATION 
>   omag-server/pom.xml 4c1c98aa3 
> 
> 
> Diff: https://reviews.apache.org/r/67060/diff/3/
> 
> 
> Testing
> ---
> 
> Built atlas
> Checked correct files in distribution archive
> Checked OMAG server launches with default config
> 
> 
> Thanks,
> 
> Nigel Jones
> 
>



Re: Review Request 67060: ATLAS-2668: Add OMAG Server to distribution

2018-05-11 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67060/#review202920
---


Ship it!




Please make the changes then ship it

- David Radley


On May 11, 2018, 11:18 a.m., Nigel Jones wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67060/
> ---
> 
> (Updated May 11, 2018, 11:18 a.m.)
> 
> 
> Review request for atlas, David Radley, Graham Wallis, Madhan Neethiraj, and 
> Mandy Chessell.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Added OMAG Server to distribution with an easy to launch jar
> (See JIRA for more information)
> 
> 
> Diffs
> -
> 
>   ATLAS-2668a.patch PRE-CREATION 
>   distro/pom.xml 6431fd86d 
>   distro/src/main/assemblies/omag-server.xml PRE-CREATION 
>   omag-server/README.md PRE-CREATION 
>   omag-server/pom.xml 4c1c98aa3 
> 
> 
> Diff: https://reviews.apache.org/r/67060/diff/3/
> 
> 
> Testing
> ---
> 
> Built atlas
> Checked correct files in distribution archive
> Checked OMAG server launches with default config
> 
> 
> Thanks,
> 
> Nigel Jones
> 
>



Re: Review Request 67061: ATLAS-2670: Check validity of entity guid when dissociating term

2018-05-11 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67061/#review202918
---




repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
Lines 103 (patched)
<https://reviews.apache.org/r/67061/#comment284974>

should we check that the associated entity and relationship are not deleted?


- David Radley


On May 10, 2018, 5:07 p.m., Apoorv Naik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67061/
> ---
> 
> (Updated May 10, 2018, 5:07 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-2670
> https://issues.apache.org/jira/browse/ATLAS-2670
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> See JIRA
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 5e3d81846 
>   repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java 
> 5400b77c3 
> 
> 
> Diff: https://reviews.apache.org/r/67061/diff/1/
> 
> 
> Testing
> ---
> 
> PreCommit: 
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/377/
> 
> 
> Thanks,
> 
> Apoorv Naik
> 
>



Re: Review Request 67060: ATLAS-2668: Add OMAG Server to distribution

2018-05-11 Thread David Radley
Hi Nigel,
Thanks for the explanations. I have indicated ship it! Once you make your 
amendments - I intend to commit this tomorrow if there are no further 
issues raised today, 

fyi - I notice you copied the dev list with incubator in the name - I have 
copied the non-incubator dev list. 
all the best, David. 


From:   Nigel Jones <nigel.l.jo...@gmail.com>
To: Mandy Chessell <mandy_chess...@uk.ibm.com>, Graham Wallis 
<graham_wal...@uk.ibm.com>, Madhan Neethiraj <mad...@apache.org>, David 
Radley <david...@apache.org>
Cc: atlas <d...@atlas.incubator.apache.org>
Date:   11/05/2018 10:51
Subject:Re: Review Request 67060: ATLAS-2668: Add OMAG Server to 
distribution
Sent by:Nigel Jones <nore...@reviews.apache.org>



This is an automatically generated e-mail. To reply, visit: 
https://reviews.apache.org/r/67060/ 

On May 11th, 2018, 10:39 a.m. BST, David Radley wrote:

distro/src/main/assemblies/omag-server.xml (Diff revision 1) 


29

I am wondering if you should add   and 
 like KafkaHook does. We should ensure that the jar and sh 
files are executable and the readme is not.


I see
-rw-r--r--  1 david  staff  1080 11 May 09:48 README.md
-rw-r--r--  1 david  staff  8022 11 May 09:55 
omag-server-1.0.0-SNAPSHOT-sources.jar
-rw-r--r--  1 david  staff  6561 11 May 09:55 
omag-server-1.0.0-SNAPSHOT-test-sources.jar
-rw-r--r--  1 david  staff  17222951 11 May 09:55 
omag-server-1.0.0-SNAPSHOT.jar


It doesn to look like the jar is executable.
Also why have we got the sources and test sources jar files?
On May 11th, 2018, 10:41 a.m. BST, David Radley wrote:
As it works - I guess we do not need to make it executable. Please could 
you review deleting the otehr jars
Intriguingly when I build it I do not see those jars in the distribution 
which only contains the files as per the jira ie
? 
~/src/atlas/distro/target/apache-atlas-1.0.0-SNAPSHOT-omag-server/omag-server-1.0.0-SNAPSHOT
 
[master ?·3?·1|…1? 3]
10:49 $ ls
README.md  omag-server-1.0.0-SNAPSHOT.jar


I did check this beforehand.. I'll do a clean build again to double check


Note I am referring here to what goes in the distribution (under 
distro/target)... I think I'd leave what's in the component build 
(omag-server/target)

On May 11th, 2018, 10:39 a.m. BST, David Radley wrote:

distro/src/main/assemblies/omag-server.xml (Diff revision 1) 


38
../omag-server
I see none of the other assemblies use ../ I am not sure why we need to?
This is the link to the location of the artifacts we want to pull into the 
distibution . Others do use it such as atlas-storm-hook-bridge.xml, so 
does hbase. Ok?

On May 11th, 2018, 10:39 a.m. BST, David Radley wrote:

omag-server/README.md (Diff revision 1) 


22
**Launching the standalone server**
I suggest changing this to be launching the standalone Open Metadata And 
Governance (OMAG) Server.
Fair comment, though as there's no regression here could do in another 
jira for expediency? Can remake patch once we are happy with other issues?

On May 11th, 2018, 10:39 a.m. BST, David Radley wrote:

omag-server/README.md (Diff revision 1) 


25
 
If you lose this whitespace - then we will not get thew white space 
warning on patch apply
Odd. hadn't noticed that. I still need to understand how it gets there. 
Will do.

On May 11th, 2018, 10:39 a.m. BST, David Radley wrote:

omag-server/pom.xml (Diff revision 1) 
124
org.apache.maven.plugins
124
org.springframework.boot
I do not understand this change. How does this effect the Atlas build?
It's to do with making an executable jar. Previously the maven-jar-plugin 
was being used, but would not build a jar you could run - nor did it 
include dependencies. To do that you can normally use the 
maven-shade-plugin instead (as we see elsewhere in atlas, and as I used 
for the gaian ranger plugin). Here you can built a composite jar with 
dependencies, and can also set the main class in the manifest. However in 
a spring environment this fails as additional configuration data is 
needed. This is where the spring maven plugin comes to the rescue and 
builds a correct jar so you can just now run the omag server as per the 
readme. The JIRA has a little more explanation

- Nigel

On May 10th, 2018, 5:23 p.m. BST, Nigel Jones wrote:

Review request for atlas, David Radley, Graham Wallis, Madhan Neethiraj, 
and Mandy Chessell.
By Nigel Jones.
Updated May 10, 2018, 5:23 p.m.
Repository: atlas 
Description 

Added OMAG Server to distribution with an easy to launch jar
(See JIRA for more information)
Testing 

Built atlas
Checked correct files in distribution archive
Checked OMAG server launches with default config
Diffs 
distro/pom.xml (6431fd86d)
distro/src/main/assemblies/omag-server.xml (PRE-CREATION)
omag-server/README.md (PRE-CREATION)
omag-server/pom.xml (4c1c98aa3)
View Diff



Unless stated otherwise above:
IBM Unit

Re: Review Request 67060: ATLAS-2668: Add OMAG Server to distribution

2018-05-11 Thread David Radley


> On May 11, 2018, 9:39 a.m., David Radley wrote:
> > distro/src/main/assemblies/omag-server.xml
> > Lines 29 (patched)
> > <https://reviews.apache.org/r/67060/diff/1/?file=2019396#file2019396line29>
> >
> > I am wondering if you should add   and 
> >  like KafkaHook does. We should ensure that the jar and 
> > sh files are executable and the readme is not.
> > 
> > I see
> > -rw-r--r--  1 david  staff  1080 11 May 09:48 README.md
> > -rw-r--r--  1 david  staff  8022 11 May 09:55 
> > omag-server-1.0.0-SNAPSHOT-sources.jar
> > -rw-r--r--  1 david  staff  6561 11 May 09:55 
> > omag-server-1.0.0-SNAPSHOT-test-sources.jar
> > -rw-r--r--  1 david  staff  17222951 11 May 09:55 
> > omag-server-1.0.0-SNAPSHOT.jar
> > 
> > 
> > It doesn to look like the jar is executable.
> > Also why have we got the sources and test sources jar files?

As it works - I guess we do not need to make it executable. Please could you 
review deleting the otehr jars


- David


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67060/#review202903
---


On May 10, 2018, 4:23 p.m., Nigel Jones wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67060/
> ---
> 
> (Updated May 10, 2018, 4:23 p.m.)
> 
> 
> Review request for atlas, David Radley, Graham Wallis, Madhan Neethiraj, and 
> Mandy Chessell.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Added OMAG Server to distribution with an easy to launch jar
> (See JIRA for more information)
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 6431fd86d 
>   distro/src/main/assemblies/omag-server.xml PRE-CREATION 
>   omag-server/README.md PRE-CREATION 
>   omag-server/pom.xml 4c1c98aa3 
> 
> 
> Diff: https://reviews.apache.org/r/67060/diff/1/
> 
> 
> Testing
> ---
> 
> Built atlas
> Checked correct files in distribution archive
> Checked OMAG server launches with default config
> 
> 
> Thanks,
> 
> Nigel Jones
> 
>



Re: Review Request 67060: ATLAS-2668: Add OMAG Server to distribution

2018-05-11 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67060/#review202903
---




distro/src/main/assemblies/omag-server.xml
Lines 29 (patched)
<https://reviews.apache.org/r/67060/#comment284944>

I am wondering if you should add   and 
 like KafkaHook does. We should ensure that the jar and sh 
files are executable and the readme is not.

I see
-rw-r--r--  1 david  staff  1080 11 May 09:48 README.md
-rw-r--r--  1 david  staff  8022 11 May 09:55 
omag-server-1.0.0-SNAPSHOT-sources.jar
-rw-r--r--  1 david  staff  6561 11 May 09:55 
omag-server-1.0.0-SNAPSHOT-test-sources.jar
-rw-r--r--  1 david  staff  17222951 11 May 09:55 
omag-server-1.0.0-SNAPSHOT.jar

It doesn to look like the jar is executable.
Also why have we got the sources and test sources jar files?



distro/src/main/assemblies/omag-server.xml
Lines 38 (patched)
<https://reviews.apache.org/r/67060/#comment284948>

I see none of the other assemblies use ../ I am not sure why we need to?



omag-server/README.md
Lines 22 (patched)
<https://reviews.apache.org/r/67060/#comment284946>

I suggest changing this to be launching the standalone Open Metadata And 
Governance (OMAG) Server.



omag-server/README.md
Lines 25 (patched)
<https://reviews.apache.org/r/67060/#comment284945>

If you lose this whitespace - then we will not get thew white space warning 
on patch apply



omag-server/pom.xml
Line 124 (original), 124 (patched)
<https://reviews.apache.org/r/67060/#comment284947>

I do not understand this change. How does this effect the Atlas build?


- David Radley


On May 10, 2018, 4:23 p.m., Nigel Jones wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67060/
> ---
> 
> (Updated May 10, 2018, 4:23 p.m.)
> 
> 
> Review request for atlas, David Radley, Graham Wallis, Madhan Neethiraj, and 
> Mandy Chessell.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Added OMAG Server to distribution with an easy to launch jar
> (See JIRA for more information)
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 6431fd86d 
>   distro/src/main/assemblies/omag-server.xml PRE-CREATION 
>   omag-server/README.md PRE-CREATION 
>   omag-server/pom.xml 4c1c98aa3 
> 
> 
> Diff: https://reviews.apache.org/r/67060/diff/1/
> 
> 
> Testing
> ---
> 
> Built atlas
> Checked correct files in distribution archive
> Checked OMAG server launches with default config
> 
> 
> Thanks,
> 
> Nigel Jones
> 
>



[jira] [Commented] (ATLAS-2655) [Glossary] Restrict category child to be within same glossary

2018-05-08 Thread David Radley (JIRA)

[ 
https://issues.apache.org/jira/browse/ATLAS-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467152#comment-16467152
 ] 

David Radley commented on ATLAS-2655:
-

It can be very useful to be able to have category children in other glossaries. 
For example re- using a 'standard' glossary categorizations across an 
organization.  
 If you need to restrict - could you put it under control of a feature toggle. 

> [Glossary] Restrict category child to be within same glossary
> -
>
> Key: ATLAS-2655
> URL: https://issues.apache.org/jira/browse/ATLAS-2655
> Project: Atlas
>  Issue Type: Bug
>Reporter: Apoorv Naik
>Assignee: Apoorv Naik
>Priority: Major
> Fix For: 1.0.0
>
>
> Currently no such check exists.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Review Request 66949: ATLAS-2523: changes to accept external GUIDs and manage homeIds

2018-05-04 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66949/#review202433
---




common/src/main/java/org/apache/atlas/repository/Constants.java
Lines 80 (patched)
<https://reviews.apache.org/r/66949/#comment284251>

I suggest a comment here to describe how an Atlas developer should 
understand this field. 

A concern I have, is that if the users go through the om aPI then the 
entities with a non-null homId will never be updated - apart from as part of 
the asynchronous cnotification process from anotehr server. The Atlas developer 
using the local Atlas API, needs to be aware not to change the entities and 
relationships with a none null homeId. Ideally we should police this.



intg/src/main/java/org/apache/atlas/type/AtlasTypeUtil.java
Lines 61 (patched)
<https://reviews.apache.org/r/66949/#comment284248>

I am wondering why you are adding a Logger and not using it om this change.



intg/src/main/java/org/apache/atlas/type/AtlasTypeUtil.java
Line 384 (original), 389 (patched)
<https://reviews.apache.org/r/66949/#comment284249>

I suggest adding a comment here, around why you are checking for '-' as the 
first character. 

What happens if another repository creates an identifier starting with a 
'-' character?

to cope with theis case - I am suggest that the logic should check for a 
null homeId and a guid starting character of '-' (or null) we should say the 
guid is not assigned.



repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1.java
Line 709 (original), 709 (patched)
<https://reviews.apache.org/r/66949/#comment284250>

why are we leaving in commented out code


- David Radley


On May 4, 2018, 10:55 a.m., Graham Wallis wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66949/
> ---
> 
> (Updated May 4, 2018, 10:55 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry and Madhan Neethiraj.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-2523: Changes to accept external GUIDs and manage homeIds
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 
> c570be2ccbc41a426b0f093b4a33163092223f2f 
>   intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java 
> e6a7f19633a1642f6b415db99e20c0641df9463b 
>   intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationship.java 
> d04daa5d283fdba90b917f38eba6c937021352df 
>   intg/src/main/java/org/apache/atlas/type/AtlasTypeUtil.java 
> 1b6af94f24eb7d99953f72815167c868a9bd9efd 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> edf10da9533803234342dc3313ea1024c5e7030f 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1.java
>  9fcba6ddaa1b74b2c81b980bcb455b614f4a4ed7 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java
>  d51adad80e133d636ef84883d395126ae0150af5 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphRetriever.java
>  3b9a287f6986ff6bca26fc20b2f94ec1700dbe1e 
> 
> 
> Diff: https://reviews.apache.org/r/66949/diff/1/
> 
> 
> Testing
> ---
> 
> Functional testing of these changes to save reference copies of entities and 
> relationships
> 
> 
> Thanks,
> 
> Graham Wallis
> 
>



[jira] [Commented] (ATLAS-2607) Classification lifecycle through metadata properties and relationships

2018-05-02 Thread David Radley (JIRA)

[ 
https://issues.apache.org/jira/browse/ATLAS-2607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460928#comment-16460928
 ] 

David Radley commented on ATLAS-2607:
-

[~madhan.neethiraj] . when you say "Asset can only be attached to 
classifications with the status of ACTIVE" -is this the general principle?

I had assumed that the Asset could be classified with a draft classification, 
this would exist in the graph but would not be seen by normal queries. We would 
need new queries that an workflow approver could use to see the draft 
classifications.

You say "Existing entity-classification-update permission will be used to 
enforce change to status". I suggest that the workflow permissions should be 
separate so we can have an approver that can only update the status and 
everyone else can update all but the status.  

In the open Metadata archive model I notice:
- there is a Draft sate in the open metadata TermRelationshipStatus. I wonder 
if you should consider using these enum values for classifications. 
- Open metadata also has the GovernanceClassificationStatus which includes 
states for Governance action classifications. 

It is worth considering enum states that will ease mapping from open metadata.. 




> Classification lifecycle through metadata properties and relationships
> --
>
> Key: ATLAS-2607
> URL: https://issues.apache.org/jira/browse/ATLAS-2607
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Reporter: Srikanth Venkat
>Assignee: Madhan Neethiraj
>Priority: Critical
>
> Currently tags or classifications in Atlas are considered active once they 
> are defined. For governance and stewardship purposes, it would be important 
> to attach the notion of what state in its lifecycle a particular 
> classification is. This would help with workflows to manage the lifecycle 
> aspects and provide any filtering needed to take appropriate actions. For 
> example only active classifications should be considered for classification 
> based policy enforcement. Additionally lifecycle status would help with 
> filtering and search as well as reporting and compliance/audit scenarios.
> Implementation Proposal:
>  * All tags or classifications have a "Lifecycle Status" property
>  * They can go through the following list of states during their lifecycle: 
> DRAFT -> ACTIVE  -> RETIRED
>  * Lifecycle Status can be set as an enum property that is mandatory or 
> required for all classifications.
>  * All existing classifications already present in Atlas before this change 
> will default to an ACTIVE status so that all pre-existing classifications 
> will continue to work as before.
>  * All new classifications after this change will start out with DRAFT status 
> and a steward or an admin with appropriate permissions can move them into an 
> ACTIVE state (controlled via Metadata security policies)
>  * Policy enforcement for authorization on classifications can ignore any 
> that are not in ACTIVE state. 
>  * Asset can only be attached to classifications with the status of ACTIVE
>  * For a classification in RETIRED state, we might have an optional 
> relationship with another classification called "Replaced By" which is the 
> new classification that the current one was remapped or replaced with. The 
> inverse relationship could be labeled "Replaces" which is on the new 
> classification and points to the removed classification that it replaces. 
>  * The state RETIRED implies this classification is no longer used and policy 
> enforcement will ignore any classifications in such states. 
>  * Additionally UI can filter and show by default only classifications that 
> are not RETIRED and a checkbox to "Show Retired"
>  * Deletion of classifications should work as it currently does with no 
> behavioral changes.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (ATLAS-2139) Map IGC meta model and Atlas OMRS open metadata types

2018-04-30 Thread David Radley (JIRA)

[ 
https://issues.apache.org/jira/browse/ATLAS-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458681#comment-16458681
 ] 

David Radley commented on ATLAS-2139:
-

[~sethv] Mandy is not available this week.

My thoughts on your questions are:
1. getSkeletonEntity is javadoc says "Return an entity with the header and type 
information filled out.  The caller only needs to add properties and 
classifications to complete the set up of the entity."  The way I am reading 
this is that I would not expect it to fill in properties, as the caller is 
expected to do this. 
2. The way the OMRS is, means that entities and relationships are created 
independently, but you need information from more than 1 entity or 
relationship, in order to issue the IGC call. If I have interpreted this 
correctly, my suggestion is :
- the IGC local connector keeps the entities and relationships until it has 
enough information (schema rids and the like) to issue the IGC request. Is 
there a way you can persist this information in IGC. 
or
-The local connector could query IGC for information that is not supplied in an 
add entity, this may allow you to access the rid information you need. 

   


> Map IGC meta model and Atlas OMRS open metadata types
> -
>
> Key: ATLAS-2139
> URL: https://issues.apache.org/jira/browse/ATLAS-2139
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Major
> Attachments: Relational Tables and Columns.png
>
>
> Create the two-way mapping between IGC meta model and Atlas OMRS open 
> metadata types.
> It is used by the IGC Event Mapper and IGC metadata repository connector.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Cassandra support for Atlas v1.0

2018-04-30 Thread David Radley
Hi Pierre,
>From what I understand, the way Apache projects work is that a group in 
the community may want a certain capability (for example running with 
hbase and solr). There may be another group in the community who want to 
run with other capabilities (Cassandra) that the first group may not be 
interested in. Each group in the community would test, raise and fix 
issues and make sure that their capability still worked. 

The committers and PMC decide what is supported,based on things like their 
confidence on what has been tested. As far as I am aware, none of the 
committers or PMCs have tested Cassandra or Scylladb  in production, and 
the code change you did and I committed was for an embedded Cassandra 
build. Embedded Cassandra seems to be a development style of build - 
embedded hbase is a different build profile from external hbase - which is 
the hbase profile used in production. 

So there are no issues I am aware of, and there is one voice in the 
community who wants Cassandra support. I would also like Cassandra support 
in Atlas, but I have not put any resources in to doing rigorous testing of 
it. I suggest adding the embedded Cassandra for development now , this may 
encourage other members of the community to use Cassandra and raise, test 
and fix issues around Cassandra; so that production use could then be 
easily supported. I suggest this is a point in time decision for v1.0, and 
does not stop people from using Cassandra in production with v1.0 or from 
Atlas supporting it in the future. I think the community and Apache Atlas 
is stronger from being inclusive of diverse usages of Atlas, 
I am interested if you or others have views on my interpretation,
all the best, David. 


 



From:   Pierre Padovani <pierre.padov...@civitaslearning.com>
To: dev@atlas.apache.org
Date:   27/04/2018 20:18
Subject:Re: Cassandra support for Atlas v1.0



I don't have an issue with #3, but we will be using it in production,
specifically with the Scylladb implementation.

Out of curiosity, what exactly are the issues?

On Fri, Apr 27, 2018 at 10:45 AM, Apoorv Naik <an...@hortonworks.com> 
wrote:

> #3 sounds like a better option for now. +1
>
> On 4/27/18, 12:42 AM, "Nigel Jones" <nigel.l.jo...@gmail.com> wrote:
>
>
>
> On 2018/04/26 16:14:06, David Radley <david_rad...@uk.ibm.com> 
wrote:
>
> > 3) Document it in the same way as the embedded solr and hbase 
build.
> That
>
> +1 for this option #3
>
> If in future someone has a need for production support I'm sure they
> will get engaged and help in testing etc :-)
>
>
>
>



Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


[jira] [Updated] (ATLAS-2622) Add extra checking for open metatdata entity type attributes

2018-04-29 Thread David Radley (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-2622:

Description: 
Jira 2604 added extra checking and units tests to ensure that relationshipdef 
enddef names do not conflict with each other or with any locally defined 
attributes. 

This Jira is raised to ensure that he same checking occurs for when add /update 
entitytype, add/update relationshipdefs are created via the API. 
Also this checking should occur when types are replicated between metadata 
repositories.

Ideally this and related name checking should be done in one place in the code 
for all these scenarios;  2604 updates code involved with the archive types 
only - I suggest this checking be pushed down into code that is used for the 
other scenarios if possible.

> Add extra checking for open metatdata entity type attributes
> 
>
> Key: ATLAS-2622
> URL: https://issues.apache.org/jira/browse/ATLAS-2622
> Project: Atlas
>  Issue Type: Improvement
>        Reporter: David Radley
>Priority: Major
>
> Jira 2604 added extra checking and units tests to ensure that relationshipdef 
> enddef names do not conflict with each other or with any locally defined 
> attributes. 
> This Jira is raised to ensure that he same checking occurs for when add 
> /update entitytype, add/update relationshipdefs are created via the API. 
> Also this checking should occur when types are replicated between metadata 
> repositories.
> Ideally this and related name checking should be done in one place in the 
> code for all these scenarios;  2604 updates code involved with the archive 
> types only - I suggest this checking be pushed down into code that is used 
> for the other scenarios if possible.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (ATLAS-2622) Add extra checking for open metatdata entity type attributes

2018-04-29 Thread David Radley (JIRA)
David Radley created ATLAS-2622:
---

 Summary: Add extra checking for open metatdata entity type 
attributes
 Key: ATLAS-2622
 URL: https://issues.apache.org/jira/browse/ATLAS-2622
 Project: Atlas
  Issue Type: Improvement
Reporter: David Radley






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: ATLAS-2569 ATLAS-2571 - commit?

2018-04-27 Thread David Radley
Hi Nigel,
I assumed Graham was going to commit  ATLAS-2569 as he had put ship it; 
Graham let me know if this is not the case. 
I do not see the link to the review board in the other Jira. I can have a 
look - if you put the link in,
all the best, David. 




From:   Nigel Jones 
To: 
Date:   27/04/2018 08:45
Subject:ATLAS-2569 ATLAS-2571 - commit?



FAO committers: Both of these now have a patch available. One +1 on the 
submission so far, but both are minor cleanups it would be good to get 
committed especially as our poms are regularly changing :-) 




Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


Cassandra support for Atlas v1.0

2018-04-26 Thread David Radley
Hi there,
We had a meeting to discuss how we would like to document the embedded 
Cassandra support in Atlas v1.0. We talked of 3 options :

1) Include it in the code base but do not document.
2) Document it as a tech preview
3) Document it in the same way as the embedded solr and hbase build. That 
means it will be document as being supported for development, but not in 
production. 

My preference is option 3). Are there any objections to this approach?
many thanks, David. 

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


[jira] [Updated] (ATLAS-2604) Minor fixups for relationshipdefs in the open metadata archive

2018-04-26 Thread David Radley (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-2604:

Attachment: ATLAS2604-3.patch

> Minor fixups for relationshipdefs in the open metadata archive 
> ---
>
> Key: ATLAS-2604
> URL: https://issues.apache.org/jira/browse/ATLAS-2604
> Project: Atlas
>  Issue Type: Bug
>        Reporter: David Radley
>    Assignee: David Radley
>Priority: Major
> Attachments: ATLAS2604-2.patch, ATLAS2604-3.patch, ATLAS2604.patch
>
>
> Some relationships were introducing the same attribute name into an entity 
> type. This Jira is to change the relationshipdefs so all the entitydef 
> attributes are unique.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Review Request 66784: ATLAS2604 Minor fixups to open metadata archive types

2018-04-26 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66784/
---

(Updated April 26, 2018, 2:37 p.m.)


Review request for atlas, Madhan Neethiraj and Mandy Chessell.


Bugs: ATLAS2604
https://issues.apache.org/jira/browse/ATLAS2604


Repository: atlas


Description
---

ATLAS2604 Minor fixups to open metadata archive types to avoid duplicate 
attributes due to relationship end names.


Diffs (updated)
-

  
omrs/src/main/java/org/apache/atlas/omrs/archivemanager/OMRSArchiveBuilder.java 
5902c4202 
  
omrs/src/main/java/org/apache/atlas/omrs/archivemanager/OMRSArchiveHelper.java 
42d2d8746 
  
omrs/src/main/java/org/apache/atlas/omrs/archivemanager/opentypes/OpenMetadataTypesArchive.java
 faab96cac 
  omrs/src/main/java/org/apache/atlas/omrs/ffdc/OMRSErrorCode.java 351c910e3 
  
omrs/src/test/java/org/apache/atlas/omrs/archivemanager/opentypes/TestOpenMetadataTypesArchive.java
 PRE-CREATION 
  
omrs/src/test/java/org/apache/atlas/omrs/archivemanager/store/TestOMRSArchiveBuilder.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/66784/diff/3/

Changes: https://reviews.apache.org/r/66784/diff/2-3/


Testing
---

Ran junit that failed without the patch applied. I ran this in Intellij using 
the  Maven view to run the test phase on the OMRS module. This successfully ran 
the test.


Thanks,

David Radley



[jira] [Updated] (ATLAS-2604) Minor fixups for relationshipdefs in the open metadata archive

2018-04-26 Thread David Radley (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-2604:

Attachment: ATLAS2604-2.patch

> Minor fixups for relationshipdefs in the open metadata archive 
> ---
>
> Key: ATLAS-2604
> URL: https://issues.apache.org/jira/browse/ATLAS-2604
> Project: Atlas
>  Issue Type: Bug
>        Reporter: David Radley
>    Assignee: David Radley
>Priority: Major
> Attachments: ATLAS2604-2.patch, ATLAS2604.patch
>
>
> Some relationships were introducing the same attribute name into an entity 
> type. This Jira is to change the relationshipdefs so all the entitydef 
> attributes are unique.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Review Request 66784: ATLAS2604 Minor fixups to open metadata archive types

2018-04-26 Thread David Radley


> On April 25, 2018, 8:40 a.m., Mandy Chessell wrote:
> > omrs/src/test/java/org/apache/atlas/omrs/archivemanager/opentypes/TestOpenMetadataTypesArchive.java
> > Lines 35 (patched)
> > <https://reviews.apache.org/r/66784/diff/1/?file=2011193#file2011193line35>
> >
> > This code is not in the most effective place.  It only ensures the open 
> > metadata types do not have duplicate attributes for an entity and only 
> > after the archive is built.  It should be in the archive builder ensuring 
> > that no archives can be built with duplicate names.  It should also be in 
> > the repository content manager validating that types from different 
> > archives do not cause an entity to have duplicate properties.  This test 
> > code should be removed and reformatted to run in the main OMRS code.

I have redone the tests.


- David


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66784/#review201902
-------


On April 26, 2018, 12:57 p.m., David Radley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66784/
> ---
> 
> (Updated April 26, 2018, 12:57 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj and Mandy Chessell.
> 
> 
> Bugs: ATLAS2604
> https://issues.apache.org/jira/browse/ATLAS2604
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS2604 Minor fixups to open metadata archive types to avoid duplicate 
> attributes due to relationship end names.
> 
> 
> Diffs
> -
> 
>   
> omrs/src/main/java/org/apache/atlas/omrs/archivemanager/OMRSArchiveBuilder.java
>  5902c4202 
>   
> omrs/src/main/java/org/apache/atlas/omrs/archivemanager/OMRSArchiveHelper.java
>  42d2d8746 
>   
> omrs/src/main/java/org/apache/atlas/omrs/archivemanager/opentypes/OpenMetadataTypesArchive.java
>  faab96cac 
>   omrs/src/main/java/org/apache/atlas/omrs/ffdc/OMRSErrorCode.java 351c910e3 
>   
> omrs/src/test/java/org/apache/atlas/omrs/archivemanager/opentypes/TestOpenMetadataTypesArchive.java
>  PRE-CREATION 
>   
> omrs/src/test/java/org/apache/atlas/omrs/archivemanager/store/TestOMRSArchiveBuilder.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/66784/diff/2/
> 
> 
> Testing
> ---
> 
> Ran junit that failed without the patch applied. I ran this in Intellij using 
> the  Maven view to run the test phase on the OMRS module. This successfully 
> ran the test.
> 
> 
> Thanks,
> 
> David Radley
> 
>



Re: Review Request 66784: ATLAS2604 Minor fixups to open metadata archive types

2018-04-26 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66784/
---

(Updated April 26, 2018, 12:57 p.m.)


Review request for atlas, Madhan Neethiraj and Mandy Chessell.


Bugs: ATLAS2604
https://issues.apache.org/jira/browse/ATLAS2604


Repository: atlas


Description
---

ATLAS2604 Minor fixups to open metadata archive types to avoid duplicate 
attributes due to relationship end names.


Diffs (updated)
-

  
omrs/src/main/java/org/apache/atlas/omrs/archivemanager/OMRSArchiveBuilder.java 
5902c4202 
  
omrs/src/main/java/org/apache/atlas/omrs/archivemanager/OMRSArchiveHelper.java 
42d2d8746 
  
omrs/src/main/java/org/apache/atlas/omrs/archivemanager/opentypes/OpenMetadataTypesArchive.java
 faab96cac 
  omrs/src/main/java/org/apache/atlas/omrs/ffdc/OMRSErrorCode.java 351c910e3 
  
omrs/src/test/java/org/apache/atlas/omrs/archivemanager/opentypes/TestOpenMetadataTypesArchive.java
 PRE-CREATION 
  
omrs/src/test/java/org/apache/atlas/omrs/archivemanager/store/TestOMRSArchiveBuilder.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/66784/diff/2/

Changes: https://reviews.apache.org/r/66784/diff/1-2/


Testing
---

Ran junit that failed without the patch applied. I ran this in Intellij using 
the  Maven view to run the test phase on the OMRS module. This successfully ran 
the test.


Thanks,

David Radley



[jira] [Updated] (ATLAS-2604) Minor fixups for relationshipdefs in the open metadata archive

2018-04-24 Thread David Radley (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-2604:

Description: Some relationships were introducing the same attribute name 
into an entity type. This Jira is to change the relationshipdefs so all the 
entitydef attributes are unique.  (was: Some relationships were introducing the 
same attribute name into an entity type. This Jira is to change the 
relationships so all the entity attributes are unique.)

> Minor fixups for relationshipdefs in the open metadata archive 
> ---
>
> Key: ATLAS-2604
> URL: https://issues.apache.org/jira/browse/ATLAS-2604
> Project: Atlas
>  Issue Type: Bug
>        Reporter: David Radley
>    Assignee: David Radley
>Priority: Major
> Attachments: ATLAS2604.patch
>
>
> Some relationships were introducing the same attribute name into an entity 
> type. This Jira is to change the relationshipdefs so all the entitydef 
> attributes are unique.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (ATLAS-2604) Minor fixups for relationshipdefs in the open metadata archive

2018-04-24 Thread David Radley (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-2604:

Description: Some relationships were introducing the same attribute name 
into an entity type. This Jira is to change the relationships so all the entity 
attributes are unique.

> Minor fixups for relationshipdefs in the open metadata archive 
> ---
>
> Key: ATLAS-2604
> URL: https://issues.apache.org/jira/browse/ATLAS-2604
> Project: Atlas
>  Issue Type: Bug
>        Reporter: David Radley
>    Assignee: David Radley
>Priority: Major
>
> Some relationships were introducing the same attribute name into an entity 
> type. This Jira is to change the relationships so all the entity attributes 
> are unique.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (ATLAS-2604) Minor fixups for relationshipdefs in the open metadata archive

2018-04-24 Thread David Radley (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-2604:

Attachment: ATLAS2604.patch

> Minor fixups for relationshipdefs in the open metadata archive 
> ---
>
> Key: ATLAS-2604
> URL: https://issues.apache.org/jira/browse/ATLAS-2604
> Project: Atlas
>  Issue Type: Bug
>        Reporter: David Radley
>    Assignee: David Radley
>Priority: Major
> Attachments: ATLAS2604.patch
>
>
> Some relationships were introducing the same attribute name into an entity 
> type. This Jira is to change the relationships so all the entity attributes 
> are unique.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Review Request 66784: ATLAS2604 Minor fixups to open metadata archive types

2018-04-24 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66784/
---

Review request for atlas, Madhan Neethiraj and Mandy Chessell.


Bugs: ATLAS2604
https://issues.apache.org/jira/browse/ATLAS2604


Repository: atlas


Description
---

ATLAS2604 Minor fixups to open metadata archive types to avoid duplicate 
attributes due to relationship end names.


Diffs
-

  
omrs/src/main/java/org/apache/atlas/omrs/archivemanager/opentypes/OpenMetadataTypesArchive.java
 faab96cac 
  
omrs/src/test/java/org/apache/atlas/omrs/archivemanager/opentypes/TestOpenMetadataTypesArchive.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/66784/diff/1/


Testing
---

Ran junit that failed without the patch applied. I ran this in Intellij using 
the  Maven view to run the test phase on the OMRS module. This successfully ran 
the test.


Thanks,

David Radley



[jira] [Created] (ATLAS-2604) Minor fixups for relationshipdefs in the open metadata archive

2018-04-24 Thread David Radley (JIRA)
David Radley created ATLAS-2604:
---

 Summary: Minor fixups for relationshipdefs in the open metadata 
archive 
 Key: ATLAS-2604
 URL: https://issues.apache.org/jira/browse/ATLAS-2604
 Project: Atlas
  Issue Type: Bug
Reporter: David Radley
Assignee: David Radley






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (ATLAS-2601) OMAG server Swagger

2018-04-24 Thread David Radley (JIRA)

[ 
https://issues.apache.org/jira/browse/ATLAS-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449807#comment-16449807
 ] 

David Radley commented on ATLAS-2601:
-

[~daniela.otelea] . Ideally we should minimize the dependancies and standardise 
on one library to implement a capability. I am wondering why we need a new 
swagger dependancy - when we already have a working one knitted in the Atlas 
buld. .

omag-api is in the atlas web server - so this is not completely independent of 
Atlas.   

> OMAG server Swagger 
> 
>
> Key: ATLAS-2601
> URL: https://issues.apache.org/jira/browse/ATLAS-2601
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Maryna Strelchuk
>Assignee: Maryna Strelchuk
>Priority: Major
> Attachments: 0001-Atlas-2601-OMAG-server-Swagger.patch
>
>
> This Jira addresses the addition of the Swagger for the OMAG server. 
> Currently Swagger was added as a dependency for
>  * omag-server
>  * omag-api
>  * omas-connectedasset



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (ATLAS-2601) OMAG server Swagger

2018-04-24 Thread David Radley (JIRA)

[ 
https://issues.apache.org/jira/browse/ATLAS-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449723#comment-16449723
 ] 

David Radley commented on ATLAS-2601:
-

The existing Atlas build uses enunciate to generate swagger. Is there a reason 
to add a second mechanism to generate Swagger? 

> OMAG server Swagger 
> 
>
> Key: ATLAS-2601
> URL: https://issues.apache.org/jira/browse/ATLAS-2601
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Maryna Strelchuk
>Assignee: Maryna Strelchuk
>Priority: Major
> Attachments: 0001-Atlas-2601-OMAG-server-Swagger.patch
>
>
> This Jira addresses the addition of the Swagger for the OMAG server. 
> Currently Swagger was added as a dependency for
>  * omag-server
>  * omag-api
>  * omas-connectedasset



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Review Request 65435: ATLAS-2298 - Review of OCF Database Connector_New

2018-04-13 Thread David Radley


> On March 26, 2018, 2 p.m., David Radley wrote:
> > ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/connectors/gaian/GaianOCFConnector.java
> > Lines 212 (patched)
> > <https://reviews.apache.org/r/65435/diff/2/?file=1985285#file1985285line212>
> >
> > should we return this for an error?
> 
> Yao Li wrote:
> i am not quite sure what do you mean

the finally is clause in java is driven in all cases - i.e even if you throw an 
error. I suggest this be moved to the end of the try block


> On March 26, 2018, 2 p.m., David Radley wrote:
> > ocf-database-connector/src/main/resources/gaian.properties
> > Lines 6 (patched)
> > <https://reviews.apache.org/r/65435/diff/2/?file=1985292#file1985292line6>
> >
> > It does not look right to include default passwords here, without 
> > talking about poc / dev scenarios and not production scenarios
> 
> Yao Li wrote:
> for now I am still not sure where do we store the information for the 
> front-end Gaian. For now I just set it up as a properties file. I dont know 
> these password will be stored in a central properties file or also OCF 
> Connection for every asset. I will keep this issue open.
> 
> Yao Li wrote:
> I have deleted the properties file for the new patch. for now if we want 
> to use Gaian, The front-end application have to know where the Gaian is. 
> GaianOCFConnector offer a constructor which contains all the set up for 
> connecting to Gaian.

Thanks Yau -  I will wait for the new patch to be uploaded to Jira and into the 
review board. Then please close out the issues - when you feel they are 
addressed.


- David


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65435/#review199965
---


On March 30, 2018, 9:33 a.m., Yao Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65435/
> ---
> 
> (Updated March 30, 2018, 9:33 a.m.)
> 
> 
> Review request for atlas and Mandy Chessell.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This is the new review request for ATLAS-2298 OCF Database Connector. The old 
> review on 
> [https://reviews.apache.org/r/65123/](https://reviews.apache.org/r/65123/) 
> will not be updated anymore.
> 
> The OCF Database Connector is the subclass of OCF Connector and it is 
> designed especially for connection to database to retrieve data. 
> Here we implement a connector for Gaian (GaianOCFConnector) as an example for 
> using OCF Database Connector. It is related to Open Connector Framework. The 
> JIRA can be found https://issues.apache.org/jira/browse/ATLAS-2298
> 
> 
> Diffs
> -
> 
>   ocf-database-connector/README.md PRE-CREATION 
>   ocf-database-connector/pom.xml PRE-CREATION 
>   
> ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/connectors/OCFDatabaseConnection.java
>  PRE-CREATION 
>   
> ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/connectors/OCFDatabaseConnector.java
>  PRE-CREATION 
>   
> ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/connectors/OCFDatabaseConnectorProviderBase.java
>  PRE-CREATION 
>   
> ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/connectors/gaian/GaianOCFConnector.java
>  PRE-CREATION 
>   
> ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/connectors/gaian/GaianOCFConnectorProvider.java
>  PRE-CREATION 
>   
> ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/ffdc/DatabaseConnectCheckedException.java
>  PRE-CREATION 
>   
> ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/ffdc/OCFDatabaseCheckedExceptionBase.java
>  PRE-CREATION 
>   
> ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/ffdc/OCFDatabaseConnectorErrorCode.java
>  PRE-CREATION 
>   
> ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/ffdc/OCFDatabaseConnectorRuntimeException.java
>  PRE-CREATION 
>   
> ocf-database-connector/src/test/java/gaianocfconnector/GaianOCFConnectorTest.java
>  PRE-CREATION 
>   
> ocf-database-connector/src/test/java/gaianocfconnector/UseGaianOCFConnector.java
>  PRE-CREATION 
>   pom.xml ee2746a905f84359d426acfb1fa6d64c6310aaaf 
> 
> 
> Diff: https://reviews.apache.org/r/65435/diff/5/
> 
> 
> Testing
> ---
> 
> see test folder.
> Gaian has to set up in advance
> 
> 
> File Attachments
> 
> 
> 0005-ATLAS-2298-05-Feb-code-review.patch
>   
> https://reviews.apache.org/media/uploaded/files/2018/02/05/292c1917-9deb-4ced-8bd6-1689fc45fbd4__0005-ATLAS-2298-05-Feb-code-review.patch
> 
> 
> Thanks,
> 
> Yao Li
> 
>



Re: Review Request 66522: ATLAS-2523: RelationshipStore accepts existing GUID

2018-04-10 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66522/#review200802
---




repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1.java
Lines 570 (patched)
<https://reviews.apache.org/r/66522/#comment281692>

For the open metadata usage - would it not be better to accept any value 
for the guid - rather than restricting it to the the Atlas format. I assume the 
archive type guids are of this form, but what about a relationship type that is 
mastered in another repository, this may not conform to the Atlas format for 
guids. 

In the Entity case the IGC RID may not conform to an assigned or unassigned 
Atlas guid. 

I think we need to consistent in how we treat the relationship and entity 
type guids - and not enforce the Atlas format on supplied guids - as they are 
likely to not match the Atlas format.


- David Radley


On April 10, 2018, 9:01 a.m., Graham Wallis wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66522/
> ---
> 
> (Updated April 10, 2018, 9:01 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-2523: RelationshipStore accepts existing GUID
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1.java
>  28636d864f28aea97ae50683a84b79a2b29cd166 
> 
> 
> Diff: https://reviews.apache.org/r/66522/diff/1/
> 
> 
> Testing
> ---
> 
> During connector development I have been saving relationships with externally 
> specified GUIDs from FV tests.
> 
> 
> Thanks,
> 
> Graham Wallis
> 
>



Re: Review Request 66478: ATLAS-2534: Atlas glossary support

2018-04-06 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66478/#review200630
---




addons/models/-Area0/0010-base_model.json
Lines 4 (patched)
<https://reviews.apache.org/r/66478/#comment281398>

I suggest it would be cleaner to have the atlas glossary entiries and 
relationships in a new model file called atlasglossary.json.



addons/models/-Area0/0010-base_model.json
Line 105 (original), 180 (patched)
<https://reviews.apache.org/r/66478/#comment281399>

You are subtyping __internal for the 3 new entities. Open metadata 
GlossaryTerm and the other glossary entities are actually referencables. I 
suggest subtypeing Referenceable to __internalReferencable and then using  
__internalReferencable as the supertype for the 3 atlas entities you are 
adding. In this way you stay mappable to the open metadata types. I think the 
qualified name is important for these glossary entities.



intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
Lines 132 (patched)
<https://reviews.apache.org/r/66478/#comment281400>

I cannot see this being used - or what the text has to do with glossary



intg/src/main/java/org/apache/atlas/model/glossary/relations/AtlasRelatedTermId.java
Lines 25 (patched)
<https://reviews.apache.org/r/66478/#comment281401>

I am wondering why you need to introduce a new custom Atlas annotation



repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
Lines 576 (patched)
<https://reviews.apache.org/r/66478/#comment281403>

why can't they be empty?



repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
Lines 590 (patched)
<https://reviews.apache.org/r/66478/#comment281402>

Is this tracked by a Jira?



repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
Lines 602 (patched)
<https://reviews.apache.org/r/66478/#comment281405>

Unless it was created by the entity API and the relationship has not been 
created yet. I suggest the error mentions this - as the user would need to know 
this to be able to fix the issue.



repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
Lines 618 (patched)
<https://reviews.apache.org/r/66478/#comment281404>

Unless it was created by the entity API and the relationship has not been 
created yet. I suggest the error mentions this - as the user would need to know 
this to be able to fix the issue.



repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
Lines 643 (patched)
<https://reviews.apache.org/r/66478/#comment281406>

Is the a Jira for this



repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
Lines 663 (patched)
<https://reviews.apache.org/r/66478/#comment281407>

Is there a Jira for this?



repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
Lines 678 (patched)
<https://reviews.apache.org/r/66478/#comment281408>

I suspect you wanted to put some text here.



repository/src/main/java/org/apache/atlas/repository/ogm/glossary/AtlasGlossaryCategoryDTO.java
Lines 18 (patched)
<https://reviews.apache.org/r/66478/#comment281409>

what does the ogm package mean?


- David Radley


On April 5, 2018, 11:57 p.m., Apoorv Naik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66478/
> ---
> 
> (Updated April 5, 2018, 11:57 p.m.)
> 
> 
> Review request for atlas, keval bhatt, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-2534
> https://issues.apache.org/jira/browse/ATLAS-2534
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Glossary implementation, no UI
> 
> 
> Diffs
> -
> 
>   addons/models/-Area0/0010-base_model.json aebe955db 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
>  7bdbeabf0 
>   
> graphdb/common/src/main/java/org/apache/atlas/repository/graphdb/tinkerpop/query/NativeTinkerpopGraphQuery.java
>  75665592e 
>   
> graphdb/common/src/main/java/org/apache/atlas/repository/graphdb/tinkerpop/query/TinkerpopGraphQuery.java
>  96b9705fa 
>   
> graphdb/common/src/main/java/org/apache/atlas/repository/graphdb/tinkerpop/query/expr/OrderByPredicate.java
>  PRE-CREATION 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
>  6820a93c2 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/query/NativeJanusGraphQuery.java
>  d3c976df5 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/reposit

[jira] [Created] (ATLAS-2537) when running the omag server, by default it create files that interfere with a build

2018-04-06 Thread David Radley (JIRA)
David Radley created ATLAS-2537:
---

 Summary: when running the omag server, by default it create files 
that interfere with a build 
 Key: ATLAS-2537
 URL: https://issues.apache.org/jira/browse/ATLAS-2537
 Project: Atlas
  Issue Type: Improvement
Reporter: David Radley
Assignee: David Radley


when running the omagapplication in a development environment, by default a 
config file is created in the root of the workspace. This file causes build 
failures due to the licence checker and could also be inadvertently checked 
into git.

This Jira is raised to exclude this file in the license checker and to put it 
into .gitignore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-04-04 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/#review200445
---


Ship it!




Ship It!

- David Radley


On March 28, 2018, 3:28 p.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65885/
> ---
> 
> (Updated March 28, 2018, 3:28 p.m.)
> 
> 
> Review request for atlas and David Radley.
> 
> 
> Bugs: ATLAS-2470
> https://issues.apache.org/jira/browse/ATLAS-2470
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This updates pom's to add the needed cassandra jars, and adds a dist profile 
> embedded-cassandra-solr.
> 
> 
> Diffs
> -
> 
>   addons/storm-bridge-shim/pom.xml c25fbfe5 
>   addons/storm-bridge/pom.xml 2627d92e 
>   distro/pom.xml 0103bef6 
>   distro/src/bin/atlas_config.py e6415cf4 
>   distro/src/bin/atlas_start.py 39be6b7c 
>   distro/src/bin/atlas_stop.py 66edd904 
>   distro/src/conf/atlas-application.properties 10fb7b86 
>   distro/src/conf/atlas-env.sh 68b24e93 
>   distro/src/conf/cassandra.yml.template PRE-CREATION 
>   distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
>   distro/src/conf/zookeeper/zoo.cfg.template PRE-CREATION 
>   distro/src/main/assemblies/standalone-package.xml 1881082e 
>   docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
>   graphdb/janus/pom.xml 143b775f 
>   pom.xml 9ddc8e18 
>   repository/pom.xml b1d6b1f9 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
>  cee7374e 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
>  4d70b7f7 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/CassandraAuditRepositoryTest.java
>  PRE-CREATION 
>   repository/src/test/resources/cassandra_test.yml PRE-CREATION 
>   
> server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java
>  7d282b30 
>   test-tools/src/main/resources/solr/solr.xml 041f0058 
>   webapp/pom.xml 284f538f 
> 
> 
> Diff: https://reviews.apache.org/r/65885/diff/6/
> 
> 
> Testing
> ---
> 
> Full build with the new embedded-cassandra-solr, and testing to make sure 
> Atlas comes up and is functional.
> 
> Be aware that we have been running Cassandra backed JanusGraph for months 
> with no issues.
> 
> This patch now includes support for Cassandra as an audit store. I've also 
> refactored the audit unit tests, and added cassandra specific tests.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



Re: Review Request 66374: ATLAS-2526: misc model updates

2018-03-30 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66374/#review200231
---




addons/models/-Area0/0010-base_model.json
Line 205 (original), 205 (patched)
<https://reviews.apache.org/r/66374/#comment280891>

I suggest there may be migration implications to renaming these base types. 
I suggest versioning this change.



addons/models/0300-Area3-SubjectArea/0310-Glossary.json
Lines 89 (patched)
<https://reviews.apache.org/r/66374/#comment280889>

I thought that we were thinking of these constraints as legacy and had 
assumed that we would not be adding any new ones. Doesn't the owned constraint 
imply that this is a one directional relationship, but the relationship 
definition is bidirectional. 

What does it mean that there is an attribute that uses the contraint but is 
not marked as legacy?

I am wondering what we gain by adding the attribute in this way?



addons/models/0300-Area3-SubjectArea/0320-CategoryHierarchy.json
Lines 32 (patched)
<https://reviews.apache.org/r/66374/#comment280890>

I wonder why we are adding legacy attributes. I can see it gives us a 
manditory parent; I thought the relationship design was such that we were 
dropping this capability.


- David Radley


On March 30, 2018, 8:16 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66374/
> ---
> 
> (Updated March 30, 2018, 8:16 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-2526
> https://issues.apache.org/jira/browse/ATLAS-2526
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> - fixed incorrect syntax/braces in 0120-Collections.json that resulted in 
> relationshipDefs not to be loaded
> - renamed sourceToProcesses as inputToProcess and sinkFromProcesses as 
> outputFromProcess
> - added terms and categories as direct attributes of Glossary
> - added anchor, parentCategory and childrenCategories as direct attributes of 
> GlossaryCategory
> - added anchor as direct attribute of GlossaryTerm
> 
> 
> Diffs
> -
> 
>   addons/models/-Area0/0010-base_model.json aebe955d 
>   addons/models/0100-Area1-Collaboration/0120-Collections.json 4dbd2adb 
>   addons/models/0300-Area3-SubjectArea/0310-Glossary.json cac2ce2a 
>   addons/models/0300-Area3-SubjectArea/0320-CategoryHierarchy.json 10637970 
>   addons/models/0300-Area3-SubjectArea/0330-Terms.json f492ddfd 
> 
> 
> Diff: https://reviews.apache.org/r/66374/diff/1/
> 
> 
> Testing
> ---
> 
> Verified Atlas server starts up successfully with updated models
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>



Re: Review Request 66203: [ATLAS-2510]: Add support to disable/enable propagated classification in entity

2018-03-27 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66203/#review200036
---




repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java
Lines 419 (patched)
<https://reviews.apache.org/r/66203/#comment280628>

If I have understood correctly; I think the proposal here is that a 
propogated classification can be disabled - this is implmented using a soft 
delete. Please could you confirm that any downstream propagated classifications 
are also disabled?


- David Radley


On March 27, 2018, 5:26 a.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66203/
> ---
> 
> (Updated March 27, 2018, 5:26 a.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, and Madhan Neethiraj.
> 
> 
> Bugs: ATLAS-2510
> https://issues.apache.org/jira/browse/ATLAS-2510
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently there is no way to remove propagated classification from an entity. 
> This Jira provides capability to disable/enable a propagated classification 
> from an entity using REST.
> 
> The propagated classification name and its associated entity guid is supplied 
> to the REST to disable/enable right propagated classification.
> 
> 
> Disable a propagated tag:
> curl -X PUT 
> 'http://localhost:21000/api/atlas/v2/entity/guid/0ea86d75-b92c-4907-b149-4fe3fe341cc3/propagatedClassifications/PII?sourceEntityGuid=fb38e88c-c9a8-4fa2-abaa-6be895b067b3=true'
>  -H 'Authorization: Basic YWRtaW46YWRtaW4='
> 
> Enable a propagated tag:
> curl -X PUT 
> 'http://localhost:21000/api/atlas/v2/entity/guid/0ea86d75-b92c-4907-b149-4fe3fe341cc3/propagatedClassifications/PII?sourceEntityGuid=fb38e88c-c9a8-4fa2-abaa-6be895b067b3=false'
>  -H 'Authorization: Basic YWRtaW46YWRtaW4='
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 605742dd 
>   intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 4a86670c 
>   intg/src/main/java/org/apache/atlas/model/instance/AtlasClassification.java 
> 008314b3 
>   intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java 
> fce46da9 
>   repository/src/main/java/org/apache/atlas/query/GremlinClause.java 454b3434 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> bfb5a711 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java
>  79e8e3e8 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java
>  b5461d4b 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  19ac6209 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java
>  c76f6405 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphRetriever.java
>  8fe635ae 
>   
> repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
> 9b4c91d1 
>   webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java fdafa2c1 
> 
> 
> Diff: https://reviews.apache.org/r/66203/diff/6/
> 
> 
> Testing
> ---
> 
> Validated disable and enable of propagated classification through REST client 
> - POSTMAN.
> 
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/213/console
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>



Re: Review Request 65435: ATLAS-2298 - Review of OCF Database Connector_New

2018-03-26 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65435/#review199965
---




ocf-database-connector/README.md
Lines 40 (patched)
<https://reviews.apache.org/r/65435/#comment280469>

bad eol character



ocf-database-connector/README.md
Lines 42 (patched)
<https://reviews.apache.org/r/65435/#comment280472>

typo sequene

the sentence mentions we and you - I suggest using one or the other.



ocf-database-connector/README.md
Lines 58 (patched)
<https://reviews.apache.org/r/65435/#comment280470>

bad eol character



ocf-database-connector/README.md
Lines 61 (patched)
<https://reviews.apache.org/r/65435/#comment280474>

I thought impersonation would require a patch on top of Gaian. If this is 
the case we need to detail where to get this patch for impersonation to be able 
to be used here.



ocf-database-connector/README.md
Lines 69 (patched)
<https://reviews.apache.org/r/65435/#comment280471>

same



ocf-database-connector/README.md
Lines 71 (patched)
<https://reviews.apache.org/r/65435/#comment280473>

'here' is an extra word
Typo Gian.
We have not mention LDAP, I think we need to be careful mentioning products 
that might not be installed.
We should not mention default passwords here - as in production this would 
not make sense.



ocf-database-connector/README.md
Lines 79 (patched)
<https://reviews.apache.org/r/65435/#comment280475>

This text is assuming that people have access to a Gaian Ranger plugin. 
This is not currently available.

You say "it is better to use **table function**. My understanding is that 
using the Ranger plugin requires the table function to be used. 

You say "You need to change cst to your own table name". What is cst?



ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/connectors/gaian/GaianOCFConnector.java
Lines 35 (patched)
<https://reviews.apache.org/r/65435/#comment280477>

incomplete sentence



ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/connectors/gaian/GaianOCFConnector.java
Lines 159 (patched)
<https://reviews.apache.org/r/65435/#comment280478>

finally is driven for errors as well - so this debug message is not 
correct. I suggst moving this debug to the end of the try section.



ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/connectors/gaian/GaianOCFConnector.java
Lines 212 (patched)
<https://reviews.apache.org/r/65435/#comment280479>

should we return this for an error?



ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/connectors/gaian/GaianOCFConnector.java
Lines 232 (patched)
<https://reviews.apache.org/r/65435/#comment280480>

this if and body is a repeat of an previous if.



ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/ffdc/OCFDatabaseConnectorErrorCode.java
Lines 76 (patched)
<https://reviews.apache.org/r/65435/#comment280481>

what is the user's valid information and valid use of OCF Connectors?



ocf-database-connector/src/main/resources/gaian.properties
Lines 6 (patched)
<https://reviews.apache.org/r/65435/#comment280482>

It does not look right to include default passwords here, without talking 
about poc / dev scenarios and not production scenarios



ocf-database-connector/src/test/java/gaianocfconnector/GaianOCFConnectorTest.java
Lines 56 (patched)
<https://reviews.apache.org/r/65435/#comment280483>

It looks like this class might a be unit test, but it seems that it 
requires Gaian to be running. If this cannot be run as a unit test - I suggest 
including instructions on how to run the tests in this file.



ocf-database-connector/src/test/java/gaianocfconnector/UseGaianOCFConnector.java
Lines 33 (patched)
<https://reviews.apache.org/r/65435/#comment280484>

I suggest the test throws the excpetions rather than gobbles them. In this 
way the test fails if there is an error


- David Radley


On March 22, 2018, 11:23 a.m., Yao Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65435/
> ---
> 
> (Updated March 22, 2018, 11:23 a.m.)
> 
> 
> Review request for atlas and Mandy Chessell.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This is the new review request for ATLAS-2298 OCF Database Connector. The old 
> review on 
> [https://reviews.apache.org/r/65123/](https://reviews.apache.org/r/65123/) 
> will not be updated anymore.
> 
> The OCF Database Connector is the subclass of OCF Connector and it is 
> designed especially 

Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-26 Thread David Radley
cutor.runWorker(ThreadPoolExecutor.java:1149)
[ERROR] at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[ERROR] at java.lang.Thread.run(Thread.java:748)
[ERROR] Caused by: java.lang.ClassNotFoundException: 
com.codahale.metrics.MetricRegistry$MetricSupplier
[ERROR] at 
java.net.URLClassLoader.findClass(URLClassLoader.java:381)
[ERROR] at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
[ERROR] at 
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
[ERROR] at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
[ERROR] ... 23 more
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, 
please read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the 
command
[ERROR]   mvn  -rf :atlas-repository



repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
Lines 200 (patched)
<https://reviews.apache.org/r/65885/#comment280468>

Do we need this TODO?


- David Radley


On March 20, 2018, 12:04 a.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65885/
> ---
> 
> (Updated March 20, 2018, 12:04 a.m.)
> 
> 
> Review request for atlas and David Radley.
> 
> 
> Bugs: ATLAS-2470
> https://issues.apache.org/jira/browse/ATLAS-2470
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This updates pom's to add the needed cassandra jars, and adds a dist profile 
> embedded-cassandra-solr.
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 0103bef6 
>   distro/src/bin/atlas_config.py e6415cf4 
>   distro/src/bin/atlas_start.py 39be6b7c 
>   distro/src/bin/atlas_stop.py 66edd904 
>   distro/src/conf/atlas-application.properties 10fb7b86 
>   distro/src/conf/atlas-env.sh 68b24e93 
>   distro/src/conf/cassandra.yml.template PRE-CREATION 
>   distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
>   distro/src/conf/zookeeper/zoo.cfg.template PRE-CREATION 
>   distro/src/main/assemblies/standalone-package.xml 1881082e 
>   docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
>   graphdb/janus/pom.xml 143b775f 
>   pom.xml 9ddc8e18 
>   repository/pom.xml b1d6b1f9 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
>  cee7374e 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
>  4d70b7f7 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/CassandraAuditRepositoryTest.java
>  PRE-CREATION 
>   repository/src/test/resources/cassandra_test.yml PRE-CREATION 
>   
> server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java
>  7d282b30 
> 
> 
> Diff: https://reviews.apache.org/r/65885/diff/4/
> 
> 
> Testing
> ---
> 
> Full build with the new embedded-cassandra-solr, and testing to make sure 
> Atlas comes up and is functional.
> 
> Be aware that we have been running Cassandra backed JanusGraph for months 
> with no issues.
> 
> This patch now includes support for Cassandra as an audit store. I've also 
> refactored the audit unit tests, and added cassandra specific tests.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



Re: Review Request 66174: ATLAS-1831 Virtualiser to dynamically configure an information virtualization layer

2018-03-20 Thread David Radley
ps as more than one consumer could be 
listening on the topic.



virtualiser/src/main/java/org/apache/atlas/virtualiser/kafka/KafkaVirtualiserConsumer.java
Lines 120 (patched)
<https://reviews.apache.org/r/66174/#comment279886>

delete



virtualiser/src/main/java/org/apache/atlas/virtualiser/views/ViewsConstructor.java
Lines 41 (patched)
<https://reviews.apache.org/r/66174/#comment279873>

I do not think you need to have this variable. As an empty view indicated 
we have nothing mapped.



virtualiser/src/main/java/org/apache/atlas/virtualiser/views/ViewsConstructor.java
Lines 47 (patched)
<https://reviews.apache.org/r/66174/#comment279874>

hich typo



virtualiser/src/main/java/org/apache/atlas/virtualiser/views/ViewsConstructor.java
Lines 54 (patched)
<https://reviews.apache.org/r/66174/#comment279876>

No views could mean this method returns null. If there is an exception then 
views will be ("",""}. Unless there is a reason I suggest views are null or 
have values.



virtualiser/src/main/java/org/apache/atlas/virtualiser/views/ViewsConstructor.java
Lines 58 (patched)
<https://reviews.apache.org/r/66174/#comment279877>

why dont we test if views == null - do we need noneMapped variable



virtualiser/src/main/java/org/apache/atlas/virtualiser/views/ViewsConstructor.java
Lines 63 (patched)
<https://reviews.apache.org/r/66174/#comment279875>

log the exception



virtualiser/src/main/java/org/apache/atlas/virtualiser/views/ViewsConstructor.java
Lines 101 (patched)
<https://reviews.apache.org/r/66174/#comment279879>

delete



virtualiser/src/main/java/org/apache/atlas/virtualiser/views/ViewsConstructor.java
Lines 109 (patched)
<https://reviews.apache.org/r/66174/#comment279880>

delete



virtualiser/src/main/java/org/apache/atlas/virtualiser/views/ViewsConstructor.java
Lines 181 (patched)
<https://reviews.apache.org/r/66174/#comment279872>

buinsess typo



virtualiser/src/main/resources/input.json
Lines 11 (patched)
<https://reviews.apache.org/r/66174/#comment279871>

I am unsure why we have specifc guid values in the main code. I would think 
I would see guid values in the test suite but not in the main java.



virtualiser/src/main/resources/virtualiser.properties
Lines 9 (patched)
<https://reviews.apache.org/r/66174/#comment279881>

This file should be a sample as it has specific values in, rather than as 
part of the main code.



virtualiser/src/main/resources/virtualiser.properties
Lines 56 (patched)
<https://reviews.apache.org/r/66174/#comment279882>

We hsould not be using the atlas API and there should not be ing urls in 
Atlas.



virtualiser/src/test/java/org/apache/atlas/virtualiser/JsonReadHelper.java
Lines 21 (patched)
<https://reviews.apache.org/r/66174/#comment279883>

delete



virtualiser/src/test/java/org/apache/atlas/virtualiser/JsonReadHelper.java
Lines 33 (patched)
<https://reviews.apache.org/r/66174/#comment279884>

It is not usual to put returns in finally sections. I suggest putting it at 
the end of the try or after the finally section.


- David Radley


On March 20, 2018, 2:25 p.m., Yao Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66174/
> ---
> 
> (Updated March 20, 2018, 2:25 p.m.)
> 
> 
> Review request for atlas, Kees van de Fliert, Mandy Chessell, and Maryna 
> Strelchuk.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Virtualiser has three main funcitons:
> 1. listen to Information View OMAS (information-view-out-topic) and retrieve 
> the json file;
> 2. update business logic table and technical logic table, send the updates to 
> Gaian;
> 3. create business view json file and technical view json file, notify 
> Information View OMAS the update(information-view-in-topic). 
> 
> The JIRA can be found https://issues.apache.org/jira/browse/ATLAS-1831
> 
> 
> Diffs
> -
> 
>   virtualiser/README.md PRE-CREATION 
>   virtualiser/pom.xml PRE-CREATION 
>   
> virtualiser/src/main/java/org/apache/atlas/omas/informationview/events/BusinessTerm.java
>  PRE-CREATION 
>   
> virtualiser/src/main/java/org/apache/atlas/omas/informationview/events/ColumnContextEvent.java
>  PRE-CREATION 
>   
> virtualiser/src/main/java/org/apache/atlas/omas/informationview/events/ColumnDetails.java
>  PRE-CREATION 
>   
> virtualiser/src/main/java/org/apache/atlas/omas/informationview/events/Connection.java
>  PRE-CREATION 
>   
> virtualiser/src/main/java/org/apache/atlas/omas/informationview/events/DerivedColumnDetail.java
>  PRE-CREATION 
>   
> virtualiser/src/main/java/org/apache/

Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-19 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/#review199456
---




docs/src/site/twiki/InstallationSteps.twiki
Lines 45 (patched)
<https://reviews.apache.org/r/65885/#comment279774>

I built Atlas as per these instructions. I then started atlas. Atlas 
started and worked - but when I looked at the application.log there was only 3 
log entries.  

2018-03-19 12:22:22,447 WARN  - [main:] ~ No meta.properties file under dir 
/Users/david/atlasreview2/distro/target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/data/kafka/kafka/meta.properties
 (Logging$class:87)
2018-03-19 12:22:22,922 WARN  - [main:] ~ No meta.properties file under dir 
/Users/david/atlasreview2/distro/target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/data/kafka/kafka/meta.properties
 (Logging$class:87)
2018-03-19 12:22:22,986 INFO  - [NotificationHookConsumer thread-0:] ~ 
[atlas-hook-consumer-thread]: Starting (Logging$class:72)

I would have expected much more logging.


- David Radley


On March 19, 2018, 11:51 a.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65885/
> ---
> 
> (Updated March 19, 2018, 11:51 a.m.)
> 
> 
> Review request for atlas and David Radley.
> 
> 
> Bugs: ATLAS-2470
> https://issues.apache.org/jira/browse/ATLAS-2470
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This updates pom's to add the needed cassandra jars, and adds a dist profile 
> embedded-cassandra-solr.
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 0103bef6 
>   distro/src/bin/atlas_config.py e6415cf4 
>   distro/src/bin/atlas_start.py 39be6b7c 
>   distro/src/bin/atlas_stop.py 66edd904 
>   distro/src/conf/atlas-application.properties 10fb7b86 
>   distro/src/conf/atlas-env.sh 68b24e93 
>   distro/src/conf/cassandra.yml.template PRE-CREATION 
>   distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
>   distro/src/conf/zookeeper/zoo.cfg.template PRE-CREATION 
>   distro/src/main/assemblies/standalone-package.xml 1881082e 
>   docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
>   graphdb/janus/pom.xml 143b775f 
>   repository/pom.xml b1d6b1f9 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
>  cee7374e 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
>  4d70b7f7 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/CassandraAuditRepositoryTest.java
>  PRE-CREATION 
>   repository/src/test/resources/cassandra_test.yml PRE-CREATION 
>   
> server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java
>  7d282b30 
> 
> 
> Diff: https://reviews.apache.org/r/65885/diff/3/
> 
> 
> Testing
> ---
> 
> Full build with the new embedded-cassandra-solr, and testing to make sure 
> Atlas comes up and is functional.
> 
> Be aware that we have been running Cassandra backed JanusGraph for months 
> with no issues.
> 
> This patch now includes support for Cassandra as an audit store. I've also 
> refactored the audit unit tests, and added cassandra specific tests.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-18 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/#review199398
---




distro/src/conf/atlas-application.properties
Line 23 (original), 23 (patched)
<https://reviews.apache.org/r/65885/#comment279722>

Sorry for the delay in reviewing. Unfortunatley the patch now does not 
apply to master I get: 
git am < ../ATLAS-2470-2.patch 
Applying: ATLAS-2470 - JanusGraph Cassandra support
.git/rebase-apply/patch:1146: trailing whitespace.
# The number of ticks that the initial 
error: patch failed: distro/src/conf/atlas-application.properties:24
error: distro/src/conf/atlas-application.properties: patch does not apply
.git/rebase-apply/patch:1039: new blank line at EOF.
+
.git/rebase-apply/patch:2761: new blank line at EOF.
+
Patch failed at 0001 ATLAS-2470 - JanusGraph Cassandra support


- David Radley


On March 8, 2018, 6:53 p.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65885/
> ---
> 
> (Updated March 8, 2018, 6:53 p.m.)
> 
> 
> Review request for atlas and David Radley.
> 
> 
> Bugs: ATLAS-2470
> https://issues.apache.org/jira/browse/ATLAS-2470
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This updates pom's to add the needed cassandra jars, and adds a dist profile 
> embedded-cassandra-solr.
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 0103bef6 
>   distro/src/bin/atlas_config.py e6415cf4 
>   distro/src/bin/atlas_start.py 39be6b7c 
>   distro/src/bin/atlas_stop.py 66edd904 
>   distro/src/conf/atlas-application.properties 355abf44 
>   distro/src/conf/atlas-env.sh 68b24e93 
>   distro/src/conf/cassandra.yml.template PRE-CREATION 
>   distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
>   distro/src/conf/zookeeper/zoo.cfg.template PRE-CREATION 
>   distro/src/main/assemblies/standalone-package.xml 1881082e 
>   docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
>   graphdb/janus/pom.xml 143b775f 
>   repository/pom.xml b1d6b1f9 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
>  cee7374e 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
>  4d70b7f7 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/CassandraAuditRepositoryTest.java
>  PRE-CREATION 
>   repository/src/test/resources/cassandra_test.yml PRE-CREATION 
>   
> server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java
>  7d282b30 
> 
> 
> Diff: https://reviews.apache.org/r/65885/diff/2/
> 
> 
> Testing
> ---
> 
> Full build with the new embedded-cassandra-solr, and testing to make sure 
> Atlas comes up and is functional.
> 
> Be aware that we have been running Cassandra backed JanusGraph for months 
> with no issues.
> 
> This patch now includes support for Cassandra as an audit store. I've also 
> refactored the audit unit tests, and added cassandra specific tests.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



Re: Review Request 66111: AtlasClientV2 should support create of relationship

2018-03-16 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66111/#review199317
---


Ship it!




Ship It!

- David Radley


On March 16, 2018, 8:58 a.m., Nigel Jones wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66111/
> ---
> 
> (Updated March 16, 2018, 8:58 a.m.)
> 
> 
> Review request for atlas, David Radley, Graham Wallis, and Madhan Neethiraj.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Add support to Atlas Client v2 api to allow create of relationships
> 
> 
> Diffs
> -
> 
>   client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 92ad923a 
> 
> 
> Diff: https://reviews.apache.org/r/66111/diff/1/
> 
> 
> Testing
> ---
> 
> See JIRA.
> Still testing, but won't be able to complete for a few weeks so posting for 
> feedback/comments & for others to try :-)
> 
> 
> Thanks,
> 
> Nigel Jones
> 
>



Re: AtlasClient API v2

2018-03-13 Thread David Radley
Hi Nigel,
I think you are saying this is a regression. 
I notice "Add jackson.core dependency in intg module" has just been 
committed. Could this make a difference?
Does it work if you recompile? 
   all the best, David. 



From:   Nigel Jones 
To: 
Date:   13/03/2018 13:02
Subject:AtlasClient API v2



I'm trying to use the AtlasClientV2 api but hitting an exception when 
trying to create entities.

This is using code from master (client and server) built in the last few 
days.

I'm able to send REST APIs directly, also my type creation worked fine.

What do I need to look at to resolve this error? Is it an error I've made 
in my types? Could it be a Jackson version issue (I did try updating some 
dependencies for my client code to 2.x)

Creating sample entities:Â 
2018-03-13 12:43:12 INFO Â AtlasBaseClient:330 - 
--
2018-03-13 12:43:12 INFO  AtlasBaseClient:331 - Call         : POST 
api/atlas/v2/entity/
2018-03-13 12:43:12 INFO Â AtlasBaseClient:332 - Content-type : 
application/json; charset=UTF-8Â 
2018-03-13 12:43:12 INFO  AtlasBaseClient:333 - Accept       : 
application/json 
2018-03-13 12:43:12 INFO  AtlasBaseClient:335 - Request      : 
AtlasEntityWithExtInfo{entity=AtlasEntity{AtlasStruct{typeName='gaianSchema', 
attributes=[qualifiedName:GAIAN, name:GAIAN, comment:Gaian Database 
Schema]}guid='-24212680184574', status=null, createdBy='null', 
updatedBy='null', createTime=null, updateTime=null, version=0, 
relationshipAttributes=[], classifications=[], 
},AtlasEntityExtInfo{referredEntities={}}}
Exception in thread "main" java.lang.AbstractMethodError
    at 
org.codehaus.jackson.map.AnnotationIntrospector$Pair.findSerializer(AnnotationIntrospector.java:1153)
    at 
org.codehaus.jackson.map.ser.BasicSerializerFactory.findSerializerFromAnnotation(BasicSerializerFactory.java:366)
    at 
org.codehaus.jackson.map.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:252)
    at 
org.codehaus.jackson.map.ser.StdSerializerProvider._createUntypedSerializer(StdSerializerProvider.java:782)
    at 
org.codehaus.jackson.map.ser.StdSerializerProvider._createAndCacheUntypedSerializer(StdSerializerProvider.java:735)
    at 
org.codehaus.jackson.map.ser.StdSerializerProvider.findValueSerializer(StdSerializerProvider.java:344)
    at 
org.codehaus.jackson.map.ser.StdSerializerProvider.findTypedValueSerializer(StdSerializerProvider.java:420)
    at 
org.codehaus.jackson.map.ser.StdSerializerProvider._serializeValue(StdSerializerProvider.java:601)
    at 
org.codehaus.jackson.map.ser.StdSerializerProvider.serializeValue(StdSerializerProvider.java:256)
    at 
org.codehaus.jackson.map.ObjectMapper.writeValue(ObjectMapper.java:1613)
    at 
org.codehaus.jackson.jaxrs.JacksonJsonProvider.writeTo(JacksonJsonProvider.java:559)
    at 
com.sun.jersey.json.impl.provider.entity.JacksonProviderProxy.writeTo(JacksonProviderProxy.java:160)
    at 
com.sun.jersey.api.client.RequestWriter.writeRequestEntity(RequestWriter.java:300)
    at 
com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:217)
    at 
com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
    at 
com.sun.jersey.api.client.filter.HTTPBasicAuthFilter.handle(HTTPBasicAuthFilter.java:105)
    at com.sun.jersey.api.client.Client.handle(Client.java:652)
    at 
com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
    at 
com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
    at 
com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:634)
    at 
org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:350)
    at 
org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:323)
    at 
org.apache.atlas.AtlasBaseClient.callAPI(AtlasBaseClient.java:211)
    at 
org.apache.atlas.AtlasClientV2.createEntity(AtlasClientV2.java:285)
    at 
org.apache.atlas.gaian.GaianSampleData.createInstance(GaianSampleData.java:176)
    at 
org.apache.atlas.gaian.GaianSampleData.createSchema(GaianSampleData.java:198)
    at 
org.apache.atlas.gaian.GaianSampleData.createEntities(GaianSampleData.java:155)
    at 
org.apache.atlas.gaian.GaianSampleData.main(GaianSampleData.java:72)




Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-05 Thread David Radley


> On March 4, 2018, 11:04 a.m., David Radley wrote:
> > distro/pom.xml
> > Lines 302 (patched)
> > <https://reviews.apache.org/r/65885/diff/1/?file=1969111#file1969111line302>
> >
> > for consistency should we have  as well?
> 
> Pierre Padovani wrote:
> You cannot just have cassandra without a solr/elasticsearch install 
> unless you are using DSE (the Datastax Enterprise with integrates with solr). 
> Based on cursory reading over on JanusGraph, they do not recommend production 
> deployments of JanusGraph with Cassandra embedded as the performance profiles 
> are not entirely predictable. Again I would address this in the main 
> documentation task.

Yes I was thinking that cassandra-embedded to differentiate between the 
production version - maybe you can add this when the extenral cassandra support 
is put in.


- David


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/#review198584
---


On March 2, 2018, 4:51 p.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65885/
> ---
> 
> (Updated March 2, 2018, 4:51 p.m.)
> 
> 
> Review request for atlas and David Radley.
> 
> 
> Bugs: ATLAS-2470
> https://issues.apache.org/jira/browse/ATLAS-2470
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This updates pom's to add the needed cassandra jars, and adds a dist profile 
> embedded-cassandra-solr.
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 0103bef6 
>   distro/src/bin/atlas_config.py e6415cf4 
>   distro/src/bin/atlas_start.py 39be6b7c 
>   distro/src/bin/atlas_stop.py 66edd904 
>   distro/src/conf/cassandra.yml PRE-CREATION 
>   distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
>   distro/src/conf/zookeeper/zoo.cfg PRE-CREATION 
>   distro/src/main/assemblies/standalone-package.xml 1881082e 
>   docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
>   graphdb/janus/pom.xml 143b775f 
> 
> 
> Diff: https://reviews.apache.org/r/65885/diff/1/
> 
> 
> Testing
> ---
> 
> Full build with the new embedded-cassandra-solr, and testing to make sure 
> Atlas comes up and is functional.
> 
> Be aware that we have been running Cassandra backed JanusGraph for months 
> with no issues.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-05 Thread David Radley


> On March 4, 2018, 11:07 a.m., David Radley wrote:
> > My review comments are from my initial look at the code; I hope to try 
> > running this patch to verify it works for me
> 
> Pierre Padovani wrote:
> I'll update the patch with the above changes today or tomorrow as I have 
> time.
> 
> Pierre Padovani wrote:
> I've updated the ATLAS-2470 with a new patch that should fix all of these 
> issues.
> 
> Pierre Padovani wrote:
> Hi David, I think I'm missing something. The instructions on the 'Using 
> GIT with Atlas' page are unclear on patch revisions. Do I create another 
> review, or do I somehow update this review?

It is best to use the same review for subsequent patches- so then it is 
possible to compare between versions. I have updated the wiki with this 
information - thanks.


- David


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/#review198593
---


On March 2, 2018, 4:51 p.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65885/
> ---
> 
> (Updated March 2, 2018, 4:51 p.m.)
> 
> 
> Review request for atlas and David Radley.
> 
> 
> Bugs: ATLAS-2470
> https://issues.apache.org/jira/browse/ATLAS-2470
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This updates pom's to add the needed cassandra jars, and adds a dist profile 
> embedded-cassandra-solr.
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 0103bef6 
>   distro/src/bin/atlas_config.py e6415cf4 
>   distro/src/bin/atlas_start.py 39be6b7c 
>   distro/src/bin/atlas_stop.py 66edd904 
>   distro/src/conf/cassandra.yml PRE-CREATION 
>   distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
>   distro/src/conf/zookeeper/zoo.cfg PRE-CREATION 
>   distro/src/main/assemblies/standalone-package.xml 1881082e 
>   docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
>   graphdb/janus/pom.xml 143b775f 
> 
> 
> Diff: https://reviews.apache.org/r/65885/diff/1/
> 
> 
> Testing
> ---
> 
> Full build with the new embedded-cassandra-solr, and testing to make sure 
> Atlas comes up and is functional.
> 
> Be aware that we have been running Cassandra backed JanusGraph for months 
> with no issues.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



[jira] [Commented] (ATLAS-2478) Elasticsearch support is broken for JanusGraph

2018-03-05 Thread David Radley (JIRA)

[ 
https://issues.apache.org/jira/browse/ATLAS-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386672#comment-16386672
 ] 

David Radley commented on ATLAS-2478:
-

[~ppadovani] I am not an expert on this part of Atlas. My thought is that most 
atlas customer will not have moved to Janus yet, so will need to perform some 
sort of migration of existing data; I would think that this sort of change 
could be made during the migration. 

The only released version of Atlas is an Alpha release, which is not intended 
for production. I suggest getting this change into master, so migration scripts 
could take account of this as well.  

> Elasticsearch support is broken for JanusGraph
> --
>
> Key: ATLAS-2478
> URL: https://issues.apache.org/jira/browse/ATLAS-2478
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 1.0.0-alpha
>Reporter: Pierre Padovani
>Assignee: Pierre Padovani
>Priority: Major
> Fix For: 1.0.0
>
>
> With JanusGraph the Elasticsearch support moved to 5.x+. This introduced a 
> change where fields that contained '.' (dots) were not allowed unless either 
> a specific cluster wide setting was enabled AND the mapping was formatted 
> such that each of the fields that contained a '.' could be considered part of 
> an object.
> Example:
> {code:java}
> foo.x
> foo.y
> foo.z{code}
>  Elasticsearch looks at these fields as if they are truly:
> {code:java}
> foo : {
>   x,
>   y,
>   z
> }{code}
> In the file:
> /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java
> {code:java}
> /**
>  * Properties for type store graph.
>  */
> public static final String TYPE_CATEGORY_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.category";
> public static final String VERTEX_TYPE_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type";
> public static final String TYPENAME_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.name";
> public static final String TYPEDESCRIPTION_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.description";
> public static final String TYPEVERSION_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.version";
> public static final String TYPEOPTIONS_PROPERTY_KEY = 
> INTERNAL_PROPERTY_KEY_PREFIX + "type.options";
> {code}
> These are the only fields that cause Elasticsearch issue. As you can see a 
> field called 'type' is created, then additional fields type.name, 
> type.description etc. This will cause a mapping conflict exception in 
> Elasticsearch and it will refuse to create the mapping.
>  
> The easy fix is to simply replace the '.' with an '_' (underscore) but this 
> will be a backwards incompatible change for existing customers. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-04 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/#review198593
---



My review comments are from my initial look at the code; I hope to try running 
this patch to verify it works for me

- David Radley


On March 2, 2018, 4:51 p.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65885/
> ---
> 
> (Updated March 2, 2018, 4:51 p.m.)
> 
> 
> Review request for atlas and David Radley.
> 
> 
> Bugs: ATLAS-2470
> https://issues.apache.org/jira/browse/ATLAS-2470
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This updates pom's to add the needed cassandra jars, and adds a dist profile 
> embedded-cassandra-solr.
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 0103bef6 
>   distro/src/bin/atlas_config.py e6415cf4 
>   distro/src/bin/atlas_start.py 39be6b7c 
>   distro/src/bin/atlas_stop.py 66edd904 
>   distro/src/conf/cassandra.yml PRE-CREATION 
>   distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
>   distro/src/conf/zookeeper/zoo.cfg PRE-CREATION 
>   distro/src/main/assemblies/standalone-package.xml 1881082e 
>   docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
>   graphdb/janus/pom.xml 143b775f 
> 
> 
> Diff: https://reviews.apache.org/r/65885/diff/1/
> 
> 
> Testing
> ---
> 
> Full build with the new embedded-cassandra-solr, and testing to make sure 
> Atlas comes up and is functional.
> 
> Be aware that we have been running Cassandra backed JanusGraph for months 
> with no issues.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas

2018-03-04 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65885/#review198584
---




distro/pom.xml
Lines 258 (patched)
<https://reviews.apache.org/r/65885/#comment278782>

I see that you are downloading Zookeeper. Many components require zookeeper 
- so it may already exist on the system. I think for production you would want 
to run with an external Zookeeper. 

I suggest we allow this as a build option. I realise this jira is for the 
embedded Cssandra build only - but I could envisage you needing non-embedded 
cassandra builds as well; I thought I would suggest this here - if this is a 
requirement for you, you  may decide you want to address this in a separate 
Jira.



distro/pom.xml
Lines 302 (patched)
<https://reviews.apache.org/r/65885/#comment278783>

for consistency should we have  as well?



distro/src/bin/atlas_start.py
Lines 121 (patched)
<https://reviews.apache.org/r/65885/#comment278784>

For an embedded hbase build, it will use the hbase zk and the embeded solr. 
I assume this line ( and the matching stop) should not be called for the 
embedded hbase build (which is not using the downloaded zk).



distro/src/conf/cassandra.yml
Lines 22 (patched)
<https://reviews.apache.org/r/65885/#comment278780>

typo



distro/src/conf/cassandra.yml
Lines 24 (patched)
<https://reviews.apache.org/r/65885/#comment278785>

I suggest the install could text replace these to the correct values. Is it 
possible for it to refer to an environment variable with the absolute path, 
then the script pick up that environment variable?



distro/src/conf/zookeeper/zoo.cfg
Lines 20 (patched)
<https://reviews.apache.org/r/65885/#comment278781>

bad end of line character here and a few places below



docs/src/site/twiki/InstallationSteps.twiki
Lines 38 (patched)
<https://reviews.apache.org/r/65885/#comment278786>

We did not have to do with before this change for embedded hbase solr. I 
think these notes should not apply to the embedded hbase solr profile.


- David Radley


On March 2, 2018, 4:51 p.m., Pierre Padovani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65885/
> ---
> 
> (Updated March 2, 2018, 4:51 p.m.)
> 
> 
> Review request for atlas and David Radley.
> 
> 
> Bugs: ATLAS-2470
> https://issues.apache.org/jira/browse/ATLAS-2470
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This updates pom's to add the needed cassandra jars, and adds a dist profile 
> embedded-cassandra-solr.
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 0103bef6 
>   distro/src/bin/atlas_config.py e6415cf4 
>   distro/src/bin/atlas_start.py 39be6b7c 
>   distro/src/bin/atlas_stop.py 66edd904 
>   distro/src/conf/cassandra.yml PRE-CREATION 
>   distro/src/conf/zookeeper/log4j.properties PRE-CREATION 
>   distro/src/conf/zookeeper/zoo.cfg PRE-CREATION 
>   distro/src/main/assemblies/standalone-package.xml 1881082e 
>   docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 
>   graphdb/janus/pom.xml 143b775f 
> 
> 
> Diff: https://reviews.apache.org/r/65885/diff/1/
> 
> 
> Testing
> ---
> 
> Full build with the new embedded-cassandra-solr, and testing to make sure 
> Atlas comes up and is functional.
> 
> Be aware that we have been running Cassandra backed JanusGraph for months 
> with no issues.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>



[jira] [Commented] (ATLAS-2465) OMRS - Kafka topic connector

2018-02-28 Thread David Radley (JIRA)

[ 
https://issues.apache.org/jira/browse/ATLAS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16380673#comment-16380673
 ] 

David Radley commented on ATLAS-2465:
-

Hi [~kanes] ,

when you want the patch to be reviewed for inclusion into Atlas,  please create 
one patch file and put the same changes into the Review Board tool. The code 
reviews occur in the Review Board tool,  kind regards,   David.

 

> OMRS - Kafka topic connector 
> -
>
> Key: ATLAS-2465
> URL: https://issues.apache.org/jira/browse/ATLAS-2465
> Project: Atlas
>  Issue Type: Improvement
>  Components: atlas-intg
>Reporter: Shrinivas Kane
>Assignee: Shrinivas Kane
>Priority: Minor
> Attachments: kafka-topic_connector-part1.patch, 
> kafka-topic_connector-part2.patch
>
>
> Add topic connector which will 
> a) listen to incoming kafka message and distribute across listeners 
> b) send message to kafka topic based on generated events 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (ATLAS-2259) Add Janus Graph Cassandra support

2018-02-26 Thread David Radley (JIRA)

[ 
https://issues.apache.org/jira/browse/ATLAS-2259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377245#comment-16377245
 ] 

David Radley commented on ATLAS-2259:
-

Hi [~ppadovani], If you submit a patch I will review it and commit it for you 
if there are no concerns with it, this means submitting attaching a patch file 
to this Jira and raising a rbt review for the code,   all the best, David.  

> Add Janus Graph Cassandra support
> -
>
> Key: ATLAS-2259
> URL: https://issues.apache.org/jira/browse/ATLAS-2259
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 1.0.0
>Reporter: Pierre Padovani
>Assignee: Pierre Padovani
>Priority: Major
> Fix For: 1.0.0
>
>
> Atlas should have support for Cassandra as a backend for Janus available by 
> default. If someone wants this type of configuration, they have to modify the 
> pom.xml and rebuild Atlas. Here is the pom.xml modification required to 
> enable this support:
> {code:java}
> 
> org.janusgraph
> janusgraph-cassandra
> ${janus.version}
> 
> 
> org.codehaus.jettison
> jettison
> 
> 
>   commons-lang
>   commons-lang
> 
> 
> 
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (ATLAS-2270) Supported combinations of persistent store and index backend

2018-02-23 Thread David Radley (JIRA)

[ 
https://issues.apache.org/jira/browse/ATLAS-2270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374297#comment-16374297
 ] 

David Radley commented on ATLAS-2270:
-

Hi Pierre,
Great; you ask to be a contributer on the dev list - this allows you to 
assign a Jira to yourself.

Then you create a patch in git and attach it to the Jira and put it into 
the review board for review. There may be review comments or a "ship it". 
After that a committer (probably me in this case)  should take your patch 
and push it to master. 
 
I was new to git and have created a wiki page with the git commands I use 
https://cwiki.apache.org/confluence/display/ATLAS/Using+Git+with+Atlas. 
You may find this useful if you have not used git too much.
There is a also a wiki page for committers 
https://cwiki.apache.org/confluence/display/ATLAS/Developer+Resources you 
may be interested in . 

I suggest that the Jira should be scoped just to the fix you want to put 
in. You could change the words of 2270 or create a new one, 
   all the best, David. 




From:   "Pierre Padovani (JIRA)" <j...@apache.org>
To: david...@apache.org
Date:   22/02/2018 16:44
Subject:[jira] [Commented] (ATLAS-2270) Supported combinations of 
persistent store and index backend




[ 
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_ATLAS-2D2270-3Fpage-3Dcom.atlassian.jira.plugin.system.issuetabpanels-3Acomment-2Dtabpanel-26focusedCommentId-3D16373048-23comment-2D16373048=DwICaQ=jf_iaSHvJObTbx-siA1ZOg=QhpUQPr5YlG95aAgCvZGStEXHg4hBbSYQ9JkRqR_svY=XQ5cRTBSZfDIanJGha8zM6Dkgnw0IjOYn8x5TlzRFBc=CvAzKNiTZy_yOqiqGZadB1tR74aieY28pWb2P8q3ucI=
 
] 

Pierre Padovani commented on ATLAS-2270:


[~davidrad] I'd be happy to be a contributor and get this in. How does one 
go about becoming one? Are there a set of guidelines somewhere specific to 
this project?

We've been running the Cassandra + ES 5.x flavor of Atlas as both a self 
contained docker container for dev purposes, as well as a full blown HA 
cluster for at least the last two months. So I think it looks like a 
pretty stable setup.

https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_ATLAS-2D2270=DwICaQ=jf_iaSHvJObTbx-siA1ZOg=QhpUQPr5YlG95aAgCvZGStEXHg4hBbSYQ9JkRqR_svY=XQ5cRTBSZfDIanJGha8zM6Dkgnw0IjOYn8x5TlzRFBc=r6UWXjJszzenyZwhDWtpds09zR_aEaPTnA245uXb-vU=

indexing backend Atlas 1.0.0 (master) should support. This includes 
building/running Atlas as a standalone package and running UTs/ITs as part 
of the Atlas build. 
databases that will be supported in master/1.0.0. This JIRA deliberately 
ignores titan1 and janusgraph 0.1.1 as the former should be 
deprecated/removed and the other is a transient state as we get to 
janusgraph 0.2.0. 
combinations of persistent store and indexer. It is suggested that this 
set is kept unchanged:
additional combinations. Cassandra is included in this discussion pending 
response to ATLAS-2259.
continued and the remaining 4 combinations, marked with '?', should be 
considered. There seems to be evidence of people using all 4 of these 
combinations, although not necessarily with Atlas.
possible to build Atlas as a standalone package with any of the 
combinations - i.e. that they are mutually exclusive and do not interfere 
with one another. They currently interfere which makes it impossible to 
build Atlas with -Pdist,berkeley-elasticsearch because the 'dist' profile 
will exclude jars that are needed by the berkeley-elasticsearch profile - 
which leads to class not found exceptions when the Atlas server is 
started. The solution to this could be very simple, or slightly more 
sophisticated, depending on how many of the combinations we choose to 
support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)




Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


> Supported combinations of persistent store and index backend
> 
>
> Key: ATLAS-2270
> URL: https://issues.apache.org/jira/browse/ATLAS-2270
> Project: Atlas
>  Issue Type: Bug
>Reporter: Graham Wallis
>Priority: Major
>
> We need to discuss and decide which combinations of persistent store and 
> indexing backend Atlas 1.0.0 (master) should support. This includes 
> building/running Atlas as a standalone package and running UTs/ITs as part of 
> the Atlas build. 
> This JIRA focusses on titan0 and janusgraph 0.2.0, as they are the graph 
> databases that will be supported in master/1.0.0. This JIRA deliberately 
> ignores titan1 and janusgraph 0.1.1 as the former should be 
&g

Re: [jira] [Commented] (ATLAS-2270) Supported combinations of persistent store and index backend

2018-02-23 Thread David Radley
Hi Pierre,
Great; you ask to be a contributer on the dev list - this allows you to 
assign a Jira to yourself.

Then you create a patch in git and attach it to the Jira and put it into 
the review board for review. There may be review comments or a "ship it". 
After that a committer (probably me in this case)  should take your patch 
and push it to master. 
 
I was new to git and have created a wiki page with the git commands I use 
https://cwiki.apache.org/confluence/display/ATLAS/Using+Git+with+Atlas. 
You may find this useful if you have not used git too much.
There is a also a wiki page for committers 
https://cwiki.apache.org/confluence/display/ATLAS/Developer+Resources you 
may be interested in . 

I suggest that the Jira should be scoped just to the fix you want to put 
in. You could change the words of 2270 or create a new one, 
   all the best, David. 




From:   "Pierre Padovani (JIRA)" 
To: david...@apache.org
Date:   22/02/2018 16:44
Subject:[jira] [Commented] (ATLAS-2270) Supported combinations of 
persistent store and index backend




[ 
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_ATLAS-2D2270-3Fpage-3Dcom.atlassian.jira.plugin.system.issuetabpanels-3Acomment-2Dtabpanel-26focusedCommentId-3D16373048-23comment-2D16373048=DwICaQ=jf_iaSHvJObTbx-siA1ZOg=QhpUQPr5YlG95aAgCvZGStEXHg4hBbSYQ9JkRqR_svY=XQ5cRTBSZfDIanJGha8zM6Dkgnw0IjOYn8x5TlzRFBc=CvAzKNiTZy_yOqiqGZadB1tR74aieY28pWb2P8q3ucI=
 
] 

Pierre Padovani commented on ATLAS-2270:


[~davidrad] I'd be happy to be a contributor and get this in. How does one 
go about becoming one? Are there a set of guidelines somewhere specific to 
this project?

We've been running the Cassandra + ES 5.x flavor of Atlas as both a self 
contained docker container for dev purposes, as well as a full blown HA 
cluster for at least the last two months. So I think it looks like a 
pretty stable setup.

> Supported combinations of persistent store and index backend
> 
>
> Key: ATLAS-2270
> URL: 
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_ATLAS-2D2270=DwICaQ=jf_iaSHvJObTbx-siA1ZOg=QhpUQPr5YlG95aAgCvZGStEXHg4hBbSYQ9JkRqR_svY=XQ5cRTBSZfDIanJGha8zM6Dkgnw0IjOYn8x5TlzRFBc=r6UWXjJszzenyZwhDWtpds09zR_aEaPTnA245uXb-vU=

> Project: Atlas
>  Issue Type: Bug
>Reporter: Graham Wallis
>Priority: Major
>
> We need to discuss and decide which combinations of persistent store and 
indexing backend Atlas 1.0.0 (master) should support. This includes 
building/running Atlas as a standalone package and running UTs/ITs as part 
of the Atlas build. 
> This JIRA focusses on titan0 and janusgraph 0.2.0, as they are the graph 
databases that will be supported in master/1.0.0. This JIRA deliberately 
ignores titan1 and janusgraph 0.1.1 as the former should be 
deprecated/removed and the other is a transient state as we get to 
janusgraph 0.2.0. 
> With titan0 as the graph provider, Atlas has supported the following 
combinations of persistent store and indexer. It is suggested that this 
set is kept unchanged:
> {{
> titan0  solr  es
> 
> berkeley   0  1
> hbase   1  0
> cassandra0  0
> }}
> With janusgraph (0.2.0) as the graph provider, Atlas *could* support 
additional combinations. Cassandra is included in this discussion pending 
response to ATLAS-2259.
> {{
> janus 0.2.0  solr  es
> 
> berkeley   ?  1
> hbase   1  ?
> cassandra?  ?
> }}
> It is suggested that the combinations marked with '1' should be 
continued and the remaining 4 combinations, marked with '?', should be 
considered. There seems to be evidence of people using all 4 of these 
combinations, although not necessarily with Atlas.
> Depending on the decision made above, we need to ensure that it is 
possible to build Atlas as a standalone package with any of the 
combinations - i.e. that they are mutually exclusive and do not interfere 
with one another. They currently interfere which makes it impossible to 
build Atlas with -Pdist,berkeley-elasticsearch because the 'dist' profile 
will exclude jars that are needed by the berkeley-elasticsearch profile - 
which leads to class not found exceptions when the Atlas server is 
started. The solution to this could be very simple, or slightly more 
sophisticated, depending on how many of the combinations we choose to 
support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)




Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 

[jira] [Commented] (ATLAS-2270) Supported combinations of persistent store and index backend

2018-02-22 Thread David Radley (JIRA)

[ 
https://issues.apache.org/jira/browse/ATLAS-2270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373016#comment-16373016
 ] 

David Radley commented on ATLAS-2270:
-

[~ppadovani]  - as far as I am aware there is no one working on this at the 
moment. The only permutation that works is the Hbase solr one at the moment in 
master. You could become a contribute and contribute a fix, which we could 
commit.  I think in these situations the person who would like it in, needs to 
contribute the fix and would then fix any issues in that stack. At this time , 
this stack is not a priority at the moment for the few committers I have talked 
to. Is this patch something you want to contribute i.e. provide a patch to 
master and submit on the review board and be up for fixing any issues in that 
stack. I so I could commit it for you if it does not break the hbase solr 
config. 

> Supported combinations of persistent store and index backend
> 
>
> Key: ATLAS-2270
> URL: https://issues.apache.org/jira/browse/ATLAS-2270
> Project: Atlas
>  Issue Type: Bug
>Reporter: Graham Wallis
>Priority: Major
>
> We need to discuss and decide which combinations of persistent store and 
> indexing backend Atlas 1.0.0 (master) should support. This includes 
> building/running Atlas as a standalone package and running UTs/ITs as part of 
> the Atlas build. 
> This JIRA focusses on titan0 and janusgraph 0.2.0, as they are the graph 
> databases that will be supported in master/1.0.0. This JIRA deliberately 
> ignores titan1 and janusgraph 0.1.1 as the former should be 
> deprecated/removed and the other is a transient state as we get to janusgraph 
> 0.2.0. 
> With titan0 as the graph provider, Atlas has supported the following 
> combinations of persistent store and indexer. It is suggested that this set 
> is kept unchanged:
> {{
> titan0  solr  es
> 
> berkeley   0  1
> hbase   1  0
> cassandra0  0
> }}
> With janusgraph (0.2.0) as the graph provider, Atlas *could* support 
> additional combinations. Cassandra is included in this discussion pending 
> response to ATLAS-2259.
> {{
> janus 0.2.0  solr  es
> 
> berkeley   ?  1
> hbase   1  ?
> cassandra?  ?
> }}
> It is suggested that the combinations marked with '1' should be continued and 
> the remaining 4 combinations, marked with '?', should be considered. There 
> seems to be evidence of people using all 4 of these combinations, although 
> not necessarily with Atlas.
> Depending on the decision made above, we need to ensure that it is possible 
> to build Atlas as a standalone package with any of the combinations - i.e. 
> that they are mutually exclusive and do not interfere with one another. They 
> currently interfere which makes it impossible to build Atlas with 
> -Pdist,berkeley-elasticsearch because the 'dist' profile will exclude jars 
> that are needed by the berkeley-elasticsearch profile - which leads to class 
> not found exceptions when the Atlas server is started. The solution to this 
> could be very simple, or slightly more sophisticated, depending on how many 
> of the combinations we choose to support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Review Request 65742: ATLAS-2457: updates to support time-boundary in AtlasClassification associations

2018-02-22 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65742/#review198099
---




repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphRetriever.java
Lines 221 (patched)
<https://reviews.apache.org/r/65742/#comment278209>

Do we handle 5 classification of the same type against an entity, but with 
non-overlapping validity periods.



repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphRetriever.java
Lines 222 (patched)
<https://reviews.apache.org/r/65742/#comment278208>

My feeling is that there should be a new state for the classification that 
says it is yet to be activated.



repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphRetriever.java
Lines 226 (patched)
<https://reviews.apache.org/r/65742/#comment278207>

We should have checking to make sure the lower bounds are sooner than the 
upper bounds.
Do we allow the same value for bottom and top. 
Do we allow adding classification validity periods in the past? Or crossing 
the present?
Maybe we round the time period to a minute to prevent timeing issues for 
very small time periods. 

I think we should police all of these bioundary conditions and have unique 
messages for each case.

Holding and specifying the time in UTC would prevent any quirky issues like 
daylight savings, 29th of Feb is valid only for leap years.


- David Radley


On Feb. 22, 2018, 12:19 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65742/
> ---
> 
> (Updated Feb. 22, 2018, 12:19 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-2457
> https://issues.apache.org/jira/browse/ATLAS-2457
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Updated AtlasClassification with addition of 'timeBoundary' attribute.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java ae528807 
>   intg/src/main/java/org/apache/atlas/model/TimeBoundary.java PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/instance/AtlasClassification.java 
> a499f793 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java
>  0224bf01 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphRetriever.java
>  d01fb9f0 
>   
> webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java
>  dabb2efa 
> 
> 
> Diff: https://reviews.apache.org/r/65742/diff/2/
> 
> 
> Testing
> ---
> 
> - added IT to validate that the new attribute is correctly stored/retrieved
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>



Re: Relationships to struct fields

2018-02-20 Thread David Radley
Hi Pierre,
That is not the way we have implemented the relationships. It is only 
possible to have relationships between 2 entities. I think this makes 
sense to restrict to entities. I am interested why you need this and 
whether there are alternative ways to model what you need,  all the best, 
David. 



From:   Pierre Padovani 
To: dev@atlas.apache.org
Date:   20/02/2018 14:26
Subject:Relationships to struct fields



Quick question, as I could not find any documentation that asserts this is
a valid use case or not.

I have a type with a struct. I'd like to create a relationship from a 
field
in the struct to another type.

Is this even possible?



Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


[jira] [Commented] (ATLAS-2456) Implement tag propagation using relationships

2018-02-20 Thread David Radley (JIRA)

[ 
https://issues.apache.org/jira/browse/ATLAS-2456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369846#comment-16369846
 ] 

David Radley commented on ATLAS-2456:
-

[~sarath.ku...@gmail.com] Thanks for your reply. 

>From your responses, is it true that: 
 * many propagated classifications of the same type can exist for an entity
 * a propagated classification can exist with the same type as an explicitly 
defined classification
 * on point 7 - does this mean that only one classification type or any of its 
sub types can be applied to an entity

 

I suggest:
 * we do not propagate a classification of a given type to an entity if there 
is already a classification of that type defined (we should account for sub 
types here)  
 * we should not allow more than one classification of a given type against an 
entity across explicitly defined classification and propagated classifications. 
This means that a policy only has to consider one classification and its 
ramifications. I am not sure how we resolve the potential conflicts; ideally 
this would be resolved using some rules and a  data steward. If we could search 
for these conflicts (maybe with a new conflicted status on the entity) , then a 
classification could be then explicitly defined on the entity and would force 
which classification values were effective.    

 

 

 

 

> Implement tag propagation using relationships
> -
>
> Key: ATLAS-2456
> URL: https://issues.apache.org/jira/browse/ATLAS-2456
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 1.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: TAG PROPAGATION IN ATLAS v.1.pdf, 
> tag_propagation_rest_api
>
>
> {color:#172b4d}Scalable way to quickly and efficiently propagate tags for 
> efficient searches and tag based security. Likewise tags for derivative 
> dataset should be inherited from the parent. For example, if an entity is 
> tagged "PII" then resulting entity created from a CTAS operation should also 
> be tagged "secret" to maintain the classification of the parent. In the case 
> where 2 or more datasets are aggregated the derivative dataset should be a 
> union of all parent tags.{color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (ATLAS-2456) Implement tag propagation using relationships

2018-02-19 Thread David Radley (JIRA)

[ 
https://issues.apache.org/jira/browse/ATLAS-2456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368955#comment-16368955
 ] 

David Radley commented on ATLAS-2456:
-

[~sarath.ku...@gmail.com] I have not had time to go through the code in a lot 
of detail. I think it looks pretty good. I wanted to check how you handle the 
following situations:

1) If there is a classification that is applied to an entity with certain 
attributes defined, and you decide to propagate the same classification to that 
entity, in this situation there must be 2 classification instances - each with 
different property values. 

So how does the code handle the case when there are 2 instances of the same 
classification with different property values that propagate to the same target 
entity.

2) I think the classifications defined in the open type 
GovernanceActionclassifications 
[https://cwiki.apache.org/confluence/display/ATLAS/Area+4+-+Governance] are 
going to be the key classifications around governance. There classifications 
are defined by their enums- which have an implied low-high order.  So for 
example confifidentialityLevel of sensitive is higher than internal. For point 
1 scenario- we would want the higher value to win.    

4) I think that any classification that is explicitly defined on an entity wins 
over propagated values.

5) Can you confirm that only one classification of a type is allowed for a 
given entity.

6) I assume that the entitytypes defined on a classificationdef will always 
constrain - so classification and propagated classification (or their subtypes) 
can only ever classify entities of those types (or their subtypes). 

7) I assume that a classification and subtypes of a classification could both 
be applied to an entity. This could compromise the idea of one classification 
per entity. How do you see this working or is this policed as invalid?  For 
example a Confidentiality classification and its subclass could be applied to 
an entity. - each specifying different levels of confidentiality - which one is 
effective?

> Implement tag propagation using relationships
> -
>
> Key: ATLAS-2456
> URL: https://issues.apache.org/jira/browse/ATLAS-2456
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 1.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: TAG PROPAGATION IN ATLAS v.1.pdf, 
> tag_propagation_rest_api
>
>
> {color:#172b4d}Scalable way to quickly and efficiently propagate tags for 
> efficient searches and tag based security. Likewise tags for derivative 
> dataset should be inherited from the parent. For example, if an entity is 
> tagged "PII" then resulting entity created from a CTAS operation should also 
> be tagged "secret" to maintain the classification of the parent. In the case 
> where 2 or more datasets are aggregated the derivative dataset should be a 
> union of all parent tags.{color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Review Request 65700: [ATLAS-2456]: Implement tag propagation using relationships

2018-02-18 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65700/#review197732
---




addons/models/-Area0/0010-base_model.json
Line 216 (original), 216 (patched)
<https://reviews.apache.org/r/65700/#comment277981>

you are adding tag propagation from a DataSet to a Process and from a 
Process to a dataset. I realise these are aggregation relationships, I wonder 
what prevents circularities.



common/src/main/java/org/apache/atlas/repository/Constants.java
Lines 72 (patched)
<https://reviews.apache.org/r/65700/#comment277982>

Can we use classification instead of traits or tags?



dashboardv2/public/js/utils/Enums.js
Lines 31 (patched)
<https://reviews.apache.org/r/65700/#comment277983>

agree with the previous reviewer that these tags seem the same as the TAG_ 
enums



repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphRetriever.java
Lines 1141 (patched)
<https://reviews.apache.org/r/65700/#comment277984>

I wonder if you could explain what ths creation of a propagation edge 
means. This implies that the propagated classifications are stored in the 
edges. I am wondering how this fits with associating the propagated 
classifications with the downstream entities, which I assume will be stored as 
propagation classifications in the downstream entities - have /i understood 
this correctly. A comment to explain how this works would be helpful.


- David Radley


On Feb. 18, 2018, 8:27 a.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65700/
> ---
> 
> (Updated Feb. 18, 2018, 8:27 a.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, and Madhan Neethiraj.
> 
> 
> Bugs: ATLAS-2456
> https://issues.apache.org/jira/browse/ATLAS-2456
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Scalable way to quickly and efficiently propagate tags for efficient searches 
> and tag based security. Likewise tags for derivative dataset should be 
> inherited from the parent. For example, if an entity is tagged "PII" then 
> resulting entity created from a CTAS operation should also be tagged "secret" 
> to maintain the classification of the parent. In the case where 2 or more 
> datasets are aggregated the derivative dataset should be a union of all 
> parent tags.
> 
> This changeset includes introduction of v2 data structures for notifications 
> and audit events.
> 
> 
> Diffs
> -
> 
>   addons/models/-Area0/0010-base_model.json 0296e8f9 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 265be788 
>   dashboardv2/public/js/utils/Enums.js a7b9a8b6 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java
>  42837f49 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java
>  a68d8ebe 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusVertex.java
>  aef20f03 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Vertex.java
>  ca48e3d9 
>   intg/src/main/java/org/apache/atlas/AtlasErrorCode.java ff09e6c9 
>   intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java 
> PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java 
> PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/instance/AtlasClassification.java 
> f594a814 
>   intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java 
> 08d1ce11 
>   
> intg/src/main/java/org/apache/atlas/model/notification/EntityNotification.java
>  b272b733 
>   
> intg/src/main/java/org/apache/atlas/v1/model/notification/EntityNotificationV2.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java
>  74d3b913 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java
>  9dc78350 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
>  774934c7 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java
>  22d2a810 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java
>  c3826019 
>   
> repository/src/main/java/org/apache/atlas/

Unable to resolve Jiras

2018-02-16 Thread David Radley
Hi Madhan,
I have pushed the changed for Jiras 1095 and 2246. I do not seem to be 
able to resolve these Jiras. Any thoughts? 
  all the best, David. 
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


[jira] [Commented] (ATLAS-2246) Define OMRS Connector interface

2018-02-16 Thread David Radley (JIRA)

[ 
https://issues.apache.org/jira/browse/ATLAS-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366808#comment-16366808
 ] 

David Radley commented on ATLAS-2246:
-

I have committed the code

commit 8a57e6571c0079c25c147732ad3a6994be8e14ea (HEAD -> master, origin/master, 
origin/HEAD)
Author: Mandy Chessell <mandy_chess...@uk.ibm.com>
Date:   Thu Feb 15 12:46:01 2018 +

    ATLAS-2246: OMRS Connector API plus REST and IGC Connector skeleton - 15th 
February 2018
    
    Signed-off-by: David Radley <david_rad...@uk.ibm.com>

> Define OMRS Connector interface
> ---
>
> Key: ATLAS-2246
> URL: https://issues.apache.org/jira/browse/ATLAS-2246
> Project: Atlas
>  Issue Type: Sub-task
>  Components: atlas-intg
>Reporter: Mandy Chessell
>Assignee: Mandy Chessell
>Priority: Critical
> Attachments: 
> 0001-ATLAS-2246-OMRS-Connector-API-plus-REST-and-IGC-Conn.patch, OMRS 
> Javadoc.zip
>
>
> This task covers the creation of the OMRS Connector APIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Review Request 63502: ATLAS-2246 - Review request for the OMRS Connector API.

2018-02-16 Thread David Radley


> On Feb. 15, 2018, 6:13 p.m., Jerry He wrote:
> > Can we put the omrs module in the top level pom so that it builds the omrs 
> > module?

I agree, this is where it needs to go. We were thinking that we would leave it 
out of the build for now until omrs is more fleshed out.
It is really useful for the community developers to have this code checked in, 
so that can code to it. 
The Atlas connector is once piece of code that is being developed - this will 
allow omrs calls to isue requests to Atlas; I think when we have that, we can 
link it into the build. Personally I have added the line to the pom.xml to 
bring omrs into my local builds.


- David


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63502/#review197628
---


On Feb. 15, 2018, 7:03 p.m., Mandy Chessell wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63502/
> ---
> 
> (Updated Feb. 15, 2018, 7:03 p.m.)
> 
> 
> Review request for atlas, David Radley and Madhan Neethiraj.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This is a request for code review of the OMRS.  There are still parts of the 
> code not written - the aim of the review is to make the connector interfaces 
> available so they can be evaluated by teams buinding components against the 
> APIs.
> 
> There is a patch connected to 
> https://issues.apache.org/jira/projects/ATLAS/issues/ATLAS-2246 that has the 
> code plus other code that is not ready for review.  The OMRS Connector API 
> depends on the OCF - see https://issues.apache.org/jira/browse/ATLAS-1095.
> 
> 
> Diffs
> -
> 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ConnectorBase.java cc2e430 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ConnectorProviderBase.java 
> fafd20c 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/Connection.java 
> d34e6f4 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/KeyPattern.java 
> ce6dae1 
>   omrs/README.md PRE-CREATION 
>   omrs/pom.xml PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/eventmapper/AtlasOMRSRepositoryEventMapper.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/eventmapper/AtlasOMRSRepositoryEventMapperProvider.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/repositoryconnector/LocalAtlasOMRSMetadataCollection.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/repositoryconnector/LocalAtlasOMRSRepositoryConnector.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/repositoryconnector/LocalAtlasOMRSRepositoryConnectorProvider.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v1/eventmapper/IGCOMRSRepositoryEventMapper.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v1/eventmapper/IGCOMRSRepositoryEventMapperProvider.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v1/repositoryconnector/IGCOMRSMetadataCollection.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v1/repositoryconnector/IGCOMRSRepositoryConnector.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v1/repositoryconnector/IGCOMRSRepositoryConnectorProvider.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v2/eventmapper/IGCV2OMRSRepositoryEventMapper.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v2/eventmapper/IGCV2OMRSRepositoryEventMapperProvider.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v2/repositoryconnector/IGCV2OMRSMetadataCollection.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v2/repositoryconnector/IGCV2OMRSRepositoryConnector.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v2/repositoryconnector/IGCV2OMRSRepositoryConnectorProvider.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/admin/OMRSConfigurationFactory.java 
> PRE-CREATION 
>   omrs/src/main/java/org/apache/atlas/omrs/admin/OMRSOperationalServices.java 
> PRE-CREATION 
>   omrs/src/main/java/org/apache/atlas/omrs/admin/properties/CohortConfig.java 
> PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/admin/properties/Enterprise

Re: Metadata security policies examples

2018-02-16 Thread David Radley
Hi Madhan,
My thoughts on the 3 points are :
1) It would seem to me that we should have one way of identifying users at 
the metadata layer and any normalisation or mapping should be done prior 
to this so that users are uniquely identified in an organization in the 
policy / metadata layer..
2)  You say "Wouldn't it be more flexible to allow different set of users 
to edit and classify entities? Similar to the following scenario." I think 
it is not an 'or ' situation but an 'and'.  I think it is beneficial to 
restrict based on a list of users as you suggest. It is also useful to be 
able to have some principles  as Mandy suggests, so if you have certain 
access rights then you get other access rights by virtual of a policy. 
Having these principles expressed in policies means that the policies can 
be simpler and more naturally expressed. 
3) Agreed.

  all the best, David. 



From:   Madhan Neethiraj 
To: "dev@atlas.apache.org" 
Date:   16/02/2018 02:23
Subject:Re: Metadata security policies examples



Mandy,

> edit access is limited to the user identified in the createdBy property.
It will be possible to support authorization as above. However, there are 
few issues to be aware of in such approach. For example:
 - 'createdBy' property may not be present in all entity-types
 - the username in 'createdBy' property might have to be normalized or 
mapped, to be able to compare with the username logged into Atlas.
   - for example john.sc...@example.com vs jscott
I would recommend to handle such username normalization/mapping in a later 
phase.

> where a classification can only be added to an entity by a user that has 
edit access to the entity.
> where a classification can only be added to any entity by a user with 
create rights on the classification.
Wouldn't it be more flexible to allow different set of users to edit and 
classify entities? Similar to the following scenario.

> where edit access to an entity is required before a relationship can 
connect it to something else
> and it would be good from a graph decoupling point of view if adding 
relationships could be done independently of the access rights to either 
entity.
I agree on decoupling authorization to create relationship from access 
rights to the entities at both ends.

Thanks,
Madhan

On 2/14/18, 10:06 AM, "Mandy Chessell"  
wrote:

Hello Madhan,
I was thinking through our common use cases for metadata security. For 

most metadata entities and relationships, we would want to enforce 
that 
metadata is readable by logged on users but edit access is limited to 
the 
user identified in the createdBy property. 
 
Then we have special cases for entities such as connections and some 
governance actions.
For example there may be a connection to an audit log and that can 
only be 
seen by members of the security team since having access to the 
connection 
means you can connect to the data store.
Some governance actions may be updateable by anyone in the governance 
team 
- not just the creator.
 
When it comes to classifications, we have 2 scenarios 
- where a classification can only be added to an entity by a user that 
has 
edit access to the entity.
- where a classification can only be added to any entity by a user 
with 
create rights on the classification.
 
I was trying to think through similar examples for relationships - for 

example, where edit access to an entity is required before a 
relationship 
can connect it to something else - but I can't think of one - and it 
would 
be good from a graph decoupling point of view if adding relationships 
could be done independently of the access rights to either entity.
 
All the best
Mandy
___
Mandy Chessell CBE FREng CEng FBCS
IBM Distinguished Engineer
 
Master Inventor
Member of the IBM Academy of Technology
Visiting Professor, Department of Computer Science, University of 
Sheffield
 
Email: mandy_chess...@uk.ibm.com
LinkedIn: 
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.linkedin.com_pub_mandy-2Dchessell_22_897_a49=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=QhpUQPr5YlG95aAgCvZGStEXHg4hBbSYQ9JkRqR_svY=Qa-FKldwlfxLK0OygoMR9IopavMa6ccJ81Xtg4l11cs=20HZiMDEZtwudUrz9j12W32Kb8AklTGhbNDPAg5AN6s=

 
Assistant: Janet Brooks - jsbrook...@uk.ibm.com





Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


Attempting to run Atlas with external HBase

2018-02-15 Thread David Radley
Hi Madhan,
Based on our meeting earlier this week, you suggested that we run with an 
external HBase rather than the embedded HBase, which you have found has 
delays around some transaction commits.
I am not very familiar with HBase and wondered if you could point me in 
the right direction. This is what I did:

1) I ran an embedded hbase build and found the solr and hbase tar.gz files
2) I then expanded these archives in a new runtime folder
3) I built Atlas without the embedded hbase option and then copied over 
the atlas tree into the runtime folder.
4)  I found https://atlas.apache.org/InstallationSteps.html .
- It indicates I should specify export ATLAS_SERVER_OPTS="-server 
-XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSClassUnloadingEnabled 
-XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled 
-XX:+PrintTenuringDistribution -XX:+HeapDumpOnOutOfMemoryError 
-XX:HeapDumpPath=dumps/atlas_server.hprof -Xloggc:logs/gc-worker.log 
-verbose:gc -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 
-XX:GCLogFileSize=1m -XX:+PrintGCDetails -XX:+PrintHeapAtGC 
-XX:+PrintGCTimeStamps"
which I put in the atlas_env.sh
- As I am on a mac it suggests I specify  :
export ATLAS_SERVER_OPTS="-Djava.awt.headless=true 
-Djava.security.krb5.realm= -Djava.security.krb5.kdc="
 which I did.
- It says I should change the config to: 
"atlas.graph.storage.hbase.table=atlas
atlas.audit.hbase.tablename=apache_atlas_entity_audit
"
Is this correct - as the application.properties that has been generated is 
"atlas.graph.storage.hbase.table=apache_atlas_titan". 
- It says I should start solr with "$SOLR_HOME/bin/solr start -c -
z  -p 8983". I do not know what to put in for <
zookeeper_host:port>. Do I need to specify this if I am using the solr 
embedded zookeeper?
- It then says I should run
"  $SOLR_BIN/solr create -c vertex_index -d SOLR_CONF -shards #numShards 
-replicationFactor #replicationFactor
  $SOLR_BIN/solr create -c edge_index -d SOLR_CONF -shards #numShards 
-replicationFactor #replicationFactor
  $SOLR_BIN/solr create -c fulltext_index -d SOLR_CONF -shards #numShards 
-replicationFactor #replicationFactor"
It seems I need to specify numbers for numshards and 
replicationFactor, can I let these default, if not what do I specify here?
- It then says I need to specify 
"atlas.graph.index.search.backend=solr5
 atlas.graph.index.search.solr.mode=cloud
 atlas.graph.index.search.solr.zookeeper-url= eg: 10.1.6.4:2181,10.1.6.5:2181
 atlas.graph.index.search.solr.zookeeper-connect-timeout=. Default value is 6 ms
 atlas.graph.index.search.solr.zookeeper-session-timeout=. Default value is 6 ms"

I am not sure what to put for atlas.graph.index.search.solr.zookeeper-url. 
should we be using the solr embedded ZK - if do I need this line?

I updated HBASE_CONF_DIR to point to the conf folder of the hbase I had 
expanded. 

I start solr using $SOLR_HOME/bin/solr start -c -p 8983
I start hbase 
I start atlas.

Atlas says it successfully has started but the last line in the 
application.log says :

2018-02-15 17:25:23,755 INFO  - [main:] ~ Not running setup per 
configuration atlas.server.run.setup.on.start. 
(SetupSteps$SetupRequired:189)

The installation twiki talks of "If the setup failed due to HBase 
JanusGraph schema setup errors, it may be necessary to repair the HBase 
schema. If no
data has been stored, one can also disable and drop the HBase tables used 
by Atlas and run setup again." . It does not indicate what commands I need 
to run and how. 


many thanks , David. 






 


 
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


Re: Review Request 63502: ATLAS-2246 - Review request for the OMRS Connector API.

2018-02-15 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63502/#review197619
---


Ship it!




Ship It!

- David Radley


On Feb. 15, 2018, 12:49 p.m., Mandy Chessell wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63502/
> ---
> 
> (Updated Feb. 15, 2018, 12:49 p.m.)
> 
> 
> Review request for atlas, David Radley and Madhan Neethiraj.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This is a request for code review of the OMRS.  There are still parts of the 
> code not written - the aim of the review is to make the connector interfaces 
> available so they can be evaluated by teams buinding components against the 
> APIs.
> 
> There is a patch connected to 
> https://issues.apache.org/jira/projects/ATLAS/issues/ATLAS-2246 that has the 
> code plus other code that is not ready for review.  The OMRS Connector API 
> depends on the OCF - see https://issues.apache.org/jira/browse/ATLAS-1095.
> 
> 
> Diffs
> -
> 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ConnectorBase.java cc2e430 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ConnectorProviderBase.java 
> fafd20c 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/Connection.java 
> d34e6f4 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/KeyPattern.java 
> ce6dae1 
>   omrs/README.md PRE-CREATION 
>   omrs/pom.xml PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/eventmapper/AtlasOMRSRepositoryEventMapper.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/eventmapper/AtlasOMRSRepositoryEventMapperProvider.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/repositoryconnector/LocalAtlasOMRSMetadataCollection.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/repositoryconnector/LocalAtlasOMRSRepositoryConnector.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/repositoryconnector/LocalAtlasOMRSRepositoryConnectorProvider.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v1/eventmapper/IGCOMRSRepositoryEventMapper.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v1/eventmapper/IGCOMRSRepositoryEventMapperProvider.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v1/repositoryconnector/IGCOMRSMetadataCollection.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v1/repositoryconnector/IGCOMRSRepositoryConnector.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v1/repositoryconnector/IGCOMRSRepositoryConnectorProvider.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v2/eventmapper/IGCV2OMRSRepositoryEventMapper.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v2/eventmapper/IGCV2OMRSRepositoryEventMapperProvider.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v2/repositoryconnector/IGCV2OMRSMetadataCollection.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v2/repositoryconnector/IGCV2OMRSRepositoryConnector.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v2/repositoryconnector/IGCV2OMRSRepositoryConnectorProvider.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/admin/OMRSConfigurationFactory.java 
> PRE-CREATION 
>   omrs/src/main/java/org/apache/atlas/omrs/admin/OMRSOperationalServices.java 
> PRE-CREATION 
>   omrs/src/main/java/org/apache/atlas/omrs/admin/properties/CohortConfig.java 
> PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/admin/properties/EnterpriseAccessConfig.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/admin/properties/LocalRepositoryConfig.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/admin/properties/OpenMetadataEventProtocolVersion.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/admin/properties/OpenMetadataExchangeRule.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/admin/properties/RepositoryServicesConfig.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/archivemanager/OMRSArchiveBuilder.java
>  PRE-CREATI

[jira] [Created] (ATLAS-2449) Latest master atlas does not build cleanly with tests enabled locally on Mac

2018-02-15 Thread David Radley (JIRA)
David Radley created ATLAS-2449:
---

 Summary: Latest master atlas does not build cleanly with tests 
enabled locally on Mac
 Key: ATLAS-2449
 URL: https://issues.apache.org/jira/browse/ATLAS-2449
 Project: Atlas
  Issue Type: Bug
Reporter: David Radley


I run mvn clean install -Pdist,embedded-hbase-solr

And it fails with:

127.0.0.1 - - [15/Feb/2018:14:06:01 +] "GET 
//localhost:31000/api/atlas/v2/types/typedef/name/hive_table_v1 HTTP/1.1" 503 
372 "-" "Java/1.8.0_151"
127.0.0.1 - - [15/Feb/2018:14:06:06 +] "GET 
//localhost:31000/api/atlas/v2/types/typedef/name/hive_table_v1 HTTP/1.1" 503 
372 "-" "Java/1.8.0_151"
127.0.0.1 - - [15/Feb/2018:14:06:11 +] "GET 
//localhost:31000/api/atlas/v2/types/typedef/name/hive_process_v1 HTTP/1.1" 503 
374 "-" "Java/1.8.0_151"
127.0.0.1 - - [15/Feb/2018:14:06:16 +] "GET 
//localhost:31000/api/atlas/v2/types/typedef/name/hive_process_v1 HTTP/1.1" 503 
374 "-" "Java/1.8.0_151"
127.0.0.1 - - [15/Feb/2018:14:06:21 +] "GET 
//localhost:31000/api/atlas/v2/types/typedef/name/hive_process_v1 HTTP/1.1" 503 
374 "-" "Java/1.8.0_151"
127.0.0.1 - - [15/Feb/2018:14:06:26 +] "GET 
//localhost:31000/api/atlas/v2/types/typedef/name/hive_process_v1 HTTP/1.1" 503 
374 "-" "Java/1.8.0_151"
127.0.0.1 - - [15/Feb/2018:14:06:31 +] "POST 
//localhost:31000/api/atlas/types HTTP/1.1" 503 342 "-" "Java/1.8.0_151"
127.0.0.1 - - [15/Feb/2018:14:06:36 +] "POST 
//localhost:31000/api/atlas/types HTTP/1.1" 503 342 "-" "Java/1.8.0_151"
127.0.0.1 - - [15/Feb/2018:14:06:41 +] "POST 
//localhost:31000/api/atlas/types HTTP/1.1" 503 342 "-" "Java/1.8.0_151"
127.0.0.1 - - [15/Feb/2018:14:06:46 +] "POST 
//localhost:31000/api/atlas/types HTTP/1.1" 503 342 "-" "Java/1.8.0_151"
Tests run: 143, Failures: 11, Errors: 0, Skipped: 125, Time elapsed: 2,422.962 
sec <<< FAILURE! - in TestSuite
setUp(org.apache.atlas.web.integration.DataSetLineageJerseyResourceIT)  Time 
elapsed: 510.086 sec  <<< FAILURE!
org.apache.atlas.AtlasServiceException: Metadata service API 
org.apache.atlas.AtlasClient$API_V1@5161ae92 failed with status 503 (Service 
Unavailable) Response Body (


Error 503 


HTTP ERROR: 503
Problem accessing /api/atlas/types. Reason:
    Service Unavailable
http://eclipse.org/jetty;>Powered by Jetty:// 
9.3.14.v20161028


)
    at 
org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:388)
    at 
org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:323)
    at org.apache.atlas.AtlasBaseClient.callAPI(AtlasBaseClient.java:211)
    at org.apache.atlas.AtlasClient.callAPIWithBody(AtlasClient.java:906)
    at org.apache.atlas.AtlasClient.createType(AtlasClient.java:257)
    at org.apache.atlas.AtlasClient.createType(AtlasClient.java:275)
    at 
org.apache.atlas.web.integration.BaseResourceIT.createType(BaseResourceIT.java:213)
    at 
org.apache.atlas.web.integration.BaseResourceIT.createTypeDefinitionsV1(BaseResourceIT.java:360)
    at 
org.apache.atlas.web.integration.DataSetLineageJerseyResourceIT.setUp(DataSetLineageJerseyResourceIT.java:60)

setUp(org.apache.atlas.notification.NotificationHookConsumerIT)  Time elapsed: 
362.578 sec  <<< FAILURE!
org.apache.atlas.AtlasServiceException: Metadata service API 
org.apache.atlas.AtlasClient$API_V1@5161ae92 failed with status 503 (Service 
Unavailable) Response Body (


Error 503 


HTTP ERROR: 503
Problem accessing /api/atlas/types. Reason:
    Service Unavailable
http://eclipse.org/jetty;>Powered by Jetty:// 
9.3.14.v20161028


)
    at 
org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:388)
    at 
org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:323)
    at org.apache.atlas.AtlasBaseClient.callAPI(AtlasBaseClient.java:211)
    at org.apache.atlas.AtlasClient.callAPIWithBody(AtlasClient.java:906)
    at org.apache.atlas.AtlasClient.createType(AtlasClient.java:257)
    at org.apache.atlas.AtlasClient.createType(AtlasClient.java:275)
    at 
org.apache.atlas.web.integration.BaseResourceIT.createType(BaseResourceIT.java:213)
    at 
org.apache.atlas.web.integration.BaseResourceIT.createTypeDefinitionsV1(BaseResourceIT.java:360)
    at 
org.apache.atlas.notification.NotificationHookConsumerIT.setUp(NotificationHookConsumerIT.java:54)

testAccessforUnauthenticatedResource(org.apache.atlas.web.filters.AtlasAuthenticationSimpleFilterIT)
  Time elapsed: 0.003 sec  <<< FAILURE!
java.lang.AssertionError: expected [200] but found [503]
    at org.testng.Assert.fail(As

Re: Server appears to start, but connections fail with 503

2018-02-14 Thread David Radley
Hi Ernie,
There are 2 ways to build and run Atlas currently:
- external Hbase and Solr and ZK
- embedded HBase Solr and ZK

I am not sure which one you are trying. Sharing your build string would be 
helpful. If you are using the embedded HBase option then you need to 
specify environment variables, for the runtime Atlas to work,

As per yesterdays call - Madhan suggests running with an external HBase, 
as he seems a performance delay using the embedded HBase. 
   all the best, David. 



From:   "Ernie Ostic" 
To: "Atlas" 
Date:   14/02/2018 11:52
Subject:Server appears to start, but connections fail with 503





Hi all...

Using "master", the build appears to be successful, and the server is
properly listening on port 21000, but quick start or even direct access
fails with 503.   Any suggestions on where to look?   Thanks!

Ernie



$ bin/quick_start.py

log4j:WARN No such property [maxFileSize] in
org.apache.log4j.PatternLayout.
log4j:WARN No such property [maxBackupIndex] in
org.apache.log4j.PatternLayout.
log4j:WARN No such property [maxFileSize] in
org.apache.log4j.PatternLayout.
log4j:WARN No such property [maxBackupIndex] in
org.apache.log4j.PatternLayout.
log4j:WARN No such property [maxFileSize] in
org.apache.log4j.PatternLayout.

Enter username for atlas :- admin
Enter password for atlas :-

Creating sample types:

Exception in thread "main" org.apache.atlas.AtlasServiceException: 
Metadata
service API org.apache.atlas.AtlasClientV2$API_V2@1033576a failed with
status 503 (Service Unavailable) Response Body (


Error 503 


HTTP ERROR: 503
Problem accessing /api/atlas/v2/types/typedefs/. Reason:
Service Unavailable
https://urldefense.proofpoint.com/v2/url?u=http-3A__eclipse.org_jetty=DwMFAg=jf_iaSHvJObTbx-siA1ZOg=QhpUQPr5YlG95aAgCvZGStEXHg4hBbSYQ9JkRqR_svY=0y7mtLMoBMEc2RNTv8nhllpgWZHIGzqdnqcU0FUYtvg=yRGEThijdpp0lFksddATlowggX7r9D_zX0z2DXZR8EA=
">Powered by Jetty://
9.3.14.v20161028


)
 at org.apache.atlas.AtlasBaseClient.callAPIWithResource
(AtlasBaseClient.java:388)
 at org.apache.atlas.AtlasBaseClient.callAPIWithResource
(AtlasBaseClient.java:323)
 at 
org.apache.atlas.AtlasBaseClient.callAPI(AtlasBaseClient.java:211)
 at org.apache.atlas.AtlasClientV2.createAtlasTypeDefs
(AtlasClientV2.java:227)
 at org.apache.atlas.examples.QuickStartV2.createTypes
(QuickStartV2.java:185)
 at org.apache.atlas.examples.QuickStartV2.runQuickstart
(QuickStartV2.java:141)
 at 
org.apache.atlas.examples.QuickStartV2.main(QuickStartV2.java:126)

No sample data added to Apache Atlas Server.



Ernie Ostic

Worldwide Technical Sales
InfoSphere Information Server
IBM Analytics

Cell: (617) 331 8238
---
Apache Atlas Update!
https://dsrealtime.wordpress.com/2017/11/16/apache-atlas-update-have-you-been-watching/


Open IGC is here!
https://dsrealtime.wordpress.com/2015/07/29/open-igc-is-here/



Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


Re: Review Request 63502: ATLAS-2246 - Review request for the OMRS Connector API.

2018-02-13 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63502/#review197399
---




omrs/src/main/java/org/apache/atlas/omrs/metadatacollection/properties/instances/EntitySummary.java
Lines 71 (patched)
<https://reviews.apache.org/r/63502/#comment277546>

It would be normal to only expose List here are the return value and not 
expose the type of List, so you do not expose the List implementation.
The same for the setter.


- David Radley


On Jan. 29, 2018, 9:51 a.m., Mandy Chessell wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63502/
> ---
> 
> (Updated Jan. 29, 2018, 9:51 a.m.)
> 
> 
> Review request for atlas, David Radley and Madhan Neethiraj.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This is a request for code review of the OMRS.  There are still parts of the 
> code not written - the aim of the review is to make the connector interfaces 
> available so they can be evaluated by teams buinding components against the 
> APIs.
> 
> There is a patch connected to 
> https://issues.apache.org/jira/projects/ATLAS/issues/ATLAS-2246 that has the 
> code plus other code that is not ready for review.  The OMRS Connector API 
> depends on the OCF - see https://issues.apache.org/jira/browse/ATLAS-1095.
> 
> 
> Diffs
> -
> 
>   omrs/README.md PRE-CREATION 
>   omrs/pom.xml PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/eventmapper/AtlasOMRSRepositoryEventMapper.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/eventmapper/AtlasOMRSRepositoryEventMapperProvider.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/repositoryconnector/LocalAtlasOMRSMetadataCollection.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/repositoryconnector/LocalAtlasOMRSRepositoryConnector.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/repositoryconnector/LocalAtlasOMRSRepositoryConnectorProvider.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/eventmapper/IGCOMRSRepositoryEventMapper.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/eventmapper/IGCOMRSRepositoryEventMapperProvider.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/repositoryconnector/IGCOMRSMetadataCollection.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/repositoryconnector/IGCOMRSRepositoryConnector.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/repositoryconnector/IGCOMRSRepositoryConnectorProvider.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/admin/OMRSConfigurationFactory.java 
> PRE-CREATION 
>   omrs/src/main/java/org/apache/atlas/omrs/admin/OMRSOperationalServices.java 
> PRE-CREATION 
>   omrs/src/main/java/org/apache/atlas/omrs/admin/properties/CohortConfig.java 
> PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/admin/properties/EnterpriseAccessConfig.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/admin/properties/LocalRepositoryConfig.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/admin/properties/OpenMetadataEventProtocolVersion.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/admin/properties/OpenMetadataExchangeRule.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/admin/properties/RepositoryServicesConfig.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/archivemanager/OMRSArchiveManager.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/archivemanager/store/OpenMetadataArchiveStore.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/archivemanager/store/OpenMetadataArchiveStoreConnectorBase.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/archivemanager/store/OpenMetadataArchiveStoreProviderBase.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/archivemanager/store/properties/OpenMetadataArchive.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/archivemanager/store/properties/OpenMetadataArchiveInstanceStore.java
>  PRE-CREATION 
>   
> omrs/src/main/java/org/apache/atlas/omrs/archivemanager/store/properties/Op

[jira] [Commented] (ATLAS-1095) Open connector framework

2018-01-27 Thread David Radley (JIRA)

[ 
https://issues.apache.org/jira/browse/ATLAS-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16342239#comment-16342239
 ] 

David Radley commented on ATLAS-1095:
-

I have committed the patch, the details are :

Author    Mandy Chessell <mandy_chess...@uk.ibm.com>    
    Fri, 19 Jan 2018 09:27:26 + (09:27 +)
committer    David Radley <david_rad...@uk.ibm.com>    
    Sat, 27 Jan 2018 00:40:56 + (00:40 +)
commit    cbfdd7fcc2ccb011c6f3000e3cb7a9de79268445
tree    bc710344fde1adac92c65ec74e1042cc34960db4    tree | snapshot
parent    804c4635e66e05fae627ad1e9344c1b11251ad35    commit | diff

> Open connector framework
> 
>
> Key: ATLAS-1095
> URL: https://issues.apache.org/jira/browse/ATLAS-1095
> Project: Atlas
>  Issue Type: New Feature
>Affects Versions: 1.0.0
>Reporter: Stephanie Hazlewood
>Assignee: David Radley
>Priority: Major
>  Labels: VirtualDataConnector
> Attachments: 0001-ATLAS-1095-final-code-drop-for-OCF.patch, OCF 
> JavaDoc.zip, Open Connector Framework - 20th June 2017.doc, Open Connector 
> Framework - 9th May 2017.doc
>
>
> Atlas provides a common approach to metadata management and governance across 
> all systems and data within an organization.  Today Atlas provides access to 
> metadata.   A connector provides access to a data source.  As connectors are 
> the proxy of all data, they can also be explicit providers of metadata.   
> This JIRA proposes an open connector framework to manage connectors that 
> provide access to both data and the metadata Atlas provides together through 
> a single connector interface.  
> This will help data tools to to better the exchange of information between 
> platforms. It also offers new opportunities for the consistent enforcement of 
> the governance policies and rules (e.g., rules of visibility).  Source 
> connector/connection metadata provides the nucleus around which all other 
> metadata describing the data builds.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (ATLAS-1095) Open connector framework

2018-01-26 Thread David Radley (JIRA)

 [ 
https://issues.apache.org/jira/browse/ATLAS-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley reassigned ATLAS-1095:
---

Assignee: David Radley  (was: Mandy Chessell)

> Open connector framework
> 
>
> Key: ATLAS-1095
> URL: https://issues.apache.org/jira/browse/ATLAS-1095
> Project: Atlas
>  Issue Type: New Feature
>Affects Versions: 1.0.0
>Reporter: Stephanie Hazlewood
>Assignee: David Radley
>Priority: Major
>  Labels: VirtualDataConnector
> Attachments: 0001-ATLAS-1095-final-code-drop-for-OCF.patch, OCF 
> JavaDoc.zip, Open Connector Framework - 20th June 2017.doc, Open Connector 
> Framework - 9th May 2017.doc
>
>
> Atlas provides a common approach to metadata management and governance across 
> all systems and data within an organization.  Today Atlas provides access to 
> metadata.   A connector provides access to a data source.  As connectors are 
> the proxy of all data, they can also be explicit providers of metadata.   
> This JIRA proposes an open connector framework to manage connectors that 
> provide access to both data and the metadata Atlas provides together through 
> a single connector interface.  
> This will help data tools to to better the exchange of information between 
> platforms. It also offers new opportunities for the consistent enforcement of 
> the governance policies and rules (e.g., rules of visibility).  Source 
> connector/connection metadata provides the nucleus around which all other 
> metadata describing the data builds.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: Review Request 63503: ATLAS-1095 - Review of OCF - the Open Connector Framework

2018-01-26 Thread David Radley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63503/#review196317
---


Ship it!




Ship It!

- David Radley


On Jan. 19, 2018, 9:30 a.m., Mandy Chessell wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63503/
> ---
> 
> (Updated Jan. 19, 2018, 9:30 a.m.)
> 
> 
> Review request for atlas and Madhan Neethiraj.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch contains the open connector framework code.  This code is located 
> in the om-fwk-ocf component and is described in JIRA 
> https://issues.apache.org/jira/browse/ATLAS-1095
> 
> I have added a new patch to the Jira with fixes from Yao's comments.  
> Upgraded Maven to 352 and rebuilt/rerun tests.
> 
> 
> Diffs
> -
> 
>   om-fwk-ocf/README.md PRE-CREATION 
>   om-fwk-ocf/pom.xml PRE-CREATION 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/Connector.java PRE-CREATION 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ConnectorBase.java 
> PRE-CREATION 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ConnectorBroker.java 
> PRE-CREATION 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ConnectorProvider.java 
> PRE-CREATION 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ConnectorProviderBase.java 
> PRE-CREATION 
>   
> om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ffdc/ConnectionCheckedException.java
>  PRE-CREATION 
>   
> om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ffdc/ConnectorCheckedException.java
>  PRE-CREATION 
>   
> om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ffdc/OCFCheckedExceptionBase.java
>  PRE-CREATION 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ffdc/OCFErrorCode.java 
> PRE-CREATION 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ffdc/OCFRuntimeException.java 
> PRE-CREATION 
>   
> om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ffdc/PropertyServerException.java
>  PRE-CREATION 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ffdc/README.md PRE-CREATION 
>   
> om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/AdditionalProperties.java
>  PRE-CREATION 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/Annotation.java 
> PRE-CREATION 
>   
> om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/AnnotationStatus.java
>  PRE-CREATION 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/Annotations.java 
> PRE-CREATION 
>   
> om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/AssetDescriptor.java 
> PRE-CREATION 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/AssetDetail.java 
> PRE-CREATION 
>   
> om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/AssetPropertyBase.java
>  PRE-CREATION 
>   
> om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/AssetPropertyIteratorBase.java
>  PRE-CREATION 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/AssetSummary.java 
> PRE-CREATION 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/AssetUniverse.java 
> PRE-CREATION 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/Certification.java 
> PRE-CREATION 
>   
> om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/Certifications.java 
> PRE-CREATION 
>   
> om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/Classification.java 
> PRE-CREATION 
>   
> om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/Classifications.java 
> PRE-CREATION 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/Comment.java 
> PRE-CREATION 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/CommentType.java 
> PRE-CREATION 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/Comments.java 
> PRE-CREATION 
>   
> om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/ConnectedAssetProperties.java
>  PRE-CREATION 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/Connection.java 
> PRE-CREATION 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/Connections.java 
> PRE-CREATION 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/ConnectorType.java 
> PRE-CREATION 
>   
> om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/DerivedSchemaElement.java
>  PRE-CREATION 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/ElementHeader.java 
> PRE-CREATION 
>   om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/ElementOrigin.java 
> PRE-CREATION 
>   om-fwk-ocf/src

  1   2   3   4   5   6   >