Build failed in Jenkins: apache-atlas-nightly #756

[Apache Jenkins Server]

See 


Changes:

[ssubramanian] Remove patch file committed as part of ATLAS-1819

--
[...truncated 273.72 KB...]
[WARNING] Note: Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ 
atlas-repository ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 54 source files to 

[WARNING] 
:
 Some input files use or override a deprecated API.
[WARNING] 
:
 Recompile with -Xlint:deprecation for details.
[WARNING] 
:
 Some input files use unchecked or unsafe operations.
[WARNING] 
:
 Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ atlas-repository 
---
[INFO] Surefire report directory: 

[INFO] Using configured provider org.apache.maven.surefire.testng.TestNGProvider
[WARNING] file.encoding cannot be set as system property, use 
-Dfile.encoding=... instead

---
 T E S T S
---
Running org.apache.atlas.repository.impexp.ImportServiceReportingTest
Running 
org.apache.atlas.repository.store.graph.v1.InverseReferenceUpdateHardDeleteV1Test
Running org.apache.atlas.service.StoreBackedTypeCacheMetadataServiceTest
Running org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1Test
Running org.apache.atlas.repository.impexp.ZipSinkTest
Running org.apache.atlas.repository.graph.ReverseReferenceUpdateSoftDeleteTest
Running org.apache.atlas.repository.graph.GraphBackedRepositorySoftDeleteTest
Running org.apache.atlas.discovery.DataSetLineageServiceTest
Running org.apache.atlas.repository.impexp.ExportServiceTest
Running org.apache.atlas.util.CompiledQueryCacheKeyTest
Running org.apache.atlas.discovery.GraphBackedDiscoveryServiceTest
Running org.apache.atlas.repository.audit.HBaseBasedAuditRepositoryTest
Running org.apache.atlas.repository.graph.ReverseReferenceUpdateHardDeleteTest
Running org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStoreTest
Running 
org.apache.atlas.repository.store.graph.v1.InverseReferenceUpdateSoftDeleteV1Test
Running org.apache.atlas.service.DefaultMetadataServiceTest
Running org.apache.atlas.repository.audit.InMemoryAuditRepositoryTest
Running org.apache.atlas.repository.store.graph.AtlasEntityDefStoreV1Test
Running org.apache.atlas.repository.store.graph.v1.AtlasDeleteHandlerV1Test
Running org.apache.atlas.repository.graph.GraphBackedMetadataRepositoryTest
Running org.apache.atlas.repository.store.graph.v1.InverseReferenceUpdateV1Test
Running org.apache.atlas.repository.store.graph.v1.HardDeleteHandlerV1Test
Running org.apache.atlas.repository.store.graph.v1.SoftDeleteHandlerV1Test
Running org.apache.atlas.RepositoryServiceLoadingTest
Running org.apache.atlas.TestModules
Running org.apache.atlas.repository.graph.GraphBackedRepositoryHardDeleteTest
Running org.apache.atlas.repository.graph.TestIntSequence
Running org.apache.atlas.repository.graph.Gremlin3QueryOptimizerTest
Running org.apache.atlas.repository.graph.GraphHelperTest
Running org.apache.atlas.repository.impexp.ZipSourceTest
Running org.apache.atlas.repository.audit.HBaseBasedAuditRepositoryHATest
Running org.apache.atlas.repository.impexp.ImportServiceTest
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.46 sec - in 
org.apache.atlas.repository.impexp.ImportServiceReportingTest
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.357 sec - in 
org.apache.atlas.repository.graph.TestIntSequence
Running org.apache.atlas.repository.graph.AbstractGremlinQueryOptimizerTest
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.024 sec - in 
org.apache.atlas.repository.store.graph.v1.AtlasDeleteHandlerV1Test
Tests run: 33, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.24 sec - in 
org.apache.atlas.repository.graph.Gremlin3QueryOptimizerTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.787 sec - in 
org.apache.atlas.util.CompiledQueryCacheKeyTest
Running org.apache.atlas.repository.graph.GraphBackedSearchIndexerMockTest

Build failed in Jenkins: apache-atlas-nightly #755

[Apache Jenkins Server]

See 


Changes:

[ssubramanian] ATLAS-1753: Fix for sandbox graph instance for each test

--
[...truncated 327.42 KB...]
[WARNING] Note: Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ 
atlas-repository ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 54 source files to 

[WARNING] 
:
 Some input files use or override a deprecated API.
[WARNING] 
:
 Recompile with -Xlint:deprecation for details.
[WARNING] 
:
 Some input files use unchecked or unsafe operations.
[WARNING] 
:
 Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ atlas-repository 
---
[INFO] Surefire report directory: 

[INFO] Using configured provider org.apache.maven.surefire.testng.TestNGProvider
[WARNING] file.encoding cannot be set as system property, use 
-Dfile.encoding=... instead

---
 T E S T S
---
Running org.apache.atlas.repository.graph.ReverseReferenceUpdateHardDeleteTest
Running org.apache.atlas.service.DefaultMetadataServiceTest
Running org.apache.atlas.repository.graph.GraphHelperMockTest
Running org.apache.atlas.repository.graph.GraphBackedMetadataRepositoryTest
Running org.apache.atlas.repository.graph.Gremlin3QueryOptimizerTest
Running org.apache.atlas.repository.graph.AbstractGremlinQueryOptimizerTest
Running org.apache.atlas.repository.graph.GraphBackedRepositorySoftDeleteTest
Running org.apache.atlas.repository.impexp.ExportServiceTest
Running org.apache.atlas.repository.impexp.ImportServiceReportingTest
Running org.apache.atlas.repository.graph.GraphHelperTest
Running org.apache.atlas.BaseRepositoryTest
Running org.apache.atlas.repository.graph.GraphBackedSearchIndexerTest
Running org.apache.atlas.discovery.DataSetLineageServiceTest
Running org.apache.atlas.service.StoreBackedTypeCacheMetadataServiceTest
Running org.apache.atlas.repository.graph.GraphRepoMapperScaleTest
Running org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStoreTest
Running org.apache.atlas.repository.typestore.GraphBackedTypeStoreTest
Running org.apache.atlas.repository.graph.GraphBackedRepositoryHardDeleteTest
Running 
org.apache.atlas.repository.store.graph.v1.InverseReferenceUpdateHardDeleteV1Test
Running org.apache.atlas.repository.impexp.ZipSourceTest
Running org.apache.atlas.repository.impexp.ZipSinkTest
Running org.apache.atlas.repository.store.graph.AtlasEntityDefStoreV1Test
Running org.apache.atlas.repository.graph.Gremlin2QueryOptimizerTest
Running org.apache.atlas.repository.typestore.StoreBackedTypeCacheTest
Running 
org.apache.atlas.repository.store.graph.v1.InverseReferenceUpdateSoftDeleteV1Test
Running org.apache.atlas.repository.store.graph.v1.SoftDeleteHandlerV1Test
Running org.apache.atlas.repository.graph.TestIntSequence
Running org.apache.atlas.repository.graph.ReverseReferenceUpdateSoftDeleteTest
Running org.apache.atlas.repository.store.graph.v1.AtlasDeleteHandlerV1Test
Running org.apache.atlas.repository.impexp.ImportServiceTest
Running org.apache.atlas.discovery.GraphBackedDiscoveryServiceTest
Running org.apache.atlas.repository.graph.GraphBackedSearchIndexerMockTest
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.759 sec - in 
org.apache.atlas.BaseRepositoryTest
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.92 sec - in 
org.apache.atlas.repository.store.graph.v1.AtlasDeleteHandlerV1Test
Running org.apache.atlas.repository.store.graph.v1.InverseReferenceUpdateV1Test
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.309 sec - in 
org.apache.atlas.repository.impexp.ZipSinkTest
Running org.apache.atlas.repository.store.graph.v1.HardDeleteHandlerV1Test
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.407 sec - in 
org.apache.atlas.repository.graph.GraphBackedSearchIndexerMockTest
Running org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1Test
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time 

[jira] [Updated] (ATLAS-1095) Open connector framework

[Mandy Chessell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mandy Chessell updated ATLAS-1095:
--
Description: 
Atlas provides a common approach to metadata management and governance across 
all systems and data within an organization.  Today Atlas provides access to 
metadata.   A connector provides access to a data source.  As connectors are 
the proxy of all data, they can also be explicit providers of metadata.   

This JIRA proposes an open connector framework to manage connectors that 
provide access to both data and the metadata Atlas provides together through a 
single connector interface.  

This will help data tools to to better the exchange of information between 
platforms. It also offers new opportunities for the consistent enforcement of 
the governance policies and rules (e.g., rules of visibility).  Source 
connector/connection metadata provides the nucleus around which all other 
metadata describing the data builds.  


  was:
Atlas provides a common approach to metadata management and governance across 
all systems and data within an organization.  Today Atlas provides access to 
metadata.   A connector provides access to a data source.  As connectors are 
the proxy of all data, they can also be explicit providers of metadata.   

This JIRA proposes an open connector framework to manage connectors that 
provide access to both data and the metadata Atlas provides together through a 
single connector interface.  

This will help data tools to to better the exchange of information between 
platforms. It also offers new opportunities for the consistent enforcement of 
the governance policies and rules (e.g., rules of visibility).  Source 
connector/connection metadata provides the nucleus around which all other 
metadata describing the data builds.  

Introducing this framework:

The Connector Broker is used to access to (finding and instantiating) a 
connector instance. 

Provides extension API supporting both partial and incremental adoption of the 
framework by existing connector provider runtimes allowing connector 
providers/contributors to adapt what they have vs re-write.

Introducing the APIs for metadata-enabled open connectors:

Provides an API for the normalized access to the metadata describing a 
connector (asset type, connection type).  This is called the Asset OMAS

The key personas using the framework are the tool developers and developers.

As a connector developer, I want to create a new connector (or register an 
existing connector) to plug into the framework so that I can retrieve and 
manage metadata about what the source provides and ensure data used from that 
source has governance policies (e.g., rules of visibility) consistently 
configured and enforced. 

The tool developer uses a connector available through the framework to 
implement an application. Leveraging the APIs of the connector framework 
applications will be able to find relevant sources systems and connect to them. 




> Open connector framework
> 
>
> Key: ATLAS-1095
> URL: https://issues.apache.org/jira/browse/ATLAS-1095
> Project: Atlas
>  Issue Type: New Feature
>Affects Versions: 0.8-incubating
>Reporter: Stephanie Hazlewood
>Assignee: Mandy Chessell
>  Labels: VirtualDataConnector
> Attachments: Open Connector Framework - 9th May 2017.doc
>
>
> Atlas provides a common approach to metadata management and governance across 
> all systems and data within an organization.  Today Atlas provides access to 
> metadata.   A connector provides access to a data source.  As connectors are 
> the proxy of all data, they can also be explicit providers of metadata.   
> This JIRA proposes an open connector framework to manage connectors that 
> provide access to both data and the metadata Atlas provides together through 
> a single connector interface.  
> This will help data tools to to better the exchange of information between 
> platforms. It also offers new opportunities for the consistent enforcement of 
> the governance policies and rules (e.g., rules of visibility).  Source 
> connector/connection metadata provides the nucleus around which all other 
> metadata describing the data builds.  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (ATLAS-1410) V2 Glossary Function

[Mandy Chessell (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025478#comment-16025478
 ] 

Mandy Chessell commented on ATLAS-1410:
---

I have copied these comments into JIRA ATLAS-1768 
(https://issues.apache.org/jira/browse/ATLAS-1768) and added responses there.

> V2 Glossary Function
> 
>
> Key: ATLAS-1410
> URL: https://issues.apache.org/jira/browse/ATLAS-1410
> Project: Atlas
>  Issue Type: Improvement
>Reporter: David Radley
>Assignee: David Radley
>  Labels: VirtualDataConnector
> Attachments: Atlas Glossary V2 proposal v1.0.pdf, Atlas Glossary V2 
> proposal v1.1.pdf, Atlas Glossary V2 proposal v1.2.pdf, Atlas Glossary V2 
> proposal v1.3.pdf, Atlas Glossary V2 proposal v1.4.pdf
>
>
> The BaseResourceDefinition uses the AttributeDefintion class from typesystem. 
> There are newer more funcitonal versions of this capability in the atlas-intg 
> project. This Jira is changing over the glossary implementation to the newer 
> entity / type classes.  
> Instread of the instanceProperties and collectionProperties in the 
> BaseResourceDefintions we should use something in this sort of style :  
> "
>  AtlasEntityDef deptTypeDef =
> AtlasTypeUtil.createClassTypeDef(DEPARTMENT_TYPE, 
> "Department"+_description, ImmutableSet.of(),
> AtlasTypeUtil.createRequiredAttrDef("name", "string"),
> new AtlasAttributeDef("employees", 
> String.format("array<%s>", "Person"), true,
> AtlasAttributeDef.Cardinality.SINGLE, 0, 1, 
> false, false,
> 
> Collections.emptyList()));
> AtlasEntityDef personTypeDef = 
> AtlasTypeUtil.createClassTypeDef("Person", "Person"+_description, 
> ImmutableSet.of(),
> AtlasTypeUtil.createRequiredAttrDef("name", "string"),
> AtlasTypeUtil.createOptionalAttrDef("address", "Address"),
> AtlasTypeUtil.createOptionalAttrDef("birthday", "date"),
> AtlasTypeUtil.createOptionalAttrDef("hasPets", "boolean"),
> AtlasTypeUtil.createOptionalAttrDef("numberOfCars", "byte"),
> AtlasTypeUtil.createOptionalAttrDef("houseNumber", "short"),
> AtlasTypeUtil.createOptionalAttrDef("carMileage", "int"),
> AtlasTypeUtil.createOptionalAttrDef("age", "float"),
> "
> For the parent child relationships with glossary categories and terms we 
> should be able to have the type system manage edge deletion. As part of this, 
> we will need to investigate whether we could get rid of the disconnect and 
> connect methods added in ATLAS-1186 
>  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (ATLAS-1691) OMAS Interfaces for Atlas

[Mandy Chessell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mandy Chessell reassigned ATLAS-1691:
-

Assignee: Mandy Chessell  (was: Nigel Jones)

> OMAS Interfaces for Atlas
> -
>
> Key: ATLAS-1691
> URL: https://issues.apache.org/jira/browse/ATLAS-1691
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nigel Jones
>Assignee: Mandy Chessell
>  Labels: VirtualDataConnector
>
> The Open Metadata Access Services (OMAS) provide consumer specific interfaces 
> to the metadata repository.   The hope is to have a community of vendor tools 
> developing around each OMAS interface.   The OMAS interfaces are independent 
> from one another to allow each community to evolve their own API at their 
> pace.
> Related Jira will cover the list of OMAS interfaces that are oriented to 
> particular tools & usage.
> Each OMAS JIRA needs to cover:
> * The Java and REST API for the OMAS call-response API
> * The appropriate OMAS Event Publisher and Listener



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (ATLAS-1689) Virtual Data Connector Project

[Mandy Chessell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mandy Chessell reassigned ATLAS-1689:
-

Assignee: Mandy Chessell

> Virtual Data Connector Project 
> ---
>
> Key: ATLAS-1689
> URL: https://issues.apache.org/jira/browse/ATLAS-1689
> Project: Atlas
>  Issue Type: Improvement
>Reporter: David Radley
>Assignee: Mandy Chessell
>  Labels: VirtualDataConnector
>
> Virtual Data Connector - this is a project to create a meta-data driven data 
> lake in which Apache Atlas is the meta-data store.
> Create an open, metadata-driven virtual connector to data repositories – a 
> Data Virtualization Platform that will satisfy the following goals:
> Give users one place to access each and every Data Lake repository 
> (search and discover)
> Give users one search interface for both data and metadata inside the 
> Data Lake (search and discover)
> To provide one place to look for and provision data for all kinds of end 
> users
> To provide one place to build and maintain data security for all data 
> repositories inside the Data Lake



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 59533: Test Sandbox fix

[Sarath Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59533/#review176138
---


Ship it!




Ship It!

- Sarath Subramanian


On May 25, 2017, 2:07 p.m., Apoorv Naik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59533/
> ---
> 
> (Updated May 25, 2017, 2:07 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Sarath Subramanian, and Suma 
> Shivaprasad.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Due to race condition, a couple of test still end up using the same directory 
> for DB/index storage. 
> 
> This change checks for the existence of the folder and generates the name 
> again in case of any discrepancy.
> 
> 
> Diffs
> -
> 
>   graphdb/common/src/test/java/org/apache/atlas/graph/GraphSandboxUtil.java 
> 44ad4fc0 
>   repository/src/test/java/org/apache/atlas/DBSandboxer.java cc8e0e27 
> 
> 
> Diff: https://reviews.apache.org/r/59533/diff/2/
> 
> 
> Testing
> ---
> 
> mvn clean package executes successfully
> mvn clean install executes successfully with one consitent failure related to 
> notification processing.
> 
> 
> Thanks,
> 
> Apoorv Naik
> 
>

Re: Review Request 56959: Import API: Added Support for Transforming Entities During Import

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/
---

(Updated May 25, 2017, 9:32 p.m.)


Review request for atlas and Madhan Neethiraj.


Changes
---

Updates:
- Refactored _AtlasImportRequest_ to make it simpler.
- Moved class to _repository_ from _intg_.
- Addressed review comments.


Bugs: ATLAS-1825
https://issues.apache.org/jira/browse/ATLAS-1825


Repository: atlas


Description (updated)
---

Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) 
for scenarios and solution approach.

Impact
==
* New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This 
holds a map of entity-type to attribute and the tranforms to be applied.
* New class _AtlasImportTransformBuiler_ reads the requests specified in the 
query string of the import API and creates a _AtlasImportTransfor_ structure.
* _AtlasImportTransformBuiler_ has new classes that support transformations. 
They are:
* _replace_ Implements find and replace on attribute values.
* _lowercase_ Converts attribute value to lower case. 
* Additional classes that unit test these classes have been added.

Implementation Approach
===
* _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the 
transform every entity that is fetched from the source.
* _Import API_ now accepts multi-part input. This allows for elegantly 
specifying import parameters as well as the binary input from the file.

CURL


*Setup*
Create file with these contents call it _importTransform.json_:
```javascript
{ "options": {
"transforms": {
"hive_db": {
"qualifiedName": [
"replace:@cl1:@cl2"
]
},
"hive_table": {
"qualifiedName": [
"lowercase",
"replace:@cl1:@cl2"
]
}
}
}
}
```

```
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H 
"Cache-Control: no-cache" -F request=@../docs/importTransform.json -F 
data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import;
```

Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is not 
required anymore.

Notice the output from the API, it has the _AtlasRequest_ that was passed as 
input.

Documentation
=
Need to update TWIKI documents.


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
7530196d 
  pom.xml a93b8ad9 
  
repository/src/main/java/org/apache/atlas/repository/impexp/EntityAttributeTransformer.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 
9ec15e00 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransformers.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransforms.java
 PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 
87c9f0eb 
  
repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/impexp/EntityAttributeTransformerTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
 08e9ee81 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformersTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/impexp/TransformEvaluatorTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ZipSourceTest.java 
8b042d7a 
  
repository/src/test/java/org/apache/atlas/services/EntityDiscoveryServiceTest.java
 d4900838 
  webapp/pom.xml 4132912d 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
01f9a860 


Diff: https://reviews.apache.org/r/56959/diff/7/

Changes: https://reviews.apache.org/r/56959/diff/6-7/


Testing
---

**Unit Test**
* Added tests for *AtlasImportTransformBuilder* and contained classes.
* Updated *ZipSourceTest*
* Added tests to *ImportServiceTest* to exercise new option.

**Functional Test**
* Small size data export & import.
* Medium size data export & import.

**Performance Tests**
Not exhaustive. Simple stop watch tests to compare against baseline.


Thanks,

Ashutosh Mestry

[jira] [Created] (ATLAS-1830) IGC Event Mapper

[Mandy Chessell (JIRA)]

Mandy Chessell created ATLAS-1830:
-

 Summary: IGC Event Mapper
 Key: ATLAS-1830
 URL: https://issues.apache.org/jira/browse/ATLAS-1830
 Project: Atlas
  Issue Type: New Feature
  Components: atlas-intg
Affects Versions: 0.9-incubating
Reporter: Mandy Chessell


This component listens for events from IBM's Information Governance Catalog and 
converts them into OMRS Event Messages and posts them onto the OMRS Topic in 
Kafka.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (ATLAS-1689) Virtual Data Connector Project

[Mandy Chessell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mandy Chessell updated ATLAS-1689:
--
Summary: Virtual Data Connector Project   (was: Virtual Data connector )

> Virtual Data Connector Project 
> ---
>
> Key: ATLAS-1689
> URL: https://issues.apache.org/jira/browse/ATLAS-1689
> Project: Atlas
>  Issue Type: Improvement
>Reporter: David Radley
>  Labels: VirtualDataConnector
>
> Virtual Data Connector - this is a project to create a meta-data driven data 
> lake in which Apache Atlas is the meta-data store.
> Create an open, metadata-driven virtual connector to data repositories – a 
> Data Virtualization Platform that will satisfy the following goals:
> Give users one place to access each and every Data Lake repository 
> (search and discover)
> Give users one search interface for both data and metadata inside the 
> Data Lake (search and discover)
> To provide one place to look for and provision data for all kinds of end 
> users
> To provide one place to build and maintain data security for all data 
> repositories inside the Data Lake



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (ATLAS-1829) OMRS Message Payloads with Event Listener and Publisher

[Mandy Chessell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mandy Chessell updated ATLAS-1829:
--
Labels: VirtualDataConnector  (was: )

> OMRS Message Payloads with Event Listener and Publisher
> ---
>
> Key: ATLAS-1829
> URL: https://issues.apache.org/jira/browse/ATLAS-1829
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: Mandy Chessell
>  Labels: VirtualDataConnector
>
> This JIRA covers the event support for OMRS.  There is an OMRS Event 
> Publisher that publishes events relating to changing metadata in the Atlas 
> repository.  These events follow the OMRS message payloads that are derived 
> from the Atlas type model.
> OMRS Messages are published to the OMRS Topic in Kafka.
> There is also an OMRS Event Listener that takes events from the OMRS Topic 
> and pushes them to an OMRS Connector if they are from a different repository 
> instance.
> This JIRA complements ATLAS-1773 
> (https://issues.apache.org/jira/browse/ATLAS-1773) OMRS REST Connector



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (ATLAS-1829) OMRS Message Payloads with Event Listener and Publisher

[Mandy Chessell (JIRA)]

Mandy Chessell created ATLAS-1829:
-

 Summary: OMRS Message Payloads with Event Listener and Publisher
 Key: ATLAS-1829
 URL: https://issues.apache.org/jira/browse/ATLAS-1829
 Project: Atlas
  Issue Type: New Feature
  Components:  atlas-core
Affects Versions: 0.9-incubating
Reporter: Mandy Chessell
Assignee: Mandy Chessell


This JIRA covers the event support for OMRS.  There is an OMRS Event Publisher 
that publishes events relating to changing metadata in the Atlas repository.  
These events follow the OMRS message payloads that are derived from the Atlas 
type model.

OMRS Messages are published to the OMRS Topic in Kafka.

There is also an OMRS Event Listener that takes events from the OMRS Topic and 
pushes them to an OMRS Connector if they are from a different repository 
instance.

This JIRA complements ATLAS-1773 
(https://issues.apache.org/jira/browse/ATLAS-1773) OMRS REST Connector



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 59533: Test Sandbox fix

[Apoorv Naik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59533/
---

(Updated May 25, 2017, 9:07 p.m.)


Review request for atlas, Madhan Neethiraj, Sarath Subramanian, and Suma 
Shivaprasad.


Changes
---

Added Sandbox naming capability, defaults to UUID.


Repository: atlas


Description
---

Due to race condition, a couple of test still end up using the same directory 
for DB/index storage. 

This change checks for the existence of the folder and generates the name again 
in case of any discrepancy.


Diffs (updated)
-

  graphdb/common/src/test/java/org/apache/atlas/graph/GraphSandboxUtil.java 
44ad4fc0 
  repository/src/test/java/org/apache/atlas/DBSandboxer.java cc8e0e27 


Diff: https://reviews.apache.org/r/59533/diff/2/

Changes: https://reviews.apache.org/r/59533/diff/1-2/


Testing
---

mvn clean package executes successfully
mvn clean install executes successfully with one consitent failure related to 
notification processing.


Thanks,

Apoorv Naik

[jira] [Updated] (ATLAS-1691) OMAS Interfaces for Atlas

[Mandy Chessell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mandy Chessell updated ATLAS-1691:
--
Description: 
The Open Metadata Access Services (OMAS) provide consumer specific interfaces 
to the metadata repository.   The hope is to have a community of vendor tools 
developing around each OMAS interface.   The OMAS interfaces are independent 
from one another to allow each community to evolve their own API at their pace.

Related Jira will cover the list of OMAS interfaces that are oriented to 
particular tools & usage.

Each OMAS JIRA needs to cover:
* The Java and REST API for the OMAS call-response API
* The appropriate OMAS Event Publisher and Listener


  was:
The Open Metadata Access Services (OMAS) provide consumer specific interfaces 
to the metadata repository.   The hope is to have a community of vendor tools 
developing around each OMAS interface.   The OMAS interfaces are independent 
from one another to allow each community to evolve their own API at their pace.

Related Jira will cover the list of OMAS interfaces that are oriented to 
particular tools & usage.


> OMAS Interfaces for Atlas
> -
>
> Key: ATLAS-1691
> URL: https://issues.apache.org/jira/browse/ATLAS-1691
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nigel Jones
>Assignee: Nigel Jones
>  Labels: VirtualDataConnector
>
> The Open Metadata Access Services (OMAS) provide consumer specific interfaces 
> to the metadata repository.   The hope is to have a community of vendor tools 
> developing around each OMAS interface.   The OMAS interfaces are independent 
> from one another to allow each community to evolve their own API at their 
> pace.
> Related Jira will cover the list of OMAS interfaces that are oriented to 
> particular tools & usage.
> Each OMAS JIRA needs to cover:
> * The Java and REST API for the OMAS call-response API
> * The appropriate OMAS Event Publisher and Listener



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: apache-atlas-nightly #754

[Apache Jenkins Server]

See 

--
[...truncated 327.83 KB...]
[WARNING] Note: Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ 
atlas-repository ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 54 source files to 

[WARNING] 
:
 Some input files use or override a deprecated API.
[WARNING] 
:
 Recompile with -Xlint:deprecation for details.
[WARNING] 
:
 Some input files use unchecked or unsafe operations.
[WARNING] 
:
 Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ atlas-repository 
---
[INFO] Surefire report directory: 

[INFO] Using configured provider org.apache.maven.surefire.testng.TestNGProvider
[WARNING] file.encoding cannot be set as system property, use 
-Dfile.encoding=... instead

---
 T E S T S
---
Running 
org.apache.atlas.repository.store.graph.v1.InverseReferenceUpdateSoftDeleteV1Test
Running org.apache.atlas.repository.store.graph.AtlasEntityDefStoreV1Test
Running org.apache.atlas.repository.audit.HBaseBasedAuditRepositoryHATest
Running org.apache.atlas.repository.audit.InMemoryAuditRepositoryTest
Running org.apache.atlas.repository.store.graph.v1.AtlasDeleteHandlerV1Test
Running org.apache.atlas.repository.audit.HBaseBasedAuditRepositoryTest
Running org.apache.atlas.repository.BaseTest
Running 
org.apache.atlas.repository.store.graph.v1.InverseReferenceUpdateHardDeleteV1Test
Running org.apache.atlas.repository.graph.AbstractGremlinQueryOptimizerTest
Running org.apache.atlas.repository.graph.GraphHelperMockTest
Running org.apache.atlas.repository.impexp.ImportServiceTest
Running org.apache.atlas.util.CompiledQueryCacheKeyTest
Running org.apache.atlas.repository.store.graph.v1.SoftDeleteHandlerV1Test
Running org.apache.atlas.repository.impexp.ImportServiceReportingTest
Running org.apache.atlas.repository.graph.ReverseReferenceUpdateSoftDeleteTest
Running org.apache.atlas.repository.impexp.ZipSinkTest
Running org.apache.atlas.repository.typestore.GraphBackedTypeStoreTest
Running org.apache.atlas.repository.store.graph.v1.HardDeleteHandlerV1Test
Running org.apache.atlas.repository.graph.ReverseReferenceUpdateHardDeleteTest
Running org.apache.atlas.repository.store.graph.v1.InverseReferenceUpdateV1Test
Running org.apache.atlas.repository.typestore.StoreBackedTypeCacheTest
Running org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStoreTest
Running org.apache.atlas.repository.graph.Gremlin2QueryOptimizerTest
Running org.apache.atlas.repository.impexp.ZipSourceTest
Running org.apache.atlas.repository.impexp.ExportServiceTest
Running org.apache.atlas.repository.graph.GraphBackedSearchIndexerTest
Running org.apache.atlas.repository.graph.GraphBackedRepositorySoftDeleteTest
Running org.apache.atlas.repository.graph.GraphBackedRepositoryHardDeleteTest
Running org.apache.atlas.repository.graph.TestIntSequence
Running org.apache.atlas.repository.graph.GraphBackedSearchIndexerMockTest
Running org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1Test
Running org.apache.atlas.repository.graph.GraphRepoMapperScaleTest
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.28 sec - in 
org.apache.atlas.repository.store.graph.v1.AtlasDeleteHandlerV1Test
Running org.apache.atlas.repository.graph.GraphHelperTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.34 sec - in 
org.apache.atlas.repository.audit.InMemoryAuditRepositoryTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.832 sec - in 
org.apache.atlas.util.CompiledQueryCacheKeyTest
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.574 sec - in 
org.apache.atlas.repository.store.graph.v1.InverseReferenceUpdateV1Test
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.62 sec - in 
org.apache.atlas.repository.BaseTest
Running org.apache.atlas.repository.graph.Gremlin3QueryOptimizerTest
Running org.apache.atlas.BaseRepositoryTest
Tests run: 0, Failures: 0, 

[jira] [Created] (ATLAS-1828) Catalog OMAS

[Mandy Chessell (JIRA)]

Mandy Chessell created ATLAS-1828:
-

 Summary: Catalog OMAS
 Key: ATLAS-1828
 URL: https://issues.apache.org/jira/browse/ATLAS-1828
 Project: Atlas
  Issue Type: New Feature
  Components:  atlas-core
Affects Versions: 0.9-incubating
Reporter: Mandy Chessell
Assignee: Mandy Chessell


This JIRA delivers the External API for catalog search applications.  This API 
is used by ATLAS-1765 (https://issues.apache.org/jira/browse/ATLAS-1765) that 
provides the Self-Service Catalog Search and Data Preview UI.

This JIRA is dependent on ATLAS-1773 
(https://issues.apache.org/jira/browse/ATLAS-1773) that implements the OMRS 
REST Connector



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (ATLAS-1768) Create common types for Open Metadata

[Mandy Chessell (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025290#comment-16025290
 ] 

Mandy Chessell commented on ATLAS-1768:
---

Added "confidence" to SemanticAssignment as per [~cmgrote] comments


> Create common types for Open Metadata
> -
>
> Key: ATLAS-1768
> URL: https://issues.apache.org/jira/browse/ATLAS-1768
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: Mandy Chessell
>  Labels: VirtualDataConnector
>
> This JIRA describes a proposal for standard types for open metadata entities 
> and relationships.  For example, glossaries, database definitions, rules, 
> policies, ...
> The value of having standard definitions for metadata is to enable type safe 
> APIs and business level UIs plus be able to exchange metadata between 
> different instances of metadata repositories.
> The implementation of these common types is divided into 8 areas:
> * Area 0 - for extensions to Apache Atlas's base model
> * Area 1 - for definitions of the data-related assets we are governing and 
> using
> * Area 2 - for a glossary of meanings and semantic relationships
> * Area 3 - for information about asset use, crowd-sourced definitions and 
> collaboration around the data-related assets
> * Area 4 - for governance such as policies, rules and classifications
> * Area 5 - for reference models and reference data
> * Area 6 - for metadata discovery processes (see 
> https://issues.apache.org/jira/browse/ATLAS-1748)
> * Area 7 - for lineage
> Adaptation and flexibility are key in metadata environments so these common 
> definitions must be extensible - and we still need to support the ad hoc 
> definition of new types in Atlas.
> Apache Atlas supports meta-types that are used in the definition of new 
> types.  These are currently enumeration, struct, classification and entity.  
> JIRA https://issues.apache.org/jira/browse/ATLAS-1690 adds relationships to 
> this list.  The open metadata models make use of all of these meta-types.  
> These are represented by sterotypes on the classes of the open metadata 
> definitions.
> The Atlas wiki has the models as a set of linked pages which are probably the 
> easiest way to view the models.
> Start here: 
> https://cwiki.apache.org/confluence/display/ATLAS/Building+out+the+Apache+Atlas+Typesystem



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (ATLAS-1773) Create the OMRS Connector for Atlas

[Mandy Chessell (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025286#comment-16025286
 ] 

Mandy Chessell commented on ATLAS-1773:
---

JIRA ATLAS-1700 (https://issues.apache.org/jira/browse/ATLAS-1700) describes 
the types of constraints that are needed to support the glossary model.  The 
OMRS Connect needs to support the specification/implementation of constraint 
classes that are called by the generated OMAS Connector.

> Create the OMRS Connector for Atlas
> ---
>
> Key: ATLAS-1773
> URL: https://issues.apache.org/jira/browse/ATLAS-1773
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: Mandy Chessell
>
> This JIRA provides the definition of the OMRS Connector API and an 
> implementation of this API for a local Apache Atlas metadata repository and 
> for the OMRS REST API.
> The OMRS Connector has 3 API groups
> * The types API - this is the metadata API for a metadata repository
> * The entity and relationships APIs that provide the type-agnostic interfaces 
> that can access any type - even those added dynamically
> * The fine-grained type-safe APIs that are generated from the addons models 
> in the build.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (ATLAS-1773) Create the OMRS Connector for Atlas

[Mandy Chessell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mandy Chessell updated ATLAS-1773:
--
Description: 
This JIRA provides the definition of the OMRS Connector API and an 
implementation of this API for a local Apache Atlas metadata repository and for 
the OMRS REST API.

The OMRS Connector has 3 API groups
* The types API - this is the metadata API for a metadata repository
* The entity and relationships APIs that provide the type-agnostic interfaces 
that can access any type - even those added dynamically
* The fine-grained type-safe APIs that are generated from the addons models in 
the build.


  was:This JIRA provides the definition of the OMRS Connector API and an 
implementation of this API for a local Apache Atlas metadata repository and for 
the OMRS REST API.


> Create the OMRS Connector for Atlas
> ---
>
> Key: ATLAS-1773
> URL: https://issues.apache.org/jira/browse/ATLAS-1773
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: Mandy Chessell
>
> This JIRA provides the definition of the OMRS Connector API and an 
> implementation of this API for a local Apache Atlas metadata repository and 
> for the OMRS REST API.
> The OMRS Connector has 3 API groups
> * The types API - this is the metadata API for a metadata repository
> * The entity and relationships APIs that provide the type-agnostic interfaces 
> that can access any type - even those added dynamically
> * The fine-grained type-safe APIs that are generated from the addons models 
> in the build.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: apache-atlas-nightly #753

[Apache Jenkins Server]

See 


Changes:

[apoorvnaik] Removing empty file causing rat check failure

--
[...truncated 327.75 KB...]
[WARNING] Note: Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ 
atlas-repository ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 54 source files to 

[WARNING] 
:
 Some input files use or override a deprecated API.
[WARNING] 
:
 Recompile with -Xlint:deprecation for details.
[WARNING] 
:
 Some input files use unchecked or unsafe operations.
[WARNING] 
:
 Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ atlas-repository 
---
[INFO] Surefire report directory: 

[INFO] Using configured provider org.apache.maven.surefire.testng.TestNGProvider
[WARNING] file.encoding cannot be set as system property, use 
-Dfile.encoding=... instead

---
 T E S T S
---
Running org.apache.atlas.repository.store.graph.v1.HardDeleteHandlerV1Test
Running org.apache.atlas.repository.graph.GraphBackedSearchIndexerTest
Running org.apache.atlas.repository.store.graph.AtlasEntityDefStoreV1Test
Running org.apache.atlas.repository.impexp.ZipSourceTest
Running 
org.apache.atlas.repository.store.graph.v1.InverseReferenceUpdateHardDeleteV1Test
Running org.apache.atlas.repository.typestore.StoreBackedTypeCacheTest
Running org.apache.atlas.repository.graph.GraphRepoMapperScaleTest
Running org.apache.atlas.repository.graph.GraphBackedMetadataRepositoryTest
Running org.apache.atlas.repository.impexp.ImportServiceTest
Running org.apache.atlas.repository.audit.InMemoryAuditRepositoryTest
Running org.apache.atlas.repository.graph.GraphHelperMockTest
Running 
org.apache.atlas.repository.store.graph.v1.InverseReferenceUpdateSoftDeleteV1Test
Running org.apache.atlas.repository.store.graph.v1.SoftDeleteHandlerV1Test
Running org.apache.atlas.repository.impexp.ImportServiceReportingTest
Running org.apache.atlas.repository.store.graph.v1.InverseReferenceUpdateV1Test
Running org.apache.atlas.repository.graph.AbstractGremlinQueryOptimizerTest
Running org.apache.atlas.repository.graph.GraphBackedRepositoryHardDeleteTest
Running org.apache.atlas.repository.BaseTest
Running org.apache.atlas.repository.impexp.ZipSinkTest
Running org.apache.atlas.repository.store.graph.v1.AtlasDeleteHandlerV1Test
Running org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStoreTest
Running org.apache.atlas.repository.typestore.GraphBackedTypeStoreTest
Running org.apache.atlas.repository.graph.ReverseReferenceUpdateHardDeleteTest
Running org.apache.atlas.repository.graph.Gremlin2QueryOptimizerTest
Running org.apache.atlas.repository.graph.GraphHelperTest
Running org.apache.atlas.repository.graph.GraphBackedRepositorySoftDeleteTest
Running org.apache.atlas.repository.graph.TestIntSequence
Running org.apache.atlas.repository.impexp.ExportServiceTest
Running org.apache.atlas.repository.audit.HBaseBasedAuditRepositoryTest
Running org.apache.atlas.repository.graph.Gremlin3QueryOptimizerTest
Running org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1Test
Running org.apache.atlas.repository.audit.HBaseBasedAuditRepositoryHATest
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.613 sec - in 
org.apache.atlas.repository.graph.TestIntSequence
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.049 sec - in 
org.apache.atlas.repository.store.graph.v1.InverseReferenceUpdateV1Test
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.086 sec - in 
org.apache.atlas.repository.BaseTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.573 sec - in 
org.apache.atlas.repository.impexp.ZipSinkTest
Running org.apache.atlas.repository.graph.ReverseReferenceUpdateSoftDeleteTest
Running org.apache.atlas.service.DefaultMetadataServiceTest
Running org.apache.atlas.repository.graph.GraphBackedSearchIndexerMockTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 

[jira] [Updated] (ATLAS-1410) V2 Glossary Function

[Mandy Chessell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mandy Chessell updated ATLAS-1410:
--
Summary: V2 Glossary Function  (was: V2 Glossary API)

> V2 Glossary Function
> 
>
> Key: ATLAS-1410
> URL: https://issues.apache.org/jira/browse/ATLAS-1410
> Project: Atlas
>  Issue Type: Improvement
>Reporter: David Radley
>Assignee: David Radley
>  Labels: VirtualDataConnector
> Attachments: Atlas Glossary V2 proposal v1.0.pdf, Atlas Glossary V2 
> proposal v1.1.pdf, Atlas Glossary V2 proposal v1.2.pdf, Atlas Glossary V2 
> proposal v1.3.pdf, Atlas Glossary V2 proposal v1.4.pdf
>
>
> The BaseResourceDefinition uses the AttributeDefintion class from typesystem. 
> There are newer more funcitonal versions of this capability in the atlas-intg 
> project. This Jira is changing over the glossary implementation to the newer 
> entity / type classes.  
> Instread of the instanceProperties and collectionProperties in the 
> BaseResourceDefintions we should use something in this sort of style :  
> "
>  AtlasEntityDef deptTypeDef =
> AtlasTypeUtil.createClassTypeDef(DEPARTMENT_TYPE, 
> "Department"+_description, ImmutableSet.of(),
> AtlasTypeUtil.createRequiredAttrDef("name", "string"),
> new AtlasAttributeDef("employees", 
> String.format("array<%s>", "Person"), true,
> AtlasAttributeDef.Cardinality.SINGLE, 0, 1, 
> false, false,
> 
> Collections.emptyList()));
> AtlasEntityDef personTypeDef = 
> AtlasTypeUtil.createClassTypeDef("Person", "Person"+_description, 
> ImmutableSet.of(),
> AtlasTypeUtil.createRequiredAttrDef("name", "string"),
> AtlasTypeUtil.createOptionalAttrDef("address", "Address"),
> AtlasTypeUtil.createOptionalAttrDef("birthday", "date"),
> AtlasTypeUtil.createOptionalAttrDef("hasPets", "boolean"),
> AtlasTypeUtil.createOptionalAttrDef("numberOfCars", "byte"),
> AtlasTypeUtil.createOptionalAttrDef("houseNumber", "short"),
> AtlasTypeUtil.createOptionalAttrDef("carMileage", "int"),
> AtlasTypeUtil.createOptionalAttrDef("age", "float"),
> "
> For the parent child relationships with glossary categories and terms we 
> should be able to have the type system manage edge deletion. As part of this, 
> we will need to investigate whether we could get rid of the disconnect and 
> connect methods added in ATLAS-1186 
>  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (ATLAS-1698) Create Glossary OMAS API

[Mandy Chessell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mandy Chessell updated ATLAS-1698:
--
Summary: Create Glossary OMAS API  (was: Create OMAS Glossary project, new 
types and Java API  )

> Create Glossary OMAS API
> 
>
> Key: ATLAS-1698
> URL: https://issues.apache.org/jira/browse/ATLAS-1698
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: David Radley
>  Labels: VirtualDataConnector
>
> The Glossary OMAS provides a specialized API for glossary applications to 
> retrieve and store their glossary metadata and link assets of different types 
> to these glossary entries.
> The Glossary OMAS makes heavy use of the Area 2 open metadata model.  See 
> https://cwiki.apache.org/confluence/display/ATLAS/Area+2+-+Glossary



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (ATLAS-1698) Create OMAS Glossary project, new types and Java API

[Mandy Chessell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mandy Chessell updated ATLAS-1698:
--
Description: 
The Glossary OMAS provides a specialized API for glossary applications to 
retrieve and store their glossary metadata and link assets of different types 
to these glossary entries.

The Glossary OMAS makes heavy use of the Area 2 open metadata model.  See 
https://cwiki.apache.org/confluence/display/ATLAS/Area+2+-+Glossary


> Create OMAS Glossary project, new types and Java API  
> --
>
> Key: ATLAS-1698
> URL: https://issues.apache.org/jira/browse/ATLAS-1698
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: David Radley
>  Labels: VirtualDataConnector
>
> The Glossary OMAS provides a specialized API for glossary applications to 
> retrieve and store their glossary metadata and link assets of different types 
> to these glossary entries.
> The Glossary OMAS makes heavy use of the Area 2 open metadata model.  See 
> https://cwiki.apache.org/confluence/display/ATLAS/Area+2+-+Glossary



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (ATLAS-520) Temporal / Versioning support for Atlas metadata

[Mandy Chessell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mandy Chessell updated ATLAS-520:
-
Summary: Temporal / Versioning support for Atlas metadata  (was: Temporal / 
Versioning support for types, traits, entites )

> Temporal / Versioning support for Atlas metadata
> 
>
> Key: ATLAS-520
> URL: https://issues.apache.org/jira/browse/ATLAS-520
> Project: Atlas
>  Issue Type: Wish
>Affects Versions: trunk
>Reporter: Nigel Jones
>  Labels: VirtualDataConnector
>
> With Atlas storing metadata & lineage data , from a governance perspective we 
> would like to ask questons such as
> - what different data sources were used to produce this report last year 
> compared to this year?
>  - how was this salary column in the employee table classified on 6 Apr 2015?
> These queries are needed to support audit, and to aid in process improvement 
> + root cause analysis
> To do so we need temporal support in atlas. A query should be able to 
> indicate the point in time that it applies to. This means keeping track of 
> times, versions within atlas. Potentially this means not just the nodes in 
> the graph, but the edges too, however this can mean an ever increasing graph 
> which can get out of control.
> opening up this JIRA to enable an evaluation/ discussion of what we might be 
> able to do



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (ATLAS-520) Temporal / Versioning support for types, traits, entites ....

[Mandy Chessell (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025238#comment-16025238
 ] 

Mandy Chessell commented on ATLAS-520:
--

The proposed data model for versioned metadata objects is show on wiki page 
https://cwiki.apache.org/confluence/display/ATLAS/Area+0+-+Extensions+to+the+base+model


> Temporal / Versioning support for types, traits, entites 
> -
>
> Key: ATLAS-520
> URL: https://issues.apache.org/jira/browse/ATLAS-520
> Project: Atlas
>  Issue Type: Wish
>Affects Versions: trunk
>Reporter: Nigel Jones
>  Labels: VirtualDataConnector
>
> With Atlas storing metadata & lineage data , from a governance perspective we 
> would like to ask questons such as
> - what different data sources were used to produce this report last year 
> compared to this year?
>  - how was this salary column in the employee table classified on 6 Apr 2015?
> These queries are needed to support audit, and to aid in process improvement 
> + root cause analysis
> To do so we need temporal support in atlas. A query should be able to 
> indicate the point in time that it applies to. This means keeping track of 
> times, versions within atlas. Potentially this means not just the nodes in 
> the graph, but the edges too, however this can mean an ever increasing graph 
> which can get out of control.
> opening up this JIRA to enable an evaluation/ discussion of what we might be 
> able to do



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (ATLAS-519) Atlas operational metrics

[Mandy Chessell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mandy Chessell updated ATLAS-519:
-
Summary: Atlas operational metrics  (was: metrics)

> Atlas operational metrics
> -
>
> Key: ATLAS-519
> URL: https://issues.apache.org/jira/browse/ATLAS-519
> Project: Atlas
>  Issue Type: Wish
>Affects Versions: trunk
>Reporter: Nigel Jones
>  Labels: VirtualDataConnector
>
> Metadata is crucial in an enterprise. Atlas can increasingly store a wide 
> variety of metadata for the organization, and this may well be within a cloud 
> type environment (be it on-prem or public cloud).
> Usage/performance metrics are needed in atlas to
> * Better understand how atlas is being used, to identify areas for improvement
> * understand performance characteristics and capacity planning
> * allow in future for usage oriented billing & QOS.
> * aid in debugging
> Opening this as a discussion focal point to think about what infrastructure 
> may be needed, and the kind of metrics that might make sense.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (ATLAS-1410) V2 Glossary API

[Mandy Chessell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mandy Chessell updated ATLAS-1410:
--
Labels: VirtualDataConnector  (was: )

> V2 Glossary API
> ---
>
> Key: ATLAS-1410
> URL: https://issues.apache.org/jira/browse/ATLAS-1410
> Project: Atlas
>  Issue Type: Improvement
>Reporter: David Radley
>Assignee: David Radley
>  Labels: VirtualDataConnector
> Attachments: Atlas Glossary V2 proposal v1.0.pdf, Atlas Glossary V2 
> proposal v1.1.pdf, Atlas Glossary V2 proposal v1.2.pdf, Atlas Glossary V2 
> proposal v1.3.pdf, Atlas Glossary V2 proposal v1.4.pdf
>
>
> The BaseResourceDefinition uses the AttributeDefintion class from typesystem. 
> There are newer more funcitonal versions of this capability in the atlas-intg 
> project. This Jira is changing over the glossary implementation to the newer 
> entity / type classes.  
> Instread of the instanceProperties and collectionProperties in the 
> BaseResourceDefintions we should use something in this sort of style :  
> "
>  AtlasEntityDef deptTypeDef =
> AtlasTypeUtil.createClassTypeDef(DEPARTMENT_TYPE, 
> "Department"+_description, ImmutableSet.of(),
> AtlasTypeUtil.createRequiredAttrDef("name", "string"),
> new AtlasAttributeDef("employees", 
> String.format("array<%s>", "Person"), true,
> AtlasAttributeDef.Cardinality.SINGLE, 0, 1, 
> false, false,
> 
> Collections.emptyList()));
> AtlasEntityDef personTypeDef = 
> AtlasTypeUtil.createClassTypeDef("Person", "Person"+_description, 
> ImmutableSet.of(),
> AtlasTypeUtil.createRequiredAttrDef("name", "string"),
> AtlasTypeUtil.createOptionalAttrDef("address", "Address"),
> AtlasTypeUtil.createOptionalAttrDef("birthday", "date"),
> AtlasTypeUtil.createOptionalAttrDef("hasPets", "boolean"),
> AtlasTypeUtil.createOptionalAttrDef("numberOfCars", "byte"),
> AtlasTypeUtil.createOptionalAttrDef("houseNumber", "short"),
> AtlasTypeUtil.createOptionalAttrDef("carMileage", "int"),
> AtlasTypeUtil.createOptionalAttrDef("age", "float"),
> "
> For the parent child relationships with glossary categories and terms we 
> should be able to have the type system manage edge deletion. As part of this, 
> we will need to investigate whether we could get rid of the disconnect and 
> connect methods added in ATLAS-1186 
>  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: apache-atlas-nightly #752

[Apache Jenkins Server]

See 

--
[...truncated 273.72 KB...]
[WARNING] Note: Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ 
atlas-repository ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 54 source files to 

[WARNING] 
:
 Some input files use or override a deprecated API.
[WARNING] 
:
 Recompile with -Xlint:deprecation for details.
[WARNING] 
:
 Some input files use unchecked or unsafe operations.
[WARNING] 
:
 Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ atlas-repository 
---
[INFO] Surefire report directory: 

[INFO] Using configured provider org.apache.maven.surefire.testng.TestNGProvider
[WARNING] file.encoding cannot be set as system property, use 
-Dfile.encoding=... instead

---
 T E S T S
---
Running org.apache.atlas.service.StoreBackedTypeCacheMetadataServiceTest
Running org.apache.atlas.util.CompiledQueryCacheKeyTest
Running org.apache.atlas.discovery.DataSetLineageServiceTest
Running org.apache.atlas.discovery.GraphBackedDiscoveryServiceTest
Running org.apache.atlas.repository.impexp.ZipSinkTest
Running org.apache.atlas.repository.audit.HBaseBasedAuditRepositoryHATest
Running org.apache.atlas.repository.audit.InMemoryAuditRepositoryTest
Running org.apache.atlas.repository.audit.HBaseBasedAuditRepositoryTest
Running org.apache.atlas.repository.graph.GraphBackedRepositorySoftDeleteTest
Running 
org.apache.atlas.repository.store.graph.v1.InverseReferenceUpdateHardDeleteV1Test
Running org.apache.atlas.repository.graph.ReverseReferenceUpdateHardDeleteTest
Running org.apache.atlas.service.DefaultMetadataServiceTest
Running org.apache.atlas.RepositoryServiceLoadingTest
Running org.apache.atlas.repository.impexp.ImportServiceReportingTest
Running org.apache.atlas.repository.store.graph.v1.AtlasDeleteHandlerV1Test
Running org.apache.atlas.repository.graph.GraphBackedMetadataRepositoryTest
Running org.apache.atlas.repository.graph.TestIntSequence
Running org.apache.atlas.repository.store.graph.v1.InverseReferenceUpdateV1Test
Running org.apache.atlas.repository.graph.GraphHelperTest
Running org.apache.atlas.TestModules
Running 
org.apache.atlas.repository.store.graph.v1.InverseReferenceUpdateSoftDeleteV1Test
Running org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStoreTest
Running org.apache.atlas.repository.store.graph.v1.SoftDeleteHandlerV1Test
Running org.apache.atlas.repository.graph.Gremlin3QueryOptimizerTest
Running org.apache.atlas.repository.impexp.ExportServiceTest
Running org.apache.atlas.repository.graph.ReverseReferenceUpdateSoftDeleteTest
Running org.apache.atlas.repository.graph.GraphBackedRepositoryHardDeleteTest
Running org.apache.atlas.repository.impexp.ImportServiceTest
Running org.apache.atlas.repository.impexp.ZipSourceTest
Running org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1Test
Running org.apache.atlas.repository.store.graph.AtlasEntityDefStoreV1Test
Running org.apache.atlas.repository.store.graph.v1.HardDeleteHandlerV1Test
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.676 sec - in 
org.apache.atlas.repository.impexp.ZipSinkTest
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.138 sec - in 
org.apache.atlas.repository.store.graph.v1.AtlasDeleteHandlerV1Test
Running org.apache.atlas.repository.graph.AbstractGremlinQueryOptimizerTest
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.072 sec - in 
org.apache.atlas.TestModules
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.441 sec - in 
org.apache.atlas.repository.impexp.ImportServiceReportingTest
Running org.apache.atlas.repository.graph.GraphBackedSearchIndexerMockTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.018 sec - in 
org.apache.atlas.util.CompiledQueryCacheKeyTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.877 sec - in 

[jira] [Commented] (ATLAS-1768) Create common types for Open Metadata

[Mandy Chessell (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025180#comment-16025180
 ] 

Mandy Chessell commented on ATLAS-1768:
---

These are the comments added by [~davidrad] in Jira ATLAS-1410.

210 I wonder if language should be a code table value - or more generally an 
valid value from reference data
===>  It is a descriptive field that we are not going to process - it is 
for human consumption so no value in creating a code table that will need to be 
synchronized between all of the metadata repositories. We can add language 
identifier - such as "En" is we need to process this in the future.

210 I am wondering about usage. Should this also be a code table - it seems 
more structural than the description
==> the usage is a description and examples of how the term is used - it is 
a string.   Look 

220 I suggest the supercategory to the subcategory be a composition (filled in 
diamond) relationship.
==>  This would not be correct because the subcategory is not deleted when 
the supercategory is deleted.  It remains linked to the Glossary object.  I 
have added an aggregation (open diamond) to show that the category is 
collecting subcategories.

230 I think the GlossaryCategory role name should be categories rather than 
category
   ==> done

240 I wonder about the "to" and "from" ends of the related term as they imply a 
direction - for a SYNONYM and TRANSLATION there is no direction. It is almost 
like synonyms and transactions should be in a synonym group or translation 
group respectively. Maybe we introduce an equivalence group concept, where 
everything in the group is related to everything else in the group. This would 
help for tag propagation for these terms.
  ==>  I think this over-complicates the model and would make it difficult to 
map to IGC.  Typically the synonyms are in different glossaries, connecting the 
canonical model.   I would be nervous about doing tag propagation along 
glossary relationships that are not from the spine model.

I don't think we have a way in the current Atlas model to constrain the number 
of classifications to 0..1.
 ==>  Classifications have a cardinality - are you saying it does not work?  Or 
something else?



> Create common types for Open Metadata
> -
>
> Key: ATLAS-1768
> URL: https://issues.apache.org/jira/browse/ATLAS-1768
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: Mandy Chessell
>  Labels: VirtualDataConnector
>
> This JIRA describes a proposal for standard types for open metadata entities 
> and relationships.  For example, glossaries, database definitions, rules, 
> policies, ...
> The value of having standard definitions for metadata is to enable type safe 
> APIs and business level UIs plus be able to exchange metadata between 
> different instances of metadata repositories.
> The implementation of these common types is divided into 8 areas:
> * Area 0 - for extensions to Apache Atlas's base model
> * Area 1 - for definitions of the data-related assets we are governing and 
> using
> * Area 2 - for a glossary of meanings and semantic relationships
> * Area 3 - for information about asset use, crowd-sourced definitions and 
> collaboration around the data-related assets
> * Area 4 - for governance such as policies, rules and classifications
> * Area 5 - for reference models and reference data
> * Area 6 - for metadata discovery processes (see 
> https://issues.apache.org/jira/browse/ATLAS-1748)
> * Area 7 - for lineage
> Adaptation and flexibility are key in metadata environments so these common 
> definitions must be extensible - and we still need to support the ad hoc 
> definition of new types in Atlas.
> Apache Atlas supports meta-types that are used in the definition of new 
> types.  These are currently enumeration, struct, classification and entity.  
> JIRA https://issues.apache.org/jira/browse/ATLAS-1690 adds relationships to 
> this list.  The open metadata models make use of all of these meta-types.  
> These are represented by sterotypes on the classes of the open metadata 
> definitions.
> The Atlas wiki has the models as a set of linked pages which are probably the 
> easiest way to view the models.
> Start here: 
> https://cwiki.apache.org/confluence/display/ATLAS/Building+out+the+Apache+Atlas+Typesystem



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (ATLAS-1768) Create common types for Open Metadata

[Mandy Chessell (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025180#comment-16025180
 ] 

Mandy Chessell edited comment on ATLAS-1768 at 5/25/17 6:37 PM:


These are the comments added by [~davidrad] in Jira ATLAS-1410.

210 I wonder if language should be a code table value - or more generally an 
valid value from reference data
===>  It is a descriptive field that we are not going to process - it is 
for human consumption so no value in creating a code table that will need to be 
synchronized between all of the metadata repositories. We can add language 
identifier - such as "En" is we need to process this in the future.

210 I am wondering about usage. Should this also be a code table - it seems 
more structural than the description
==> the usage is a description and examples of how the term is used - it is 
a string.   Look at dictionary definitions 

220 I suggest the supercategory to the subcategory be a composition (filled in 
diamond) relationship.
==>  This would not be correct because the subcategory is not deleted when 
the supercategory is deleted.  It remains linked to the Glossary object.  I 
have added an aggregation (open diamond) to show that the category is 
collecting subcategories.

230 I think the GlossaryCategory role name should be categories rather than 
category
   ==> done

240 I wonder about the "to" and "from" ends of the related term as they imply a 
direction - for a SYNONYM and TRANSLATION there is no direction. It is almost 
like synonyms and transactions should be in a synonym group or translation 
group respectively. Maybe we introduce an equivalence group concept, where 
everything in the group is related to everything else in the group. This would 
help for tag propagation for these terms.
  ==>  I think this over-complicates the model and would make it difficult to 
map to IGC.  Typically the synonyms are in different glossaries, connecting the 
canonical model.   I would be nervous about doing tag propagation along 
glossary relationships that are not from the spine model.

I don't think we have a way in the current Atlas model to constrain the number 
of classifications to 0..1.
 ==>  Classifications have a cardinality - are you saying it does not work?  Or 
something else?




was (Author: mandy_chessell):
These are the comments added by [~davidrad] in Jira ATLAS-1410.

210 I wonder if language should be a code table value - or more generally an 
valid value from reference data
===>  It is a descriptive field that we are not going to process - it is 
for human consumption so no value in creating a code table that will need to be 
synchronized between all of the metadata repositories. We can add language 
identifier - such as "En" is we need to process this in the future.

210 I am wondering about usage. Should this also be a code table - it seems 
more structural than the description
==> the usage is a description and examples of how the term is used - it is 
a string.   Look 

220 I suggest the supercategory to the subcategory be a composition (filled in 
diamond) relationship.
==>  This would not be correct because the subcategory is not deleted when 
the supercategory is deleted.  It remains linked to the Glossary object.  I 
have added an aggregation (open diamond) to show that the category is 
collecting subcategories.

230 I think the GlossaryCategory role name should be categories rather than 
category
   ==> done

240 I wonder about the "to" and "from" ends of the related term as they imply a 
direction - for a SYNONYM and TRANSLATION there is no direction. It is almost 
like synonyms and transactions should be in a synonym group or translation 
group respectively. Maybe we introduce an equivalence group concept, where 
everything in the group is related to everything else in the group. This would 
help for tag propagation for these terms.
  ==>  I think this over-complicates the model and would make it difficult to 
map to IGC.  Typically the synonyms are in different glossaries, connecting the 
canonical model.   I would be nervous about doing tag propagation along 
glossary relationships that are not from the spine model.

I don't think we have a way in the current Atlas model to constrain the number 
of classifications to 0..1.
 ==>  Classifications have a cardinality - are you saying it does not work?  Or 
something else?



> Create common types for Open Metadata
> -
>
> Key: ATLAS-1768
> URL: https://issues.apache.org/jira/browse/ATLAS-1768
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: Mandy Chessell
>  Labels: VirtualDataConnector
>
> This JIRA describes a 

[jira] [Comment Edited] (ATLAS-1410) V2 Glossary API

[Mandy Chessell (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024937#comment-16024937
 ] 

Mandy Chessell edited comment on ATLAS-1410 at 5/25/17 6:25 PM:


A proposed model for the Apache Atlas Glossary is shown on wiki page: 
https://cwiki.apache.org/confluence/display/ATLAS/Area+2+-+Glossary.

This model is defined in JIRA ATLAS-1768, 
https://issues.apache.org/jira/browse/ATLAS-1768


was (Author: mandy_chessell):
A proposed model for the Apache Atlas Glossary is shown on wiki page: 
https://cwiki.apache.org/confluence/display/ATLAS/Area+2+-+Glossary

> V2 Glossary API
> ---
>
> Key: ATLAS-1410
> URL: https://issues.apache.org/jira/browse/ATLAS-1410
> Project: Atlas
>  Issue Type: Improvement
>Reporter: David Radley
>Assignee: David Radley
> Attachments: Atlas Glossary V2 proposal v1.0.pdf, Atlas Glossary V2 
> proposal v1.1.pdf, Atlas Glossary V2 proposal v1.2.pdf, Atlas Glossary V2 
> proposal v1.3.pdf, Atlas Glossary V2 proposal v1.4.pdf
>
>
> The BaseResourceDefinition uses the AttributeDefintion class from typesystem. 
> There are newer more funcitonal versions of this capability in the atlas-intg 
> project. This Jira is changing over the glossary implementation to the newer 
> entity / type classes.  
> Instread of the instanceProperties and collectionProperties in the 
> BaseResourceDefintions we should use something in this sort of style :  
> "
>  AtlasEntityDef deptTypeDef =
> AtlasTypeUtil.createClassTypeDef(DEPARTMENT_TYPE, 
> "Department"+_description, ImmutableSet.of(),
> AtlasTypeUtil.createRequiredAttrDef("name", "string"),
> new AtlasAttributeDef("employees", 
> String.format("array<%s>", "Person"), true,
> AtlasAttributeDef.Cardinality.SINGLE, 0, 1, 
> false, false,
> 
> Collections.emptyList()));
> AtlasEntityDef personTypeDef = 
> AtlasTypeUtil.createClassTypeDef("Person", "Person"+_description, 
> ImmutableSet.of(),
> AtlasTypeUtil.createRequiredAttrDef("name", "string"),
> AtlasTypeUtil.createOptionalAttrDef("address", "Address"),
> AtlasTypeUtil.createOptionalAttrDef("birthday", "date"),
> AtlasTypeUtil.createOptionalAttrDef("hasPets", "boolean"),
> AtlasTypeUtil.createOptionalAttrDef("numberOfCars", "byte"),
> AtlasTypeUtil.createOptionalAttrDef("houseNumber", "short"),
> AtlasTypeUtil.createOptionalAttrDef("carMileage", "int"),
> AtlasTypeUtil.createOptionalAttrDef("age", "float"),
> "
> For the parent child relationships with glossary categories and terms we 
> should be able to have the type system manage edge deletion. As part of this, 
> we will need to investigate whether we could get rid of the disconnect and 
> connect methods added in ATLAS-1186 
>  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (ATLAS-1827) HiveHookIT failures due to missing files in target folder

[Sarath Subramanian (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025086#comment-16025086
 ] 

Sarath Subramanian commented on ATLAS-1827:
---

[~yussufshaikh], The mentioned HiveHookIT tests passes for me. Also the file 
creation logic in HiveHookIT.file() creates file in the target directory of 
hive-bridge project and not its parent (Validated this).
Could you mention how you are trying to run the ITs? using maven? 

> HiveHookIT failures due to missing files in target folder
> -
>
> Key: ATLAS-1827
> URL: https://issues.apache.org/jira/browse/ATLAS-1827
> Project: Atlas
>  Issue Type: Bug
>  Components: atlas-intg
>Affects Versions: 0.9-incubating
> Environment: # cat /etc/lsb-release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=14.04
> DISTRIB_CODENAME=trusty
> DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS"
> # uname -a
> Linux 3b9700711ca1 3.19.0-37-generic #42-Ubuntu SMP Fri Nov 20 18:22:05 UTC 
> 2015 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Yussuf Shaikh
> Fix For: 0.9-incubating
>
> Attachments: ATLAS-1827.patch
>
>
> Below tests fail because Hive cannot read the test files from 
> addons/hive-bridge/target folder.
> org.apache.atlas.hive.hook.HiveHookIT.testLoadDFSPathPartitioned
> org.apache.atlas.hive.hook.HiveHookIT.testLoadLocalPath
> org.apache.atlas.hive.hook.HiveHookIT.testLoadLocalPathIntoPartition
> Stack trace for testLoadLocalPath:
> 2017-05-24 06:37:37,150 ERROR - [main:] ~ FAILED: SemanticException Line 1:23 
> Invalid path 
> ''file:///var/lib/jenkins/workspace/atlas-test/addons/hive-bridge/./target/load-data-7KV7AalW6D'':
>  No files matching path 
> file:/var/lib/jenkins/workspace/atlas-test/addons/hive-bridge/target/load-data-7KV7AalW6D
> org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:23 Invalid path 
> ''file:///var/lib/jenkins/workspace/atlas-test/addons/hive-bridge/./target/load-data-7KV7AalW6D'':
>  No files matching path 
> file:/var/lib/jenkins/workspace/atlas-test/addons/hive-bridge/target/load-data-7KV7AalW6D
>   at 
> org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.applyConstraints(LoadSemanticAnalyzer.java:139)
>   at 
> org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:230)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:424)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
>   at 
> org.apache.atlas.hive.HiveITBase.runCommandWithDelay(HiveITBase.java:119)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (ATLAS-1410) V2 Glossary API

[David Radley (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025052#comment-16025052
 ] 

David Radley commented on ATLAS-1410:
-

[~mandy_chessell] Looks really good. Some thoughts: 

- 210 I wonder if language should be a code table value - or more generally an 
valid value from reference data   
-210 I am wondering about usage. Should this also be a code table - it seems 
more structural than the description 
-220 I suggest the supercategory to the subcategory be a composition (filled in 
diamond) relationship.  
- 230 I think the GlossaryCategory role name should be categories rather than 
category 
- 240 I wonder about the "to" and "from" ends of the related term as they imply 
a direction - for a SYNONYM and TRANSLATION there is no direction. It is almost 
like synonyms and transactions should be in a synonym group or translation 
group respectively. Maybe we introduce an equivalence group concept, where 
everything in the group is related to everything else in the group. This would 
help for tag propagation for these terms.

I don't think we have a way in the current Atlas model to constrain the number 
of classifications to  0..1. 

> V2 Glossary API
> ---
>
> Key: ATLAS-1410
> URL: https://issues.apache.org/jira/browse/ATLAS-1410
> Project: Atlas
>  Issue Type: Improvement
>Reporter: David Radley
>Assignee: David Radley
> Attachments: Atlas Glossary V2 proposal v1.0.pdf, Atlas Glossary V2 
> proposal v1.1.pdf, Atlas Glossary V2 proposal v1.2.pdf, Atlas Glossary V2 
> proposal v1.3.pdf, Atlas Glossary V2 proposal v1.4.pdf
>
>
> The BaseResourceDefinition uses the AttributeDefintion class from typesystem. 
> There are newer more funcitonal versions of this capability in the atlas-intg 
> project. This Jira is changing over the glossary implementation to the newer 
> entity / type classes.  
> Instread of the instanceProperties and collectionProperties in the 
> BaseResourceDefintions we should use something in this sort of style :  
> "
>  AtlasEntityDef deptTypeDef =
> AtlasTypeUtil.createClassTypeDef(DEPARTMENT_TYPE, 
> "Department"+_description, ImmutableSet.of(),
> AtlasTypeUtil.createRequiredAttrDef("name", "string"),
> new AtlasAttributeDef("employees", 
> String.format("array<%s>", "Person"), true,
> AtlasAttributeDef.Cardinality.SINGLE, 0, 1, 
> false, false,
> 
> Collections.emptyList()));
> AtlasEntityDef personTypeDef = 
> AtlasTypeUtil.createClassTypeDef("Person", "Person"+_description, 
> ImmutableSet.of(),
> AtlasTypeUtil.createRequiredAttrDef("name", "string"),
> AtlasTypeUtil.createOptionalAttrDef("address", "Address"),
> AtlasTypeUtil.createOptionalAttrDef("birthday", "date"),
> AtlasTypeUtil.createOptionalAttrDef("hasPets", "boolean"),
> AtlasTypeUtil.createOptionalAttrDef("numberOfCars", "byte"),
> AtlasTypeUtil.createOptionalAttrDef("houseNumber", "short"),
> AtlasTypeUtil.createOptionalAttrDef("carMileage", "int"),
> AtlasTypeUtil.createOptionalAttrDef("age", "float"),
> "
> For the parent child relationships with glossary categories and terms we 
> should be able to have the type system manage edge deletion. As part of this, 
> we will need to investigate whether we could get rid of the disconnect and 
> connect methods added in ATLAS-1186 
>  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 58803: ATLAS-1390: IBM graph implementation

[Apoorv Naik]

> On May 10, 2017, 2:28 p.m., Graham Wallis wrote:
> > I managed to get this to build and ran it against an IBM Graph instance. 
> > But there are a number of things that needed changes/workarounds to get it 
> > working:
> > 
> > 1. The atlas-graphdb-ibm-graph build relies on the java-8 build profile 
> > being activated so that it can set skipMain and skip to false, so that the 
> > ibm-graph component gets built.  The java-8 profile has an activation rule 
> > of:
> > 
> > 
> > java-8
> > 
> > 1.8
> > 
> > 
> >   .. etc
> > ..but even when running with a 1.8 Java environment the profile was not 
> > activated. It was therefore necessary to explicitly activate the profile, 
> > by including -Pjava-8 on the maven command line. With both the java-8 and 
> > ibm-graph profiles explicitly named, it worked and the ibm-graph component 
> > was built.
> > 
> > 2. When the ibm-graph profile is built, the resulting packaged jar contains 
> > both ibm-graph and titan0. The different graph providers are assumed to be 
> > mutually exclusive as indicated by the maven dependency exclusions (e.g. if 
> > titan0, exclude titan0 and ibm-graph). There must be an unintended 
> > transitive dependency on titan0. I didn't bother to fix this (in the pom 
> > files) because I'm not sure that mutual exclusion is actually the best 
> > approach so I raised this as a question on JIRA ATLAS-1757.
> > 
> > 3. With the above changes, the ibm-graph profile builds, but when testing 
> > it (atlas_start.py followed by quick_start.py) there is an NPE when trying 
> > to loadGraphUpdateScript. This is because the readStream is null because 
> > the script file path is incorrectly specified: the web root is 
> > "src/main/resources" so the groovy script uri should be 
> > "graphUpdater.groovy" rather than "src/main/resources/graphUpdater.groovy".
> > 
> > 4. With the above changes, it is possible to connect to an IBM Graph 
> > instance and to run through the quick_start.py, but not completely cleanly. 
> > I detected the following errors:
> > 
> > 2017-05-09 15:42:53,586 INFO  - [pool-1-thread-10:] ~ 
> > test-1-graph:Test-1-Graph: e26d0d19-7163-4687-9608-6113d915cfc5:  begin 
> > transaction e26d0d19-7163-4687-9608-6113d915cfc5 (IBMGraphGraph:974)
> > 2017-05-09 15:42:53,587 INFO  - [pool-1-thread-10 - 
> > 55ad3ab1-8acb-4647-9874-5b7741b8d2b6:] ~ test-1-graph:Test-1-Graph: 
> > bf922368-abaa-4354-b942-d63b40044c56:  begin transaction 
> > bf922368-abaa-4354-b942-d63b40044c56 (IBMGraphGraph:974)
> > 2017-05-09 15:42:53,678 ERROR - [pool-1-thread-10 - 
> > 55ad3ab1-8acb-4647-9874-5b7741b8d2b6:] ~ Error handling a request: 
> > e417ee56dcfcd88d (ExceptionMapperUtil:32)
> > org.apache.atlas.ibmgraph.exception.BadRequestException: Request POST 
> > https://ibmgraph-alpha.eu-gb.bluemix.net/be6aad23-26ca-4776-9b2a-b3ccffa90f63/test-1-graph/gremlin
> > {"gremlin":"import java.util.function.Function;import 
> > org.apache.tinkerpop.gremlin.process.traversal.strategy.decoration.PartitionStrategy;import
> >  static org.apache.tinkerpop.gremlin.process.traversal.P.*;def 
> > g=graph.traversal();g.V().has('__typeName', T.in, typeNames) 
> > [startIdx.. >  failed.  Failure response: HttpResponse[Status = 400 - Bad Request, IBM 
> > Graph Request Id = c20f2b5e-b693-41f3-982c-2dc6c14791b5]
> > Response Body:
> > {"code":"BadRequestError","message":"[Static type checking] - No such 
> > property: in for class: java.lang.Class 
> > , [Static type checking] - Cannot 
> > find matching method 
> > org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversal#has(java.lang.String,
> >  java.lang.Object, java.util.ArrayList ). 
> > Please check if the declared type is right and if the method exists."}
> > at 
> > org.apache.atlas.ibmgraph.http.HttpRequestHandler.handleFailures(HttpRequestHandler.java:250)
> > at 
> > org.apache.atlas.ibmgraph.http.HttpRequestHandler.translateResponse(HttpRequestHandler.java:199)
> > at 
> > org.apache.atlas.ibmgraph.http.HttpRequestHandler.processRequest_(HttpRequestHandler.java:140)
> > at 
> > org.apache.atlas.ibmgraph.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:108)
> > at 
> > org.apache.atlas.ibmgraph.api.GraphDatabaseClient.processMultiValuedRequest(GraphDatabaseClient.java:124)
> > at 
> > org.apache.atlas.ibmgraph.api.GraphDatabaseClient.executeGremlin(GraphDatabaseClient.java:239)
> > at 
> > org.apache.atlas.ibmgraph.api.GraphDatabaseClient.executeGremlin(GraphDatabaseClient.java:227)
> > 
> > I think the problem here is that Atlas is trying to use Tinkerpop 2 syntax 
> > with a Tinkerpop 3 graph; hence the confusion looking for an 'in' property 
> > for the class 'T'. In TP3, there are first-class predicates, such as 
> > within(), that I think should be used for this 

Re: Review Request 56959: Import API: Added Support for Transforming Entities During Import

[Ashutosh Mestry]

> On May 25, 2017, 2:19 p.m., Apoorv Naik wrote:
> > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
> > Line 366 (original), 367 (patched)
> > 
> >
> > Can you check if the data can directly be serialized the Java POJO here 
> > ? I did see a few similar usecases on StackOverflow and they were 
> > de-serializing into the POJO directly at the interface level.

I got an exception when i tried it earlier. I have simplified the 
_AtlasImportRequest_ class further. Let me re-attempt it.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/#review176074
---


On May 25, 2017, 5:16 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56959/
> ---
> 
> (Updated May 25, 2017, 5:16 a.m.)
> 
> 
> Review request for atlas and Madhan Neethiraj.
> 
> 
> Bugs: ATLAS-1825
> https://issues.apache.org/jira/browse/ATLAS-1825
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Please refer to 
> [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios 
> and solution approach.
> 
> Impact
> ==
> * New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This 
> holds a map of entity-type to attribute and the tranforms to be applied.
> * New class _AtlasImportTransformBuiler_ reads the requests specified in the 
> query string of the import API and creates a _AtlasImportTransfor_ structure.
> * _AtlasImportTransformBuiler_ has new classes that support transformations. 
> They are:
> * _replace_ Implements find and replace on attribute values.
> * _lowercase_ Converts attribute value to lower case. 
> * Additional classes that unit test these classes have been added.
> 
> Implementation Approach
> ===
> * _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the 
> transform every entity that is fetched from the source.
> * _Import API_ now accepts multi-part input. This allows for elegantly 
> specifying import parameters as well as the binary input from the file.
> * Please take note of _ImportTransformerEnum_. This is first case where Enum 
> has state variables.
> 
> CURL
> 
> 
> *Setup*
> Create file with these contents call it _importTransform.json_:
> ```javascript
> {
> "options": {},
> "importTransform": {
> "transforms": {
> "hive_table": {
> "qualifiedName": [
> "lowercase",
> "replace:@cl1:@cl2"
> ]
> }
> }
> }
> }
> ```
> 
> ```
> curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H 
> "Cache-Control: no-cache" -F request=@../docs/importTransform.json -F 
> data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import;
> ```
> 
> Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is 
> not required anymore.
> 
> Notice the output from the API, it has the _AtlasRequest_ that was passed as 
> input.
> 
> Documentation
> =
> Need to update TWIKI documents.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
> 7530196d 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransforms.java 
> PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/impexp/ImportTransformerEnum.java 
> PRE-CREATION 
>   
> intg/src/test/java/org/apache/atlas/model/impexp/ImportTransformerEnumTest.java
>  PRE-CREATION 
>   pom.xml a93b8ad9 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportEntityTransformer.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
>  9ec15e00 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 
> 87c9f0eb 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportEntityTransformerTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
>  08e9ee81 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/TransformEvaluatorTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/services/EntityDiscoveryServiceTest.java
>  d4900838 
>   webapp/pom.xml 4132912d 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
> 01f9a860 
> 
> 
> Diff: https://reviews.apache.org/r/56959/diff/6/
> 
> 
> Testing
> ---
> 
> **Unit Test**
> * Added tests for *AtlasImportTransformBuilder* and contained classes.
> * Updated *ZipSourceTest*
> * Added tests to *ImportServiceTest* to exercise new 

[jira] [Commented] (ATLAS-1768) Create common types for Open Metadata

[Christopher Grote (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024945#comment-16024945
 ] 

Christopher Grote commented on ATLAS-1768:
--

Looks great!

For the Dictionary model, would it make sense to add a numeric "confidence" to 
the SemanticAssignment object, given the allowance in TermAssignmentStatus for 
a DISCOVERED status (i.e. to capture not just that a relationship was 
discovered, but the confidence of its accuracy)?

> Create common types for Open Metadata
> -
>
> Key: ATLAS-1768
> URL: https://issues.apache.org/jira/browse/ATLAS-1768
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: Mandy Chessell
>  Labels: VirtualDataConnector
>
> This JIRA describes a proposal for standard types for open metadata entities 
> and relationships.  For example, glossaries, database definitions, rules, 
> policies, ...
> The value of having standard definitions for metadata is to enable type safe 
> APIs and business level UIs plus be able to exchange metadata between 
> different instances of metadata repositories.
> The implementation of these common types is divided into 8 areas:
> * Area 0 - for extensions to Apache Atlas's base model
> * Area 1 - for definitions of the data-related assets we are governing and 
> using
> * Area 2 - for a glossary of meanings and semantic relationships
> * Area 3 - for information about asset use, crowd-sourced definitions and 
> collaboration around the data-related assets
> * Area 4 - for governance such as policies, rules and classifications
> * Area 5 - for reference models and reference data
> * Area 6 - for metadata discovery processes (see 
> https://issues.apache.org/jira/browse/ATLAS-1748)
> * Area 7 - for lineage
> Adaptation and flexibility are key in metadata environments so these common 
> definitions must be extensible - and we still need to support the ad hoc 
> definition of new types in Atlas.
> Apache Atlas supports meta-types that are used in the definition of new 
> types.  These are currently enumeration, struct, classification and entity.  
> JIRA https://issues.apache.org/jira/browse/ATLAS-1690 adds relationships to 
> this list.  The open metadata models make use of all of these meta-types.  
> These are represented by sterotypes on the classes of the open metadata 
> definitions.
> The Atlas wiki has the models as a set of linked pages which are probably the 
> easiest way to view the models.
> Start here: 
> https://cwiki.apache.org/confluence/display/ATLAS/Building+out+the+Apache+Atlas+Typesystem



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (ATLAS-1410) V2 Glossary API

[Mandy Chessell (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024937#comment-16024937
 ] 

Mandy Chessell commented on ATLAS-1410:
---

A proposed model for the Apache Atlas Glossary is shown on wiki page: 
https://cwiki.apache.org/confluence/display/ATLAS/Area+2+-+Glossary

> V2 Glossary API
> ---
>
> Key: ATLAS-1410
> URL: https://issues.apache.org/jira/browse/ATLAS-1410
> Project: Atlas
>  Issue Type: Improvement
>Reporter: David Radley
>Assignee: David Radley
> Attachments: Atlas Glossary V2 proposal v1.0.pdf, Atlas Glossary V2 
> proposal v1.1.pdf, Atlas Glossary V2 proposal v1.2.pdf, Atlas Glossary V2 
> proposal v1.3.pdf, Atlas Glossary V2 proposal v1.4.pdf
>
>
> The BaseResourceDefinition uses the AttributeDefintion class from typesystem. 
> There are newer more funcitonal versions of this capability in the atlas-intg 
> project. This Jira is changing over the glossary implementation to the newer 
> entity / type classes.  
> Instread of the instanceProperties and collectionProperties in the 
> BaseResourceDefintions we should use something in this sort of style :  
> "
>  AtlasEntityDef deptTypeDef =
> AtlasTypeUtil.createClassTypeDef(DEPARTMENT_TYPE, 
> "Department"+_description, ImmutableSet.of(),
> AtlasTypeUtil.createRequiredAttrDef("name", "string"),
> new AtlasAttributeDef("employees", 
> String.format("array<%s>", "Person"), true,
> AtlasAttributeDef.Cardinality.SINGLE, 0, 1, 
> false, false,
> 
> Collections.emptyList()));
> AtlasEntityDef personTypeDef = 
> AtlasTypeUtil.createClassTypeDef("Person", "Person"+_description, 
> ImmutableSet.of(),
> AtlasTypeUtil.createRequiredAttrDef("name", "string"),
> AtlasTypeUtil.createOptionalAttrDef("address", "Address"),
> AtlasTypeUtil.createOptionalAttrDef("birthday", "date"),
> AtlasTypeUtil.createOptionalAttrDef("hasPets", "boolean"),
> AtlasTypeUtil.createOptionalAttrDef("numberOfCars", "byte"),
> AtlasTypeUtil.createOptionalAttrDef("houseNumber", "short"),
> AtlasTypeUtil.createOptionalAttrDef("carMileage", "int"),
> AtlasTypeUtil.createOptionalAttrDef("age", "float"),
> "
> For the parent child relationships with glossary categories and terms we 
> should be able to have the type system manage edge deletion. As part of this, 
> we will need to investigate whether we could get rid of the disconnect and 
> connect methods added in ATLAS-1186 
>  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (ATLAS-1768) Create common types for Open Metadata

[Mandy Chessell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mandy Chessell updated ATLAS-1768:
--
Description: 
This JIRA describes a proposal for standard types for open metadata entities 
and relationships.  For example, glossaries, database definitions, rules, 
policies, ...

The value of having standard definitions for metadata is to enable type safe 
APIs and business level UIs plus be able to exchange metadata between different 
instances of metadata repositories.

The implementation of these common types is divided into 8 areas:
* Area 0 - for extensions to Apache Atlas's base model
* Area 1 - for definitions of the data-related assets we are governing and using
* Area 2 - for a glossary of meanings and semantic relationships
* Area 3 - for information about asset use, crowd-sourced definitions and 
collaboration around the data-related assets
* Area 4 - for governance such as policies, rules and classifications
* Area 5 - for reference models and reference data
* Area 6 - for metadata discovery processes (see 
https://issues.apache.org/jira/browse/ATLAS-1748)
* Area 7 - for lineage

Adaptation and flexibility are key in metadata environments so these common 
definitions must be extensible - and we still need to support the ad hoc 
definition of new types in Atlas.

Apache Atlas supports meta-types that are used in the definition of new types.  
These are currently enumeration, struct, classification and entity.  JIRA 
https://issues.apache.org/jira/browse/ATLAS-1690 adds relationships to this 
list.  The open metadata models make use of all of these meta-types.  These are 
represented by sterotypes on the classes of the open metadata definitions.

The Atlas wiki has the models as a set of linked pages which are probably the 
easiest way to view the models.
Start here: 
https://cwiki.apache.org/confluence/display/ATLAS/Building+out+the+Apache+Atlas+Typesystem


  was:
This JIRA describes a proposal for standard types for common metadata entities 
and relationships.  For example, glossaries, database definitions, rules, 
policies, ...

The value of having standard definitions for metadata is to enable type safe 
APIs and business level UIs plus be able to exchange metadata between different 
instances of metadata repositories.

The implementation of these common definitions can be divided into 2 groups:
 - standard system attributes for all entities and relationships
 - standard types for commonly used metadata

Adaptation and flexibility are key in metadata environments so these common 
definitions must be extensible - and we still need to support the ad hoc 
definition of new types in Atlas.


> Create common types for Open Metadata
> -
>
> Key: ATLAS-1768
> URL: https://issues.apache.org/jira/browse/ATLAS-1768
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: Mandy Chessell
>  Labels: VirtualDataConnector
>
> This JIRA describes a proposal for standard types for open metadata entities 
> and relationships.  For example, glossaries, database definitions, rules, 
> policies, ...
> The value of having standard definitions for metadata is to enable type safe 
> APIs and business level UIs plus be able to exchange metadata between 
> different instances of metadata repositories.
> The implementation of these common types is divided into 8 areas:
> * Area 0 - for extensions to Apache Atlas's base model
> * Area 1 - for definitions of the data-related assets we are governing and 
> using
> * Area 2 - for a glossary of meanings and semantic relationships
> * Area 3 - for information about asset use, crowd-sourced definitions and 
> collaboration around the data-related assets
> * Area 4 - for governance such as policies, rules and classifications
> * Area 5 - for reference models and reference data
> * Area 6 - for metadata discovery processes (see 
> https://issues.apache.org/jira/browse/ATLAS-1748)
> * Area 7 - for lineage
> Adaptation and flexibility are key in metadata environments so these common 
> definitions must be extensible - and we still need to support the ad hoc 
> definition of new types in Atlas.
> Apache Atlas supports meta-types that are used in the definition of new 
> types.  These are currently enumeration, struct, classification and entity.  
> JIRA https://issues.apache.org/jira/browse/ATLAS-1690 adds relationships to 
> this list.  The open metadata models make use of all of these meta-types.  
> These are represented by sterotypes on the classes of the open metadata 
> definitions.
> The Atlas wiki has the models as a set of linked pages which are probably the 
> easiest way to view the models.
> Start here: 
> 

[jira] [Issue Comment Deleted] (ATLAS-1768) Create common types for Open Metadata

[Mandy Chessell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mandy Chessell updated ATLAS-1768:
--
Comment: was deleted

(was: The Apache Atlas Repository Core model is a working document covering the 
properties of vertices and edges proposed for the Apache Atlas graph database.)

> Create common types for Open Metadata
> -
>
> Key: ATLAS-1768
> URL: https://issues.apache.org/jira/browse/ATLAS-1768
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: Mandy Chessell
>  Labels: VirtualDataConnector
>
> This JIRA describes a proposal for standard types for common metadata 
> entities and relationships.  For example, glossaries, database definitions, 
> rules, policies, ...
> The value of having standard definitions for metadata is to enable type safe 
> APIs and business level UIs plus be able to exchange metadata between 
> different instances of metadata repositories.
> The implementation of these common definitions can be divided into 2 groups:
>  - standard system attributes for all entities and relationships
>  - standard types for commonly used metadata
> Adaptation and flexibility are key in metadata environments so these common 
> definitions must be extensible - and we still need to support the ad hoc 
> definition of new types in Atlas.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (ATLAS-1768) Create common types for Open Metadata

[Mandy Chessell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mandy Chessell updated ATLAS-1768:
--
Attachment: (was: Apache Atlas Repository Core Model - 7th May 2017.pdf)

> Create common types for Open Metadata
> -
>
> Key: ATLAS-1768
> URL: https://issues.apache.org/jira/browse/ATLAS-1768
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: Mandy Chessell
>  Labels: VirtualDataConnector
>
> This JIRA describes a proposal for standard types for common metadata 
> entities and relationships.  For example, glossaries, database definitions, 
> rules, policies, ...
> The value of having standard definitions for metadata is to enable type safe 
> APIs and business level UIs plus be able to exchange metadata between 
> different instances of metadata repositories.
> The implementation of these common definitions can be divided into 2 groups:
>  - standard system attributes for all entities and relationships
>  - standard types for commonly used metadata
> Adaptation and flexibility are key in metadata environments so these common 
> definitions must be extensible - and we still need to support the ad hoc 
> definition of new types in Atlas.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Atlas model for the glossary

[Mandy Chessell]

Dear All,
Please link to 
https://cwiki.apache.org/confluence/display/ATLAS/Area+2+-+Glossary to see 
a proposed model for the open metadata glossary.  Feedback welcome - 
either by the dev list - or Jira 
https://issues.apache.org/jira/browse/ATLAS-1768

All the best
Mandy
___
Mandy Chessell CBE FREng CEng FBCS
IBM Distinguished Engineer

Master Inventor
Member of the IBM Academy of Technology
Visiting Professor, Department of Computer Science, University of 
Sheffield

Email: mandy_chess...@uk.ibm.com
LinkedIn: http://www.linkedin.com/pub/mandy-chessell/22/897/a49

Assistant: Janet Brooks - jsbrook...@uk.ibm.com

[jira] [Commented] (ATLAS-1768) Create common types for Open Metadata

[Mandy Chessell (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024871#comment-16024871
 ] 

Mandy Chessell commented on ATLAS-1768:
---

The model for the glossary is described on the Atlas Wiki - 
https://cwiki.apache.org/confluence/display/ATLAS/Area+2+-+Glossary

> Create common types for Open Metadata
> -
>
> Key: ATLAS-1768
> URL: https://issues.apache.org/jira/browse/ATLAS-1768
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: Mandy Chessell
>  Labels: VirtualDataConnector
> Attachments: Apache Atlas Repository Core Model - 7th May 2017.pdf
>
>
> This JIRA describes a proposal for standard types for common metadata 
> entities and relationships.  For example, glossaries, database definitions, 
> rules, policies, ...
> The value of having standard definitions for metadata is to enable type safe 
> APIs and business level UIs plus be able to exchange metadata between 
> different instances of metadata repositories.
> The implementation of these common definitions can be divided into 2 groups:
>  - standard system attributes for all entities and relationships
>  - standard types for commonly used metadata
> Adaptation and flexibility are key in metadata environments so these common 
> definitions must be extensible - and we still need to support the ad hoc 
> definition of new types in Atlas.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 56959: Import API: Added Support for Transforming Entities During Import

[Apoorv Naik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/#review176074
---




webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
Line 366 (original), 367 (patched)


Can you check if the data can directly be serialized the Java POJO here ? I 
did see a few similar usecases on StackOverflow and they were de-serializing 
into the POJO directly at the interface level.


- Apoorv Naik


On May 25, 2017, 5:16 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56959/
> ---
> 
> (Updated May 25, 2017, 5:16 a.m.)
> 
> 
> Review request for atlas and Madhan Neethiraj.
> 
> 
> Bugs: ATLAS-1825
> https://issues.apache.org/jira/browse/ATLAS-1825
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Please refer to 
> [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios 
> and solution approach.
> 
> Impact
> ==
> * New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This 
> holds a map of entity-type to attribute and the tranforms to be applied.
> * New class _AtlasImportTransformBuiler_ reads the requests specified in the 
> query string of the import API and creates a _AtlasImportTransfor_ structure.
> * _AtlasImportTransformBuiler_ has new classes that support transformations. 
> They are:
> * _replace_ Implements find and replace on attribute values.
> * _lowercase_ Converts attribute value to lower case. 
> * Additional classes that unit test these classes have been added.
> 
> Implementation Approach
> ===
> * _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the 
> transform every entity that is fetched from the source.
> * _Import API_ now accepts multi-part input. This allows for elegantly 
> specifying import parameters as well as the binary input from the file.
> * Please take note of _ImportTransformerEnum_. This is first case where Enum 
> has state variables.
> 
> CURL
> 
> 
> *Setup*
> Create file with these contents call it _importTransform.json_:
> ```javascript
> {
> "options": {},
> "importTransform": {
> "transforms": {
> "hive_table": {
> "qualifiedName": [
> "lowercase",
> "replace:@cl1:@cl2"
> ]
> }
> }
> }
> }
> ```
> 
> ```
> curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H 
> "Cache-Control: no-cache" -F request=@../docs/importTransform.json -F 
> data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import;
> ```
> 
> Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is 
> not required anymore.
> 
> Notice the output from the API, it has the _AtlasRequest_ that was passed as 
> input.
> 
> Documentation
> =
> Need to update TWIKI documents.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
> 7530196d 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportTransforms.java 
> PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/impexp/ImportTransformerEnum.java 
> PRE-CREATION 
>   
> intg/src/test/java/org/apache/atlas/model/impexp/ImportTransformerEnumTest.java
>  PRE-CREATION 
>   pom.xml a93b8ad9 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportEntityTransformer.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
>  9ec15e00 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 
> 87c9f0eb 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportEntityTransformerTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
>  08e9ee81 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/TransformEvaluatorTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/services/EntityDiscoveryServiceTest.java
>  d4900838 
>   webapp/pom.xml 4132912d 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
> 01f9a860 
> 
> 
> Diff: https://reviews.apache.org/r/56959/diff/6/
> 
> 
> Testing
> ---
> 
> **Unit Test**
> * Added tests for *AtlasImportTransformBuilder* and contained classes.
> * Updated *ZipSourceTest*
> * Added tests to *ImportServiceTest* to exercise new option.
> 
> **Functional Test**
> * Small size data export & import.
> * Medium size data export & import.
> 
> **Performance Tests**
> Not exhaustive. Simple stop watch tests to compare against baseline.
> 
> 
> Thanks,
> 

[jira] [Updated] (ATLAS-1811) UI : Audit Events are not shown in sorted order based on time sometimes.

[Kalyani Kashikar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kalyani Kashikar updated ATLAS-1811:

Attachment: ATLAS-1811.patch

> UI : Audit Events are not shown in sorted order based on time sometimes.
> 
>
> Key: ATLAS-1811
> URL: https://issues.apache.org/jira/browse/ATLAS-1811
> Project: Atlas
>  Issue Type: Bug
>  Components: atlas-webui
>Affects Versions: trunk, 0.9-incubating
>Reporter: Sharmadha Sainath
>Assignee: Kalyani Kashikar
> Attachments: ATLAS-1811.patch, AuditDetailsUnordered.png
>
>
> Created an entity and made did many tag association , dis-association and tag 
> update to that entity. All audit details are captured but not displayed in 
> sorted order based on timestamp value present in audit response. A few 
> entries are missing order.
> Attached the screenshot .



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

RE: [VOTE] Graduate Apache Atlas Project from Incubator

[Nigel L Jones]

+1 from me.

I'm sure together we can take Apache Atlas from strength to strength by
working together in a vibrant and diverse community in terms of improving
it's capabilities through good architecture, design & coding, and also
ensuring these are appreciated by users and potential users through
testing, documentation, papers & similar. It has massive potential both in
hadoop and in a much broader ecosystem and I'm thoroughly excited to be
part of this.

Whilst graduation doesn't automatically make this happen, it does send a
message that Atlas is ready for prime time, and has serious commitment
behind it.

Well done Atlas!

Nigel Jones
IBM

-Original Message-
From: Suma Shivaprasad [mailto:sumasai.shivapra...@gmail.com]
Sent: 25 May 2017 00:23
To: dev@atlas.incubator.apache.org
Subject: [VOTE] Graduate Apache Atlas Project from Incubator

Dear Atlas Community Members,

Following the discussion in Atlas community on graduating Apache Atlas
Project from Incubation to a Top Level Project, starting this voting thread
within the Atlas community. I appreciate mentors and community members in
making this happen and thanks for their support. Apache Atlas Project has
come a long way since entering into incubation during May 2015 and the
project summary shows the details on this.

Please vote on the Apache Atlas Project resolution that is found in bottom
of this email.

[ ] +1 Graduate Apache Atlas from the Incubator.
[ ] +0 No opinion
[ ] -1 Don't graduate Apache Atlas from the Incubator ( please provide the
reason)

This VOTE will be opened for next 72 hours.

If this VOTE succeeds, a similar VOTE will be done in general@incubator
mailing list. If that succeeds, a resolution will be included in the next
Apache Board Meeting.

Here is my Vote +1 (binding)

Thanks all for your participation
Suma


Project Summary:

=
http://incubator.apache.org/projects/atlas.html

Project website:
=

http://atlas.incubator.apache.org

Project Documentation:
===

http://atlas.incubator.apache.org/index.html
Architecture - http://atlas.incubator.apache.org/Architecture.html
Installation guide - http://atlas.incubator.apache.org/InstallationSteps.
html
QuickStart Guide- http://atlas.incubator.apache.org/QuickStart.html
REST API - http://atlas.incubator.apache.org/api/v2/index.html
Atlas Release Guide - https://cwiki.apache.org/confluence/display/ATLAS/
Developer+Resources

Project maturity Assessment:
===

https://cwiki.apache.org/confluence/display/ATLAS/
Apache+Atlas+Project+Maturity+Model

Proposed PMC size: 38

Total number of committers  : 38 members Total number of contributors : 62
members

PMC affiliation (* indicated chair)

* Hortonworks (13)
   IBM (3)
   Aetna(4)
   Freestone (3)
   Merck (2)
   Oracle(2)
   Target (2)
   Schlumberger(2)
   Dataguise(1)
   Inmobi(1)
   JPMC(1)
   MPR(1)
   SAP(1)
   SparklineData(1)
   Others(1)

1660 commits on master
80 contributors across all branches
Dev list averaged ~500-800 msgs/month in 2017 User list was created
recently in April 2017. Hence not considering stats for this.
1827 issues created

Committer¹s affiliation:
===
* Hortonworks (13)
   IBM (3)
   Aetna(4)
   Freestone (3)
   Merck (2)
   Oracle(2)
   Target (2)
   Schlumberger(2)
   Dataguise(1)
   Inmobi(1)
   JPMC(1)
   MPR(1)
   SAP(1)
   SparklineData(1)
   Others(1)


Apache Atlas Top Level Project Resolution:


Establish the Apache Atlas Project

WHEREAS, the Board of Directors deems it to be in the best interests of the
Foundation and consistent with the Foundation¹s purpose to establish a
Project Management Committee charged with the creation and maintenance of
open-source software, for distribution at no charge to the public, related
to a data management platform That provides real-time, consistent access to
data-intensive applications throughout widely distributed cloud
architectures.

NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee (PMC),
to be known as the "Apache Atlas Project", be and hereby is established
pursuant to Bylaws of the Foundation; and be it further RESOLVED,that the
Apache Atlas Project be and hereby is responsible for the creation and
maintenance of software related to a data management platform that provides
real-time, consistent access to data-intensive applications throughout
widely distributed cloud architectures.

RESOLVED, that the office of "Vice President, Apache Atlas" be and hereby
is created, the person holding such office to serve at the direction of the
Board of Directors as the chair of the Apache Atlas Project, and to have
primary responsibility for management of the projects within the scope of
responsibility of the Apache Atlas Project; and be it Further.

>RESOLVED,that the persons listed immediately below be and hereby are
>appointed to serve as the initial members of the Apache Atlas Project:

Aaron Dossett adoss...@target.com 

[jira] [Commented] (ATLAS-1821) Classification propagation from entity to a derivative or child entity

[David Radley (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024460#comment-16024460
 ] 

David Radley commented on ATLAS-1821:
-

I think having the classifications propagate is powerful. I am concerned that 
we need to restrict which classifications propagate across which relationships. 
If we want to pursue a declarative approach to this, I would like to understand 
the process by which conflicts are resolved and by who and how we know that 
these conflict resolutions meet with the companies governance policies. 

We have a use case where the glossary is set up with a Glossary Term National 
Insurance number, which is tagged as confidential. It is mapped to a masked 
column and an unmasked column. The masked column can be public, but the 
unmasked column need to get the Terms security classification. This would not 
be a conflict. 

I suggest a rules based approach be used instead. In this case a governance 
team could define a set of rules around how classifications flow , including 
special cases. Maybe something like :

for all Glossary terms that have assigned assets, flow the terms 
confidentiality level classification to the assigned asset, except in the case 
where if the assigned asset is masked -  then classify it as public 

We are then in a position to author rules that encapsulate best governance 
practices and play a part to enforce governance standards.   






> Classification propagation from entity to a derivative or child entity
> --
>
> Key: ATLAS-1821
> URL: https://issues.apache.org/jira/browse/ATLAS-1821
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core, atlas-webui
>Reporter: Srikanth Venkat
> Fix For: 0.9-incubating
>
>
> User Story:
> As a data steward, I need a scalable way to quickly and efficiently propagate 
> classification across the information supply chain to support efficient 
> searches and classification based security for compliance and audit purposes. 
> This requires:
> 1. Classifications for derivative entities should be inherited from the 
> originator and to child entities from parent. 
> For example, if a Hive column is classified "Confidential" then resulting 
> column created from a CTAS operation should also be tagged "Confidential" to 
> maintain the classification of the original entity. In the case where 2 or 
> more entities are composed, the derivative entity should have the union of 
> all classifications of each source entity.
> 2. Business Terms:
> a. Child business terms should inherit the classifications associated with 
> the parent term.
> b. The option to propagate classification to child business terms in a 
> hierarchy should be provided
> c. Ability to update the propagated tags manually via UI or through the API
> d. Tagging a term should propagate to data assets that are already attached 
> to that business term as well
> 3. Data assets
> a. For all supported data asset types in Atlas, if a derivative asset is 
> created it should inherit the tags and attributes from the original asset.
> b. the option to propagate tags to child entities should be provided (e.g. if 
> you tag a folder in HDFS optionally tag all the files within it)
> c. Ability to update the propagated tags manually via UI or through the API
> d. Tagging a parent object should be inherited after child creation 
> dynamically (unless a flag is set not to do this)
> e. Derived data assets should have the tags of the original data asset.
> Conflict resolution - if there are different values for attributes on tags 
> (classifications) on upstream or parent entities used to derive a data asset 
> then user needs to be prompted for action to resolve the conflict. Once 
> resolved, the resolved value should be carried forth to derived assets.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] Graduate Apache Atlas Project from Incubator

[David Radley]

+1



From:   Suma Shivaprasad 
To: dev@atlas.incubator.apache.org
Date:   25/05/2017 00:23
Subject:[VOTE] Graduate Apache Atlas Project from Incubator



Dear Atlas Community Members,

Following the discussion in Atlas community on graduating Apache Atlas
Project from Incubation to a Top Level Project, starting this
voting thread within the Atlas community. I appreciate mentors and
community members in making this happen and thanks for their
support. Apache Atlas Project has come a long way since entering into
incubation during May 2015 and the project summary shows the
details on this.

Please vote on the Apache Atlas Project resolution that is found in
bottom of this email.

[ ] +1 Graduate Apache Atlas from the Incubator.
[ ] +0 No opinion
[ ] -1 Don't graduate Apache Atlas from the Incubator ( please provide
the reason)

This VOTE will be opened for next 72 hours.

If this VOTE succeeds, a similar VOTE will be done in general@incubator
mailing list. If that succeeds, a resolution will be
included in the next Apache Board Meeting.

Here is my Vote +1 (binding)

Thanks all for your participation
Suma


Project Summary:

=
http://incubator.apache.org/projects/atlas.html

Project website:
=

http://atlas.incubator.apache.org

Project Documentation:
===

http://atlas.incubator.apache.org/index.html
Architecture - http://atlas.incubator.apache.org/Architecture.html
Installation guide - http://atlas.incubator.apache.org/InstallationSteps.
html
QuickStart Guide- http://atlas.incubator.apache.org/QuickStart.html
REST API - http://atlas.incubator.apache.org/api/v2/index.html
Atlas Release Guide - https://cwiki.apache.org/confluence/display/ATLAS/
Developer+Resources

Project maturity Assessment:
===

https://cwiki.apache.org/confluence/display/ATLAS/
Apache+Atlas+Project+Maturity+Model

Proposed PMC size: 38

Total number of committers  : 38 members
Total number of contributors : 62 members

PMC affiliation (* indicated chair)

* Hortonworks (13)
   IBM (3)
   Aetna(4)
   Freestone (3)
   Merck (2)
   Oracle(2)
   Target (2)
   Schlumberger(2)
   Dataguise(1)
   Inmobi(1)
   JPMC(1)
   MPR(1)
   SAP(1)
   SparklineData(1)
   Others(1)

1660 commits on master
80 contributors across all branches
Dev list averaged ~500-800 msgs/month in 2017
User list was created recently in April 2017. Hence not considering stats
for this.
1827 issues created

Committer¹s affiliation:
===
* Hortonworks (13)
   IBM (3)
   Aetna(4)
   Freestone (3)
   Merck (2)
   Oracle(2)
   Target (2)
   Schlumberger(2)
   Dataguise(1)
   Inmobi(1)
   JPMC(1)
   MPR(1)
   SAP(1)
   SparklineData(1)
   Others(1)


Apache Atlas Top Level Project Resolution:


Establish the Apache Atlas Project

WHEREAS, the Board of Directors deems it to be in the best interests of
the Foundation and consistent with the Foundation¹s purpose to establish a
Project Management Committee charged with the creation and maintenance of
open-source software, for distribution at no charge to the public, related
to a data management platform That provides real-time, consistent access
to data-intensive applications throughout widely distributed cloud
architectures.

NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee
(PMC), to be known as the "Apache Atlas Project", be and hereby is
established pursuant to Bylaws of the Foundation; and be it further
RESOLVED,that the Apache Atlas Project be and hereby is responsible for
the creation and maintenance of software related to a data management
platform that provides real-time, consistent access to data-intensive
applications throughout widely distributed cloud architectures.

RESOLVED, that the office of "Vice President, Apache Atlas" be and
hereby is created, the person holding such office to serve at the
direction of the Board of Directors as the chair of the Apache Atlas
Project, and to have primary responsibility for management of the projects
within the scope of responsibility of the Apache Atlas Project; and be it
Further.

>RESOLVED,that the persons listed immediately below be and hereby are
>appointed to serve as the initial members of the Apache Atlas Project:

Aaron Dossett adoss...@target.com 
Andrew Ahn a...@apache.org 
Anil Varma ava...@slb.com 
Arun C. Murthy acmur...@apache.org 
Apoorv Naik apoorvn...@apache.org 
Ayub Pathan ayubpat...@apache.org 
Barbara Stortz bsto...@sap.com 
Chris Hyzer chy...@aetna.com 
Daniel Markwat dmark...@aetna.com 
Darshan Kumar darshanku...@apache.org 
Dave Kantor dkan...@apache.org 
David Kaspar dkas...@merck.com 

RE: [VOTE] Graduate Apache Atlas Project from Incubator

[Helvoirt, S.A.G. van (Stefhan)]

+1

Kind regards,
Stefhan

Van: Mandy Chessell
Verzonden: ‎25-‎5-‎2017 09:30
Aan: dev@atlas.incubator.apache.org
Onderwerp: Re: [VOTE] Graduate Apache Atlas Project from Incubator

+1

This is good news, thank you to the Atlas team for moving this forward.

All the best
Mandy
___
Mandy Chessell CBE FREng CEng FBCS
IBM Distinguished Engineer

Master Inventor
Member of the IBM Academy of Technology
Visiting Professor, Department of Computer Science, University of
Sheffield

Email: mandy_chess...@uk.ibm.com
LinkedIn: http://www.linkedin.com/pub/mandy-chessell/22/897/a49

Assistant: Janet Brooks - jsbrook...@uk.ibm.com



From:   Kalyani Kashikar 
To: dev@atlas.incubator.apache.org
Date:   25/05/2017 07:20
Subject:Re: [VOTE] Graduate Apache Atlas Project from Incubator



+1.

On Thu, May 25, 2017 at 11:40 AM, Keval Bhatt 
wrote:

> +1
>
> Happy to hear that Atlas is graduating.
>
> On Thu, May 25, 2017 at 10:57 AM, Hemanth Yamijala <
> hyamij...@hortonworks.com> wrote:
>
> > +1.
> > 
> > From: Suma Shivaprasad 
> > Sent: Thursday, May 25, 2017 4:53 AM
> > To: dev@atlas.incubator.apache.org
> > Subject: [VOTE] Graduate Apache Atlas Project from Incubator
> >
> > Dear Atlas Community Members,
> >
> > Following the discussion in Atlas community on graduating Apache Atlas
> > Project from Incubation to a Top Level Project, starting this
> > voting thread within the Atlas community. I appreciate mentors and
> > community members in making this happen and thanks for their
> > support. Apache Atlas Project has come a long way since entering into
> > incubation during May 2015 and the project summary shows the
> > details on this.
> >
> > Please vote on the Apache Atlas Project resolution that is found in
> > bottom of this email.
> >
> > [ ] +1 Graduate Apache Atlas from the Incubator.
> > [ ] +0 No opinion
> > [ ] -1 Don't graduate Apache Atlas from the Incubator ( please provide
> > the reason)
> >
> > This VOTE will be opened for next 72 hours.
> >
> > If this VOTE succeeds, a similar VOTE will be done in
general@incubator
> > mailing list. If that succeeds, a resolution will be
> > included in the next Apache Board Meeting.
> >
> > Here is my Vote +1 (binding)
> >
> > Thanks all for your participation
> > Suma
> >
> >
> > Project Summary:
> >
> > =
> > http://incubator.apache.org/projects/atlas.html
> >
> > Project website:
> > =
> >
> > http://atlas.incubator.apache.org
> >
> > Project Documentation:
> > ===
> >
> > http://atlas.incubator.apache.org/index.html
> > Architecture - http://atlas.incubator.apache.org/Architecture.html
> > Installation guide -
http://atlas.incubator.apache.org/InstallationSteps
> .
> > html
> > QuickStart Guide- http://atlas.incubator.apache.org/QuickStart.html
> > REST API - http://atlas.incubator.apache.org/api/v2/index.html
> > Atlas Release Guide -
https://cwiki.apache.org/confluence/display/ATLAS/
> > Developer+Resources
> >
> > Project maturity Assessment:
> > ===
> >
> > https://cwiki.apache.org/confluence/display/ATLAS/
> > Apache+Atlas+Project+Maturity+Model
> >
> > Proposed PMC size: 38
> >
> > Total number of committers  : 38 members
> > Total number of contributors : 62 members
> >
> > PMC affiliation (* indicated chair)
> >
> > * Hortonworks (13)
> >IBM (3)
> >Aetna(4)
> >Freestone (3)
> >Merck (2)
> >Oracle(2)
> >Target (2)
> >Schlumberger(2)
> >Dataguise(1)
> >Inmobi(1)
> >JPMC(1)
> >MPR(1)
> >SAP(1)
> >SparklineData(1)
> >Others(1)
> >
> > 1660 commits on master
> > 80 contributors across all branches
> > Dev list averaged ~500-800 msgs/month in 2017
> > User list was created recently in April 2017. Hence not considering
stats
> > for this.
> > 1827 issues created
> >
> > Committer¹s affiliation:
> > ===
> > * Hortonworks (13)
> >IBM (3)
> >Aetna(4)
> >Freestone (3)
> >Merck (2)
> >Oracle(2)
> >Target (2)
> >Schlumberger(2)
> >Dataguise(1)
> >Inmobi(1)
> >JPMC(1)
> >MPR(1)
> >SAP(1)
> >SparklineData(1)
> >Others(1)
> >
> >
> > Apache Atlas Top Level Project Resolution:
> > 
> >
> > Establish the Apache Atlas Project
> >
> > WHEREAS, the Board of Directors deems it to be in the best interests
of
> > the Foundation and consistent with the Foundation¹s purpose to
establish
> a
> > Project Management Committee charged with the creation and maintenance
of
> > open-source software, for distribution at no charge to the public,
> related
> > to a data management platform That provides real-time, consistent
access
> > to data-intensive applications throughout widely 

Re: [VOTE] Graduate Apache Atlas Project from Incubator

[Mandy Chessell]

+1

This is good news, thank you to the Atlas team for moving this forward.

All the best
Mandy
___
Mandy Chessell CBE FREng CEng FBCS
IBM Distinguished Engineer

Master Inventor
Member of the IBM Academy of Technology
Visiting Professor, Department of Computer Science, University of 
Sheffield

Email: mandy_chess...@uk.ibm.com
LinkedIn: http://www.linkedin.com/pub/mandy-chessell/22/897/a49

Assistant: Janet Brooks - jsbrook...@uk.ibm.com



From:   Kalyani Kashikar 
To: dev@atlas.incubator.apache.org
Date:   25/05/2017 07:20
Subject:Re: [VOTE] Graduate Apache Atlas Project from Incubator



+1.

On Thu, May 25, 2017 at 11:40 AM, Keval Bhatt 
wrote:

> +1
>
> Happy to hear that Atlas is graduating.
>
> On Thu, May 25, 2017 at 10:57 AM, Hemanth Yamijala <
> hyamij...@hortonworks.com> wrote:
>
> > +1.
> > 
> > From: Suma Shivaprasad 
> > Sent: Thursday, May 25, 2017 4:53 AM
> > To: dev@atlas.incubator.apache.org
> > Subject: [VOTE] Graduate Apache Atlas Project from Incubator
> >
> > Dear Atlas Community Members,
> >
> > Following the discussion in Atlas community on graduating Apache Atlas
> > Project from Incubation to a Top Level Project, starting this
> > voting thread within the Atlas community. I appreciate mentors and
> > community members in making this happen and thanks for their
> > support. Apache Atlas Project has come a long way since entering into
> > incubation during May 2015 and the project summary shows the
> > details on this.
> >
> > Please vote on the Apache Atlas Project resolution that is found in
> > bottom of this email.
> >
> > [ ] +1 Graduate Apache Atlas from the Incubator.
> > [ ] +0 No opinion
> > [ ] -1 Don't graduate Apache Atlas from the Incubator ( please provide
> > the reason)
> >
> > This VOTE will be opened for next 72 hours.
> >
> > If this VOTE succeeds, a similar VOTE will be done in 
general@incubator
> > mailing list. If that succeeds, a resolution will be
> > included in the next Apache Board Meeting.
> >
> > Here is my Vote +1 (binding)
> >
> > Thanks all for your participation
> > Suma
> >
> >
> > Project Summary:
> >
> > =
> > http://incubator.apache.org/projects/atlas.html
> >
> > Project website:
> > =
> >
> > http://atlas.incubator.apache.org
> >
> > Project Documentation:
> > ===
> >
> > http://atlas.incubator.apache.org/index.html
> > Architecture - http://atlas.incubator.apache.org/Architecture.html
> > Installation guide - 
http://atlas.incubator.apache.org/InstallationSteps
> .
> > html
> > QuickStart Guide- http://atlas.incubator.apache.org/QuickStart.html
> > REST API - http://atlas.incubator.apache.org/api/v2/index.html
> > Atlas Release Guide - 
https://cwiki.apache.org/confluence/display/ATLAS/
> > Developer+Resources
> >
> > Project maturity Assessment:
> > ===
> >
> > https://cwiki.apache.org/confluence/display/ATLAS/
> > Apache+Atlas+Project+Maturity+Model
> >
> > Proposed PMC size: 38
> >
> > Total number of committers  : 38 members
> > Total number of contributors : 62 members
> >
> > PMC affiliation (* indicated chair)
> >
> > * Hortonworks (13)
> >IBM (3)
> >Aetna(4)
> >Freestone (3)
> >Merck (2)
> >Oracle(2)
> >Target (2)
> >Schlumberger(2)
> >Dataguise(1)
> >Inmobi(1)
> >JPMC(1)
> >MPR(1)
> >SAP(1)
> >SparklineData(1)
> >Others(1)
> >
> > 1660 commits on master
> > 80 contributors across all branches
> > Dev list averaged ~500-800 msgs/month in 2017
> > User list was created recently in April 2017. Hence not considering 
stats
> > for this.
> > 1827 issues created
> >
> > Committer¹s affiliation:
> > ===
> > * Hortonworks (13)
> >IBM (3)
> >Aetna(4)
> >Freestone (3)
> >Merck (2)
> >Oracle(2)
> >Target (2)
> >Schlumberger(2)
> >Dataguise(1)
> >Inmobi(1)
> >JPMC(1)
> >MPR(1)
> >SAP(1)
> >SparklineData(1)
> >Others(1)
> >
> >
> > Apache Atlas Top Level Project Resolution:
> > 
> >
> > Establish the Apache Atlas Project
> >
> > WHEREAS, the Board of Directors deems it to be in the best interests 
of
> > the Foundation and consistent with the Foundation¹s purpose to 
establish
> a
> > Project Management Committee charged with the creation and maintenance 
of
> > open-source software, for distribution at no charge to the public,
> related
> > to a data management platform That provides real-time, consistent 
access
> > to data-intensive applications throughout widely distributed cloud
> > architectures.
> >
> > NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee
> > (PMC), to be known as the "Apache Atlas Project", be and hereby is
> > established pursuant to Bylaws of the Foundation; and be it further
> > RESOLVED,that the 

Re: [VOTE] Graduate Apache Atlas Project from Incubator

[Keval Bhatt]

+1

Happy to hear that Atlas is graduating.

On Thu, May 25, 2017 at 10:57 AM, Hemanth Yamijala <
hyamij...@hortonworks.com> wrote:

> +1.
> 
> From: Suma Shivaprasad 
> Sent: Thursday, May 25, 2017 4:53 AM
> To: dev@atlas.incubator.apache.org
> Subject: [VOTE] Graduate Apache Atlas Project from Incubator
>
> Dear Atlas Community Members,
>
> Following the discussion in Atlas community on graduating Apache Atlas
> Project from Incubation to a Top Level Project, starting this
> voting thread within the Atlas community. I appreciate mentors and
> community members in making this happen and thanks for their
> support. Apache Atlas Project has come a long way since entering into
> incubation during May 2015 and the project summary shows the
> details on this.
>
> Please vote on the Apache Atlas Project resolution that is found in
> bottom of this email.
>
> [ ] +1 Graduate Apache Atlas from the Incubator.
> [ ] +0 No opinion
> [ ] -1 Don't graduate Apache Atlas from the Incubator ( please provide
> the reason)
>
> This VOTE will be opened for next 72 hours.
>
> If this VOTE succeeds, a similar VOTE will be done in general@incubator
> mailing list. If that succeeds, a resolution will be
> included in the next Apache Board Meeting.
>
> Here is my Vote +1 (binding)
>
> Thanks all for your participation
> Suma
>
>
> Project Summary:
>
> =
> http://incubator.apache.org/projects/atlas.html
>
> Project website:
> =
>
> http://atlas.incubator.apache.org
>
> Project Documentation:
> ===
>
> http://atlas.incubator.apache.org/index.html
> Architecture - http://atlas.incubator.apache.org/Architecture.html
> Installation guide - http://atlas.incubator.apache.org/InstallationSteps.
> html
> QuickStart Guide- http://atlas.incubator.apache.org/QuickStart.html
> REST API - http://atlas.incubator.apache.org/api/v2/index.html
> Atlas Release Guide - https://cwiki.apache.org/confluence/display/ATLAS/
> Developer+Resources
>
> Project maturity Assessment:
> ===
>
> https://cwiki.apache.org/confluence/display/ATLAS/
> Apache+Atlas+Project+Maturity+Model
>
> Proposed PMC size: 38
>
> Total number of committers  : 38 members
> Total number of contributors : 62 members
>
> PMC affiliation (* indicated chair)
>
> * Hortonworks (13)
>IBM (3)
>Aetna(4)
>Freestone (3)
>Merck (2)
>Oracle(2)
>Target (2)
>Schlumberger(2)
>Dataguise(1)
>Inmobi(1)
>JPMC(1)
>MPR(1)
>SAP(1)
>SparklineData(1)
>Others(1)
>
> 1660 commits on master
> 80 contributors across all branches
> Dev list averaged ~500-800 msgs/month in 2017
> User list was created recently in April 2017. Hence not considering stats
> for this.
> 1827 issues created
>
> Committer¹s affiliation:
> ===
> * Hortonworks (13)
>IBM (3)
>Aetna(4)
>Freestone (3)
>Merck (2)
>Oracle(2)
>Target (2)
>Schlumberger(2)
>Dataguise(1)
>Inmobi(1)
>JPMC(1)
>MPR(1)
>SAP(1)
>SparklineData(1)
>Others(1)
>
>
> Apache Atlas Top Level Project Resolution:
> 
>
> Establish the Apache Atlas Project
>
> WHEREAS, the Board of Directors deems it to be in the best interests of
> the Foundation and consistent with the Foundation¹s purpose to establish a
> Project Management Committee charged with the creation and maintenance of
> open-source software, for distribution at no charge to the public, related
> to a data management platform That provides real-time, consistent access
> to data-intensive applications throughout widely distributed cloud
> architectures.
>
> NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee
> (PMC), to be known as the "Apache Atlas Project", be and hereby is
> established pursuant to Bylaws of the Foundation; and be it further
> RESOLVED,that the Apache Atlas Project be and hereby is responsible for
> the creation and maintenance of software related to a data management
> platform that provides real-time, consistent access to data-intensive
> applications throughout widely distributed cloud architectures.
>
> RESOLVED, that the office of "Vice President, Apache Atlas" be and
> hereby is created, the person holding such office to serve at the
> direction of the Board of Directors as the chair of the Apache Atlas
> Project, and to have primary responsibility for management of the projects
> within the scope of responsibility of the Apache Atlas Project; and be it
> Further.
>
> >RESOLVED,that the persons listed immediately below be and hereby are
> >appointed to serve as the initial members of the Apache Atlas Project:
>
> Aaron Dossett adoss...@target.com 
> Andrew Ahn a...@apache.org 
> Anil Varma ava...@slb.com 
> Arun C. Murthy acmur...@apache.org 
> Apoorv Naik apoorvn...@apache.org