date:20170720

Re: Review Request 60980: ATLAS-1961: Basic search improvement in use of Solr index for attribute filtering (# 3)

2017-07-20 Thread Madhan Neethiraj



> On July 21, 2017, 3:12 a.m., Apoorv Naik wrote:
> > repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
> > Line 56 (original), 56 (patched)
> > 
> >
> > Should we add all solr reserved characters here ?
> > 
> > + - && || ! ( ) { } [ ] ^ " ~ * ? : /
> > 
> > Otherwise overall it looks good.

For these special characters (or keywords - && ||), Solr recommends to escape 
with a backslash. However, we use double-quotes when value includes one of "@/ 
" - which serves well for our searches. This needs to be investigated further. 
I would suggest we take this up with Solr experts (mailing lists?).


- Madhan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60980/#review181094
---


On July 21, 2017, 1:39 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60980/
> ---
> 
> (Updated July 21, 2017, 1:39 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-1961
> https://issues.apache.org/jira/browse/ATLAS-1961
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> - fixed NPE when search includes a free-text query & a classification that 
> has large number of sub-types
> - fixed entity search in Solr to include state filter
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java
>  a3525c93 
>   
> repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java
>  1b19a0e2 
>   repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
> 7950127d 
> 
> 
> Diff: https://reviews.apache.org/r/60980/diff/4/
> 
> 
> Testing
> ---
> 
> - validated that the fix addressed the error cases listed above
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

[jira] [Resolved] (ATLAS-1970) Export/Import - When updateTypeDefinition set to false , new types are not imported

2017-07-20 Thread Sharmadha Sainath (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sharmadha Sainath resolved ATLAS-1970.
--
Resolution: Not A Bug

> Export/Import - When updateTypeDefinition set to false , new types are not 
> imported
> ---
>
> Key: ATLAS-1970
> URL: https://issues.apache.org/jira/browse/ATLAS-1970
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 0.9-incubating, 0.8.1-incubating
>Reporter: Sharmadha Sainath
>Assignee: Sharmadha Sainath
>Priority: Critical
> Attachments: ImportFailureDueToUnknownType.txt
>
>
> Import has option "updateTypeDefinition" which is used to update the type 
> definitions in the backup cluster (cluster on which import is done) when the 
> value is set to true. (Default is set to true). When its value is set to 
> false , types in backup cluster are not updated with types present in 
> exported zip file.
> This works fine when , say a type type1 is present in both clusters , an 
> entity of type type1 is exported and imported into backup cluster with 
> updateTypeDefinition is set to false - Import is done successfully and type 
> is not updated.
> When the zip file contains type5 *which is not present in backup cluster* and 
> the when import is fired , import fails with following exception:
> {code}
> {"errorCode":"ATLAS-500-00-001","errorMessage":"org.apache.atlas.exception.AtlasBaseException:
>  Type ENTITY with name type5 does not exist"}
> {code}
> Attached the complete exception stack trace found in backup cluster's 
> application logs. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ATLAS-1900) Test NotificationHookConsumerIT#testUpdateEntityPartial fails intermittently

2017-07-20 Thread Sneha Kanekar (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095731#comment-16095731
 ] 

Sneha Kanekar commented on ATLAS-1900:
--

Any update on this?

> Test NotificationHookConsumerIT#testUpdateEntityPartial fails intermittently
> 
>
> Key: ATLAS-1900
> URL: https://issues.apache.org/jira/browse/ATLAS-1900
> Project: Atlas
>  Issue Type: Bug
>  Components: atlas-webui
>Affects Versions: 0.9-incubating
> Environment: Ubuntu:14.04
> $ java -version
> openjdk version "1.8.0_111"
> OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-3~14.04.1-b14)
> OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)
>Reporter: Sneha Kanekar
>  Labels: ppc64le, x86
> Attachments: application.log
>
>
> The integration test 
> org.apache.atlas.notification.NotificationHookConsumerIT.testUpdateEntityPartial
>  fails intermittently with Internal server error 500. This test fails on x86 
> and ppc64le platform. 
> {code:borderStyle=solid}
> testUpdateEntityPartial(org.apache.atlas.notification.NotificationHookConsumerIT)
>   Time elapsed: 0.646 sec  <<< FAILURE!
> org.apache.atlas.AtlasServiceException: Metadata service API 
> org.apache.atlas.AtlasBaseClient$APIInfo@3c2809c9 failed with status 500 
> (Internal Server Error) Response Body ({"error":"Could not execute operation 
> due to backend exception"})
> at 
> org.apache.atlas.notification.NotificationHookConsumerIT.testUpdateEntityPartial(NotificationHookConsumerIT.java:127)
> {code}
> The full error message from standard output is as follows:
> {code:borderStyle=solid}
> Jun 27, 2017 5:49:29 AM com.sun.jersey.spi.container.ContainerResponse 
> logException
> SEVERE: Mapped exception to response: 500 (Internal Server Error)
> javax.ws.rs.WebApplicationException
> at 
> org.apache.atlas.web.resources.EntityResource.getEntityDefinitionByAttribute(EntityResource.java:838)
> at 
> org.apache.atlas.web.resources.EntityResource.getEntity(EntityResource.java:764)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at 
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
> at 
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at 
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> at 
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at 
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at 
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
> at 
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
> at 
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
> at 
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
> at 
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
> at 
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
> at 
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at 
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
> at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
> at 
> org.apache.atlas.web.filters.AuditFilter.doFilter(AuditFilter.java:76)
> at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at 
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
> at 
> org.apache.atlas.web.filters.AtlasAuthorizationFilter.doFilter(AtlasAuthorizationFilter.java:157)
> at 
> org.springframework.security.web.FilterChainProxy$VirtualFi

Re: Review Request 60980: ATLAS-1961: Basic search improvement in use of Solr index for attribute filtering (# 3)

2017-07-20 Thread Apoorv Naik


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60980/#review181094
---




repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
Line 56 (original), 56 (patched)


Should we add all solr reserved characters here ?

+ - && || ! ( ) { } [ ] ^ " ~ * ? : /

Otherwise overall it looks good.


- Apoorv Naik


On July 21, 2017, 1:39 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60980/
> ---
> 
> (Updated July 21, 2017, 1:39 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-1961
> https://issues.apache.org/jira/browse/ATLAS-1961
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> - fixed NPE when search includes a free-text query & a classification that 
> has large number of sub-types
> - fixed entity search in Solr to include state filter
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java
>  a3525c93 
>   
> repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java
>  1b19a0e2 
>   repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
> 7950127d 
> 
> 
> Diff: https://reviews.apache.org/r/60980/diff/4/
> 
> 
> Testing
> ---
> 
> - validated that the fix addressed the error cases listed above
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Review Request 61007: [ATLAS-1979]: Update storm model relationship category and fix for UT and Coverity scan issues

2017-07-20 Thread Madhan Neethiraj


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61007/#review181089
---


Ship it!




Ship It!

- Madhan Neethiraj


On July 20, 2017, 9:16 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61007/
> ---
> 
> (Updated July 20, 2017, 9:16 p.m.)
> 
> 
> Review request for atlas and Madhan Neethiraj.
> 
> 
> Bugs: ATLAS-1979
> https://issues.apache.org/jira/browse/ATLAS-1979
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Update storm_topology_nodes relationshipCategory from ASSOCIATION to 
> AGGREGATION
> Fix the following UT failures:
> _Failed tests: 
> AtlasRelationshipStoreHardDeleteV1Test>AtlasRelationshipStoreV1Test.testDepartmentEmployeeEntitiesUsingRelationship:172
>  expected: but was:<[]>
> AtlasRelationshipStoreSoftDeleteV1Test>AtlasRelationshipStoreV1Test.testDepartmentEmployeeEntitiesUsingRelationship:172
>  expected: but was:<[]>_
> Fix coverity scan issues flagged in last report related to ATLAS-1959
> 
> 
> Diffs
> -
> 
>   addons/models/0080-storm_model.json b008c7ab 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java
>  1282be5e 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateSoftDeleteTest.java
>  c5eda370 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java
>  31efe868 
> 
> 
> Diff: https://reviews.apache.org/r/61007/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean package succeeded
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

Re: Review Request 60980: ATLAS-1961: Basic search improvement in use of Solr index for attribute filtering (# 3)

2017-07-20 Thread Madhan Neethiraj


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60980/
---

(Updated July 21, 2017, 1:39 a.m.)


Review request for atlas.


Changes
---

updates to handle spaces in tag-names, while searcing in Solr index


Bugs: ATLAS-1961
https://issues.apache.org/jira/browse/ATLAS-1961


Repository: atlas


Description
---

- fixed NPE when search includes a free-text query & a classification that has 
large number of sub-types
- fixed entity search in Solr to include state filter


Diffs (updated)
-

  
repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 
a3525c93 
  
repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java
 1b19a0e2 
  repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
7950127d 


Diff: https://reviews.apache.org/r/60980/diff/4/

Changes: https://reviews.apache.org/r/60980/diff/3-4/


Testing
---

- validated that the fix addressed the error cases listed above


Thanks,

Madhan Neethiraj

[jira] [Assigned] (ATLAS-1960) Import command fired on passive server throws Exception

2017-07-20 Thread Ashutosh Mestry (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry reassigned ATLAS-1960:
--

Assignee: Ashutosh Mestry

> Import command fired on passive server throws Exception
> ---
>
> Key: ATLAS-1960
> URL: https://issues.apache.org/jira/browse/ATLAS-1960
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Sharmadha Sainath
>Assignee: Ashutosh Mestry
>Priority: Critical
> Attachments: ImportOnPassiveHost.txt
>
>
> 1.Fired import command on an ACTIVE host which resulted in successful import.
> 2. On firing import command  with a zip file containing exported items of a 
> hive_table ,on PASSIVE host, following exception is thrown :
> {code}
> {"errorCode":"ATLAS-500-00-001","errorMessage":"org.apache.atlas.exception.AtlasBaseException:
>  org.apache.atlas.typesystem.exception.TypeNotFoundException: Unknown 
> datatype: hive_table"}
> {code}
> Expected 30X with redirection URL.
> Attached the complete exception stack trace found in PASSIVE host's 
> application logs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (ATLAS-1889) Failure in simultaneous updates to an entity tag association

2017-07-20 Thread Ashutosh Mestry (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry closed ATLAS-1889.
--

> Failure in simultaneous updates to an entity tag association
> 
>
> Key: ATLAS-1889
> URL: https://issues.apache.org/jira/browse/ATLAS-1889
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 0.8-incubating
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
> Fix For: trunk, 0.8.1-incubating
>
> Attachments: ATLAS-1889-Handling-multiple-entities.patch
>
>
> *Background*
> When multiple requests simultaneously attempt to change the tags associated 
> with an entity, the current implementation might result in APIs returning 
> fewer tags for the entity than is actually associated.
> The problem thus causes UI not to behave correctly.
>  
> *Solution*
> Implements a synchronization mechanism.
>  
> Steps to duplicate the problem:
> * Create several tags. Eg. tag1, tag2, tag3, tag4.
> * Attach these tags to and entity say ‘testtable1’
> * Use the following CURL commands to delete the tags:
> {code}
> curl -X DELETE -u admin:admin 
> 'http://localhost:21000/api/atlas/entities/a2d0b023-aafe-4ddd-bb90-65164af8796c/traits/tag1'
>   -H 'Content-Type: application/json' &
> curl -X DELETE -u admin:admin 
> 'http://localhost:21000/api/atlas/entities/a2d0b023-aafe-4ddd-bb90-65164af8796c/traits/tag2'
>   -H 'Content-Type: application/json' &
> curl -X DELETE -u admin:admin 
> 'http://localhost:21000/api/atlas/entities/a2d0b023-aafe-4ddd-bb90-65164af8796c/traits/tag3'
>   -H 'Content-Type: application/json' &
> curl -X DELETE -u admin:admin 
> 'http://localhost:21000/api/atlas/entities/a2d0b023-aafe-4ddd-bb90-65164af8796c/traits/tag4'
>   -H 'Content-Type: application/json'
> {code}
> You will notice that entity _testtable1_ ends up with some tags not deleted.
> Additional querying with gremlin will show that the edges are deleted 
> correctly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (ATLAS-1734) Import API: Add Support to Update Attributes of Existing Types During Import

2017-07-20 Thread Ashutosh Mestry (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry closed ATLAS-1734.
--

> Import API: Add Support to Update Attributes of Existing Types During Import
> 
>
> Key: ATLAS-1734
> URL: https://issues.apache.org/jira/browse/ATLAS-1734
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: trunk, 0.8-incubating
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>  Labels: patch
> Fix For: trunk
>
> Attachments: 
> ATLAS-1734-Import-with-additional-attribute-processi.patch
>
>
> *Background*
> Existing version of Import API allows for importing types that are not 
> already present in the system being imported in. This causes import to fail 
> in the cases where the data being imported happens to have the additional 
> attribute.
> *Solution*
> During import, existing types are checked to determine if the types being 
> imported have additional attributes. If additional attributes exist, then the 
> existing type is updated with the new attributes. The import then proceeds.
> *Impact Assessment*
> - Import API: 
>   -- Type import: Additional capability (mentioned above).
>   -- Entity creation and processing: No impact.
> - Export API: No impact.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Review Request 61007: [ATLAS-1979]: Update storm model relationship category and fix for UT and Coverity scan issues

2017-07-20 Thread Sarath Subramanian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61007/
---

Review request for atlas and Madhan Neethiraj.


Bugs: ATLAS-1979
https://issues.apache.org/jira/browse/ATLAS-1979


Repository: atlas


Description
---

Update storm_topology_nodes relationshipCategory from ASSOCIATION to AGGREGATION
Fix the following UT failures:
_Failed tests: 
AtlasRelationshipStoreHardDeleteV1Test>AtlasRelationshipStoreV1Test.testDepartmentEmployeeEntitiesUsingRelationship:172
 expected: but was:<[]>
AtlasRelationshipStoreSoftDeleteV1Test>AtlasRelationshipStoreV1Test.testDepartmentEmployeeEntitiesUsingRelationship:172
 expected: but was:<[]>_
Fix coverity scan issues flagged in last report related to ATLAS-1959


Diffs
-

  addons/models/0080-storm_model.json b008c7ab 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java
 1282be5e 
  
repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateSoftDeleteTest.java
 c5eda370 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java
 31efe868 


Diff: https://reviews.apache.org/r/61007/diff/1/


Testing
---

mvn clean package succeeded


Thanks,

Sarath Subramanian

[jira] [Updated] (ATLAS-1979) Update storm model relationship category and fix for UT and Coverity scan issues

2017-07-20 Thread Sarath Subramanian (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-1979:
--
Attachment: ATLAS-1979.1.patch

> Update storm model relationship category and fix for UT and Coverity scan 
> issues
> 
>
> Key: ATLAS-1979
> URL: https://issues.apache.org/jira/browse/ATLAS-1979
> Project: Atlas
>  Issue Type: Bug
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
> Attachments: ATLAS-1979.1.patch
>
>
> * Update storm_topology_nodes relationshipCategory from ASSOCIATION to 
> AGGREGATION
> * Fix the following UT failures:
> _Failed tests: 
>   
> AtlasRelationshipStoreHardDeleteV1Test>AtlasRelationshipStoreV1Test.testDepartmentEmployeeEntitiesUsingRelationship:172
>  expected: but was:<[]>
>   
> AtlasRelationshipStoreSoftDeleteV1Test>AtlasRelationshipStoreV1Test.testDepartmentEmployeeEntitiesUsingRelationship:172
>  expected: but was:<[]>_
> * Fix coverity scan issues flagged in last report related to ATLAS-1959



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 60980: ATLAS-1961: Basic search improvement in use of Solr index for attribute filtering (# 3)

2017-07-20 Thread Apoorv Naik


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60980/#review181076
---


Ship it!




Looks good to me

- Apoorv Naik


On July 20, 2017, 8:31 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60980/
> ---
> 
> (Updated July 20, 2017, 8:31 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-1961
> https://issues.apache.org/jira/browse/ATLAS-1961
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> - fixed NPE when search includes a free-text query & a classification that 
> has large number of sub-types
> - fixed entity search in Solr to include state filter
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java
>  b6e0de5c 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java
>  6f629eb4 
>   
> repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java
>  22d91e04 
>   repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
> 2e75dfeb 
> 
> 
> Diff: https://reviews.apache.org/r/60980/diff/3/
> 
> 
> Testing
> ---
> 
> - validated that the fix addressed the error cases listed above
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

[jira] [Updated] (ATLAS-1979) Update storm model relationship category and fix for UT and Coverity scan issues

2017-07-20 Thread Sarath Subramanian (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-1979:
--
Description: 
* Update storm_topology_nodes relationshipCategory from ASSOCIATION to 
AGGREGATION
* Fix the following UT failures:
_Failed tests: 
  
AtlasRelationshipStoreHardDeleteV1Test>AtlasRelationshipStoreV1Test.testDepartmentEmployeeEntitiesUsingRelationship:172
 expected: but was:<[]>
  
AtlasRelationshipStoreSoftDeleteV1Test>AtlasRelationshipStoreV1Test.testDepartmentEmployeeEntitiesUsingRelationship:172
 expected: but was:<[]>_

* Fix coverity scan issues flagged in last report related to ATLAS-1959

  was:Update storm model


> Update storm model relationship category and fix for UT and Coverity scan 
> issues
> 
>
> Key: ATLAS-1979
> URL: https://issues.apache.org/jira/browse/ATLAS-1979
> Project: Atlas
>  Issue Type: Bug
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>
> * Update storm_topology_nodes relationshipCategory from ASSOCIATION to 
> AGGREGATION
> * Fix the following UT failures:
> _Failed tests: 
>   
> AtlasRelationshipStoreHardDeleteV1Test>AtlasRelationshipStoreV1Test.testDepartmentEmployeeEntitiesUsingRelationship:172
>  expected: but was:<[]>
>   
> AtlasRelationshipStoreSoftDeleteV1Test>AtlasRelationshipStoreV1Test.testDepartmentEmployeeEntitiesUsingRelationship:172
>  expected: but was:<[]>_
> * Fix coverity scan issues flagged in last report related to ATLAS-1959



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (ATLAS-1979) Update storm model relationship category and fix for UT and Coverity scan issues

2017-07-20 Thread Sarath Subramanian (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-1979:
--
Description: Update storm model

> Update storm model relationship category and fix for UT and Coverity scan 
> issues
> 
>
> Key: ATLAS-1979
> URL: https://issues.apache.org/jira/browse/ATLAS-1979
> Project: Atlas
>  Issue Type: Bug
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>
> Update storm model



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (ATLAS-1979) Update storm model relationship category and fix for UT and Coverity scan issues

2017-07-20 Thread Sarath Subramanian (JIRA)

Sarath Subramanian created ATLAS-1979:
-

 Summary: Update storm model relationship category and fix for UT 
and Coverity scan issues
 Key: ATLAS-1979
 URL: https://issues.apache.org/jira/browse/ATLAS-1979
 Project: Atlas
  Issue Type: Bug
Reporter: Sarath Subramanian
Assignee: Sarath Subramanian






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (ATLAS-1839) Area 2 of the open metadata model

2017-07-20 Thread David Radley (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-1839:

Attachment: (was: 0230Terms.json)

> Area 2 of the open metadata model
> -
>
> Key: ATLAS-1839
> URL: https://issues.apache.org/jira/browse/ATLAS-1839
> Project: Atlas
>  Issue Type: Task
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: David Radley
>  Labels: OpenMetadata, VirtualDataConnector
> Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, 
> 0220CategoryHierarchy.json, 0230Terms.json, 0240Dictionary.json, 
> 0250RelatedTerms.json, 0260Contexts.json, 0270SemanticAssignment.json, 
> 0280SpineObjects.json
>
>
> This task delivers the JSON files for the new models that describe types for 
> Area 2 in the open metadata model. This area covers the glossary.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (ATLAS-1839) Area 2 of the open metadata model

2017-07-20 Thread David Radley (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-1839:

Attachment: 0230Terms.json

> Area 2 of the open metadata model
> -
>
> Key: ATLAS-1839
> URL: https://issues.apache.org/jira/browse/ATLAS-1839
> Project: Atlas
>  Issue Type: Task
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: David Radley
>  Labels: OpenMetadata, VirtualDataConnector
> Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, 
> 0220CategoryHierarchy.json, 0230Terms.json, 0240Dictionary.json, 
> 0250RelatedTerms.json, 0260Contexts.json, 0270SemanticAssignment.json, 
> 0280SpineObjects.json
>
>
> This task delivers the JSON files for the new models that describe types for 
> Area 2 in the open metadata model. This area covers the glossary.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 60938: [ATLAS-1959]: Enhance relationship attributes to support different cardinality mappings

2017-07-20 Thread Sarath Subramanian



> On July 19, 2017, 8:11 a.m., David Radley wrote:
> > addons/models/0080-storm_model.json
> > Line 150 (original), 150 (patched)
> > 
> >
> > shouldn't this be aggregation ?
> 
> Sarath Subramanian wrote:
> since a storm node can be shared acoss multiple storm topologies and not 
> tied to a single topology made it as ASSOCIATION
> 
> David Radley wrote:
> It seems better to have this as aggregation to should some element of 
> containment. From your description , the topology contains the storm node. 
> The storm node could also be contained in otehr topologies . If htis is the 
> case - it should be AGGREGATION I think.

will change to AGGREGATION


- Sarath


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60938/#review180926
---


On July 19, 2017, 12:27 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60938/
> ---
> 
> (Updated July 19, 2017, 12:27 p.m.)
> 
> 
> Review request for atlas and Madhan Neethiraj.
> 
> 
> Bugs: ATLAS-1959
> https://issues.apache.org/jira/browse/ATLAS-1959
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> * Improve relationship model to support create/update operations and the 
> following cardinalities (previously supported using inverseReference):
> 1 to 1
> 1 to many
> many to 1
> many to many
> 
> 
> * Change legacyLabel flag in AtlasRelationshipEndDef to boolean flag.
> * Add unit tests for the above cases.
> 
> 
> Diffs
> -
> 
>   addons/models/0010-base_model.json 303f3796 
>   addons/models/0030-hive_model.json a795f0f3 
>   addons/models/0050-falcon_model.json 7755fa86 
>   addons/models/0060-hbase_model.json 1d264df4 
>   addons/models/0080-storm_model.json 25360ff0 
>   intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java 
> 68da6af1 
>   intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationship.java 
> 2de9bdf0 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasRelationshipDef.java 
> fc820d49 
>   
> intg/src/main/java/org/apache/atlas/model/typedef/AtlasRelationshipEndDef.java
>  f80ea895 
>   intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java e94dd190 
>   intg/src/main/java/org/apache/atlas/type/AtlasRelationshipType.java 
> 841b66f7 
>   intg/src/main/java/org/apache/atlas/type/AtlasStructType.java f97d7674 
>   intg/src/test/java/org/apache/atlas/TestRelationshipUtilsV2.java 
> PRE-CREATION 
>   intg/src/test/java/org/apache/atlas/TestUtilsV2.java 9774583d 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 6f6d74bc 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityGraphDiscoveryV1.java
>  12e8bb1f 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasGraphUtilsV1.java
>  cd9a47ad 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1.java
>  3ff6fbef 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  d4fdc257 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java
>  157f8cd2 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphRetriever.java
>  f4257be7 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
>  404225c2 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java
>  d9017319 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreHardDeleteV1Test.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreSoftDeleteV1Test.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java
>  67702231 
> 
> 
> Diff: https://reviews.apache.org/r/60938/diff/4/
> 
> 
> Testing
> ---
> 
> added unit test - AtlasRelationshipStoreV1Test
> 
> mvn clean package - succeeded with no errors.
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

[jira] [Updated] (ATLAS-1839) Area 2 of the open metadata model

2017-07-20 Thread David Radley (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-1839:

Attachment: (was: 0220CategoryHierarchy.json)

> Area 2 of the open metadata model
> -
>
> Key: ATLAS-1839
> URL: https://issues.apache.org/jira/browse/ATLAS-1839
> Project: Atlas
>  Issue Type: Task
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: David Radley
>  Labels: OpenMetadata, VirtualDataConnector
> Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, 
> 0220CategoryHierarchy.json, 0230Terms.json, 0240Dictionary.json, 
> 0250RelatedTerms.json, 0260Contexts.json, 0270SemanticAssignment.json, 
> 0280SpineObjects.json
>
>
> This task delivers the JSON files for the new models that describe types for 
> Area 2 in the open metadata model. This area covers the glossary.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (ATLAS-1839) Area 2 of the open metadata model

2017-07-20 Thread David Radley (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-1839:

Attachment: 0220CategoryHierarchy.json

> Area 2 of the open metadata model
> -
>
> Key: ATLAS-1839
> URL: https://issues.apache.org/jira/browse/ATLAS-1839
> Project: Atlas
>  Issue Type: Task
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: David Radley
>  Labels: OpenMetadata, VirtualDataConnector
> Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, 
> 0220CategoryHierarchy.json, 0230Terms.json, 0240Dictionary.json, 
> 0250RelatedTerms.json, 0260Contexts.json, 0270SemanticAssignment.json, 
> 0280SpineObjects.json
>
>
> This task delivers the JSON files for the new models that describe types for 
> Area 2 in the open metadata model. This area covers the glossary.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ATLAS-1970) Export/Import - When updateTypeDefinition set to false , new types are not imported

2017-07-20 Thread Ashutosh Mestry (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-1970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095085#comment-16095085
 ] 

Ashutosh Mestry commented on ATLAS-1970:


I think this is expected behavior.

> Export/Import - When updateTypeDefinition set to false , new types are not 
> imported
> ---
>
> Key: ATLAS-1970
> URL: https://issues.apache.org/jira/browse/ATLAS-1970
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 0.9-incubating, 0.8.1-incubating
>Reporter: Sharmadha Sainath
>Assignee: Sharmadha Sainath
>Priority: Critical
> Attachments: ImportFailureDueToUnknownType.txt
>
>
> Import has option "updateTypeDefinition" which is used to update the type 
> definitions in the backup cluster (cluster on which import is done) when the 
> value is set to true. (Default is set to true). When its value is set to 
> false , types in backup cluster are not updated with types present in 
> exported zip file.
> This works fine when , say a type type1 is present in both clusters , an 
> entity of type type1 is exported and imported into backup cluster with 
> updateTypeDefinition is set to false - Import is done successfully and type 
> is not updated.
> When the zip file contains type5 *which is not present in backup cluster* and 
> the when import is fired , import fails with following exception:
> {code}
> {"errorCode":"ATLAS-500-00-001","errorMessage":"org.apache.atlas.exception.AtlasBaseException:
>  Type ENTITY with name type5 does not exist"}
> {code}
> Attached the complete exception stack trace found in backup cluster's 
> application logs. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 61002: ATLAS-1968: Import File Signature Change

2017-07-20 Thread Madhan Neethiraj


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61002/#review181062
---


Ship it!




Ship It!

- Madhan Neethiraj


On July 20, 2017, 5:22 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61002/
> ---
> 
> (Updated July 20, 2017, 5:22 p.m.)
> 
> 
> Review request for atlas and Madhan Neethiraj.
> 
> 
> Bugs: ATLAS-1968
> https://issues.apache.org/jira/browse/ATLAS-1968
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> The Import API that takes file as input was changed to make it consistent 
> with the other API that takes ZIP stream as input.
> 
> This change did not correctly reflect within _ImportService_.
> 
> **Solution**
> _AtlasImportRequest_ has been updated to add the _fileName_ option.
> 
> **CURL**
> 
> Note that the file path should be a path on the server.
> 
> _importOptions.json_ contents:
> ```javascript
> {
> "options": {
> "fileName": "/root/smalldb.zip"
> }
> }
> ```
> 
> CURL
> ```bash
> curl -g -X POST -u admin:admin -H "Content-Type: application/json" -H 
> "Cache-Control: no-cache" -d @./importOptions.json 
> "http://localhost:21000/api/atlas/admin/importFile";
> ```
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/Import-API-Options.twiki 85887844 
>   docs/src/site/twiki/Import-API.twiki b5de113f 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
> 4d2ac627 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
>  92217178 
> 
> 
> Diff: https://reviews.apache.org/r/61002/diff/2/
> 
> 
> Testing
> ---
> 
> **Functional test**
> Imported ZIP file using the new API.
> 
> **Unit tests**
> None.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 61002: ATLAS-1968: Import File Signature Change

2017-07-20 Thread Ashutosh Mestry


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61002/
---

(Updated July 20, 2017, 5:22 p.m.)


Review request for atlas and Madhan Neethiraj.


Changes
---

Updates include:
- Updated TWIKI documentation.


Bugs: ATLAS-1968
https://issues.apache.org/jira/browse/ATLAS-1968


Repository: atlas


Description
---

**Background**
The Import API that takes file as input was changed to make it consistent with 
the other API that takes ZIP stream as input.

This change did not correctly reflect within _ImportService_.

**Solution**
_AtlasImportRequest_ has been updated to add the _fileName_ option.

**CURL**

Note that the file path should be a path on the server.

_importOptions.json_ contents:
```javascript
{
"options": {
"fileName": "/root/smalldb.zip"
}
}
```

CURL
```bash
curl -g -X POST -u admin:admin -H "Content-Type: application/json" -H 
"Cache-Control: no-cache" -d @./importOptions.json 
"http://localhost:21000/api/atlas/admin/importFile";
```


Diffs (updated)
-

  docs/src/site/twiki/Import-API-Options.twiki 85887844 
  docs/src/site/twiki/Import-API.twiki b5de113f 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
4d2ac627 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 
92217178 


Diff: https://reviews.apache.org/r/61002/diff/2/

Changes: https://reviews.apache.org/r/61002/diff/1-2/


Testing
---

**Functional test**
Imported ZIP file using the new API.

**Unit tests**
None.


Thanks,

Ashutosh Mestry

Review Request 61002: ATLAS-1968: Import File Signature Change

2017-07-20 Thread Ashutosh Mestry


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61002/
---

Review request for atlas and Madhan Neethiraj.


Bugs: ATLAS-1968
https://issues.apache.org/jira/browse/ATLAS-1968


Repository: atlas


Description
---

**Background**
The Import API that takes file as input was changed to make it consistent with 
the other API that takes ZIP stream as input.

This change did not correctly reflect within _ImportService_.

**Solution**
_AtlasImportRequest_ has been updated to add the _fileName_ option.

**CURL**

Note that the file path should be a path on the server.

_importOptions.json_ contents:
```javascript
{
"options": {
"fileName": "/root/smalldb.zip"
}
}
```

CURL
```bash
curl -g -X POST -u admin:admin -H "Content-Type: application/json" -H 
"Cache-Control: no-cache" -d @./importOptions.json 
"http://localhost:21000/api/atlas/admin/importFile";
```


Diffs
-

  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
4d2ac627 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 
92217178 


Diff: https://reviews.apache.org/r/61002/diff/1/


Testing
---

**Functional test**
Imported ZIP file using the new API.

**Unit tests**
None.


Thanks,

Ashutosh Mestry

[jira] [Updated] (ATLAS-1968) Export/Import - Exception thrown when Import fired with zip file found on server

2017-07-20 Thread Ashutosh Mestry (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry updated ATLAS-1968:
---
Attachment: ATLAS-1968-Importfile-option.patch

> Export/Import -  Exception thrown when Import fired with zip file found on 
> server
> -
>
> Key: ATLAS-1968
> URL: https://issues.apache.org/jira/browse/ATLAS-1968
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 0.9-incubating, 0.8.1-incubating
>Reporter: Sharmadha Sainath
>Assignee: Ashutosh Mestry
>Priority: Blocker
> Attachments: ATLAS-1968-Importfile-option.patch, 
> ImportExceptionWhenZipFileOnServer.txt
>
>
> 1. Exported an entity on the cluster1 ,created zip file and placed it in /tmp 
> location of cluster2 .
> 2. Fired the following import command against cluster2 :
> {code}
> curl -v -X POST -u admin:admin  
> "http://cluster2:21000/api/atlas/admin/importFile?FILENAME=/tmp/entity.zip"; 
> {code}
> The import command failed with 500 internal server error.
> Attached the exception stack trace found in cluster2's application logs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (ATLAS-1968) Export/Import - Exception thrown when Import fired with zip file found on server

2017-07-20 Thread Ashutosh Mestry (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry reassigned ATLAS-1968:
--

Assignee: Ashutosh Mestry

> Export/Import -  Exception thrown when Import fired with zip file found on 
> server
> -
>
> Key: ATLAS-1968
> URL: https://issues.apache.org/jira/browse/ATLAS-1968
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 0.9-incubating, 0.8.1-incubating
>Reporter: Sharmadha Sainath
>Assignee: Ashutosh Mestry
>Priority: Blocker
> Attachments: ATLAS-1968-Importfile-option.patch, 
> ImportExceptionWhenZipFileOnServer.txt
>
>
> 1. Exported an entity on the cluster1 ,created zip file and placed it in /tmp 
> location of cluster2 .
> 2. Fired the following import command against cluster2 :
> {code}
> curl -v -X POST -u admin:admin  
> "http://cluster2:21000/api/atlas/admin/importFile?FILENAME=/tmp/entity.zip"; 
> {code}
> The import command failed with 500 internal server error.
> Attached the exception stack trace found in cluster2's application logs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (ATLAS-1969) Cannot create a second child relationship

2017-07-20 Thread David Radley (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley resolved ATLAS-1969.
-
Resolution: Not A Problem

I had found a bug in my test - I have it working now :-) 

> Cannot create a second child relationship
> -
>
> Key: ATLAS-1969
> URL: https://issues.apache.org/jira/browse/ATLAS-1969
> Project: Atlas
>  Issue Type: Bug
>Reporter: David Radley
> Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, 
> 0220CategoryHierarchy.json
>
>
> On the latest code I create 3 GlossaryCategories CatA CatB and CatC. I can 
> successfully create a relationship with type CategoryHierarchyLink with CatA 
> as the parent and CatB as the child.. I then try to try to create a 
> relationship with type CategoryHierarchyLink with catA as the parent and CatC 
> as the child; and it fails with
> 2017-07-20 12:14:51,188 ERROR - [pool-1-thread-7 - 
> 7d5fbe39-0e26-4af0-9137-651b51f9a69c:] ~ graph rollback due to exception 
> (GraphTransactionInterceptor:71)
> org.apache.atlas.exception.AtlasBaseException: invalid parameters: found null 
> entity
> at 
> org.apache.atlas.repository.store.graph.v1.AtlasEntityGraphDiscoveryV1.discover(AtlasEntityGraphDiscoveryV1.java:139)
> at 
> org.apache.atlas.repository.store.graph.v1.AtlasEntityGraphDiscoveryV1.discoverEntities(AtlasEntityGraphDiscoveryV1.java:69)
> at 
> org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.preCreateOrUpdate(AtlasEntityStoreV1.java:612)
> at org



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (ATLAS-1978) gas omas : add outbound publishing/kafka

2017-07-20 Thread Nigel Jones (JIRA)

Nigel Jones created ATLAS-1978:
--

 Summary: gas omas : add outbound publishing/kafka
 Key: ATLAS-1978
 URL: https://issues.apache.org/jira/browse/ATLAS-1978
 Project: Atlas
  Issue Type: Sub-task
Reporter: Nigel Jones


GAF omas must publish changes on topics for consumers.
This subtask will be used to add that support after the initial rest api has 
been created



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (ATLAS-1977) gas omas : add inbound messaging/kafka support

2017-07-20 Thread Nigel Jones (JIRA)

Nigel Jones created ATLAS-1977:
--

 Summary: gas omas : add inbound messaging/kafka support
 Key: ATLAS-1977
 URL: https://issues.apache.org/jira/browse/ATLAS-1977
 Project: Atlas
  Issue Type: Sub-task
Reporter: Nigel Jones






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (ATLAS-1976) gaf omas : Add implementation of roles

2017-07-20 Thread Nigel Jones (JIRA)

Nigel Jones created ATLAS-1976:
--

 Summary: gaf omas : Add implementation of roles
 Key: ATLAS-1976
 URL: https://issues.apache.org/jira/browse/ATLAS-1976
 Project: Atlas
  Issue Type: Sub-task
Reporter: Nigel Jones


This subtask will add roles support (this is dependent on model changes)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (ATLAS-1975) gaf omas : Implement classification support

2017-07-20 Thread Nigel Jones (JIRA)

Nigel Jones created ATLAS-1975:
--

 Summary: gaf omas : Implement classification support
 Key: ATLAS-1975
 URL: https://issues.apache.org/jira/browse/ATLAS-1975
 Project: Atlas
  Issue Type: Sub-task
Reporter: Nigel Jones


This subtask will add the initial implementation for classifications (tags)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (ATLAS-1974) gas omas : create skeleton implementation + generated swagger

2017-07-20 Thread Nigel Jones (JIRA)

Nigel Jones created ATLAS-1974:
--

 Summary: gas omas : create skeleton implementation + generated 
swagger
 Key: ATLAS-1974
 URL: https://issues.apache.org/jira/browse/ATLAS-1974
 Project: Atlas
  Issue Type: Sub-task
Reporter: Nigel Jones


In this task I'll take the previously generated definition in yaml & take the 
first step to making it closer to implementation by creating the appropriate 
directory structure, maven projects & java classes which would generate the 
interface & generate swagger output



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (ATLAS-1973) gaf omas : high level swagger API + supporting doc

2017-07-20 Thread Nigel Jones (JIRA)

Nigel Jones created ATLAS-1973:
--

 Summary: gaf omas : high level swagger API + supporting doc
 Key: ATLAS-1973
 URL: https://issues.apache.org/jira/browse/ATLAS-1973
 Project: Atlas
  Issue Type: Sub-task
Reporter: Nigel Jones


Complete/update swagger.io high level definition of api (yaml)
Add supporting notes as documentation 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ATLAS-1696) Governance Action Framework OMAS

2017-07-20 Thread Nigel Jones (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094883#comment-16094883
 ] 

Nigel Jones commented on ATLAS-1696:


[~davidrad] I will update the swagger as per some of our recent discussions. 
Tags is the term still used by ranger, whilst classifications are used by atlas 
(there is of course also a distinction between the classification (type), the 
classification instance, and that association of a classification 
definition+instance to an asset (which in ranger is termed a resource).  I 
initially went with tag being in a ranger-mindset, plus it's shorter... BUT To 
answer your question I will change to classification since this is an Atlas 
API. 

On your roles question, I will clarify -- it's still dependent on the 
definition of roles which we haven't worked through in the model yet.  I'll go 
with all roles atlas knows about for now, and elaborate once the model is 
refined. 

The GAF implementation will be neutral. There are references in the swagger doc 
as I hoped it would form a basis for discussion rather than being the final 
result. I will update references to ensure if ranger is mentioned it's clear 
it's an example. Further I am creating some subtasks which will include 
creating a swagger output off a skeleton implementation as you've done for 
glossary OMAS.

> Governance Action Framework OMAS
> 
>
> Key: ATLAS-1696
> URL: https://issues.apache.org/jira/browse/ATLAS-1696
> Project: Atlas
>  Issue Type: New Feature
>Reporter: Nigel Jones
>Assignee: Nigel Jones
>  Labels: VirtualDataConnector
>
> Governance Action OMAS is one of multiple consumer-centric based interfaces 
> that will be added to Apache Atlas, & provides the API (REST and messaging) 
> to support policy enforcement frameworks such as Apache Ranger. Detailed 
> knowledge of the Atlas data models and structure can then be hidden from 
> these consumers.
> The functionality of gaf includes
>  - ability to retrieve classifications associated to assets
>   - restricted to "interesting" classifications 
>   - restricted to interesting assets being managed by the requesting endpoint
>  - to retrieve a list of interesting roles that relate to enforcement
>  - to retrieve any template rule definitions/lookup tables that might be used 
> to construct executable rules
> The scoping constructs supported in the API will include
>  - Only get classifications that are relevant for security enforcement (ie: 
> only those inheriting from a specified supertype? Verify in ATLAS-1839)
>  - only get information about assets (resources) in a certain part of the 
> datalake (Q: HOW. By zone? How to specify? by asset type? By associated 
> endpoint?)
>  - pagination
>  
> See ATLAS-1839 for more information on the model and classifications
> In the Atlas data model classifications propagate - for example
>  * An database column DOB has no explicit classification
>  * It's containing table CDB  is classified as "customer personal details"
>  * The "SPI" classification is attached to this table with the value 
> "sensitive"
> At enforcement time all that an engine such as ranger cares about is that the 
> column "DOB" is sensitive, how we got there isn't important.  In the example 
> above the propogation occurs
>  * Along the assigned term relationship 
>  * along the structural containment relationship (table->column)
> Therefore gaf omas will "flatten" the structure - so in this case we'll see
>  table/CDB - SPI:sensitive
>  column/DOB - SPI:sensitive
> There will be cases where multiple classifications (of the same type) can be 
> navigated to from an asset like DOB. This may not make logical sense, 
> however, Until precedence is resolved in ATLAS-1839 & related Jiras, OMAS 
> will pass through multiple classifications
> This interface will also support message notifications of changes to managed 
> resources such as a new role, classification. A single kafka topic will be 
> used. 
>  
> A first pass swagger can be found at 
> https://app.swaggerhub.com/apis/planetf1/GovernanceActionOMAS/0.1



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (ATLAS-1972) Allow only a named set of classifications to be associated with an entity.

2017-07-20 Thread David Radley (JIRA)

David Radley created ATLAS-1972:
---

 Summary: Allow only a named set of classifications to be 
associated with an entity.
 Key: ATLAS-1972
 URL: https://issues.apache.org/jira/browse/ATLAS-1972
 Project: Atlas
  Issue Type: Improvement
Reporter: David Radley
Assignee: David Radley


Enhance EntityDef to have a list of classification names - that if specified 
can be used at runtime to reject the association of a classification to an 
Entity. The EntityDef Classifications are inherited by any children EntityDefs. 

If a parent classification is already associated with an entityDef then we 
should not allow a child classificationDef to be added - as this could be 
confusing. 


  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ATLAS-1696) Governance Action Framework OMAS

2017-07-20 Thread Nigel Jones (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094861#comment-16094861
 ] 

Nigel Jones commented on ATLAS-1696:


Glossary omas is being defined in 
https://issues.apache.org/jira/browse/ATLAS-1698 - the discussion includes some 
general discussion that would apply to all OMAS interfaces. 

Creating sub-tasks for review and implementation:

> Governance Action Framework OMAS
> 
>
> Key: ATLAS-1696
> URL: https://issues.apache.org/jira/browse/ATLAS-1696
> Project: Atlas
>  Issue Type: New Feature
>Reporter: Nigel Jones
>Assignee: Nigel Jones
>  Labels: VirtualDataConnector
>
> Governance Action OMAS is one of multiple consumer-centric based interfaces 
> that will be added to Apache Atlas, & provides the API (REST and messaging) 
> to support policy enforcement frameworks such as Apache Ranger. Detailed 
> knowledge of the Atlas data models and structure can then be hidden from 
> these consumers.
> The functionality of gaf includes
>  - ability to retrieve classifications associated to assets
>   - restricted to "interesting" classifications 
>   - restricted to interesting assets being managed by the requesting endpoint
>  - to retrieve a list of interesting roles that relate to enforcement
>  - to retrieve any template rule definitions/lookup tables that might be used 
> to construct executable rules
> The scoping constructs supported in the API will include
>  - Only get classifications that are relevant for security enforcement (ie: 
> only those inheriting from a specified supertype? Verify in ATLAS-1839)
>  - only get information about assets (resources) in a certain part of the 
> datalake (Q: HOW. By zone? How to specify? by asset type? By associated 
> endpoint?)
>  - pagination
>  
> See ATLAS-1839 for more information on the model and classifications
> In the Atlas data model classifications propagate - for example
>  * An database column DOB has no explicit classification
>  * It's containing table CDB  is classified as "customer personal details"
>  * The "SPI" classification is attached to this table with the value 
> "sensitive"
> At enforcement time all that an engine such as ranger cares about is that the 
> column "DOB" is sensitive, how we got there isn't important.  In the example 
> above the propogation occurs
>  * Along the assigned term relationship 
>  * along the structural containment relationship (table->column)
> Therefore gaf omas will "flatten" the structure - so in this case we'll see
>  table/CDB - SPI:sensitive
>  column/DOB - SPI:sensitive
> There will be cases where multiple classifications (of the same type) can be 
> navigated to from an asset like DOB. This may not make logical sense, 
> however, Until precedence is resolved in ATLAS-1839 & related Jiras, OMAS 
> will pass through multiple classifications
> This interface will also support message notifications of changes to managed 
> resources such as a new role, classification. A single kafka topic will be 
> used. 
>  
> A first pass swagger can be found at 
> https://app.swaggerhub.com/apis/planetf1/GovernanceActionOMAS/0.1



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ATLAS-1969) Cannot create a second child relationship

2017-07-20 Thread Madhan Neethiraj (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-1969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094809#comment-16094809
 ] 

Madhan Neethiraj commented on ATLAS-1969:
-

[~davidrad] - can you please add JSONs for the entities and relationship 
creation?

> Cannot create a second child relationship
> -
>
> Key: ATLAS-1969
> URL: https://issues.apache.org/jira/browse/ATLAS-1969
> Project: Atlas
>  Issue Type: Bug
>Reporter: David Radley
> Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, 
> 0220CategoryHierarchy.json
>
>
> On the latest code I create 3 GlossaryCategories CatA CatB and CatC. I can 
> successfully create a relationship with type CategoryHierarchyLink with CatA 
> as the parent and CatB as the child.. I then try to try to create a 
> relationship with type CategoryHierarchyLink with catA as the parent and CatC 
> as the child; and it fails with
> 2017-07-20 12:14:51,188 ERROR - [pool-1-thread-7 - 
> 7d5fbe39-0e26-4af0-9137-651b51f9a69c:] ~ graph rollback due to exception 
> (GraphTransactionInterceptor:71)
> org.apache.atlas.exception.AtlasBaseException: invalid parameters: found null 
> entity
> at 
> org.apache.atlas.repository.store.graph.v1.AtlasEntityGraphDiscoveryV1.discover(AtlasEntityGraphDiscoveryV1.java:139)
> at 
> org.apache.atlas.repository.store.graph.v1.AtlasEntityGraphDiscoveryV1.discoverEntities(AtlasEntityGraphDiscoveryV1.java:69)
> at 
> org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.preCreateOrUpdate(AtlasEntityStoreV1.java:612)
> at org



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

RE: Unsubscribe me from Mailing List

2017-07-20 Thread Bilal Arshad

Kindly unsubscribe me from the mailing list too.

Kind Regards

-Original Message-
From: Muhammad Waris [mailto:muhammad.wa...@northbaysolutions.net]
Sent: 20 July 2017 14:46
To: dev@atlas.apache.org
Cc: d...@atlas.incubator.apache.org
Subject: Re: Unsubscribe me from Mailing List

Hi,

Also please Unsubscribe me from the mailing list.

Thanks.

On Thu, Jul 20, 2017 at 2:48 PM, naseem rafique < 
naseem.rafi...@northbaysolutions.net> wrote:

> Hi,
>
> Please Unsubscribe me from the mailing list.
>
> Thanks.
>

The University of Derby has a published policy regarding email and reserves the 
right to monitor email traffic.
If you believe this was sent to you in error, please reply to the sender and 
let them know.

Key University contacts: http://www.derby.ac.uk/its/contacts/

[jira] [Updated] (ATLAS-1970) Export/Import - When updateTypeDefinition set to false , new types are not imported

2017-07-20 Thread Sharmadha Sainath (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sharmadha Sainath updated ATLAS-1970:
-
Attachment: (was: ATLAS-1970.patch)

> Export/Import - When updateTypeDefinition set to false , new types are not 
> imported
> ---
>
> Key: ATLAS-1970
> URL: https://issues.apache.org/jira/browse/ATLAS-1970
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 0.9-incubating, 0.8.1-incubating
>Reporter: Sharmadha Sainath
>Assignee: Sharmadha Sainath
>Priority: Critical
> Attachments: ImportFailureDueToUnknownType.txt
>
>
> Import has option "updateTypeDefinition" which is used to update the type 
> definitions in the backup cluster (cluster on which import is done) when the 
> value is set to true. (Default is set to true). When its value is set to 
> false , types in backup cluster are not updated with types present in 
> exported zip file.
> This works fine when , say a type type1 is present in both clusters , an 
> entity of type type1 is exported and imported into backup cluster with 
> updateTypeDefinition is set to false - Import is done successfully and type 
> is not updated.
> When the zip file contains type5 *which is not present in backup cluster* and 
> the when import is fired , import fails with following exception:
> {code}
> {"errorCode":"ATLAS-500-00-001","errorMessage":"org.apache.atlas.exception.AtlasBaseException:
>  Type ENTITY with name type5 does not exist"}
> {code}
> Attached the complete exception stack trace found in backup cluster's 
> application logs. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Issue Comment Deleted] (ATLAS-1970) Export/Import - When updateTypeDefinition set to false , new types are not imported

2017-07-20 Thread Sharmadha Sainath (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sharmadha Sainath updated ATLAS-1970:
-
Comment: was deleted

(was: Added a patch with fix to allow creation of types even when 
updateTypeDefinition is set to false.

As there can be many types in the exported zip file ( which may / may not be 
present in the backup cluster ) , it would be a little overhead to check if 
type is already present and update if present.

Also ,  updateTypeDefinition's default value is true. So , if 
updateTypeDefinition is set to true or not set at all , Atlas would still go 
ahead and create/update the types.

Please let me know/re-assign if there is be a better way of fixing this.

Tested the patch with following :
1. updateTypeDefinition set to false
2. updateTypeDefinition set to true
3. updateTypeDefinition is not specified
4. updateTypeDefinition with true/false with other options such as transforms

CC : [~ashutoshm] [~mad...@apache.org])

> Export/Import - When updateTypeDefinition set to false , new types are not 
> imported
> ---
>
> Key: ATLAS-1970
> URL: https://issues.apache.org/jira/browse/ATLAS-1970
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 0.9-incubating, 0.8.1-incubating
>Reporter: Sharmadha Sainath
>Assignee: Sharmadha Sainath
>Priority: Critical
> Attachments: ATLAS-1970.patch, ImportFailureDueToUnknownType.txt
>
>
> Import has option "updateTypeDefinition" which is used to update the type 
> definitions in the backup cluster (cluster on which import is done) when the 
> value is set to true. (Default is set to true). When its value is set to 
> false , types in backup cluster are not updated with types present in 
> exported zip file.
> This works fine when , say a type type1 is present in both clusters , an 
> entity of type type1 is exported and imported into backup cluster with 
> updateTypeDefinition is set to false - Import is done successfully and type 
> is not updated.
> When the zip file contains type5 *which is not present in backup cluster* and 
> the when import is fired , import fails with following exception:
> {code}
> {"errorCode":"ATLAS-500-00-001","errorMessage":"org.apache.atlas.exception.AtlasBaseException:
>  Type ENTITY with name type5 does not exist"}
> {code}
> Attached the complete exception stack trace found in backup cluster's 
> application logs. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (ATLAS-1971) Relationship end1 and end 2 are the wrong way round in the instances

2017-07-20 Thread David Radley (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley resolved ATLAS-1971.
-
Resolution: Invalid

My mistake I was confused with the types and names - I think it is good. 

> Relationship end1 and end 2 are the wrong way round in the instances
> 
>
> Key: ATLAS-1971
> URL: https://issues.apache.org/jira/browse/ATLAS-1971
> Project: Atlas
>  Issue Type: Bug
>Reporter: David Radley
>
> If you use the files that have been attached to ATLAS-1969, And create a 
> Glossary and a GlossaryCategory instance. I notice :
> "relationshipDefs":[
>  {
>  "name": "CategoryAnchor",
>  "typeVersion": "1.0",
>  "endDef1": {
>"name": "anchor",
>"type": "GlossaryCategory",
>"cardinality":"SINGLE"
>  },
>  "endDef2": {
>"name": "categories",
>"type": "Glossary",
>"cardinality":"SET",
> "isContainer":true
>  },
>  "relationshipCategory":"COMPOSITION",
>  "propagateTags":"NONE"
>  },
> when I create this relationship I had expected to see GlossaryCategory with 
> new attribute anchor of type Glossary. I create an instance and see that the 
> Category CatA is  : 
> {
> "referredEntities": {},
> "entity": {
> "typeName": "GlossaryCategory",
> "attributes": {
> "shortDescription": "CatA desc",
> "qualifiedName": "DomainA.CatA",
> "longDescription": "CatA desc is described here and we could say 
> alot more about it",
> "displayName": "CatA"
> },
> "guid": "e2d4fe5c-7e20-447e-a1d2-a4eacd093174",
> "status": "ACTIVE",
> "createdBy": "admin",
> "updatedBy": "admin",
> "createTime": 1500549089631,
> "updateTime": 1500549089631,
> "version": 0,
> "relationshipAttributes": {
> "externalGlossaryLinks": [],
> "meanings": [],
> "parentCategory": {
> "guid": "d6df28ed-1179-4f4d-bead-ca632bc2fe14",
> "typeName": "GlossaryCategory"
> },
> "terms": [],
> "externalReference": [],
> "embeddedMetadata": [],
> "categories": [
> {
> "guid": "de1890ce-c923-4b79-809b-f33e4054e672",
> "typeName": "Glossary"
> }
> ],
> "childrenCategories": []
> },
> "classifications": []
> }
> }
> This is not correct, it should have injected a relationship attribute anchor. 
> Also the Glossary is : 
> {
> "referredEntities": {},
> "entity": {
> "typeName": "Glossary",
> "attributes": {
> "shortDescription": "TestGlossary1 is described here",
> "qualifiedName": "DomainA.TestGlossary1",
> "longDescription": "TestGlossary1 is described here and we could 
> say alot more about it",
> "usage": "Sample for test purposes",
> "language": "English (UK)",
> "displayName": "TestGlossary1"
> },
> "guid": "de1890ce-c923-4b79-809b-f33e4054e672",
> "status": "ACTIVE",
> "createdBy": "admin",
> "updatedBy": "admin",
> "createTime": 1500549085716,
> "updateTime": 1500549085716,
> "version": 0,
> "relationshipAttributes": {
> "meanings": [],
> "externalGlossary": [],
> "externalReference": [],
> "embeddedMetadata": [],
> "anchor": {
> "guid": "e2d4fe5c-7e20-447e-a1d2-a4eacd093174",
> "typeName": "GlossaryCategory"
> }
> },
> "classifications": []
> }
> }
>



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Unsubscribe me from Mailing List

2017-07-20 Thread Muhammad Waris

Hi,

Also please Unsubscribe me from the mailing list.

Thanks.

On Thu, Jul 20, 2017 at 2:48 PM, naseem rafique <
naseem.rafi...@northbaysolutions.net> wrote:

> Hi,
>
> Please Unsubscribe me from the mailing list.
>
> Thanks.
>

[jira] [Created] (ATLAS-1971) Relationship end1 and end 2 are the wrong way round in the instances

2017-07-20 Thread David Radley (JIRA)

David Radley created ATLAS-1971:
---

 Summary: Relationship end1 and end 2 are the wrong way round in 
the instances
 Key: ATLAS-1971
 URL: https://issues.apache.org/jira/browse/ATLAS-1971
 Project: Atlas
  Issue Type: Bug
Reporter: David Radley


If you use the files that have been attached to ATLAS-1969, And create a 
Glossary and a GlossaryCategory instance. I notice :

"relationshipDefs":[
 {
 "name": "CategoryAnchor",
 "typeVersion": "1.0",

 "endDef1": {
   "name": "anchor",
   "type": "GlossaryCategory",
   "cardinality":"SINGLE"

 },
 "endDef2": {
   "name": "categories",
   "type": "Glossary",
   "cardinality":"SET",
"isContainer":true
 },
 "relationshipCategory":"COMPOSITION",
 "propagateTags":"NONE"
 },
when I create this relationship I had expected to see GlossaryCategory with new 
attribute anchor of type Glossary. I create an instance and see that the 
Category CatA is  : 
{
"referredEntities": {},
"entity": {
"typeName": "GlossaryCategory",
"attributes": {
"shortDescription": "CatA desc",
"qualifiedName": "DomainA.CatA",
"longDescription": "CatA desc is described here and we could say 
alot more about it",
"displayName": "CatA"
},
"guid": "e2d4fe5c-7e20-447e-a1d2-a4eacd093174",
"status": "ACTIVE",
"createdBy": "admin",
"updatedBy": "admin",
"createTime": 1500549089631,
"updateTime": 1500549089631,
"version": 0,
"relationshipAttributes": {
"externalGlossaryLinks": [],
"meanings": [],
"parentCategory": {
"guid": "d6df28ed-1179-4f4d-bead-ca632bc2fe14",
"typeName": "GlossaryCategory"
},
"terms": [],
"externalReference": [],
"embeddedMetadata": [],
"categories": [
{
"guid": "de1890ce-c923-4b79-809b-f33e4054e672",
"typeName": "Glossary"
}
],
"childrenCategories": []
},
"classifications": []
}
}

This is not correct, it should have injected a relationship attribute anchor. 

Also the Glossary is : 
{
"referredEntities": {},
"entity": {
"typeName": "Glossary",
"attributes": {
"shortDescription": "TestGlossary1 is described here",
"qualifiedName": "DomainA.TestGlossary1",
"longDescription": "TestGlossary1 is described here and we could 
say alot more about it",
"usage": "Sample for test purposes",
"language": "English (UK)",
"displayName": "TestGlossary1"
},
"guid": "de1890ce-c923-4b79-809b-f33e4054e672",
"status": "ACTIVE",
"createdBy": "admin",
"updatedBy": "admin",
"createTime": 1500549085716,
"updateTime": 1500549085716,
"version": 0,
"relationshipAttributes": {
"meanings": [],
"externalGlossary": [],
"externalReference": [],
"embeddedMetadata": [],
"anchor": {
"guid": "e2d4fe5c-7e20-447e-a1d2-a4eacd093174",
"typeName": "GlossaryCategory"
}
},
"classifications": []
}
}



   





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (ATLAS-1970) Export/Import - When updateTypeDefinition set to false , new types are not imported

2017-07-20 Thread Sharmadha Sainath (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-1970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094679#comment-16094679
 ] 

Sharmadha Sainath edited comment on ATLAS-1970 at 7/20/17 1:30 PM:
---

Added a patch with fix to allow creation of types even when 
updateTypeDefinition is set to false.

As there can be many types in the exported zip file ( which may / may not be 
present in the backup cluster ) , it would be a little overhead to check if 
type is already present and update if present.

Also ,  updateTypeDefinition's default value is true. So , if 
updateTypeDefinition is set to true or not set at all , Atlas would still go 
ahead and create/update the types.

Please let me know/re-assign if there is be a better way of fixing this.

Tested the patch with following :
1. updateTypeDefinition set to false
2. updateTypeDefinition set to true
3. updateTypeDefinition is not specified
4. updateTypeDefinition with true/false with other options such as transforms

CC : [~ashutoshm] [~mad...@apache.org]


was (Author: ssainath):
Added a patch with fix to allow creation of types even when 
updateTypeDefinition is set to false.

As there can be many types in the exported zip file ( which may / may not be 
present in the backup cluster ) , it would be a little overhead to check if 
type is already present and update if present.

Also ,  updateTypeDefinition's default value is true. So , if 
updateTypeDefinition is set to true or not set at all , Atlas would still go 
ahead and create/update the types.

Please let me know/re-assign if there could be a better way of fixing this.

Tested the patch with following :
1. updateTypeDefinition set to false
2. updateTypeDefinition set to true
3. updateTypeDefinition is not specified
4. updateTypeDefinition with true/false with other options such as transforms



> Export/Import - When updateTypeDefinition set to false , new types are not 
> imported
> ---
>
> Key: ATLAS-1970
> URL: https://issues.apache.org/jira/browse/ATLAS-1970
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 0.9-incubating, 0.8.1-incubating
>Reporter: Sharmadha Sainath
>Assignee: Sharmadha Sainath
>Priority: Critical
> Attachments: ATLAS-1970.patch, ImportFailureDueToUnknownType.txt
>
>
> Import has option "updateTypeDefinition" which is used to update the type 
> definitions in the backup cluster (cluster on which import is done) when the 
> value is set to true. (Default is set to true). When its value is set to 
> false , types in backup cluster are not updated with types present in 
> exported zip file.
> This works fine when , say a type type1 is present in both clusters , an 
> entity of type type1 is exported and imported into backup cluster with 
> updateTypeDefinition is set to false - Import is done successfully and type 
> is not updated.
> When the zip file contains type5 *which is not present in backup cluster* and 
> the when import is fired , import fails with following exception:
> {code}
> {"errorCode":"ATLAS-500-00-001","errorMessage":"org.apache.atlas.exception.AtlasBaseException:
>  Type ENTITY with name type5 does not exist"}
> {code}
> Attached the complete exception stack trace found in backup cluster's 
> application logs. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (ATLAS-1970) Export/Import - When updateTypeDefinition set to false , new types are not imported

2017-07-20 Thread Sharmadha Sainath (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sharmadha Sainath updated ATLAS-1970:
-
Attachment: ATLAS-1970.patch

Added a patch with fix to allow creation of types even when 
updateTypeDefinition is set to false.

As there can be many types in the exported zip file ( which may / may not be 
present in the backup cluster ) , it would be a little overhead to check if 
type is already present and update if present.

Also ,  updateTypeDefinition's default value is true. So , if 
updateTypeDefinition is set to true or not set at all , Atlas would still go 
ahead and create/update the types.

Please let me know/re-assign if there could be a better way of fixing this.

Tested the patch with following :
1. updateTypeDefinition set to false
2. updateTypeDefinition set to true
3. updateTypeDefinition is not specified
4. updateTypeDefinition with true/false with other options such as transforms



> Export/Import - When updateTypeDefinition set to false , new types are not 
> imported
> ---
>
> Key: ATLAS-1970
> URL: https://issues.apache.org/jira/browse/ATLAS-1970
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 0.9-incubating, 0.8.1-incubating
>Reporter: Sharmadha Sainath
>Assignee: Sharmadha Sainath
>Priority: Critical
> Attachments: ATLAS-1970.patch, ImportFailureDueToUnknownType.txt
>
>
> Import has option "updateTypeDefinition" which is used to update the type 
> definitions in the backup cluster (cluster on which import is done) when the 
> value is set to true. (Default is set to true). When its value is set to 
> false , types in backup cluster are not updated with types present in 
> exported zip file.
> This works fine when , say a type type1 is present in both clusters , an 
> entity of type type1 is exported and imported into backup cluster with 
> updateTypeDefinition is set to false - Import is done successfully and type 
> is not updated.
> When the zip file contains type5 *which is not present in backup cluster* and 
> the when import is fired , import fails with following exception:
> {code}
> {"errorCode":"ATLAS-500-00-001","errorMessage":"org.apache.atlas.exception.AtlasBaseException:
>  Type ENTITY with name type5 does not exist"}
> {code}
> Attached the complete exception stack trace found in backup cluster's 
> application logs. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (ATLAS-1969) Cannot create a second child relationship

2017-07-20 Thread David Radley (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-1969:

Description: 
On the latest code I create 3 GlossaryCategories CatA CatB and CatC. I can 
successfully create a relationship with type CategoryHierarchyLink with CatA as 
the parent and CatB as the child.. I then try to try to create a relationship 
with type CategoryHierarchyLink with catA as the parent and CatC as the child; 
and it fails with
2017-07-20 12:14:51,188 ERROR - [pool-1-thread-7 - 
7d5fbe39-0e26-4af0-9137-651b51f9a69c:] ~ graph rollback due to exception 
(GraphTransactionInterceptor:71)
org.apache.atlas.exception.AtlasBaseException: invalid parameters: found null 
entity
at 
org.apache.atlas.repository.store.graph.v1.AtlasEntityGraphDiscoveryV1.discover(AtlasEntityGraphDiscoveryV1.java:139)
at 
org.apache.atlas.repository.store.graph.v1.AtlasEntityGraphDiscoveryV1.discoverEntities(AtlasEntityGraphDiscoveryV1.java:69)
at 
org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.preCreateOrUpdate(AtlasEntityStoreV1.java:612)
at org

  was:
On the latest code I create 3 categories A B and C. I can successfully create a 
relationship with type CategoryHierarchyLink with catA as the parent and CatB 
as the child.. I then try to try to create a relationship with type 
CategoryHierarchyLink with catA as the parent and CatC as the child; and it 
fails with
2017-07-20 12:14:51,188 ERROR - [pool-1-thread-7 - 
7d5fbe39-0e26-4af0-9137-651b51f9a69c:] ~ graph rollback due to exception 
(GraphTransactionInterceptor:71)
org.apache.atlas.exception.AtlasBaseException: invalid parameters: found null 
entity
at 
org.apache.atlas.repository.store.graph.v1.AtlasEntityGraphDiscoveryV1.discover(AtlasEntityGraphDiscoveryV1.java:139)
at 
org.apache.atlas.repository.store.graph.v1.AtlasEntityGraphDiscoveryV1.discoverEntities(AtlasEntityGraphDiscoveryV1.java:69)
at 
org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.preCreateOrUpdate(AtlasEntityStoreV1.java:612)
at org


> Cannot create a second child relationship
> -
>
> Key: ATLAS-1969
> URL: https://issues.apache.org/jira/browse/ATLAS-1969
> Project: Atlas
>  Issue Type: Bug
>Reporter: David Radley
> Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, 
> 0220CategoryHierarchy.json
>
>
> On the latest code I create 3 GlossaryCategories CatA CatB and CatC. I can 
> successfully create a relationship with type CategoryHierarchyLink with CatA 
> as the parent and CatB as the child.. I then try to try to create a 
> relationship with type CategoryHierarchyLink with catA as the parent and CatC 
> as the child; and it fails with
> 2017-07-20 12:14:51,188 ERROR - [pool-1-thread-7 - 
> 7d5fbe39-0e26-4af0-9137-651b51f9a69c:] ~ graph rollback due to exception 
> (GraphTransactionInterceptor:71)
> org.apache.atlas.exception.AtlasBaseException: invalid parameters: found null 
> entity
> at 
> org.apache.atlas.repository.store.graph.v1.AtlasEntityGraphDiscoveryV1.discover(AtlasEntityGraphDiscoveryV1.java:139)
> at 
> org.apache.atlas.repository.store.graph.v1.AtlasEntityGraphDiscoveryV1.discoverEntities(AtlasEntityGraphDiscoveryV1.java:69)
> at 
> org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.preCreateOrUpdate(AtlasEntityStoreV1.java:612)
> at org



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (ATLAS-1970) Export/Import - When updateTypeDefinition set to false , new types are not imported

2017-07-20 Thread Sharmadha Sainath (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sharmadha Sainath reassigned ATLAS-1970:


Assignee: Sharmadha Sainath

> Export/Import - When updateTypeDefinition set to false , new types are not 
> imported
> ---
>
> Key: ATLAS-1970
> URL: https://issues.apache.org/jira/browse/ATLAS-1970
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 0.9-incubating, 0.8.1-incubating
>Reporter: Sharmadha Sainath
>Assignee: Sharmadha Sainath
>Priority: Critical
> Attachments: ImportFailureDueToUnknownType.txt
>
>
> Import has option "updateTypeDefinition" which is used to update the type 
> definitions in the backup cluster (cluster on which import is done) when the 
> value is set to true. (Default is set to true). When its value is set to 
> false , types in backup cluster are not updated with types present in 
> exported zip file.
> This works fine when , say a type type1 is present in both clusters , an 
> entity of type type1 is exported and imported into backup cluster with 
> updateTypeDefinition is set to false - Import is done successfully and type 
> is not updated.
> When the zip file contains type5 *which is not present in backup cluster* and 
> the when import is fired , import fails with following exception:
> {code}
> {"errorCode":"ATLAS-500-00-001","errorMessage":"org.apache.atlas.exception.AtlasBaseException:
>  Type ENTITY with name type5 does not exist"}
> {code}
> Attached the complete exception stack trace found in backup cluster's 
> application logs. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (ATLAS-1970) Export/Import - When updateTypeDefinition set to false , new types are not imported

2017-07-20 Thread Sharmadha Sainath (JIRA)

Sharmadha Sainath created ATLAS-1970:


 Summary: Export/Import - When updateTypeDefinition set to false , 
new types are not imported
 Key: ATLAS-1970
 URL: https://issues.apache.org/jira/browse/ATLAS-1970
 Project: Atlas
  Issue Type: Bug
  Components:  atlas-core
Affects Versions: 0.9-incubating, 0.8.1-incubating
Reporter: Sharmadha Sainath
Priority: Critical
 Attachments: ImportFailureDueToUnknownType.txt

Import has option "updateTypeDefinition" which is used to update the type 
definitions in the backup cluster (cluster on which import is done) when the 
value is set to true. (Default is set to true). When its value is set to false 
, types in backup cluster are not updated with types present in exported zip 
file.

This works fine when , say a type type1 is present in both clusters , an entity 
of type type1 is exported and imported into backup cluster with 
updateTypeDefinition is set to false - Import is done successfully and type is 
not updated.

When the zip file contains type5 *which is not present in backup cluster* and 
the when import is fired , import fails with following exception:

{code}
{"errorCode":"ATLAS-500-00-001","errorMessage":"org.apache.atlas.exception.AtlasBaseException:
 Type ENTITY with name type5 does not exist"}
{code}

Attached the complete exception stack trace found in backup cluster's 
application logs. 




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (ATLAS-1839) Area 2 of the open metadata model

2017-07-20 Thread Nigel Jones (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094613#comment-16094613
 ] 

Nigel Jones edited comment on ATLAS-1839 at 7/20/17 12:37 PM:
--

[~davidrad] the toxic combination support in ranger policies is primarily 
geared to controlling what a user may access, whilst the validation [~ivarea] 
is suggesting is primarily about creates and updates, ie defining the data 
model itself. That's not to say ranger couldn't do this (since it can address 
any operation such as a create) but I don't think that's ranger's intent. But I 
agree it's a fine line and could well vary significantly in different 
environments

As such I think it makes sense to define validation in atlas and be able to 
link to code artifacts, services that implement those validations probably 
through a combination of discovery & stewardship , plus making it easier when 
writing pipelines for say ETL or streaming, to be able to easily pull in atlas 
metadata and capture a link between a validation implemented by an pipeline 
author (or being used from a library) and it's definition in atlas. Thus atlas 
ends up with both the "intent" (the business spec if you like) as well as links 
to the implementation yet does not constrain those implementations since they 
can be so varied. It's also incremental, easy to adopt, and means one can get 
value without everything being in place?

Following on from this, absolutely some of those validations could be 
implemented as complex rules using a full featured rules engine, but I think it 
would be tricky and constraining to capture all that in atlas, hence why I'd go 
for the link approach & some relatively loose coupling

So with that done, sure we could have a more complex rules engine embedded in, 
or used by ranger plugins... but this could be one of a number of different 
approaches

I'd be inclined to start off with us figuring out how to model, and some use 
cases where we can explore the authoring (ie in atlas), assisted authoring 
(when writing a job), metadata capture (from those other systems, also relates 
to lineage) & probably best to do that in ATLAS-1995? This also touches on 
RANGER-1869 (metadata capture)

Certainly this is an interesting area !


was (Author: jonesn):
[~davidrad] the toxic combination support in ranger policies is primarily 
geared to controlling what a user may access, whilst the validation [~ivarea] 
is suggesting is primarily about creates and updates, ie defining the data 
model itself. That's not to say ranger couldn't do this (since it can address 
any operation such as a create) but I don't think that's ranger's intent. But I 
agree it's a fine line and could well vary significantly in different 
environments

As such I think it makes sense to define validation in atlas and be able to 
link to code artifacts, services that implement those validations probably 
through a combination of discovery & stewardship , plus making it easier when 
writing pipelines for say ETL or streaming, to be able to easily pull in atlas 
metadata and capture a link between a validation implemented by an pipeline 
author (or being used from a library) and it's definition in atlas. Thus atlas 
ends up with both the "intent" (the business spec if you like) as well as links 
to the implementation yet does not constrain those implementations since they 
can be so varied. 

Following on from this, absolutely some of those validations could be 
implemented as complex rules using a full featured rules engine, but I think it 
would be tricky and constraining to capture all that in atlas, hence why I'd go 
for the link approach & some relatively loose coupling

So with that done, sure we could have a more complex rules engine embedded in, 
or used by ranger plugins... but this could be one of a number of different 
approaches

I'd be inclined to start off with us figuring out how to model, and some use 
cases where we can explore the authoring (ie in atlas), assisted authoring 
(when writing a job), metadata capture (from those other systems, also relates 
to lineage) & probably best to do that in ATLAS-1995? This also touches on 
RANGER-1869 (metadata capture)

Certainly this is an interesting area !

> Area 2 of the open metadata model
> -
>
> Key: ATLAS-1839
> URL: https://issues.apache.org/jira/browse/ATLAS-1839
> Project: Atlas
>  Issue Type: Task
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: David Radley
>  Labels: OpenMetadata, VirtualDataConnector
> Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, 
> 0220CategoryHierarchy.json, 0230Terms.json, 0240Dictionary.json, 
> 0250RelatedTerms.json, 0260Contexts.json, 0270SemanticAssignme

[jira] [Comment Edited] (ATLAS-1839) Area 2 of the open metadata model

2017-07-20 Thread Nigel Jones (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094613#comment-16094613
 ] 

Nigel Jones edited comment on ATLAS-1839 at 7/20/17 12:36 PM:
--

[~davidrad] the toxic combination support in ranger policies is primarily 
geared to controlling what a user may access, whilst the validation [~ivarea] 
is suggesting is primarily about creates and updates, ie defining the data 
model itself. That's not to say ranger couldn't do this (since it can address 
any operation such as a create) but I don't think that's ranger's intent. But I 
agree it's a fine line and could well vary significantly in different 
environments

As such I think it makes sense to define validation in atlas and be able to 
link to code artifacts, services that implement those validations probably 
through a combination of discovery & stewardship , plus making it easier when 
writing pipelines for say ETL or streaming, to be able to easily pull in atlas 
metadata and capture a link between a validation implemented by an pipeline 
author (or being used from a library) and it's definition in atlas. Thus atlas 
ends up with both the "intent" (the business spec if you like) as well as links 
to the implementation yet does not constrain those implementations since they 
can be so varied. 

Following on from this, absolutely some of those validations could be 
implemented as complex rules using a full featured rules engine, but I think it 
would be tricky and constraining to capture all that in atlas, hence why I'd go 
for the link approach & some relatively loose coupling

So with that done, sure we could have a more complex rules engine embedded in, 
or used by ranger plugins... but this could be one of a number of different 
approaches

I'd be inclined to start off with us figuring out how to model, and some use 
cases where we can explore the authoring (ie in atlas), assisted authoring 
(when writing a job), metadata capture (from those other systems, also relates 
to lineage) & probably best to do that in ATLAS-1995? This also touches on 
RANGER-1869 (metadata capture)

Certainly this is an interesting area !


was (Author: jonesn):
[~davidrad] the toxic combination support in ranger policies is primarily 
geared to controlling what a user may access, whilst the validation [~ivarea] 
is suggesting is primarily about creates and updates, ie defining the data 
model itself. That's not to say ranger couldn't do this (since it can address 
any operation such as a create) but I don't think that's ranger's intent. But I 
agree it's a fine line and could well vary significantly in different 
environments

As such I think it makes sense to define validation in atlas and be able to 
link to code artifacts, services that implement those validations probably 
through a combination of discovery & stewardship , plus making it easier when 
writing pipelines for say ETL or streaming, to be able to easily pull in atlas 
metadata and capture a link between a validation implemented by an pipeline 
author (or being used from a library) and it's definition in atlas. Thus atlas 
ends up with both the "intent" (the business spec if you like) as well as links 
to the implementation yet does not constrain those implementations since they 
can be so varied. 

Following on from this, absolutely some of those validations could be 
implemented as complex rules, but I think it would be tricky and constraining 
to capture all that in atlas, hence why I'd go for the link approach & some 
relatively loose coupling

So with that done, sure we could have a more complex rules engine embedded in, 
or used by ranger plugins... but this could be one of a number of different 
approaches

I'd be inclined to start off with us figuring out how to model, and some use 
cases where we can explore the authoring (ie in atlas), assisted authoring 
(when writing a job), metadata capture (from those other systems, also relates 
to lineage) & probably best to do that in ATLAS-1995? This also touches on 
RANGER-1869 (metadata capture)

Certainly this is an interesting area !

> Area 2 of the open metadata model
> -
>
> Key: ATLAS-1839
> URL: https://issues.apache.org/jira/browse/ATLAS-1839
> Project: Atlas
>  Issue Type: Task
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: David Radley
>  Labels: OpenMetadata, VirtualDataConnector
> Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, 
> 0220CategoryHierarchy.json, 0230Terms.json, 0240Dictionary.json, 
> 0250RelatedTerms.json, 0260Contexts.json, 0270SemanticAssignment.json, 
> 0280SpineObjects.json
>
>
> This task delivers the JSON files for the new models that describe types for 
> Area 2 in the op

[jira] [Commented] (ATLAS-1839) Area 2 of the open metadata model

2017-07-20 Thread Nigel Jones (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094613#comment-16094613
 ] 

Nigel Jones commented on ATLAS-1839:


[~davidrad] the toxic combination support in ranger policies is primarily 
geared to controlling what a user may access, whilst the validation [~ivarea] 
is suggesting is primarily about creates and updates, ie defining the data 
model itself. That's not to say ranger couldn't do this (since it can address 
any operation such as a create) but I don't think that's ranger's intent. But I 
agree it's a fine line and could well vary significantly in different 
environments

As such I think it makes sense to define validation in atlas and be able to 
link to code artifacts, services that implement those validations probably 
through a combination of discovery & stewardship , plus making it easier when 
writing pipelines for say ETL or streaming, to be able to easily pull in atlas 
metadata and capture a link between a validation implemented by an pipeline 
author (or being used from a library) and it's definition in atlas. Thus atlas 
ends up with both the "intent" (the business spec if you like) as well as links 
to the implementation yet does not constrain those implementations since they 
can be so varied. 

Following on from this, absolutely some of those validations could be 
implemented as complex rules, but I think it would be tricky and constraining 
to capture all that in atlas, hence why I'd go for the link approach & some 
relatively loose coupling

So with that done, sure we could have a more complex rules engine embedded in, 
or used by ranger plugins... but this could be one of a number of different 
approaches

I'd be inclined to start off with us figuring out how to model, and some use 
cases where we can explore the authoring (ie in atlas), assisted authoring 
(when writing a job), metadata capture (from those other systems, also relates 
to lineage) & probably best to do that in ATLAS-1995? This also touches on 
RANGER-1869 (metadata capture)

Certainly this is an interesting area !

> Area 2 of the open metadata model
> -
>
> Key: ATLAS-1839
> URL: https://issues.apache.org/jira/browse/ATLAS-1839
> Project: Atlas
>  Issue Type: Task
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: David Radley
>  Labels: OpenMetadata, VirtualDataConnector
> Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, 
> 0220CategoryHierarchy.json, 0230Terms.json, 0240Dictionary.json, 
> 0250RelatedTerms.json, 0260Contexts.json, 0270SemanticAssignment.json, 
> 0280SpineObjects.json
>
>
> This task delivers the JSON files for the new models that describe types for 
> Area 2 in the open metadata model. This area covers the glossary.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Issue Comment Deleted] (ATLAS-1969) Cannot create a second child relationship

2017-07-20 Thread David Radley (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-1969:

Comment: was deleted

(was: On the latest code I create 3 categories A B and C. I can successfully 
create a relationship with type  CategoryHierarchyLink with catA as the parent 
and CatB as the child.. I then try to try to create a relationship with type  
CategoryHierarchyLink with catA as the parent and CatC as the child; and it 
fails with 
2017-07-20 12:14:51,188 ERROR - [pool-1-thread-7 - 
7d5fbe39-0e26-4af0-9137-651b51f9a69c:] ~ graph rollback due to exception  
(GraphTransactionInterceptor:71)
org.apache.atlas.exception.AtlasBaseException: invalid parameters: found null 
entity
at 
org.apache.atlas.repository.store.graph.v1.AtlasEntityGraphDiscoveryV1.discover(AtlasEntityGraphDiscoveryV1.java:139)
at 
org.apache.atlas.repository.store.graph.v1.AtlasEntityGraphDiscoveryV1.discoverEntities(AtlasEntityGraphDiscoveryV1.java:69)
at 
org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.preCreateOrUpdate(AtlasEntityStoreV1.java:612)
at org)

> Cannot create a second child relationship
> -
>
> Key: ATLAS-1969
> URL: https://issues.apache.org/jira/browse/ATLAS-1969
> Project: Atlas
>  Issue Type: Bug
>Reporter: David Radley
> Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, 
> 0220CategoryHierarchy.json
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (ATLAS-1969) Cannot create a second child relationship

2017-07-20 Thread David Radley (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-1969:

Description: 
On the latest code I create 3 categories A B and C. I can successfully create a 
relationship with type CategoryHierarchyLink with catA as the parent and CatB 
as the child.. I then try to try to create a relationship with type 
CategoryHierarchyLink with catA as the parent and CatC as the child; and it 
fails with
2017-07-20 12:14:51,188 ERROR - [pool-1-thread-7 - 
7d5fbe39-0e26-4af0-9137-651b51f9a69c:] ~ graph rollback due to exception 
(GraphTransactionInterceptor:71)
org.apache.atlas.exception.AtlasBaseException: invalid parameters: found null 
entity
at 
org.apache.atlas.repository.store.graph.v1.AtlasEntityGraphDiscoveryV1.discover(AtlasEntityGraphDiscoveryV1.java:139)
at 
org.apache.atlas.repository.store.graph.v1.AtlasEntityGraphDiscoveryV1.discoverEntities(AtlasEntityGraphDiscoveryV1.java:69)
at 
org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.preCreateOrUpdate(AtlasEntityStoreV1.java:612)
at org

> Cannot create a second child relationship
> -
>
> Key: ATLAS-1969
> URL: https://issues.apache.org/jira/browse/ATLAS-1969
> Project: Atlas
>  Issue Type: Bug
>Reporter: David Radley
> Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, 
> 0220CategoryHierarchy.json
>
>
> On the latest code I create 3 categories A B and C. I can successfully create 
> a relationship with type CategoryHierarchyLink with catA as the parent and 
> CatB as the child.. I then try to try to create a relationship with type 
> CategoryHierarchyLink with catA as the parent and CatC as the child; and it 
> fails with
> 2017-07-20 12:14:51,188 ERROR - [pool-1-thread-7 - 
> 7d5fbe39-0e26-4af0-9137-651b51f9a69c:] ~ graph rollback due to exception 
> (GraphTransactionInterceptor:71)
> org.apache.atlas.exception.AtlasBaseException: invalid parameters: found null 
> entity
> at 
> org.apache.atlas.repository.store.graph.v1.AtlasEntityGraphDiscoveryV1.discover(AtlasEntityGraphDiscoveryV1.java:139)
> at 
> org.apache.atlas.repository.store.graph.v1.AtlasEntityGraphDiscoveryV1.discoverEntities(AtlasEntityGraphDiscoveryV1.java:69)
> at 
> org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.preCreateOrUpdate(AtlasEntityStoreV1.java:612)
> at org



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ATLAS-1969) Cannot create a second child relationship

2017-07-20 Thread David Radley (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-1969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094558#comment-16094558
 ] 

David Radley commented on ATLAS-1969:
-

On the latest code I create 3 categories A B and C. I can successfully create a 
relationship with type  CategoryHierarchyLink with catA as the parent and CatB 
as the child.. I then try to try to create a relationship with type  
CategoryHierarchyLink with catA as the parent and CatC as the child; and it 
fails with 
2017-07-20 12:14:51,188 ERROR - [pool-1-thread-7 - 
7d5fbe39-0e26-4af0-9137-651b51f9a69c:] ~ graph rollback due to exception  
(GraphTransactionInterceptor:71)
org.apache.atlas.exception.AtlasBaseException: invalid parameters: found null 
entity
at 
org.apache.atlas.repository.store.graph.v1.AtlasEntityGraphDiscoveryV1.discover(AtlasEntityGraphDiscoveryV1.java:139)
at 
org.apache.atlas.repository.store.graph.v1.AtlasEntityGraphDiscoveryV1.discoverEntities(AtlasEntityGraphDiscoveryV1.java:69)
at 
org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.preCreateOrUpdate(AtlasEntityStoreV1.java:612)
at org

> Cannot create a second child relationship
> -
>
> Key: ATLAS-1969
> URL: https://issues.apache.org/jira/browse/ATLAS-1969
> Project: Atlas
>  Issue Type: Bug
>Reporter: David Radley
> Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, 
> 0220CategoryHierarchy.json
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (ATLAS-1969) Cannot create a second child relationship

2017-07-20 Thread David Radley (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-1969:

Attachment: 0005LinkedMediaTypes.json
0210Glossary.json
0220CategoryHierarchy.json

> Cannot create a second child relationship
> -
>
> Key: ATLAS-1969
> URL: https://issues.apache.org/jira/browse/ATLAS-1969
> Project: Atlas
>  Issue Type: Bug
>Reporter: David Radley
> Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, 
> 0220CategoryHierarchy.json
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (ATLAS-1969) Cannot create a second child relationship

2017-07-20 Thread David Radley (JIRA)

David Radley created ATLAS-1969:
---

 Summary: Cannot create a second child relationship
 Key: ATLAS-1969
 URL: https://issues.apache.org/jira/browse/ATLAS-1969
 Project: Atlas
  Issue Type: Bug
Reporter: David Radley






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (ATLAS-1963) Relationship instance creation does not result in the expected entity instance content

2017-07-20 Thread David Radley (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley resolved ATLAS-1963.
-
Resolution: Duplicate

> Relationship instance creation does not result in the expected entity 
> instance content 
> ---
>
> Key: ATLAS-1963
> URL: https://issues.apache.org/jira/browse/ATLAS-1963
> Project: Atlas
>  Issue Type: Bug
>Reporter: David Radley
> Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, 
> 0220CategoryHierarchy.json
>
>
> I create types using the attached  json files. I then create 2 
> GlossaryCategory instances. 
> I then create a relationship from one category to another with 
> CategoryHierarchyLink.
> {
> "end1": {
> "guid" : "12bcfbd3-1574-452f-8139-4d14147fa56b",
> "typeName" : "GlossaryCategory"
> },
> "end2": {
> "guid" : "1b7b5834-abc6-4ea4-bb6e-b10ae066724e",
> "typeName" : "GlossaryCategory"
> },
> "typeName"  : "CategoryHierarchyLink"
> } 
> I get one of the categories and do not see the populated parent or children 
> category relationship in the relationship attributes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (ATLAS-1968) Export/Import - Exception thrown when Import fired with zip file found on server

2017-07-20 Thread Sharmadha Sainath (JIRA)

Sharmadha Sainath created ATLAS-1968:


 Summary: Export/Import -  Exception thrown when Import fired with 
zip file found on server
 Key: ATLAS-1968
 URL: https://issues.apache.org/jira/browse/ATLAS-1968
 Project: Atlas
  Issue Type: Bug
  Components:  atlas-core
Affects Versions: 0.9-incubating, 0.8.1-incubating
Reporter: Sharmadha Sainath
Priority: Blocker
 Attachments: ImportExceptionWhenZipFileOnServer.txt

1. Exported an entity on the cluster1 ,created zip file and placed it in /tmp 
location of cluster2 .
2. Fired the following import command against cluster2 :

{code}
curl -v -X POST -u admin:admin  
"http://cluster2:21000/api/atlas/admin/importFile?FILENAME=/tmp/entity.zip"; 
{code}

The import command failed with 500 internal server error.

Attached the exception stack trace found in cluster2's application logs.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ATLAS-1927) UI - changing the look and feel of attribute in details page.

2017-07-20 Thread Kalyani Kashikar (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094479#comment-16094479
 ] 

Kalyani Kashikar commented on ATLAS-1927:
-

Thank Madhan Neethiraj for review and commit.
* Committed to master - 
http://git-wip-us.apache.org/repos/asf/incubator-atlas/commit/71f9c065
* Committed to 0.8 -   
http://git-wip-us.apache.org/repos/asf/incubator-atlas/commit/6d062c6b

> UI - changing the look and feel of attribute in details page.
> -
>
> Key: ATLAS-1927
> URL: https://issues.apache.org/jira/browse/ATLAS-1927
> Project: Atlas
>  Issue Type: Bug
>Reporter: Kalyani Kashikar
>Assignee: Kalyani Kashikar
> Attachments: ATLAS-1927.1.patch, ATLAS-1927.2.patch, 
> ATLAS-1927.3.patch, ATLAS-1927.4.patch, ATLAS-1927.patch
>
>
> Show the attributes of entities in tag format instead of showing comma(,) 
> separator.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 60938: [ATLAS-1959]: Enhance relationship attributes to support different cardinality mappings

2017-07-20 Thread David Radley

> On July 19, 2017, 3:11 p.m., David Radley wrote:
> > addons/models/0080-storm_model.json
> > Line 150 (original), 150 (patched)
> > 
> >
> > shouldn't this be aggregation ?
> 
> Sarath Subramanian wrote:
> since a storm node can be shared acoss multiple storm topologies and not 
> tied to a single topology made it as ASSOCIATION

It seems better to have this as aggregation to should some element of 
containment. From your description , the topology contains the storm node. The 
storm node could also be contained in otehr topologies . If htis is the case - 
it should be AGGREGATION I think.

> On July 19, 2017, 3:11 p.m., David Radley wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
> > Lines 256 (patched)
> > 
> >
> > Does the API still allow creation of entities with constaints - how 
> > will this come through to thie code.
> 
> Sarath Subramanian wrote:
> yes, to support backward compatibility entity creation with constraints 
> is still allowed.

Thanks Sarath for the explanation. I think I get it. I wonder if there is a 
cleaner way we can design this.   

What I do not like about this design is that the user could specify 
RelationshipDefs that do not quite match the existing constraint definitions. 
Also I am not keen on an flag with legacy in - as todays new functions is 
tomorrow legacy. the models also look messy with all the duplication.   

I assume for the 'legacy case', the requirement is to be able to specify 
propagate tags- I assume there is no compelling requirement to add 
relationshipDef attributes to the old API form.

If this is the case, I suggest we remove the legacy flag from RelationshipDef 
and remove the duplicate RelationshipDefs from the models. 
 Instead we should add a flag to the constraint API form as an indicator to 
propagate tags. This would be so much cleaner - and would not need to expose 
anything legacy in the RelationshipDef API and would mean that the user could 
not mismatch relationshipDefs with existing entityDef attributeDef constraint 
definitions. The model files would also not have all the duplication that they 
currently have.

- David

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60938/#review180926
---

On July 19, 2017, 7:27 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60938/
> ---
> 
> (Updated July 19, 2017, 7:27 p.m.)
> 
> 
> Review request for atlas and Madhan Neethiraj.
> 
> 
> Bugs: ATLAS-1959
> https://issues.apache.org/jira/browse/ATLAS-1959
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> * Improve relationship model to support create/update operations and the 
> following cardinalities (previously supported using inverseReference):
> 1 to 1
> 1 to many
> many to 1
> many to many
> 
> 
> * Change legacyLabel flag in AtlasRelationshipEndDef to boolean flag.
> * Add unit tests for the above cases.
> 
> 
> Diffs
> -
> 
>   addons/models/0010-base_model.json 303f3796 
>   addons/models/0030-hive_model.json a795f0f3 
>   addons/models/0050-falcon_model.json 7755fa86 
>   addons/models/0060-hbase_model.json 1d264df4 
>   addons/models/0080-storm_model.json 25360ff0 
>   intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java 
> 68da6af1 
>   intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationship.java 
> 2de9bdf0 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasRelationshipDef.java 
> fc820d49 
>   
> intg/src/main/java/org/apache/atlas/model/typedef/AtlasRelationshipEndDef.java
>  f80ea895 
>   intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java e94dd190 
>   intg/src/main/java/org/apache/atlas/type/AtlasRelationshipType.java 
> 841b66f7 
>   intg/src/main/java/org/apache/atlas/type/AtlasStructType.java f97d7674 
>   intg/src/test/java/org/apache/atlas/TestRelationshipUtilsV2.java 
> PRE-CREATION 
>   intg/src/test/java/org/apache/atlas/TestUtilsV2.java 9774583d 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 6f6d74bc 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityGraphDiscoveryV1.java
>  12e8bb1f 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasGraphUtilsV1.java
>  cd9a47ad 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1.java
>  3ff6fbef 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  d4fdc257

Unsubscribe me from Mailing List

2017-07-20 Thread naseem rafique

Hi,

Please Unsubscribe me from the mailing list.

Thanks.

[jira] [Created] (ATLAS-1967) Search UI : Render attribute filter based on browser URL

2017-07-20 Thread Keval Bhatt (JIRA)

Keval Bhatt created ATLAS-1967:
--

 Summary: Search UI : Render attribute filter based on browser URL
 Key: ATLAS-1967
 URL: https://issues.apache.org/jira/browse/ATLAS-1967
 Project: Atlas
  Issue Type: Improvement
Affects Versions: 0.9-incubating
Reporter: Keval Bhatt
Assignee: Keval Bhatt






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 60980: ATLAS-1961: Basic search improvement in use of Solr index for attribute filtering (# 3)

2017-07-20 Thread Madhan Neethiraj


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60980/
---

(Updated July 20, 2017, 8:31 a.m.)


Review request for atlas.


Changes
---

updated to choose appropriate start-offset, to handle 'state filter' (to 
exclude deleted entities)


Bugs: ATLAS-1961
https://issues.apache.org/jira/browse/ATLAS-1961


Repository: atlas


Description
---

- fixed NPE when search includes a free-text query & a classification that has 
large number of sub-types
- fixed entity search in Solr to include state filter


Diffs (updated)
-

  
repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java
 b6e0de5c 
  
repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 
6f629eb4 
  
repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java
 22d91e04 
  repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
2e75dfeb 


Diff: https://reviews.apache.org/r/60980/diff/3/

Changes: https://reviews.apache.org/r/60980/diff/2-3/


Testing
---

- validated that the fix addressed the error cases listed above


Thanks,

Madhan Neethiraj

[jira] [Commented] (ATLAS-1839) Area 2 of the open metadata model

2017-07-20 Thread Israel Varea (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094313#comment-16094313
 ] 

Israel Varea commented on ATLAS-1839:
-

I can think in a more complex validation when it depends on a different 
attribute. A simple example is when we want to model an Event, with two 
attributes, start_time, end_time. The validation of end_time should be that 
start_time <= end_time.

However, the most common are pattern matching by regular expressions and 
validations from a reference table columns (e.g. country codes).

> Area 2 of the open metadata model
> -
>
> Key: ATLAS-1839
> URL: https://issues.apache.org/jira/browse/ATLAS-1839
> Project: Atlas
>  Issue Type: Task
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: David Radley
>  Labels: OpenMetadata, VirtualDataConnector
> Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, 
> 0220CategoryHierarchy.json, 0230Terms.json, 0240Dictionary.json, 
> 0250RelatedTerms.json, 0260Contexts.json, 0270SemanticAssignment.json, 
> 0280SpineObjects.json
>
>
> This task delivers the JSON files for the new models that describe types for 
> Area 2 in the open metadata model. This area covers the glossary.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

64 matches

Mail list logo