[
https://issues.apache.org/jira/browse/ATLAS-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066140#comment-16066140
]
Péter Gergő Barna edited comment on ATLAS-1879 at 6/28/17 1:05 PM:
-------------------------------------------------------------------
[~madhan.neethiraj],
I understand the problem is the updateTypesDef method updates the TypeRegistry
first, then the GraphStorage. First the TypeRegistry is updated with the user
input, which does not necessarily contain all fields. Then the updateGraphStore
method updates the storage, and this method does not updates missing fields
back into the registry.
repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java
{noformat}
@Override
@GraphTransaction
public AtlasTypesDef updateTypesDef(AtlasTypesDef typesDef) throws
AtlasBaseException {
if (LOG.isDebugEnabled()) {
LOG.debug("==> AtlasTypeDefGraphStore.updateTypesDef(enums={},
structs={}, classfications={}, entities={})",
CollectionUtils.size(typesDef.getEnumDefs()),
CollectionUtils.size(typesDef.getStructDefs()),
CollectionUtils.size(typesDef.getClassificationDefs()),
CollectionUtils.size(typesDef.getEntityDefs()));
}
AtlasTransientTypeRegistry ttr = lockTypeRegistryAndReleasePostCommit();
// Translate any NOT FOUND errors to BAD REQUEST
try {
ttr.updateTypes(typesDef);
} catch (AtlasBaseException e) {
if (AtlasErrorCode.TYPE_NAME_NOT_FOUND == e.getAtlasErrorCode()) {
throw new AtlasBaseException(AtlasErrorCode.BAD_REQUEST,
e.getMessage());
} else {
throw e;
}
}
AtlasTypesDef ret = updateGraphStore(typesDef, ttr);
if (LOG.isDebugEnabled()) {
LOG.debug("<== AtlasTypeDefGraphStore.updateTypesDef(enums={},
structs={}, classfications={}, entities={})",
CollectionUtils.size(typesDef.getEnumDefs()),
CollectionUtils.size(typesDef.getStructDefs()),
CollectionUtils.size(typesDef.getClassificationDefs()),
CollectionUtils.size(typesDef.getEntityDefs()));
}
return ret;
{noformat}
I figured, a possible simple solution would be the GraphStorage being updated
first, and then the TypeRegistry:
{noformat}
AtlasTypesDef ret = updateGraphStore(typesDef, ttr);
ttr.updateTypes(ret);
{noformat}
Unfortunately, that will cause a NullPointerException:
{noformat}
2017-06-26 19:42:59,802 ERROR - [pool-1-thread-7 - /api/atlas/v2/types/typedefs
- 23dc280b-d4e9-4c98-9e80-5496b77a98e3:] ~ graph rollback due to exception
(GraphTransactionInterceptor:71)
java.lang.NullPointerException
at
org.apache.atlas.repository.store.graph.v1.AtlasStructDefStoreV1.toJsonFromAttribute(AtlasStructDefStoreV1.java:508)
at
org.apache.atlas.repository.store.graph.v1.AtlasStructDefStoreV1.updateVertexPreUpdate(AtlasStructDefStoreV1.java:448)
at
org.apache.atlas.repository.store.graph.v1.AtlasClassificationDefStoreV1.updateVertexPreUpdate(AtlasClassificationDefStoreV1.java:365)
at
org.apache.atlas.repository.store.graph.v1.AtlasClassificationDefStoreV1.updateByName(AtlasClassificationDefStoreV1.java:215)
at
org.apache.atlas.repository.store.graph.v1.AtlasClassificationDefStoreV1.update(AtlasClassificationDefStoreV1.java:184)
at
org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStore.updateGraphStore(AtlasTypeDefGraphStore.java:793)
at
org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStore.updateTypesDef(AtlasTypeDefGraphStore.java:372)
at
org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStore$$FastClassBySpringCGLIB$$5226c80b.invoke(<generated>)
at
org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at
org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:60)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at
org.apache.atlas.repository.store.graph.v1.AtlasTypeDefGraphStoreV1$$EnhancerBySpringCGLIB$$f4b1ae3b.updateTypesDef(<generated>)
at
org.apache.atlas.web.rest.TypesREST.updateAtlasTypeDefs(TypesREST.java:326)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
...
{noformat}
I made some tests and I found, that if TypeRegistry update is called both
before and after the GraphStorage update, than it will fix the issue. Of course
that would be a silly way to fix this problem. Must go deeper...
{noformat}
AtlasTransientTypeRegistry ttr = lockTypeRegistryAndReleasePostCommit();
// Translate any NOT FOUND errors to BAD REQUEST
try {
ttr.updateTypes(typesDef);
} catch (AtlasBaseException e) {
if (AtlasErrorCode.TYPE_NAME_NOT_FOUND == e.getAtlasErrorCode()) {
throw new AtlasBaseException(AtlasErrorCode.BAD_REQUEST,
e.getMessage());
} else {
throw e;
}
}
AtlasTypesDef ret = updateGraphStore(typesDef, ttr);
ttr.updateTypes(ret);
{noformat}
I also noticed, that the addToGraphStore method (used during creation of a new
type) calls an updateGuid method on the TypeRegistry:
{noformat}
ttr.updateGuid(createdDef.getName(), createdDef.getGuid());
{noformat}
Calling updateGuid in the updateGraphStore method (or in the updateTypesDef
method) would possibly fix the Guid problem. I am looking into a similar
solution for the createTime, updateTime, etc. fields.
Conclusion:
I am still looking into this to find an elegant solution, any comments or hints
are welcome, thx
was (Author: bpgergo):
[~madhan.neethiraj],
I understand the problem may be in the updateTypesDef method:
(repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java)
{noformat}
@Override
@GraphTransaction
public AtlasTypesDef updateTypesDef(AtlasTypesDef typesDef) throws
AtlasBaseException {
if (LOG.isDebugEnabled()) {
LOG.debug("==> AtlasTypeDefGraphStore.updateTypesDef(enums={},
structs={}, classfications={}, entities={})",
CollectionUtils.size(typesDef.getEnumDefs()),
CollectionUtils.size(typesDef.getStructDefs()),
CollectionUtils.size(typesDef.getClassificationDefs()),
CollectionUtils.size(typesDef.getEntityDefs()));
}
AtlasTransientTypeRegistry ttr = lockTypeRegistryAndReleasePostCommit();
// Translate any NOT FOUND errors to BAD REQUEST
try {
ttr.updateTypes(typesDef);
} catch (AtlasBaseException e) {
if (AtlasErrorCode.TYPE_NAME_NOT_FOUND == e.getAtlasErrorCode()) {
throw new AtlasBaseException(AtlasErrorCode.BAD_REQUEST,
e.getMessage());
} else {
throw e;
}
}
AtlasTypesDef ret = updateGraphStore(typesDef, ttr);
if (LOG.isDebugEnabled()) {
LOG.debug("<== AtlasTypeDefGraphStore.updateTypesDef(enums={},
structs={}, classfications={}, entities={})",
CollectionUtils.size(typesDef.getEnumDefs()),
CollectionUtils.size(typesDef.getStructDefs()),
CollectionUtils.size(typesDef.getClassificationDefs()),
CollectionUtils.size(typesDef.getEntityDefs()));
}
return ret;
{noformat}
This method updates the TypeRegistry first, then the GraphStorage.
The TypeRegistry is updated with the user input, which does not necessarily
contain all fields.
I figured, maybe the GraphStorage should be updated first, and then the
TypeRegistry:
{noformat}
AtlasTypesDef ret = updateGraphStore(typesDef, ttr);
ttr.updateTypes(ret);
{noformat}
Unfortunately, it will cause a NullPointerException:
{noformat}
2017-06-26 19:42:59,802 ERROR - [pool-1-thread-7 - /api/atlas/v2/types/typedefs
- 23dc280b-d4e9-4c98-9e80-5496b77a98e3:] ~ graph rollback due to exception
(GraphTransactionInterceptor:71)
java.lang.NullPointerException
at
org.apache.atlas.repository.store.graph.v1.AtlasStructDefStoreV1.toJsonFromAttribute(AtlasStructDefStoreV1.java:508)
at
org.apache.atlas.repository.store.graph.v1.AtlasStructDefStoreV1.updateVertexPreUpdate(AtlasStructDefStoreV1.java:448)
at
org.apache.atlas.repository.store.graph.v1.AtlasClassificationDefStoreV1.updateVertexPreUpdate(AtlasClassificationDefStoreV1.java:365)
at
org.apache.atlas.repository.store.graph.v1.AtlasClassificationDefStoreV1.updateByName(AtlasClassificationDefStoreV1.java:215)
at
org.apache.atlas.repository.store.graph.v1.AtlasClassificationDefStoreV1.update(AtlasClassificationDefStoreV1.java:184)
at
org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStore.updateGraphStore(AtlasTypeDefGraphStore.java:793)
at
org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStore.updateTypesDef(AtlasTypeDefGraphStore.java:372)
at
org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStore$$FastClassBySpringCGLIB$$5226c80b.invoke(<generated>)
at
org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at
org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:60)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at
org.apache.atlas.repository.store.graph.v1.AtlasTypeDefGraphStoreV1$$EnhancerBySpringCGLIB$$f4b1ae3b.updateTypesDef(<generated>)
at
org.apache.atlas.web.rest.TypesREST.updateAtlasTypeDefs(TypesREST.java:326)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
...
{noformat}
I made some tests and I found, that if TypeRegistry update is called both
before and after the GraphStorage update, than it will fix the issue:
{noformat}
AtlasTransientTypeRegistry ttr = lockTypeRegistryAndReleasePostCommit();
// Translate any NOT FOUND errors to BAD REQUEST
try {
ttr.updateTypes(typesDef);
} catch (AtlasBaseException e) {
if (AtlasErrorCode.TYPE_NAME_NOT_FOUND == e.getAtlasErrorCode()) {
throw new AtlasBaseException(AtlasErrorCode.BAD_REQUEST,
e.getMessage());
} else {
throw e;
}
}
AtlasTypesDef ret = updateGraphStore(typesDef, ttr);
ttr.updateTypes(ret);
{noformat}
I am still looking into this to find a more elegant solution, any comments or
hints are welcome, thx
> Updating classification removes some properties
> -----------------------------------------------
>
> Key: ATLAS-1879
> URL: https://issues.apache.org/jira/browse/ATLAS-1879
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
> Affects Versions: 0.8-incubating
> Reporter: Laura Ngo
> Attachments: Atlas-1789.postman_collection.json
>
>
> * Created classification via POST.
> * Updated via PUT
> * Lost properties
> POST http://127.0.0.1:21000/api/atlas/v2/types/typedefs
> {code}
> {
> "classificationDefs": [{
> "name": "test_classification_11",
> "description": "",
> "createdBy" : "admin",
> "superTypes": [],
> "attributeDefs": [{
> "name" : "test_class_11",
> "typeName" : "string",
> "isOptional" : true,
> "isUnique" : true,
> "isIndexable" : true,
> "cardinality": "SINGLE",
> "valuesMinCount": 0,
> "valuesMaxCount": 1
> }]
> }],
> "entityDefs": [],
> "enumDefs": [],
> "structDefs": []
> }
> {code}
> GET
> http://127.0.0.1:21000/api/atlas/v2/types/classification/name/test_classification_11
> {code}
> {
> "category": "CLASSIFICATION",
> "guid": "83162fe1-4bb4-4a87-b2b8-364e751a1265",
> "createdBy": "admin",
> "createTime": 1497485890857,
> "updateTime": 1497485890857,
> "version": 1,
> "name": "test_classification_11",
> "description": "",
> "typeVersion": "1.0",
> "attributeDefs": [
> {
> "name": "test_class_11",
> "typeName": "string",
> "isOptional": true,
> "cardinality": "SINGLE",
> "valuesMinCount": 0,
> "valuesMaxCount": 1,
> "isUnique": true,
> "isIndexable": true
> }
> ],
> "superTypes": [],
> }
> {code}
> PUT http://127.0.0.1:21000/api/atlas/v2/types/typedefs
> Update attribute.
> GET
> http://127.0.0.1:21000/api/atlas/v2/types/classification/name/test_classification_11
> {code}
> {
> "category": "CLASSIFICATION",
> "createdBy": "admin",
> "name": "test_classification_11",
> "description": "",
> "attributeDefs": [
> {
> "name": "test_class_11",
> "typeName": "string",
> "isOptional": true,
> "cardinality": "SINGLE",
> "valuesMinCount": 0,
> "valuesMaxCount": 1,
> "isUnique": true,
> "isIndexable": false
> }
> ],
> "superTypes": [],
> }
> {code}
> Some properties are missing after PUT update of attribute "isIndexable"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
