[jira] [Commented] (ATLAS-158) Provide Atlas Entity Change Notification
[ https://issues.apache.org/jira/browse/ATLAS-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001788#comment-15001788 ] liutongfeng commented on ATLAS-158: --- Yes, it's my environment cause the issue. I use the vagrant, and put the atlas in the mounted directory, the issue will occur, but put the atlas int the vm's directory, it will be ok. Thanks! > Provide Atlas Entity Change Notification > > > Key: ATLAS-158 > URL: https://issues.apache.org/jira/browse/ATLAS-158 > Project: Atlas > Issue Type: Task >Reporter: Tom Beerbower >Assignee: Tom Beerbower > Fix For: 0.6-incubating > > Attachments: ATLAS-158-13.patch, ATLAS-158.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-158) Provide Atlas Entity Change Notification
[ https://issues.apache.org/jira/browse/ATLAS-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001162#comment-15001162 ] Tom Beerbower commented on ATLAS-158: - [~ltfxyz], It looks like the issue running quick start has been fixed with https://issues.apache.org/jira/browse/ATLAS-296. I can run quick start cleanly now. Thanks! > Provide Atlas Entity Change Notification > > > Key: ATLAS-158 > URL: https://issues.apache.org/jira/browse/ATLAS-158 > Project: Atlas > Issue Type: Task >Reporter: Tom Beerbower >Assignee: Tom Beerbower > Fix For: 0.6-incubating > > Attachments: ATLAS-158-13.patch, ATLAS-158.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-158) Provide Atlas Entity Change Notification
[
https://issues.apache.org/jira/browse/ATLAS-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000297#comment-15000297
]
liutongfeng commented on ATLAS-158:
---
Error occurred when running the quickstart.
...
Submitting new entity= {
"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
"id":{
"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
"id":"-815787746768834",
"version":0,
"typeName":"DB"
},
"typeName":"DB",
"values":{
"name":"Sales",
"createTime":1447244389968,
"description":"Sales Database",
"locationUri":"hdfs://host:8000/apps/warehouse/sales",
"owner":"John ETL"
},
"traitNames":[
],
"traits":{
}
}
Exception in thread "main" org.apache.atlas.AtlasServiceException: Metadata
service API CREATE_ENTITY failed with status 400(Bad Request) Response Body
({"error":"java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is
not the leader for that
topic-partition.","stackTrace":"org.apache.atlas.notification.NotificationException:
java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is
not the leader for that topic-partition.\n\tat
org.apache.atlas.kafka.KafkaNotification.send(KafkaNotification.java:196)\n\tat
org.apache.atlas.notification.entity.NotificationEntityChangeListener.notifyOfEntityEvent(NotificationEntityChangeListener.java:97)\n\tat
org.apache.atlas.notification.entity.NotificationEntityChangeListener.onEntitiesAdded(NotificationEntityChangeListener.java:62)\n\tat
org.apache.atlas.services.DefaultMetadataService.onEntitiesAddedToRepo(DefaultMetadataService.java:465)\n\tat
org.apache.atlas.services.DefaultMetadataService.createEntities(DefaultMetadataService.java:259)\n\tat
org.apache.atlas.web.resources.EntityResource.submit(EntityResource.java:113)\n\tat
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\n\tat
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat
java.lang.reflect.Method.invoke(Method.java:606)\n\tat
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)\n\tat
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)\n\tat
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)\n\tat
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)\n\tat
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)\n\tat
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)\n\tat
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)\n\tat
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)\n\tat
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)\n\tat
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)\n\tat
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)\n\tat
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)\n\tat
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)\n\tat
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)\n\tat
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)\n\tat
com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)\n\tat
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)\n\tat
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)\n\tat
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)\n\tat
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)\n\tat
org.apache.atlas.web.filters.AuditFilter.doFilter(AuditFilter.java:67)\n\tat
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)\n\tat
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)\n\tat
com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)\n\tat
com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)\n\tat
com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)\n\tat
com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)\n\tat
org.eclipse.
[jira] [Commented] (ATLAS-158) Provide Atlas Entity Change Notification
[ https://issues.apache.org/jira/browse/ATLAS-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14998803#comment-14998803 ] Shwetha G S commented on ATLAS-158: --- You can run individual IT - but some tests are not idempotent and required cleaning graph DB which is in target. https://maven.apache.org/surefire/maven-failsafe-plugin/examples/single-test.html > Provide Atlas Entity Change Notification > > > Key: ATLAS-158 > URL: https://issues.apache.org/jira/browse/ATLAS-158 > Project: Atlas > Issue Type: Task >Reporter: Tom Beerbower >Assignee: Tom Beerbower > Fix For: 0.6-incubating > > Attachments: ATLAS-158-13.patch, ATLAS-158.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-158) Provide Atlas Entity Change Notification
[ https://issues.apache.org/jira/browse/ATLAS-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14991724#comment-14991724 ] Tom Beerbower commented on ATLAS-158: - [~shwethags], Okay thanks. So if I understand, you must do a clean between runs and you can not run an individual integration test. If I run 'mvn clean package' from the top level followed by 'mvn install' from webapp, it will run all of the tests and show the single failure in NotificationHookConsumerIT. If I skip the clean build then I see other failures. I guess that is expected since the data directory is not cleaned up. If I try to run just the NotificationHookConsumerIT test (-Dtest=NotificationHookConsumerIT) then it runs the test but Atlas is not started so it fails. I'll look into the failure. Thanks again! > Provide Atlas Entity Change Notification > > > Key: ATLAS-158 > URL: https://issues.apache.org/jira/browse/ATLAS-158 > Project: Atlas > Issue Type: Task >Reporter: Tom Beerbower >Assignee: Tom Beerbower > Fix For: 0.6-incubating > > Attachments: ATLAS-158.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-158) Provide Atlas Entity Change Notification
[ https://issues.apache.org/jira/browse/ATLAS-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14991151#comment-14991151 ] Shwetha G S commented on ATLAS-158: --- Here is the jenkins build that ran the tests successfully with the committed code: https://builds.apache.org/job/apache-atlas-nightly/109/console > Provide Atlas Entity Change Notification > > > Key: ATLAS-158 > URL: https://issues.apache.org/jira/browse/ATLAS-158 > Project: Atlas > Issue Type: Task >Reporter: Tom Beerbower >Assignee: Tom Beerbower > Fix For: 0.6-incubating > > Attachments: ATLAS-158.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-158) Provide Atlas Entity Change Notification
[ https://issues.apache.org/jira/browse/ATLAS-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14991136#comment-14991136 ] Shwetha G S commented on ATLAS-158: --- All the tests succeed on a fresh checkout of code. So, I don't think there is any issue in the committed code. Please check the patch again > Provide Atlas Entity Change Notification > > > Key: ATLAS-158 > URL: https://issues.apache.org/jira/browse/ATLAS-158 > Project: Atlas > Issue Type: Task >Reporter: Tom Beerbower >Assignee: Tom Beerbower > Fix For: 0.6-incubating > > Attachments: ATLAS-158.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-158) Provide Atlas Entity Change Notification
[ https://issues.apache.org/jira/browse/ATLAS-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14991130#comment-14991130 ] Shwetha G S commented on ATLAS-158: --- The test classes that end with 'IT' are integration tests and integration tests don't run in 'test' lifecycle. https://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html has details of mvn lifecycles. You need to run mvn clean package/install to run all tests. The tests use target as the graph data directory. Make sure to use 'clean' goal to cleanup the data from previous build In the webapp module, jetty-maven-plugin runs in the pre integration test phase that starts Atlas server. Then the integration tests use this atlas server for end to end testing. So, you don't need to start atlas manually > Provide Atlas Entity Change Notification > > > Key: ATLAS-158 > URL: https://issues.apache.org/jira/browse/ATLAS-158 > Project: Atlas > Issue Type: Task >Reporter: Tom Beerbower >Assignee: Tom Beerbower > Fix For: 0.6-incubating > > Attachments: ATLAS-158.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-158) Provide Atlas Entity Change Notification
[
https://issues.apache.org/jira/browse/ATLAS-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14990756#comment-14990756
]
Tom Beerbower commented on ATLAS-158:
-
[~shwethags],
Okay, I can run the test if I manually start Atlas first. Is that the right
way to do it? I thought that would be part of the test setup and would happen
automatically when maven was run. Anyway, I now see ...
{code}
---
T E S T S
---
Picked up _JAVA_OPTIONS: -XX:MaxPermSize=128M -Xmx4096m -Djava.awt.headless=true
Running org.apache.atlas.notification.NotificationHookConsumerIT
Tests run: 2, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 4.912 sec <<<
FAILURE! - in org.apache.atlas.notification.NotificationHookConsumerIT
testConsumeHookMessage(org.apache.atlas.notification.NotificationHookConsumerIT)
Time elapsed: 2.705 sec <<< FAILURE!
java.lang.Exception: Waiting timed out after 1000 msec
at
org.apache.atlas.web.resources.BaseResourceIT.waitFor(BaseResourceIT.java:279)
at
org.apache.atlas.notification.NotificationHookConsumerIT.testConsumeHookMessage(NotificationHookConsumerIT.java:66)
Results :
Failed tests:
NotificationHookConsumerIT.testConsumeHookMessage:66->BaseResourceIT.waitFor:279
»
Tests run: 2, Failures: 1, Errors: 0, Skipped: 1
{code}
Is that what you are seeing?
> Provide Atlas Entity Change Notification
>
>
> Key: ATLAS-158
> URL: https://issues.apache.org/jira/browse/ATLAS-158
> Project: Atlas
> Issue Type: Task
>Reporter: Tom Beerbower
>Assignee: Tom Beerbower
> Fix For: 0.6-incubating
>
> Attachments: ATLAS-158.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ATLAS-158) Provide Atlas Entity Change Notification
[
https://issues.apache.org/jira/browse/ATLAS-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14990272#comment-14990272
]
Tom Beerbower commented on ATLAS-158:
-
[~shwethags],
Could you give more details on this?
{quote}
1. Debug NotificationHookConsumerIT failure
{quote}
If I do 'mvn test' from webapp, it succeeds but I don't think it is actually
running any tests.
Same thing if I do 'mvn test -Dtest=NotificationHookConsumerIT'.
The only way I can get it to actually run the test is with 'mvn install
-Dtest=NotificationHookConsumerIT'. In that case it fails with the output
below but does so whether I have the patch applied or not. Also that seems to
be the case for other tests as well ('mvn install
-Dtest=TypesJerseyResourceIT').
{code}
Picked up _JAVA_OPTIONS: -XX:MaxPermSize=128M -Xmx4096m -Djava.awt.headless=true
Running org.apache.atlas.notification.NotificationHookConsumerIT
Tests run: 4, Failures: 1, Errors: 0, Skipped: 3, Time elapsed: 2.314 sec <<<
FAILURE! - in org.apache.atlas.notification.NotificationHookConsumerIT
setUp(org.apache.atlas.notification.NotificationHookConsumerIT) Time elapsed:
2.266 sec <<< FAILURE!
com.sun.jersey.api.client.ClientHandlerException: java.io.IOException:
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.http.HttpClient.(HttpClient.java:211)
at sun.net.www.http.HttpClient.New(HttpClient.java:308)
at sun.net.www.http.HttpClient.New(HttpClient.java:326)
at
sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:997)
at
sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:933)
at
sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:851)
at
org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:76)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128)
at
org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:322)
at
org.apache.atlas.security.SecureClientUtils$1$1.run(SecureClientUtils.java:102)
at
org.apache.atlas.security.SecureClientUtils$1$1.run(SecureClientUtils.java:98)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.atlas.security.SecureClientUtils$1.getHttpURLConnection(SecureClientUtils.java:98)
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:159)
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
at com.sun.jersey.api.client.Client.handle(Client.java:648)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at
com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:623)
at
org.apache.atlas.AtlasClient.callAPIWithResource(AtlasClient.java:444)
at
org.apache.atlas.AtlasClient.callAPIWithResource(AtlasClient.java:439)
at org.apache.atlas.AtlasClient.getType(AtlasClient.java:196)
at
org.apache.atlas.web.resources.BaseResourceIT.createType(BaseResourceIT.java:85)
at
org.apache.atlas.web.resources.BaseResourceIT.createTypeDefinitions(BaseResourceIT.java:195)
at
org.apache.atlas.notification.NotificationHookConsumerIT.setUp(NotificationHookConsumerIT.java:42)
Results :
Failed tests:
NotificationHookConsumerIT.setUp:42->BaseResourceIT.createTypeDefinitions:195->BaseResourceIT.createType:85
» ClientHandler
Tests run: 4, Failures: 1, Errors: 0, Skipped: 3
{code}
> Provide Atlas Entity Change Notification
>
>
> Key: ATLAS-1
[jira] [Commented] (ATLAS-158) Provide Atlas Entity Change Notification
[ https://issues.apache.org/jira/browse/ATLAS-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14988893#comment-14988893 ] Shwetha G S commented on ATLAS-158: --- Attaching the patch here as I won't have permission to update the patch on review board. Changes done: 1. Removed dependency of repository module in notification. Moved listeners to common module 2. Added atlas.kafka.hook.group.id in distro/src/conf/application.properties - required for atlas deployment. The consumer group id changed to 'atlas' as the value should represent the consumer so that any debugging from kafka end can relate to the actual consumer. Also, removed the property atlas.kafka.entities.group.id to force users to set this value 3. Replaced TraitInfo with IStruct 4. Changed EntityNotification interface. I think the new interface makes sense for the following reasons: a. The entity returned IReferenceableInstance already has getTraits(). So, EntityNotification.getAllTraits() can represent the flattened traits(including super types) b. Traits as map doesn't work - Assume traits t1, t2, t3. traits t2 and t3 extend t1. If traits t2 and t3 are associated with the entity, getAllTraits() should return t1, t2, t1, t3. So, it has to be a list and not map c. For trait_add/delete operation, the traits added/deleted won't be a single trait, it will be multiple traits because of inheritance. And just the trait name is not enough as the same super trait may be shared across multiple traits. In general, its better to just give the whole entity and all its latest traits even in case of entity/trait add/update/delete. This will work even for ranger as ranger maintains resource-tag mapping and they can just replace all entity mappings. 5. Deleted AbstractNotificationConsumerTest and EntityNotificationImplTest as they weren't compiling with the new changes. They didn't really test anything much. If you want, you can add them back The patch compiles and all the tests pass except for NotificationHookConsumerIT. Looks like kafka doesn't stop/start clean. Pending are: 1. Debug NotificationHookConsumerIT failure 2. Add UT for EntityNotificationImpl.getAllTraits() 3. There are no tests at all for the end to end flow of entity notification. Please add ITs for all operation types 4. Add documentation I think without the ITs, its risky to commit the patch 2. > Provide Atlas Entity Change Notification > > > Key: ATLAS-158 > URL: https://issues.apache.org/jira/browse/ATLAS-158 > Project: Atlas > Issue Type: Task >Reporter: Tom Beerbower >Assignee: Tom Beerbower > Fix For: 0.6-incubating > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
