Review Request 70672: [ATLAS-3153] Add keycloak authentication
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70672/ --- Review request for atlas, Ashutosh Mestry and Madhan Neethiraj. Bugs: https://issues.apache.org/jira/browse/ATLAS-3153 https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3153 Repository: atlas Description --- Keycloak is an open source Identity and Access Management solution aimed at modern applications and services. It makes it easy to secure applications and services with little to no code. This enables Atlas to use OpenID Connect (OAUTH2) and allows integration with more services. Diffs - docs/src/site/twiki/Atlas-Authentication.twiki ddaa7fec2 pom.xml 7ad24ea5b webapp/pom.xml 994ee86f0 webapp/src/main/java/org/apache/atlas/web/security/AtlasAuthenticationProvider.java fb21d7563 webapp/src/main/java/org/apache/atlas/web/security/AtlasKeycloakAuthenticationProvider.java PRE-CREATION webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java bf6b85bea Diff: https://reviews.apache.org/r/70672/diff/1/ Testing --- Thanks, Bolke de Bruin
[jira] [Commented] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849115#comment-16849115 ] Bolke de Bruin commented on ATLAS-3153: --- Ping? > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement >Affects Versions: 2.0.0 > Reporter: Bolke de Bruin >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16855708#comment-16855708 ] Bolke de Bruin commented on ATLAS-3153: --- [~saqeeb.shaikh136] Let me verify. > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement > Components: atlas-core, atlas-webui >Affects Versions: 2.0.0 > Reporter: Bolke de Bruin >Priority: Major > Labels: authentication, authorization > Attachments: 0001-ATLAS-3153-Add-keycloak-authentication.patch, > application.log, keycloak.json, openid_connect_atlas.md > > Time Spent: 20m > Remaining Estimate: 0h > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850029#comment-16850029 ] Bolke de Bruin commented on ATLAS-3153: --- [~sarath.ku...@gmail.com] sure, can you explain what you would like to see in the design doc? Both OpenID connect and spring security are well understood. The roles / groups might be something as they can be obtained from keycloak instead of UGI. That’s also pretty straightforward. Can you give me some guidance? > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement >Affects Versions: 2.0.0 > Reporter: Bolke de Bruin >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850080#comment-16850080 ] Bolke de Bruin commented on ATLAS-3153: --- [~sarath.ku...@gmail.com] as mentioned this is already part of the PR? Isn’t the twiki used for this? > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement >Affects Versions: 2.0.0 > Reporter: Bolke de Bruin >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850144#comment-16850144 ] Bolke de Bruin commented on ATLAS-3153: --- [~sarath.ku...@gmail.com] I have added the file. Mostly copied from the PR with some additions. > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement >Affects Versions: 2.0.0 > Reporter: Bolke de Bruin >Priority: Major > Attachments: openid_connect_atlas.md > > Time Spent: 20m > Remaining Estimate: 0h > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3153: -- Attachment: openid_connect_atlas.md > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement >Affects Versions: 2.0.0 > Reporter: Bolke de Bruin >Priority: Major > Attachments: openid_connect_atlas.md > > Time Spent: 20m > Remaining Estimate: 0h > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852947#comment-16852947 ] Bolke de Bruin commented on ATLAS-3153: --- Ping? Can this be mergel now? > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement > Components: atlas-core, atlas-webui >Affects Versions: 2.0.0 > Reporter: Bolke de Bruin >Priority: Major > Labels: authentication, authorization > Attachments: 0001-ATLAS-3153-Add-keycloak-authentication.patch, > openid_connect_atlas.md > > Time Spent: 20m > Remaining Estimate: 0h > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3153: -- Component/s: atlas-webui atlas-core > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement > Components: atlas-core, atlas-webui >Affects Versions: 2.0.0 > Reporter: Bolke de Bruin >Priority: Major > Attachments: 0001-ATLAS-3153-Add-keycloak-authentication.patch, > openid_connect_atlas.md > > Time Spent: 20m > Remaining Estimate: 0h > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3153: -- Labels: authentication authorization (was: ) > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement > Components: atlas-core, atlas-webui >Affects Versions: 2.0.0 > Reporter: Bolke de Bruin >Priority: Major > Labels: authentication, authorization > Attachments: 0001-ATLAS-3153-Add-keycloak-authentication.patch, > openid_connect_atlas.md > > Time Spent: 20m > Remaining Estimate: 0h > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864630#comment-16864630 ] Bolke de Bruin commented on ATLAS-3153: --- Ping [~saqeeb.shaikh136] ? I really need some more info in order to reproduce > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement > Components: atlas-core, atlas-webui >Affects Versions: 2.0.0 > Reporter: Bolke de Bruin >Priority: Major > Labels: authentication, authorization > Attachments: 0001-ATLAS-3153-Add-keycloak-authentication.patch, > application.log, keycloak.json, openid_connect_atlas.md > > Time Spent: 20m > Remaining Estimate: 0h > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866719#comment-16866719 ] Bolke de Bruin commented on ATLAS-3153: --- Sure will do. We should have some overlap :) > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement > Components: atlas-core, atlas-webui >Affects Versions: 2.0.0 > Reporter: Bolke de Bruin >Priority: Major > Labels: authentication, authorization > Attachments: 0001-ATLAS-3153-Add-keycloak-authentication.patch, > application.log, keycloak.json, openid_connect_atlas.md > > Time Spent: 20m > Remaining Estimate: 0h > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16872629#comment-16872629 ] Bolke de Bruin commented on ATLAS-3153: --- [~saqeeb.shaikh136] both would need changes. Can I suggest doing that outside of this PR? I’m happy to do so, but it seem not the ‘same unit’ of work. > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement > Components: atlas-core, atlas-webui >Affects Versions: 2.0.0 > Reporter: Bolke de Bruin >Priority: Major > Labels: authentication, authorization > Attachments: 0001-ATLAS-3153-Add-keycloak-authentication.patch, > application.log, keycloak.json, openid_connect_atlas.md > > Time Spent: 20m > Remaining Estimate: 0h > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16870629#comment-16870629 ] Bolke de Bruin commented on ATLAS-3153: --- After our discussion I verified: * MIT kdc, with Kerberized Atlas * HDP 3.1, FreeIPA, with Kerberized Atlas both are working fine (ie. client is redirected). > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement > Components: atlas-core, atlas-webui >Affects Versions: 2.0.0 > Reporter: Bolke de Bruin >Priority: Major > Labels: authentication, authorization > Attachments: 0001-ATLAS-3153-Add-keycloak-authentication.patch, > application.log, keycloak.json, openid_connect_atlas.md > > Time Spent: 20m > Remaining Estimate: 0h > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16859322#comment-16859322 ] Bolke de Bruin commented on ATLAS-3153: --- [~saqeeb.shaikh136] can you share a bit more on the flow you did and your configuration? Im having difficulty replicating the behavior I think you are describing.I have tested this with a manually configured KDC. I do see that while a Kerberos credential can be available a redirect still happens due to the fact the Keycloak's filters are earlier in the chain. This is equal to Knox integration (I have never used Knox, but its filter as also earlier in the chain) it seems. In short I can turn on Kerberos and Keycloak and Atlas will always use Keycloak. > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement > Components: atlas-core, atlas-webui >Affects Versions: 2.0.0 > Reporter: Bolke de Bruin >Priority: Major > Labels: authentication, authorization > Attachments: 0001-ATLAS-3153-Add-keycloak-authentication.patch, > application.log, keycloak.json, openid_connect_atlas.md > > Time Spent: 20m > Remaining Estimate: 0h > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16859322#comment-16859322 ] Bolke de Bruin edited comment on ATLAS-3153 at 6/8/19 10:50 PM: [~saqeeb.shaikh136] can you share a bit more on the flow you did and your configuration? Im having difficulty replicating the behavior I think you are describing.I have tested this with a manually configured KDC. Considering that you are using "j_spring_security_check" [1] it seems you are submitting the form somehow, with a redirect that should not be possible? If I run [2] I get a redirect. Without "login.jsp" a "Negotiate" and then a redirect. I do see that while a Kerberos credential can be available a redirect still happens due to the fact the Keycloak's filters are earlier in the chain. This is equal to Knox integration (I have never used Knox, but its filter as also earlier in the chain) it seems. In short I can turn on Kerberos and Keycloak and Atlas will always use Keycloak. A session with debug logging configured for Keycloak's adapter might help. [1] adminRequest http://172.22.92.244:21000/j_spring_security_check (PreAuthActionsHandler:75) [2] curl --negotiate -u : -v http://atlas-dev.novalocal:21000/login.jsp was (Author: bolke): [~saqeeb.shaikh136] can you share a bit more on the flow you did and your configuration? Im having difficulty replicating the behavior I think you are describing.I have tested this with a manually configured KDC. I do see that while a Kerberos credential can be available a redirect still happens due to the fact the Keycloak's filters are earlier in the chain. This is equal to Knox integration (I have never used Knox, but its filter as also earlier in the chain) it seems. In short I can turn on Kerberos and Keycloak and Atlas will always use Keycloak. > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement > Components: atlas-core, atlas-webui > Affects Versions: 2.0.0 >Reporter: Bolke de Bruin >Priority: Major > Labels: authentication, authorization > Attachments: 0001-ATLAS-3153-Add-keycloak-authentication.patch, > application.log, keycloak.json, openid_connect_atlas.md > > Time Spent: 20m > Remaining Estimate: 0h > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 70672: [ATLAS-3153] Add keycloak authentication
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70672/ --- (Updated May 18, 2019, 6:40 p.m.) Review request for atlas, Ashutosh Mestry and Madhan Neethiraj. Bugs: https://issues.apache.org/jira/browse/ATLAS-3153 https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3153 Repository: atlas Description --- Keycloak is an open source Identity and Access Management solution aimed at modern applications and services. It makes it easy to secure applications and services with little to no code. This enables Atlas to use OpenID Connect (OAUTH2) and allows integration with more services. Diffs (updated) - docs/src/site/twiki/Atlas-Authentication.twiki ddaa7fec2 pom.xml 7ad24ea5b webapp/pom.xml 994ee86f0 webapp/src/main/java/org/apache/atlas/web/security/AtlasAuthenticationProvider.java fb21d7563 webapp/src/main/java/org/apache/atlas/web/security/AtlasKeycloakAuthenticationProvider.java PRE-CREATION webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java bf6b85bea Diff: https://reviews.apache.org/r/70672/diff/2/ Changes: https://reviews.apache.org/r/70672/diff/1-2/ Testing --- Thanks, Bolke de Bruin
[jira] [Commented] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846470#comment-16846470 ] Bolke de Bruin commented on ATLAS-3153: --- [~srikvenk] I have already included documentation in the PR (twiki) that describes this. Do you want me to extend that? We don't use Azure but the keycloak client should work with any oauth provider or (preferred) OpenID Connect (a layer on top of oauth). Azure supports both so with proper configuration in keycloak.json and maybe a mapper defined in Azure's service definition this should 'just' work. AuthN/Z are then both supported. If you disable Hadoop's UGI integration as documented you have roles/groups (exclusive this is a limitation of atlas at the moment not of keycloak/OpenID) > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement >Affects Versions: 2.0.0 > Reporter: Bolke de Bruin >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3153: -- Affects Version/s: (was: 1.1.0) 2.0.0 > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement >Affects Versions: 2.0.0 > Reporter: Bolke de Bruin >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3309) Support OIDC bearer token in QuickStart and Hive import
[ https://issues.apache.org/jira/browse/ATLAS-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16879035#comment-16879035 ] Bolke de Bruin commented on ATLAS-3309: --- Cc [~nixonrodrigues] > Support OIDC bearer token in QuickStart and Hive import > --- > > Key: ATLAS-3309 > URL: https://issues.apache.org/jira/browse/ATLAS-3309 > Project: Atlas > Issue Type: Improvement > Reporter: Bolke de Bruin >Priority: Major > Attachments: > 0002-ATLAS-3309-Add-support-for-OIDC-bearer-token-in-exam.patch > > > In ATLAS-3153 support for keycloak (OIDC) authentication was added. The > QuickStart examples and Hive import need to be updated to support bearer > token authentication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873590#comment-16873590 ] Bolke de Bruin commented on ATLAS-3153: --- [~saqeeb.shaikh136] I have created ATLAS-3309 to track this. Can this now be merged? > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement > Components: atlas-core, atlas-webui >Affects Versions: 2.0.0 > Reporter: Bolke de Bruin >Priority: Major > Labels: authentication, authorization > Attachments: 0001-ATLAS-3153-Add-keycloak-authentication.patch, > application.log, keycloak.json, openid_connect_atlas.md > > Time Spent: 20m > Remaining Estimate: 0h > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3309) Support OIDC bearer token in QuickStart and Hive import
[ https://issues.apache.org/jira/browse/ATLAS-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873589#comment-16873589 ] Bolke de Bruin commented on ATLAS-3309: --- Cc [~saqeeb.shaikh136] > Support OIDC bearer token in QuickStart and Hive import > --- > > Key: ATLAS-3309 > URL: https://issues.apache.org/jira/browse/ATLAS-3309 > Project: Atlas > Issue Type: Improvement > Reporter: Bolke de Bruin >Priority: Major > > In ATLAS-3153 support for keycloak (OIDC) authentication was added. The > QuickStart examples and Hive import need to be updated to support bearer > token authentication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3309) Support OIDC bearer token in QuickStart and Hive import
[ https://issues.apache.org/jira/browse/ATLAS-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3309: -- Description: In ATLAS-3153 support for keycloak (OIDC) authentication was added. The QuickStart examples and Hive import need to be updated to support bearer token authentication. Summary: Support OIDC bearer token in QuickStart and Hive import (was: Support OIDC bearer to) > Support OIDC bearer token in QuickStart and Hive import > --- > > Key: ATLAS-3309 > URL: https://issues.apache.org/jira/browse/ATLAS-3309 > Project: Atlas > Issue Type: Improvement > Reporter: Bolke de Bruin >Priority: Major > > In ATLAS-3153 support for keycloak (OIDC) authentication was added. The > QuickStart examples and Hive import need to be updated to support bearer > token authentication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ATLAS-3309) Support OIDC bearer to
Bolke de Bruin created ATLAS-3309: - Summary: Support OIDC bearer to Key: ATLAS-3309 URL: https://issues.apache.org/jira/browse/ATLAS-3309 Project: Atlas Issue Type: Improvement Reporter: Bolke de Bruin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878538#comment-16878538 ] Bolke de Bruin commented on ATLAS-3153: --- [~saqeeb.shaikh136] can we have this merged please? > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement > Components: atlas-core, atlas-webui >Affects Versions: 2.0.0 > Reporter: Bolke de Bruin >Priority: Major > Labels: authentication, authorization > Attachments: 0001-ATLAS-3153-Add-keycloak-authentication.patch, > application.log, keycloak.json, openid_connect_atlas.md > > Time Spent: 20m > Remaining Estimate: 0h > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875578#comment-16875578 ] Bolke de Bruin commented on ATLAS-3153: --- ping [~saqeeb.shaikh136] . I have created the first version for bridges, quickstart and exampkes, but it is dependent on this one. Also note that Knox does not have support in Atlas client side. > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement > Components: atlas-core, atlas-webui >Affects Versions: 2.0.0 > Reporter: Bolke de Bruin >Priority: Major > Labels: authentication, authorization > Attachments: 0001-ATLAS-3153-Add-keycloak-authentication.patch, > application.log, keycloak.json, openid_connect_atlas.md > > Time Spent: 20m > Remaining Estimate: 0h > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3203) Incorrect qualifiedName for QuickStartV2 examples
[ https://issues.apache.org/jira/browse/ATLAS-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875579#comment-16875579 ] Bolke de Bruin commented on ATLAS-3203: --- [~saqeeb.shaikh136] can this be merged please? its a simple fix > Incorrect qualifiedName for QuickStartV2 examples > - > > Key: ATLAS-3203 > URL: https://issues.apache.org/jira/browse/ATLAS-3203 > Project: Atlas > Issue Type: Bug >Reporter: Nanne Wielinga >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > The qualifiedName for columns should be: 'database.table.column@cluster'. > The qualifiedName for columns currently is: 'column@cluster' > > The qualifiedName for tables should be: 'database.table@cluster'. > The qualifiedName for tables currently is: 'table@cluster' -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880583#comment-16880583 ] Bolke de Bruin edited comment on ATLAS-3153 at 7/8/19 5:53 PM: --- [~nixonrodrigues] can you share a bit more what is failing exactly? I assume IT is integration test? Can I reproduce the test somehow? Is it possible to get access to the jenkins logs? was (Author: bolke): [~nixonrodrigues]can you share a bit more what is failing exactly? I assume IT is integration test? Can I reproduce the test somehow? > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement > Components: atlas-core, atlas-webui >Affects Versions: 2.0.0 > Reporter: Bolke de Bruin >Priority: Major > Labels: authentication, authorization > Fix For: 3.0.0 > > Attachments: 0001-ATLAS-3153-Add-keycloak-authentication.patch, > 0001-ATLAS-3153-Testcase-fix-due-to-Keycloak-authenticati.patch, > application.log, keycloak.json, openid_connect_atlas.md > > Time Spent: 40m > Remaining Estimate: 0h > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3317) java.lang.Exception: Timeout scanning annotations while running Integration Tests
[
https://issues.apache.org/jira/browse/ATLAS-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880638#comment-16880638
]
Bolke de Bruin commented on ATLAS-3317:
---
Ok. Do you know if there is a way to get to the application logs when running
the IT? I see only client side at the moment. By default Keycloak is inactive
so this looks a bit weird.
> java.lang.Exception: Timeout scanning annotations while running Integration
> Tests
> -
>
> Key: ATLAS-3317
> URL: https://issues.apache.org/jira/browse/ATLAS-3317
> Project: Atlas
> Issue Type: Bug
>Reporter: Nixon Rodrigues
>Priority: Major
>
> run :- *mvn clean install -DskipUTs*
>
> {noformat}
> [INFO] --- jetty-maven-plugin:9.3.14.v20161028:deploy-war (start-jetty) @
> atlas-webapp ---
> [INFO] Logging initialized @220653ms
> [INFO] Configuring Jetty for project: Apache Atlas Web Application
> [INFO] Context path = /
> [INFO] Tmp directory =
> /home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/tmp
> [INFO] Web defaults = org/eclipse/jetty/webapp/webdefault.xml
> [INFO] Web overrides = none
> [INFO] jetty-9.3.14.v20161028
> [INFO] Scanning elapsed time=67258ms
> [WARNING] Failed startup of context o.e.j.m.p.JettyWebAppContext@2eb2d10c{/,
> [file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE}\{/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war}|file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE%7D%7B/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war%7D]
> java.lang.Exception: Timeout scanning annotations
> at
> org.eclipse.jetty.annotations.AnnotationConfiguration.scanForAnnotations
> (AnnotationConfiguration.java:578)
> at org.eclipse.jetty.annotations.AnnotationConfiguration.configure
> (AnnotationConfiguration.java:447)
> at org.eclipse.jetty.webapp.WebAppContext.configure
> (WebAppContext.java:494)
> at org.eclipse.jetty.webapp.WebAppContext.startContext
> (WebAppContext.java:1361)
> at org.eclipse.jetty.server.handler.ContextHandler.doStart
> (ContextHandler.java:778)
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart
> (ServletContextHandler.java:262)
> at org.eclipse.jetty.webapp.WebAppContext.doStart (WebAppContext.java:520)
> at org.eclipse.jetty.maven.plugin.JettyWebAppContext.doStart
> (JettyWebAppContext.java:398)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart
> (ContextHandlerCollection.java:161)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.server.Server.start (Server.java:422)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:105)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.Server.doStart (Server.java:389)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.startJetty
> (AbstractJettyMojo.java:460)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.execute
> (AbstractJettyMojo.java:328)
> at org.eclipse.jetty.maven.plugin.JettyRunWarMojo.execute
> (JettyRunWarMojo.java:64)
> at org.eclipse.jetty.maven.plugin.JettyDeployWar.execute
> (JettyDeployWar.java:65)
> at org.apache.
[jira] [Commented] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880583#comment-16880583 ] Bolke de Bruin commented on ATLAS-3153: --- [~nixonrodrigues]can you share a bit more what is failing exactly? I assume IT is integration test? Can I reproduce the test somehow? > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement > Components: atlas-core, atlas-webui >Affects Versions: 2.0.0 > Reporter: Bolke de Bruin >Priority: Major > Labels: authentication, authorization > Fix For: 3.0.0 > > Attachments: 0001-ATLAS-3153-Add-keycloak-authentication.patch, > 0001-ATLAS-3153-Testcase-fix-due-to-Keycloak-authenticati.patch, > application.log, keycloak.json, openid_connect_atlas.md > > Time Spent: 40m > Remaining Estimate: 0h > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3317) java.lang.Exception: Timeout scanning annotations while running Integration Tests
[
https://issues.apache.org/jira/browse/ATLAS-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880588#comment-16880588
]
Bolke de Bruin commented on ATLAS-3317:
---
[~nixonrodrigues] I'll have a look. Please note that the keycloak patch was
tested (mvn clean install) with 2.0 rcX. I'll check that build and will check
out master.
> java.lang.Exception: Timeout scanning annotations while running Integration
> Tests
> -
>
> Key: ATLAS-3317
> URL: https://issues.apache.org/jira/browse/ATLAS-3317
> Project: Atlas
> Issue Type: Bug
>Reporter: Nixon Rodrigues
>Priority: Major
>
> run :- *mvn clean install -DskipUTs*
>
> {noformat}
> [INFO] --- jetty-maven-plugin:9.3.14.v20161028:deploy-war (start-jetty) @
> atlas-webapp ---
> [INFO] Logging initialized @220653ms
> [INFO] Configuring Jetty for project: Apache Atlas Web Application
> [INFO] Context path = /
> [INFO] Tmp directory =
> /home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/tmp
> [INFO] Web defaults = org/eclipse/jetty/webapp/webdefault.xml
> [INFO] Web overrides = none
> [INFO] jetty-9.3.14.v20161028
> [INFO] Scanning elapsed time=67258ms
> [WARNING] Failed startup of context o.e.j.m.p.JettyWebAppContext@2eb2d10c{/,
> [file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE}\{/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war}|file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE%7D%7B/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war%7D]
> java.lang.Exception: Timeout scanning annotations
> at
> org.eclipse.jetty.annotations.AnnotationConfiguration.scanForAnnotations
> (AnnotationConfiguration.java:578)
> at org.eclipse.jetty.annotations.AnnotationConfiguration.configure
> (AnnotationConfiguration.java:447)
> at org.eclipse.jetty.webapp.WebAppContext.configure
> (WebAppContext.java:494)
> at org.eclipse.jetty.webapp.WebAppContext.startContext
> (WebAppContext.java:1361)
> at org.eclipse.jetty.server.handler.ContextHandler.doStart
> (ContextHandler.java:778)
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart
> (ServletContextHandler.java:262)
> at org.eclipse.jetty.webapp.WebAppContext.doStart (WebAppContext.java:520)
> at org.eclipse.jetty.maven.plugin.JettyWebAppContext.doStart
> (JettyWebAppContext.java:398)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart
> (ContextHandlerCollection.java:161)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.server.Server.start (Server.java:422)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:105)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.Server.doStart (Server.java:389)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.startJetty
> (AbstractJettyMojo.java:460)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.execute
> (AbstractJettyMojo.java:328)
> at org.eclipse.jetty.maven.plugin.JettyRunWarMojo.execute
> (JettyRunWarMojo.java:64)
> at org.eclipse.jetty.maven.plugin.JettyDeployWar.execute
> (JettyDeployWar.java:65)
> at org.apache.maven.plugin.DefaultBuildP
[jira] [Commented] (ATLAS-3317) java.lang.Exception: Timeout scanning annotations while running Integration Tests
[ https://issues.apache.org/jira/browse/ATLAS-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880624#comment-16880624 ] Bolke de Bruin commented on ATLAS-3317: --- I'm also seeing errors with commit f75d7f409f3323e799bb33e6ae1da89526fc475c although a bit different (this is the commit before my patch) ``` 127.0.0.1 - - [08/Jul/2019:18:40:22 +] "GET //localhost:31000/api/atlas/lineage/d8cc5cd7-625b-4c0b-874d-703feba86fd9/inputs/graph HTTP/1.1" 200 2423 "-" "Java/1.8.0_212" Jul 08, 2019 6:40:23 PM com.sun.jersey.spi.container.ContainerResponse logException SEVERE: Mapped exception to response: 500 (Internal Server Error) javax.ws.rs.WebApplicationException at org.apache.atlas.web.resources.DataSetLineageResource.schema(DataSetLineageResource.java:203) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1712) at org.apache.atlas.web.filters.AuditFilter.doFilter(AuditFilter.java:100) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.apache.atlas.web.filters.AtlasCSRFPreventionFilter$ServletFilterHttpInteraction.proceed(AtlasCSRFPreventionFilter.java:235) at org.apache.atlas.web.filters.AtlasCSRFPreventionFilter.handleHttpInteraction(AtlasCSRFPreventionFilter.java:177) at org.apache.atlas.web.filters.AtlasCSRFPreventionFilter.doFilter(AtlasCSRFPreventionFilter.java:190) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.apache.atlas.web.filters.AtlasAuthenticationFilter.doFilter(Atla
[jira] [Comment Edited] (ATLAS-3317) java.lang.Exception: Timeout scanning annotations while running Integration Tests
[
https://issues.apache.org/jira/browse/ATLAS-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880624#comment-16880624
]
Bolke de Bruin edited comment on ATLAS-3317 at 7/8/19 6:46 PM:
---
I'm also seeing errors with commit f75d7f409f3323e799bb33e6ae1da89526fc475c
although a bit different (this is the commit before my patch)
{code:java}
127.0.0.1 - - [08/Jul/2019:18:40:22 +] "GET
//localhost:31000/api/atlas/lineage/d8cc5cd7-625b-4c0b-874d-703feba86fd9/inputs/graph
HTTP/1.1" 200 2423 "-" "Java/1.8.0_212"
Jul 08, 2019 6:40:23 PM com.sun.jersey.spi.container.ContainerResponse
logException
SEVERE: Mapped exception to response: 500 (Internal Server Error)
javax.ws.rs.WebApplicationException
at
org.apache.atlas.web.resources.DataSetLineageResource.schema(DataSetLineageResource.java:203)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1712)
at org.apache.atlas.web.filters.AuditFilter.doFilter(AuditFilter.java:100)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
at
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.apache.atlas.web.filters.AtlasCSRFPreventionFilter$ServletFilterHttpInteraction.proceed(AtlasCSRFPreventionFilter.java:235)
at
org.apache.atlas.web.filters.AtlasCSRFPreventionFilter.handleHttpInteraction(AtlasCSRFPreventionFilter.java:177)
at
org.apache.atlas.web.filters.AtlasCSRFPreventionFilter.doFilter(AtlasCSRFPreventionFilter.java:190)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.apache.atlas.web.filters.Atla
[jira] [Comment Edited] (ATLAS-3317) java.lang.Exception: Timeout scanning annotations while running Integration Tests
[
https://issues.apache.org/jira/browse/ATLAS-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880638#comment-16880638
]
Bolke de Bruin edited comment on ATLAS-3317 at 7/8/19 7:05 PM:
---
Ok. Do you know if there is a way to get to the application logs when running
the IT? I see only client side at the moment. By default Keycloak is inactive
so this looks a bit weird. Where can I find the config of the IT (ie. server
config)?
was (Author: bolke):
Ok. Do you know if there is a way to get to the application logs when running
the IT? I see only client side at the moment. By default Keycloak is inactive
so this looks a bit weird.
> java.lang.Exception: Timeout scanning annotations while running Integration
> Tests
> -
>
> Key: ATLAS-3317
> URL: https://issues.apache.org/jira/browse/ATLAS-3317
> Project: Atlas
> Issue Type: Bug
>Reporter: Nixon Rodrigues
>Priority: Major
>
> run :- *mvn clean install -DskipUTs*
>
> {noformat}
> [INFO] --- jetty-maven-plugin:9.3.14.v20161028:deploy-war (start-jetty) @
> atlas-webapp ---
> [INFO] Logging initialized @220653ms
> [INFO] Configuring Jetty for project: Apache Atlas Web Application
> [INFO] Context path = /
> [INFO] Tmp directory =
> /home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/tmp
> [INFO] Web defaults = org/eclipse/jetty/webapp/webdefault.xml
> [INFO] Web overrides = none
> [INFO] jetty-9.3.14.v20161028
> [INFO] Scanning elapsed time=67258ms
> [WARNING] Failed startup of context o.e.j.m.p.JettyWebAppContext@2eb2d10c{/,
> [file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE}\{/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war}|file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE%7D%7B/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war%7D]
> java.lang.Exception: Timeout scanning annotations
> at
> org.eclipse.jetty.annotations.AnnotationConfiguration.scanForAnnotations
> (AnnotationConfiguration.java:578)
> at org.eclipse.jetty.annotations.AnnotationConfiguration.configure
> (AnnotationConfiguration.java:447)
> at org.eclipse.jetty.webapp.WebAppContext.configure
> (WebAppContext.java:494)
> at org.eclipse.jetty.webapp.WebAppContext.startContext
> (WebAppContext.java:1361)
> at org.eclipse.jetty.server.handler.ContextHandler.doStart
> (ContextHandler.java:778)
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart
> (ServletContextHandler.java:262)
> at org.eclipse.jetty.webapp.WebAppContext.doStart (WebAppContext.java:520)
> at org.eclipse.jetty.maven.plugin.JettyWebAppContext.doStart
> (JettyWebAppContext.java:398)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart
> (ContextHandlerCollection.java:161)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.server.Server.start (Server.java:422)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:105)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.Server.doStart (Server.java:389)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.startJe
[jira] [Commented] (ATLAS-3317) java.lang.Exception: Timeout scanning annotations while running Integration Tests
[
https://issues.apache.org/jira/browse/ATLAS-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880650#comment-16880650
]
Bolke de Bruin commented on ATLAS-3317:
---
Trying now. If that works, I'm suspecting a lookup timeout somehow.
the failsafe reports don't give me a server side trace btw, just the trace of
the failed test. A would assume a 503 error being returned should somehow end
up being logged server side as well.
> java.lang.Exception: Timeout scanning annotations while running Integration
> Tests
> -
>
> Key: ATLAS-3317
> URL: https://issues.apache.org/jira/browse/ATLAS-3317
> Project: Atlas
> Issue Type: Bug
>Reporter: Nixon Rodrigues
>Priority: Major
> Attachments: maxWait.diff
>
>
> run :- *mvn clean install -DskipUTs*
>
> {noformat}
> [INFO] --- jetty-maven-plugin:9.3.14.v20161028:deploy-war (start-jetty) @
> atlas-webapp ---
> [INFO] Logging initialized @220653ms
> [INFO] Configuring Jetty for project: Apache Atlas Web Application
> [INFO] Context path = /
> [INFO] Tmp directory =
> /home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/tmp
> [INFO] Web defaults = org/eclipse/jetty/webapp/webdefault.xml
> [INFO] Web overrides = none
> [INFO] jetty-9.3.14.v20161028
> [INFO] Scanning elapsed time=67258ms
> [WARNING] Failed startup of context o.e.j.m.p.JettyWebAppContext@2eb2d10c{/,
> [file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE}\{/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war}|file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE%7D%7B/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war%7D]
> java.lang.Exception: Timeout scanning annotations
> at
> org.eclipse.jetty.annotations.AnnotationConfiguration.scanForAnnotations
> (AnnotationConfiguration.java:578)
> at org.eclipse.jetty.annotations.AnnotationConfiguration.configure
> (AnnotationConfiguration.java:447)
> at org.eclipse.jetty.webapp.WebAppContext.configure
> (WebAppContext.java:494)
> at org.eclipse.jetty.webapp.WebAppContext.startContext
> (WebAppContext.java:1361)
> at org.eclipse.jetty.server.handler.ContextHandler.doStart
> (ContextHandler.java:778)
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart
> (ServletContextHandler.java:262)
> at org.eclipse.jetty.webapp.WebAppContext.doStart (WebAppContext.java:520)
> at org.eclipse.jetty.maven.plugin.JettyWebAppContext.doStart
> (JettyWebAppContext.java:398)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart
> (ContextHandlerCollection.java:161)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.server.Server.start (Server.java:422)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:105)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.Server.doStart (Server.java:389)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.startJetty
> (AbstractJettyMojo.java:460)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.execute
> (AbstractJettyMojo.java:328)
> at org.eclipse.jetty.maven.plugin.JettyRunWa
[jira] [Comment Edited] (ATLAS-3317) java.lang.Exception: Timeout scanning annotations while running Integration Tests
[
https://issues.apache.org/jira/browse/ATLAS-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880657#comment-16880657
]
Bolke de Bruin edited comment on ATLAS-3317 at 7/8/19 7:42 PM:
---
That indeed fixes the issue. Now I am also getting an application.log, which
wasn't there previously.
Ah okay I was thrown of by the reference to my patch i guess. Obviously in the
patch itself annotations have been added and keycloak probably brings some
itself as well. This increases scanning time. The default of 60s might just not
have been enough anymore now.
Note that the setting is in seconds so it is set to an hour now.
was (Author: bolke):
That indeed fixes the issue. Now I am also getting an application.log, which
wasn't there previously. Maybe server didn't have time to startup?
> java.lang.Exception: Timeout scanning annotations while running Integration
> Tests
> -
>
> Key: ATLAS-3317
> URL: https://issues.apache.org/jira/browse/ATLAS-3317
> Project: Atlas
> Issue Type: Bug
>Reporter: Nixon Rodrigues
>Priority: Major
> Attachments: maxWait.diff
>
>
> run :- *mvn clean install -DskipUTs*
>
> {noformat}
> [INFO] --- jetty-maven-plugin:9.3.14.v20161028:deploy-war (start-jetty) @
> atlas-webapp ---
> [INFO] Logging initialized @220653ms
> [INFO] Configuring Jetty for project: Apache Atlas Web Application
> [INFO] Context path = /
> [INFO] Tmp directory =
> /home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/tmp
> [INFO] Web defaults = org/eclipse/jetty/webapp/webdefault.xml
> [INFO] Web overrides = none
> [INFO] jetty-9.3.14.v20161028
> [INFO] Scanning elapsed time=67258ms
> [WARNING] Failed startup of context o.e.j.m.p.JettyWebAppContext@2eb2d10c{/,
> [file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE}\{/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war}|file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE%7D%7B/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war%7D]
> java.lang.Exception: Timeout scanning annotations
> at
> org.eclipse.jetty.annotations.AnnotationConfiguration.scanForAnnotations
> (AnnotationConfiguration.java:578)
> at org.eclipse.jetty.annotations.AnnotationConfiguration.configure
> (AnnotationConfiguration.java:447)
> at org.eclipse.jetty.webapp.WebAppContext.configure
> (WebAppContext.java:494)
> at org.eclipse.jetty.webapp.WebAppContext.startContext
> (WebAppContext.java:1361)
> at org.eclipse.jetty.server.handler.ContextHandler.doStart
> (ContextHandler.java:778)
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart
> (ServletContextHandler.java:262)
> at org.eclipse.jetty.webapp.WebAppContext.doStart (WebAppContext.java:520)
> at org.eclipse.jetty.maven.plugin.JettyWebAppContext.doStart
> (JettyWebAppContext.java:398)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart
> (ContextHandlerCollection.java:161)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.server.Server.start (Server.java:422)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:105)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.Server.
[jira] [Commented] (ATLAS-3317) java.lang.Exception: Timeout scanning annotations while running Integration Tests
[
https://issues.apache.org/jira/browse/ATLAS-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880657#comment-16880657
]
Bolke de Bruin commented on ATLAS-3317:
---
That indeed fixes the issue. Now I am also getting an application.log, which
wasn't there previously. Maybe server didn't have time to startup?
> java.lang.Exception: Timeout scanning annotations while running Integration
> Tests
> -
>
> Key: ATLAS-3317
> URL: https://issues.apache.org/jira/browse/ATLAS-3317
> Project: Atlas
> Issue Type: Bug
>Reporter: Nixon Rodrigues
>Priority: Major
> Attachments: maxWait.diff
>
>
> run :- *mvn clean install -DskipUTs*
>
> {noformat}
> [INFO] --- jetty-maven-plugin:9.3.14.v20161028:deploy-war (start-jetty) @
> atlas-webapp ---
> [INFO] Logging initialized @220653ms
> [INFO] Configuring Jetty for project: Apache Atlas Web Application
> [INFO] Context path = /
> [INFO] Tmp directory =
> /home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/tmp
> [INFO] Web defaults = org/eclipse/jetty/webapp/webdefault.xml
> [INFO] Web overrides = none
> [INFO] jetty-9.3.14.v20161028
> [INFO] Scanning elapsed time=67258ms
> [WARNING] Failed startup of context o.e.j.m.p.JettyWebAppContext@2eb2d10c{/,
> [file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE}\{/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war}|file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE%7D%7B/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war%7D]
> java.lang.Exception: Timeout scanning annotations
> at
> org.eclipse.jetty.annotations.AnnotationConfiguration.scanForAnnotations
> (AnnotationConfiguration.java:578)
> at org.eclipse.jetty.annotations.AnnotationConfiguration.configure
> (AnnotationConfiguration.java:447)
> at org.eclipse.jetty.webapp.WebAppContext.configure
> (WebAppContext.java:494)
> at org.eclipse.jetty.webapp.WebAppContext.startContext
> (WebAppContext.java:1361)
> at org.eclipse.jetty.server.handler.ContextHandler.doStart
> (ContextHandler.java:778)
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart
> (ServletContextHandler.java:262)
> at org.eclipse.jetty.webapp.WebAppContext.doStart (WebAppContext.java:520)
> at org.eclipse.jetty.maven.plugin.JettyWebAppContext.doStart
> (JettyWebAppContext.java:398)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart
> (ContextHandlerCollection.java:161)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.server.Server.start (Server.java:422)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:105)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.Server.doStart (Server.java:389)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.startJetty
> (AbstractJettyMojo.java:460)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.execute
> (AbstractJettyMojo.java:328)
> at org.eclipse.jetty.maven.plugin.JettyRunWarMojo.execute
> (JettyRunWarMojo.java:64)
> at org.eclipse.jetty.maven.plugin.JettyDeployWar.execute
> (JettyDeployWar.java:65)
> at org.a
[jira] [Comment Edited] (ATLAS-3317) java.lang.Exception: Timeout scanning annotations while running Integration Tests
[
https://issues.apache.org/jira/browse/ATLAS-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880657#comment-16880657
]
Bolke de Bruin edited comment on ATLAS-3317 at 7/8/19 7:43 PM:
---
That indeed fixes the issue. Now I am also getting an application.log, which
wasn't there previously.
Ah okay I was thrown of by the reference to my patch i guess. Obviously in the
patch itself annotations have been added and keycloak probably brings some
itself as well. This increases scanning time. The default of 60s might just not
have been enough anymore now.
Note that the setting is in seconds so it is set to 3000 seconds now, which
might not be what you want.
was (Author: bolke):
That indeed fixes the issue. Now I am also getting an application.log, which
wasn't there previously.
Ah okay I was thrown of by the reference to my patch i guess. Obviously in the
patch itself annotations have been added and keycloak probably brings some
itself as well. This increases scanning time. The default of 60s might just not
have been enough anymore now.
Note that the setting is in seconds so it is set to an hour now.
> java.lang.Exception: Timeout scanning annotations while running Integration
> Tests
> -
>
> Key: ATLAS-3317
> URL: https://issues.apache.org/jira/browse/ATLAS-3317
> Project: Atlas
> Issue Type: Bug
>Reporter: Nixon Rodrigues
>Priority: Major
> Attachments: maxWait.diff
>
>
> run :- *mvn clean install -DskipUTs*
>
> {noformat}
> [INFO] --- jetty-maven-plugin:9.3.14.v20161028:deploy-war (start-jetty) @
> atlas-webapp ---
> [INFO] Logging initialized @220653ms
> [INFO] Configuring Jetty for project: Apache Atlas Web Application
> [INFO] Context path = /
> [INFO] Tmp directory =
> /home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/tmp
> [INFO] Web defaults = org/eclipse/jetty/webapp/webdefault.xml
> [INFO] Web overrides = none
> [INFO] jetty-9.3.14.v20161028
> [INFO] Scanning elapsed time=67258ms
> [WARNING] Failed startup of context o.e.j.m.p.JettyWebAppContext@2eb2d10c{/,
> [file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE}\{/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war}|file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE%7D%7B/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war%7D]
> java.lang.Exception: Timeout scanning annotations
> at
> org.eclipse.jetty.annotations.AnnotationConfiguration.scanForAnnotations
> (AnnotationConfiguration.java:578)
> at org.eclipse.jetty.annotations.AnnotationConfiguration.configure
> (AnnotationConfiguration.java:447)
> at org.eclipse.jetty.webapp.WebAppContext.configure
> (WebAppContext.java:494)
> at org.eclipse.jetty.webapp.WebAppContext.startContext
> (WebAppContext.java:1361)
> at org.eclipse.jetty.server.handler.ContextHandler.doStart
> (ContextHandler.java:778)
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart
> (ServletContextHandler.java:262)
> at org.eclipse.jetty.webapp.WebAppContext.doStart (WebAppContext.java:520)
> at org.eclipse.jetty.maven.plugin.JettyWebAppContext.doStart
> (JettyWebAppContext.java:398)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart
> (ContextHandlerCollection.java:161)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jett
[jira] [Commented] (ATLAS-3317) java.lang.Exception: Timeout scanning annotations while running Integration Tests
[
https://issues.apache.org/jira/browse/ATLAS-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880664#comment-16880664
]
Bolke de Bruin commented on ATLAS-3317:
---
I'll have a stab at it, but it is bedtime here as well to be honest.
> java.lang.Exception: Timeout scanning annotations while running Integration
> Tests
> -
>
> Key: ATLAS-3317
> URL: https://issues.apache.org/jira/browse/ATLAS-3317
> Project: Atlas
> Issue Type: Bug
>Reporter: Nixon Rodrigues
>Priority: Major
> Attachments: maxWait.diff
>
>
> run :- *mvn clean install -DskipUTs*
>
> {noformat}
> [INFO] --- jetty-maven-plugin:9.3.14.v20161028:deploy-war (start-jetty) @
> atlas-webapp ---
> [INFO] Logging initialized @220653ms
> [INFO] Configuring Jetty for project: Apache Atlas Web Application
> [INFO] Context path = /
> [INFO] Tmp directory =
> /home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/tmp
> [INFO] Web defaults = org/eclipse/jetty/webapp/webdefault.xml
> [INFO] Web overrides = none
> [INFO] jetty-9.3.14.v20161028
> [INFO] Scanning elapsed time=67258ms
> [WARNING] Failed startup of context o.e.j.m.p.JettyWebAppContext@2eb2d10c{/,
> [file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE}\{/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war}|file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE%7D%7B/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war%7D]
> java.lang.Exception: Timeout scanning annotations
> at
> org.eclipse.jetty.annotations.AnnotationConfiguration.scanForAnnotations
> (AnnotationConfiguration.java:578)
> at org.eclipse.jetty.annotations.AnnotationConfiguration.configure
> (AnnotationConfiguration.java:447)
> at org.eclipse.jetty.webapp.WebAppContext.configure
> (WebAppContext.java:494)
> at org.eclipse.jetty.webapp.WebAppContext.startContext
> (WebAppContext.java:1361)
> at org.eclipse.jetty.server.handler.ContextHandler.doStart
> (ContextHandler.java:778)
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart
> (ServletContextHandler.java:262)
> at org.eclipse.jetty.webapp.WebAppContext.doStart (WebAppContext.java:520)
> at org.eclipse.jetty.maven.plugin.JettyWebAppContext.doStart
> (JettyWebAppContext.java:398)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart
> (ContextHandlerCollection.java:161)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.server.Server.start (Server.java:422)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart
> (ContainerLifeCycle.java:105)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.Server.doStart (Server.java:389)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.startJetty
> (AbstractJettyMojo.java:460)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.execute
> (AbstractJettyMojo.java:328)
> at org.eclipse.jetty.maven.plugin.JettyRunWarMojo.execute
> (JettyRunWarMojo.java:64)
> at org.eclipse.jetty.maven.plugin.JettyDeployWar.execute
> (JettyDeployWar.java:65)
> at org.apache.maven.plugin.Default
[jira] [Created] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
Bolke de Bruin created ATLAS-3153: - Summary: Support OpenID Connect directly rather than through Knox Key: ATLAS-3153 URL: https://issues.apache.org/jira/browse/ATLAS-3153 Project: Atlas Issue Type: Improvement Reporter: Bolke de Bruin The current SSO implementation with Apache Knox is limiting SSO interoperability to Apache Knox. Knox uses JWT verification which could easily be extended to allow for direct OpenID Connect support and doesn't require organizations to deploy Knox. Required changes: * Pickup bearer token from headers * Improve and standardize redirecting * Optionally: obtain certificates from well_known uri * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3153) Support OpenID Connect directly rather than through Knox
[ https://issues.apache.org/jira/browse/ATLAS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3153: -- Affects Version/s: 1.1.0 > Support OpenID Connect directly rather than through Knox > > > Key: ATLAS-3153 > URL: https://issues.apache.org/jira/browse/ATLAS-3153 > Project: Atlas > Issue Type: Improvement >Affects Versions: 1.1.0 > Reporter: Bolke de Bruin >Priority: Major > > The current SSO implementation with Apache Knox is limiting SSO > interoperability to Apache Knox. Knox uses JWT verification which could > easily be extended to allow for direct OpenID Connect support and doesn't > require organizations to deploy Knox. > Required changes: > * Pickup bearer token from headers > * Improve and standardize redirecting > * Optionally: obtain certificates from well_known uri > * Optionally: obtain user groups from userinfo endpoint rather than UGI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3165) Re-enable gremlin search
[ https://issues.apache.org/jira/browse/ATLAS-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3165: -- Description: .We are integrating Atlas with Lyft’s Amundsen ([https://github.com/lyft/amundsenfrontendlibrary)] and we are finding that the standard search DSL is limited in such a way that it does not support simple useful queries like ```Column where name like '*amount*' groupby(table) select count()``` Or ```Column where name like '*amount*' select count(table)``` This can be done by using a gremlin query but the interface was removed in ATLAS-2229 for some reason. We think it should be reinstated. Limiting access to this interface could be done by Ranger policies. was: We are integrating Atlas with Lyft’s Amundsen ([https://github.com/lyft/amundsenfrontendlibrary)] and we are finding that the standard search DSL is limited in such a way that it does not support simple useful queries like ```Column where name like '*amount*' groupby(table) select count()``` Or ```Column where name like '*amount*' select count(table)``` This can be done by using a gremlin query but the interface to this is marked Private which indicates that it might be changed or be removed in the future. We would like this to be a publicly supported interface. Limiting access to this interface could be done by Ranger policies. Issue Type: Bug (was: Improvement) Summary: Re-enable gremlin search (was: Mark InterfaceAudience “search/gremlin” as public) > Re-enable gremlin search > > > Key: ATLAS-3165 > URL: https://issues.apache.org/jira/browse/ATLAS-3165 > Project: Atlas > Issue Type: Bug >Affects Versions: 1.1.0 > Reporter: Bolke de Bruin >Priority: Major > > .We are integrating Atlas with Lyft’s Amundsen > ([https://github.com/lyft/amundsenfrontendlibrary)] and we are finding that > the standard search DSL is limited in such a way that it does not support > simple useful queries like > ```Column where name like '*amount*' groupby(table) select count()``` > Or > ```Column where name like '*amount*' select count(table)``` > > This can be done by using a gremlin query but the interface was removed in > ATLAS-2229 for some reason. We think it should be reinstated. > Limiting access to this interface could be done by Ranger policies. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ATLAS-3165) Mark InterfaceAudience “search/gremlin” as public
Bolke de Bruin created ATLAS-3165: - Summary: Mark InterfaceAudience “search/gremlin” as public Key: ATLAS-3165 URL: https://issues.apache.org/jira/browse/ATLAS-3165 Project: Atlas Issue Type: Improvement Affects Versions: 1.1.0 Reporter: Bolke de Bruin We are integrating Atlas with Lyft’s Amundsen ([https://github.com/lyft/amundsenfrontendlibrary)] and we are finding that the standard search DSL is limited in such a way that it does not support simple useful queries like ```Column where name like '*amount*' groupby(table) select count()``` Or ```Column where name like '*amount*' select count(table)``` This can be done by using a gremlin query but the interface to this is marked Private which indicates that it might be changed or be removed in the future. We would like this to be a publicly supported interface. Limiting access to this interface could be done by Ranger policies. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3081) Expose Gremlin Search API
[ https://issues.apache.org/jira/browse/ATLAS-3081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825868#comment-16825868 ] Bolke de Bruin commented on ATLAS-3081: --- Bringing in the challenges from ATLAS-3165 , I think the notion of "user" within Atlas is a bit too narrow. There can be services that want to expose different information which can currently not be expressed in the DSL that Atlas supports. I want to be able to find the most popular tables/schemas/columns/documents by lineage/usage/. Ie. expressions like this are not possible: ```Column where name like '*amount*' groupby(table) select count()``` Or ```Column where name like '*amount*' select count(table)``` So yes as an administrator I want to expose advanced search functionality for integration with other services. Extending the current DSL might work, but it will always be an abstraction except when you are able to fully express in your DSL what Gremlin offers, which defeats the purpose I suppose. In my consideration the DSL exists to give users a kind of familar query language not to protect them from the power of Gremlin. > Expose Gremlin Search API > - > > Key: ATLAS-3081 > URL: https://issues.apache.org/jira/browse/ATLAS-3081 > Project: Atlas > Issue Type: New Feature >Reporter: Ayush Nigam >Assignee: Ayush Nigam >Priority: Minor > Attachments: ATLAS_3081.patch > > > Expose Gremlin Search API to solve more complex usecases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3081) Expose Gremlin Search API
[ https://issues.apache.org/jira/browse/ATLAS-3081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16826238#comment-16826238 ] Bolke de Bruin commented on ATLAS-3081: --- [~madhan.neethiraj] I understand that point of view for *end-users.* I would be happy with a DSL that allows querying like the types of queries I mentioned above. How do you suggest solving these? > Expose Gremlin Search API > - > > Key: ATLAS-3081 > URL: https://issues.apache.org/jira/browse/ATLAS-3081 > Project: Atlas > Issue Type: New Feature >Reporter: Ayush Nigam >Assignee: Ayush Nigam >Priority: Minor > Attachments: ATLAS_3081.patch > > > Expose Gremlin Search API to solve more complex usecases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ATLAS-3367) Expose total search results when using IndexQuery
Bolke de Bruin created ATLAS-3367: - Summary: Expose total search results when using IndexQuery Key: ATLAS-3367 URL: https://issues.apache.org/jira/browse/ATLAS-3367 Project: Atlas Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Bolke de Bruin ElasticSearch and SOLR both can expose the total number of search results without returning all results. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (ATLAS-3367) Expose total search results when using IndexQuery
[ https://issues.apache.org/jira/browse/ATLAS-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3367: -- Attachment: 0001-ATLAS-3367-Expose-total-count-of-index-searches.patch > Expose total search results when using IndexQuery > - > > Key: ATLAS-3367 > URL: https://issues.apache.org/jira/browse/ATLAS-3367 > Project: Atlas > Issue Type: Improvement >Affects Versions: 2.0.0 > Reporter: Bolke de Bruin >Priority: Major > Attachments: > 0001-ATLAS-3367-Expose-total-count-of-index-searches.patch > > > ElasticSearch and SOLR both can expose the total number of search results > without returning all results. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (ATLAS-3367) Expose total search results when using IndexQuery
[ https://issues.apache.org/jira/browse/ATLAS-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3367: -- Attachment: (was: 0001-ATLAS-3367-Expose-total-count-of-index-searches.patch) > Expose total search results when using IndexQuery > - > > Key: ATLAS-3367 > URL: https://issues.apache.org/jira/browse/ATLAS-3367 > Project: Atlas > Issue Type: Improvement >Affects Versions: 2.0.0 > Reporter: Bolke de Bruin >Priority: Major > Attachments: > 0001-ATLAS-3367-Expose-total-count-of-index-searches.patch > > > ElasticSearch and SOLR both can expose the total number of search results > without returning all results. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
Re: Review Request 71282: Expose total count of index searches
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71282/ --- (Updated Aug. 13, 2019, 7:22 p.m.) Review request for atlas, Nixon Rodrigues and Saqeeb Shaikh. Bugs: https://issues.apache.org/jira/browse/ATLAS-3367 https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3367 Repository: atlas Description --- Both ElasticSearch and SOLR can expose the total count of results without returning all results. This is useful to give the user or client an idea how many results there are. This patch ensures the total is returned if available. This total is an approximate as scrubbing of the results still needs to happen. Therefore, one should not only rely on this information to provide ,for example, pagination. Diffs (updated) - intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java d274cc07e repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 479ddfd89 repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java c67e34772 repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java f7d8f08c7 repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java 1dd1afaa3 repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java 0ffd61c07 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java a7ccaebc8 repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java c253d544b Diff: https://reviews.apache.org/r/71282/diff/2/ Changes: https://reviews.apache.org/r/71282/diff/1-2/ Testing --- Thanks, Bolke de Bruin
[jira] [Created] (ATLAS-3368) Very slow creation of entities with referred entities without relationshipDef
Bolke de Bruin created ATLAS-3368:
-
Summary: Very slow creation of entities with referred entities
without relationshipDef
Key: ATLAS-3368
URL: https://issues.apache.org/jira/browse/ATLAS-3368
Project: Atlas
Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Bolke de Bruin
If an entity is created that has a definition with a reference to a non built
in type, e.g.
{code:java}
,{
"name" : "server",
"typeName" : "server",
"isOptional" : false,
"cardinality" : "SINGLE",
"valuesMinCount" : 1,
"valuesMaxCount" : 1,
"isUnique" : false,
"isIndexable" : true
}{code}
It will trigger a legacy code path in EntityGraphMapper. This path can take
excessive amounts of time:
{code}2019-08-12 12:10:23,323 DEBUG - [NotificationHookConsumer thread-0:] ~ No
RelationshipDef defined between rokku_client and server on attribute: server
(EntityGraphMapper:907)
2019-08-12 12:10:23,323 DEBUG - [NotificationHookConsumer thread-0:] ~ ==>
mapObjectIdValue(org.apache.atlas.repository.store.graph.v2.AttributeMutationContext@3f7c5b66)
(EntityGraphMapper:791)
2019-08-12 12:10:28,825 DEBUG - [NotificationHookConsumer thread-0:] ~ ==>
setProperty(edge[id=v6kwld-biutqg-xel1-x8oqg label=__rokku_client.server from
vertex[id=696799240 type=rokku_client
guid=30db90a8-8236-41c6-9438-4fa2ec14f349] -> to vertex[id=55832632 type=server
guid=9144e224-089f-4ae3-83bd-3472175d92a4]], __state, ACTIVE)
(AtlasGraphUtilsV2:212)
2019-08-12 12:10:23,323 DEBUG - [NotificationHookConsumer thread-0:] ~ ==>
mapObjectIdValue(org.apache.atlas.repository.store.graph.v2.AttributeMutationContext@3f7c5b66)
(EntityGraphMapper:791)
2019-08-12 12:10:28,825 DEBUG - [NotificationHookConsumer thread-0:] ~ ==>
setProperty(edge[id=v6kwld-biutqg-xel1-x8oqg label=__rokku_client.server from
vertex[id=696799240 type=rokku_client
guid=30db90a8-8236-41c6-9438-4fa2ec14f349] -> to vertex[id=55832632 type=server
guid=9144e224-089f-4ae3-83bd-3472175d92a4]], __state, ACTIVE)
(AtlasGraphUtilsV2:212)
{code}
Notice the 5 second(!) jump between the last two log lines. At a minimum the
log about triggering the legacy code path should be at WARN level and extended
with a suggestion to update the definition for better performance.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
Re: Review Request 71282: Expose total count of index searches
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71282/ --- (Updated Aug. 20, 2019, 9:47 a.m.) Review request for atlas, Nixon Rodrigues and Saqeeb Shaikh. Changes --- use approx Bugs: https://issues.apache.org/jira/browse/ATLAS-3367 https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3367 Repository: atlas Description --- Both ElasticSearch and SOLR can expose the total count of results without returning all results. This is useful to give the user or client an idea how many results there are. This patch ensures the total is returned if available. This total is an approximate as scrubbing of the results still needs to happen. Therefore, one should not only rely on this information to provide ,for example, pagination. Diffs (updated) - intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java d274cc07e repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 479ddfd89 repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java c67e34772 repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java f7d8f08c7 repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java 1dd1afaa3 repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java 0ffd61c07 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java a7ccaebc8 repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java c253d544b Diff: https://reviews.apache.org/r/71282/diff/3/ Changes: https://reviews.apache.org/r/71282/diff/2-3/ Testing --- Thanks, Bolke de Bruin
[jira] [Created] (ATLAS-3378) Update JanusGraph to 0.4.0
Bolke de Bruin created ATLAS-3378: - Summary: Update JanusGraph to 0.4.0 Key: ATLAS-3378 URL: https://issues.apache.org/jira/browse/ATLAS-3378 Project: Atlas Issue Type: Improvement Reporter: Bolke de Bruin This release supports hbase 2 out of the box -- This message was sent by Atlassian JIRA (v7.6.14#76016)
Re: Review Request 71278: Set log level to warn for relationships without relationship definition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71278/ --- (Updated Aug. 21, 2019, 7:42 p.m.) Review request for atlas, Nixon Rodrigues and Saqeeb Shaikh. Bugs: https://issues.apache.org/jira/browse/ATLAS-3368 https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3368 Repository: atlas Description --- If an entity is created that has a definition with a reference to a non built in type, it will trigger a legacy code path in EntityGraphMapper. This path can take excessive amounts of time (>5s per commit), but operators can be caught unaware as the log message for this is at debug level and explain the portential issue. Diffs - repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java f9010fefb Diff: https://reviews.apache.org/r/71278/diff/1/ Testing --- - reviewed patch logging to warn Thanks, Bolke de Bruin
[jira] [Commented] (ATLAS-3370) Aggregation Metrics with quick search, Counts don't add up
[ https://issues.apache.org/jira/browse/ATLAS-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912945#comment-16912945 ] Bolke de Bruin commented on ATLAS-3370: --- why is the indexing changed from fulltext to string based? this changes search behavior. per documentation of janus: When a string mapping is configured, the string value is indexed and can be queried "as-is" - including stop words and non-letter characters. However, in this case the query must match the entire string value. Hence, the string mapping is useful when indexing short character sequences that are considered to be one token. > Aggregation Metrics with quick search, Counts don't add up > -- > > Key: ATLAS-3370 > URL: https://issues.apache.org/jira/browse/ATLAS-3370 > Project: Atlas > Issue Type: Bug >Reporter: Sridhar >Assignee: Sridhar >Priority: Major > > The issue was happening because of tokenization done for the fields in issue. -- This message was sent by Atlassian Jira (v8.3.2#803003)
Re: Review Request 71344: Use fulltext indices for dsl search
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71344/ --- (Updated Aug. 22, 2019, 3:52 a.m.) Review request for atlas and Nixon Rodrigues. Changes --- fix test Bugs: https://issues.apache.org/jira/browse/ATLAS-3384 https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3384 Repository: atlas Description --- Per janusgraph documentation https://docs.janusgraph.org/latest/index-parameters.html strings are indexed as text by default. Atlas uses string search which is suboptimal and leads to significant performance loss. This switches to use fulltext predicates which give a significant speedup. Diffs (updated) - repository/src/main/java/org/apache/atlas/query/GremlinClause.java ca8419a8c repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 568667054 Diff: https://reviews.apache.org/r/71344/diff/2/ Changes: https://reviews.apache.org/r/71344/diff/1-2/ Testing --- Tested in prod with Solr/Hbase non embedded Thanks, Bolke de Bruin
[jira] [Commented] (ATLAS-3378) Update JanusGraph to 0.4.0
[ https://issues.apache.org/jira/browse/ATLAS-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916317#comment-16916317 ] Bolke de Bruin commented on ATLAS-3378: --- JanusGraph 0.4.0 now also supports native OrderBy giving a nice performance boost to sort clauses (when refactored to use OrderBy) > Update JanusGraph to 0.4.0 > -- > > Key: ATLAS-3378 > URL: https://issues.apache.org/jira/browse/ATLAS-3378 > Project: Atlas > Issue Type: Improvement > Reporter: Bolke de Bruin >Priority: Major > > This release supports hbase 2 out of the box -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (ATLAS-3397) Remove Solr6Index and use upstream version
[ https://issues.apache.org/jira/browse/ATLAS-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3397: -- Summary: Remove Solr6Index and use upstream version (was: Push Solr6Index improvements upstream and use Janus' version) > Remove Solr6Index and use upstream version > -- > > Key: ATLAS-3397 > URL: https://issues.apache.org/jira/browse/ATLAS-3397 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0 > Reporter: Bolke de Bruin >Priority: Critical > > Solr6Index has changes to support Kerberos and multiple Zookeeper clients. > There are several issues with Atlas' version: > * LICENSE for this file is incorrect as it is not licensed to the ASF. The > file says it is copied from Janus. > * It is outdated and is not benefitting from upstream improvements (like > indexed OrderBy of Janus 0.4) > * Kerberos has been integrated into Janus' version > * multiple Zookeeper clients have been implemented (although still using > ZOOKEEPER_URL, but it supports multiple entries: > [https://github.com/JanusGraph/janusgraph/blob/master/janusgraph-solr/src/main/java/org/janusgraph/diskstorage/solr/SolrIndex.java#L176] > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (ATLAS-3393) Fix inconsistency with attribute createtime in aws_s3_bucket
[ https://issues.apache.org/jira/browse/ATLAS-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921622#comment-16921622 ] Bolke de Bruin commented on ATLAS-3393: --- ping [~madhan.neethiraj] > Fix inconsistency with attribute createtime in aws_s3_bucket > > > Key: ATLAS-3393 > URL: https://issues.apache.org/jira/browse/ATLAS-3393 > Project: Atlas > Issue Type: Bug >Affects Versions: 1.2.0, 2.0.0 > Reporter: Bolke de Bruin >Priority: Critical > > attributes are in camelcase. in aws_s3_bukcket createtime isnt > > (fix available in linked issue/pr) > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (ATLAS-3397) Push Solr6Index improvements upstream and use Janus' version
[ https://issues.apache.org/jira/browse/ATLAS-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3397: -- Description: Solr6Index has changes to support Kerberos and multiple Zookeeper clients. There are several issues with Atlas' version: * LICENSE for this file is incorrect as it is not licensed to the ASF. The file says it is copied from Janus. * It is outdated and is not benefitting from upstream improvements (like indexed OrderBy of Janus 0.4) * Kerberos has been integrated into Janus' version * multiple Zookeeper clients have been implemented (although still using ZOOKEEPER_URL, but it supports multiple entries: [https://github.com/JanusGraph/janusgraph/blob/master/janusgraph-solr/src/main/java/org/janusgraph/diskstorage/solr/SolrIndex.java#L176] was: Solr6Index has changes to support Kerberos and multiple Zookeeper clients. There are several issues with Atlas' version: * LICENSE for this file is incorrect as it is not licensed to the ASF. * It is outdated and is not benefitting from upstream improvements (like indexed OrderBy of Janus 0.4) * Kerberos has been integrated into Janus' version * multiple Zookeeper clients have been implemented (although still using ZOOKEEPER_URL, but it supports multiple entries: [https://github.com/JanusGraph/janusgraph/blob/master/janusgraph-solr/src/main/java/org/janusgraph/diskstorage/solr/SolrIndex.java#L176] > Push Solr6Index improvements upstream and use Janus' version > > > Key: ATLAS-3397 > URL: https://issues.apache.org/jira/browse/ATLAS-3397 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0 > Reporter: Bolke de Bruin >Priority: Critical > > Solr6Index has changes to support Kerberos and multiple Zookeeper clients. > There are several issues with Atlas' version: > * LICENSE for this file is incorrect as it is not licensed to the ASF. The > file says it is copied from Janus. > * It is outdated and is not benefitting from upstream improvements (like > indexed OrderBy of Janus 0.4) > * Kerberos has been integrated into Janus' version > * multiple Zookeeper clients have been implemented (although still using > ZOOKEEPER_URL, but it supports multiple entries: > [https://github.com/JanusGraph/janusgraph/blob/master/janusgraph-solr/src/main/java/org/janusgraph/diskstorage/solr/SolrIndex.java#L176] > -- This message was sent by Atlassian Jira (v8.3.2#803003)
Please add me as a contributor to Atlas
Hello! Re subject. Can someone please add me as a contributor? Apache id “bolke”. Cheers Bolke.
[jira] [Comment Edited] (ATLAS-3393) Fix inconsistency with attribute createtime in aws_s3_bucket
[ https://issues.apache.org/jira/browse/ATLAS-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918344#comment-16918344 ] Bolke de Bruin edited comment on ATLAS-3393 at 8/29/19 7:11 AM: I understand that backward compatibility is important. However, we encountered this bug due to integration issues with our tooling expecting "createTime" as it is used everywhere else[1]. I do think it will create more problem down the road if you leave this inconstency in. "createTime" is also used hardcoded in [2]. If it is considered to be an "internal" attribute what happens if you mix the two? Is there a way to alias one to the other? So the trade off is that this inconsistency needs to be tested for in donwstream tooling and its not described in the documentation. So the question becomes how much tooling is already integrating with "createtime" and how much future tooling will assume "createTime". I think the latter will be bigger. [1] [https://github.com/apache/atlas/search?q=createTime_q=createTime] [2] [https://github.com/apache/atlas/blob/7e4788ed00c6a0ef4e8c89f0486b8073a56409e4/intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationship.java#L63] was (Author: bolke): I understand that backward compatibility is important. However, we encountered this bug due to integration issues with our tooling expecting "createTime" as it is used everywhere else[1]. I do think it will create more problem down the road if you leave this inconstency in. "createTime" is also used hardcoded in [2]. If it is considered to be an "internal" attribute what happens if you mix the two? Is there a way to alias one to the other? [1] https://github.com/apache/atlas/search?q=createTime_q=createTime [2] https://github.com/apache/atlas/blob/7e4788ed00c6a0ef4e8c89f0486b8073a56409e4/intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationship.java#L63 > Fix inconsistency with attribute createtime in aws_s3_bucket > > > Key: ATLAS-3393 > URL: https://issues.apache.org/jira/browse/ATLAS-3393 > Project: Atlas > Issue Type: Bug >Affects Versions: 1.2.0, 2.0.0 >Reporter: Bolke de Bruin >Priority: Critical > > attributes are in camelcase. in aws_s3_bukcket createtime isnt > > (fix available in linked issue/pr) > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (ATLAS-3393) Fix inconsistency with attribute createtime in aws_s3_bucket
[ https://issues.apache.org/jira/browse/ATLAS-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918344#comment-16918344 ] Bolke de Bruin edited comment on ATLAS-3393 at 8/29/19 7:12 AM: I understand that backward compatibility is important. However, we encountered this bug due to integration issues with our tooling expecting "createTime" as it is used everywhere else[1]. I do think it will create more problems down the road if you leave this inconstency in. "createTime" is also used hardcoded in [2]. If it is considered to be an "internal" attribute what happens if you mix the two? Is there a way to alias one to the other? So the trade off is that this inconsistency needs to be tested for in downstream tooling and it is not described in the documentation. So the question becomes how much tooling is already integrating with "createtime" and how much future tooling will assume "createTime". I think the latter will be bigger. [1] [https://github.com/apache/atlas/search?q=createTime_q=createTime] [2] [https://github.com/apache/atlas/blob/7e4788ed00c6a0ef4e8c89f0486b8073a56409e4/intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationship.java#L63] was (Author: bolke): I understand that backward compatibility is important. However, we encountered this bug due to integration issues with our tooling expecting "createTime" as it is used everywhere else[1]. I do think it will create more problems down the road if you leave this inconstency in. "createTime" is also used hardcoded in [2]. If it is considered to be an "internal" attribute what happens if you mix the two? Is there a way to alias one to the other? So the trade off is that this inconsistency needs to be tested for in donwstream tooling and it is not described in the documentation. So the question becomes how much tooling is already integrating with "createtime" and how much future tooling will assume "createTime". I think the latter will be bigger. [1] [https://github.com/apache/atlas/search?q=createTime_q=createTime] [2] [https://github.com/apache/atlas/blob/7e4788ed00c6a0ef4e8c89f0486b8073a56409e4/intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationship.java#L63] > Fix inconsistency with attribute createtime in aws_s3_bucket > > > Key: ATLAS-3393 > URL: https://issues.apache.org/jira/browse/ATLAS-3393 > Project: Atlas > Issue Type: Bug >Affects Versions: 1.2.0, 2.0.0 >Reporter: Bolke de Bruin >Priority: Critical > > attributes are in camelcase. in aws_s3_bukcket createtime isnt > > (fix available in linked issue/pr) > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (ATLAS-3393) Fix inconsistency with attribute createtime in aws_s3_bucket
[ https://issues.apache.org/jira/browse/ATLAS-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918344#comment-16918344 ] Bolke de Bruin edited comment on ATLAS-3393 at 8/29/19 7:12 AM: I understand that backward compatibility is important. However, we encountered this bug due to integration issues with our tooling expecting "createTime" as it is used everywhere else[1]. I do think it will create more problems down the road if you leave this inconstency in. "createTime" is also used hardcoded in [2]. If it is considered to be an "internal" attribute what happens if you mix the two? Is there a way to alias one to the other? So the trade off is that this inconsistency needs to be tested for in donwstream tooling and its not described in the documentation. So the question becomes how much tooling is already integrating with "createtime" and how much future tooling will assume "createTime". I think the latter will be bigger. [1] [https://github.com/apache/atlas/search?q=createTime_q=createTime] [2] [https://github.com/apache/atlas/blob/7e4788ed00c6a0ef4e8c89f0486b8073a56409e4/intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationship.java#L63] was (Author: bolke): I understand that backward compatibility is important. However, we encountered this bug due to integration issues with our tooling expecting "createTime" as it is used everywhere else[1]. I do think it will create more problem down the road if you leave this inconstency in. "createTime" is also used hardcoded in [2]. If it is considered to be an "internal" attribute what happens if you mix the two? Is there a way to alias one to the other? So the trade off is that this inconsistency needs to be tested for in donwstream tooling and its not described in the documentation. So the question becomes how much tooling is already integrating with "createtime" and how much future tooling will assume "createTime". I think the latter will be bigger. [1] [https://github.com/apache/atlas/search?q=createTime_q=createTime] [2] [https://github.com/apache/atlas/blob/7e4788ed00c6a0ef4e8c89f0486b8073a56409e4/intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationship.java#L63] > Fix inconsistency with attribute createtime in aws_s3_bucket > > > Key: ATLAS-3393 > URL: https://issues.apache.org/jira/browse/ATLAS-3393 > Project: Atlas > Issue Type: Bug >Affects Versions: 1.2.0, 2.0.0 >Reporter: Bolke de Bruin >Priority: Critical > > attributes are in camelcase. in aws_s3_bukcket createtime isnt > > (fix available in linked issue/pr) > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (ATLAS-3393) Fix inconsistency with attribute createtime in aws_s3_bucket
[ https://issues.apache.org/jira/browse/ATLAS-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918344#comment-16918344 ] Bolke de Bruin edited comment on ATLAS-3393 at 8/29/19 7:12 AM: I understand that backward compatibility is important. However, we encountered this bug due to integration issues with our tooling expecting "createTime" as it is used everywhere else[1]. I do think it will create more problems down the road if you leave this inconstency in. "createTime" is also used hardcoded in [2]. If it is considered to be an "internal" attribute what happens if you mix the two? Is there a way to alias one to the other? So the trade off is that this inconsistency needs to be tested for in donwstream tooling and it is not described in the documentation. So the question becomes how much tooling is already integrating with "createtime" and how much future tooling will assume "createTime". I think the latter will be bigger. [1] [https://github.com/apache/atlas/search?q=createTime_q=createTime] [2] [https://github.com/apache/atlas/blob/7e4788ed00c6a0ef4e8c89f0486b8073a56409e4/intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationship.java#L63] was (Author: bolke): I understand that backward compatibility is important. However, we encountered this bug due to integration issues with our tooling expecting "createTime" as it is used everywhere else[1]. I do think it will create more problems down the road if you leave this inconstency in. "createTime" is also used hardcoded in [2]. If it is considered to be an "internal" attribute what happens if you mix the two? Is there a way to alias one to the other? So the trade off is that this inconsistency needs to be tested for in donwstream tooling and its not described in the documentation. So the question becomes how much tooling is already integrating with "createtime" and how much future tooling will assume "createTime". I think the latter will be bigger. [1] [https://github.com/apache/atlas/search?q=createTime_q=createTime] [2] [https://github.com/apache/atlas/blob/7e4788ed00c6a0ef4e8c89f0486b8073a56409e4/intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationship.java#L63] > Fix inconsistency with attribute createtime in aws_s3_bucket > > > Key: ATLAS-3393 > URL: https://issues.apache.org/jira/browse/ATLAS-3393 > Project: Atlas > Issue Type: Bug >Affects Versions: 1.2.0, 2.0.0 >Reporter: Bolke de Bruin >Priority: Critical > > attributes are in camelcase. in aws_s3_bukcket createtime isnt > > (fix available in linked issue/pr) > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (ATLAS-3393) Fix inconsistency with attribute createtime in aws_s3_bucket
[ https://issues.apache.org/jira/browse/ATLAS-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3393: -- Affects Version/s: 1.2.0 > Fix inconsistency with attribute createtime in aws_s3_bucket > > > Key: ATLAS-3393 > URL: https://issues.apache.org/jira/browse/ATLAS-3393 > Project: Atlas > Issue Type: Bug >Affects Versions: 1.2.0, 2.0.0 > Reporter: Bolke de Bruin >Priority: Critical > > attributes are in camelcase. in aws_s3_bukcket createtime isnt > > (fix available in linked issue/pr) > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (ATLAS-3393) Fix inconsistency with attribute createtime in aws_s3_bucket
[ https://issues.apache.org/jira/browse/ATLAS-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918344#comment-16918344 ] Bolke de Bruin commented on ATLAS-3393: --- I understand that backward compatibility is important. However, we encountered this bug due to integration issues with our tooling expecting "createTime" as it is used everywhere else[1]. I do think it will create more problem down the road if you leave this inconstency in. "createTime" is also used hardcoded in [2]. If it is considered to be an "internal" attribute what happens if you mix the two? Is there a way to alias one to the other? [1] https://github.com/apache/atlas/search?q=createTime_q=createTime [2] https://github.com/apache/atlas/blob/7e4788ed00c6a0ef4e8c89f0486b8073a56409e4/intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationship.java#L63 > Fix inconsistency with attribute createtime in aws_s3_bucket > > > Key: ATLAS-3393 > URL: https://issues.apache.org/jira/browse/ATLAS-3393 > Project: Atlas > Issue Type: Bug > Affects Versions: 2.0.0 >Reporter: Bolke de Bruin >Priority: Critical > > attributes are in camelcase. in aws_s3_bukcket createtime isnt > > (fix available in linked issue/pr) > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (ATLAS-3393) Fix inconsistency with attribute createtime in aws_s3_bucket
Bolke de Bruin created ATLAS-3393: - Summary: Fix inconsistency with attribute createtime in aws_s3_bucket Key: ATLAS-3393 URL: https://issues.apache.org/jira/browse/ATLAS-3393 Project: Atlas Issue Type: Bug Affects Versions: 2.0.0 Reporter: Bolke de Bruin attributes are in camelcase. in aws_s3_bukcket createtime isnt (fix available in linked issue/pr) -- This message was sent by Atlassian Jira (v8.3.2#803003)
Review requests pending
Hi All, Sorry for reaching out directly, but I have a few outstanding review requests that didn’t get any attention yet. It would be awesome if you could take a look at it. These are improvements in performance and usability. They are not overly intrusive. We have a couple more upcoming hence I am reaching out now to ensure we won’t get a very big backlog ;-). https://reviews.apache.org/r/71278/ -> Set log level to warn for relationships without relationship definition https://reviews.apache.org/r/71344/ -> Use fulltext indices for dsl search https://reviews.apache.org/r/71282/ -> Expose total count of index searches Cheers Bolke
Re: Review Request 71344: Use fulltext indices for dsl search
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71344/ --- (Updated Aug. 31, 2019, 8:21 p.m.) Review request for atlas, Ashutosh Mestry and Nixon Rodrigues. Changes --- This select the right predicate based on the specified index. This was required after merging of ATLAS-3370 Bugs: https://issues.apache.org/jira/browse/ATLAS-3384 https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3384 Repository: atlas Description --- Per janusgraph documentation https://docs.janusgraph.org/latest/index-parameters.html strings are indexed as text by default. Atlas uses string search which is suboptimal and leads to significant performance loss. This switches to use fulltext predicates which give a significant speedup. Diffs (updated) - repository/src/main/java/org/apache/atlas/query/GremlinClause.java ca8419a8c repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java e64a89436 repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java b73d42716 Diff: https://reviews.apache.org/r/71344/diff/3/ Changes: https://reviews.apache.org/r/71344/diff/2-3/ Testing --- Tested in prod with Solr/Hbase non embedded Thanks, Bolke de Bruin
[jira] [Created] (ATLAS-3384) Use proper indexed queries for DSL search
Bolke de Bruin created ATLAS-3384:
-
Summary: Use proper indexed queries for DSL search
Key: ATLAS-3384
URL: https://issues.apache.org/jira/browse/ATLAS-3384
Project: Atlas
Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Bolke de Bruin
Per janusgraph documentation
[https://docs.janusgraph.org/latest/index-parameters.html] strings are indexed
as text by default. To use the indexes one is required to use
* {{textContains}}: is true if (at least) one word inside the text string
matches the query string
* {{textContainsPrefix}}: is true if (at least) one word inside the text
string begins with the query string
* {{textContainsRegex}}: is true if (at least) one word inside the text string
matches the given regular expression
* {{textContainsFuzzy}}: is true if (at least) one word inside the text string
is similar to the query String (based on Levenshtein edit distance)
Atlas uses string search which is suboptimal and leads to significant
performance loss. We have observed subsecond queries vs half minute pkus
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Comment Edited] (ATLAS-3370) Aggregation Metrics with quick search, Counts don't add up
[
https://issues.apache.org/jira/browse/ATLAS-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913656#comment-16913656
]
Bolke de Bruin edited comment on ATLAS-3370 at 8/22/19 8:04 PM:
[~Koritala] This is not congruent with our observations and I think there there
was at a minimum an oversight that all fields that are not indexed with
"STRING" are automatically indexed as "TEXT" _("By default, strings are indexed
as text. To make this indexing option explicit, one can define a mapping when
indexing a property key as text." [1])_, this means all text attributes from
the past. This is not tied to the Fulltext index created by Atlas. Also note
that Freetext is not supported on Elastic.
We were debugging DSL search performance issues on Atlas 2.0 configured with
Solr (non embedded) and hbase 2. It uses the Freetext search (confirmed there
is no fulltext index in Solr). The basic search was returning subsecond, but
for the same query the DSL search came back 20 seconds later or more. We traced
it down to the use of incorrect predicates per the documentation of
JanusGraph[1]. When we changed it to use the correct predicates we had the same
response times.
So this change has the consequence that indices are now mixed instead of one
type, however the predicate usage in master is always geared to use STRING and
thus reverts to in memory sorting in many cases which is slow. The change we
were proposing in the linked issue would change the predicate to use the
correct one for the index.
This change complicates matters in that it requires the GremlinQueryComposer to
be aware of the index used for the attribute and then select the right
predicate usage. Or expose this complexity to the user by having something like
"FULLTEXT_LIKE".
1. [https://docs.janusgraph.org/latest/index-parameters.html]
was (Author: bolke):
[~Koritala] This is not congruent with our observations and I think there there
was at a minimum an oversight that all fields that are not indexed with
"STRING" are automatically indexed as "TEXT" _("By default, strings are indexed
as text. To make this indexing option explicit, one can define a mapping when
indexing a property key as text." [1])_, this means all text attributes from
the past. This is not tight to the Fulltext index created by Atlas. Also note
that Freetext is not supported on Elastic.
We were debugging DSL search performance issues on Atlas 2.0 configured with
Solr (non embedded) and hbase 2. It uses the Freetext search (confirmed there
is no fulltext index in Solr). The basic search was returning subsecond, but
for the same query the DSL search came back 20 seconds later or more. We traced
it down to the use of incorrect predicates per the documentation of
JanusGraph[1]. When we changed it to use the correct predicates we had the same
response times.
So this change has the consequence that indices are now mixed instead of one
type, however the predicate usage in master is always geared to use STRING and
thus reverts to in memory sorting in many cases which is slow. The change we
were proposing in the linked issue would change the predicate to use the
correct one for the index.
This change complicates matters in that it requires the GremlinQueryComposer to
be aware of the index used for the attribute and then select the right
predicate usage. Or expose this complexity to the user by having something like
"FULLTEXT_LIKE".
1. https://docs.janusgraph.org/latest/index-parameters.html
> Aggregation Metrics with quick search, Counts don't add up
> --
>
> Key: ATLAS-3370
> URL: https://issues.apache.org/jira/browse/ATLAS-3370
> Project: Atlas
> Issue Type: Bug
>Reporter: Sridhar
>Assignee: Sridhar
>Priority: Major
>
> The issue was happening because of tokenization done for the fields in issue.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (ATLAS-3398) Duplicates for unique attributes
[ https://issues.apache.org/jira/browse/ATLAS-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3398: -- Description: We are seeing issues with entities being added to Atlas with duplicate "qualifiedName". The guids differ and other attributes do also differ. Below a graph that shows the distribution over time for duplicates. We have difficulty determining which one is the right one (as they are different) in order to clean them up. We are also not the only ones encountering this as you can in the linked issue. We have noticed that Atlas does not use the [locking|https://docs.janusgraph.org/master/advanced-topics/eventual-consistency/] mechanism of Janus to prevent this: !zrzut_ekranu_2019-09-03_o_10.28.50.png! was: We are seeing issues with entities being added to Atlas with duplicate "qualifiedName". The guids differ and other attributes do also differ. Below a graph that shows the distribution over time for duplicates. We have difficulty determining which one is the right one (as they are different) in order to clean them up. We are also not the only ones encountering this as you can in the linked issue. !zrzut_ekranu_2019-09-03_o_10.28.50.png! > Duplicates for unique attributes > - > > Key: ATLAS-3398 > URL: https://issues.apache.org/jira/browse/ATLAS-3398 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.0.0, trunk >Reporter: Bolke de Bruin >Priority: Blocker > Attachments: zrzut_ekranu_2019-09-03_o_10.28.50.png > > > We are seeing issues with entities being added to Atlas with duplicate > "qualifiedName". The guids differ and other attributes do also differ. Below > a graph that shows the distribution over time for duplicates. We have > difficulty determining which one is the right one (as they are different) in > order to clean them up. > We are also not the only ones encountering this as you can in the linked > issue. > We have noticed that Atlas does not use the > [locking|https://docs.janusgraph.org/master/advanced-topics/eventual-consistency/] > mechanism of Janus to prevent this: > > !zrzut_ekranu_2019-09-03_o_10.28.50.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (ATLAS-3398) Duplicates for unique attributes
[ https://issues.apache.org/jira/browse/ATLAS-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3398: -- Component/s: atlas-core > Duplicates for unique attributes > - > > Key: ATLAS-3398 > URL: https://issues.apache.org/jira/browse/ATLAS-3398 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0, trunk > Reporter: Bolke de Bruin >Priority: Blocker > Attachments: zrzut_ekranu_2019-09-03_o_10.28.50.png > > > We are seeing issues with entities being added to Atlas with duplicate > "qualifiedName". The guids differ and other attributes do also differ. Below > a graph that shows the distribution over time for duplicates. We have > difficulty determining which one is the right one (as they are different) in > order to clean them up. > We are also not the only ones encountering this as you can in the linked > issue. > We have noticed that Atlas does not use the > [locking|https://docs.janusgraph.org/master/advanced-topics/eventual-consistency/] > mechanism of Janus to prevent this: > > !zrzut_ekranu_2019-09-03_o_10.28.50.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (ATLAS-3398) Duplicates for unique attributes
Bolke de Bruin created ATLAS-3398: - Summary: Duplicates for unique attributes Key: ATLAS-3398 URL: https://issues.apache.org/jira/browse/ATLAS-3398 Project: Atlas Issue Type: Bug Affects Versions: 2.0.0, trunk Reporter: Bolke de Bruin We are seeing issues with entities being added to Atlas with duplicate "qualifiedName". The guids differ and other attributes do also differ. Below a graph that shows the distribution over time for duplicates. We have difficulty determining which one is the right one (as they are different) in order to clean them up. We are also not the only ones encountering this as you can in the linked issue. !https://files.slack.com/files-pri/T9LB9QRT8-FMM67SSQK/zrzut_ekranu_2019-09-03_o_10.28.50.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (ATLAS-3398) Duplicates for unique attributes
[ https://issues.apache.org/jira/browse/ATLAS-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3398: -- Attachment: zrzut_ekranu_2019-09-03_o_10.28.50.png > Duplicates for unique attributes > - > > Key: ATLAS-3398 > URL: https://issues.apache.org/jira/browse/ATLAS-3398 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.0.0, trunk > Reporter: Bolke de Bruin >Priority: Blocker > Attachments: zrzut_ekranu_2019-09-03_o_10.28.50.png > > > We are seeing issues with entities being added to Atlas with duplicate > "qualifiedName". The guids differ and other attributes do also differ. Below > a graph that shows the distribution over time for duplicates. We have > difficulty determining which one is the right one (as they are different) in > order to clean them up. > We are also not the only ones encountering this as you can in the linked > issue. > > !https://files.slack.com/files-pri/T9LB9QRT8-FMM67SSQK/zrzut_ekranu_2019-09-03_o_10.28.50.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (ATLAS-3398) Duplicates for unique attributes
[ https://issues.apache.org/jira/browse/ATLAS-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated ATLAS-3398: -- Description: We are seeing issues with entities being added to Atlas with duplicate "qualifiedName". The guids differ and other attributes do also differ. Below a graph that shows the distribution over time for duplicates. We have difficulty determining which one is the right one (as they are different) in order to clean them up. We are also not the only ones encountering this as you can in the linked issue. !zrzut_ekranu_2019-09-03_o_10.28.50.png! was: We are seeing issues with entities being added to Atlas with duplicate "qualifiedName". The guids differ and other attributes do also differ. Below a graph that shows the distribution over time for duplicates. We have difficulty determining which one is the right one (as they are different) in order to clean them up. We are also not the only ones encountering this as you can in the linked issue. !https://files.slack.com/files-pri/T9LB9QRT8-FMM67SSQK/zrzut_ekranu_2019-09-03_o_10.28.50.png! > Duplicates for unique attributes > - > > Key: ATLAS-3398 > URL: https://issues.apache.org/jira/browse/ATLAS-3398 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.0.0, trunk >Reporter: Bolke de Bruin >Priority: Blocker > Attachments: zrzut_ekranu_2019-09-03_o_10.28.50.png > > > We are seeing issues with entities being added to Atlas with duplicate > "qualifiedName". The guids differ and other attributes do also differ. Below > a graph that shows the distribution over time for duplicates. We have > difficulty determining which one is the right one (as they are different) in > order to clean them up. > We are also not the only ones encountering this as you can in the linked > issue. > > !zrzut_ekranu_2019-09-03_o_10.28.50.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (ATLAS-3393) Fix inconsistency with attribute createtime in aws_s3_bucket
[ https://issues.apache.org/jira/browse/ATLAS-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin reassigned ATLAS-3393: - Assignee: Bolke de Bruin > Fix inconsistency with attribute createtime in aws_s3_bucket > > > Key: ATLAS-3393 > URL: https://issues.apache.org/jira/browse/ATLAS-3393 > Project: Atlas > Issue Type: Bug >Affects Versions: 1.2.0, 2.0.0 > Reporter: Bolke de Bruin > Assignee: Bolke de Bruin >Priority: Critical > > attributes are in camelcase. in aws_s3_bukcket createtime isnt > > (fix available in linked issue/pr) > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (ATLAS-3393) Fix inconsistency with attribute createtime in aws_s3_bucket
[ https://issues.apache.org/jira/browse/ATLAS-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922253#comment-16922253 ] Bolke de Bruin commented on ATLAS-3393: --- [~madhan.neethiraj] got your point. Thanks for being elaborate. > Fix inconsistency with attribute createtime in aws_s3_bucket > > > Key: ATLAS-3393 > URL: https://issues.apache.org/jira/browse/ATLAS-3393 > Project: Atlas > Issue Type: Bug >Affects Versions: 1.2.0, 2.0.0 > Reporter: Bolke de Bruin >Priority: Critical > > attributes are in camelcase. in aws_s3_bukcket createtime isnt > > (fix available in linked issue/pr) > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
Re: Review Request 71278: Set log level to warn for relationships without relationship definition
> On sep 5, 2019, 8:15 p.m., Ashutosh Mestry wrote: > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > > Line 907 (original), 907 (patched) > > <https://reviews.apache.org/r/71278/diff/1/?file=2160649#file2160649line907> > > > > Are you OK with this warning being under debug log level? thanks for reviewing. I dont follow you comment? We noticed performance (10s per entity for ingestion) issues but we had to turn on debugging to find that we were triggering an unoptimized code path. Having this at warn would have saved us a lot of time and maybe others. INFO might work to. happy to change it to that if that is what you require. - Bolke --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71278/#review217597 ------- On aug 21, 2019, 7:42 p.m., Bolke de Bruin wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71278/ > --- > > (Updated aug 21, 2019, 7:42 p.m.) > > > Review request for atlas, Nixon Rodrigues and Saqeeb Shaikh. > > > Bugs: https://issues.apache.org/jira/browse/ATLAS-3368 > > https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3368 > > > Repository: atlas > > > Description > --- > > If an entity is created that has a definition with a reference to a > non built in type, it will trigger a legacy code path in > EntityGraphMapper. > This path can take excessive amounts of time (>5s per commit), but > operators > can be caught unaware as the log message for this is at debug > level and explain the portential issue. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > f9010fefb > > > Diff: https://reviews.apache.org/r/71278/diff/1/ > > > Testing > --- > > - reviewed patch logging to warn > > > Thanks, > > Bolke de Bruin > >
Re: Review Request 71278: Set log level to warn for relationships without relationship definition
> On sep 5, 2019, 8:15 p.m., Ashutosh Mestry wrote: > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > > Line 907 (original), 907 (patched) > > <https://reviews.apache.org/r/71278/diff/1/?file=2160649#file2160649line907> > > > > Are you OK with this warning being under debug log level? > > Bolke de Bruin wrote: > thanks for reviewing. I dont follow you comment? We noticed performance > (10s per entity for ingestion) issues but we had to turn on debugging to find > that we were triggering an unoptimized code path. Having this at warn would > have saved us a lot of time and maybe others. INFO might work to. happy to > change it to that if that is what you require. arghhh got what you mean. will fix that - Bolke --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71278/#review217597 ----------- On aug 21, 2019, 7:42 p.m., Bolke de Bruin wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71278/ > --- > > (Updated aug 21, 2019, 7:42 p.m.) > > > Review request for atlas, Nixon Rodrigues and Saqeeb Shaikh. > > > Bugs: https://issues.apache.org/jira/browse/ATLAS-3368 > > https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3368 > > > Repository: atlas > > > Description > --- > > If an entity is created that has a definition with a reference to a > non built in type, it will trigger a legacy code path in > EntityGraphMapper. > This path can take excessive amounts of time (>5s per commit), but > operators > can be caught unaware as the log message for this is at debug > level and explain the portential issue. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > f9010fefb > > > Diff: https://reviews.apache.org/r/71278/diff/1/ > > > Testing > --- > > - reviewed patch logging to warn > > > Thanks, > > Bolke de Bruin > >
[jira] [Created] (ATLAS-3399) Add order by (sort) to basic search
Bolke de Bruin created ATLAS-3399: - Summary: Add order by (sort) to basic search Key: ATLAS-3399 URL: https://issues.apache.org/jira/browse/ATLAS-3399 Project: Atlas Issue Type: New Feature Components: atlas-core Affects Versions: 2.0.0 Reporter: Bolke de Bruin Fix For: trunk JanusGraph 0.4.0 has added support for indexed "Order By" using the search backend. This would allow sorting on names, popularity scores etc en improve the capabilities for data discovery -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (ATLAS-3399) Add order by (sort) to basic search
[ https://issues.apache.org/jira/browse/ATLAS-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin reassigned ATLAS-3399: - Assignee: Bolke de Bruin > Add order by (sort) to basic search > --- > > Key: ATLAS-3399 > URL: https://issues.apache.org/jira/browse/ATLAS-3399 > Project: Atlas > Issue Type: New Feature > Components: atlas-core >Affects Versions: 2.0.0 > Reporter: Bolke de Bruin > Assignee: Bolke de Bruin >Priority: Major > Fix For: trunk > > > JanusGraph 0.4.0 has added support for indexed "Order By" using the search > backend. This would allow sorting on names, popularity scores etc en improve > the capabilities for data discovery -- This message was sent by Atlassian Jira (v8.3.2#803003)
Outstanding review requests
Hello Team! I have some outstanding review requests and I would appreciate feedback or (duh) merging. Can someone take a look: * Add order by (sort) to basic search : https://reviews.apache.org/r/71431/ * Use fulltext indices for dsl search : https://reviews.apache.org/r/71344/ * Expose total count of index searches : https://reviews.apache.org/r/71282/ * Set log level to warn for relationships without relationship definition : https://reviews.apache.org/r/71278/ Thanks! Bolke
[jira] [Commented] (ATLAS-3409) Remove the duplicate code in Solr6Index.
[ https://issues.apache.org/jira/browse/ATLAS-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928633#comment-16928633 ] Bolke de Bruin commented on ATLAS-3409: --- This is a duplicate of atlas-3397 > Remove the duplicate code in Solr6Index. > > > Key: ATLAS-3409 > URL: https://issues.apache.org/jira/browse/ATLAS-3409 > Project: Atlas > Issue Type: Improvement >Reporter: Sridhar >Assignee: Sridhar >Priority: Major > > Historically, we made a copy of SolrIndex from Janus graph and referred to it > as Solr6Index. Over time, SolrIndex in janus graph has mode enough progress > w.r.t Kerberos support etc. So, we should remove the duplicate code > Solr6Index. > This ticket should be done after ATLAS-3378 is completed. -- This message was sent by Atlassian Jira (v8.3.2#803003)
Fwd: Outstanding review requests (patches!)
Ping! I wonder what to do to grab some attention, because I see there is even some duplication of effort going on now - ATLAS-3399 is partially duplicated ATLAS-3378 which did receive reviews quickly? Do I need to add reviewers? I do think the added functionality is worthwhile and will benefit others too, but please let me know if those patches don’t make sense. Thanks for responding. Bolke On 4 September 2019 at 14:47:41, Bolke de Bruin (bdbr...@gmail.com) wrote: Hello Team! I have some outstanding review requests and I would appreciate feedback or (duh) merging. Can someone take a look: * Add order by (sort) to basic search : https://reviews.apache.org/r/71431/ * Use fulltext indices for dsl search : https://reviews.apache.org/r/71344/ (Received one “ship it!”, no further action) * Expose total count of index searches : https://reviews.apache.org/r/71282/ * Set log level to warn for relationships without relationship definition : https://reviews.apache.org/r/71278/. (This one is under review) Thanks! Bolke
[jira] [Commented] (ATLAS-3378) Update JanusGraph to 0.4.0
[ https://issues.apache.org/jira/browse/ATLAS-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926837#comment-16926837 ] Bolke de Bruin commented on ATLAS-3378: --- see also: ATLAS-3399 > Update JanusGraph to 0.4.0 > -- > > Key: ATLAS-3378 > URL: https://issues.apache.org/jira/browse/ATLAS-3378 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0 > Reporter: Bolke de Bruin >Assignee: Sarath Subramanian >Priority: Major > Fix For: 2.1.0 > > > This release supports hbase 2 out of the box -- This message was sent by Atlassian Jira (v8.3.2#803003)
Re: Review Request 71278: Set log level to warn for relationships without relationship definition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71278/ --- (Updated Sept. 14, 2019, 1:09 p.m.) Review request for atlas, Nixon Rodrigues and Saqeeb Shaikh. Changes --- Addressed comments Bugs: https://issues.apache.org/jira/browse/ATLAS-3368 https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3368 Repository: atlas Description --- If an entity is created that has a definition with a reference to a non built in type, it will trigger a legacy code path in EntityGraphMapper. This path can take excessive amounts of time (>5s per commit), but operators can be caught unaware as the log message for this is at debug level and explain the portential issue. Diffs (updated) - docs/src/site/twiki/Configuration.twiki 6190a7c5f intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 345b105ff repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java c02f80905 Diff: https://reviews.apache.org/r/71278/diff/2/ Changes: https://reviews.apache.org/r/71278/diff/1-2/ Testing --- - reviewed patch logging to warn Thanks, Bolke de Bruin
Re: Review Request 71431: Add indexed order by (sort) to basic search
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71431/ --- (Updated Sept. 16, 2019, 6:38 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, Madhan Neethiraj, and Nixon Rodrigues. Changes --- addressing comments Bugs: https://issues.apache.org/jira/browse/ATLAS-3399 https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3399 Repository: atlas Description --- Add indexed order by to basic search. Diffs (updated) - graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java f3722b827 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusIndexQuery.java 7c258b7a3 graphdb/janus/src/main/java/org/janusgraph/diskstorage/solr/Solr6Index.java e457866af intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java aac6b5aa3 repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 479ddfd89 repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java f7d8f08c7 repository/src/main/java/org/apache/atlas/discovery/SearchContext.java 9bd0382eb webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java e6b338b84 webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java 06931b38c webapp/src/test/resources/json/search-parameters/entity-filters.json f4b2efe63 webapp/src/test/resources/json/search-parameters/tag-filters.json 5e74328d6 Diff: https://reviews.apache.org/r/71431/diff/3/ Changes: https://reviews.apache.org/r/71431/diff/2-3/ Testing --- - production and tests added Thanks, Bolke de Bruin
Re: Review Request 71431: Add indexed order by (sort) to basic search
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71431/ --- (Updated Sept. 16, 2019, 7:03 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, Madhan Neethiraj, and Nixon Rodrigues. Changes --- Addressed comments. Only verification of attributename against entity Bugs: https://issues.apache.org/jira/browse/ATLAS-3399 https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3399 Repository: atlas Description --- Add indexed order by to basic search. Diffs (updated) - graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java f3722b827 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusIndexQuery.java 7c258b7a3 graphdb/janus/src/main/java/org/janusgraph/diskstorage/solr/Solr6Index.java e457866af intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java aac6b5aa3 repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 479ddfd89 repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java f7d8f08c7 repository/src/main/java/org/apache/atlas/discovery/SearchContext.java 9bd0382eb webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java e6b338b84 webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java 06931b38c webapp/src/test/resources/json/search-parameters/entity-filters.json f4b2efe63 webapp/src/test/resources/json/search-parameters/tag-filters.json 5e74328d6 Diff: https://reviews.apache.org/r/71431/diff/5/ Changes: https://reviews.apache.org/r/71431/diff/4-5/ Testing --- - production and tests added Thanks, Bolke de Bruin
Re: Review Request 71431: Add indexed order by (sort) to basic search
> On Sept. 14, 2019, 7:16 p.m., Madhan Neethiraj wrote: > > repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java > > Lines 205 (patched) > > <https://reviews.apache.org/r/71431/diff/2/?file=2165082#file2165082line205> > > > > graphQuery requires 'qualifiedName' of the attribute (i.e. Asset.name, > > Asset.owner) - refer to SearchProcessor.toGraphFilterQuery(). Is 'sortBy' > > here qualifiedName of the attribute? > > Bolke de Bruin wrote: > Good point. It seems (but please correct me if I am wrong) given the > tests that it is not required. However, I assume I will need to update > SearchParameters to validate whether the attribute exists? > > Madhan Neethiraj wrote: > Validation of attribute names is handled in > SearchContext.validateAttributes(); consider updating this method to validate > the new field SearchParameters.sortBy. Can you search on classificationType and entityType at the same time? I'm assuming that I need to verify the sortBy parameter against entityType and classifictionType. It seems a bit odd though - Bolke --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71431/#review217738 ------- On Sept. 14, 2019, 12:45 p.m., Bolke de Bruin wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71431/ > --- > > (Updated Sept. 14, 2019, 12:45 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, > Madhan Neethiraj, and Nixon Rodrigues. > > > Bugs: https://issues.apache.org/jira/browse/ATLAS-3399 > > https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3399 > > > Repository: atlas > > > Description > --- > > Add indexed order by to basic search. > > > Diffs > - > > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java > f3722b827 > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusIndexQuery.java > 7c258b7a3 > graphdb/janus/src/main/java/org/janusgraph/diskstorage/solr/Solr6Index.java > e457866af > intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java > aac6b5aa3 > > repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java > f7d8f08c7 > webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java e6b338b84 > webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java > 06931b38c > webapp/src/test/resources/json/search-parameters/entity-filters.json > f4b2efe63 > > > Diff: https://reviews.apache.org/r/71431/diff/2/ > > > Testing > --- > > - production and tests added > > > Thanks, > > Bolke de Bruin > >
Re: Review Request 71431: Add indexed order by (sort) to basic search
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71431/ --- (Updated Sept. 16, 2019, 7:10 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, Madhan Neethiraj, and Nixon Rodrigues. Changes --- Addressed comment. Bugs: https://issues.apache.org/jira/browse/ATLAS-3399 https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3399 Repository: atlas Description --- Add indexed order by to basic search. Diffs (updated) - graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java f3722b827 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusIndexQuery.java 7c258b7a3 graphdb/janus/src/main/java/org/janusgraph/diskstorage/solr/Solr6Index.java e457866af intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java aac6b5aa3 repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 479ddfd89 repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java f7d8f08c7 repository/src/main/java/org/apache/atlas/discovery/SearchContext.java 9bd0382eb webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java e6b338b84 webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java 06931b38c webapp/src/test/resources/json/search-parameters/entity-filters.json f4b2efe63 webapp/src/test/resources/json/search-parameters/tag-filters.json 5e74328d6 Diff: https://reviews.apache.org/r/71431/diff/6/ Changes: https://reviews.apache.org/r/71431/diff/5-6/ Testing --- - production and tests added Thanks, Bolke de Bruin
Re: Review Request 71431: Add indexed order by (sort) to basic search
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71431/ --- (Updated Sept. 16, 2019, 6:45 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, Madhan Neethiraj, and Nixon Rodrigues. Changes --- Addressed other comments. - Question: how to test the classification orderby properly. What 'attribute name' is possible? Bugs: https://issues.apache.org/jira/browse/ATLAS-3399 https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3399 Repository: atlas Description --- Add indexed order by to basic search. Diffs (updated) - graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java f3722b827 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusIndexQuery.java 7c258b7a3 graphdb/janus/src/main/java/org/janusgraph/diskstorage/solr/Solr6Index.java e457866af intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java aac6b5aa3 repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 479ddfd89 repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java f7d8f08c7 repository/src/main/java/org/apache/atlas/discovery/SearchContext.java 9bd0382eb webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java e6b338b84 webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java 06931b38c webapp/src/test/resources/json/search-parameters/entity-filters.json f4b2efe63 webapp/src/test/resources/json/search-parameters/tag-filters.json 5e74328d6 Diff: https://reviews.apache.org/r/71431/diff/4/ Changes: https://reviews.apache.org/r/71431/diff/3-4/ Testing --- - production and tests added Thanks, Bolke de Bruin
Re: Review Request 71431: Add indexed order by (sort) to basic search
> On Sept. 14, 2019, 7:16 p.m., Madhan Neethiraj wrote: > > repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java > > Lines 205 (patched) > > <https://reviews.apache.org/r/71431/diff/2/?file=2165082#file2165082line205> > > > > graphQuery requires 'qualifiedName' of the attribute (i.e. Asset.name, > > Asset.owner) - refer to SearchProcessor.toGraphFilterQuery(). Is 'sortBy' > > here qualifiedName of the attribute? > > Bolke de Bruin wrote: > Good point. It seems (but please correct me if I am wrong) given the > tests that it is not required. However, I assume I will need to update > SearchParameters to validate whether the attribute exists? > > Madhan Neethiraj wrote: > Validation of attribute names is handled in > SearchContext.validateAttributes(); consider updating this method to validate > the new field SearchParameters.sortBy. > > Bolke de Bruin wrote: > Can you search on classificationType and entityType at the same time? I'm > assuming that I need to verify the sortBy parameter against entityType and > classifictionType. It seems a bit odd though Ah used the ui to find out a bit more. Now I am going to assume I just need to validate against entityType. - Bolke --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71431/#review217738 ------- On Sept. 16, 2019, 6:45 p.m., Bolke de Bruin wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71431/ > --- > > (Updated Sept. 16, 2019, 6:45 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, > Madhan Neethiraj, and Nixon Rodrigues. > > > Bugs: https://issues.apache.org/jira/browse/ATLAS-3399 > > https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3399 > > > Repository: atlas > > > Description > --- > > Add indexed order by to basic search. > > > Diffs > - > > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java > f3722b827 > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusIndexQuery.java > 7c258b7a3 > graphdb/janus/src/main/java/org/janusgraph/diskstorage/solr/Solr6Index.java > e457866af > intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java > aac6b5aa3 > > repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java > 479ddfd89 > > repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java > f7d8f08c7 > repository/src/main/java/org/apache/atlas/discovery/SearchContext.java > 9bd0382eb > webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java e6b338b84 > webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java > 06931b38c > webapp/src/test/resources/json/search-parameters/entity-filters.json > f4b2efe63 > webapp/src/test/resources/json/search-parameters/tag-filters.json 5e74328d6 > > > Diff: https://reviews.apache.org/r/71431/diff/4/ > > > Testing > --- > > - production and tests added > > > Thanks, > > Bolke de Bruin > >
Re: Review Request 71282: Expose total count of index searches
> On Sept. 14, 2019, 4:53 p.m., Madhan Neethiraj wrote: > > intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java > > Lines 71 (patched) > > <https://reviews.apache.org/r/71282/diff/3/?file=2161896#file2161896line71> > > > > I suggest to initialize approximateCount to "-1", to enable > > distinguishing "empty results" from "unavailability of approximate count". Will do. > On Sept. 14, 2019, 4:53 p.m., Madhan Neethiraj wrote: > > repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java > > Lines 131 (patched) > > <https://reviews.apache.org/r/71282/diff/3/?file=2161902#file2161902line131> > > > > - given "getResultCount()" can only return approximate count, that too > > not in all cases, consider renaming this to "getApproximateResultCount()" > > - and the implementations should return "-1" when approximate count is > > not available That's not true in this case although maybe a bit semantic. This call does get the total result count from the search backend scrubbing happens only after the API call. I suggest leaving it as it might be used in different circumstances in the future (hypothetical: if it would become possible to do scrubbing in the search engine for example). - Bolke --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71282/#review217737 --- On Sept. 14, 2019, 12:46 p.m., Bolke de Bruin wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71282/ > --- > > (Updated Sept. 14, 2019, 12:46 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, > Madhan Neethiraj, and Nixon Rodrigues. > > > Bugs: https://issues.apache.org/jira/browse/ATLAS-3367 > > https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3367 > > > Repository: atlas > > > Description > --- > > Both ElasticSearch and SOLR can expose the total count > of results without returning all results. This is useful > to give the user or client an idea how many results there are. > > This patch ensures the total is returned if available. This total > is an approximate as scrubbing of the results still needs to happen. > Therefore, one should not only rely on this information to provide > ,for example, pagination. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java > d274cc07e > > repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java > 479ddfd89 > > repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java > c67e34772 > > repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java > f7d8f08c7 > > repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java > 1dd1afaa3 > > repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java > 0ffd61c07 > repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java > a7ccaebc8 > > repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java > c253d544b > > > Diff: https://reviews.apache.org/r/71282/diff/3/ > > > Testing > --- > > > Thanks, > > Bolke de Bruin > >
Re: Review Request 71282: Expose total count of index searches
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71282/ --- (Updated Sept. 14, 2019, 6:03 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, Madhan Neethiraj, and Nixon Rodrigues. Changes --- Use -1 to initialize Bugs: https://issues.apache.org/jira/browse/ATLAS-3367 https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3367 Repository: atlas Description --- Both ElasticSearch and SOLR can expose the total count of results without returning all results. This is useful to give the user or client an idea how many results there are. This patch ensures the total is returned if available. This total is an approximate as scrubbing of the results still needs to happen. Therefore, one should not only rely on this information to provide ,for example, pagination. Diffs (updated) - intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java d274cc07e repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 479ddfd89 repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java f667aa399 repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java f7d8f08c7 repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java 1dd1afaa3 repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java 0ffd61c07 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java f7847edff repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java c253d544b Diff: https://reviews.apache.org/r/71282/diff/4/ Changes: https://reviews.apache.org/r/71282/diff/3-4/ Testing --- Thanks, Bolke de Bruin
Re: Review Request 71431: Add indexed order by (sort) to basic search
> On Sept. 14, 2019, 7:16 p.m., Madhan Neethiraj wrote: > > repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java > > Line 50 (original), 57 (patched) > > <https://reviews.apache.org/r/71431/diff/2/?file=2165082#file2165082line57> > > > > ClassificationSearchProcessor needs to be updated as well to support > > sortBy. Please review and update. Good catch. Im planning to add the same logic as in EntitySearchProcessor and add it to "tagGraphQueryWithAttributes" and "entityGraphQueryTraitNames". Does that make sense? > On Sept. 14, 2019, 7:16 p.m., Madhan Neethiraj wrote: > > repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java > > Lines 205 (patched) > > <https://reviews.apache.org/r/71431/diff/2/?file=2165082#file2165082line205> > > > > graphQuery requires 'qualifiedName' of the attribute (i.e. Asset.name, > > Asset.owner) - refer to SearchProcessor.toGraphFilterQuery(). Is 'sortBy' > > here qualifiedName of the attribute? Good point. It seems (but please correct me if I am wrong) given the tests that it is not required. However, I assume I will need to update SearchParameters to validate whether the attribute exists? - Bolke --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71431/#review217738 --- On Sept. 14, 2019, 12:45 p.m., Bolke de Bruin wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71431/ > --- > > (Updated Sept. 14, 2019, 12:45 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, > Madhan Neethiraj, and Nixon Rodrigues. > > > Bugs: https://issues.apache.org/jira/browse/ATLAS-3399 > > https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3399 > > > Repository: atlas > > > Description > --- > > Add indexed order by to basic search. > > > Diffs > - > > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java > f3722b827 > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusIndexQuery.java > 7c258b7a3 > graphdb/janus/src/main/java/org/janusgraph/diskstorage/solr/Solr6Index.java > e457866af > intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java > aac6b5aa3 > > repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java > f7d8f08c7 > webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java e6b338b84 > webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java > 06931b38c > webapp/src/test/resources/json/search-parameters/entity-filters.json > f4b2efe63 > > > Diff: https://reviews.apache.org/r/71431/diff/2/ > > > Testing > --- > > - production and tests added > > > Thanks, > > Bolke de Bruin > >
Re: Review Request 71282: Expose total count of index searches
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71282/ --- (Updated Sept. 15, 2019, 8:10 a.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, Madhan Neethiraj, and Nixon Rodrigues. Changes --- Addressed comments. Bugs: https://issues.apache.org/jira/browse/ATLAS-3367 https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3367 Repository: atlas Description --- Both ElasticSearch and SOLR can expose the total count of results without returning all results. This is useful to give the user or client an idea how many results there are. This patch ensures the total is returned if available. This total is an approximate as scrubbing of the results still needs to happen. Therefore, one should not only rely on this information to provide ,for example, pagination. Diffs (updated) - intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java d274cc07e repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 479ddfd89 repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java f667aa399 repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java f7d8f08c7 repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java 1dd1afaa3 repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java 0ffd61c07 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java f7847edff repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java c253d544b Diff: https://reviews.apache.org/r/71282/diff/5/ Changes: https://reviews.apache.org/r/71282/diff/4-5/ Testing --- Thanks, Bolke de Bruin
Fwd: Review Request 71431: Add indexed order by (sort) to basic search
Friendly ping.
Sent from my iPhone
Begin forwarded message:
> From: Bolke de Bruin
> Date: 16 September 2019 at 21:11:17 CEST
> To: Aadarsh Jajodia , Ashutosh Mestry
> , Sridhar K , Le Ma
> , Nixon Rodrigues
> , Madhan Neethiraj
> Cc: Sarath Subramanian , atlas
> , Bolke de Bruin
> Subject: Re: Review Request 71431: Add indexed order by (sort) to basic search
> Reply-To: Bolke de Bruin
>
>
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71431/
>
> On September 16th, 2019, 7:05 p.m. UTC, Madhan Neethiraj wrote:
>
> repository/src/main/java/org/apache/atlas/discovery/SearchContext.java (Diff
> revision 4)
> 272
> FilterCriteria filterCriteria = new FilterCriteria();
> Instead of creating FilterCriteria, consider using lines #263 - #265 here:
> if (StringUtils.isNotEmpty(attributeName) &&
> structType.getAttributeType(attributeName) == null) {
> throw new AtlasBaseException(AtlasErrorCode.UNKNOWN_ATTRIBUTE,
> attributeName, structType.getTypeName());
>
> Better yet, update 'else' block at #260 to call this method:
> } else {
> validateAttribute(structType, filterCriteria.getAttributeName();
> }
> Nice. I was struggling with that a bit. Fixed.
>
> - Bolke
>
>
> On September 16th, 2019, 7:10 p.m. UTC, Bolke de Bruin wrote:
>
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma,
> Madhan Neethiraj, and Nixon Rodrigues.
> By Bolke de Bruin.
> Updated Sept. 16, 2019, 7:10 p.m.
>
> Bugs: https://issues.apache.org/jira/browse/ATLAS-3399
> Repository: atlas
> Description
>
> Add indexed order by to basic search.
> Testing
>
> production and tests added
> Diffs
>
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
> (f3722b827)
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusIndexQuery.java
> (7c258b7a3)
> graphdb/janus/src/main/java/org/janusgraph/diskstorage/solr/Solr6Index.java
> (e457866af)
> intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java
> (aac6b5aa3)
> repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java
> (479ddfd89)
> repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java
> (f7d8f08c7)
> repository/src/main/java/org/apache/atlas/discovery/SearchContext.java
> (9bd0382eb)
> webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java (e6b338b84)
> webapp/src/test/java/org/apache/atlas/web/integration/BasicSearchIT.java
> (06931b38c)
> webapp/src/test/resources/json/search-parameters/entity-filters.json
> (f4b2efe63)
> webapp/src/test/resources/json/search-parameters/tag-filters.json (5e74328d6)
> View Diff
[jira] [Commented] (ATLAS-3305) Unable to scale atlas kafka consumers
[
https://issues.apache.org/jira/browse/ATLAS-3305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903328#comment-16903328
]
Bolke de Bruin commented on ATLAS-3305:
---
Why not use partitions with hashed ordering by qualified name? Or is this what
you meant [~arempter]?
Can we have consensus here [~sarath.ku...@gmail.com] cause this scalability
issue is stopping deployment at the moment for us
> Unable to scale atlas kafka consumers
> -
>
> Key: ATLAS-3305
> URL: https://issues.apache.org/jira/browse/ATLAS-3305
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core, atlas-intg
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Adam Rempter
>Priority: Major
> Labels: performance
>
> We wanted to scale kafka consumers for atlas, as we are getting many lineage
> messages and processing them just with one consumer is not enough.
>
> There is parameter atlas.notification.hook.numthreads to scale consumers in
> NotificationHookConsumer.
> But the method:
>
> notificationInterface.createConsumers(NotificationType.HOOK, numThreads)
>
> is always returning one element list, which effectively always starts one
> consumer
> List> consumers =
> Collections.singletonList(kafkaConsumer);
>
> Log incorrectly says that nuber of consumers has been created:
> LOG.info("<== KafkaNotification.createConsumers(notificationType={},
> numConsumers={}, autoCommitEnabled={})", notificationType, numConsumers,
> autoCommitEnabled)
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
Re: Atlas import taking huge amount of time
Will this solve / improve the issue with extremely slow Kafka consumption as
well?
Cheers
Bolke
Sent from my iPhone
> On 25 Jul 2019, at 09:54, Nallapati, Sreenivasulu
> wrote:
>
> Sure Ashutosh,
>
> Please let me know once it is done.
>
>
>
> ---
> Regards,
> Sreeni
>
> On 25/07/19, 4:52 AM, "Ashutosh Mestry" wrote:
>
>This email is from an external sender.
>
>
>There are few dependent patches. I will try to put out a patch on version
> 2.0. It will take me few days to get this. Please bear with me.
>
>~ ashutosh
>...
>No hurry, no pause. – Tim Ferriss, Life Hacker, Author
>
>
>On 7/24/19, 4:31 AM, "Nallapati, Sreenivasulu"
> wrote:
>
>Hi Ashutosh,
>
>Thanks for your reply.
>
>Currently we are using Atlas 2.0.0 and I am not able to apply this
> patch. It has lot of compilation errors.
>
>Do you have a patch for 2.0.0?
>
>
>---
>Regards,
>Sreeni
>
>On 23/07/19, 11:36 PM, "Ashutosh Mestry" wrote:
>
>This email is from an external sender.
>
>
>Hi
>
>Existing import processes 1 entity at a time. Thus time taken is
> linear. There is a JIRA that improves the situation. It is being tested right
> now.
>
>Please take a look at:
>JIRA: https://issues.apache.org/jira/browse/ATLAS-3320
>Review: https://reviews.apache.org/r/71025/ (latest patch is here)
>
>Best regards,
>
>~ ashutosh
>...
>No hurry, no pause. – Tim Ferriss, Life Hacker, Author
>
>
>On 7/23/19, 6:46 AM, "Nallapati, Sreenivasulu"
> wrote:
>
>Hello folks,
>
>We are trying to export and import the existing data to
> different atlas system.
>We have around 1 entities in the exported zip file. The
> export is taking around 2-3 mins.
>Total zip file size is 14 MB. The largest file in the zip is
> around 7 MB which has almost 1000 relationshipAttributes in it.
>When we try to import this, the import is running for more
> than 25 hours. Is this expected behaviour? Is there any way to speed up this
> process?
>
>Export command
>curl -igk -X POST -u admin:admin -H "Content-Type:
> application/json" -H "Cache-Control: no-cache" -d '{
>"itemsToExport": [
> { "typeName": "kafka_topic" }
>],
>"options": {
>"matchType": "forType"
>}
>}' "http:// localhost:21000/api/atlas/admin/export" >
> /tmp/kafka_topic.zip
>
>
>Import command
>curl -ikg -X POST -u admin:admin -H "Content-Type:
> multipart/form-data" -H "Cache-Control: no-cache" -F data=@ kafka_topic.zip
> "http://localhost:21000/api/atlas/admin/import;
>
>let us know if we are missing in the import process
>
>
>---
>Regards,
>Sreeni
>
>
>
>
>
>
