Re: Review Request 70600: ATLAS-3214 Create Spark Models in Atlas to Support Spark Lineage
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70600/#review215344 --- Sorry for the last deal, we're discussing to remove attribute `durationMs` as we found SAC has no way to fill the right value. (The value is query-based whereas `spark_process` model is application-based.) Could we hold off this to early next week as we will update the PR here https://github.com/hortonworks-spark/spark-atlas-connector/pull/249 and merge shortly? - Jungtaek Lim On 5 17, 2019, 5:47 오후, Aadarsh Jajodia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70600/ > --- > > (Updated 5 17, 2019, 5:47 오후) > > > Review request for atlas, Ankur Gupta, Arun Mahadevan, Ashutosh Mestry, > Jungtaek Lim, Sridhar K, Le Ma, Madhan Neethiraj, and Sarath Subramanian. > > > Repository: atlas > > > Description > --- > > This is an initial mode file for the SPARK models. Please refer to the file > added here. This file is a GET call on typeDefs on a cluster which had SAC > enabled and SAC has created the models. The SAC version used was > https://github.com/hortonworks/spark-atlas-connector/tree/SAC-CDP-1.0-dev > > > Diffs > - > > addons/models/1000-Hadoop/1100-spark_model.json PRE-CREATION > > > Diff: https://reviews.apache.org/r/70600/diff/4/ > > > Testing > --- > > > File Attachments > > > SAC Models > > https://reviews.apache.org/media/uploaded/files/2019/05/07/e31827b2-04c5-4fab-b5b7-4271398831e4__spark_sac_api_model_definition.txt > > > Thanks, > > Aadarsh Jajodia > >
[jira] [Created] (ATLAS-3216) Minor typos in What's New for 2.0 markdown
Carol Drummond created ATLAS-3216: - Summary: Minor typos in What's New for 2.0 markdown Key: ATLAS-3216 URL: https://issues.apache.org/jira/browse/ATLAS-3216 Project: Atlas Issue Type: Bug Affects Versions: 2.0.0 Reporter: Carol Drummond Assignee: Carol Drummond Fix For: 2.0.0 Minor typos in the release notes for 2.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Review Request 70672: [ATLAS-3153] Add keycloak authentication
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70672/ --- Review request for atlas, Ashutosh Mestry and Madhan Neethiraj. Bugs: https://issues.apache.org/jira/browse/ATLAS-3153 https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/ATLAS-3153 Repository: atlas Description --- Keycloak is an open source Identity and Access Management solution aimed at modern applications and services. It makes it easy to secure applications and services with little to no code. This enables Atlas to use OpenID Connect (OAUTH2) and allows integration with more services. Diffs - docs/src/site/twiki/Atlas-Authentication.twiki ddaa7fec2 pom.xml 7ad24ea5b webapp/pom.xml 994ee86f0 webapp/src/main/java/org/apache/atlas/web/security/AtlasAuthenticationProvider.java fb21d7563 webapp/src/main/java/org/apache/atlas/web/security/AtlasKeycloakAuthenticationProvider.java PRE-CREATION webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java bf6b85bea Diff: https://reviews.apache.org/r/70672/diff/1/ Testing --- Thanks, Bolke de Bruin
[GitHub] [atlas] bolkedebruin commented on issue #53: [ATLAS-3153] Add keycloak authentication
bolkedebruin commented on issue #53: [ATLAS-3153] Add keycloak authentication URL: https://github.com/apache/atlas/pull/53#issuecomment-493608872 cc @mneethiraj @ashutoshm This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [atlas] bolkedebruin opened a new pull request #53: [ATLAS-3153] Add keycloak authentication
bolkedebruin opened a new pull request #53: [ATLAS-3153] Add keycloak authentication URL: https://github.com/apache/atlas/pull/53 Keycloak is an open source Identity and Access Management solution aimed at modern applications and services. It makes it easy to secure applications and services with little to no code. This enabled Atlas to use OpenID Connect (OAUTH2) and allows integration with more services. This is a working implementation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (ATLAS-3184) Add support of ineage integration for more Impala commands
[ https://issues.apache.org/jira/browse/ATLAS-3184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842423#comment-16842423 ] Na Li commented on ATLAS-3184: -- pre-commit test for patch 2 is at https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1109/console > Add support of ineage integration for more Impala commands > -- > > Key: ATLAS-3184 > URL: https://issues.apache.org/jira/browse/ATLAS-3184 > Project: Atlas > Issue Type: New Feature > Components: atlas-core >Affects Versions: 1.1.0 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: ATLAS-3184.001.patch, ATLAS-3184.002.patch > > > [ATLAS-3183|https://issues.apache.org/jira/browse/ATLAS-3183] creates the > Impala integration with Atlas. It only supports "create view" command. > We need to add support for all other Impala commands that generate lineage > records. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3184) Add support of ineage integration for more Impala commands
[ https://issues.apache.org/jira/browse/ATLAS-3184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated ATLAS-3184: - Attachment: ATLAS-3184.002.patch > Add support of ineage integration for more Impala commands > -- > > Key: ATLAS-3184 > URL: https://issues.apache.org/jira/browse/ATLAS-3184 > Project: Atlas > Issue Type: New Feature > Components: atlas-core >Affects Versions: 1.1.0 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: ATLAS-3184.001.patch, ATLAS-3184.002.patch > > > [ATLAS-3183|https://issues.apache.org/jira/browse/ATLAS-3183] creates the > Impala integration with Atlas. It only supports "create view" command. > We need to add support for all other Impala commands that generate lineage > records. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3181) The URL produced by the swagger interactive API is missing "/api/atlas/"
[
https://issues.apache.org/jira/browse/ATLAS-3181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842407#comment-16842407
]
ASF subversion and git services commented on ATLAS-3181:
Commit ff7415d8e45e5955764e522c28d9a914dfcc35f3 in atlas's branch
refs/heads/master from CDrummond
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=ff7415d ]
ATLAS-3181: fixed swagger interactive example URLs
Signed-off-by: Sarath Subramanian
> The URL produced by the swagger interactive API is missing "/api/atlas/"
>
>
> Key: ATLAS-3181
> URL: https://issues.apache.org/jira/browse/ATLAS-3181
> Project: Atlas
> Issue Type: Bug
> Components: atlas-webui
>Affects Versions: 1.0.0, 1.1.0, 2.0.0
>Reporter: Carol Drummond
>Priority: Minor
> Attachments: ATLAS-3181.patch
>
>
> Swagger isn't building the API url correctly: it's missing the "api/atlas"
> parts of the URL:
>
> Here's what it builds:
> [https://localhost:21000/v2/entity/guid/4b2aa848-fb3a-401a-9e05-29a08ab2c846/classifications]
> {quote}curl -X GET --header 'Accept: application/json'
> '[https://localhost:21000/v2/entity/guid/4b2aa848-fb3a-401a-9e05-29a08ab2c846/classifications]'{quote}
>
> Here's what works (added "/api/atlas"):
>
> [http://localhost:21000|http://localhost:21000/]*/api/atlas/*v2/entity/guid/4b2aa848-fb3a-401a-9e05-29a08ab2c846/classifications
>
> {quote}curl -X GET
> '[http://localhost:21000/api/atlas/v2/entity/guid/4b2aa848-fb3a-401a-9e05-29a08ab2c846/classifications]'
> -u user:password{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (ATLAS-3181) The URL produced by the swagger interactive API is missing "/api/atlas/"
[
https://issues.apache.org/jira/browse/ATLAS-3181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian reassigned ATLAS-3181:
-
Assignee: Carol Drummond
> The URL produced by the swagger interactive API is missing "/api/atlas/"
>
>
> Key: ATLAS-3181
> URL: https://issues.apache.org/jira/browse/ATLAS-3181
> Project: Atlas
> Issue Type: Bug
> Components: atlas-webui
>Affects Versions: 1.0.0, 1.1.0, 2.0.0
>Reporter: Carol Drummond
>Assignee: Carol Drummond
>Priority: Minor
> Attachments: ATLAS-3181.patch
>
>
> Swagger isn't building the API url correctly: it's missing the "api/atlas"
> parts of the URL:
>
> Here's what it builds:
> [https://localhost:21000/v2/entity/guid/4b2aa848-fb3a-401a-9e05-29a08ab2c846/classifications]
> {quote}curl -X GET --header 'Accept: application/json'
> '[https://localhost:21000/v2/entity/guid/4b2aa848-fb3a-401a-9e05-29a08ab2c846/classifications]'{quote}
>
> Here's what works (added "/api/atlas"):
>
> [http://localhost:21000|http://localhost:21000/]*/api/atlas/*v2/entity/guid/4b2aa848-fb3a-401a-9e05-29a08ab2c846/classifications
>
> {quote}curl -X GET
> '[http://localhost:21000/api/atlas/v2/entity/guid/4b2aa848-fb3a-401a-9e05-29a08ab2c846/classifications]'
> -u user:password{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: Review Request 70575: ATLAS-3181 Corrected missing components in the example URLs in Swagger interactive API doc
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70575/#review215333 --- Ship it! Ship It! - Sarath Subramanian On May 1, 2019, 1:08 p.m., Carol Drummond wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70575/ > --- > > (Updated May 1, 2019, 1:08 p.m.) > > > Review request for atlas and Sarath Subramanian. > > > Bugs: ATLAS-3181 > https://issues.apache.org/jira/browse/ATLAS-3181 > > > Repository: atlas > > > Description > --- > > This patch inserts a prefix to the URL produced by enunciate for the Swagger > interactive API examples. Without the prefix, the example URLs do not work. > > > Diffs > - > > build-tools/src/main/resources/enunciate.xml e01d69680 > > > Diff: https://reviews.apache.org/r/70575/diff/1/ > > > Testing > --- > > Validated that my prefix appears in the right place in the Swagger UI for a > selection of APIs. > Validated that the interactivity works as expected (no regressions). > Validated that the examples now produce URLs that work as expected. > > > Thanks, > > Carol Drummond > >
Re: Review Request 70600: ATLAS-3214 Create Spark Models in Atlas to Support Spark Lineage
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70600/#review215332 --- Ship it! Ship It! - Sarath Subramanian On May 17, 2019, 10:47 a.m., Aadarsh Jajodia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70600/ > --- > > (Updated May 17, 2019, 10:47 a.m.) > > > Review request for atlas, Ankur Gupta, Arun Mahadevan, Ashutosh Mestry, > Jungtaek Lim, Sridhar K, Le Ma, Madhan Neethiraj, and Sarath Subramanian. > > > Repository: atlas > > > Description > --- > > This is an initial mode file for the SPARK models. Please refer to the file > added here. This file is a GET call on typeDefs on a cluster which had SAC > enabled and SAC has created the models. The SAC version used was > https://github.com/hortonworks/spark-atlas-connector/tree/SAC-CDP-1.0-dev > > > Diffs > - > > addons/models/1000-Hadoop/1100-spark_model.json PRE-CREATION > > > Diff: https://reviews.apache.org/r/70600/diff/4/ > > > Testing > --- > > > File Attachments > > > SAC Models > > https://reviews.apache.org/media/uploaded/files/2019/05/07/e31827b2-04c5-4fab-b5b7-4271398831e4__spark_sac_api_model_definition.txt > > > Thanks, > > Aadarsh Jajodia > >
Re: Review Request 70600: ATLAS-3214 Create Spark Models in Atlas to Support Spark Lineage
> On May 17, 2019, 4:28 a.m., Jungtaek Lim wrote: > > > > Jungtaek Lim wrote: > > Removing isLegacy Attributes since spark is not released yet. All > fields that are referenced will show up in the relations tab. > > I think we've talked yesterday and decided to keep the flag on since we > want these attributes to also show up in properties tab. For example, see > `spark_ml_pipeline`. If we can't show up these attributes in properties tab, > end users have to do couple of clicks to see these value in relationship tab, > and properties tab is no longer working as `overview`. Hi Jungtaek, Traditionally Atlas used to follow that. But with recent changes to Atlas, all relationship attributes will show up on the relationship tab only. For any new models that is the convention. Thank you - Aadarsh --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70600/#review215314 --- On May 16, 2019, 11:54 p.m., Aadarsh Jajodia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70600/ > --- > > (Updated May 16, 2019, 11:54 p.m.) > > > Review request for atlas, Ankur Gupta, Arun Mahadevan, Ashutosh Mestry, > Jungtaek Lim, Sridhar K, Le Ma, Madhan Neethiraj, and Sarath Subramanian. > > > Repository: atlas > > > Description > --- > > This is an initial mode file for the SPARK models. Please refer to the file > added here. This file is a GET call on typeDefs on a cluster which had SAC > enabled and SAC has created the models. The SAC version used was > https://github.com/hortonworks/spark-atlas-connector/tree/SAC-CDP-1.0-dev > > > Diffs > - > > addons/models/1000-Hadoop/1100-spark_model.json PRE-CREATION > > > Diff: https://reviews.apache.org/r/70600/diff/3/ > > > Testing > --- > > > File Attachments > > > SAC Models > > https://reviews.apache.org/media/uploaded/files/2019/05/07/e31827b2-04c5-4fab-b5b7-4271398831e4__spark_sac_api_model_definition.txt > > > Thanks, > > Aadarsh Jajodia > >
[jira] [Assigned] (ATLAS-3080) Impala should send notification for DDL operation to Atlas through Kafka
[ https://issues.apache.org/jira/browse/ATLAS-3080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reassigned ATLAS-3080: Assignee: Na Li > Impala should send notification for DDL operation to Atlas through Kafka > > > Key: ATLAS-3080 > URL: https://issues.apache.org/jira/browse/ATLAS-3080 > Project: Atlas > Issue Type: Task > Components: atlas-intg >Affects Versions: 1.1.0 >Reporter: Na Li >Assignee: Na Li >Priority: Blocker > > if user needs Tag based security policy for tables in Impala, we need the > table metadata in Atlas, like assigning a PII tag for an Impala table in > atlas and enforce security policy in Ranger for the tag ‘PII’ on who can > access/modify it. > Atlas does not allow user to configure tag on a table if Atlas has not > received notification about that table creation. Right now, Impala does not > send notifications for DDL operation. Therefore, we need to add Atlas code to > integrate with Impala. > Besides this use case, sending notifications to Atlas allow other > functionalities to consume Impala DDL operation activities. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
