[GitHub] [atlas] aadarshjajodia commented on issue #64: [ATLAS-3314] Loosen the relationship between spark_table and spark_storagedesc
aadarshjajodia commented on issue #64: [ATLAS-3314] Loosen the relationship between spark_table and spark_storagedesc URL: https://github.com/apache/atlas/pull/64#issuecomment-507965067 +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (ATLAS-3314) Update relationshipCategory between spark_table and spark_storagedesc type
[ https://issues.apache.org/jira/browse/ATLAS-3314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian resolved ATLAS-3314. --- Resolution: Fixed > Update relationshipCategory between spark_table and spark_storagedesc type > -- > > Key: ATLAS-3314 > URL: https://issues.apache.org/jira/browse/ATLAS-3314 > Project: Atlas > Issue Type: Task > Components: atlas-intg >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Spark models define the relationship of "spark_table" and "spark_storagedesc" > as "COMPOSITION", which requires "spark_storagedesc" to refer "spark_table". > (Note that Atlas 1.x didn't require in same relationship.) > This enforces the sequence of creating entities (as Atlas doesn't support > referring other entity in same creation request) to "spark_table" -> > "spark_storagedesc", which doesn't seem to work in every usages, at least > this requires major changes on Spark Atlas Connector. > As a workaround, we can loosen the relationship via defining the relationship > as "ASSOCIATION", so that they can refer each other at any side. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3314) Update relationshipCategory between spark_table and spark_storagedesc type
[ https://issues.apache.org/jira/browse/ATLAS-3314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16877480#comment-16877480 ] ASF subversion and git services commented on ATLAS-3314: Commit 2c375b08a52ac8d1039abb4b612a41cb9d89b420 in atlas's branch refs/heads/master from Jungtaek Lim (HeartSaVioR) [ https://gitbox.apache.org/repos/asf?p=atlas.git;h=2c375b0 ] ATLAS-3314: Update relationshipCategory between spark_table and spark_storagedesc type Signed-off-by: Sarath Subramanian > Update relationshipCategory between spark_table and spark_storagedesc type > -- > > Key: ATLAS-3314 > URL: https://issues.apache.org/jira/browse/ATLAS-3314 > Project: Atlas > Issue Type: Task > Components: atlas-intg >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Spark models define the relationship of "spark_table" and "spark_storagedesc" > as "COMPOSITION", which requires "spark_storagedesc" to refer "spark_table". > (Note that Atlas 1.x didn't require in same relationship.) > This enforces the sequence of creating entities (as Atlas doesn't support > referring other entity in same creation request) to "spark_table" -> > "spark_storagedesc", which doesn't seem to work in every usages, at least > this requires major changes on Spark Atlas Connector. > As a workaround, we can loosen the relationship via defining the relationship > as "ASSOCIATION", so that they can refer each other at any side. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3314) Update relationshipCategory between spark_table and spark_storagedesc type
[ https://issues.apache.org/jira/browse/ATLAS-3314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3314: -- Summary: Update relationshipCategory between spark_table and spark_storagedesc type (was: Update the relationship between spark_table and spark_storagedesc) > Update relationshipCategory between spark_table and spark_storagedesc type > -- > > Key: ATLAS-3314 > URL: https://issues.apache.org/jira/browse/ATLAS-3314 > Project: Atlas > Issue Type: Task > Components: atlas-intg >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Assignee: Sarath Subramanian >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Spark models define the relationship of "spark_table" and "spark_storagedesc" > as "COMPOSITION", which requires "spark_storagedesc" to refer "spark_table". > (Note that Atlas 1.x didn't require in same relationship.) > This enforces the sequence of creating entities (as Atlas doesn't support > referring other entity in same creation request) to "spark_table" -> > "spark_storagedesc", which doesn't seem to work in every usages, at least > this requires major changes on Spark Atlas Connector. > As a workaround, we can loosen the relationship via defining the relationship > as "ASSOCIATION", so that they can refer each other at any side. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (ATLAS-3314) Update relationshipCategory between spark_table and spark_storagedesc type
[ https://issues.apache.org/jira/browse/ATLAS-3314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian reassigned ATLAS-3314: - Assignee: (was: Sarath Subramanian) > Update relationshipCategory between spark_table and spark_storagedesc type > -- > > Key: ATLAS-3314 > URL: https://issues.apache.org/jira/browse/ATLAS-3314 > Project: Atlas > Issue Type: Task > Components: atlas-intg >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Spark models define the relationship of "spark_table" and "spark_storagedesc" > as "COMPOSITION", which requires "spark_storagedesc" to refer "spark_table". > (Note that Atlas 1.x didn't require in same relationship.) > This enforces the sequence of creating entities (as Atlas doesn't support > referring other entity in same creation request) to "spark_table" -> > "spark_storagedesc", which doesn't seem to work in every usages, at least > this requires major changes on Spark Atlas Connector. > As a workaround, we can loosen the relationship via defining the relationship > as "ASSOCIATION", so that they can refer each other at any side. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3314) Update the relationship between spark_table and spark_storagedesc
[ https://issues.apache.org/jira/browse/ATLAS-3314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3314: -- Summary: Update the relationship between spark_table and spark_storagedesc (was: Loosen the relationship between spark_table and spark_storagedesc) > Update the relationship between spark_table and spark_storagedesc > - > > Key: ATLAS-3314 > URL: https://issues.apache.org/jira/browse/ATLAS-3314 > Project: Atlas > Issue Type: Task > Components: atlas-intg >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Assignee: Sarath Subramanian >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Spark models define the relationship of "spark_table" and "spark_storagedesc" > as "COMPOSITION", which requires "spark_storagedesc" to refer "spark_table". > (Note that Atlas 1.x didn't require in same relationship.) > This enforces the sequence of creating entities (as Atlas doesn't support > referring other entity in same creation request) to "spark_table" -> > "spark_storagedesc", which doesn't seem to work in every usages, at least > this requires major changes on Spark Atlas Connector. > As a workaround, we can loosen the relationship via defining the relationship > as "ASSOCIATION", so that they can refer each other at any side. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (ATLAS-3314) Loosen the relationship between spark_table and spark_storagedesc
[ https://issues.apache.org/jira/browse/ATLAS-3314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian reassigned ATLAS-3314: - Assignee: Sarath Subramanian > Loosen the relationship between spark_table and spark_storagedesc > - > > Key: ATLAS-3314 > URL: https://issues.apache.org/jira/browse/ATLAS-3314 > Project: Atlas > Issue Type: Task > Components: atlas-intg >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Assignee: Sarath Subramanian >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Spark models define the relationship of "spark_table" and "spark_storagedesc" > as "COMPOSITION", which requires "spark_storagedesc" to refer "spark_table". > (Note that Atlas 1.x didn't require in same relationship.) > This enforces the sequence of creating entities (as Atlas doesn't support > referring other entity in same creation request) to "spark_table" -> > "spark_storagedesc", which doesn't seem to work in every usages, at least > this requires major changes on Spark Atlas Connector. > As a workaround, we can loosen the relationship via defining the relationship > as "ASSOCIATION", so that they can refer each other at any side. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [atlas] sarathsubramanian commented on issue #64: [ATLAS-3314] Loosen the relationship between spark_table and spark_storagedesc
sarathsubramanian commented on issue #64: [ATLAS-3314] Loosen the relationship between spark_table and spark_storagedesc URL: https://github.com/apache/atlas/pull/64#issuecomment-507935624 +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [atlas] HeartSaVioR commented on issue #64: [ATLAS-3314] Loosen the relationship between spark_table and spark_storagedesc
HeartSaVioR commented on issue #64: [ATLAS-3314] Loosen the relationship between spark_table and spark_storagedesc URL: https://github.com/apache/atlas/pull/64#issuecomment-507935368 cc. @aadarshjajodia @sarathsubramanian This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
Re: Review Request 70913: ATLAS-3286: Populated dynamic attribute flags for AtlasAttributes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70913/ --- (Updated July 3, 2019, 12:13 a.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, Madhan Neethiraj, and Sarath Subramanian. Changes --- Added 1 interface and 3 subclasses to keep track of the type of each part of the template. The classes can either be constant, an attribute, or one that's dependent on it's relationship attributes. I changed also the logic in AtlasEntityType because saving the split parts of the template simplifies the code in AtlasEntityType. Repository: atlas Description --- ATLAS-3286: Populated dynamic attribute flags for AtlasAttributes Diffs (updated) - addons/models/1000-Hadoop/1030-hive_model.json 8901aa4aa86fb2802a9e9b1e65c2ff8aad8855ad intg/src/main/java/org/apache/atlas/model/AttributeToken.java PRE-CREATION intg/src/main/java/org/apache/atlas/model/ConstantToken.java PRE-CREATION intg/src/main/java/org/apache/atlas/model/DependentToken.java PRE-CREATION intg/src/main/java/org/apache/atlas/model/TemplateToken.java PRE-CREATION intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java d9ae9e36773146fad652a1e28fc8822bae5c8557 intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 254eee4c9d18ab87a8862ce479fc2274227c7c29 intg/src/test/java/org/apache/atlas/type/TestAtlasEntityType.java 3c53c02b48747515217c9327c98209a48ee84237 Diff: https://reviews.apache.org/r/70913/diff/8/ Changes: https://reviews.apache.org/r/70913/diff/7-8/ Testing --- Added unit test Thanks, Merryle Wang
[GitHub] [atlas] HeartSaVioR opened a new pull request #64: [ATLAS-3314] Loosen the relationship between spark_table and spark_storagedesc
HeartSaVioR opened a new pull request #64: [ATLAS-3314] Loosen the relationship between spark_table and spark_storagedesc URL: https://github.com/apache/atlas/pull/64 This patch loosens the relationship between `spark_table` and `spark_storagedesc` to ensure either side of setting relationship works instead of requiring entity of `spark_storagedesc` to refer entity of `spark_table`. Please refer https://issues.apache.org/jira/browse/ATLAS-3314 for more details on rationalization. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (ATLAS-3314) Loosen the relationship between spark_table and spark_storagedesc
Jungtaek Lim created ATLAS-3314: --- Summary: Loosen the relationship between spark_table and spark_storagedesc Key: ATLAS-3314 URL: https://issues.apache.org/jira/browse/ATLAS-3314 Project: Atlas Issue Type: Task Components: atlas-intg Affects Versions: 2.0.0 Reporter: Jungtaek Lim Spark models define the relationship of "spark_table" and "spark_storagedesc" as "COMPOSITION", which requires "spark_storagedesc" to refer "spark_table". (Note that Atlas 1.x didn't require in same relationship.) This enforces the sequence of creating entities (as Atlas doesn't support referring other entity in same creation request) to "spark_table" -> "spark_storagedesc", which doesn't seem to work in every usages, at least this requires major changes on Spark Atlas Connector. As a workaround, we can loosen the relationship via defining the relationship as "ASSOCIATION", so that they can refer each other at any side. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 70913: ATLAS-3286: Populated dynamic attribute flags for AtlasAttributes
> On July 2, 2019, 3:50 p.m., Ashutosh Mestry wrote:
> > This an edge case: When temporary tables are created, their qualifiedName
> > gets suffixed with GUID, how will this scheme handle that?
>
> Aadarsh Jajodia wrote:
> Hi Ashutosh, Thats a good point you mentioned. Right now we don't have a
> way in the templates to refer to a system attribute(we might need this as you
> mentioned for a temp table). We can take that up later.
See this: AtlasHiveHookContext
public String getQualifiedName(Table table) {
String tableName = table.getTableName();
if (table.isTemporary()) {
if (SessionState.get() != null && SessionState.get().getSessionId()
!= null) {
tableName = tableName + TEMP_TABLE_PREFIX +
SessionState.get().getSessionId();
} else {
tableName = tableName + TEMP_TABLE_PREFIX +
RandomStringUtils.random(10);
}
}
return (table.getDbName() + QNAME_SEP_ENTITY_NAME + tableName +
QNAME_SEP_CLUSTER_NAME).toLowerCase() + getClusterName();
}
- Ashutosh
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70913/#review216308
---
On July 1, 2019, 6:10 p.m., Merryle Wang wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70913/
> ---
>
> (Updated July 1, 2019, 6:10 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma,
> Madhan Neethiraj, and Sarath Subramanian.
>
>
> Repository: atlas
>
>
> Description
> ---
>
> ATLAS-3286: Populated dynamic attribute flags for AtlasAttributes
>
>
> Diffs
> -
>
> addons/models/1000-Hadoop/1030-hive_model.json
> 8901aa4aa86fb2802a9e9b1e65c2ff8aad8855ad
> intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
> d9ae9e36773146fad652a1e28fc8822bae5c8557
> intg/src/main/java/org/apache/atlas/type/AtlasStructType.java
> 254eee4c9d18ab87a8862ce479fc2274227c7c29
> intg/src/test/java/org/apache/atlas/type/TestAtlasEntityType.java
> 3c53c02b48747515217c9327c98209a48ee84237
>
>
> Diff: https://reviews.apache.org/r/70913/diff/7/
>
>
> Testing
> ---
>
> Added unit test
>
>
> Thanks,
>
> Merryle Wang
>
>
Re: Review Request 70913: ATLAS-3286: Populated dynamic attribute flags for AtlasAttributes
> On July 2, 2019, 3:50 p.m., Ashutosh Mestry wrote: > > This an edge case: When temporary tables are created, their qualifiedName > > gets suffixed with GUID, how will this scheme handle that? Hi Ashutosh, Thats a good point you mentioned. Right now we don't have a way in the templates to refer to a system attribute(we might need this as you mentioned for a temp table). We can take that up later. - Aadarsh --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70913/#review216308 --- On July 1, 2019, 6:10 p.m., Merryle Wang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70913/ > --- > > (Updated July 1, 2019, 6:10 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, > Madhan Neethiraj, and Sarath Subramanian. > > > Repository: atlas > > > Description > --- > > ATLAS-3286: Populated dynamic attribute flags for AtlasAttributes > > > Diffs > - > > addons/models/1000-Hadoop/1030-hive_model.json > 8901aa4aa86fb2802a9e9b1e65c2ff8aad8855ad > intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java > d9ae9e36773146fad652a1e28fc8822bae5c8557 > intg/src/main/java/org/apache/atlas/type/AtlasStructType.java > 254eee4c9d18ab87a8862ce479fc2274227c7c29 > intg/src/test/java/org/apache/atlas/type/TestAtlasEntityType.java > 3c53c02b48747515217c9327c98209a48ee84237 > > > Diff: https://reviews.apache.org/r/70913/diff/7/ > > > Testing > --- > > Added unit test > > > Thanks, > > Merryle Wang > >
[jira] [Created] (ATLAS-3313) Enhance Atlas server to process notifications from multiple Kafka topics
Madhan Neethiraj created ATLAS-3313: --- Summary: Enhance Atlas server to process notifications from multiple Kafka topics Key: ATLAS-3313 URL: https://issues.apache.org/jira/browse/ATLAS-3313 Project: Atlas Issue Type: Improvement Components: atlas-core Reporter: Madhan Neethiraj Currently Atlas server processes notifications from Kafka topic named ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). The name of the Kafka topic to which an Atlas Hook publishes notifications can be configured via configuration atlas.notification.hook.topic.name. However, since Atlas server processes notifications from a single topic, all hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic. To enable different hooks to publish to different Kafka topics, Atlas server should support processing notifications from multiple Kafka topics. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 70913: ATLAS-3286: Populated dynamic attribute flags for AtlasAttributes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70913/#review216308 --- This an edge case: When temporary tables are created, their qualifiedName gets suffixed with GUID, how will this scheme handle that? - Ashutosh Mestry On July 1, 2019, 6:10 p.m., Merryle Wang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70913/ > --- > > (Updated July 1, 2019, 6:10 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, > Madhan Neethiraj, and Sarath Subramanian. > > > Repository: atlas > > > Description > --- > > ATLAS-3286: Populated dynamic attribute flags for AtlasAttributes > > > Diffs > - > > addons/models/1000-Hadoop/1030-hive_model.json > 8901aa4aa86fb2802a9e9b1e65c2ff8aad8855ad > intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java > d9ae9e36773146fad652a1e28fc8822bae5c8557 > intg/src/main/java/org/apache/atlas/type/AtlasStructType.java > 254eee4c9d18ab87a8862ce479fc2274227c7c29 > intg/src/test/java/org/apache/atlas/type/TestAtlasEntityType.java > 3c53c02b48747515217c9327c98209a48ee84237 > > > Diff: https://reviews.apache.org/r/70913/diff/7/ > > > Testing > --- > > Added unit test > > > Thanks, > > Merryle Wang > >
