[jira] [Updated] (ATLAS-3591) Improve user-defined properties search
[
https://issues.apache.org/jira/browse/ATLAS-3591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-3591:
--
Attachment: ATLAS-3591.001.patch
> Improve user-defined properties search
> --
>
> Key: ATLAS-3591
> URL: https://issues.apache.org/jira/browse/ATLAS-3591
> Project: Atlas
> Issue Type: Task
> Components: atlas-core
>Affects Versions: 2.0.0
> Reporter: Sarath Subramanian
> Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: ATLAS-3591.001.patch
>
>
> User-defined properties search is enabled in system attribute search. This
> Jira is improve the usability of the user-defined properties search.
>
> For. e.g
>
> {code:java}
> "entityFilters": { "condition": "AND", "criterion": [ { "attributeName":
> "__customAttributes", "operator": "contains", "attributeValue": "key2=value2"
> } ] }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-3591) Improve user-defined properties search
[
https://issues.apache.org/jira/browse/ATLAS-3591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-3591:
--
Description:
User-defined properties search is enabled in system attribute search. This Jira
is improve the usability of the user-defined properties search.
For. e.g
{code:java}
"entityFilters": { "condition": "AND", "criterion": [ { "attributeName":
"__customAttributes", "operator": "contains", "attributeValue": "key2=value2" }
] }
{code}
was:
User-defined properties search is enabled in system attribute search. This Jira
is improve the usability of the user-defined properties search.
For. e.g
"entityFilters": {
"condition": "AND",
"criterion": [
{
"attributeName" : "*__customAttributes*",
"operator" : "*contains*",
"attributeValue": *"key2=value2"*
}
]
}
> Improve user-defined properties search
> --
>
> Key: ATLAS-3591
> URL: https://issues.apache.org/jira/browse/ATLAS-3591
> Project: Atlas
> Issue Type: Task
> Components: atlas-core
>Affects Versions: 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 2.1.0
>
>
> User-defined properties search is enabled in system attribute search. This
> Jira is improve the usability of the user-defined properties search.
>
> For. e.g
>
> {code:java}
> "entityFilters": { "condition": "AND", "criterion": [ { "attributeName":
> "__customAttributes", "operator": "contains", "attributeValue": "key2=value2"
> } ] }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ATLAS-3591) Improve user-defined properties search
Sarath Subramanian created ATLAS-3591:
-
Summary: Improve user-defined properties search
Key: ATLAS-3591
URL: https://issues.apache.org/jira/browse/ATLAS-3591
Project: Atlas
Issue Type: Task
Components: atlas-core
Affects Versions: 2.0.0
Reporter: Sarath Subramanian
Assignee: Sarath Subramanian
Fix For: 2.1.0
User-defined properties search is enabled in system attribute search. This Jira
is improve the usability of the user-defined properties search.
For. e.g
"entityFilters": {
"condition": "AND",
"criterion": [
{
"attributeName" : "*__customAttributes*",
"operator" : "*contains*",
"attributeValue": *"key2=value2"*
}
]
}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-3559) Asset Type Definition: Add attribute displayName & userDescription
[ https://issues.apache.org/jira/browse/ATLAS-3559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3559: -- Component/s: atlas-core > Asset Type Definition: Add attribute displayName & userDescription > -- > > Key: ATLAS-3559 > URL: https://issues.apache.org/jira/browse/ATLAS-3559 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Nikhil Bonte >Assignee: Nikhil Bonte >Priority: Major > Fix For: 2.1.0 > > Attachments: > ATLAS-3559-Add-attribute-displayName-in-Asset-typeDe.patch, > ATLAS-3559-Additional-attributes-to-Asset.patch > > > *Background* > Currently _name_ and _description_ get set as part of setting technical > metadata. Example, Hive hook sets these values as part of posting hook > messages. Once this data is available within Atlas, there is no way to > override these attributes. It would not make sense to change these values > since these are set as part of initial ingest. > *Solution* > Add _displayName_ and _userDescription_ to _Asset_ optional attributes. > Later, UI can be developed to let end user update these properties > interactively. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3559) Asset Type Definition: Add attribute displayName & userDescription
[ https://issues.apache.org/jira/browse/ATLAS-3559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3559: -- Affects Version/s: 2.0.0 > Asset Type Definition: Add attribute displayName & userDescription > -- > > Key: ATLAS-3559 > URL: https://issues.apache.org/jira/browse/ATLAS-3559 > Project: Atlas > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Nikhil Bonte >Assignee: Nikhil Bonte >Priority: Major > Attachments: > ATLAS-3559-Add-attribute-displayName-in-Asset-typeDe.patch, > ATLAS-3559-Additional-attributes-to-Asset.patch > > > *Background* > Currently _name_ and _description_ get set as part of setting technical > metadata. Example, Hive hook sets these values as part of posting hook > messages. Once this data is available within Atlas, there is no way to > override these attributes. It would not make sense to change these values > since these are set as part of initial ingest. > *Solution* > Add _displayName_ and _userDescription_ to _Asset_ optional attributes. > Later, UI can be developed to let end user update these properties > interactively. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3559) Asset Type Definition: Add attribute displayName & userDescription
[ https://issues.apache.org/jira/browse/ATLAS-3559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3559: -- Fix Version/s: 2.1.0 > Asset Type Definition: Add attribute displayName & userDescription > -- > > Key: ATLAS-3559 > URL: https://issues.apache.org/jira/browse/ATLAS-3559 > Project: Atlas > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Nikhil Bonte >Assignee: Nikhil Bonte >Priority: Major > Fix For: 2.1.0 > > Attachments: > ATLAS-3559-Add-attribute-displayName-in-Asset-typeDe.patch, > ATLAS-3559-Additional-attributes-to-Asset.patch > > > *Background* > Currently _name_ and _description_ get set as part of setting technical > metadata. Example, Hive hook sets these values as part of posting hook > messages. Once this data is available within Atlas, there is no way to > override these attributes. It would not make sense to change these values > since these are set as part of initial ingest. > *Solution* > Add _displayName_ and _userDescription_ to _Asset_ optional attributes. > Later, UI can be developed to let end user update these properties > interactively. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (ATLAS-3559) Asset Type Definition: Add attribute displayName & userDescription
[ https://issues.apache.org/jira/browse/ATLAS-3559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017597#comment-17017597 ] Sarath Subramanian edited comment on ATLAS-3559 at 1/17/20 1:02 AM: update description in patch file (005-base_model_add_display_attributes.json) to "Add displayName and *userDescription* to Asset" +1 was (Author: sarath.ku...@gmail.com): update description of patch to "Add displayName and *userDescription* to Asset" +1 > Asset Type Definition: Add attribute displayName & userDescription > -- > > Key: ATLAS-3559 > URL: https://issues.apache.org/jira/browse/ATLAS-3559 > Project: Atlas > Issue Type: Improvement >Reporter: Nikhil Bonte >Assignee: Nikhil Bonte >Priority: Major > Attachments: > ATLAS-3559-Add-attribute-displayName-in-Asset-typeDe.patch, > ATLAS-3559-Additional-attributes-to-Asset.patch > > > *Background* > Currently _name_ and _description_ get set as part of setting technical > metadata. Example, Hive hook sets these values as part of posting hook > messages. Once this data is available within Atlas, there is no way to > override these attributes. It would not make sense to change these values > since these are set as part of initial ingest. > *Solution* > Add _displayName_ and _userDescription_ to _Asset_ optional attributes. > Later, UI can be developed to let end user update these properties > interactively. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (ATLAS-3559) Asset Type Definition: Add attribute displayName & userDescription
[ https://issues.apache.org/jira/browse/ATLAS-3559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017597#comment-17017597 ] Sarath Subramanian edited comment on ATLAS-3559 at 1/17/20 12:58 AM: - update description of patch to "Add displayName and *userDescription* to Asset" +1 was (Author: sarath.ku...@gmail.com): update description of patch to "Add displayName and *userDescription* to Asset", + "action": "ADD_ATTRIBUTE"" +1 > Asset Type Definition: Add attribute displayName & userDescription > -- > > Key: ATLAS-3559 > URL: https://issues.apache.org/jira/browse/ATLAS-3559 > Project: Atlas > Issue Type: Improvement >Reporter: Nikhil Bonte >Assignee: Nikhil Bonte >Priority: Major > Attachments: > ATLAS-3559-Add-attribute-displayName-in-Asset-typeDe.patch, > ATLAS-3559-Additional-attributes-to-Asset.patch > > > *Background* > Currently _name_ and _description_ get set as part of setting technical > metadata. Example, Hive hook sets these values as part of posting hook > messages. Once this data is available within Atlas, there is no way to > override these attributes. It would not make sense to change these values > since these are set as part of initial ingest. > *Solution* > Add _displayName_ and _userDescription_ to _Asset_ optional attributes. > Later, UI can be developed to let end user update these properties > interactively. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3559) Asset Type Definition: Add attribute displayName & userDescription
[ https://issues.apache.org/jira/browse/ATLAS-3559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017597#comment-17017597 ] Sarath Subramanian commented on ATLAS-3559: --- update description of patch to "Add displayName and *userDescription* to Asset", + "action": "ADD_ATTRIBUTE"" +1 > Asset Type Definition: Add attribute displayName & userDescription > -- > > Key: ATLAS-3559 > URL: https://issues.apache.org/jira/browse/ATLAS-3559 > Project: Atlas > Issue Type: Improvement >Reporter: Nikhil Bonte >Assignee: Nikhil Bonte >Priority: Major > Attachments: > ATLAS-3559-Add-attribute-displayName-in-Asset-typeDe.patch, > ATLAS-3559-Additional-attributes-to-Asset.patch > > > *Background* > Currently _name_ and _description_ get set as part of setting technical > metadata. Example, Hive hook sets these values as part of posting hook > messages. Once this data is available within Atlas, there is no way to > override these attributes. It would not make sense to change these values > since these are set as part of initial ingest. > *Solution* > Add _displayName_ and _userDescription_ to _Asset_ optional attributes. > Later, UI can be developed to let end user update these properties > interactively. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Review Request 72021: ATLAS-3590: Hive hook should ignore capturing lineage for temporary tables in CTAS queries
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72021/ --- Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Mandar Ambawane, mayank jain, Nixon Rodrigues, and Sidharth Mishra. Bugs: ATLAS-3590 https://issues.apache.org/jira/browse/ATLAS-3590 Repository: atlas Description --- Atlas hive hook ignores capturing metadata for temporary tables in CREATE queries. Similarly it should ignore capturing lineage for temporary tables in CTAS queries Diffs - addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 5b1f61b63 addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java 425be5952 addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java 82df57650 Diff: https://reviews.apache.org/r/72021/diff/1/ Testing --- validated manually hive hook is not sending lineage for temporary table. Precommit: https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1606/console Thanks, Sarath Subramanian
[jira] [Updated] (ATLAS-3590) Hive hook should ignore capturing lineage for temporary tables in CTAS queries
[ https://issues.apache.org/jira/browse/ATLAS-3590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3590: -- Attachment: ATLAS-3590.001.patch > Hive hook should ignore capturing lineage for temporary tables in CTAS queries > -- > > Key: ATLAS-3590 > URL: https://issues.apache.org/jira/browse/ATLAS-3590 > Project: Atlas > Issue Type: Task > Components: atlas-core >Affects Versions: 2.0.0 > Reporter: Sarath Subramanian > Assignee: Sarath Subramanian >Priority: Major > Fix For: 2.1.0 > > Attachments: ATLAS-3590.001.patch > > > Atlas hive hook ignores capturing metadata for temporary tables in CREATE > queries. Similarly it should ignore capturing lineage for temporary tables in > CTAS queries -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3590) Hive hook should ignore capturing lineage for temporary tables in CTAS queries
Sarath Subramanian created ATLAS-3590: - Summary: Hive hook should ignore capturing lineage for temporary tables in CTAS queries Key: ATLAS-3590 URL: https://issues.apache.org/jira/browse/ATLAS-3590 Project: Atlas Issue Type: Task Components: atlas-core Affects Versions: 2.0.0 Reporter: Sarath Subramanian Assignee: Sarath Subramanian Fix For: 2.1.0 Atlas hive hook ignores capturing metadata for temporary tables in CREATE queries. Similarly it should ignore capturing lineage for temporary tables in CTAS queries -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3586) Hive hook should set startTime and endTime for hive_process entities to System.currentTimeMillis()
[ https://issues.apache.org/jira/browse/ATLAS-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3586: -- Description: Hive hook should set startTime and endTime for hive_process entities to System.currentTimeMillis() The server side (ATLAS-3565) sets startTime and endTime to System.currentTimeMillis() if it is empty or null, to be consistent we may need to update the hook. was: Hive hook should set startTime and endTime for hive_process entities to System.currentTimeMillis() The server side sets startTime and endTime to System.currentTimeMillis() if it is empty or null, to be consistent we may need to update the hook. > Hive hook should set startTime and endTime for hive_process entities to > System.currentTimeMillis() > -- > > Key: ATLAS-3586 > URL: https://issues.apache.org/jira/browse/ATLAS-3586 > Project: Atlas > Issue Type: Task > Components: atlas-core >Affects Versions: 2.0.0 > Reporter: Sarath Subramanian >Assignee: Sarath Subramanian >Priority: Major > Fix For: 2.1.0 > > Attachments: ATLAS-3568.001.patch > > > Hive hook should set startTime and endTime for hive_process entities to > System.currentTimeMillis() > > The server side (ATLAS-3565) sets startTime and endTime to > System.currentTimeMillis() if it is empty or null, to be consistent we may > need to update the hook. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3586) Hive hook should set startTime and endTime for hive_process entities to System.currentTimeMillis()
[ https://issues.apache.org/jira/browse/ATLAS-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3586: -- Attachment: ATLAS-3568.001.patch > Hive hook should set startTime and endTime for hive_process entities to > System.currentTimeMillis() > -- > > Key: ATLAS-3586 > URL: https://issues.apache.org/jira/browse/ATLAS-3586 > Project: Atlas > Issue Type: Task > Components: atlas-core >Affects Versions: 2.0.0 > Reporter: Sarath Subramanian >Assignee: Sarath Subramanian >Priority: Major > Fix For: 2.1.0 > > Attachments: ATLAS-3568.001.patch > > > Hive hook should set startTime and endTime for hive_process entities to > System.currentTimeMillis() > > The server side sets startTime and endTime to System.currentTimeMillis() if > it is empty or null, to be consistent we may need to update the hook. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3586) Hive hook should set startTime and endTime for hive_process entities to System.currentTimeMillis()
Sarath Subramanian created ATLAS-3586: - Summary: Hive hook should set startTime and endTime for hive_process entities to System.currentTimeMillis() Key: ATLAS-3586 URL: https://issues.apache.org/jira/browse/ATLAS-3586 Project: Atlas Issue Type: Task Components: atlas-core Affects Versions: 2.0.0 Reporter: Sarath Subramanian Assignee: Sarath Subramanian Fix For: 2.1.0 Hive hook should set startTime and endTime for hive_process entities to System.currentTimeMillis() The server side sets startTime and endTime to System.currentTimeMillis() if it is empty or null, to be consistent we may need to update the hook. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3581) HBase hook converts qualifiedName to lowercase for namespace, table, column-family and column
[ https://issues.apache.org/jira/browse/ATLAS-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3581: -- Fix Version/s: 2.1.0 > HBase hook converts qualifiedName to lowercase for namespace, table, > column-family and column > - > > Key: ATLAS-3581 > URL: https://issues.apache.org/jira/browse/ATLAS-3581 > Project: Atlas > Issue Type: Bug > Components: atlas-intg >Affects Versions: 2.0.0 >Reporter: Ramesh Mani >Assignee: Ramesh Mani >Priority: Major > Fix For: 2.1.0 > > > HBase hook converts qualifiedName to lowercase for namespace, table, > column-family and column -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3581) HBase hook converts qualifiedName to lowercase for namespace, table, column-family and column
[ https://issues.apache.org/jira/browse/ATLAS-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3581: -- Affects Version/s: 2.0.0 > HBase hook converts qualifiedName to lowercase for namespace, table, > column-family and column > - > > Key: ATLAS-3581 > URL: https://issues.apache.org/jira/browse/ATLAS-3581 > Project: Atlas > Issue Type: Bug > Components: atlas-intg >Affects Versions: 2.0.0 >Reporter: Ramesh Mani >Assignee: Ramesh Mani >Priority: Major > > HBase hook converts qualifiedName to lowercase for namespace, table, > column-family and column -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 71982: ATLAS-3581:HBase hook converts qualifiedName to lowercase for namespace, table, column-family and column
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71982/#review219260 --- Ship it! Ship It! - Sarath Subramanian On Jan. 10, 2020, 1:56 p.m., Ramesh Mani wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71982/ > --- > > (Updated Jan. 10, 2020, 1:56 p.m.) > > > Review request for atlas, Ashutosh Mestry, Abhay Kulkarni, Madhan Neethiraj, > and Sarath Subramanian. > > > Bugs: ATLAS-3581 > https://issues.apache.org/jira/browse/ATLAS-3581 > > > Repository: atlas > > > Description > --- > > ATLAS-3581:HBase hook converts qualifiedName to lowercase for namespace, > table, column-family and column > > > Diffs > - > > > addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java > 26be9b3 > > addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java > fde70f1 > > > Diff: https://reviews.apache.org/r/71982/diff/1/ > > > Testing > --- > > Verified in local VM > > > Thanks, > > Ramesh Mani > >
[jira] [Updated] (ATLAS-3564) New AWS S3 model addition
[ https://issues.apache.org/jira/browse/ATLAS-3564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3564: -- Affects Version/s: 2.0.0 > New AWS S3 model addition > - > > Key: ATLAS-3564 > URL: https://issues.apache.org/jira/browse/ATLAS-3564 > Project: Atlas > Issue Type: New Feature > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > Fix For: 2.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ATLAS-3564) New AWS S3 model addition
[ https://issues.apache.org/jira/browse/ATLAS-3564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian resolved ATLAS-3564. --- Fix Version/s: 2.1.0 Resolution: Fixed > New AWS S3 model addition > - > > Key: ATLAS-3564 > URL: https://issues.apache.org/jira/browse/ATLAS-3564 > Project: Atlas > Issue Type: New Feature > Components: atlas-core >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > Fix For: 2.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3576) UI: Improve system attributes presentation
[ https://issues.apache.org/jira/browse/ATLAS-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014062#comment-17014062 ] Sarath Subramanian commented on ATLAS-3576: --- Thanks for the patch [~kevalbhatt]. +1 > UI: Improve system attributes presentation > --- > > Key: ATLAS-3576 > URL: https://issues.apache.org/jira/browse/ATLAS-3576 > Project: Atlas > Issue Type: Improvement >Reporter: Keval Bhatt >Assignee: Keval Bhatt >Priority: Major > Attachments: ATLAS-3576-3.patch, ATLAS-3576-v1.patch, > ATLAS-3576.patch, Screen Shot 2020-01-08 at 9.19.00 PM.png, Screen Shot > 2020-01-08 at 9.19.20 PM.png > > > Before Patch: > !Screen Shot 2020-01-08 at 9.19.00 PM.png|width=618,height=284! > > After Patch: > !Screen Shot 2020-01-08 at 9.19.20 PM.png|width=622,height=271! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 71922: ATLAS-3564: New version of AWS S3 model addition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71922/#review219185 --- Ship it! Ship It! - Sarath Subramanian On Jan. 8, 2020, 3:12 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71922/ > --- > > (Updated Jan. 8, 2020, 3:12 p.m.) > > > Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, and > Sarath Subramanian. > > > Bugs: ATLAS-3564 > https://issues.apache.org/jira/browse/ATLAS-3564 > > > Repository: atlas > > > Description > --- > > This is the new version) of aws s3 model at atlas to allow the hierarchical > structure and attributes same as AWS S3 Console. The existing aws s3 models > has limitations like pseudo directory not containing another pseudo > directory, pseudo directory doesn't have attributes of aws object (as some > case it gets populated), both object and bucket doesn't containing all the > attributes of AWS S3 etc. > > > Diffs > - > > addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json PRE-CREATION > > > Diff: https://reviews.apache.org/r/71922/diff/3/ > > > Testing > --- > > > Thanks, > > Sidharth Mishra > >
Re: Review Request 71969: ATLAS-3486: introduce Namespace typedef
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71969/#review219184 --- Fix it, then Ship it! intg/src/main/java/org/apache/atlas/type/AtlasNamespaceType.java Lines 115 (patched) <https://reviews.apache.org/r/71969/#comment307305> this check always mandates ATTR_MAX_STRING_LENGTH to be set for string attributes. should we set a default value for string sttributes if max string length is not specified? repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java Lines 303 (patched) <https://reviews.apache.org/r/71969/#comment307304> 'typeDefStore' is class variable. not needed to be passed in method params. please review. - Sarath Subramanian On Jan. 7, 2020, 5:23 p.m., Aadarsh Jajodia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71969/ > --- > > (Updated Jan. 7, 2020, 5:23 p.m.) > > > Review request for atlas, Ashutosh Mestry, Sridhar K, Le Ma, Madhan > Neethiraj, and Sarath Subramanian. > > > Repository: atlas > > > Description > --- > > ATLAS-3486: introduce Namespace typedef > > > Diffs > - > > intg/src/main/java/org/apache/atlas/AtlasErrorCode.java > 3ebd70d981a734035416fdf9d880cebd68fe2f30 > intg/src/main/java/org/apache/atlas/model/TypeCategory.java > f06f64f450f407e3f9a0e742726ff4dd12ccc695 > intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java > PRE-CREATION > intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java > bb7ead0f9f8bab3094eb82e9e286dd58e8a6e3de > intg/src/main/java/org/apache/atlas/model/typedef/AtlasTypesDef.java > 3634fdfd313639eb97b3c4698e091487b0e44a80 > intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java > 4ee68a936f99bb4c819b5335da2cc8bf7d539397 > intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java > 4742d1c0ea7d7a24fee0453d6fe052579ba6bbd2 > intg/src/main/java/org/apache/atlas/type/AtlasNamespaceType.java > PRE-CREATION > intg/src/main/java/org/apache/atlas/type/AtlasStructType.java > 3475ce6aa8dd4cad4caf08ab5501fc8dc37c1339 > intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java > b071dc9d664cee9e1ffc54726ffbf15f4f602d30 > intg/src/main/java/org/apache/atlas/type/AtlasTypeUtil.java > 0883d54f490e22c6510e6fc0cb804b87713a7ecb > intg/src/main/java/org/apache/atlas/typesystem/types/DataTypes.java > dba2d88146eff314191ae6bb24ad7337b0ea10ae > intg/src/test/java/org/apache/atlas/TestRelationshipUtilsV2.java > 02613b5f7250b14324ed294c22de079b74d55b08 > > intg/src/test/java/org/apache/atlas/model/typedef/TestAtlasNamespaceDef.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > 7c551304b2b65b90302f6e5fa9fc5b9f1b8e2c12 > > repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java > 6016723d82abd47fa15c5f3a94b70c3ce0673413 > > repository/src/main/java/org/apache/atlas/repository/impexp/ExportTypeProcessor.java > 5bad615cd5c223b8246d8443ee5b33925a8b22e9 > > repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java > 08b00e76a5a96aa898f4fc252d3abc22845be50b > > repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java > b04f188beff345d417141d194f0a61e584352111 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasTypeDefGraphStoreV2.java > a5ccfb5b2055c88f596312f4033bc0034d3d165c > > repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2Test.java > PRE-CREATION > webapp/src/main/java/org/apache/atlas/examples/QuickStartV2.java > 6cd0ee331b7ae24757b58e76ec47bf556106846a > webapp/src/main/java/org/apache/atlas/web/rest/TypesREST.java > fb56fad6412079e20dd3e345b81a08d9e5ace657 > > > Diff: https://reviews.apache.org/r/71969/diff/2/ > > > Testing > --- > > As suggested by Madhan in this review here > https://reviews.apache.org/r/71898/, it makes sense to have AtlasNamespaceDef > inherit from AtlasStructDef. This change builds on top of that > > > Thanks, > > Aadarsh Jajodia > >
Re: Review Request 71922: ATLAS-3564: New version of AWS S3 model addition
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71922/#review219183
---
Fix it, then Ship it!
addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
Lines 266 (patched)
<https://reviews.apache.org/r/71922/#comment307303>
remove ending comma (','), same for line 272
- Sarath Subramanian
On Jan. 8, 2020, 10:54 a.m., Sidharth Mishra wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71922/
> ---
>
> (Updated Jan. 8, 2020, 10:54 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, and
> Sarath Subramanian.
>
>
> Bugs: ATLAS-3564
> https://issues.apache.org/jira/browse/ATLAS-3564
>
>
> Repository: atlas
>
>
> Description
> ---
>
> This is the new version) of aws s3 model at atlas to allow the hierarchical
> structure and attributes same as AWS S3 Console. The existing aws s3 models
> has limitations like pseudo directory not containing another pseudo
> directory, pseudo directory doesn't have attributes of aws object (as some
> case it gets populated), both object and bucket doesn't containing all the
> attributes of AWS S3 etc.
>
>
> Diffs
> -
>
> addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/71922/diff/2/
>
>
> Testing
> ---
>
>
> Thanks,
>
> Sidharth Mishra
>
>
[jira] [Commented] (ATLAS-3576) UI: Improve system attributes presentation
[ https://issues.apache.org/jira/browse/ATLAS-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010971#comment-17010971 ] Sarath Subramanian commented on ATLAS-3576: --- one more issue - *Modified By (date)* should be *Last Modified Timestamp (date)* > UI: Improve system attributes presentation > --- > > Key: ATLAS-3576 > URL: https://issues.apache.org/jira/browse/ATLAS-3576 > Project: Atlas > Issue Type: Improvement >Reporter: Keval Bhatt >Assignee: Keval Bhatt >Priority: Major > Attachments: ATLAS-3576.patch, Screen Shot 2020-01-08 at 9.19.00 > PM.png, Screen Shot 2020-01-08 at 9.19.20 PM.png > > > Before Patch: > !Screen Shot 2020-01-08 at 9.19.00 PM.png|width=618,height=284! > > After Patch: > !Screen Shot 2020-01-08 at 9.19.20 PM.png|width=622,height=271! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 71922: ATLAS-3564: New version of AWS S3 model addition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71922/#review219181 --- addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json Lines 7 (patched) <https://reviews.apache.org/r/71922/#comment307297> aws_s3_v2_container => aws_s3_container_v2 addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json Lines 15 (patched) <https://reviews.apache.org/r/71922/#comment307298> aws_s3_v2_containee => aws_s3_containee_v2 addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json Lines 23 (patched) <https://reviews.apache.org/r/71922/#comment307299> aws_s3_v2_object => aws_s3_object_v2 addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json Lines 102 (patched) <https://reviews.apache.org/r/71922/#comment307300> aws_s3_v2_pseudo_dir => aws_s3_pseudo_dir_v2 update line 106 and 107 as well addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json Lines 147 (patched) <https://reviews.apache.org/r/71922/#comment307301> aws_s3_v2_bucket => aws_s3_bucket_v2 addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json Lines 258 (patched) <https://reviews.apache.org/r/71922/#comment307302> aws_s3_v2_container_containees => aws_s3_container_containees_v2 update 264 and 269 - Sarath Subramanian On Jan. 8, 2020, 10:54 a.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71922/ > --- > > (Updated Jan. 8, 2020, 10:54 a.m.) > > > Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, and > Sarath Subramanian. > > > Bugs: ATLAS-3564 > https://issues.apache.org/jira/browse/ATLAS-3564 > > > Repository: atlas > > > Description > --- > > This is the new version) of aws s3 model at atlas to allow the hierarchical > structure and attributes same as AWS S3 Console. The existing aws s3 models > has limitations like pseudo directory not containing another pseudo > directory, pseudo directory doesn't have attributes of aws object (as some > case it gets populated), both object and bucket doesn't containing all the > attributes of AWS S3 etc. > > > Diffs > - > > addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json PRE-CREATION > > > Diff: https://reviews.apache.org/r/71922/diff/2/ > > > Testing > --- > > > Thanks, > > Sidharth Mishra > >
[jira] [Commented] (ATLAS-3576) UI: Improve system attributes presentation
[ https://issues.apache.org/jira/browse/ATLAS-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010921#comment-17010921 ] Sarath Subramanian commented on ATLAS-3576: --- [~kevalbhatt], the build is failing with the patch. can you check > UI: Improve system attributes presentation > --- > > Key: ATLAS-3576 > URL: https://issues.apache.org/jira/browse/ATLAS-3576 > Project: Atlas > Issue Type: Improvement >Reporter: Keval Bhatt >Assignee: Keval Bhatt >Priority: Major > Attachments: ATLAS-3576.patch, Screen Shot 2020-01-08 at 9.19.00 > PM.png, Screen Shot 2020-01-08 at 9.19.20 PM.png > > > Before Patch: > !Screen Shot 2020-01-08 at 9.19.00 PM.png|width=618,height=284! > > After Patch: > !Screen Shot 2020-01-08 at 9.19.20 PM.png|width=622,height=271! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 71933: ATLAS-3568: Performance improvements in writing audit logs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71933/ --- (Updated Dec. 20, 2019, 10:21 a.m.) Review request for atlas, Ashutosh Mestry, Nikhil Bonte, Nixon Rodrigues, Saqeeb Shaikh, and Sarath Subramanian. Bugs: ATLAS-3568 https://issues.apache.org/jira/browse/ATLAS-3568 Repository: atlas Description --- Atlas server generates audit logs on changes to entities - like add/remove classifications, entity attribute updates, etc. Add/remove of classifications to an entity can result in multiple entities to be updated due to tag-propagation. In such cases, the time taken to create audit logs can be reduced by writing audits for all updated entities in one call to the audit store, instead of current implementation that writes audit logs for one entity at a time. Diffs (updated) - intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java d36582b repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 706da16 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java bd1ba58 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java 3fbcecb repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java 7a2ccb9 webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java 8893380 Diff: https://reviews.apache.org/r/71933/diff/2/ Changes: https://reviews.apache.org/r/71933/diff/1-2/ Testing --- manually validated. precommit - https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1595/console Thanks, Sarath Subramanian
Review Request 71933: ATLAS-3568: Performance improvements in writing audit logs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71933/ --- Review request for atlas, Ashutosh Mestry, Nikhil Bonte, Nixon Rodrigues, Saqeeb Shaikh, and Sarath Subramanian. Bugs: ATLAS-3568 https://issues.apache.org/jira/browse/ATLAS-3568 Repository: atlas Description --- Atlas server generates audit logs on changes to entities - like add/remove classifications, entity attribute updates, etc. Add/remove of classifications to an entity can result in multiple entities to be updated due to tag-propagation. In such cases, the time taken to create audit logs can be reduced by writing audits for all updated entities in one call to the audit store, instead of current implementation that writes audit logs for one entity at a time. Diffs - intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java d36582b07 repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 706da1664 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java bd1ba5898 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java 3fbcecb42 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java 7a2ccb977 webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java 8893380fe Diff: https://reviews.apache.org/r/71933/diff/1/ Testing --- manually validated. precommit - https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1595/console Thanks, Sarath Subramanian
[jira] [Commented] (ATLAS-3568) Performance improvements in writing audit logs
[ https://issues.apache.org/jira/browse/ATLAS-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17001028#comment-17001028 ] Sarath Subramanian commented on ATLAS-3568: --- attaching patch from [~saqeeb.s] > Performance improvements in writing audit logs > -- > > Key: ATLAS-3568 > URL: https://issues.apache.org/jira/browse/ATLAS-3568 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Saqeeb Shaikh >Assignee: Saqeeb Shaikh >Priority: Major > Attachments: ATLAS-3568.001.patch > > > Atlas server generates audit logs on changes to entities - like add/remove > classifications, entity attribute updates, etc. Add/remove of classifications > to an entity can result in multiple entities to be updated due to > tag-propagation. In such cases, the time taken to create audit logs can be > reduced by writing audits for all updated entities in one call to the audit > store, instead of current implementation that writes audit logs for one > entity at a time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3568) Performance improvements in writing audit logs
[ https://issues.apache.org/jira/browse/ATLAS-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3568: -- Attachment: ATLAS-3568.001.patch > Performance improvements in writing audit logs > -- > > Key: ATLAS-3568 > URL: https://issues.apache.org/jira/browse/ATLAS-3568 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Saqeeb Shaikh >Assignee: Saqeeb Shaikh >Priority: Major > Attachments: ATLAS-3568.001.patch > > > Atlas server generates audit logs on changes to entities - like add/remove > classifications, entity attribute updates, etc. Add/remove of classifications > to an entity can result in multiple entities to be updated due to > tag-propagation. In such cases, the time taken to create audit logs can be > reduced by writing audits for all updated entities in one call to the audit > store, instead of current implementation that writes audit logs for one > entity at a time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 71930: ATLAS-3566: improvements in upgrade patches, to avoid full-scan
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71930/#review219072 --- Ship it! Ship It! - Sarath Subramanian On Dec. 19, 2019, 5:14 p.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71930/ > --- > > (Updated Dec. 19, 2019, 5:14 p.m.) > > > Review request for atlas, Ashutosh Mestry, Nikhil Bonte, Nixon Rodrigues, > Saqeeb Shaikh, and Sarath Subramanian. > > > Bugs: ATLAS-3566 > https://issues.apache.org/jira/browse/ATLAS-3566 > > > Repository: atlas > > > Description > --- > > Patches that update for vertices for unique-attributes and > classification-text fields currently perform a full-scan, which can cause a > long delay before processing can begin. Updated this to process entities one > type after another - which avoids full-scan, as the query uses indexes on > typeName field. > > For classfication-text field update, the processing is updated to scan for > classfication vertices and find the entity vertices for each classification > found. This would also avoid full-scan, and in addition avoid additional cost > of fetching entities that don't have classifications attached. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/patches/ClassificationTextPatch.java > 2af50ba4d > > repository/src/main/java/org/apache/atlas/repository/patches/ConcurrentPatchProcessor.java > 3eedb9839 > > repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatch.java > bd5e32b30 > > > Diff: https://reviews.apache.org/r/71930/diff/1/ > > > Testing > --- > > - verified manually > - pre-commit tests run: > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1594/ > > > Thanks, > > Madhan Neethiraj > >
[jira] [Assigned] (ATLAS-3566) Improve UniqueAttributePatchProcessor and ClassificationTextPatch performance
[ https://issues.apache.org/jira/browse/ATLAS-3566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian reassigned ATLAS-3566: - Assignee: (was: Sarath Subramanian) > Improve UniqueAttributePatchProcessor and ClassificationTextPatch performance > - > > Key: ATLAS-3566 > URL: https://issues.apache.org/jira/browse/ATLAS-3566 > Project: Atlas > Issue Type: Task > Components: atlas-core >Affects Versions: 2.0.0 > Reporter: Sarath Subramanian >Priority: Major > Fix For: 2.1.0 > > > UniqueAttributePatchProcessor and ClassificationTextPatchProcessor uses full > table scan to get vertex Ids to apply patches. This approach is very slow in > atlas with lot of vertices. Optimization needs to be added to look for > vertices which has classification and unique attribute entity vertices. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3566) Improve UniqueAttributePatchProcessor and ClassificationTextPatch performance
Sarath Subramanian created ATLAS-3566: - Summary: Improve UniqueAttributePatchProcessor and ClassificationTextPatch performance Key: ATLAS-3566 URL: https://issues.apache.org/jira/browse/ATLAS-3566 Project: Atlas Issue Type: Task Components: atlas-core Affects Versions: 2.0.0 Reporter: Sarath Subramanian Assignee: Sarath Subramanian Fix For: 2.1.0 UniqueAttributePatchProcessor and ClassificationTextPatchProcessor uses full table scan to get vertex Ids to apply patches. This approach is very slow in atlas with lot of vertices. Optimization needs to be added to look for vertices which has classification and unique attribute entity vertices. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ATLAS-3565) Atlas server should set startTime and endTime for hive_process to System.currentTimeMillis() if either of these are null
[ https://issues.apache.org/jira/browse/ATLAS-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian resolved ATLAS-3565. --- Fix Version/s: 2.1.0 Resolution: Fixed > Atlas server should set startTime and endTime for hive_process to > System.currentTimeMillis() if either of these are null > > > Key: ATLAS-3565 > URL: https://issues.apache.org/jira/browse/ATLAS-3565 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Saqeeb Shaikh >Assignee: Saqeeb Shaikh >Priority: Major > Fix For: 2.1.0 > > > For entity type hive_process startTime and endTime are compulsory attributes, > if they are null then atlas server should set current system time for those > attributes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 71929: ATLAS-3565 :: Atlas server should set startTime and endTime for hive_process to System.currentTimeMillis() if either of these are null.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71929/#review219071 --- Ship it! Ship It! - Sarath Subramanian On Dec. 19, 2019, 6:20 a.m., Saqeeb Shaikh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71929/ > --- > > (Updated Dec. 19, 2019, 6:20 a.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Nixon Rodrigues, > and Sarath Subramanian. > > > Bugs: ATLAS-3565 > https://issues.apache.org/jira/browse/ATLAS-3565 > > > Repository: atlas > > > Description > --- > > For entity type hive_process startTime and endTime are compulsory attributes, > if they are null then atlas server should set current system time for those > attributes. > > > Diffs > - > > > webapp/src/main/java/org/apache/atlas/notification/preprocessor/EntityPreprocessor.java > 0cddd41 > > webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java > d31495c > > > Diff: https://reviews.apache.org/r/71929/diff/1/ > > > Testing > --- > > Pre commit build :: > https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1593/console > > Manually verified with JSON having huge number of columns. > > > Thanks, > > Saqeeb Shaikh > >
[jira] [Resolved] (ATLAS-3562) Hive metadata has the same classification multiple times
[ https://issues.apache.org/jira/browse/ATLAS-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian resolved ATLAS-3562. --- Resolution: Fixed > Hive metadata has the same classification multiple times > > > Key: ATLAS-3562 > URL: https://issues.apache.org/jira/browse/ATLAS-3562 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Mandar Ambawane >Assignee: Mandar Ambawane >Priority: Major > Fix For: 2.1.0 > > > # Create multiple table entities of a hive entity. > # Associate one classification to the parent entity. > # Simultaneously associate same classification to same entity from another > browser. > EXPECTED: Application should throw error message. > ACTUAL RESULT: Same classification gets associated with the entity twice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3562) Hive metadata has the same classification multiple times
[ https://issues.apache.org/jira/browse/ATLAS-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3562: -- Affects Version/s: 2.0.0 > Hive metadata has the same classification multiple times > > > Key: ATLAS-3562 > URL: https://issues.apache.org/jira/browse/ATLAS-3562 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Mandar Ambawane >Assignee: Mandar Ambawane >Priority: Major > > # Create multiple table entities of a hive entity. > # Associate one classification to the parent entity. > # Simultaneously associate same classification to same entity from another > browser. > EXPECTED: Application should throw error message. > ACTUAL RESULT: Same classification gets associated with the entity twice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3562) Hive metadata has the same classification multiple times
[ https://issues.apache.org/jira/browse/ATLAS-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3562: -- Fix Version/s: 2.1.0 > Hive metadata has the same classification multiple times > > > Key: ATLAS-3562 > URL: https://issues.apache.org/jira/browse/ATLAS-3562 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Mandar Ambawane >Assignee: Mandar Ambawane >Priority: Major > Fix For: 2.1.0 > > > # Create multiple table entities of a hive entity. > # Associate one classification to the parent entity. > # Simultaneously associate same classification to same entity from another > browser. > EXPECTED: Application should throw error message. > ACTUAL RESULT: Same classification gets associated with the entity twice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3562) Hive metadata has the same classification multiple times
[ https://issues.apache.org/jira/browse/ATLAS-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3562: -- Component/s: atlas-core > Hive metadata has the same classification multiple times > > > Key: ATLAS-3562 > URL: https://issues.apache.org/jira/browse/ATLAS-3562 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Mandar Ambawane >Assignee: Mandar Ambawane >Priority: Major > Fix For: 2.1.0 > > > # Create multiple table entities of a hive entity. > # Associate one classification to the parent entity. > # Simultaneously associate same classification to same entity from another > browser. > EXPECTED: Application should throw error message. > ACTUAL RESULT: Same classification gets associated with the entity twice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 71924: ATLAS-3562: Hive metadata has the same classification multiple times
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71924/#review219055 --- Ship it! Ship It! - Sarath Subramanian On Dec. 17, 2019, 11:43 p.m., Mandar Ambawane wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71924/ > --- > > (Updated Dec. 17, 2019, 11:43 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Nixon Rodrigues, > and Sarath Subramanian. > > > Bugs: ATLAS-3562 > https://issues.apache.org/jira/browse/ATLAS-3562 > > > Repository: atlas > > > Description > --- > > Put Lock on entity before its gets cached. > > Moved GraphTransactionInterceptor.lockObjectAndReleasePostCommit(guid); > before code gets AtlasVertex for the entity guid. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java > ea5e6ab > > > Diff: https://reviews.apache.org/r/71924/diff/1/ > > > Testing > --- > > TESTING: > Testing done by sending 2 simultaneous curl resquests to associate same > Classification to the same entity. > > RESULT: > > Classification gets associted with the entity only once. > > Application throws exception for the other simultaneous curl request: > org.apache.atlas.exception.AtlasBaseException: invalid parameters: entity: > , already associated with classification: > > > Thanks, > > Mandar Ambawane > >
Re: Review Request 71919: ATLAS-3563: Improve tag propagation performance using in-memory traversal
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71919/ --- (Updated Dec. 17, 2019, 5:31 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, keval bhatt, Sridhar K, Le Ma, Mandar Ambawane, mayank jain, Nixon Rodrigues, Sameer Shaikh, and Sarath Subramanian. Bugs: ATLAS-3563 https://issues.apache.org/jira/browse/ATLAS-3563 Repository: atlas Description --- Tag propagation uses gremlin query to find entities to which the tag has to be propagated to. Gremlin query doesn't scale well for entities with large lineage (with many depth). In-memory traversal seems to have improved performance significantly since it avoids the overhead added by gremlin script engine initialization, query execution time. Performance improvement in tag propagation from 3004 ms to 180 ms is seen Diffs (updated) - graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java 6de4dcf10 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusVertex.java 71b285731 intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 7a2aae2e9 intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 928ac0d8b repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 1e7acf1e7 repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java c9ed79750 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java 1c8b057ba repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java a415d3084 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 8a24fa127 repository/src/main/java/org/apache/atlas/util/AtlasGremlin3QueryProvider.java 20c570f7f repository/src/main/java/org/apache/atlas/util/AtlasGremlinQueryProvider.java d201db338 repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java 6f9c05e7a Diff: https://reviews.apache.org/r/71919/diff/4/ Changes: https://reviews.apache.org/r/71919/diff/3-4/ Testing --- Manually validated tag propagation works. * Add classification * Block propagation * Change Propagation direction * Remove Classification Thanks, Sarath Subramanian
Re: Review Request 71919: ATLAS-3563: Improve tag propagation performance using in-memory traversal
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71919/ --- (Updated Dec. 17, 2019, 10:05 a.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, keval bhatt, Sridhar K, Le Ma, Mandar Ambawane, mayank jain, Nixon Rodrigues, Sameer Shaikh, and Sarath Subramanian. Bugs: ATLAS-3563 https://issues.apache.org/jira/browse/ATLAS-3563 Repository: atlas Description --- Tag propagation uses gremlin query to find entities to which the tag has to be propagated to. Gremlin query doesn't scale well for entities with large lineage (with many depth). In-memory traversal seems to have improved performance significantly since it avoids the overhead added by gremlin script engine initialization, query execution time. Performance improvement in tag propagation from 3004 ms to 180 ms is seen Diffs (updated) - graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java 6de4dcf10 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusVertex.java 71b285731 intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 928ac0d8b repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 1e7acf1e7 repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java c9ed79750 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java 1c8b057ba repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java a415d3084 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 8a24fa127 repository/src/main/java/org/apache/atlas/util/AtlasGremlin3QueryProvider.java 20c570f7f repository/src/main/java/org/apache/atlas/util/AtlasGremlinQueryProvider.java d201db338 Diff: https://reviews.apache.org/r/71919/diff/3/ Changes: https://reviews.apache.org/r/71919/diff/2-3/ Testing --- Manually validated tag propagation works. * Add classification * Block propagation * Change Propagation direction * Remove Classification Thanks, Sarath Subramanian
[jira] [Updated] (ATLAS-3563) Improve tag propagation performance using in-memory traversal
[ https://issues.apache.org/jira/browse/ATLAS-3563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3563: -- Description: Tag propagation uses gremlin query to find entities to which the tag has to be propagated to. Gremlin query doesn't scale well for entities with large lineage (with many depth). In-memory traversal seems to have improved performance significantly since it avoids the overhead added by gremlin script engine initialization, query execution time. Performance improvement in tag propagation from *3004 ms* to *180 ms* is seen was: Tag propagation uses gremlin query to find entities to which the tag has to be propagated to. Gremlin query is not scaling well for entities with large lineage (with many depth). In-memory traversal seems to have improved performance significantly since it avoids the overhead added by gremlin script engine initialization, query execution time. Performance improvement in tag propagation from 3004 ms to 180 ms is seen > Improve tag propagation performance using in-memory traversal > - > > Key: ATLAS-3563 > URL: https://issues.apache.org/jira/browse/ATLAS-3563 > Project: Atlas > Issue Type: Task > Components: atlas-core >Affects Versions: 2.0.0 > Reporter: Sarath Subramanian > Assignee: Sarath Subramanian >Priority: Major > Fix For: 2.1.0 > > > Tag propagation uses gremlin query to find entities to which the tag has to > be propagated to. > Gremlin query doesn't scale well for entities with large lineage (with many > depth). In-memory traversal seems to have improved performance significantly > since it avoids the overhead added by gremlin script engine initialization, > query execution time. > Performance improvement in tag propagation from *3004 ms* to *180 ms* is seen -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3563) Improve tag propagation performance using in-memory traversal
[ https://issues.apache.org/jira/browse/ATLAS-3563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3563: -- Attachment: (was: ATLAS-3563.001.patch) > Improve tag propagation performance using in-memory traversal > - > > Key: ATLAS-3563 > URL: https://issues.apache.org/jira/browse/ATLAS-3563 > Project: Atlas > Issue Type: Task > Components: atlas-core >Affects Versions: 2.0.0 > Reporter: Sarath Subramanian > Assignee: Sarath Subramanian >Priority: Major > Fix For: 2.1.0 > > > Tag propagation uses gremlin query to find entities to which the tag has to > be propagated to. > Gremlin query is not scaling well for entities with large lineage (with many > depth). In-memory traversal seems to have improved performance significantly > since it avoids the overhead added by gremlin script engine initialization, > query execution time. > > Performance improvement in tag propagation from 3004 ms to 180 ms is seen -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3563) Improve tag propagation performance using in-memory traversal
[ https://issues.apache.org/jira/browse/ATLAS-3563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3563: -- Attachment: ATLAS-3563.001.patch > Improve tag propagation performance using in-memory traversal > - > > Key: ATLAS-3563 > URL: https://issues.apache.org/jira/browse/ATLAS-3563 > Project: Atlas > Issue Type: Task > Components: atlas-core >Affects Versions: 2.0.0 > Reporter: Sarath Subramanian > Assignee: Sarath Subramanian >Priority: Major > Fix For: 2.1.0 > > Attachments: ATLAS-3563.001.patch > > > Tag propagation uses gremlin query to find entities to which the tag has to > be propagated to. > Gremlin query is not scaling well for entities with large lineage (with many > depth). In-memory traversal seems to have improved performance significantly > since it avoids the overhead added by gremlin script engine initialization, > query execution time. > > Performance improvement in tag propagation from 3004 ms to 180 ms is seen -- This message was sent by Atlassian Jira (v8.3.4#803005)
Review Request 71919: ATLAS-3563: Improve tag propagation performance using in-memory traversal
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71919/ --- Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, keval bhatt, Sridhar K, Le Ma, Mandar Ambawane, mayank jain, Nixon Rodrigues, Sameer Shaikh, and Sarath Subramanian. Bugs: ATLAS-3563 https://issues.apache.org/jira/browse/ATLAS-3563 Repository: atlas Description --- Tag propagation uses gremlin query to find entities to which the tag has to be propagated to. Gremlin query is not scaling well for entities with large lineage (with many depth). In-memory traversal seems to have improved performance significantly since it avoids the overhead added by gremlin script engine initialization, query execution time. Performance improvement in tag propagation from 3004 ms to 180 ms is seen Diffs - graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java 6de4dcf10 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusVertex.java 71b285731 intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 928ac0d8b repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 1e7acf1e7 repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java c9ed79750 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java 1c8b057ba repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java a415d3084 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 8a24fa127 repository/src/main/java/org/apache/atlas/util/AtlasGremlin3QueryProvider.java 20c570f7f repository/src/main/java/org/apache/atlas/util/AtlasGremlinQueryProvider.java d201db338 Diff: https://reviews.apache.org/r/71919/diff/1/ Testing --- Manually validated tag propagation works. * Add classification * Block propagation * Change Propagation direction * Remove Classification Thanks, Sarath Subramanian
[jira] [Created] (ATLAS-3563) Improve tag propagation performance using in-memory traversal
Sarath Subramanian created ATLAS-3563: - Summary: Improve tag propagation performance using in-memory traversal Key: ATLAS-3563 URL: https://issues.apache.org/jira/browse/ATLAS-3563 Project: Atlas Issue Type: Task Components: atlas-core Affects Versions: 2.0.0 Reporter: Sarath Subramanian Assignee: Sarath Subramanian Fix For: 2.1.0 Tag propagation uses gremlin query to find entities to which the tag has to be propagated to. Gremlin query is not scaling well for entities with large lineage (with many depth). In-memory traversal seems to have improved performance significantly since it avoids the overhead added by gremlin script engine initialization, query execution time. Performance improvement in tag propagation from 3004 ms to 180 ms is seen -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3561) Atlas start fails in embedded-hbase mode with zookeeper error
[
https://issues.apache.org/jira/browse/ATLAS-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16996647#comment-16996647
]
Sarath Subramanian commented on ATLAS-3561:
---
the error log shows about embedded HBase failing to start. This might be due to
$JAVA_HOME env variable not set. Can you explain how setting
hbase.unsafe.stream.capability.enforce = false
solved the issue.
> Atlas start fails in embedded-hbase mode with zookeeper error
> -
>
> Key: ATLAS-3561
> URL: https://issues.apache.org/jira/browse/ATLAS-3561
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 3.0.0
>Reporter: chaitali borole
>Assignee: chaitali borole
>Priority: Minor
> Fix For: 3.0.0
>
>
> After compiling Atlas with {{mvn clean package -Pdist,embedded-hbase-solr}}
> and starting Atlas with embedded services hbase, solr and kafka using
> {{atlas_start.py}}, the Atlas start fails with below error in
> {{application.log}}
> {noformat}
> 2019-12-09 16:01:28,839 INFO - [main:] ~ Not running setup per configuration
> atlas.server.run.setup.on.start. (SetupSteps$SetupRequired:189)
> 2019-12-09 16:01:32,786 WARN -
> [ReadOnlyZKClient-localhost:2181@0x0fa5f81c-SendThread(localhost:2181):] ~
> Session 0x16eea36b27b0003 for server null, unexpected error, closing socket
> connection and attempting reconnect (ClientCnxn$SendThread:1102)
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
> 2019-12-09 16:01:32,889 WARN - [ReadOnlyZKClient-localhost:2181@0x0fa5f81c:]
> ~ 0x0fa5f81c to localhost:2181 failed for get of /hbase/meta-region-server,
> code = CONNECTIONLOSS, retries = 1 (ReadOnlyZKClient$ZKTask$1:183)
> 2019-12-09 16:01:34,004 WARN -
> [ReadOnlyZKClient-localhost:2181@0x0fa5f81c-SendThread(localhost:2181):] ~
> Session 0x16eea36b27b0003 for server null, unexpected error, closing socket
> connection and attempting reconnect (ClientCnxn$SendThread:1102)
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
> {noformat}
>
> *Workaround*
> Adding below property in {{hbase-site.xml.template}} and
> running {{mvn clean package -Pdist,embedded-hbase-solr}} the issue is
> resolved.
>
> {code:none}
>
> hbase.unsafe.stream.capability.enforce
> false
>
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-3558) Improve lineage performance using in-memory traversal
[ https://issues.apache.org/jira/browse/ATLAS-3558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3558: -- Description: Lineage in atlas uses graph query to compute lineage across entities (inputs, outputs or both). Lineage rendering performance have degraded after using janusgraph version 0.4.0 On investigation, lineage graph query initialization and execution using gremlin script engine has been found to be the bottleneck. Alternate in-memory computation of lineage has improved performance by many folds (~90% improvement). This Jira is about adding alternate in-memory computation of lineage. *"atlas.lineage.query.use.gremlin"* property can be used to toggle between graph query and in-memory computation of lineage. The default option will be in-memory. was: Lineage in atlas uses graph query to compute lineage across entities (inputs, outputs or both). Lineage rendering performance have degraded after using janusgraph version 0.4.0 On investigation, lineage graph query initialization and execution using gremlin script engine has been found to be the bottleneck. Alternate in-memory computation of lineage has improved performance by many folds (~90% improvement). This Jira is about adding alternate in-memory computation of lineage. *"atlas.use.graph.query.for.lineage"* property can be used to toggle between graph query and in-memory computation of lineage. The default option will be in-memory. > Improve lineage performance using in-memory traversal > - > > Key: ATLAS-3558 > URL: https://issues.apache.org/jira/browse/ATLAS-3558 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Sarath Subramanian >Assignee: Sarath Subramanian >Priority: Major > Fix For: 2.1.0 > > Attachments: image-2019-12-11-15-54-30-483.png > > > Lineage in atlas uses graph query to compute lineage across entities (inputs, > outputs or both). Lineage rendering performance have degraded after using > janusgraph version 0.4.0 > On investigation, lineage graph query initialization and execution using > gremlin script engine has been found to be the bottleneck. > Alternate in-memory computation of lineage has improved performance by many > folds (~90% improvement). This Jira is about adding alternate in-memory > computation of lineage. > *"atlas.lineage.query.use.gremlin"* property can be used to toggle between > graph query and in-memory computation of lineage. The default option will be > in-memory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 71902: ATLAS-3558: Improve lineage performance using in-memory traversal
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71902/ --- (Updated Dec. 13, 2019, 7:53 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, keval bhatt, Sridhar K, Le Ma, Mandar Ambawane, mayank jain, Nixon Rodrigues, Sameer Shaikh, and Sarath Subramanian. Bugs: ATLAS-3558 https://issues.apache.org/jira/browse/ATLAS-3558 Repository: atlas Description --- Lineage in atlas uses graph query to compute lineage across entities (inputs, outputs or both). Lineage rendering performance have degraded after using janusgraph version 0.4.0 On investigation, lineage graph query initialization and execution using gremlin script engine has been found to be the bottleneck. Alternate in-memory computation of lineage has improved performance by many folds (~90% improvement). This Jira is about adding alternate in-memory computation of lineage. "atlas.use.graph.query.for.lineage" property can be used to toggle between graph query and in-memory computation of lineage. The default option will be in-memory. Diffs (updated) - intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 979bd0ae3 repository/src/main/java/org/apache/atlas/discovery/EntityLineageService.java 9a020468d Diff: https://reviews.apache.org/r/71902/diff/5/ Changes: https://reviews.apache.org/r/71902/diff/4-5/ Testing --- Precommit: https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1585/console Manually validated lineage rendering works fine for simple, complex and circular lineages. Thanks, Sarath Subramanian
Re: Review Request 71902: ATLAS-3558: Improve lineage performance using in-memory traversal
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71902/ --- (Updated Dec. 13, 2019, 4:38 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, keval bhatt, Sridhar K, Le Ma, Mandar Ambawane, mayank jain, Nixon Rodrigues, Sameer Shaikh, and Sarath Subramanian. Bugs: ATLAS-3558 https://issues.apache.org/jira/browse/ATLAS-3558 Repository: atlas Description --- Lineage in atlas uses graph query to compute lineage across entities (inputs, outputs or both). Lineage rendering performance have degraded after using janusgraph version 0.4.0 On investigation, lineage graph query initialization and execution using gremlin script engine has been found to be the bottleneck. Alternate in-memory computation of lineage has improved performance by many folds (~90% improvement). This Jira is about adding alternate in-memory computation of lineage. "atlas.use.graph.query.for.lineage" property can be used to toggle between graph query and in-memory computation of lineage. The default option will be in-memory. Diffs (updated) - intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 979bd0ae3 repository/src/main/java/org/apache/atlas/discovery/EntityLineageService.java 9a020468d Diff: https://reviews.apache.org/r/71902/diff/4/ Changes: https://reviews.apache.org/r/71902/diff/3-4/ Testing --- Precommit: https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1585/console Manually validated lineage rendering works fine for simple, complex and circular lineages. Thanks, Sarath Subramanian
Re: Review Request 71902: ATLAS-3558: Improve lineage performance using in-memory traversal
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71902/ --- (Updated Dec. 13, 2019, 2:54 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, keval bhatt, Sridhar K, Le Ma, Mandar Ambawane, mayank jain, Nixon Rodrigues, Sameer Shaikh, and Sarath Subramanian. Changes --- addressed review comments. Bugs: ATLAS-3558 https://issues.apache.org/jira/browse/ATLAS-3558 Repository: atlas Description --- Lineage in atlas uses graph query to compute lineage across entities (inputs, outputs or both). Lineage rendering performance have degraded after using janusgraph version 0.4.0 On investigation, lineage graph query initialization and execution using gremlin script engine has been found to be the bottleneck. Alternate in-memory computation of lineage has improved performance by many folds (~90% improvement). This Jira is about adding alternate in-memory computation of lineage. "atlas.use.graph.query.for.lineage" property can be used to toggle between graph query and in-memory computation of lineage. The default option will be in-memory. Diffs (updated) - intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 979bd0ae3 repository/src/main/java/org/apache/atlas/discovery/EntityLineageService.java 9a020468d Diff: https://reviews.apache.org/r/71902/diff/3/ Changes: https://reviews.apache.org/r/71902/diff/2-3/ Testing --- Precommit: https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1581/console Manually validated lineage rendering works fine for simple, complex and circular lineages. Thanks, Sarath Subramanian
Re: Review Request 71902: ATLAS-3558: Improve lineage performance using in-memory traversal
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71902/ --- (Updated Dec. 12, 2019, 3:20 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, keval bhatt, Sridhar K, Le Ma, Mandar Ambawane, mayank jain, Nixon Rodrigues, Sameer Shaikh, and Sarath Subramanian. Changes --- addressed review comments. Bugs: ATLAS-3558 https://issues.apache.org/jira/browse/ATLAS-3558 Repository: atlas Description --- Lineage in atlas uses graph query to compute lineage across entities (inputs, outputs or both). Lineage rendering performance have degraded after using janusgraph version 0.4.0 On investigation, lineage graph query initialization and execution using gremlin script engine has been found to be the bottleneck. Alternate in-memory computation of lineage has improved performance by many folds (~90% improvement). This Jira is about adding alternate in-memory computation of lineage. "atlas.use.graph.query.for.lineage" property can be used to toggle between graph query and in-memory computation of lineage. The default option will be in-memory. Diffs (updated) - intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 979bd0ae3 repository/src/main/java/org/apache/atlas/discovery/EntityLineageService.java 9a020468d Diff: https://reviews.apache.org/r/71902/diff/2/ Changes: https://reviews.apache.org/r/71902/diff/1-2/ Testing --- Precommit: https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1581/console Manually validated lineage rendering works fine for simple, complex and circular lineages. Thanks, Sarath Subramanian
[jira] [Comment Edited] (ATLAS-3558) Improve lineage performance using in-memory traversal
[ https://issues.apache.org/jira/browse/ATLAS-3558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994005#comment-16994005 ] Sarath Subramanian edited comment on ATLAS-3558 at 12/12/19 12:00 AM: -- !image-2019-12-11-15-54-30-483.png! improvement in above lineage by over *92%* (from *4516 ms* to *325.4 ms*) was (Author: sarath.ku...@gmail.com): !image-2019-12-11-15-54-30-483.png! improved in above lineage by over *92%* (from *4516 ms* to *325.4 ms*) > Improve lineage performance using in-memory traversal > - > > Key: ATLAS-3558 > URL: https://issues.apache.org/jira/browse/ATLAS-3558 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0 > Reporter: Sarath Subramanian > Assignee: Sarath Subramanian >Priority: Major > Fix For: 2.1.0 > > Attachments: image-2019-12-11-15-54-30-483.png > > > Lineage in atlas uses graph query to compute lineage across entities (inputs, > outputs or both). Lineage rendering performance have degraded after using > janusgraph version 0.4.0 > On investigation, lineage graph query initialization and execution using > gremlin script engine has been found to be the bottleneck. > Alternate in-memory computation of lineage has improved performance by many > folds (~90% improvement). This Jira is about adding alternate in-memory > computation of lineage. > *"atlas.use.graph.query.for.lineage"* property can be used to toggle between > graph query and in-memory computation of lineage. The default option will be > in-memory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Review Request 71902: ATLAS-3558: Improve lineage performance using in-memory traversal
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71902/ --- Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, keval bhatt, Sridhar K, Le Ma, Mandar Ambawane, mayank jain, Nixon Rodrigues, Sameer Shaikh, and Sarath Subramanian. Bugs: ATLAS-3558 https://issues.apache.org/jira/browse/ATLAS-3558 Repository: atlas Description --- Lineage in atlas uses graph query to compute lineage across entities (inputs, outputs or both). Lineage rendering performance have degraded after using janusgraph version 0.4.0 On investigation, lineage graph query initialization and execution using gremlin script engine has been found to be the bottleneck. Alternate in-memory computation of lineage has improved performance by many folds (~90% improvement). This Jira is about adding alternate in-memory computation of lineage. "atlas.use.graph.query.for.lineage" property can be used to toggle between graph query and in-memory computation of lineage. The default option will be in-memory. Diffs - intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 979bd0ae3 repository/src/main/java/org/apache/atlas/discovery/EntityLineageService.java 9a020468d Diff: https://reviews.apache.org/r/71902/diff/1/ Testing --- Precommit: https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1581/console Thanks, Sarath Subramanian
[jira] [Commented] (ATLAS-3558) Improve lineage performance using in-memory traversal
[ https://issues.apache.org/jira/browse/ATLAS-3558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994005#comment-16994005 ] Sarath Subramanian commented on ATLAS-3558: --- !image-2019-12-11-15-54-30-483.png! improved in above lineage by over *92%* (from *4516 ms* to *325.4 ms*) > Improve lineage performance using in-memory traversal > - > > Key: ATLAS-3558 > URL: https://issues.apache.org/jira/browse/ATLAS-3558 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0 > Reporter: Sarath Subramanian > Assignee: Sarath Subramanian >Priority: Major > Fix For: 2.1.0 > > Attachments: image-2019-12-11-15-54-30-483.png > > > Lineage in atlas uses graph query to compute lineage across entities (inputs, > outputs or both). Lineage rendering performance have degraded after using > janusgraph version 0.4.0 > On investigation, lineage graph query initialization and execution using > gremlin script engine has been found to be the bottleneck. > Alternate in-memory computation of lineage has improved performance by many > folds (~90% improvement). This Jira is about adding alternate in-memory > computation of lineage. > *"atlas.use.graph.query.for.lineage"* property can be used to toggle between > graph query and in-memory computation of lineage. The default option will be > in-memory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3558) Improve lineage performance using in-memory traversal
[ https://issues.apache.org/jira/browse/ATLAS-3558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3558: -- Attachment: lineage1.png > Improve lineage performance using in-memory traversal > - > > Key: ATLAS-3558 > URL: https://issues.apache.org/jira/browse/ATLAS-3558 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0 > Reporter: Sarath Subramanian > Assignee: Sarath Subramanian >Priority: Major > Fix For: 2.1.0 > > Attachments: image-2019-12-11-15-54-30-483.png > > > Lineage in atlas uses graph query to compute lineage across entities (inputs, > outputs or both). Lineage rendering performance have degraded after using > janusgraph version 0.4.0 > On investigation, lineage graph query initialization and execution using > gremlin script engine has been found to be the bottleneck. > Alternate in-memory computation of lineage has improved performance by many > folds (~90% improvement). This Jira is about adding alternate in-memory > computation of lineage. > *"atlas.use.graph.query.for.lineage"* property can be used to toggle between > graph query and in-memory computation of lineage. The default option will be > in-memory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3558) Improve lineage performance using in-memory traversal
[ https://issues.apache.org/jira/browse/ATLAS-3558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3558: -- Attachment: (was: lineage1.png) > Improve lineage performance using in-memory traversal > - > > Key: ATLAS-3558 > URL: https://issues.apache.org/jira/browse/ATLAS-3558 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0 > Reporter: Sarath Subramanian > Assignee: Sarath Subramanian >Priority: Major > Fix For: 2.1.0 > > Attachments: image-2019-12-11-15-54-30-483.png > > > Lineage in atlas uses graph query to compute lineage across entities (inputs, > outputs or both). Lineage rendering performance have degraded after using > janusgraph version 0.4.0 > On investigation, lineage graph query initialization and execution using > gremlin script engine has been found to be the bottleneck. > Alternate in-memory computation of lineage has improved performance by many > folds (~90% improvement). This Jira is about adding alternate in-memory > computation of lineage. > *"atlas.use.graph.query.for.lineage"* property can be used to toggle between > graph query and in-memory computation of lineage. The default option will be > in-memory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3558) Improve lineage performance using in-memory traversal
[ https://issues.apache.org/jira/browse/ATLAS-3558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3558: -- Attachment: image-2019-12-11-15-54-30-483.png > Improve lineage performance using in-memory traversal > - > > Key: ATLAS-3558 > URL: https://issues.apache.org/jira/browse/ATLAS-3558 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0 > Reporter: Sarath Subramanian > Assignee: Sarath Subramanian >Priority: Major > Fix For: 2.1.0 > > Attachments: image-2019-12-11-15-54-30-483.png > > > Lineage in atlas uses graph query to compute lineage across entities (inputs, > outputs or both). Lineage rendering performance have degraded after using > janusgraph version 0.4.0 > On investigation, lineage graph query initialization and execution using > gremlin script engine has been found to be the bottleneck. > Alternate in-memory computation of lineage has improved performance by many > folds (~90% improvement). This Jira is about adding alternate in-memory > computation of lineage. > *"atlas.use.graph.query.for.lineage"* property can be used to toggle between > graph query and in-memory computation of lineage. The default option will be > in-memory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3558) Improve lineage performance using in-memory traversal
Sarath Subramanian created ATLAS-3558: - Summary: Improve lineage performance using in-memory traversal Key: ATLAS-3558 URL: https://issues.apache.org/jira/browse/ATLAS-3558 Project: Atlas Issue Type: Improvement Components: atlas-core Affects Versions: 2.0.0 Reporter: Sarath Subramanian Assignee: Sarath Subramanian Fix For: 2.1.0 Lineage in atlas uses graph query to compute lineage across entities (inputs, outputs or both). Lineage rendering performance have degraded after using janusgraph version 0.4.0 On investigation, lineage graph query initialization and execution using gremlin script engine has been found to be the bottleneck. Alternate in-memory computation of lineage has improved performance by many folds (~90% improvement). This Jira is about adding alternate in-memory computation of lineage. *"atlas.use.graph.query.for.lineage"* property can be used to toggle between graph query and in-memory computation of lineage. The default option will be in-memory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DRAFT] Board report for Apache Atlas: December 2019
Thanks for compiling the board report. +1 On Tue, Dec 10, 2019 at 5:57 PM Madhan Neethiraj wrote: > Atlas team, > > > > Please review the draft board report below and send your feedback/comments. > > > > Thanks, > > Madhan > > > > > > ## Description: > > Apache Atlas is a scalable and extensible set of core foundational > > governance services that enables enterprises to effectively and > efficiently > > meet their compliance requirements within Hadoop and allows integration > with > > the complete enterprise data ecosystem > > > > ## Issues: > > There are no issues requiring board attention at this time. > > > > ## Activity: > > - enhancements to support user-defined attributes in entity instances > > - enhancements to support labels to entity instances > > - search enhancements to find entities with multiple classifications, > by labels > > - added an option to authorize notifications using the username in the > message > > - added REST APIs to: retrieve entity-header given its unique > attributes, purge deleted entities > > - upgraded JanusGraph version to the most recent release, 0.4.0 > > - improvements to stats/metrics API and UI to include JVM details > > - working on a new feature to introduce namespace attributes > > - working on search enhancements to support system-attributes > > > > ## Health report: > > - 1 new contributor added in last 3 months: leslizhang > > > > ## PMC changes: > > - Currently 34 PMC members > > - Last addition to PMC role was on 5/13/2019 > > > > ## Committer base changes: > > - Currently 38 committers > > - No new committers added in last 3 months > > - Last addition to committer role was on 10/15/2018 > > > > ## Releases: > > 0.8.4was released on 06/21/2019 > > 1.2.0was released on 06/12/2019 > > 2.0.0was released on 05/14/2019 > > 0.8.3was released on 10/31/2018 > > 1.1.0was released on 09/17/2018 > > 1.0.0was released on 06/02/2018 > > 0.8.2was released on 02/05/2018 > > 1.0.0-alpha was released on 01/25/2018 > > 0.8.1was released on 08/29/2017 > > 0.8-incubating was released on 03/16/2017 > > 0.7.1-incubating was released on 01/26/2017 > > 0.7-incubating was released on 07/09/2016 > > 0.6-incubating was released on 12/31/2015 > > 0.5-incubating was released on 07/11/2015 > > > >
[jira] [Commented] (ATLAS-3550) Add authorize check for partialUpdateEntityAttrByGuid API call
[ https://issues.apache.org/jira/browse/ATLAS-3550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989428#comment-16989428 ] Sarath Subramanian commented on ATLAS-3550: --- Thanks for the patch. +1 > Add authorize check for partialUpdateEntityAttrByGuid API call > --- > > Key: ATLAS-3550 > URL: https://issues.apache.org/jira/browse/ATLAS-3550 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Nixon Rodrigues >Assignee: Nixon Rodrigues >Priority: Minor > Fix For: 2.1.0 > > Attachments: > 0001-ATLAS-3550-Add-authorize-check-for-partialUpdateEnti.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (ATLAS-3543) UI: Support basic search to search with System attributes only for entity.
[ https://issues.apache.org/jira/browse/ATLAS-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian closed ATLAS-3543. - > UI: Support basic search to search with System attributes only for entity. > --- > > Key: ATLAS-3543 > URL: https://issues.apache.org/jira/browse/ATLAS-3543 > Project: Atlas > Issue Type: Improvement >Reporter: Keval Bhatt >Priority: Major > > Support basic search to search with System attributes only for entity: > __guid, __historicalGuids, __labels, __customAttributes, > __classificationsText, > __classificationNames, __propagatedClassificationNames, __isIncomplete -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ATLAS-3549) Add a new REST endpoint to get EntityHeader using unique attributes
[ https://issues.apache.org/jira/browse/ATLAS-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian resolved ATLAS-3549. --- Resolution: Fixed > Add a new REST endpoint to get EntityHeader using unique attributes > --- > > Key: ATLAS-3549 > URL: https://issues.apache.org/jira/browse/ATLAS-3549 > Project: Atlas > Issue Type: Bug >Reporter: Mandar Ambawane >Assignee: Mandar Ambawane >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 71883: ATLAS-3549: Add a new REST endpoint to get EntityHeader using unique attributes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71883/#review218951 --- Ship it! Ship It! - Sarath Subramanian On Dec. 5, 2019, 6:22 a.m., Mandar Ambawane wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71883/ > --- > > (Updated Dec. 5, 2019, 6:22 a.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Nixon Rodrigues, > and Sarath Subramanian. > > > Bugs: ATLAS-3549 > https://issues.apache.org/jira/browse/ATLAS-3549 > > > Repository: atlas > > > Description > --- > > This patch address requirement to add new api to get AtlasEntityHeader by > unique attributes. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java > 49dd5c5 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java > c8e65ef > webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 33f4828 > webapp/src/test/java/org/apache/atlas/web/adapters/TestEntityREST.java > b747124 > > > Diff: https://reviews.apache.org/r/71883/diff/1/ > > > Testing > --- > > 1.Basic testing done. > > 2.Test cases added in TestEntityREST > > 3.https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1578/ > > > Thanks, > > Mandar Ambawane > >
[jira] [Updated] (ATLAS-3545) NullPointerException while trying to delete classification
[ https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3545: -- Attachment: ATLAS-3545.002.patch > NullPointerException while trying to delete classification > -- > > Key: ATLAS-3545 > URL: https://issues.apache.org/jira/browse/ATLAS-3545 > Project: Atlas > Issue Type: Bug >Affects Versions: trunk >Reporter: Bolke de Bruin > Assignee: Sarath Subramanian >Priority: Critical > Attachments: ATLAS-3545.002.patch, Screenshot 2019-11-30 at > 22.16.44.png, Screenshot 2019-11-30 at 22.28.00.png, Screenshot 2019-12-02 at > 21.43.29.png > > > We see an issue where there is a NullPointerException while trying to delete > a classification that is propagating. > It seems (stack trace is as of yet unavailable) that it is caused due to a > typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see > screenshots. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3545) NullPointerException while trying to delete classification
[ https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3545: -- Attachment: (was: ATLAS-3545.001.patch) > NullPointerException while trying to delete classification > -- > > Key: ATLAS-3545 > URL: https://issues.apache.org/jira/browse/ATLAS-3545 > Project: Atlas > Issue Type: Bug >Affects Versions: trunk >Reporter: Bolke de Bruin > Assignee: Sarath Subramanian >Priority: Critical > Attachments: Screenshot 2019-11-30 at 22.16.44.png, Screenshot > 2019-11-30 at 22.28.00.png, Screenshot 2019-12-02 at 21.43.29.png > > > We see an issue where there is a NullPointerException while trying to delete > a classification that is propagating. > It seems (stack trace is as of yet unavailable) that it is caused due to a > typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see > screenshots. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 71870: ATLAS-3545: NullPointerException while trying to delete classification
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71870/ --- (Updated Dec. 4, 2019, 8:17 p.m.) Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, Madhan Neethiraj, Mandar Ambawane, Nikhil Bonte, Nixon Rodrigues, Sameer Shaikh, and Sidharth Mishra. Bugs: ATLAS-3545 https://issues.apache.org/jira/browse/ATLAS-3545 Repository: atlas Description --- Corrupt/Null classification vertices is causing NPE during entity search and retrieval. Null checks have been added to avoid NPE and logging added to identify potential corrupt classification vertex. Diffs (updated) - repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java a114d252e repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 6b111732a Diff: https://reviews.apache.org/r/71870/diff/2/ Changes: https://reviews.apache.org/r/71870/diff/1-2/ Testing --- Precommit: https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1575/ Manually verified that null/corrupt classification vertices are ignored during search and entity retrieval. Approproate WARN messages are printed in the log file: *2019-12-03 23:26:45,243 WARN - [pool-2-thread-10 - 305383de-19f2-4f93-89db-3b54db870c3f:] ~ Ignoring invalid classification vertex: vertex[id=81932528] (EntityGraphRetriever:284)* Thanks, Sarath Subramanian
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71791/#review218927 --- client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java Lines 598 (patched) <https://reviews.apache.org/r/71791/#comment306863> 'GET_ATLAS_AUDIT' => 'GET_ATLAS_AUDITS' intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java Lines 56 (patched) <https://reviews.apache.org/r/71791/#comment306860> unused constructor. consider removing it. intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java Lines 124 (patched) <https://reviews.apache.org/r/71791/#comment306859> define equals() and hashCode() method as well. intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java Lines 125 (patched) <https://reviews.apache.org/r/71791/#comment306861> incorrect toString() implementation. please format in alignment with other model classes. intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java Lines 122 (patched) <https://reviews.apache.org/r/71791/#comment306868> assign ret to null and handle null in callers. intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java Lines 128 (patched) <https://reviews.apache.org/r/71791/#comment306867> => StringUtils.join(entityIds, ",")) intg/src/main/java/org/apache/atlas/model/typedef/AtlasBaseTypeDef.java Line 143 (original), 143 (patched) <https://reviews.apache.org/r/71791/#comment306862> revert if change not intended. repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java Lines 76 (patched) <https://reviews.apache.org/r/71791/#comment306857> move debug log to beginning of method. repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java Lines 92 (patched) <https://reviews.apache.org/r/71791/#comment306858> avoid too many ternary operations in a single line. add null checks/default values inside getSearchCriteria() method. repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java Lines 272 (patched) <https://reviews.apache.org/r/71791/#comment306864> nit: revert new line change repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java Lines 1050 (patched) <https://reviews.apache.org/r/71791/#comment306866> consider refactoring this method webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java Lines 437 (patched) <https://reviews.apache.org/r/71791/#comment306855> shouldn't purge events be audited only on successful purge on entities? consider checking EntityMutationResponse for any purge events present and then audit them. webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java Lines 438 (patched) <https://reviews.apache.org/r/71791/#comment306856> 'addAtlasAudit()' => 'add()' 'startTimestamp' can be got from RequestContext.get().getRequestTime() - Sarath Subramanian On Dec. 4, 2019, 1:57 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71791/ > ------- > > (Updated Dec. 4, 2019, 1:57 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3518 > https://issues.apache.org/jira/browse/ATLAS-3518 > > > Repository: atlas > > > Description > --- > > Right now for purge, delete entity, etc. at Atlas we create audit entry at > HBase. User can go to entity and click on audit to see the audit information > for that particular entity. But if user purge one entity there will not be > any vertex at Janus graph and user will not able to get the purge audit which > is important for governance purpose. > > As part of fix, implemented Atlas Audit Framework which is being used for > Purge operation and can be extended for other operations. In Future once we > change the AtlasAudit type as super type for Import export as well we can use > this framework for all Atlas operations. > > admin/audit rest has been added which will return all the Atlas audit > operations including import export at Atlas. > > > Diffs > - > > addons/models/-Area0/0010-base_model.json 2f5fdaf14 > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > 8c0a6407c > dashboardv2/public/js/utils/Enums.js 58542ec59 > dashboardv3/public/js/utils/Enums.js
[jira] [Commented] (ATLAS-3545) NullPointerException while trying to delete classification
[ https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16987617#comment-16987617 ] Sarath Subramanian commented on ATLAS-3545: --- [~bolke], while we continue to investigate the cause of corrupt classification vertex being created (due to race condition). Attached patch should avoid NPE and print the corrupt classification vertex id in the logs and continue entity search/retrieval. > NullPointerException while trying to delete classification > -- > > Key: ATLAS-3545 > URL: https://issues.apache.org/jira/browse/ATLAS-3545 > Project: Atlas > Issue Type: Bug >Affects Versions: trunk >Reporter: Bolke de Bruin > Assignee: Sarath Subramanian >Priority: Critical > Attachments: ATLAS-3545.001.patch, Screenshot 2019-11-30 at > 22.16.44.png, Screenshot 2019-11-30 at 22.28.00.png, Screenshot 2019-12-02 at > 21.43.29.png > > > We see an issue where there is a NullPointerException while trying to delete > a classification that is propagating. > It seems (stack trace is as of yet unavailable) that it is caused due to a > typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see > screenshots. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3545) NullPointerException while trying to delete classification
[ https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3545: -- Attachment: ATLAS-3545.001.patch > NullPointerException while trying to delete classification > -- > > Key: ATLAS-3545 > URL: https://issues.apache.org/jira/browse/ATLAS-3545 > Project: Atlas > Issue Type: Bug >Affects Versions: trunk >Reporter: Bolke de Bruin > Assignee: Sarath Subramanian >Priority: Critical > Attachments: ATLAS-3545.001.patch, Screenshot 2019-11-30 at > 22.16.44.png, Screenshot 2019-11-30 at 22.28.00.png, Screenshot 2019-12-02 at > 21.43.29.png > > > We see an issue where there is a NullPointerException while trying to delete > a classification that is propagating. > It seems (stack trace is as of yet unavailable) that it is caused due to a > typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see > screenshots. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
Review Request 71870: ATLAS-3545: NullPointerException while trying to delete classification
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71870/ --- Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, Madhan Neethiraj, Mandar Ambawane, Nikhil Bonte, Nixon Rodrigues, Sameer Shaikh, and Sidharth Mishra. Bugs: ATLAS-3545 https://issues.apache.org/jira/browse/ATLAS-3545 Repository: atlas Description --- Corrupt/Null classification vertices is causing NPE during entity search and retrieval. Null checks have been added to avoid NPE and logging added to identify potential corrupt classification vertex. Diffs - repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java a114d252e repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 6b111732a Diff: https://reviews.apache.org/r/71870/diff/1/ Testing --- Precommit: https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1575/ Manually verified that null/corrupt classification vertices are ignored during search and entity retrieval. Approproate WARN messages are printed in the log file: *2019-12-03 23:26:45,243 WARN - [pool-2-thread-10 - 305383de-19f2-4f93-89db-3b54db870c3f:] ~ Ignoring invalid classification vertex: vertex[id=81932528] (EntityGraphRetriever:284)* Thanks, Sarath Subramanian
[jira] [Commented] (ATLAS-3545) NullPointerException while trying to delete classification
[ https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16987301#comment-16987301 ] Sarath Subramanian commented on ATLAS-3545: --- [~bolke] thanks for the repro steps, this is useful. I will investigate this issue. > NullPointerException while trying to delete classification > -- > > Key: ATLAS-3545 > URL: https://issues.apache.org/jira/browse/ATLAS-3545 > Project: Atlas > Issue Type: Bug >Affects Versions: trunk >Reporter: Bolke de Bruin > Assignee: Sarath Subramanian >Priority: Critical > Attachments: Screenshot 2019-11-30 at 22.16.44.png, Screenshot > 2019-11-30 at 22.28.00.png, Screenshot 2019-12-02 at 21.43.29.png > > > We see an issue where there is a NullPointerException while trying to delete > a classification that is propagating. > It seems (stack trace is as of yet unavailable) that it is caused due to a > typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see > screenshots. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ATLAS-3545) NullPointerException while trying to delete classification
[ https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian reassigned ATLAS-3545: - Assignee: Sarath Subramanian > NullPointerException while trying to delete classification > -- > > Key: ATLAS-3545 > URL: https://issues.apache.org/jira/browse/ATLAS-3545 > Project: Atlas > Issue Type: Bug >Affects Versions: trunk >Reporter: Bolke de Bruin > Assignee: Sarath Subramanian >Priority: Critical > Attachments: Screenshot 2019-11-30 at 22.16.44.png, Screenshot > 2019-11-30 at 22.28.00.png, Screenshot 2019-12-02 at 21.43.29.png > > > We see an issue where there is a NullPointerException while trying to delete > a classification that is propagating. > It seems (stack trace is as of yet unavailable) that it is caused due to a > typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see > screenshots. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 71818: ATLAS-3534 Changes to allow namespaces to be added to an entity instance
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71818/#review218895 --- Patch fails to apply in apache master. can you please rebase. - Sarath Subramanian On Nov. 25, 2019, 6:04 p.m., Aadarsh Jajodia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71818/ > --- > > (Updated Nov. 25, 2019, 6:04 p.m.) > > > Review request for atlas, Ashutosh Mestry, Sridhar K, Le Ma, Madhan > Neethiraj, and Sarath Subramanian. > > > Bugs: ATLAS-3534 > https://issues.apache.org/jira/browse/ATLAS-3534 > > > Repository: atlas > > > Description > --- > > The changes made in these include adding/updating namespaces to an entity > instance > as part of create or update > > > Diffs > - > > common/src/main/java/org/apache/atlas/repository/Constants.java > 0b28243c47d9e025ecc38a05ce1a54efd71e6149 > intg/src/main/java/org/apache/atlas/AtlasErrorCode.java > 7a2aae2e9ae8174c5309164f3e41c940cbf3ddf8 > intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java > 1b033b9ac0d413c04269b480562940fb5b305939 > intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java > 557ef74a95c2a939b4b89cd1db8fa4c73d52dd51 > intg/src/main/java/org/apache/atlas/type/AtlasNamespaceType.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java > 1e7acf1e728f472a01a26fe4be69a11e21ca3e77 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityGraphDiscoveryV2.java > d3d9367fb158f2bafbcc44f3c25ee1a63c5c2aa7 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java > c8e65ef15fcf38cb4b5f1d9f1f37cf1578c13b48 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > a114d252e7462a08c14ebef598ed13b6b43e247b > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java > 6b111732a11b2aef7dbeb050087bde3c70fd21bd > > > Diff: https://reviews.apache.org/r/71818/diff/1/ > > > Testing > --- > > > Thanks, > > Aadarsh Jajodia > >
[jira] [Assigned] (ATLAS-3254) Atlas entity with large array of refs causes performance issues for lineage
[ https://issues.apache.org/jira/browse/ATLAS-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian reassigned ATLAS-3254: - Assignee: Sarath Subramanian > Atlas entity with large array of refs causes performance issues for lineage > --- > > Key: ATLAS-3254 > URL: https://issues.apache.org/jira/browse/ATLAS-3254 > Project: Atlas > Issue Type: Bug > Components: atlas-core, atlas-webui >Affects Versions: 1.0.0, 2.0.0 >Reporter: Adam Rempter > Assignee: Sarath Subramanian >Priority: Major > Labels: performance > Attachments: Screenshot 2019-11-28 at 21.18.44.png, > example_create_entities.json, rest_entity_get_pseudodir.json > > > We use “aws_s3_pseudo_dir” type from 3020-aws_s3_typedefs.json model. > It has following property: > "name": "s3Objects", > "typeName": "array" > > Now in AWS buckets you can have thousands of objects. This causes that > s3Objects array grows quite quickly, causing aws_s3_pseudo_dir entity Json to > rich easly few MBs. > > Then we start seeing problems like: > * UI is dying on displaying entity properties or lineage > * Error in logs: audit record too long: entityType=aws_s3_pseudo_dir, > guid=24398271-6ba0-4db5-adfa-38e432dc55ce, size=1053931; maxSize=1048576. > entity attribute values not stored in audit (EntityAuditListenerV2:234) > * Some errors with write to HBase (java.lang.IllegalArgumentException: > KeyValue size too large, as workaround we set hbase.client.keyvalue.maxsize > param to 0) > * kafka consumer errors (we can of course set some parameters on consumer, > but I think it is just workaround) > … > Exception in NotificationHookConsumer (NotificationHookConsumer:332) > org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be > completed since the group has already rebalanced and assigned the partitions > to another member. This means that the time between subsequen > t calls to poll() was longer than the configured max.poll.interval.ms, which > typically implies that the poll loop is spending too much time message > processing. You can address this either by increasing the sessio > n timeout or by reducing the maximum size of batches returned in poll() with > max.poll.records. > … > Specifying pseudo_dir is required for s3objects: > name": "pseudoDirectory", > "typeName": "aws_s3_pseudo_dir", > "cardinality": "SINGLE", > "isIndexable": false, > *"isOptional": false,* > "isUnique": false, > -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71649/#review218869
---
intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java
Lines 75 (patched)
<https://reviews.apache.org/r/71649/#comment306778>
this check is incorrect - adding null/empty 'attributeDefs' will reset the
attributeDefs to emplty list. (old entries are removed). Please review
if (CollectionUtils.isNotEmpty(attributeDefs)) {
this.attributeDefs.addAll(attributeDefs);
}
intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java
Lines 162 (patched)
<https://reviews.apache.org/r/71649/#comment306779>
'maxStrLength' and 'validPattern' are string attribute specific
constraints, consider adding this constraints in AtlasAttributeDef.options map
(define allowed string constraint enums). This will be extensible to add any
condition in the future.
intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
Lines 49 (patched)
<https://reviews.apache.org/r/71649/#comment306776>
nit: remove new line
intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
Lines 1000 (patched)
<https://reviews.apache.org/r/71649/#comment306777>
consider refactoring this method:
private void addNamespaceAttribute(AtlasNamespaceAttribute nsAttribute) {
String nsName = nsAttribute.getDefinedInType().getTypeName();
if (!namespaceAttributes.containsKey(nsName)) {
namespaceAttributes.put(nsName, new ArrayList<>());
}
namespaceAttributes.get(nsName).add(nsAttribute);
}
intg/src/main/java/org/apache/atlas/type/AtlasNamespaceType.java
Lines 31 (patched)
<https://reviews.apache.org/r/71649/#comment306782>
nit: remove unused import.
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
Lines 586 (patched)
<https://reviews.apache.org/r/71649/#comment306783>
this method is similar to method in line 505. consider refactoring to a
single method and change behavior for AtlasNamespaceDef and AtlasStructDef
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasAbstractDefStoreV2.java
Lines 67 (patched)
<https://reviews.apache.org/r/71649/#comment306775>
consider refactoring line 67-81 to avoid duplicates:
if (typeDef instanceof AtlasStructDef) {
for (AtlasStructDef.AtlasAttributeDef attrDef : ((AtlasStructDef)
typeDef).getAttributeDefs()) {
validateAttributeName(typeDef, attrDef.getName());
}
} else if (typeDef instanceof AtlasNamespaceDef) {
for (AtlasNamespaceDef.AtlasNamespaceAttributeDef attrDef :
((AtlasNamespaceDef) typeDef).getAttributeDefs()) {
validateAttributeName(typeDef, attrDef.getName());
}
}
private void validateAttributeName(AtlasBaseTypeDef typeDef, String
attributeName) throws AtlasBaseException {
if (AtlasDSL.Parser.isKeyword(attributeName)) {
throw new AtlasBaseException(AtlasErrorCode.ATTRIBUTE_NAME_INVALID,
attributeName, typeDef.getCategory().name());
}
}
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasAbstractDefStoreV2.java
Lines 93 (patched)
<https://reviews.apache.org/r/71649/#comment306774>
nit: remove new line
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 370 (patched)
<https://reviews.apache.org/r/71649/#comment306780>
encodedtypeNamePropertyKey => encodedTypeNamePropertyKey
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2Test.java
Lines 32 (patched)
<https://reviews.apache.org/r/71649/#comment306781>
nit: remove unused imports in line 32-33
- Sarath Subramanian
On Nov. 21, 2019, 6:03 p.m., Aadarsh Jajodia wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71649/
> ---
>
> (Updated Nov. 21, 2019, 6:03 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Sridhar K, Le Ma, Madhan
> Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3486
> https://issues.apache.org/jira/browse/ATLAS-3486
>
>
> Repository: atlas
>
>
> Description
> ---
>
> This change is the first part of the bigger task defined as part of
> ATLAS-3485.
> This adds the data model needed for supporting namespaces and namespace
> attributes and also updates the type registry to include the applicable
> namespace attributes for every entity type
>
>
Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71649/#review218868 --- Patch fails to apply, can you please rebase. - Sarath Subramanian On Nov. 21, 2019, 6:03 p.m., Aadarsh Jajodia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71649/ > --- > > (Updated Nov. 21, 2019, 6:03 p.m.) > > > Review request for atlas, Ashutosh Mestry, Sridhar K, Le Ma, Madhan > Neethiraj, and Sarath Subramanian. > > > Bugs: ATLAS-3486 > https://issues.apache.org/jira/browse/ATLAS-3486 > > > Repository: atlas > > > Description > --- > > This change is the first part of the bigger task defined as part of > ATLAS-3485. > This adds the data model needed for supporting namespaces and namespace > attributes and also updates the type registry to include the applicable > namespace attributes for every entity type > > > Diffs > - > > intg/src/main/java/org/apache/atlas/AtlasErrorCode.java > 7a2aae2e9ae8174c5309164f3e41c940cbf3ddf8 > intg/src/main/java/org/apache/atlas/model/TypeCategory.java > f06f64f450f407e3f9a0e742726ff4dd12ccc695 > intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java > PRE-CREATION > intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java > bb7ead0f9f8bab3094eb82e9e286dd58e8a6e3de > intg/src/main/java/org/apache/atlas/model/typedef/AtlasTypesDef.java > 3634fdfd313639eb97b3c4698e091487b0e44a80 > intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java > 4ee68a936f99bb4c819b5335da2cc8bf7d539397 > intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java > 557ef74a95c2a939b4b89cd1db8fa4c73d52dd51 > intg/src/main/java/org/apache/atlas/type/AtlasNamespaceType.java > PRE-CREATION > intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java > b071dc9d664cee9e1ffc54726ffbf15f4f602d30 > intg/src/main/java/org/apache/atlas/type/AtlasTypeUtil.java > 0883d54f490e22c6510e6fc0cb804b87713a7ecb > intg/src/main/java/org/apache/atlas/typesystem/types/DataTypes.java > dba2d88146eff314191ae6bb24ad7337b0ea10ae > intg/src/test/java/org/apache/atlas/TestRelationshipUtilsV2.java > 02613b5f7250b14324ed294c22de079b74d55b08 > intg/src/test/java/org/apache/atlas/TestUtilsV2.java > ff79994c519702e90b2e478d00cae0008889f956 > > intg/src/test/java/org/apache/atlas/model/typedef/TestAtlasNamespaceDef.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > 7c551304b2b65b90302f6e5fa9fc5b9f1b8e2c12 > > repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java > 2e2ab1a664171555c57560e1c0b4cbdbc20c0f6f > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasAbstractDefStoreV2.java > 2cb2b47898ded4d6c5c84800ff93fa58b2c480da > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasTypeDefGraphStoreV2.java > a5ccfb5b2055c88f596312f4033bc0034d3d165c > > repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2Test.java > PRE-CREATION > webapp/src/main/java/org/apache/atlas/examples/QuickStartV2.java > 6cd0ee331b7ae24757b58e76ec47bf556106846a > webapp/src/main/java/org/apache/atlas/web/rest/TypesREST.java > fb56fad6412079e20dd3e345b81a08d9e5ace657 > > > Diff: https://reviews.apache.org/r/71649/diff/8/ > > > Testing > --- > > Added unit tests > > > Thanks, > > Aadarsh Jajodia > >
Re: Review Request 71848: ATLAS-3544: fix NPE during entity-delete
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71848/#review218862 --- Ship it! Ship It! - Sarath Subramanian On Nov. 30, 2019, 10:20 a.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71848/ > --- > > (Updated Nov. 30, 2019, 10:20 a.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Le Ma, Nixon > Rodrigues, Sarath Subramanian, and Sidharth Mishra. > > > Bugs: ATLAS-3544 > https://issues.apache.org/jira/browse/ATLAS-3544 > > > Repository: atlas > > > Description > --- > > fixed NPE caused by incorrect condition check > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > 621458406 > > > Diff: https://reviews.apache.org/r/71848/diff/1/ > > > Testing > --- > > - pre-commit tests run: > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1573/ > - verified manually > > > Thanks, > > Madhan Neethiraj > >
[jira] [Updated] (ATLAS-3457) Support Multi-Classification search without attributes
[ https://issues.apache.org/jira/browse/ATLAS-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3457: -- Fix Version/s: 2.1.0 > Support Multi-Classification search without attributes > -- > > Key: ATLAS-3457 > URL: https://issues.apache.org/jira/browse/ATLAS-3457 > Project: Atlas > Issue Type: New Feature > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Le Ma >Assignee: Le Ma >Priority: Major > Fix For: 2.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ATLAS-3457) Support Multi-Classification search without attributes
[ https://issues.apache.org/jira/browse/ATLAS-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian resolved ATLAS-3457. --- Resolution: Fixed > Support Multi-Classification search without attributes > -- > > Key: ATLAS-3457 > URL: https://issues.apache.org/jira/browse/ATLAS-3457 > Project: Atlas > Issue Type: New Feature >Reporter: Le Ma >Assignee: Le Ma >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3457) Support Multi-Classification search without attributes
[ https://issues.apache.org/jira/browse/ATLAS-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3457: -- Affects Version/s: 2.0.0 > Support Multi-Classification search without attributes > -- > > Key: ATLAS-3457 > URL: https://issues.apache.org/jira/browse/ATLAS-3457 > Project: Atlas > Issue Type: New Feature > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Le Ma >Assignee: Le Ma >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3457) Support Multi-Classification search without attributes
[ https://issues.apache.org/jira/browse/ATLAS-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3457: -- Component/s: atlas-core > Support Multi-Classification search without attributes > -- > > Key: ATLAS-3457 > URL: https://issues.apache.org/jira/browse/ATLAS-3457 > Project: Atlas > Issue Type: New Feature > Components: atlas-core >Reporter: Le Ma >Assignee: Le Ma >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71649/#review218727
---
I don't see namespace attribuites indexed. This should be indexed.
Check GraphBackSearchIndexer.addIndexForType() method
intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java
Lines 174 (patched)
<https://reviews.apache.org/r/71649/#comment306592>
removed unused constructors in line 174, 179, 193 and 203
intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java
Lines 162 (patched)
<https://reviews.apache.org/r/71649/#comment306600>
'maxStrLength' and 'validPattern' are string specific attribute options.
Consider adding a enum - namespaceAttributeOptions {MAX_STRING_LENGTH,
VALID_PATTERN} and use 'options' map in AtlasAttributeDef to set the namespace
attribute options. This will be easier to extend for any int, date options as
well.
intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java
Lines 83 (patched)
<https://reviews.apache.org/r/71649/#comment306617>
add REST endpoints in TypesREST for:
/namespacedef/name/{name}
/namespacedef/guid/{guid}
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 22 (patched)
<https://reviews.apache.org/r/71649/#comment306601>
nit: remove unused imports in line 22,42
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 59 (patched)
<https://reviews.apache.org/r/71649/#comment306604>
validateType() checks for invalid types only for structDef attruibutes, add
check in validateType() to check for namespace attributes as well.
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 133 (patched)
<https://reviews.apache.org/r/71649/#comment306605>
is this line needed? please review.
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 226 (patched)
<https://reviews.apache.org/r/71649/#comment306613>
"update classification-def " => "update namespace-def "
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 260 (patched)
<https://reviews.apache.org/r/71649/#comment306614>
"delete struct-def " => "delete namespace-def "
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 297 (patched)
<https://reviews.apache.org/r/71649/#comment306610>
AtlasBaseException is never thrown
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 349 (patched)
<https://reviews.apache.org/r/71649/#comment306602>
'if' condition in line 349 and 353 is duplicate. please review.
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 357 (patched)
<https://reviews.apache.org/r/71649/#comment306603>
check for null as well for 'applicableEntityTypes'. =>
CollectionUtils.isNotEmpty(attributeDef.getApplicableEntityTypes())
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 382 (patched)
<https://reviews.apache.org/r/71649/#comment306612>
=> CollectionUtils.isNotEmpty(attributeDef.getApplicableEntityTypes())
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 470 (patched)
<https://reviews.apache.org/r/71649/#comment306606>
refactor to:
if (!currAttrNames.containsAll(attrNames)) {
//throw exception
}
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 496 (patched)
<https://reviews.apache.org/r/71649/#comment306607>
check for null attributeDef.getApplicableEntityTypes() as well ?
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 515 (patched)
<https://reviews.apache.org/r/71649/#comment306608>
=>
if(!updatedApplicableEntityTypes.containsAll(existingAttribute.getApplicableEntityTypes()))
{
// throw exception
}
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 526 (patched)
<https://reviews.apache.org/r/71649/#comment306609>
set using encoded property key here:
AtlasGraphUtilsV2.encodePropertyKey(propertyKey);
repository/src/test/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStoreTest.java
Line 694 (original), 694 (patched)
<https://reviews.apache.org/r/71649/#comment306616>
c
[jira] [Updated] (ATLAS-3519) Reuse script engine to improve lineage query performance
[ https://issues.apache.org/jira/browse/ATLAS-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3519: -- Summary: Reuse script engine to improve lineage query performance (was: pool performance in hive lineage query ) > Reuse script engine to improve lineage query performance > > > Key: ATLAS-3519 > URL: https://issues.apache.org/jira/browse/ATLAS-3519 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 1.1.0, 2.0.0 >Reporter: zhangxiong >Assignee: zhangxiong >Priority: Minor > Labels: lineage, performance > Fix For: 2.1.0 > > Attachments: ATLAS-3519.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > when query lineage info for hive, the throughput reachs only 3 query per > second. > we use jstack and jfr+flame graph tools and found that the > GremlinGroovyScriptEngine object loading class logic takes much of the time. > so we make the GremlinGroovyScriptEngine a global singleton object, which > would be shared accross all query threads. finally the query performance > improved much and got 200 qps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3519) pool performance in hive lineage query
[ https://issues.apache.org/jira/browse/ATLAS-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979569#comment-16979569 ] Sarath Subramanian commented on ATLAS-3519: --- Thanks for the patch [~leslizhang]. +1 > pool performance in hive lineage query > --- > > Key: ATLAS-3519 > URL: https://issues.apache.org/jira/browse/ATLAS-3519 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 1.1.0, 2.0.0 >Reporter: zhangxiong >Assignee: zhangxiong >Priority: Minor > Labels: lineage, performance > Fix For: 2.1.0 > > Attachments: ATLAS-3519.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > when query lineage info for hive, the throughput reachs only 3 query per > second. > we use jstack and jfr+flame graph tools and found that the > GremlinGroovyScriptEngine object loading class logic takes much of the time. > so we make the GremlinGroovyScriptEngine a global singleton object, which > would be shared accross all query threads. finally the query performance > improved much and got 200 qps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 71795: DSL Queries: LIKE Clause with Special Characters
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71795/#review218738 --- Ship it! Ship It! - Sarath Subramanian On Nov. 20, 2019, 1:52 p.m., Ashutosh Mestry wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71795/ > --- > > (Updated Nov. 20, 2019, 1:52 p.m.) > > > Review request for atlas, Nikhil Bonte, Nixon Rodrigues, and Sarath > Subramanian. > > > Bugs: ATLAS-3520 > https://issues.apache.org/jira/browse/ATLAS-3520 > > > Repository: atlas > > > Description > --- > > **Background** > Please refer to the bug description for details. > > **Approach** > - Detect special chacters in the right-hand-side of LIKE clause. > - Use textRegex instead of textContainsRegex. > > > Diffs > - > > repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java > 294dc00e5 > repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java > 724ae9f7a > > repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java > ca32ffc99 > > > Diff: https://reviews.apache.org/r/71795/diff/1/ > > > Testing > --- > > **Unit tests** > - _DSLQueriesTest_ updated to handle this case. > - _GremlinQueryComposerTest_ updated to handle this case. > > **Functional tests** > Via UI. > > **Pre-commit** > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1545/ > > > Thanks, > > Ashutosh Mestry > >
Review Request 71796: ATLAS-3530: Retrieve entityDefinition for '_ALL_ENTITY_TYPES' in /api/atlas/v2/types/entitydef/name/{name}
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71796/ --- Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, Madhan Neethiraj, Mandar Ambawane, Nikhil Bonte, Nixon Rodrigues, Sameer Shaikh, and Sidharth Mishra. Bugs: ATLAS-3530 https://issues.apache.org/jira/browse/ATLAS-3530 Repository: atlas Description --- '_ALL_ENTITY_TYPES' is a new system type with all system attributes of an entity in ATLAS-3482. A new TypesREST endpoint to retrieve definition of '_ALL_ENTITY_TYPES' is necessary to render system attributes in UI. GET /api/atlas/v2/types/entitydef/name/ALL_ENTITY_TYPES should return entityDef. Diffs - intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 557ef74a9 repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java 2e2ab1a66 repository/src/test/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStoreTest.java 51dd16b85 Diff: https://reviews.apache.org/r/71796/diff/1/ Testing --- Precommit: https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/ Added unit test to check if 'ALL_ENTITY_TYPES' definition is returned correctly. Thanks, Sarath Subramanian
[jira] [Updated] (ATLAS-3530) Retrieve entityDefinition for '_ALL_ENTITY_TYPES' in /api/atlas/v2/types/entitydef/name/{name}
[
https://issues.apache.org/jira/browse/ATLAS-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-3530:
--
Attachment: 0001-ATLAS-3530-Retrieve-entityDefinition-for-_ALL_ENTITY.patch
> Retrieve entityDefinition for '_ALL_ENTITY_TYPES' in
> /api/atlas/v2/types/entitydef/name/{name}
> --
>
> Key: ATLAS-3530
> URL: https://issues.apache.org/jira/browse/ATLAS-3530
> Project: Atlas
> Issue Type: Improvement
> Components: atlas-core
>Affects Versions: 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 2.1.0
>
> Attachments:
> 0001-ATLAS-3530-Retrieve-entityDefinition-for-_ALL_ENTITY.patch
>
>
> '_ALL_ENTITY_TYPES' is a new system type with all system attributes of an
> entity in ATLAS-3482.
> A new TypesREST endpoint to retrieve definition of '_ALL_ENTITY_TYPES' is
> necessary to render system attributes in UI.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ATLAS-3530) Retrieve entityDefinition for '_ALL_ENTITY_TYPES' in /api/atlas/v2/types/entitydef/name/{name}
Sarath Subramanian created ATLAS-3530:
-
Summary: Retrieve entityDefinition for '_ALL_ENTITY_TYPES' in
/api/atlas/v2/types/entitydef/name/{name}
Key: ATLAS-3530
URL: https://issues.apache.org/jira/browse/ATLAS-3530
Project: Atlas
Issue Type: Improvement
Components: atlas-core
Affects Versions: 2.0.0
Reporter: Sarath Subramanian
Assignee: Sarath Subramanian
Fix For: 2.1.0
'_ALL_ENTITY_TYPES' is a new system type with all system attributes of an
entity in ATLAS-3482.
A new TypesREST endpoint to retrieve definition of '_ALL_ENTITY_TYPES' is
necessary to render system attributes in UI.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-3519) pool performance in hive lineage query
[ https://issues.apache.org/jira/browse/ATLAS-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3519: -- Fix Version/s: 2.1.0 > pool performance in hive lineage query > --- > > Key: ATLAS-3519 > URL: https://issues.apache.org/jira/browse/ATLAS-3519 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 1.1.0, 2.0.0 >Reporter: zhangxiong >Assignee: zhangxiong >Priority: Minor > Labels: lineage, performance > Fix For: 2.1.0 > > Attachments: ATLAS-3519.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > when query lineage info for hive, the throughput reachs only 3 query per > second. > we use jstack and jfr+flame graph tools and found that the > GremlinGroovyScriptEngine object loading class logic takes much of the time. > so we make the GremlinGroovyScriptEngine a global singleton object, which > would be shared accross all query threads. finally the query performance > improved much and got 200 qps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3482) Enhance basic search to enable search on entity system attributes
[ https://issues.apache.org/jira/browse/ATLAS-3482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3482: -- Affects Version/s: 2.0.0 > Enhance basic search to enable search on entity system attributes > - > > Key: ATLAS-3482 > URL: https://issues.apache.org/jira/browse/ATLAS-3482 > Project: Atlas > Issue Type: New Feature > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Le Ma >Assignee: Le Ma >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3482) Enhance basic search to enable search on entity system attributes
[ https://issues.apache.org/jira/browse/ATLAS-3482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3482: -- Fix Version/s: 2.1.0 > Enhance basic search to enable search on entity system attributes > - > > Key: ATLAS-3482 > URL: https://issues.apache.org/jira/browse/ATLAS-3482 > Project: Atlas > Issue Type: New Feature > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Le Ma >Assignee: Le Ma >Priority: Major > Fix For: 2.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 71736: ATLAS-3482 Enhance basic search to enable search on entity system attributes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71736/#review218692 --- Ship it! Ship It! - Sarath Subramanian On Nov. 19, 2019, 12:04 p.m., Le Ma wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71736/ > --- > > (Updated Nov. 19, 2019, 12:04 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan > Neethiraj, and Sarath Subramanian. > > > Bugs: ATLAS-3482 > https://issues.apache.org/jira/browse/ATLAS-3482 > > > Repository: atlas > > > Description > --- > > Introduce ALL_ENTITY_TYPE to support search on system attributes across all > entity types. System attributes will be passed in as normal entity attributes. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java > 8f0e5912d > intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java > e10965b87 > intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java > 417194202 > intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 884447f81 > intg/src/main/java/org/apache/atlas/type/AtlasStructType.java e8bf7f9eb > intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 8b4fd1c3b > intg/src/main/java/org/apache/atlas/type/Constants.java PRE-CREATION > intg/src/test/java/org/apache/atlas/TestUtilsV2.java 530d5cda4 > intg/src/test/java/org/apache/atlas/type/TestAtlasRelationshipType.java > 107539598 > intg/src/test/java/org/apache/atlas/type/TestAtlasTypeRegistry.java > 476bc3300 > > repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java > 672f38132 > > repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java > 03eb92bcc > > repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java > 152ade8d4 > repository/src/main/java/org/apache/atlas/discovery/SearchContext.java > 7ad32bdb9 > repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java > b56d8e83a > webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 825cda30b > webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java > cd8f8981c > > > Diff: https://reviews.apache.org/r/71736/diff/5/ > > > Testing > --- > > - Added new unit tests, passed. > - pre-commit job succeeds. > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1540/ > > > Thanks, > > Le Ma > >
Re: Review Request 71736: ATLAS-3482 Enhance basic search to enable search on entity system attributes
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71736/#review218667
---
Fix it, then Ship it!
intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java
Line 45 (original), 54 (patched)
<https://reviews.apache.org/r/71736/#comment306527>
nit: revert whitespace changes from line 54-60
intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java
Lines 97 (patched)
<https://reviews.apache.org/r/71736/#comment306525>
nit: unused method; consider removing it.
intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java
Lines 515 (patched)
<https://reviews.apache.org/r/71736/#comment306524>
consider creating a constant for "__CLASSIFICATION_ROOT" in
AtlasClassificationType
private static final String CLASSIFICATION_ROOT_NAME =
"__CLASSIFICATION_ROOT";
intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
Lines 633 (patched)
<https://reviews.apache.org/r/71736/#comment306529>
SUPER_TYPES_PROPERTY_KEY, TRAIT_NAMES_PROPERTY_KEY and
PROPAGATED_TRAIT_NAMES_PROPERTY_KEY are array attributes not indexed in solr.
Do we need to include them in system attributes (since they cannot be index
searched)?
intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
Lines 648 (patched)
<https://reviews.apache.org/r/71736/#comment306528>
create static constant for "__ENTITY_ROOT"
intg/src/main/java/org/apache/atlas/type/AtlasStructType.java
Lines 937 (patched)
<https://reviews.apache.org/r/71736/#comment306530>
consider refactoring if condition check to a method:
if (isRootType(structDef)) { ... }
private static boolean isRootType(AtlasStructDef structDef) {
return StringUtils.equals(structDef.getName(),
AtlasEntityType.ENTITY_ROOT.getTypeName()) ||
StringUtils.equals(structDef.getName(),
AtlasClassificationType.CLASSIFICATION_ROOT.getTypeName());
}
same for line #948
intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java
Lines 63 (patched)
<https://reviews.apache.org/r/71736/#comment306531>
refactor line 63-64 and 74-75 to a mthod: resolveReferencesForRootTypes().
define getters in AtlasEntityType and AtlasClassificationType
resolveReferencesForRootTypes() {
AtlasEntityType.getEntityRoot().resolveReferences(this);
AtlasClassificationType.getClassificationRoot().resolveReferences(this);
}
repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java
Lines 108 (patched)
<https://reviews.apache.org/r/71736/#comment306532>
consider refactoring if condition to method:
if (!entityRootType()) {...}
repository/src/main/java/org/apache/atlas/discovery/SearchContext.java
Line 59 (original), 61 (patched)
<https://reviews.apache.org/r/71736/#comment306533>
avoid unnecessary whitespace/newline changes
line 61-75
line 83-90
repository/src/main/java/org/apache/atlas/discovery/SearchContext.java
Lines 80 (patched)
<https://reviews.apache.org/r/71736/#comment306534>
AtlasEntityType.ENTITY_ROOT => AtlasEntityType.getEntityRoot();
repository/src/main/java/org/apache/atlas/discovery/SearchContext.java
Lines 315 (patched)
<https://reviews.apache.org/r/71736/#comment306535>
entityName => entityTypeName
consider refactoring method with a single return:
return StringUtils.equals(entityName, ALL_ENTITY_TYPES) ?
MATCH_ALL_ENTITY_TYPES : typeRegistry.getEntityTypeByName(entityName);
- Sarath Subramanian
On Nov. 18, 2019, 2:30 p.m., Le Ma wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71736/
> ---
>
> (Updated Nov. 18, 2019, 2:30 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan
> Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3482
> https://issues.apache.org/jira/browse/ATLAS-3482
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Introduce ALL_ENTITY_TYPE to support search on system attributes across all
> entity types. System attributes will be passed in as normal entity attributes.
>
>
> Diffs
> -
>
> intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java
> 8f0e5912d
> intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java
> e10965b87
> intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java
> 417194202
> intg/src/main/java/org/apache/atlas/type/AtlasEntityType.jav
Re: Review Request 71736: ATLAS-3482 Enhance basic search to enable search on entity system attributes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71736/#review218642 --- intg/src/main/java/org/apache/atlas/type/AtlasBuiltInTypes.java Lines 53 (patched) <https://reviews.apache.org/r/71736/#comment306458> getType() is never used. please review. Also in line #274 and line #700 intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java Lines 102 (patched) <https://reviews.apache.org/r/71736/#comment306460> can we refactor to do resolveReferences() inside initBaseEntityType() method? => baseType = initBaseEntityType(typeRegistry); intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java Lines 650 (patched) <https://reviews.apache.org/r/71736/#comment306459> consider creating a constant string in AtlasStructType for "ALL_ENTITY_TYPES" - Sarath Subramanian On Nov. 14, 2019, 3:57 p.m., Le Ma wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71736/ > --- > > (Updated Nov. 14, 2019, 3:57 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Madhan > Neethiraj, and Sarath Subramanian. > > > Bugs: ATLAS-3482 > https://issues.apache.org/jira/browse/ATLAS-3482 > > > Repository: atlas > > > Description > --- > > Introduce ALL_ENTITY_TYPE to support search on system attributes across all > entity types. System attributes will be passed in as normal entity attributes. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java > 8f0e5912d > intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java > e10965b87 > intg/src/main/java/org/apache/atlas/type/AtlasBuiltInTypes.java ed1e5ded2 > intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java > 417194202 > intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 884447f81 > intg/src/main/java/org/apache/atlas/type/AtlasStructType.java e8bf7f9eb > intg/src/main/java/org/apache/atlas/type/Constants.java PRE-CREATION > > repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java > 672f38132 > > repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java > 03eb92bcc > > repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java > 152ade8d4 > repository/src/main/java/org/apache/atlas/discovery/SearchContext.java > 7ad32bdb9 > repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java > b56d8e83a > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > 7c551304b > webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 825cda30b > webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java > cd8f8981c > > > Diff: https://reviews.apache.org/r/71736/diff/3/ > > > Testing > --- > > - Added new unit tests, passed. > - will run pre-commit job. > > > Thanks, > > Le Ma > >
Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71649/#review218640 --- intg/src/main/java/org/apache/atlas/AtlasErrorCode.java Lines 163 (patched) <https://reviews.apache.org/r/71649/#comment306444> it will be useful to include namespace name and attribute name in the error message. intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java Lines 27 (patched) <https://reviews.apache.org/r/71649/#comment306443> nit: unused import intg/src/test/java/org/apache/atlas/TestUtilsV2.java Lines 1466 (patched) <https://reviews.apache.org/r/71649/#comment306457> defineNamespaceTypesDef() is never used. please review repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java Lines 559 (patched) <https://reviews.apache.org/r/71649/#comment306454> preDeleteByGuid() and preDeleteByName() may not be needed. Please review. repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java Lines 253 (patched) <https://reviews.apache.org/r/71649/#comment306447> preDeleteByName() and preDeleteByGuid() may not be needed. We will not be deleting any reference edges from the namespaceDef vertex (since no edges exists). repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java Lines 264 (patched) <https://reviews.apache.org/r/71649/#comment306446> this is not valid. NamespaceDef should be deleted only when no entity has reference to that namespace name. Maybe check if there are any entity referring this namespace (using index query?) before deleting. Same for line #294 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java Lines 311 (patched) <https://reviews.apache.org/r/71649/#comment306450> toNamespaceDef => toAtlasNamespaceDef repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java Lines 321 (patched) <https://reviews.apache.org/r/71649/#comment306451> toNamespaceDef => toAtlasNamespaceDef repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java Lines 382 (patched) <https://reviews.apache.org/r/71649/#comment306449> toJsonFromAttribute => toJsonFromNamespaceAttribute repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java Lines 420 (patched) <https://reviews.apache.org/r/71649/#comment306448> toAttributeDefFromJson => toNamespaceAttributeDefFromJson repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java Lines 421 (patched) <https://reviews.apache.org/r/71649/#comment306452> AtlasBaseException is never thrown repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java Lines 524 (patched) <https://reviews.apache.org/r/71649/#comment306453> we may need to check for NamespaceAttributeDef with empty/null 'applicableTypes' => CollectionUtils.isEmpty(attributeDef.getApplicableEntityTypes()) repository/src/test/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStoreTest.java Lines 697 (patched) <https://reviews.apache.org/r/71649/#comment306455> empty test? webapp/src/main/java/org/apache/atlas/examples/QuickStartV2.java Lines 22 (patched) <https://reviews.apache.org/r/71649/#comment306456> nit: review unused imports - Sarath Subramanian On Nov. 13, 2019, 4:08 p.m., Aadarsh Jajodia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71649/ > --- > > (Updated Nov. 13, 2019, 4:08 p.m.) > > > Review request for atlas, Ashutosh Mestry, Sridhar K, Le Ma, Madhan > Neethiraj, and Sarath Subramanian. > > > Bugs: ATLAS-3486 > https://issues.apache.org/jira/browse/ATLAS-3486 > > > Repository: atlas > > > Description > --- > > This change is the first part of the bigger task defined as part of > ATLAS-3485. > This adds the data model needed for supporting namespaces and namespace > attributes and also updates the type registry to include the applicable > namespace attributes for every entity type > > > Diffs > - > > intg/src/main/java/org/apache/atlas/AtlasErrorCode.java > 7a2aae2e9ae8174c5309164f3e41c940cbf3ddf8 > intg/src/main/java/org/apache/atlas/model/TypeCategory.java > f06f64f450f407e3f9a0e742726ff4dd12ccc695 > intg/src/main/java/org/apache/atlas/mod
Re: Review Request 71737: ATLAS-3477: Introduce purging of entity in Atlas
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71737/#review218637
---
Ship it!
Ship It!
- Sarath Subramanian
On Nov. 14, 2019, 11:10 a.m., Sidharth Mishra wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71737/
> ---
>
> (Updated Nov. 14, 2019, 11:10 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath
> Subramanian.
>
>
> Bugs: ATLAS-3477
> https://issues.apache.org/jira/browse/ATLAS-3477
>
>
> Repository: atlas
>
>
> Description
> ---
>
> As part of this solution users can purge the entities which are already soft
> deleted. This will be useful as someone may be interested to purge some list
> of soft deleted entities from Atlas. Due to governance compliances, only
> admin can perform purge of entities.
>
> Purge will also create separate Audit entry at HBase similar to delete entity
> operation.
>
>
> Diffs
> -
>
> authorization/src/main/java/org/apache/atlas/authorize/AtlasPrivilege.java
> 59c596d66
> client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
> 199d6bf58
> intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java
> 444167eae
> intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java
> e9cc7cd3d
>
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java
> 7ace00dff
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutations.java
> daf4ca88c
>
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
> 43a9b842c
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java
> b94590bf1
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
> d2544df4f
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
> 3389d2465
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
> 17a96493f
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
> 4d57d8bc8
> server-api/src/main/java/org/apache/atlas/RequestContext.java 3d2a18f0c
>
> webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java
> 48f0cd3c2
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
> 464d46f1e
>
> webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java
> 5d506bbc2
>
>
> Diff: https://reviews.apache.org/r/71737/diff/6/
>
>
> Testing
> ---
>
> IT has been added.
> Validated manually by running below mentioned queries and REST:
>
>
> a) create table thursday_07_11_1 (id int);
>create table thursday_07_11_1_ctas as select * from thursday_07_11_1;
>
>create table thursday_07_11_2 (id int);
>create table thursday_07_11_2_ctas as select * from thursday_07_11_2;
> b) The above will create two lineages. Go to Atlas UI and get the vertices
> for atleast any two.
>
>
> c) Using rest delete those two entities or use below query:
> drop table thursday_07_11_1;
> drop table thursday_07_11_1_ctas;
>
> drop table thursday_07_11_2;
> drop table thursday_07_11_2_ctas;
>
> REST delete example:
> DELETE
> http://localhopst:31000/api/atlas/v2/entity/guid/7dd7fd9c-0185-41a5-aff0-4212fa7eb87c
>
> d) Now use the same guids and call purge REST:
> DELETE
> http://localhost:31000/api/atlas/admin/purge/?guid=75983487-9e29-41ee-9bae-53f0a0b54c05&guid=98bee463-8810-402a-bcdc-3eb00bf50365
>
> Check the response which will return all the successfully purged entities
>
> e) For each guid check if the audit has separate new entry for purge. REST
> used:
> GET
> http://localhost:31000/api/atlas/v2/entity/7b6806c1-8daf-4e0f-8d10-672a192968f1/audit
>
> Example Response:
>
> [
> {
> "entityId": "7b6806c1-8daf-4e0f-8d10-672a192968f1",
> "timestamp": 1573161789065,
> "user": "admin",
> "action": "ENTITY_PURGE",
> "details": "Purged entity",
> "eventKey": "7b6806c1-8daf-4e0f-8d10-672a192968f1:1573161789065:1",
> "entity": null,
&
Re: Review Request 71737: ATLAS-3477: Introduce purging of entity in Atlas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71737/#review218622 --- Fix it, then Ship it! repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java Lines 484 (patched) <https://reviews.apache.org/r/71737/#comment306418> should this be AtlasPrivilege.ADMIN_PURGE ? webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java Lines 25 (patched) <https://reviews.apache.org/r/71737/#comment306416> nit: unused import; consider removing it webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java Lines 441 (patched) <https://reviews.apache.org/r/71737/#comment306417> purgeByGuids => purgeByIds (to align with method names in EntityREST), updated logger at line 452 as well. - Sarath Subramanian On Nov. 12, 2019, 10:04 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71737/ > --- > > (Updated Nov. 12, 2019, 10:04 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3477 > https://issues.apache.org/jira/browse/ATLAS-3477 > > > Repository: atlas > > > Description > --- > > As part of this solution users can purge the entities which are already soft > deleted. This will be useful as someone may be interested to purge some list > of soft deleted entities from Atlas. Due to governance compliances, only > admin can perform purge of entities. > > Purge will also create separate Audit entry at HBase similar to delete entity > operation. > > > Diffs > - > > authorization/src/main/java/org/apache/atlas/authorize/AtlasPrivilege.java > 59c596d66 > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java > 199d6bf58 > intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java > 444167eae > intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java > e9cc7cd3d > > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java > 7ace00dff > intg/src/main/java/org/apache/atlas/model/instance/EntityMutations.java > daf4ca88c > > repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java > 43a9b842c > > repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java > b94590bf1 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > d2544df4f > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java > 3389d2465 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java > 17a96493f > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java > 4d57d8bc8 > server-api/src/main/java/org/apache/atlas/RequestContext.java 3d2a18f0c > > webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java > 48f0cd3c2 > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > 464d46f1e > > webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java > 5d506bbc2 > > > Diff: https://reviews.apache.org/r/71737/diff/4/ > > > Testing > --- > > IT has been added. > Validated manually by running below mentioned queries and REST: > > > a) create table thursday_07_11_1 (id int); >create table thursday_07_11_1_ctas as select * from thursday_07_11_1; > >create table thursday_07_11_2 (id int); >create table thursday_07_11_2_ctas as select * from thursday_07_11_2; > b) The above will create two lineages. Go to Atlas UI and get the vertices > for atleast any two. > > > c) Using rest delete those two entities or use below query: > drop table thursday_07_11_1; > drop table thursday_07_11_1_ctas; > > drop table thursday_07_11_2; > drop table thursday_07_11_2_ctas; > > REST delete example: > DELETE > http://localhopst:31000/api/atlas/v2/entity/guid/7dd7fd9c-0185-41a5-aff0-4212fa7eb87c > > d) Now use the same guids and call purge REST: > DELETE > http://localhost:31000/api/atlas/admin/purge/?guid=75983487-9e29-41ee-9bae-53f0a0b54c05&guid=98bee463-8810-402a-bcdc-3eb00bf50365 > > Check the response which will return all the successfully purged entities >
