Re: Review Request 70358: ATLAS-3113: Use index query to search for active entities and better logging in java patch framework
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70358/#review214257 --- Ship it! Ship It! - Madhan Neethiraj On April 2, 2019, 4:43 a.m., Sarath Subramanian wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70358/ > --- > > (Updated April 2, 2019, 4:43 a.m.) > > > Review request for atlas, Ashutosh Mestry, Kapildeo Nayak, Madhan Neethiraj, > Nikhil Bonte, and Nixon Rodrigues. > > > Bugs: ATLAS-3113 > https://issues.apache.org/jira/browse/ATLAS-3113 > > > Repository: atlas > > > Description > --- > > In java patch framework - Unique attribute patch handler - to retrieve all > active entities graph query is used, graph query is expensive when number of > entities are large. This Jira updates the graph query to use index query and > also logging enhancements. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatchHandler.java > 0e707a570 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java > 2882f097a > > > Diff: https://reviews.apache.org/r/70358/diff/1/ > > > Testing > --- > > validated that index query retrieved active entities much faster. > > > Thanks, > > Sarath Subramanian > >
Review Request 70358: ATLAS-3113: Use index query to search for active entities and better logging in java patch framework
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70358/ --- Review request for atlas, Ashutosh Mestry, Kapildeo Nayak, Madhan Neethiraj, Nikhil Bonte, and Nixon Rodrigues. Bugs: ATLAS-3113 https://issues.apache.org/jira/browse/ATLAS-3113 Repository: atlas Description --- In java patch framework - Unique attribute patch handler - to retrieve all active entities graph query is used, graph query is expensive when number of entities are large. This Jira updates the graph query to use index query and also logging enhancements. Diffs - repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatchHandler.java 0e707a570 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java 2882f097a Diff: https://reviews.apache.org/r/70358/diff/1/ Testing --- validated that index query retrieved active entities much faster. Thanks, Sarath Subramanian
Re: Review Request 70357: ATLAS-3112: Allow Indexing of array attributes (LIST or SET) in indexing store
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70357/#review214256 --- Ship it! Ship It! - Madhan Neethiraj On April 2, 2019, 4:37 a.m., Sarath Subramanian wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70357/ > --- > > (Updated April 2, 2019, 4:37 a.m.) > > > Review request for atlas, Ashutosh Mestry, Kapildeo Nayak, Madhan Neethiraj, > Nikhil Bonte, and Nixon Rodrigues. > > > Bugs: ATLAS-3112 > https://issues.apache.org/jira/browse/ATLAS-3112 > > > Repository: atlas > > > Description > --- > > Array attributes with cardinality LIST or SET is not indexed in the indexing > store. To allow index query to search on array attributes, array attributes > needs to be indexed. > > > Diffs > - > > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java > b1e2c5dd8 > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java > 4e18432e7 > > graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/AbstractGraphDatabaseTest.java > f72b41214 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > c57f8e3c7 > > > Diff: https://reviews.apache.org/r/70357/diff/1/ > > > Testing > --- > > Verified in Solr that array attributes - superTypeNames, traitNames and > propagatedTraitNames are indexed. > > Precommit job: > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1009/console > > > Thanks, > > Sarath Subramanian > >
[jira] [Commented] (ATLAS-3113) Use index query to search for active entities and better logging in java patch framework
[ https://issues.apache.org/jira/browse/ATLAS-3113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807405#comment-16807405 ] Sarath Subramanian commented on ATLAS-3113: --- Code review: [https://reviews.apache.org/r/70358/] > Use index query to search for active entities and better logging in java > patch framework > > > Key: ATLAS-3113 > URL: https://issues.apache.org/jira/browse/ATLAS-3113 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 1.1.0 >Reporter: Sarath Subramanian >Assignee: Sarath Subramanian >Priority: Major > Fix For: 2.0.0 > > > In java patch framework - Unique attribute patch handler - to retrieve all > active entities graph query is used, graph query is expensive when number of > entities are large. This Jira updates the graph query to use index query and > also logging enhancements. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ATLAS-3113) Use index query to search for active entities and better logging in java patch framework
Sarath Subramanian created ATLAS-3113: - Summary: Use index query to search for active entities and better logging in java patch framework Key: ATLAS-3113 URL: https://issues.apache.org/jira/browse/ATLAS-3113 Project: Atlas Issue Type: Bug Components: atlas-core Affects Versions: 1.1.0 Reporter: Sarath Subramanian Assignee: Sarath Subramanian Fix For: 2.0.0 In java patch framework - Unique attribute patch handler - to retrieve all active entities graph query is used, graph query is expensive when number of entities are large. This Jira updates the graph query to use index query and also logging enhancements. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3112) Allow Indexing of array attributes (LIST or SET) in indexing store
[ https://issues.apache.org/jira/browse/ATLAS-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807404#comment-16807404 ] Sarath Subramanian commented on ATLAS-3112: --- Code review: [https://reviews.apache.org/r/70357/] > Allow Indexing of array attributes (LIST or SET) in indexing store > -- > > Key: ATLAS-3112 > URL: https://issues.apache.org/jira/browse/ATLAS-3112 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 1.1.0 >Reporter: Sarath Subramanian >Assignee: Sarath Subramanian >Priority: Major > Fix For: 2.0.0 > > > Array attributes with cardinality LIST or SET is not indexed in the indexing > store. To allow index query to search on array attributes, array attributes > needs to be indexed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Review Request 70357: ATLAS-3112: Allow Indexing of array attributes (LIST or SET) in indexing store
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70357/ --- Review request for atlas, Ashutosh Mestry, Kapildeo Nayak, Madhan Neethiraj, Nikhil Bonte, and Nixon Rodrigues. Bugs: ATLAS-3112 https://issues.apache.org/jira/browse/ATLAS-3112 Repository: atlas Description --- Array attributes with cardinality LIST or SET is not indexed in the indexing store. To allow index query to search on array attributes, array attributes needs to be indexed. Diffs - graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java b1e2c5dd8 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java 4e18432e7 graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/AbstractGraphDatabaseTest.java f72b41214 repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java c57f8e3c7 Diff: https://reviews.apache.org/r/70357/diff/1/ Testing --- Verified in Solr that array attributes - superTypeNames, traitNames and propagatedTraitNames are indexed. Precommit job: https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1009/console Thanks, Sarath Subramanian
[jira] [Created] (ATLAS-3112) Allow Indexing of array attributes (LIST or SET) in indexing store
Sarath Subramanian created ATLAS-3112: - Summary: Allow Indexing of array attributes (LIST or SET) in indexing store Key: ATLAS-3112 URL: https://issues.apache.org/jira/browse/ATLAS-3112 Project: Atlas Issue Type: Bug Components: atlas-core Affects Versions: 1.1.0 Reporter: Sarath Subramanian Assignee: Sarath Subramanian Fix For: 2.0.0 Array attributes with cardinality LIST or SET is not indexed in the indexing store. To allow index query to search on array attributes, array attributes needs to be indexed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-2902) Export & Import Process: Audit Logs Are Not Ordered
[ https://issues.apache.org/jira/browse/ATLAS-2902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807339#comment-16807339 ] Na Li commented on ATLAS-2902: -- the code changes we need to make in order to support sorting is 1) Add sorting parameter in SearchParameters that specifies the sorting field and the sorting order. For example, PROPERTY_START_TIME + " desc". 2) Change API of AtlasJanusGraph.java.IndexQuery() to pass in the index query parameters. 3) In EntitySearchProcessor, is sorting parameter is present in SearchParameters, pass it as one of the index query parameters when calling AtlasJanusGraph.java.IndexQuery() 4) In ExportImportAuditService.get(), set the sorting parameter in SearchParameters in order to for index processor (Solr for example) to sort the result before returning the list of entities. > Export & Import Process: Audit Logs Are Not Ordered > --- > > Key: ATLAS-2902 > URL: https://issues.apache.org/jira/browse/ATLAS-2902 > Project: Atlas > Issue Type: Bug >Reporter: Ashutosh Mestry >Assignee: Na Li >Priority: Major > Attachments: ATLAS-2902-Export-Import-Audits-logs-sorted.patch > > > Export-Import audit, are not ordered based on _operationStartTime_. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (ATLAS-2902) Export & Import Process: Audit Logs Are Not Ordered
[ https://issues.apache.org/jira/browse/ATLAS-2902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reassigned ATLAS-2902: Assignee: Sridhar (was: Na Li) > Export & Import Process: Audit Logs Are Not Ordered > --- > > Key: ATLAS-2902 > URL: https://issues.apache.org/jira/browse/ATLAS-2902 > Project: Atlas > Issue Type: Bug >Reporter: Ashutosh Mestry >Assignee: Sridhar >Priority: Major > Attachments: ATLAS-2902-Export-Import-Audits-logs-sorted.patch > > > Export-Import audit, are not ordered based on _operationStartTime_. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 70304: Improvements to PC Framework
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70304/ --- (Updated April 1, 2019, 10:15 p.m.) Review request for atlas, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian. Changes --- Updates include: Addressed timeout calculation to eliminate division operation. Bugs: ATLAS-3090 https://issues.apache.org/jira/browse/ATLAS-3090 Repository: atlas Description --- **Approach** - Additional methods added to _WorkItemManager_ and _WorkItemConsumer_. - Added ability to return results from consumers. - Added ability to restart tasks if they are done. **Description** - _getResults_ Fetch results from consumers. - _drain_ wait until existing tasks are completed. - _checkAndProduce_ add tasks only after adding to executor. Diffs (updated) - graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java fb1e68448 intg/src/main/java/org/apache/atlas/pc/WorkItemConsumer.java df2cb67dd intg/src/main/java/org/apache/atlas/pc/WorkItemManager.java 8ac6f115d intg/src/test/java/org/apache/atlas/pc/WorkItemConsumerTest.java 6c88b9e6b intg/src/test/java/org/apache/atlas/pc/WorkItemConsumerWithResultsTest.java PRE-CREATION intg/src/test/java/org/apache/atlas/pc/WorkItemManagerWithResultsTest.java PRE-CREATION Diff: https://reviews.apache.org/r/70304/diff/2/ Changes: https://reviews.apache.org/r/70304/diff/1-2/ Testing --- **Unit tests** New tests added. **Pre-commit Build** https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/999/ Thanks, Ashutosh Mestry
[jira] [Updated] (ATLAS-3071) Add Functionalities to Collect Notification Metrics/Entity Lifecyle
[
https://issues.apache.org/jira/browse/ATLAS-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Le Ma updated ATLAS-3071:
-
Attachment: ATLAS-3071.patch
> Add Functionalities to Collect Notification Metrics/Entity Lifecyle
> ---
>
> Key: ATLAS-3071
> URL: https://issues.apache.org/jira/browse/ATLAS-3071
> Project: Atlas
> Issue Type: New Feature
>Reporter: Le Ma
>Assignee: Le Ma
>Priority: Major
> Attachments: ATLAS-3071.patch, MetricsDataModel.json
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Extend api/atlas/admin/metrics to provide metrics for notifications/entity
> lifycycle:
> # notification
> processed
> # notification
> failed
> today/thisHour/pastHour/total { # entity created
> # entity updated
> # entity deleted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ATLAS-3071) Add Functionalities to Collect Notification Metrics/Entity Lifecyle
[
https://issues.apache.org/jira/browse/ATLAS-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Le Ma updated ATLAS-3071:
-
Attachment: (was: ATLAS-3071.patch)
> Add Functionalities to Collect Notification Metrics/Entity Lifecyle
> ---
>
> Key: ATLAS-3071
> URL: https://issues.apache.org/jira/browse/ATLAS-3071
> Project: Atlas
> Issue Type: New Feature
>Reporter: Le Ma
>Assignee: Le Ma
>Priority: Major
> Attachments: ATLAS-3071.patch, MetricsDataModel.json
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Extend api/atlas/admin/metrics to provide metrics for notifications/entity
> lifycycle:
> # notification
> processed
> # notification
> failed
> today/thisHour/pastHour/total { # entity created
> # entity updated
> # entity deleted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (ATLAS-3111) Ranger Hive Plugin enhancement for KILL query and Replication commands authorization
[ https://issues.apache.org/jira/browse/ATLAS-3111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Mani resolved ATLAS-3111. Resolution: Invalid Wrongly assigned project. > Ranger Hive Plugin enhancement for KILL query and Replication commands > authorization > > > Key: ATLAS-3111 > URL: https://issues.apache.org/jira/browse/ATLAS-3111 > Project: Atlas > Issue Type: Bug >Reporter: Ramesh Mani >Priority: Major > > 1) Hive KILL Query > With the HIVE-17483 JIRA, Hive has introduced a way to kill query and in > hive its a privileged action for Hive Admin Role. In order for the Ranger > Hive Authorizer to support authorization, we need to enhance the ranger hive > authorizer. Current Hive implementation is to Kill Query in a HiveService > which can be LLAP / HIVESERVER2 , later these HIVE SERVICEs can be grouped > into NAME SPACEs and kill query can be run against them. When > HiveServer2/LLAP Ranger Plugin sends the request to Ranger for Authorization, > it will be sending the HIVE SERVICE in the context with the COMMAND that is > executed. > With all the details proposal is to have > 1) In Ranger Hive Service Definition, we will have a new Resource "Hive > Service" to authorize. > 2) In Ranger Hive Permission Model, we will have a new Permission "Service > Admin" to group Kill Query operation. > "Service Admin" permission will enable hive ranger plugin to isolate various > admin operations in this case "Kill Query" and in future if hive introduces > other operations which are done at "HIVE SERVICE level" , group them under > this and authorize. > "Service Admin" won't be able to do DATABASE / TABLE / COLUMN operations as > this will all be taken care by the existing DATABASE/TABLE/COLUMN level > permission model. > 2) Replication Command > Hive has enhanced it authorization for Replication Task > https://issues.apache.org/jira/browse/HIVE-17005. The proposal from Ranger > side is to have "Repl Admin" permission in RangerHive privilege model and > command REPL DUMP and REPL LOAD should be authorized for the users with > "Admin" privilege on Database / Table level. > For REPL STATUS command, the user should have SELECT privilege on the > Database/ Table Level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ATLAS-3111) Ranger Hive Plugin enhancement for KILL query and Replication commands authorization
Ramesh Mani created ATLAS-3111: -- Summary: Ranger Hive Plugin enhancement for KILL query and Replication commands authorization Key: ATLAS-3111 URL: https://issues.apache.org/jira/browse/ATLAS-3111 Project: Atlas Issue Type: Bug Reporter: Ramesh Mani 1) Hive KILL Query With the HIVE-17483 JIRA, Hive has introduced a way to kill query and in hive its a privileged action for Hive Admin Role. In order for the Ranger Hive Authorizer to support authorization, we need to enhance the ranger hive authorizer. Current Hive implementation is to Kill Query in a HiveService which can be LLAP / HIVESERVER2 , later these HIVE SERVICEs can be grouped into NAME SPACEs and kill query can be run against them. When HiveServer2/LLAP Ranger Plugin sends the request to Ranger for Authorization, it will be sending the HIVE SERVICE in the context with the COMMAND that is executed. With all the details proposal is to have 1) In Ranger Hive Service Definition, we will have a new Resource "Hive Service" to authorize. 2) In Ranger Hive Permission Model, we will have a new Permission "Service Admin" to group Kill Query operation. "Service Admin" permission will enable hive ranger plugin to isolate various admin operations in this case "Kill Query" and in future if hive introduces other operations which are done at "HIVE SERVICE level" , group them under this and authorize. "Service Admin" won't be able to do DATABASE / TABLE / COLUMN operations as this will all be taken care by the existing DATABASE/TABLE/COLUMN level permission model. 2) Replication Command Hive has enhanced it authorization for Replication Task https://issues.apache.org/jira/browse/HIVE-17005. The proposal from Ranger side is to have "Repl Admin" permission in RangerHive privilege model and command REPL DUMP and REPL LOAD should be authorized for the users with "Admin" privilege on Database / Table level. For REPL STATUS command, the user should have SELECT privilege on the Database/ Table Level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (ATLAS-3110) Add a bulk api to get entities by unique attributes
[ https://issues.apache.org/jira/browse/ATLAS-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Nigam reassigned ATLAS-3110: -- Assignee: Ayush Nigam > Add a bulk api to get entities by unique attributes > --- > > Key: ATLAS-3110 > URL: https://issues.apache.org/jira/browse/ATLAS-3110 > Project: Atlas > Issue Type: New Feature >Reporter: Ayush Nigam >Assignee: Ayush Nigam >Priority: Minor > > Add a bulk api to get entities by unique attributes,currently there is bulk > api for guids only -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ATLAS-3110) Add a bulk api to get entities by unique attributes
Ayush Nigam created ATLAS-3110: -- Summary: Add a bulk api to get entities by unique attributes Key: ATLAS-3110 URL: https://issues.apache.org/jira/browse/ATLAS-3110 Project: Atlas Issue Type: New Feature Reporter: Ayush Nigam Add a bulk api to get entities by unique attributes,currently there is bulk api for guids only -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-3081) Expose Gremlin Search API
[ https://issues.apache.org/jira/browse/ATLAS-3081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806459#comment-16806459 ] Ayush Nigam commented on ATLAS-3081: [~madhan.neethiraj] So our usecase is like this: There can be many levels of hierarchy for Atlas entity.So starting from top we may want to go to the bottom by traversing whole graph and find all those entities which have a classification,say GDPR attached to it,and all this navigation we will do based on the type of relationship. So basically we need a way to navigate the graph based on the type of relationship. It can also work if in advanced DSL we have a way to do so. > Expose Gremlin Search API > - > > Key: ATLAS-3081 > URL: https://issues.apache.org/jira/browse/ATLAS-3081 > Project: Atlas > Issue Type: New Feature >Reporter: Ayush Nigam >Assignee: Ayush Nigam >Priority: Minor > Attachments: ATLAS_3081.patch > > > Expose Gremlin Search API to solve more complex usecases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
