[jira] [Updated] (ATLAS-4191) Creation of relations does not work for glossary import
[ https://issues.apache.org/jira/browse/ATLAS-4191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4191: -- Component/s: atlas-core > Creation of relations does not work for glossary import > --- > > Key: ATLAS-4191 > URL: https://issues.apache.org/jira/browse/ATLAS-4191 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > Attachments: ATLAS-4191.patch, Atlas BG template - smallish.csv > > > When importing glossary data in CSV format, any relations within the imported > data lead to an error; import can only find referenced Terms that already > exist in Atlas. > Example: Term GA -> T1 has a 'seeAlso' relation to GA -> T2 and vice versa. > (See also attached CSV file) > This makes using relations with larger glossaries almost impossible to use: > there will be circular references (as with 'seeAlso'), so even a two-step > approach to import will not work. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4191) Creation of relations does not work for glossary import
[ https://issues.apache.org/jira/browse/ATLAS-4191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4191: -- Labels: glossary import (was: ) > Creation of relations does not work for glossary import > --- > > Key: ATLAS-4191 > URL: https://issues.apache.org/jira/browse/ATLAS-4191 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > Labels: glossary, import > Attachments: ATLAS-4191.patch, Atlas BG template - smallish.csv > > > When importing glossary data in CSV format, any relations within the imported > data lead to an error; import can only find referenced Terms that already > exist in Atlas. > Example: Term GA -> T1 has a 'seeAlso' relation to GA -> T2 and vice versa. > (See also attached CSV file) > This makes using relations with larger glossaries almost impossible to use: > there will be circular references (as with 'seeAlso'), so even a two-step > approach to import will not work. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 73076: Deferred Actions Implementation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73076/#review222659 --- Ship it! Ship It! - Sarath Subramanian On March 8, 2021, 10:19 a.m., Ashutosh Mestry wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73076/ > --- > > (Updated March 8, 2021, 10:19 a.m.) > > > Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, > Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-3919 > https://issues.apache.org/jira/browse/ATLAS-3919 > > > Repository: atlas > > > Description > --- > > **Approach** > Model: _TaskDef_ > - Definition of task. > - Contains status of the task for querying. > > _AbstractTask_ > - Takes TaskDef as input and calls run. > - The concrete tasks has logic for doing real work. > > _TaskFactory_ > > - Creates tasks given task type. > - Enumerates all the tasks that are supported. > > _TaskManagement_ > > - Component/Service. Constructed after all existing components are > constructed. > - Provides a simple interface for adding tasks. > - Processes tasks that are in pending state. > - Supports HA mode, where the tasks are not executed on passive instance. > > _TaskManagement.Registry_ > > - CRUD operations with TaskDef vertex. > > _TaskManagement.Processor_ > - Provides mechanism for provides single queue for processing tasks in a way > that is non-blocking. > - Uses task type and the associated TaskFactory to construct concrete task. > > Changes to _EntityGraphMapper_ > There are 3 new methods added that use portion of existing classification > propagation logic and create a separate method. > > New class _EntityGraphMapperWithTasks_ acts as a proxy for EntityGraphMapper. > All calls that operate without tasks get channeled to the existing > implementation. The ones involving tasks are modified to use the new task > framework. > > **Concurrent Changes to Classifications** > Via REST APIs and operations from web UI, it is possible to make changes to > the same entity. This situation will be more pronounced in the new setup. > > The existing framework has > _GraphTransactionInterceptor.lockObjectAndReleasePostCommit_. This provides > ReentrantLock on an entity that is identified by the supplied GUID. This will > cause a lock on the entity and all the impacted vertices. This has potential > for adversely impacting scalability. > > **REST APIs** > +-++--+--+ > | api/admin/tasks | GET | guids | (optional) List of guids to be > fetched. | > | | | | If list of guids supplied is empty > all tasks | > | | | | that are in PENDING state are > fetched. If list | > | | | | guids is specified, tasks > corresponding to that guid are fetched.| > +---+--+--+--+ > |api/admin/task |DELETE| guids| List of guids to be deleted. No > action is taken if list of guids | > | | | | supplied is empty or a Task guid does > not exist. | > +---+--+--+--+ > > **New Updates** > - Separate log file for tasks. > - TaskDef now has _startTime_ and _endTime_ to indicating start and end of > task. > - Refactored parameter passing during task creation. It now accepts Map > instead of Object array. Each task creates its own set of parameters. > - Reduced use of TASKS_ENABLE flag checks. > - Added _AtlasEntity.getPendingTasks()_ that indicidate presence of tasks for > a given entity. > > > Diffs > - > > common/src/main/java/org/apache/atlas/repository/Constants.java 771287f75 > distro/src/conf/atlas-log4j.xml 7df963eb2 > intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 08d6c9d4a > intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java > 4d8c94894 > intg/src/main/java/org/apache/atlas/model/tasks/AtlasTask.java PRE-CREATION > intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 27c7f7391 > intg/src/main/java/
Re: Review Request 73076: Deferred Actions Implementation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73076/#review222654 --- Fix it, then Ship it! intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java Lines 98 (patched) <https://reviews.apache.org/r/73076/#comment311824> 'pendingTasks'is never used? repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java Lines 2781 (patched) <https://reviews.apache.org/r/73076/#comment311820> why createAndQueueTask() need both 'entityVertex' and 'entityGuid'. Consider passing only 'entityVertex' and retrieve 'entityGuid' from entityVertex to avoid duplicates in methods params. repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java Lines 2795 (patched) <https://reviews.apache.org/r/73076/#comment311821> 'entityGuid' and 'entityVertex' doesn't need to be duplicated in methods params. Fetch entityGuid from entityVertex. repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationTask.java Lines 78 (patched) <https://reviews.apache.org/r/73076/#comment311823> remove task guid from entity vertex after task completion? entityGraphMapper.removePendingTaskFromEntityVertex(entityGuid, getTaskGuid()) repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java Lines 199 (patched) <https://reviews.apache.org/r/73076/#comment311822> REPORT_FREQUENCEY => REPORT_FREQUENCY - Sarath Subramanian On Feb. 25, 2021, 10:08 p.m., Ashutosh Mestry wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73076/ > --- > > (Updated Feb. 25, 2021, 10:08 p.m.) > > > Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, > Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-3919 > https://issues.apache.org/jira/browse/ATLAS-3919 > > > Repository: atlas > > > Description > --- > > **Approach** > Model: _TaskDef_ > - Definition of task. > - Contains status of the task for querying. > > _AbstractTask_ > - Takes TaskDef as input and calls run. > - The concrete tasks has logic for doing real work. > > _TaskFactory_ > > - Creates tasks given task type. > - Enumerates all the tasks that are supported. > > _TaskManagement_ > > - Component/Service. Constructed after all existing components are > constructed. > - Provides a simple interface for adding tasks. > - Processes tasks that are in pending state. > - Supports HA mode, where the tasks are not executed on passive instance. > > _TaskManagement.Registry_ > > - CRUD operations with TaskDef vertex. > > _TaskManagement.Processor_ > - Provides mechanism for provides single queue for processing tasks in a way > that is non-blocking. > - Uses task type and the associated TaskFactory to construct concrete task. > > Changes to _EntityGraphMapper_ > There are 3 new methods added that use portion of existing classification > propagation logic and create a separate method. > > New class _EntityGraphMapperWithTasks_ acts as a proxy for EntityGraphMapper. > All calls that operate without tasks get channeled to the existing > implementation. The ones involving tasks are modified to use the new task > framework. > > **Concurrent Changes to Classifications** > Via REST APIs and operations from web UI, it is possible to make changes to > the same entity. This situation will be more pronounced in the new setup. > > The existing framework has > _GraphTransactionInterceptor.lockObjectAndReleasePostCommit_. This provides > ReentrantLock on an entity that is identified by the supplied GUID. This will > cause a lock on the entity and all the impacted vertices. This has potential > for adversely impacting scalability. > > **REST APIs** > +-++--+--+ > | api/admin/tasks | GET | guids | (optional) List of guids to be > fetched. | > | | | | If list of guids supplied is empty > all tasks | > | | | | that are in PENDING state are > fetched. If list | > | | | | guids is specified, tasks > corresponding to that guid are fetched.| > +---+--+--
[jira] [Updated] (ATLAS-4181) Add mandatory attributes to existing entity definition
[ https://issues.apache.org/jira/browse/ATLAS-4181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4181: -- Fix Version/s: 2.2.0 3.0.0 > Add mandatory attributes to existing entity definition > -- > > Key: ATLAS-4181 > URL: https://issues.apache.org/jira/browse/ATLAS-4181 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Fix For: 3.0.0, 2.2.0 > > > Currently, Atlas allows to add only optional attributes to existing entity > type and it doesn’t allow to add any mandatory attribute. > Atlas should allow users to add mandatory attributes to existing entity > types. To provide backward compatibility without breaking any contract > between updated entity definition and existing entities, once entity def gets > updated with mandatory attribute it should update all entities of same type > and all subtypes with mandatory attribute. > This feature should allow only if new mandatory attribute has non-empty > default value. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4181) Add mandatory attributes to existing entity definition
[ https://issues.apache.org/jira/browse/ATLAS-4181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4181: -- Affects Version/s: 2.1.0 > Add mandatory attributes to existing entity definition > -- > > Key: ATLAS-4181 > URL: https://issues.apache.org/jira/browse/ATLAS-4181 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > > Currently, Atlas allows to add only optional attributes to existing entity > type and it doesn’t allow to add any mandatory attribute. > Atlas should allow users to add mandatory attributes to existing entity > types. To provide backward compatibility without breaking any contract > between updated entity definition and existing entities, once entity def gets > updated with mandatory attribute it should update all entities of same type > and all subtypes with mandatory attribute. > This feature should allow only if new mandatory attribute has non-empty > default value. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4181) Add mandatory attributes to existing entity definition
[ https://issues.apache.org/jira/browse/ATLAS-4181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4181: -- Component/s: atlas-core > Add mandatory attributes to existing entity definition > -- > > Key: ATLAS-4181 > URL: https://issues.apache.org/jira/browse/ATLAS-4181 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Fix For: 3.0.0, 2.2.0 > > > Currently, Atlas allows to add only optional attributes to existing entity > type and it doesn’t allow to add any mandatory attribute. > Atlas should allow users to add mandatory attributes to existing entity > types. To provide backward compatibility without breaking any contract > between updated entity definition and existing entities, once entity def gets > updated with mandatory attribute it should update all entities of same type > and all subtypes with mandatory attribute. > This feature should allow only if new mandatory attribute has non-empty > default value. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4169) Add Hive Location Path entities to HMS Hook
[ https://issues.apache.org/jira/browse/ATLAS-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4169: -- Labels: hive-hooks (was: ) > Add Hive Location Path entities to HMS Hook > --- > > Key: ATLAS-4169 > URL: https://issues.apache.org/jira/browse/ATLAS-4169 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Labels: hive-hooks > Fix For: 3.0.0, 2.2.0 > > > In ATLAS-4098 added location path entity to hive_db entity. This feature was > handled only for HS2 but missing for HMS. > When hive_db created either using HMS or HS2, hive_db entity should have path > entity as new relationship attribute. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4169) Add Hive Location Path entities to HMS Hook
[ https://issues.apache.org/jira/browse/ATLAS-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4169: -- Component/s: atlas-core > Add Hive Location Path entities to HMS Hook > --- > > Key: ATLAS-4169 > URL: https://issues.apache.org/jira/browse/ATLAS-4169 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Fix For: 3.0.0, 2.2.0 > > > In ATLAS-4098 added location path entity to hive_db entity. This feature was > handled only for HS2 but missing for HMS. > When hive_db created either using HMS or HS2, hive_db entity should have path > entity as new relationship attribute. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4169) Add Hive Location Path entities to HMS Hook
[ https://issues.apache.org/jira/browse/ATLAS-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4169: -- Fix Version/s: 2.2.0 3.0.0 > Add Hive Location Path entities to HMS Hook > --- > > Key: ATLAS-4169 > URL: https://issues.apache.org/jira/browse/ATLAS-4169 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Fix For: 3.0.0, 2.2.0 > > > In ATLAS-4098 added location path entity to hive_db entity. This feature was > handled only for HS2 but missing for HMS. > When hive_db created either using HMS or HS2, hive_db entity should have path > entity as new relationship attribute. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4169) Add Hive Location Path entities to HMS Hook
[ https://issues.apache.org/jira/browse/ATLAS-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4169: -- Affects Version/s: 2.1.0 > Add Hive Location Path entities to HMS Hook > --- > > Key: ATLAS-4169 > URL: https://issues.apache.org/jira/browse/ATLAS-4169 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > > In ATLAS-4098 added location path entity to hive_db entity. This feature was > handled only for HS2 but missing for HMS. > When hive_db created either using HMS or HS2, hive_db entity should have path > entity as new relationship attribute. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4170) v2/entity/bulk Entity GET API is able to read unauthorised entities too when skipFailedEntities is passed as True
[
https://issues.apache.org/jira/browse/ATLAS-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4170:
--
Component/s: atlas-core
> v2/entity/bulk Entity GET API is able to read unauthorised entities too when
> skipFailedEntities is passed as True
> -
>
> Key: ATLAS-4170
> URL: https://issues.apache.org/jira/browse/ATLAS-4170
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 2.1.0
>Reporter: Sidharth Kumar Mishra
>Assignee: Sidharth Kumar Mishra
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-4170.patch
>
>
> As part of https://issues.apache.org/jira/browse/ATLAS-3855,
> skipFailedEntities was introduced to ignore the entities where it fails to
> read
> When skipFailedEntities is not passed or is passed as
> skipFailedEntities=False, then we get 403 with below error as expected
> {code:java}
> {
> "errorCode": "ATLAS-403-00-001",
> "errorMessage": "hrt is not authorized to perform read entity:
> guid=ad0f349c-1fe6-46f0-be6d-98ca2e754e1c"
> } {code}
> But when we pass skipFailedEntities=True, then API is able to retrieve the
> data for even those entities on which the user has explicit deny conditions.
> Ideally, we should be ignoring these unauthorised entities and return data
> only for authorised ones.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4170) v2/entity/bulk Entity GET API is able to read unauthorised entities too when skipFailedEntities is passed as True
[
https://issues.apache.org/jira/browse/ATLAS-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4170:
--
Affects Version/s: 2.1.0
> v2/entity/bulk Entity GET API is able to read unauthorised entities too when
> skipFailedEntities is passed as True
> -
>
> Key: ATLAS-4170
> URL: https://issues.apache.org/jira/browse/ATLAS-4170
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Sidharth Kumar Mishra
>Assignee: Sidharth Kumar Mishra
>Priority: Major
> Attachments: ATLAS-4170.patch
>
>
> As part of https://issues.apache.org/jira/browse/ATLAS-3855,
> skipFailedEntities was introduced to ignore the entities where it fails to
> read
> When skipFailedEntities is not passed or is passed as
> skipFailedEntities=False, then we get 403 with below error as expected
> {code:java}
> {
> "errorCode": "ATLAS-403-00-001",
> "errorMessage": "hrt is not authorized to perform read entity:
> guid=ad0f349c-1fe6-46f0-be6d-98ca2e754e1c"
> } {code}
> But when we pass skipFailedEntities=True, then API is able to retrieve the
> data for even those entities on which the user has explicit deny conditions.
> Ideally, we should be ignoring these unauthorised entities and return data
> only for authorised ones.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4170) v2/entity/bulk Entity GET API is able to read unauthorised entities too when skipFailedEntities is passed as True
[
https://issues.apache.org/jira/browse/ATLAS-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4170:
--
Fix Version/s: 2.2.0
3.0.0
> v2/entity/bulk Entity GET API is able to read unauthorised entities too when
> skipFailedEntities is passed as True
> -
>
> Key: ATLAS-4170
> URL: https://issues.apache.org/jira/browse/ATLAS-4170
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Sidharth Kumar Mishra
>Assignee: Sidharth Kumar Mishra
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-4170.patch
>
>
> As part of https://issues.apache.org/jira/browse/ATLAS-3855,
> skipFailedEntities was introduced to ignore the entities where it fails to
> read
> When skipFailedEntities is not passed or is passed as
> skipFailedEntities=False, then we get 403 with below error as expected
> {code:java}
> {
> "errorCode": "ATLAS-403-00-001",
> "errorMessage": "hrt is not authorized to perform read entity:
> guid=ad0f349c-1fe6-46f0-be6d-98ca2e754e1c"
> } {code}
> But when we pass skipFailedEntities=True, then API is able to retrieve the
> data for even those entities on which the user has explicit deny conditions.
> Ideally, we should be ignoring these unauthorised entities and return data
> only for authorised ones.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Review Request 73193: ATLAS-4170: v2/entity/bulk Entity GET API is able to read unauthorised entities too when skipFailedEntities is passed as True
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73193/#review222616
---
Ship it!
Ship It!
- Sarath Subramanian
On Feb. 18, 2021, 6:17 p.m., Sidharth Mishra wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73193/
> ---
>
> (Updated Feb. 18, 2021, 6:17 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Deep Singh, Madhan Neethiraj,
> Radhika Kundam, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4170
> https://issues.apache.org/jira/browse/ATLAS-4170
>
>
> Repository: atlas
>
>
> Description
> ---
>
> ATLAS-4170: Fixed Bulk Entity GET API to skip unauthorised
> entities when skipFailedEntities is passed as True
>
>
> Diffs
> -
>
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
> ce58e9aa4
>
>
> Diff: https://reviews.apache.org/r/73193/diff/1/
>
>
> Testing
> ---
>
> Manually tested:
> Added a Ranger policy to deny read access on hive_db for admin user and then
> passed one of the hive_db guid and skipFailedEntities=True (curl -
> http://atlas.test.site:31000/api/atlas/v2/entity/bulk?guid=ad0f349c-1fe6-46f0-be6d-98ca2e754e1c=True)
> This returned empty after the fix as expected:
>
> {
> referredEntities: { },
> entities: [ ]
> }
>
> Along with above when I passed one more hive_tale guid, it returns the
> hive_table entity details (except the hive_db)
>
>
> Thanks,
>
> Sidharth Mishra
>
>
Re: Review Request 73076: Deferred Actions Implementation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73076/#review222610 --- Fix it, then Ship it! one improvement to add (can be taken up in a follow-up patch) - add the task id to the entity vertex as well, so from entity details page we will know how many pending tasks are in process. intg/src/main/java/org/apache/atlas/model/instance/AtlasClassification.java Lines 93 (patched) <https://reviews.apache.org/r/73076/#comment311733> nit: setPendingPropagations(other.getPendingPropagations()); repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java Lines 102 (patched) <https://reviews.apache.org/r/73076/#comment311740> revert changes to this file (not related to deferred action) repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java Lines 353 (patched) <https://reviews.apache.org/r/73076/#comment311736> getPendingProagations => getPendingPropagations repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java Lines 50 (patched) <https://reviews.apache.org/r/73076/#comment311741> update classification vertex to remove taskDef id from pendingPropagations propery here. same applies for line 69, 87 repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java Lines 89 (patched) <https://reviews.apache.org/r/73076/#comment311738> this method dispatches all taskDefs, consider renaming to "dispatchAllTasks" - Sarath Subramanian On Feb. 16, 2021, 9:55 p.m., Ashutosh Mestry wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73076/ > --- > > (Updated Feb. 16, 2021, 9:55 p.m.) > > > Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, > Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-3919 > https://issues.apache.org/jira/browse/ATLAS-3919 > > > Repository: atlas > > > Description > --- > > **Approach** > Model: _TaskDef_ > - Definition of task. > - Contains status of the task for querying. > > _AbstractTask_ > - Takes TaskDef as input and calls run. > - The concrete tasks has logic for doing real work. > > _TaskFactory_ > > - Creates tasks given task type. > - Enumerates all the tasks that are supported. > > _TaskManagement_ > > - Component/Service. Constructed after all existing components are > constructed. > - Provides a simple interface for adding tasks. > - Processes tasks that are in pending state. > - Supports HA mode, where the tasks are not executed on passive instance. > > _TaskManagement.Registry_ > > - CRUD operations with TaskDef vertex. > > _TaskManagement.Processor_ > - Provides mechanism for provides single queue for processing tasks in a way > that is non-blocking. > - Uses task type and the associated TaskFactory to construct concrete task. > > Changes to _EntityGraphMapper_ > There are 3 new methods added that use portion of existing classification > propagation logic and create a separate method. > > New class _EntityGraphMapperWithTasks_ acts as a proxy for EntityGraphMapper. > All calls that operate without tasks get channeled to the existing > implementation. The ones involving tasks are modified to use the new task > framework. > > **Concurrent Changes to Classifications** > Via REST APIs and operations from web UI, it is possible to make changes to > the same entity. This situation will be more pronounced in the new setup. > > The existing framework has > _GraphTransactionInterceptor.lockObjectAndReleasePostCommit_. This provides > ReentrantLock on an entity that is identified by the supplied GUID. This will > cause a lock on the entity and all the impacted vertices. This has potential > for adversely impacting scalability. > > **New Updates** > - Separate log file for tasks. > - TaskDef now has _startTime_ and _endTime_ to indicating start and end of > task. > - Refactored parameter passing during task creation. It now accepts Map > instead of Object array. Each task creates its own set of parameters. > - Reduced use of TASKS_ENABLE flag checks. > > > Diffs > - > > common/src/main/java/org/apache/atlas/repository/Constants.java 61abfcaca > distro/src/conf/atlas-log4j.xml 7df963eb2 > intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 08d6c9d4a > intg/src/main/java/org/apache/a
Re: Review Request 73076: Deferred Actions Implementation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73076/#review222609 --- repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java Lines 2780 (patched) <https://reviews.apache.org/r/73076/#comment311731> the taskDef guid should be removed from CLASSIFICATION_VERTEX_PENDING_PROPAGATIONS_KEY in classification vertex once the deferred action is complete. Please review. - Sarath Subramanian On Feb. 16, 2021, 9:55 p.m., Ashutosh Mestry wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73076/ > --- > > (Updated Feb. 16, 2021, 9:55 p.m.) > > > Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, > Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-3919 > https://issues.apache.org/jira/browse/ATLAS-3919 > > > Repository: atlas > > > Description > --- > > **Approach** > Model: _TaskDef_ > - Definition of task. > - Contains status of the task for querying. > > _AbstractTask_ > - Takes TaskDef as input and calls run. > - The concrete tasks has logic for doing real work. > > _TaskFactory_ > > - Creates tasks given task type. > - Enumerates all the tasks that are supported. > > _TaskManagement_ > > - Component/Service. Constructed after all existing components are > constructed. > - Provides a simple interface for adding tasks. > - Processes tasks that are in pending state. > - Supports HA mode, where the tasks are not executed on passive instance. > > _TaskManagement.Registry_ > > - CRUD operations with TaskDef vertex. > > _TaskManagement.Processor_ > - Provides mechanism for provides single queue for processing tasks in a way > that is non-blocking. > - Uses task type and the associated TaskFactory to construct concrete task. > > Changes to _EntityGraphMapper_ > There are 3 new methods added that use portion of existing classification > propagation logic and create a separate method. > > New class _EntityGraphMapperWithTasks_ acts as a proxy for EntityGraphMapper. > All calls that operate without tasks get channeled to the existing > implementation. The ones involving tasks are modified to use the new task > framework. > > **Concurrent Changes to Classifications** > Via REST APIs and operations from web UI, it is possible to make changes to > the same entity. This situation will be more pronounced in the new setup. > > The existing framework has > _GraphTransactionInterceptor.lockObjectAndReleasePostCommit_. This provides > ReentrantLock on an entity that is identified by the supplied GUID. This will > cause a lock on the entity and all the impacted vertices. This has potential > for adversely impacting scalability. > > **New Updates** > - Separate log file for tasks. > - TaskDef now has _startTime_ and _endTime_ to indicating start and end of > task. > - Refactored parameter passing during task creation. It now accepts Map > instead of Object array. Each task creates its own set of parameters. > - Reduced use of TASKS_ENABLE flag checks. > > > Diffs > - > > common/src/main/java/org/apache/atlas/repository/Constants.java 61abfcaca > distro/src/conf/atlas-log4j.xml 7df963eb2 > intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 08d6c9d4a > intg/src/main/java/org/apache/atlas/model/instance/AtlasClassification.java > 1edf4eeaf > intg/src/main/java/org/apache/atlas/model/tasks/TaskDef.java PRE-CREATION > intg/src/main/java/org/apache/atlas/utils/AtlasJson.java abeddf640 > intg/src/test/resources/atlas-application.properties 7e74d5107 > repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java > 86b369fc9 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > d0ffb853f > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java > ce58e9aa4 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > 2cfcc0bc1 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java > 8208d11c2 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagateTaskFactory.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagation
Re: Review Request 73185: NotificationHookConsumer: Large Compressed Message Processing Problem
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73185/#review222606 --- Fix it, then Ship it! notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java Lines 23 (patched) <https://reviews.apache.org/r/73185/#comment311717> nit: remove unused import webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java Line 927 (original), 920 (patched) <https://reviews.apache.org/r/73185/#comment311716> consider adding to 'lastCommittedPartitionOffset' after successful commit. => move line 920 after 921. - Sarath Subramanian On Feb. 17, 2021, 11:12 a.m., Ashutosh Mestry wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73185/ > --- > > (Updated Feb. 17, 2021, 11:12 a.m.) > > > Review request for atlas, Madhan Neethiraj, Nixon Rodrigues, and Sarath > Subramanian. > > > Bugs: ATLAS-4155 > https://issues.apache.org/jira/browse/ATLAS-4155 > > > Repository: atlas > > > Description > --- > > **Approach** > - Modify: _NotificaitonHookConsumer_: Add lastOffsetToCommit. Update this on > every commit called. Call the new method on > _AtlasKafkaConsumer.receiveWithCheckedCommit_ > - Modify: _AtlasKafkaConsumer_: Add new method _receiveWithCheckedCommit_. > > > Diffs > - > > notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java > c38a504fa > > notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java > f3e81ecc2 > > notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java > 05d0d8137 > > webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java > 0e58dacff > > webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java > 33191a7a6 > > > Diff: https://reviews.apache.org/r/73185/diff/4/ > > > Testing > --- > > **PC Build** > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/392/ > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/406/ > > **Functional tests** > Large message payload. > > > Thanks, > > Ashutosh Mestry > >
[jira] [Updated] (ATLAS-4155) NotificationHookConsumer: Large Compressed Message Processing Problem
[ https://issues.apache.org/jira/browse/ATLAS-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4155: -- Labels: consumer hooks (was: ) > NotificationHookConsumer: Large Compressed Message Processing Problem > - > > Key: ATLAS-4155 > URL: https://issues.apache.org/jira/browse/ATLAS-4155 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Labels: consumer, hooks > Attachments: ATLAS-4155-Kafka-commit-supplied-offset.patch > > > *Background* > Notification messages can be large in size. To get over Kafka's limitation on > message size, Atlas has compressed and split messages. If message size goes > beyond stipulated threshold, the message is compressed. If compressed message > goes beyond the size, it is split into multiple messages. > *Situation* > Consider a message that is so large that uncompressing it takes longer than > Kafka's timeout for message. This causes the problem where the large message > offset is not committed in time and that causes Kafka to present the same > message again. > Message Description: > Number of splits: 8 > Compressed message size: 7,452,640 > Uncompressed message size: 520,803,946 > Time taken to uncompress and stitch messages: > 90 seconds > > Sequence: > 2021-02-10 14:57:24,221: first message received > 2021-02-10 14:58:36,052: all splits combined – 72 seconds > 2021-02-10 15:01:06,971: message processing completed – 90 seconds > 2021-02-10 15:01:17,158: Kafka commit failed. Elapsed time since first > message: 197 seconds > 2021-02-10 15:01:19,857: attempt #2: first message received > 2021-02-10 15:03:01,993: attempt #2: all splits combined – 102 seconds > 2021-02-10 15:04:44,896: attempt #2: Kafka commit failed. Elapsed time since > first message: 205 seconds > Back to #5 > *Solution* > Maintain last offset received. If the same offset is presented, commit the > offset and move on to the next message. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4155) NotificationHookConsumer: Large Compressed Message Processing Problem
[ https://issues.apache.org/jira/browse/ATLAS-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4155: -- Component/s: atlas-core > NotificationHookConsumer: Large Compressed Message Processing Problem > - > > Key: ATLAS-4155 > URL: https://issues.apache.org/jira/browse/ATLAS-4155 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Attachments: ATLAS-4155-Kafka-commit-supplied-offset.patch > > > *Background* > Notification messages can be large in size. To get over Kafka's limitation on > message size, Atlas has compressed and split messages. If message size goes > beyond stipulated threshold, the message is compressed. If compressed message > goes beyond the size, it is split into multiple messages. > *Situation* > Consider a message that is so large that uncompressing it takes longer than > Kafka's timeout for message. This causes the problem where the large message > offset is not committed in time and that causes Kafka to present the same > message again. > Message Description: > Number of splits: 8 > Compressed message size: 7,452,640 > Uncompressed message size: 520,803,946 > Time taken to uncompress and stitch messages: > 90 seconds > > Sequence: > 2021-02-10 14:57:24,221: first message received > 2021-02-10 14:58:36,052: all splits combined – 72 seconds > 2021-02-10 15:01:06,971: message processing completed – 90 seconds > 2021-02-10 15:01:17,158: Kafka commit failed. Elapsed time since first > message: 197 seconds > 2021-02-10 15:01:19,857: attempt #2: first message received > 2021-02-10 15:03:01,993: attempt #2: all splits combined – 102 seconds > 2021-02-10 15:04:44,896: attempt #2: Kafka commit failed. Elapsed time since > first message: 205 seconds > Back to #5 > *Solution* > Maintain last offset received. If the same offset is presented, commit the > offset and move on to the next message. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4155) NotificationHookConsumer: Large Compressed Message Processing Problem
[ https://issues.apache.org/jira/browse/ATLAS-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4155: -- Affects Version/s: 2.1.0 > NotificationHookConsumer: Large Compressed Message Processing Problem > - > > Key: ATLAS-4155 > URL: https://issues.apache.org/jira/browse/ATLAS-4155 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Attachments: ATLAS-4155-Kafka-commit-supplied-offset.patch > > > *Background* > Notification messages can be large in size. To get over Kafka's limitation on > message size, Atlas has compressed and split messages. If message size goes > beyond stipulated threshold, the message is compressed. If compressed message > goes beyond the size, it is split into multiple messages. > *Situation* > Consider a message that is so large that uncompressing it takes longer than > Kafka's timeout for message. This causes the problem where the large message > offset is not committed in time and that causes Kafka to present the same > message again. > Message Description: > Number of splits: 8 > Compressed message size: 7,452,640 > Uncompressed message size: 520,803,946 > Time taken to uncompress and stitch messages: > 90 seconds > > Sequence: > 2021-02-10 14:57:24,221: first message received > 2021-02-10 14:58:36,052: all splits combined – 72 seconds > 2021-02-10 15:01:06,971: message processing completed – 90 seconds > 2021-02-10 15:01:17,158: Kafka commit failed. Elapsed time since first > message: 197 seconds > 2021-02-10 15:01:19,857: attempt #2: first message received > 2021-02-10 15:03:01,993: attempt #2: all splits combined – 102 seconds > 2021-02-10 15:04:44,896: attempt #2: Kafka commit failed. Elapsed time since > first message: 205 seconds > Back to #5 > *Solution* > Maintain last offset received. If the same offset is presented, commit the > offset and move on to the next message. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4163) Upgrade elasticsearch version to 6.8.14
[ https://issues.apache.org/jira/browse/ATLAS-4163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4163: -- Affects Version/s: 2.1.0 > Upgrade elasticsearch version to 6.8.14 > --- > > Key: ATLAS-4163 > URL: https://issues.apache.org/jira/browse/ATLAS-4163 > Project: Atlas > Issue Type: Task >Affects Versions: 2.1.0 >Reporter: Kevin Risden >Priority: Minor > Attachments: ATLAS-4163.patch > > > ATLAS-4052 upgraded to Elasticsearch 6.8.3 which isn't the latest. There are > some CVEs associated with Elasticsearch 6.8.3 and so should upgrade to 6.8.14 > which is the latest as of Feb 2021. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 73181: ATLAS-4146 [Atlas: Glossary] On updating the related terms, UI allows the user to add the already added term
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73181/#review222599 --- Ship it! Ship It! - Sarath Subramanian On Feb. 10, 2021, 5 a.m., Mandar Ambawane wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73181/ > --- > > (Updated Feb. 10, 2021, 5 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, > Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-4146 > https://issues.apache.org/jira/browse/ATLAS-4146 > > > Repository: atlas > > > Description > --- > > This patch checks whether any duplicate Related Glossary Terms present in the > payload sent by UI for Update operation. > > This check is done for all the GlossaryTerm Relations. > > If duplicate entry is found then Exception is thrown. > > > Diffs > - > > repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java > 82243e2 > > > Diff: https://reviews.apache.org/r/73181/diff/1/ > > > Testing > --- > > Manual Testing is done > > > Thanks, > > Mandar Ambawane > >
Re: Review Request 73182: ATLAS-4139 [Atlas: Glossary Term Bulk Import] When AdditionalAttributes is not provided in key:value format, the error message does not convey that.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73182/#review222598 --- Ship it! Ship It! - Sarath Subramanian On Feb. 15, 2021, 11:20 p.m., Mandar Ambawane wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73182/ > --- > > (Updated Feb. 15, 2021, 11:20 p.m.) > > > Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, > and Sarath Subramanian. > > > Bugs: ATLAS-4135 and ATLAS-4139 > https://issues.apache.org/jira/browse/ATLAS-4135 > https://issues.apache.org/jira/browse/ATLAS-4139 > > > Repository: atlas > > > Description > --- > > This patch changes the error message > > This patch ignore the glossaries and terms that are already present instead > of failing the bulk import > > > Diffs > - > > repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java > 82243e2 > repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java > 2c84ec7 > > > Diff: https://reviews.apache.org/r/73182/diff/2/ > > > Testing > --- > > Basic Testing is done > > > Thanks, > > Mandar Ambawane > >
Re: Review Request 73176: ATLAS-4147 : [Atlas: Glossary Term Bulk Import] When un_supported file type is passed error message displays file type {0}/ATLAS-4148 : [Atlas: Glossary Term Bulk Import] Inc
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73176/#review222597
---
Ship it!
Ship It!
- Sarath Subramanian
On Feb. 16, 2021, 12:18 a.m., chaitali wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73176/
> ---
>
> (Updated Feb. 16, 2021, 12:18 a.m.)
>
>
> Review request for atlas, Jayendra Parab, Nikhil Bonte, Nixon Rodrigues, and
> Sarath Subramanian.
>
>
> Bugs: ATLAS-4147
> https://issues.apache.org/jira/browse/ATLAS-4147
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Pass unsupported file type to the glossary term bulk import
>
> curl --request POST -u user:password
> 'https:///api/atlas/v2/glossary/import' \
> -H 'Accept: application/json' \
> -H 'Cache-Control: no-cache' \
> -H 'X-Requested-With: XMLHttpRequest' \
> -H 'X-XSRF-HEADER: ""' \
> -H 'Content-Type: multipart/form-data' \
> -F file=@abc.py \
> --compressed \
> --insecure
>
> Expectation:
> Instead of "file type {0}" actual file type should be displayed
>
> Suggestion:
>
> Along with the above message, display the supported file format too
>
> ATLAS-4148:
>
> Steps to repro:
>
> GlossaryName, TermName, ShortDescription, LongDescription, Examples,
> Abbreviation, Usage, AdditionalAttributes, TranslationTerms, ValidValuesFor,
> Synonyms, ReplacedBy, ValidValues, ReplacementTerms, SeeAlso,
> TranslatedTerms, IsA, Antonyms, Classifies, PreferredToTerms, PreferredTerms
> This file is written for testing supported format with incorrect data format.
> So this file contains some random text.
>
> When this file is used as an input, the error code displayed is incorrect:
>
> {
> "errorCode":"ATLAS-404-00-014",
> "errorMessage":"No data found in the uploaded file"
> }
> Expectation:
>
> Since this is client side error 400 should be displayed
>
> ATLAS-4134:
> Currently we display the following error message for conflicts:
>
> {
> "errorCode":"ATLAS-409-00-007",
> "errorMessage":"Glossary with qualifiedName abc already exists"
> }
> It will be user friendly if the message is "Glossary with name abc already
> exists"
>
>
> Diffs
> -
>
> intg/src/main/java/org/apache/atlas/AtlasErrorCode.java ad67fa917
> repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
> d630f669e
> repository/src/main/java/org/apache/atlas/util/FileUtils.java 71ac9e309
> repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java
> 54e00a4bd
>
>
> Diff: https://reviews.apache.org/r/73176/diff/4/
>
>
> Testing
> ---
>
> When passed above request:
> {"errorCode":"ATLAS-400-00-098","errorMessage":"The provided file type:
> template_2122.py is not supported expected file formats .csv, .xls."
>
>
> Thanks,
>
> chaitali
>
>
Re: Review Request 73076: Deferred Actions Implementation
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73076/#review222588
---
common/src/main/java/org/apache/atlas/repository/Constants.java
Lines 97 (patched)
<https://reviews.apache.org/r/73076/#comment311699>
register this new property - 'PENDING_PROPAGATIONS' as SET type in
GraphBackedSearchIndexer ('createCommonVertexIndex')
intg/src/main/java/org/apache/atlas/model/tasks/TaskDef.java
Lines 56 (patched)
<https://reviews.apache.org/r/73076/#comment311700>
consider making 'parameters' of Map type. serialize to string during
writing to vertex (TaskRegistry.createVertex()) and deserialize when reading
from vertex (TaskRegistry.toTaskDef()).
intg/src/main/java/org/apache/atlas/type/AtlasType.java
Lines 147 (patched)
<https://reviews.apache.org/r/73076/#comment311702>
remove unused method - 'fromLinkedHashMap', also remove usunsed method
AtlasJson.fromLinkedHashMap()
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
Lines 2780 (patched)
<https://reviews.apache.org/r/73076/#comment311701>
for LIST/SET type attributes, use 'AtlasGraphUtilsV2.addEncodedProperty()'
to add multi-values to vertex. Please review.
for (String pendingPropagationGuid :
classification.getPendingPropagations()) {
AtlasGraphUtilsV2.addEncodedProperty(classificationVertex,
PENDING_PROPAGATIONS, classification.getPendingPropagations());
}
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java
Lines 30 (patched)
<https://reviews.apache.org/r/73076/#comment311698>
"classificationVertexIds" => "classificationVertexId"
- Sarath Subramanian
On Feb. 11, 2021, 11:12 a.m., Ashutosh Mestry wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73076/
> ---
>
> (Updated Feb. 11, 2021, 11:12 a.m.)
>
>
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte,
> Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3919
> https://issues.apache.org/jira/browse/ATLAS-3919
>
>
> Repository: atlas
>
>
> Description
> ---
>
> **Approach**
> Model: _TaskDef_
> - Definition of task.
> - Contains status of the task for querying.
>
> _AbstractTask_
> - Takes TaskDef as input and calls run.
> - The concrete tasks has logic for doing real work.
>
> _TaskFactory_
>
> - Creates tasks given task type.
> - Enumerates all the tasks that are supported.
>
> _TaskManagement_
>
> - Component/Service. Constructed after all existing components are
> constructed.
> - Provides a simple interface for adding tasks.
> - Processes tasks that are in pending state.
> - Supports HA mode, where the tasks are not executed on passive instance.
>
> _TaskManagement.Registry_
>
> - CRUD operations with TaskDef vertex.
>
> _TaskManagement.Processor_
> - Provides mechanism for provides single queue for processing tasks in a way
> that is non-blocking.
> - Uses task type and the associated TaskFactory to construct concrete task.
>
> Changes to _EntityGraphMapper_
> There are 3 new methods added that use portion of existing classification
> propagation logic and create a separate method.
>
> New class _EntityGraphMapperWithTasks_ acts as a proxy for EntityGraphMapper.
> All calls that operate without tasks get channeled to the existing
> implementation. The ones involving tasks are modified to use the new task
> framework.
>
> **Concurrent Changes to Classifications**
> Via REST APIs and operations from web UI, it is possible to make changes to
> the same entity. This situation will be more pronounced in the new setup.
>
> The existing framework has
> _GraphTransactionInterceptor.lockObjectAndReleasePostCommit_. This provides
> ReentrantLock on an entity that is identified by the supplied GUID. This will
> cause a lock on the entity and all the impacted vertices. This has potential
> for adversely impacting scalability.
>
> **New Updates**
> - Separate log file for tasks.
> - TaskDef now has _startTime_ and _endTime_ to indicating start and end of
> task.
> - Refactored parameter passing during task creation. It now accepts Map
> instead of Object array. Each task creates its own set of parameters.
> - Reduced use of TASKS_ENABLE flag checks.
>
>
> Diffs
> -
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 61a
Re: Review Request 73181: ATLAS-4146 [Atlas: Glossary] On updating the related terms, UI allows the user to add the already added term
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73181/#review222584 --- repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java Lines 1055 (patched) <https://reviews.apache.org/r/73181/#comment311697> why qualifiedName needs to be sent back? if 'duplicateTermHeaders' has any value then this confirms the terms are duplicated? Please review. - Sarath Subramanian On Feb. 10, 2021, 5 a.m., Mandar Ambawane wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73181/ > --- > > (Updated Feb. 10, 2021, 5 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, > Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-4146 > https://issues.apache.org/jira/browse/ATLAS-4146 > > > Repository: atlas > > > Description > --- > > This patch checks whether any duplicate Related Glossary Terms present in the > payload sent by UI for Update operation. > > This check is done for all the GlossaryTerm Relations. > > If duplicate entry is found then Exception is thrown. > > > Diffs > - > > repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java > 82243e2 > > > Diff: https://reviews.apache.org/r/73181/diff/1/ > > > Testing > --- > > Manual Testing is done > > > Thanks, > > Mandar Ambawane > >
Re: Review Request 73181: ATLAS-4146 [Atlas: Glossary] On updating the related terms, UI allows the user to add the already added term
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73181/#review222583 --- Mandar, this check should be handled from UI side - not to allow saving an already assigned 'Term'. - Sarath Subramanian On Feb. 10, 2021, 5 a.m., Mandar Ambawane wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73181/ > --- > > (Updated Feb. 10, 2021, 5 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, > Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-4146 > https://issues.apache.org/jira/browse/ATLAS-4146 > > > Repository: atlas > > > Description > --- > > This patch checks whether any duplicate Related Glossary Terms present in the > payload sent by UI for Update operation. > > This check is done for all the GlossaryTerm Relations. > > If duplicate entry is found then Exception is thrown. > > > Diffs > - > > repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java > 82243e2 > > > Diff: https://reviews.apache.org/r/73181/diff/1/ > > > Testing > --- > > Manual Testing is done > > > Thanks, > > Mandar Ambawane > >
[jira] [Updated] (ATLAS-4127) Reduce number of Hive table calls in Atlas Hook
[ https://issues.apache.org/jira/browse/ATLAS-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4127: -- Fix Version/s: 2.2.0 3.0.0 > Reduce number of Hive table calls in Atlas Hook > --- > > Key: ATLAS-4127 > URL: https://issues.apache.org/jira/browse/ATLAS-4127 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Deep Singh >Assignee: Deep Singh >Priority: Major > Labels: hive-hooks > Fix For: 3.0.0, 2.2.0 > > > In Atlas Hive bridge, while calculating processQualifiedName, all partitions > are processed. > For each partition, a hive call is made to get table details. Since multiple > partitions can belong to the same table, there is a scope for improvement by > avoiding repeated hive calls for the same table information. > This improvement becomes crucial particularly in the case where there are > several partitions for a single table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ATLAS-4127) Reduce number of Hive table calls in Atlas Hook
[ https://issues.apache.org/jira/browse/ATLAS-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian resolved ATLAS-4127. --- Resolution: Fixed > Reduce number of Hive table calls in Atlas Hook > --- > > Key: ATLAS-4127 > URL: https://issues.apache.org/jira/browse/ATLAS-4127 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Deep Singh >Assignee: Deep Singh >Priority: Major > Labels: hive-hooks > Fix For: 3.0.0, 2.2.0 > > > In Atlas Hive bridge, while calculating processQualifiedName, all partitions > are processed. > For each partition, a hive call is made to get table details. Since multiple > partitions can belong to the same table, there is a scope for improvement by > avoiding repeated hive calls for the same table information. > This improvement becomes crucial particularly in the case where there are > several partitions for a single table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 73179: ATLAS-4113: Atlas UI: Entity Lineage Details Box layout rendering and centring issue.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73179/#review222580 --- Ship it! Ship It! - Sarath Subramanian On Feb. 8, 2021, 11:02 p.m., Prasad Pawar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73179/ > --- > > (Updated Feb. 8, 2021, 11:02 p.m.) > > > Review request for atlas, Jayendra Parab, keval bhatt, Nixon Rodrigues, and > Sarath Subramanian. > > > Bugs: ATLAS-4113 > https://issues.apache.org/jira/browse/ATLAS-4113 > > > Repository: atlas > > > Description > --- > > The lineage layout doesn't seem to be centred on the page, also there is an > effect where the lineage appears from top-right. > > This patch has the following fix: > 1) Entity Lineage Detail box layout position fix. > 2) White space from Detail box removed. > 3) Add option Display full name in the Lineage settings panel. > 4) https://issues.apache.org/jira/browse/ATLAS-4074 > > > Diffs > - > > dashboardv2/public/css/scss/graph.scss 19636c5 > dashboardv2/public/js/external_lib/atlas-lineage/dist/index.js e82cb48 > dashboardv2/public/js/external_lib/atlas-lineage/dist/styles.css 2cd29ed > dashboardv2/public/js/templates/graph/LineageLayoutView_tmpl.html f6f61e2 > dashboardv2/public/js/templates/graph/TypeSystemTreeView_tmpl.html 005c061 > dashboardv2/public/js/views/graph/LineageLayoutView.js aa26580 > dashboardv2/public/js/views/graph/TypeSystemTreeView.js 90734a4 > dashboardv3/public/css/scss/graph.scss ee774ce > dashboardv3/public/js/external_lib/atlas-lineage/dist/index.js e82cb48 > dashboardv3/public/js/external_lib/atlas-lineage/dist/styles.css 2cd29ed > dashboardv3/public/js/templates/graph/LineageLayoutView_tmpl.html f6f61e2 > dashboardv3/public/js/templates/graph/TypeSystemTreeView_tmpl.html 005c061 > dashboardv3/public/js/views/graph/LineageLayoutView.js aa26580 > dashboardv3/public/js/views/graph/TypeSystemTreeView.js 90734a4 > > > Diff: https://reviews.apache.org/r/73179/diff/1/ > > > Testing > --- > > Tested Lineage View, Entity details panel layout position change to right, > Entity details panel remove white space, Display full name option in lineage > setting panel shows label in full name or trunc name, Export image selected > node color fix. > > > Thanks, > > Prasad Pawar > >
[jira] [Commented] (ATLAS-4151) FixedBufferList: Change Log Level to Debug
[ https://issues.apache.org/jira/browse/ATLAS-4151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17281889#comment-17281889 ] Sarath Subramanian commented on ATLAS-4151: --- Thanks [~amestry]. +1 > FixedBufferList: Change Log Level to Debug > -- > > Key: ATLAS-4151 > URL: https://issues.apache.org/jira/browse/ATLAS-4151 > Project: Atlas > Issue Type: Bug >Affects Versions: trunk >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Trivial > Fix For: trunk > > Attachments: > ATLAS-4151-FixedBufferList-Change-Log-Level-to-Debug.patch > > > *Background* > Existing implementation emits updates from _FixedBufferList_ indicating inner > working. This fills logs with lot of entries that are not useful for > debugging. > *Solution* > Change the log level to _debug_. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 73176: ATLAS-4147 : [Atlas: Glossary Term Bulk Import] When un_supported file type is passed error message displays "file type {0}"
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73176/#review222576
---
Ship it!
Ship It!
- Sarath Subramanian
On Feb. 8, 2021, 1:38 a.m., chaitali wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73176/
> ---
>
> (Updated Feb. 8, 2021, 1:38 a.m.)
>
>
> Review request for atlas, Jayendra Parab, Nikhil Bonte, and Nixon Rodrigues.
>
>
> Bugs: ATLAS-4147
> https://issues.apache.org/jira/browse/ATLAS-4147
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Pass unsupported file type to the glossary term bulk import
>
> curl --request POST -u user:password
> 'https:///api/atlas/v2/glossary/import' \
> -H 'Accept: application/json' \
> -H 'Cache-Control: no-cache' \
> -H 'X-Requested-With: XMLHttpRequest' \
> -H 'X-XSRF-HEADER: ""' \
> -H 'Content-Type: multipart/form-data' \
> -F file=@abc.py \
> --compressed \
> --insecure
>
> Expectation:
> Instead of "file type {0}" actual file type should be displayed
>
> Suggestion:
>
> Along with the above message, display the supported file format too
>
>
> Diffs
> -
>
> intg/src/main/java/org/apache/atlas/AtlasErrorCode.java ad67fa917
> repository/src/main/java/org/apache/atlas/util/FileUtils.java 71ac9e309
>
>
> Diff: https://reviews.apache.org/r/73176/diff/1/
>
>
> Testing
> ---
>
> When passed above request:
> {"errorCode":"ATLAS-400-00-098","errorMessage":"The provided file type:
> template_2122.py is not supported expected file formats .csv, .xls."
>
>
> Thanks,
>
> chaitali
>
>
[jira] [Updated] (ATLAS-4117) Rename hive_db relationship attributes location
[ https://issues.apache.org/jira/browse/ATLAS-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4117: -- Component/s: atlas-core > Rename hive_db relationship attributes location > --- > > Key: ATLAS-4117 > URL: https://issues.apache.org/jira/browse/ATLAS-4117 > Project: Atlas > Issue Type: Task > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Fix For: 3.0.0, 2.2.0 > > > Rename hive_db relationship attributes as below to make differentiation > between attributes & relationship attributes. > Rename "location" to "locationPath" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4117) Rename hive_db relationship attributes location
[ https://issues.apache.org/jira/browse/ATLAS-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4117: -- Labels: hive-hooks (was: ) > Rename hive_db relationship attributes location > --- > > Key: ATLAS-4117 > URL: https://issues.apache.org/jira/browse/ATLAS-4117 > Project: Atlas > Issue Type: Task > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Labels: hive-hooks > Fix For: 3.0.0, 2.2.0 > > > Rename hive_db relationship attributes as below to make differentiation > between attributes & relationship attributes. > Rename "location" to "locationPath" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4117) Rename hive_db relationship attributes location
[ https://issues.apache.org/jira/browse/ATLAS-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4117: -- Fix Version/s: 2.2.0 3.0.0 > Rename hive_db relationship attributes location > --- > > Key: ATLAS-4117 > URL: https://issues.apache.org/jira/browse/ATLAS-4117 > Project: Atlas > Issue Type: Task >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Fix For: 3.0.0, 2.2.0 > > > Rename hive_db relationship attributes as below to make differentiation > between attributes & relationship attributes. > Rename "location" to "locationPath" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4117) Rename hive_db relationship attributes location
[ https://issues.apache.org/jira/browse/ATLAS-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4117: -- Affects Version/s: 2.1.0 > Rename hive_db relationship attributes location > --- > > Key: ATLAS-4117 > URL: https://issues.apache.org/jira/browse/ATLAS-4117 > Project: Atlas > Issue Type: Task >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > > Rename hive_db relationship attributes as below to make differentiation > between attributes & relationship attributes. > Rename "location" to "locationPath" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4098) Add location relationship attribute for hive_db
[ https://issues.apache.org/jira/browse/ATLAS-4098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4098: -- Fix Version/s: 2.2.0 > Add location relationship attribute for hive_db > --- > > Key: ATLAS-4098 > URL: https://issues.apache.org/jira/browse/ATLAS-4098 > Project: Atlas > Issue Type: New Feature >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Fix For: 3.0.0, 2.2.0 > > > Requirement is to add "location" as one of the relationships to hive_db. With > this we can create a "location" entity based on the type of file storage and > assign it to hive_db relationship attributes. > Since we are creating entities for location we can get more details about the > path. To support location entity with all types of file storage, need to > create a super type for all types of location entities. > For that purpose, introduce new layer between DataSet and other location > entities which are derived from DataSet. Create "Path" entity which should be > derived from "DataSet" entity and "Path" should be a super type of all other > location entities. AWS s3 directory(aws_s3_v2_directory ), Azure ADLS > storage(adls_gen2_directory), HDFS(hdfs_path ) are the few location entities > which will be subtypes of "Path" entity. > This location relationship attribute should be created with endDef1 as > "hive_db" and endDef2 as "Path". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4098) Add location relationship attribute for hive_db
[ https://issues.apache.org/jira/browse/ATLAS-4098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4098: -- Component/s: atlas-core > Add location relationship attribute for hive_db > --- > > Key: ATLAS-4098 > URL: https://issues.apache.org/jira/browse/ATLAS-4098 > Project: Atlas > Issue Type: New Feature > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Fix For: 3.0.0, 2.2.0 > > > Requirement is to add "location" as one of the relationships to hive_db. With > this we can create a "location" entity based on the type of file storage and > assign it to hive_db relationship attributes. > Since we are creating entities for location we can get more details about the > path. To support location entity with all types of file storage, need to > create a super type for all types of location entities. > For that purpose, introduce new layer between DataSet and other location > entities which are derived from DataSet. Create "Path" entity which should be > derived from "DataSet" entity and "Path" should be a super type of all other > location entities. AWS s3 directory(aws_s3_v2_directory ), Azure ADLS > storage(adls_gen2_directory), HDFS(hdfs_path ) are the few location entities > which will be subtypes of "Path" entity. > This location relationship attribute should be created with endDef1 as > "hive_db" and endDef2 as "Path". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4125) Create JavaPatch to add new supertypes to existing entities
[ https://issues.apache.org/jira/browse/ATLAS-4125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4125: -- Fix Version/s: 2.2.0 3.0.0 > Create JavaPatch to add new supertypes to existing entities > --- > > Key: ATLAS-4125 > URL: https://issues.apache.org/jira/browse/ATLAS-4125 > Project: Atlas > Issue Type: Task > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Fix For: 3.0.0, 2.2.0 > > > As part of ATLAS-4098 introduced adding new super type for existing entity > type which is done by new action type "ADD_SUPER_TYPES" for Typedef patches. > With this patch, it will update entityType's definition with new superTypes. > To provide backward compatibility for the above feature, we need to update > __superTypes for below. > 1. All vertices with type of updated entityDef > 2. All vertices of all sub-types of updated entityDef -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4098) Add location relationship attribute for hive_db
[ https://issues.apache.org/jira/browse/ATLAS-4098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4098: -- Labels: hive-hooks model (was: ) > Add location relationship attribute for hive_db > --- > > Key: ATLAS-4098 > URL: https://issues.apache.org/jira/browse/ATLAS-4098 > Project: Atlas > Issue Type: New Feature > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Labels: hive-hooks, model > Fix For: 3.0.0, 2.2.0 > > > Requirement is to add "location" as one of the relationships to hive_db. With > this we can create a "location" entity based on the type of file storage and > assign it to hive_db relationship attributes. > Since we are creating entities for location we can get more details about the > path. To support location entity with all types of file storage, need to > create a super type for all types of location entities. > For that purpose, introduce new layer between DataSet and other location > entities which are derived from DataSet. Create "Path" entity which should be > derived from "DataSet" entity and "Path" should be a super type of all other > location entities. AWS s3 directory(aws_s3_v2_directory ), Azure ADLS > storage(adls_gen2_directory), HDFS(hdfs_path ) are the few location entities > which will be subtypes of "Path" entity. > This location relationship attribute should be created with endDef1 as > "hive_db" and endDef2 as "Path". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4098) Add location relationship attribute for hive_db
[ https://issues.apache.org/jira/browse/ATLAS-4098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4098: -- Fix Version/s: 3.0.0 > Add location relationship attribute for hive_db > --- > > Key: ATLAS-4098 > URL: https://issues.apache.org/jira/browse/ATLAS-4098 > Project: Atlas > Issue Type: New Feature >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Fix For: 3.0.0 > > > Requirement is to add "location" as one of the relationships to hive_db. With > this we can create a "location" entity based on the type of file storage and > assign it to hive_db relationship attributes. > Since we are creating entities for location we can get more details about the > path. To support location entity with all types of file storage, need to > create a super type for all types of location entities. > For that purpose, introduce new layer between DataSet and other location > entities which are derived from DataSet. Create "Path" entity which should be > derived from "DataSet" entity and "Path" should be a super type of all other > location entities. AWS s3 directory(aws_s3_v2_directory ), Azure ADLS > storage(adls_gen2_directory), HDFS(hdfs_path ) are the few location entities > which will be subtypes of "Path" entity. > This location relationship attribute should be created with endDef1 as > "hive_db" and endDef2 as "Path". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4098) Add location relationship attribute for hive_db
[ https://issues.apache.org/jira/browse/ATLAS-4098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4098: -- Affects Version/s: 2.1.0 > Add location relationship attribute for hive_db > --- > > Key: ATLAS-4098 > URL: https://issues.apache.org/jira/browse/ATLAS-4098 > Project: Atlas > Issue Type: New Feature >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > > Requirement is to add "location" as one of the relationships to hive_db. With > this we can create a "location" entity based on the type of file storage and > assign it to hive_db relationship attributes. > Since we are creating entities for location we can get more details about the > path. To support location entity with all types of file storage, need to > create a super type for all types of location entities. > For that purpose, introduce new layer between DataSet and other location > entities which are derived from DataSet. Create "Path" entity which should be > derived from "DataSet" entity and "Path" should be a super type of all other > location entities. AWS s3 directory(aws_s3_v2_directory ), Azure ADLS > storage(adls_gen2_directory), HDFS(hdfs_path ) are the few location entities > which will be subtypes of "Path" entity. > This location relationship attribute should be created with endDef1 as > "hive_db" and endDef2 as "Path". -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 73076: Deferred Actions Implementation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73076/#review222552 --- repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java Lines 2324 (patched) <https://reviews.apache.org/r/73076/#comment311649> 'if' check is duplicated in line 2324 and 2323. review and remove 2324 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java Lines 43 (patched) <https://reviews.apache.org/r/73076/#comment311650> consider using classificationVertexIds as argument to entityGraphMapper.propagateClassifications() and entityGraphMapper.updateClassificationsPropagation(); internally this fires a graph query to get the classification vertex which can fetched with classification verttex id if passed. repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java Lines 83 (patched) <https://reviews.apache.org/r/73076/#comment311651> why is 0th element always passed? should it be classificationVertexIds.get(i) ? repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java Lines 133 (patched) <https://reviews.apache.org/r/73076/#comment311652> duplicated check. This check is already made in the caller - startInternal() review and remove. - Sarath Subramanian On Jan. 25, 2021, 3:08 p.m., Ashutosh Mestry wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73076/ > --- > > (Updated Jan. 25, 2021, 3:08 p.m.) > > > Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, > Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-3919 > https://issues.apache.org/jira/browse/ATLAS-3919 > > > Repository: atlas > > > Description > --- > > **Approach** > Model: _TaskDef_ > - Definition of task. > - Contains status of the task for querying. > > _AbstractTask_ > - Takes TaskDef as input and calls run. > - The concrete tasks has logic for doing real work. > > _TaskFactory_ > > - Creates tasks given task type. > - Enumerates all the tasks that are supported. > > _TaskManagement_ > > - Component/Service. Constructed after all existing components are > constructed. > - Provides a simple interface for adding tasks. > - Processes tasks that are in pending state. > - Supports HA mode, where the tasks are not executed on passive instance. > > _TaskManagement.Registry_ > > - CRUD operations with TaskDef vertex. > > _TaskManagement.Processor_ > - Provides mechanism for provides single queue for processing tasks in a way > that is non-blocking. > - Uses task type and the associated TaskFactory to construct concrete task. > > Changes to _EntityGraphMapper_ > There are 3 new methods added that use portion of existing classification > propagation logic and create a separate method. > > New class _EntityGraphMapperWithTasks_ acts as a proxy for EntityGraphMapper. > All calls that operate without tasks get channeled to the existing > implementation. The ones involving tasks are modified to use the new task > framework. > > **Concurrent Changes to Classifications** > Via REST APIs and operations from web UI, it is possible to make changes to > the same entity. This situation will be more pronounced in the new setup. > > The existing framework has > _GraphTransactionInterceptor.lockObjectAndReleasePostCommit_. This provides > ReentrantLock on an entity that is identified by the supplied GUID. This will > cause a lock on the entity and all the impacted vertices. This has potential > for adversely impacting scalability. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 08d6c9d4a > intg/src/main/java/org/apache/atlas/model/tasks/TaskDef.java PRE-CREATION > intg/src/main/java/org/apache/atlas/type/AtlasType.java b30b483e0 > intg/src/main/java/org/apache/atlas/utils/AtlasJson.java abeddf640 > intg/src/test/resources/atlas-application.properties 7e74d5107 > repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java > 86b369fc9 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java > ce58e9aa4 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > 2cfcc0bc1 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/Ent
[jira] [Updated] (ATLAS-4137) Entity Audit V1 api broken
[ https://issues.apache.org/jira/browse/ATLAS-4137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4137: -- Labels: audits v1_api (was: ) > Entity Audit V1 api broken > -- > > Key: ATLAS-4137 > URL: https://issues.apache.org/jira/browse/ATLAS-4137 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Deep Singh >Assignee: Deep Singh >Priority: Major > Labels: audits, v1_api > Fix For: 3.0.0, 2.2.0 > > > Due to the recent introduction of action type Business-attribute-update and > Custom-attribute-update, Entity Audits V1 api is broken. > Problem is in the conversion of EntityAuditEventV2 to EntityAuditEvent (V1 > event). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4137) Entity Audit V1 api broken
[ https://issues.apache.org/jira/browse/ATLAS-4137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4137: -- Affects Version/s: 2.1.0 > Entity Audit V1 api broken > -- > > Key: ATLAS-4137 > URL: https://issues.apache.org/jira/browse/ATLAS-4137 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Deep Singh >Assignee: Deep Singh >Priority: Major > > Due to the recent introduction of action type Business-attribute-update and > Custom-attribute-update, Entity Audits V1 api is broken. > Problem is in the conversion of EntityAuditEventV2 to EntityAuditEvent (V1 > event). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4137) Entity Audit V1 api broken
[ https://issues.apache.org/jira/browse/ATLAS-4137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4137: -- Fix Version/s: 2.2.0 3.0.0 > Entity Audit V1 api broken > -- > > Key: ATLAS-4137 > URL: https://issues.apache.org/jira/browse/ATLAS-4137 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Deep Singh >Assignee: Deep Singh >Priority: Major > Fix For: 3.0.0, 2.2.0 > > > Due to the recent introduction of action type Business-attribute-update and > Custom-attribute-update, Entity Audits V1 api is broken. > Problem is in the conversion of EntityAuditEventV2 to EntityAuditEvent (V1 > event). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-4114) Atlas UI : Atlas Entity Audit UI changes for Sorting
[
https://issues.apache.org/jira/browse/ATLAS-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17279835#comment-17279835
]
Sarath Subramanian commented on ATLAS-4114:
---
thank you [~prasadpp13]. Changes looks good. +1
> Atlas UI : Atlas Entity Audit UI changes for Sorting
> -
>
> Key: ATLAS-4114
> URL: https://issues.apache.org/jira/browse/ATLAS-4114
> Project: Atlas
> Issue Type: Bug
> Components: atlas-webui
>Affects Versions: 2.1.0
>Reporter: Nixon Rodrigues
>Assignee: Prasad P. Pawar
>Priority: Major
> Labels: audit, sorting
> Fix For: 3.0.0
>
> Attachments:
> 0001-Atlas-4114-UI-Atlas-UI-Atlas-Entity-Audit-UI-changes.patch,
> API_sorting.png
>
>
> In ATLAS-4094 Sorting functionality on Atlas entity audit api was added
> {quote}/api/atlas/v2/entity/\{entity-guid}/audit?sortBy=
> Unknown macro: \{user|action|timestamp}
> =\{asc|desc}=10=20
> {quote}
> Now we can sort entity audits based on the following columns:
> * user
> * action
> * timestamp
> and also enable sort in ascending or descending order with limit and offset.
> This jira is to add UI end changes for Atlas entity audit sorting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4127) Reduce number of Hive table calls in Atlas Hook
[ https://issues.apache.org/jira/browse/ATLAS-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4127: -- Component/s: atlas-core > Reduce number of Hive table calls in Atlas Hook > --- > > Key: ATLAS-4127 > URL: https://issues.apache.org/jira/browse/ATLAS-4127 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Deep Singh >Priority: Major > > In Atlas Hive bridge, while calculating processQualifiedName, all partitions > are processed. > For each partition, a hive call is made to get table details. Since multiple > partitions can belong to the same table, there is a scope for improvement by > avoiding repeated hive calls for the same table information. > This improvement becomes crucial particularly in the case where there are > several partitions for a single table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ATLAS-4127) Reduce number of Hive table calls in Atlas Hook
[ https://issues.apache.org/jira/browse/ATLAS-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian reassigned ATLAS-4127: - Assignee: Deep Singh > Reduce number of Hive table calls in Atlas Hook > --- > > Key: ATLAS-4127 > URL: https://issues.apache.org/jira/browse/ATLAS-4127 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Deep Singh >Assignee: Deep Singh >Priority: Major > Labels: hive-hooks > > In Atlas Hive bridge, while calculating processQualifiedName, all partitions > are processed. > For each partition, a hive call is made to get table details. Since multiple > partitions can belong to the same table, there is a scope for improvement by > avoiding repeated hive calls for the same table information. > This improvement becomes crucial particularly in the case where there are > several partitions for a single table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4127) Reduce number of Hive table calls in Atlas Hook
[ https://issues.apache.org/jira/browse/ATLAS-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4127: -- Labels: hive-hooks (was: ) > Reduce number of Hive table calls in Atlas Hook > --- > > Key: ATLAS-4127 > URL: https://issues.apache.org/jira/browse/ATLAS-4127 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Deep Singh >Priority: Major > Labels: hive-hooks > > In Atlas Hive bridge, while calculating processQualifiedName, all partitions > are processed. > For each partition, a hive call is made to get table details. Since multiple > partitions can belong to the same table, there is a scope for improvement by > avoiding repeated hive calls for the same table information. > This improvement becomes crucial particularly in the case where there are > several partitions for a single table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4127) Reduce number of Hive table calls in Atlas Hook
[ https://issues.apache.org/jira/browse/ATLAS-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4127: -- Affects Version/s: 2.1.0 > Reduce number of Hive table calls in Atlas Hook > --- > > Key: ATLAS-4127 > URL: https://issues.apache.org/jira/browse/ATLAS-4127 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Deep Singh >Priority: Major > > In Atlas Hive bridge, while calculating processQualifiedName, all partitions > are processed. > For each partition, a hive call is made to get table details. Since multiple > partitions can belong to the same table, there is a scope for improvement by > avoiding repeated hive calls for the same table information. > This improvement becomes crucial particularly in the case where there are > several partitions for a single table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 73170: ATLAS-4127 Reduce number of Hive table calls in Atlas Hook
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73170/#review222555 --- Ship it! Ship It! - Sarath Subramanian On Feb. 3, 2021, 6:10 p.m., Deep Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73170/ > --- > > (Updated Feb. 3, 2021, 6:10 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Rajesh > Balamohan, and Sarath Subramanian. > > > Bugs: ATLAS-4127 > https://issues.apache.org/jira/browse/ATLAS-4127 > > > Repository: atlas > > > Description > --- > > In Atlas Hive bridge, while calculating processQualifiedName, all partitions > are processed. > For each partition, a hive call is made to get table details. Since multiple > partitions can belong to the same table, there is a scope for improvement by > avoiding repeated hive calls for the same table information. > This improvement becomes crucial particularly in the case where there are > several partitions for a single table. > > In DWX, query to populate store_sales takes 15 seconds to compile. 10+ > seconds in just Atlas hook for getting table details. > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java > 2b809a637 > > > Diff: https://reviews.apache.org/r/73170/diff/1/ > > > Testing > --- > > > Thanks, > > Deep Singh > >
[jira] [Updated] (ATLAS-4125) Create JavaPatch to add new supertypes to existing entities
[ https://issues.apache.org/jira/browse/ATLAS-4125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4125: -- Component/s: atlas-core > Create JavaPatch to add new supertypes to existing entities > --- > > Key: ATLAS-4125 > URL: https://issues.apache.org/jira/browse/ATLAS-4125 > Project: Atlas > Issue Type: Task > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > > As part of ATLAS-4098 introduced adding new super type for existing entity > type which is done by new action type "ADD_SUPER_TYPES" for Typedef patches. > With this patch, it will update entityType's definition with new superTypes. > To provide backward compatibility for the above feature, we need to update > __superTypes for below. > 1. All vertices with type of updated entityDef > 2. All vertices of all sub-types of updated entityDef -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4125) Create JavaPatch to add new supertypes to existing entities
[ https://issues.apache.org/jira/browse/ATLAS-4125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4125: -- Affects Version/s: 2.1.0 > Create JavaPatch to add new supertypes to existing entities > --- > > Key: ATLAS-4125 > URL: https://issues.apache.org/jira/browse/ATLAS-4125 > Project: Atlas > Issue Type: Task >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > > As part of ATLAS-4098 introduced adding new super type for existing entity > type which is done by new action type "ADD_SUPER_TYPES" for Typedef patches. > With this patch, it will update entityType's definition with new superTypes. > To provide backward compatibility for the above feature, we need to update > __superTypes for below. > 1. All vertices with type of updated entityDef > 2. All vertices of all sub-types of updated entityDef -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4112) Basic Search : Incorrect results when searching with value having special char and contains/beginsWith/endsWith operator
[ https://issues.apache.org/jira/browse/ATLAS-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4112: -- Fix Version/s: 2.2.0 3.0.0 > Basic Search : Incorrect results when searching with value having special > char and contains/beginsWith/endsWith operator > > > Key: ATLAS-4112 > URL: https://issues.apache.org/jira/browse/ATLAS-4112 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Pinal >Assignee: Pinal >Priority: Major > Labels: BasicSearch > Fix For: 3.0.0, 2.2.0 > > > *Problem:* While searching with wildcard search i.e contains, beginsWith, > endsWith operator, which has special characters in search attribute value > gives incorrect results > *Example:* While searching qualifiedName beginsWith > "default.datagen_table_sensitive_725_1." > *Solution:* Remove quotes and wildcard '*' from query > *Steps to reproduce:* > 1. Create a hive_table with name ends with a digit: > datagen_table_sensitive_725_1 > 2. Search By Type: hive_columns and select attribute filter: qualifiedName as > "default.datagen_table_sensitive_725__1." shows no Records found_ but if > qualifiedName is changed to "default.datagen_table_sensitive_725_1" _it shows > results with qualifiedName of column > "default.datagen_table_sensitive_725___1.col1@cm"_ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4112) Basic Search : Incorrect results when searching with value having special char and contains/beginsWith/endsWith operator
[ https://issues.apache.org/jira/browse/ATLAS-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4112: -- Affects Version/s: 2.1.0 > Basic Search : Incorrect results when searching with value having special > char and contains/beginsWith/endsWith operator > > > Key: ATLAS-4112 > URL: https://issues.apache.org/jira/browse/ATLAS-4112 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Pinal >Assignee: Pinal >Priority: Major > Labels: BasicSearch > > *Problem:* While searching with wildcard search i.e contains, beginsWith, > endsWith operator, which has special characters in search attribute value > gives incorrect results > *Example:* While searching qualifiedName beginsWith > "default.datagen_table_sensitive_725_1." > *Solution:* Remove quotes and wildcard '*' from query > *Steps to reproduce:* > 1. Create a hive_table with name ends with a digit: > datagen_table_sensitive_725_1 > 2. Search By Type: hive_columns and select attribute filter: qualifiedName as > "default.datagen_table_sensitive_725__1." shows no Records found_ but if > qualifiedName is changed to "default.datagen_table_sensitive_725_1" _it shows > results with qualifiedName of column > "default.datagen_table_sensitive_725___1.col1@cm"_ -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 73160: ATLAS-4112 : Basic Search : Incorrect results when searching with value having special char and contains/beginsWith/endsWith operator
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73160/#review222551 --- Ship it! Ship It! - Sarath Subramanian On Feb. 3, 2021, 8:16 a.m., Pinal Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73160/ > --- > > (Updated Feb. 3, 2021, 8:16 a.m.) > > > Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, > and Sarath Subramanian. > > > Bugs: ATLAS-4112 > https://issues.apache.org/jira/browse/ATLAS-4112 > > > Repository: atlas > > > Description > --- > > **Problem:** While searching with wildcard search i.e contains, beginsWith, > endsWith operator, which has escape characters in search attribute value > gives incorrect results > > **Example:** While searching qualifiedName beginsWith > "default.datagen_table_sensitive_725_1." > > **Solution:** Remove quotes and wildcard '*' from query > > **Steps to reproduce:** > 1. Create a hive_table with name ends with a digit: > datagen_table_sensitive_725_1 > 2. Search By Type: hive_columns and select attribute filter: qualifiedName as > "default.datagen_table_sensitive_725_1." shows no Records found but if > qualifiedName is changed to "default.datagen_table_sensitive_725_1" it shows > results with qualifiedName of column > "default.datagen_table_sensitive_725_1.col1@cm" > > > Diffs > - > > intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 050980994 > repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java > 01daf53d2 > > repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java > 0da60d37f > > > Diff: https://reviews.apache.org/r/73160/diff/3/ > > > Testing > --- > > Added unit test cases > Precommit: > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/353/console > > > Thanks, > > Pinal Shah > >
[jira] [Updated] (ATLAS-4123) Lineage missing in case of CREATE MATERIALIZED VIEW query at Hive
[ https://issues.apache.org/jira/browse/ATLAS-4123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4123: -- Component/s: atlas-core > Lineage missing in case of CREATE MATERIALIZED VIEW query at Hive > - > > Key: ATLAS-4123 > URL: https://issues.apache.org/jira/browse/ATLAS-4123 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: ATLAS-4123.patch > > > Issue: > At hive run the below queries: > create table test_1 (id integer); > CREATE MATERIALIZED VIEW test_mv_ctas_1 as select * from test_1; > Expected behaviour: > At Atlas we should have the lineage created between hive_table entity test_1 > and test_mv_ctas_1 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4123) Lineage missing in case of CREATE MATERIALIZED VIEW query at Hive
[ https://issues.apache.org/jira/browse/ATLAS-4123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4123: -- Fix Version/s: 2.2.0 3.0.0 > Lineage missing in case of CREATE MATERIALIZED VIEW query at Hive > - > > Key: ATLAS-4123 > URL: https://issues.apache.org/jira/browse/ATLAS-4123 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: ATLAS-4123.patch > > > Issue: > At hive run the below queries: > create table test_1 (id integer); > CREATE MATERIALIZED VIEW test_mv_ctas_1 as select * from test_1; > Expected behaviour: > At Atlas we should have the lineage created between hive_table entity test_1 > and test_mv_ctas_1 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4123) Lineage missing in case of CREATE MATERIALIZED VIEW query at Hive
[ https://issues.apache.org/jira/browse/ATLAS-4123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4123: -- Labels: hivehook (was: ) > Lineage missing in case of CREATE MATERIALIZED VIEW query at Hive > - > > Key: ATLAS-4123 > URL: https://issues.apache.org/jira/browse/ATLAS-4123 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > Labels: hivehook > Fix For: 3.0.0, 2.2.0 > > Attachments: ATLAS-4123.patch > > > Issue: > At hive run the below queries: > create table test_1 (id integer); > CREATE MATERIALIZED VIEW test_mv_ctas_1 as select * from test_1; > Expected behaviour: > At Atlas we should have the lineage created between hive_table entity test_1 > and test_mv_ctas_1 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4123) Lineage missing in case of CREATE MATERIALIZED VIEW query at Hive
[ https://issues.apache.org/jira/browse/ATLAS-4123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4123: -- Affects Version/s: 2.1.0 > Lineage missing in case of CREATE MATERIALIZED VIEW query at Hive > - > > Key: ATLAS-4123 > URL: https://issues.apache.org/jira/browse/ATLAS-4123 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Sidharth Kumar Mishra >Assignee: Sidharth Kumar Mishra >Priority: Major > Attachments: ATLAS-4123.patch > > > Issue: > At hive run the below queries: > create table test_1 (id integer); > CREATE MATERIALIZED VIEW test_mv_ctas_1 as select * from test_1; > Expected behaviour: > At Atlas we should have the lineage created between hive_table entity test_1 > and test_mv_ctas_1 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 73169: ATLAS-4123: Lineage missing in case of CREATE MATERIALIZED VIEW query at Hive
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73169/#review222548
---
Ship it!
Ship It!
- Sarath Subramanian
On Feb. 2, 2021, 5:52 p.m., Sidharth Mishra wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73169/
> ---
>
> (Updated Feb. 2, 2021, 5:52 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Deep Singh, Madhan Neethiraj,
> Radhika Kundam, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4123
> https://issues.apache.org/jira/browse/ATLAS-4123
>
>
> Repository: atlas
>
>
> Description
> ---
>
> ATLAS-4123: Lineage missing in case of CREATE MATERIALIZED VIEW query at Hive
>
>
> Diffs
> -
>
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
> e48967d63
>
>
> Diff: https://reviews.apache.org/r/73169/diff/1/
>
>
> Testing
> ---
>
> Atlas hook topic message from the hive hook before fix:
> ===
>
> {
> "version": {
> "version": "1.0.0",
> "versionParts": [
> 1
> ]
> },
> "msgCompressionKind": "NONE",
> "msgSplitIdx": 1,
> "msgSplitCount": 1,
> "msgSourceIP": "172.27.10.2",
> "msgCreatedBy": "hive",
> "msgCreationTime": 1612311707304,
> "message": {
> "type": "ENTITY_CREATE_V2",
> "user": "hive",
> "entities": {
> "referredEntities": {
> "-73285185733726514": {
> "typeName": "hive_storagedesc",
> "attributes": {
> "qualifiedName": "default.test_mv_ctas_2@cm_storage",
> "storedAsSubDirectories": false,
> "location":
> "hdfs://sarath-secure-2.sarath-secure.root.hwx.site:8020/warehouse/tablespace/managed/hive/test_mv_ctas_2",
> "compressed": false,
> "inputFormat":
> "org.apache.hadoop.hive.ql.io.orc.OrcInputFormat",
> "parameters": {},
> "outputFormat":
> "org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat",
> "serdeInfo": {
> "typeName": "hive_serde",
> "attributes": {
> "serializationLib":
> "org.apache.hadoop.hive.ql.io.orc.OrcSerde",
> "name": null,
> "parameters": {}
> }
> },
> "numBuckets": -1
> },
> "guid": "-73285185733726514",
> "isIncomplete": false,
> "provenanceType": 0,
> "version": 0,
> "relationshipAttributes": {
> "table": {
> "guid": "-73285185733726513",
> "typeName": "hive_table",
> "uniqueAttributes": {
> "qualifiedName": "default.test_mv_ctas_2@cm"
> },
> "relationshipType": "hive_table_storagedesc"
> }
> },
> "proxy": false
> },
> "-73285185733726515": {
> "typeName": "hive_column",
> "attributes": {
> "owner": "admin",
> "qualifiedName": "default.test_mv_ctas_2.id@cm",
> "name": "id",
> "co
[jira] [Commented] (ATLAS-4122) Advanced Search: Fix for within clause with Double Quote Values
[ https://issues.apache.org/jira/browse/ATLAS-4122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17277521#comment-17277521 ] Sarath Subramanian commented on ATLAS-4122: --- thanks for the patch. Looks good. +1 > Advanced Search: Fix for within clause with Double Quote Values > --- > > Key: ATLAS-4122 > URL: https://issues.apache.org/jira/browse/ATLAS-4122 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: trunk, 2.1.0 >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: trunk, 2.1.0 > > Attachments: > ATLAS-4122-Advanced-Search-Literals-with-double-quot.patch > > > *Steps to Duplicate* > # In 'Advance Search' fire a query: _hive_db where name = ["Sales", > "Reporting"]_ > _Expected results:_ With appropriate data being present, the query should > return the correct results. > _Actual results:_ Query is not recognized as valid query and no results are > returned. > Root cause: The change is attributed to the implementation of ATLAS-2932, > where script execution was replaced with traversal. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 73160: ATLAS-4112 : Basic Search : Incorrect results when searching with value having special char and contains/beginsWith/endsWith operator
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73160/#review222539
---
repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
Line 461 (original), 462 (patched)
<https://reviews.apache.org/r/73160/#comment311645>
consider refactoring this method for better readability:
private boolean isIndexSearchable(FilterCriteria filterCriteria,
AtlasStructType structType) throws AtlasBaseException {
String attributeName = filterCriteria.getAttributeName();
String attributeValue = filterCriteria.getAttributeValue();
AtlasType attributeType =
structType.getAttributeType(attributeName);
String typeName = attributeType.getTypeName();
String qualifiedName =
structType.getVertexPropertyName(attributeName);
Set indexedKeys= context.getIndexedKeys();
boolean ret= indexedKeys != null &&
indexedKeys.contains(qualifiedName);
SearchParameters.Operator operator =
filterCriteria.getOperator();
AtlasStructDef.AtlasAttributeDef.IndexType indexType =
structType.getAttributeDef(attributeName).getIndexType();
if (ret) { // index exists
// for string type attributes, don't use index query in the
following cases:
// - operation is NEQ, as it might return fewer entries due
to tokenization of vertex property value
// - value-to-compare has special characters
if (AtlasBaseTypeDef.ATLAS_TYPE_STRING.equals(typeName)) {
if (operator == SearchParameters.Operator.NEQ || operator
== SearchParameters.Operator.NOT_CONTAINS) {
if (LOG.isDebugEnabled()) {
LOG.debug("{} operator found for string attribute
{}, deferring to in-memory or graph query (might cause poor performance)",
operator, qualifiedName);
}
ret = false;
} else if (operator == SearchParameters.Operator.CONTAINS
&& AtlasAttribute.hastokenizeChar(attributeValue) && indexType == null) {
if (LOG.isDebugEnabled()) {
LOG.debug("{} operator found for string (TEXT)
attribute {} and special characters found in filter value {}, deferring to
in-memory or graph query (might cause poor performance)", attributeValue);
}
ret = false;
}
}
}
if (LOG.isDebugEnabled()) {
if (!ret) {
LOG.debug("Not using index query for: attribute='{}',
operator='{}', value='{}'", qualifiedName, operator, attributeValue);
}
}
return ret;
}
repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
Lines 32 (patched)
<https://reviews.apache.org/r/73160/#comment311644>
nit: review/remove unused imports
- Sarath Subramanian
On Jan. 27, 2021, 8:56 p.m., Pinal Shah wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73160/
> ---
>
> (Updated Jan. 27, 2021, 8:56 p.m.)
>
>
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues,
> and Sarath Subramanian.
>
>
> Bugs: ATLAS-4112
> https://issues.apache.org/jira/browse/ATLAS-4112
>
>
> Repository: atlas
>
>
> Description
> ---
>
> **Problem:** While searching with wildcard search i.e contains, beginsWith,
> endsWith operator, which has escape characters in search attribute value
> gives incorrect results
>
> **Example:** While searching qualifiedName beginsWith
> "default.datagen_table_sensitive_725_1."
>
> **Solution:** Remove quotes and wildcard '*' from query
>
> **Steps to reproduce:**
> 1. Create a hive_table with name ends with a digit:
> datagen_table_sensitive_725_1
> 2. Search By Type: hive_columns and select attribute filter: qualifiedName as
> "default.datagen_table_sensitive_725_1." shows no Records found but if
> qualifiedName is changed to "default.datagen_table_sensitive_725_1" it shows
> results with qualifiedName of column
> "default.datagen_table_sensitive_725_1.col1@cm"
>
>
> Diffs
> -
>
> intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 050980994
> repository/src/m
[jira] [Updated] (ATLAS-4052) Upgrade elasticsearch version
[ https://issues.apache.org/jira/browse/ATLAS-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4052: -- Affects Version/s: (was: 3.0.0) 2.1.0 > Upgrade elasticsearch version > - > > Key: ATLAS-4052 > URL: https://issues.apache.org/jira/browse/ATLAS-4052 > Project: Atlas > Issue Type: Improvement >Affects Versions: 2.1.0 >Reporter: chaitali borole >Assignee: chaitali borole >Priority: Major > Fix For: 3.0.0 > > > Currently Atlas is pulling in elasticsearch-rest-client 6.6.0 which is really > old. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4052) Upgrade elasticsearch version
[ https://issues.apache.org/jira/browse/ATLAS-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4052: -- Fix Version/s: 2.2.0 > Upgrade elasticsearch version > - > > Key: ATLAS-4052 > URL: https://issues.apache.org/jira/browse/ATLAS-4052 > Project: Atlas > Issue Type: Improvement >Affects Versions: 2.1.0 >Reporter: chaitali borole >Assignee: chaitali borole >Priority: Major > Fix For: 3.0.0, 2.2.0 > > > Currently Atlas is pulling in elasticsearch-rest-client 6.6.0 which is really > old. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4052) Upgrade elasticsearch version
[ https://issues.apache.org/jira/browse/ATLAS-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4052: -- Component/s: atlas-core > Upgrade elasticsearch version > - > > Key: ATLAS-4052 > URL: https://issues.apache.org/jira/browse/ATLAS-4052 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: chaitali borole >Assignee: chaitali borole >Priority: Major > Fix For: 3.0.0, 2.2.0 > > > Currently Atlas is pulling in elasticsearch-rest-client 6.6.0 which is really > old. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 73147: ATLAS-4052 : Upgrade elasticsearch version
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73147/#review222541 --- Ship it! Ship It! - Sarath Subramanian On Jan. 31, 2021, 10:35 p.m., chaitali wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73147/ > --- > > (Updated Jan. 31, 2021, 10:35 p.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, > Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-4052 > https://issues.apache.org/jira/browse/ATLAS-4052 > > > Repository: atlas > > > Description > --- > > Currently Atlas is pulling in elasticsearch-rest-client 6.6.0. > this patch upgrades elasticsearch version to 6.8.3 > > > Diffs > - > > pom.xml 7bfd68b94 > > > Diff: https://reviews.apache.org/r/73147/diff/1/ > > > Testing > --- > > mvn clean install -DskipTests > mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip > atlas server running. > latest precommit : > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/362/console > > > Thanks, > > chaitali > >
[jira] [Commented] (ATLAS-4115) Ranger Tag Sync via ATLAS
[
https://issues.apache.org/jira/browse/ATLAS-4115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17275227#comment-17275227
]
Sarath Subramanian commented on ATLAS-4115:
---
[https://github.com/apache/atlas/blob/master/addons/models/-Area0/0011-glossary_model.json#L228]
AtlasGlossaryCategory is a valid type within Atlas. Can you check with Ranger
team if the types has been initialized completely within Ranger
> Ranger Tag Sync via ATLAS
> -
>
> Key: ATLAS-4115
> URL: https://issues.apache.org/jira/browse/ATLAS-4115
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Tansheet Izhad
>Priority: Major
>
> Currently trying to do ranger tag sync with a new in-build apache Atlas.
> Using the ATLAS_REST_ENDPOINT for the tag sync.
> But getting the following error:
>
> {code:java}
> 28 Jan 2021 17:17:57 ERROR AtlasRESTTagSource [Thread-7] - 287 failed to
> download tags from Atlas
> org.apache.atlas.exception.AtlasBaseException: Given typename
> AtlasGlossaryCategory was invalid
> at
> org.apache.atlas.type.AtlasTypeRegistry.getType(AtlasTypeRegistry.java:108)
> at
> org.apache.atlas.type.AtlasRelationshipType.resolveReferences(AtlasRelationshipType.java:94)
> at
> org.apache.atlas.type.AtlasTypeRegistry$AtlasTransientTypeRegistry.resolveReferences(AtlasTypeRegistry.java:373)
> at
> org.apache.atlas.type.AtlasTypeRegistry$AtlasTransientTypeRegistry.addTypes(AtlasTypeRegistry.java:445)
> at
> org.apache.ranger.tagsync.source.atlasrest.AtlasRESTTagSource.getAtlasActiveEntities(AtlasRESTTagSource.java:284)
> at
> org.apache.ranger.tagsync.source.atlasrest.AtlasRESTTagSource.synchUp(AtlasRESTTagSource.java:217)
> at
> org.apache.ranger.tagsync.source.atlasrest.AtlasRESTTagSource.run(AtlasRESTTagSource.java:200)
> {code}
> Looks like it's failing because of invalid Type. But not able to find
> AtlasGlossaryCategory type on the atlas UI.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (ATLAS-4107) Atlas not picking the ldap bind password from the correct jceks file.
[
https://issues.apache.org/jira/browse/ATLAS-4107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian resolved ATLAS-4107.
---
Resolution: Fixed
> Atlas not picking the ldap bind password from the correct jceks file.
> -
>
> Key: ATLAS-4107
> URL: https://issues.apache.org/jira/browse/ATLAS-4107
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 2.1.0
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Labels: jceks, ldap
> Fix For: 3.0.0, 2.2.0
>
>
> There are two jceks file in Atlas, ldap bind password should be retrieved
> from {{hadoop.security.credential.provider.path}} instead
> {{cert.stores.credential.provider.path}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4107) Atlas not picking the ldap bind password from the correct jceks file.
[
https://issues.apache.org/jira/browse/ATLAS-4107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4107:
--
Labels: jceks ldap (was: )
> Atlas not picking the ldap bind password from the correct jceks file.
> -
>
> Key: ATLAS-4107
> URL: https://issues.apache.org/jira/browse/ATLAS-4107
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 2.1.0
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Labels: jceks, ldap
> Fix For: 3.0.0, 2.2.0
>
>
> There are two jceks file in Atlas, ldap bind password should be retrieved
> from {{hadoop.security.credential.provider.path}} instead
> {{cert.stores.credential.provider.path}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4107) Atlas not picking the ldap bind password from the correct jceks file.
[
https://issues.apache.org/jira/browse/ATLAS-4107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4107:
--
Fix Version/s: 2.2.0
3.0.0
> Atlas not picking the ldap bind password from the correct jceks file.
> -
>
> Key: ATLAS-4107
> URL: https://issues.apache.org/jira/browse/ATLAS-4107
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
>
> There are two jceks file in Atlas, ldap bind password should be retrieved
> from {{hadoop.security.credential.provider.path}} instead
> {{cert.stores.credential.provider.path}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4107) Atlas not picking the ldap bind password from the correct jceks file.
[
https://issues.apache.org/jira/browse/ATLAS-4107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4107:
--
Affects Version/s: 2.1.0
> Atlas not picking the ldap bind password from the correct jceks file.
> -
>
> Key: ATLAS-4107
> URL: https://issues.apache.org/jira/browse/ATLAS-4107
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
>
> There are two jceks file in Atlas, ldap bind password should be retrieved
> from {{hadoop.security.credential.provider.path}} instead
> {{cert.stores.credential.provider.path}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4107) Atlas not picking the ldap bind password from the correct jceks file.
[
https://issues.apache.org/jira/browse/ATLAS-4107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4107:
--
Component/s: atlas-core
> Atlas not picking the ldap bind password from the correct jceks file.
> -
>
> Key: ATLAS-4107
> URL: https://issues.apache.org/jira/browse/ATLAS-4107
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 2.1.0
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
>
> There are two jceks file in Atlas, ldap bind password should be retrieved
> from {{hadoop.security.credential.provider.path}} instead
> {{cert.stores.credential.provider.path}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Review Request 73151: ATLAS-4107 :- Atlas not picking the ldap bind password from the correct jceks file.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73151/#review222538 --- Ship it! Ship It! - Sarath Subramanian On Jan. 28, 2021, 7:11 a.m., Nixon Rodrigues wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73151/ > --- > > (Updated Jan. 28, 2021, 7:11 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, > Nikhil Bonte, and Sarath Subramanian. > > > Bugs: ATLAS-4107 > https://issues.apache.org/jira/browse/ATLAS-4107 > > > Repository: atlas > > > Description > --- > > Atlas not picking the ldap bind password from the correct jceks file. > > There are two jceks file in Atlas configured from CM, ldap bind password > should be retrieved from hadoop.security.credential.provider.path instead > cert.stores.credential.provider.path. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/ApplicationProperties.java e662c8fae > intg/src/main/java/org/apache/atlas/security/SecurityProperties.java > 2147cd17c > intg/src/main/java/org/apache/atlas/security/SecurityUtil.java 082c6378c > > > Diff: https://reviews.apache.org/r/73151/diff/2/ > > > Testing > --- > > Tested on normal cluster with password stored in jceks file and TLS cluster > with password stored in jceks file in Atlas. > > > Thanks, > > Nixon Rodrigues > >
[jira] [Updated] (ATLAS-4111) Add replicationFactor attribute to kafka topic entity
[ https://issues.apache.org/jira/browse/ATLAS-4111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4111: -- Affects Version/s: 2.1.0 > Add replicationFactor attribute to kafka topic entity > - > > Key: ATLAS-4111 > URL: https://issues.apache.org/jira/browse/ATLAS-4111 > Project: Atlas > Issue Type: Improvement > Components: kafka-integration >Affects Versions: 2.1.0 >Reporter: Barnabas Maidics >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Currently, the replicationFactor is missing from kafka topic entities in > Atlas. We should add them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ATLAS-4111) Add replicationFactor attribute to kafka topic entity
[ https://issues.apache.org/jira/browse/ATLAS-4111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian resolved ATLAS-4111. --- Resolution: Fixed > Add replicationFactor attribute to kafka topic entity > - > > Key: ATLAS-4111 > URL: https://issues.apache.org/jira/browse/ATLAS-4111 > Project: Atlas > Issue Type: Improvement > Components: kafka-integration >Affects Versions: 2.1.0 >Reporter: Barnabas Maidics >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Currently, the replicationFactor is missing from kafka topic entities in > Atlas. We should add them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4111) Add replicationFactor attribute to kafka topic entity
[ https://issues.apache.org/jira/browse/ATLAS-4111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4111: -- Fix Version/s: 2.2.0 3.0.0 > Add replicationFactor attribute to kafka topic entity > - > > Key: ATLAS-4111 > URL: https://issues.apache.org/jira/browse/ATLAS-4111 > Project: Atlas > Issue Type: Improvement > Components: kafka-integration >Affects Versions: 2.1.0 >Reporter: Barnabas Maidics >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Currently, the replicationFactor is missing from kafka topic entities in > Atlas. We should add them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 73151: ATLAS-4107 :- Atlas not picking the ldap bind password from the correct jceks file.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73151/#review222534 --- Ship it! Ship It! - Sarath Subramanian On Jan. 21, 2021, 7:58 a.m., Nixon Rodrigues wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73151/ > --- > > (Updated Jan. 21, 2021, 7:58 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, > Nikhil Bonte, and Sarath Subramanian. > > > Bugs: ATLAS-4107 > https://issues.apache.org/jira/browse/ATLAS-4107 > > > Repository: atlas > > > Description > --- > > Atlas not picking the ldap bind password from the correct jceks file. > > There are two jceks file in Atlas configured from CM, ldap bind password > should be retrieved from hadoop.security.credential.provider.path instead > cert.stores.credential.provider.path. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/ApplicationProperties.java e662c8fae > intg/src/main/java/org/apache/atlas/security/SecurityProperties.java > 2147cd17c > intg/src/main/java/org/apache/atlas/security/SecurityUtil.java 082c6378c > notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java > 3d1b3ccf1 > webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java > 7a045749f > > > Diff: https://reviews.apache.org/r/73151/diff/1/ > > > Testing > --- > > Tested on normal cluster with password stored in jceks file and TLS cluster > with password stored in jceks file in Atlas. > > > Thanks, > > Nixon Rodrigues > >
Re: Review Request 73151: ATLAS-4107 :- Atlas not picking the ldap bind password from the correct jceks file.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73151/#review222533
---
intg/src/main/java/org/apache/atlas/security/SecurityUtil.java
Line 42 (original), 43 (patched)
<https://reviews.apache.org/r/73151/#comment311636>
consider retaining existing getPassword() method which internally calls the
new method. If providerPath not specified will fetch creds from cred_store path.
something like:
public static String
getPassword(org.apache.commons.configuration.Configuration config, String key)
throws IOException {
return getPassword(org.apache.commons.configuration.Configuration
config, String key , CERT_STORES_CREDENTIAL_PROVIDER_PATH)
}
This will avoid changes to existing usage of this method (which needs
credstore provider path).
For fetching LDAP creds, you can pass the hadoop provider path explicitly.
- Sarath Subramanian
On Jan. 21, 2021, 7:58 a.m., Nixon Rodrigues wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73151/
> ---
>
> (Updated Jan. 21, 2021, 7:58 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj,
> Nikhil Bonte, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4107
> https://issues.apache.org/jira/browse/ATLAS-4107
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Atlas not picking the ldap bind password from the correct jceks file.
>
> There are two jceks file in Atlas configured from CM, ldap bind password
> should be retrieved from hadoop.security.credential.provider.path instead
> cert.stores.credential.provider.path.
>
>
> Diffs
> -
>
> intg/src/main/java/org/apache/atlas/ApplicationProperties.java e662c8fae
> intg/src/main/java/org/apache/atlas/security/SecurityProperties.java
> 2147cd17c
> intg/src/main/java/org/apache/atlas/security/SecurityUtil.java 082c6378c
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java
> 3d1b3ccf1
> webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java
> 7a045749f
>
>
> Diff: https://reviews.apache.org/r/73151/diff/1/
>
>
> Testing
> ---
>
> Tested on normal cluster with password stored in jceks file and TLS cluster
> with password stored in jceks file in Atlas.
>
>
> Thanks,
>
> Nixon Rodrigues
>
>
Re: Review Request 73157: ATLAS-4110 Inconsistency in the result of DSL query and basic query.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73157/#review222530 --- Ship it! Ship It! - Sarath Subramanian On Jan. 25, 2021, 1:52 p.m., Deep Singh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73157/ > --- > > (Updated Jan. 25, 2021, 1:52 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Pinal Shah, and > Sarath Subramanian. > > > Bugs: ATLAS-4110 > https://issues.apache.org/jira/browse/ATLAS-4110 > > > Repository: atlas > > > Description > --- > > After upgrading Janusgraph to V-0.5.3, behaviour of DSL queries with neq in > where-clause changed. > > /* > Janusgraph 0.5.3 introduced changes to NOT-EQUAL operator > https://github.com/JanusGraph/janusgraph/issues/2205 > Earlier behavior doesn't check if property exists, so all vertices without > this property was also returned in 'neq'. > Now 'neq' checks if property exists and property is not-equal to property > value > */ > > To handle this case I have introduced HAS_NOT_OPERATOR which basically is an > OR operation of HAS_OPERATOR(with neq check) and HAS_NOT_PROPERTY(checks > absence of a property). > > > Diffs > - > > repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java > c7d01cfb1 > repository/src/main/java/org/apache/atlas/query/RegistryBasedLookup.java > 587a0990e > > repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java > be55c8ea2 > repository/src/test/java/org/apache/atlas/BasicTestSetup.java 4be094053 > repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java > 3f4559a47 > > > Diff: https://reviews.apache.org/r/73157/diff/4/ > > > Testing > --- > > Manual testing was done. > Unit test is added. > > > Thanks, > > Deep Singh > >
[jira] [Updated] (ATLAS-4094) Sorting functionality on Atlas entity audit api
[ https://issues.apache.org/jira/browse/ATLAS-4094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4094: -- Fix Version/s: 3.0.0 > Sorting functionality on Atlas entity audit api > --- > > Key: ATLAS-4094 > URL: https://issues.apache.org/jira/browse/ATLAS-4094 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Deep Singh >Assignee: Deep Singh >Priority: Major > Fix For: 3.0.0 > > > We are implementing Atlas audits UI in Data Catalog. For that, we need the > ability to sort the events by given field. Right now there is no such support > in the API. We'll need the ability to sort the events based on timestamp, > action or user. > > The API is documented here - > http://atlas.apache.org/api/v2/resource_EntityREST.html#resource_EntityREST_getAuditEvents_GET -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4094) Sorting functionality on Atlas entity audit api
[ https://issues.apache.org/jira/browse/ATLAS-4094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4094: -- Affects Version/s: 2.1.0 > Sorting functionality on Atlas entity audit api > --- > > Key: ATLAS-4094 > URL: https://issues.apache.org/jira/browse/ATLAS-4094 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Deep Singh >Assignee: Deep Singh >Priority: Major > > We are implementing Atlas audits UI in Data Catalog. For that, we need the > ability to sort the events by given field. Right now there is no such support > in the API. We'll need the ability to sort the events based on timestamp, > action or user. > > The API is documented here - > http://atlas.apache.org/api/v2/resource_EntityREST.html#resource_EntityREST_getAuditEvents_GET -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4094) Sorting functionality on Atlas entity audit api
[ https://issues.apache.org/jira/browse/ATLAS-4094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4094: -- Labels: audit sorting (was: ) > Sorting functionality on Atlas entity audit api > --- > > Key: ATLAS-4094 > URL: https://issues.apache.org/jira/browse/ATLAS-4094 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Deep Singh >Assignee: Deep Singh >Priority: Major > Labels: audit, sorting > Fix For: 3.0.0 > > > We are implementing Atlas audits UI in Data Catalog. For that, we need the > ability to sort the events by given field. Right now there is no such support > in the API. We'll need the ability to sort the events based on timestamp, > action or user. > > The API is documented here - > http://atlas.apache.org/api/v2/resource_EntityREST.html#resource_EntityREST_getAuditEvents_GET -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 73119: ATLAS-4094 Sorting functionality on Atlas entity audit api
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73119/#review222495
---
Ship it!
Ship It!
- Sarath Subramanian
On Jan. 19, 2021, 7:12 a.m., Deep Singh wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73119/
> ---
>
> (Updated Jan. 19, 2021, 7:12 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath
> Subramanian, and Sidharth Mishra.
>
>
> Bugs: ATLAS-4094
> https://issues.apache.org/jira/browse/ATLAS-4094
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Modified the existing API to take few new parameters. If new parameters are
> provided, the result is genrated by in-memory sort.
> new parameters are offset, sortBy, and sortOrder.
>
> /api/atlas/v2/entity/{entity-guid}/audit?sortBy={user|actor|timestamp}={asc|desc}=10=20
>
> HBase does not support query with sorted results. To support this API
> inmemory sort has to be performed.
> Audit entry can potentially have entire entity dumped into it. Loading entire
> audit entries for an entity can be memory intensive. Therefore we load audit
> entries with limited columns first, perform sort on this light weight list,
> then get the relevant section by removing offsets and reducing to limits.
> With this reduced list we create MultiRowRangeFilter and then again scan the
> table to get all the columns from the table this time.
>
>
> Diffs
> -
>
> intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java
> 083acac73
>
> repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
> 8a453fd43
>
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java
> 07784d1c4
>
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
> 9fca74470
>
> repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java
> 900df0205
>
> repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java
> ef9e259ea
>
> repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
> bf4f3950f
> webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 0d6d0c845
>
>
> Diff: https://reviews.apache.org/r/73119/diff/5/
>
>
> Testing
> ---
>
> Manual testing was done.
>
> Did testing on a setup with 1 million audit entries spread across 1000
> entities.
> Existing Rest API took 6-12 milliseconds for preparing the result.
> In-memory sort and double scan approach took 55-75 milliseconds.
> Single Full scan and in-memory approach took 250-300 milliseconds.
>
> As it was expected, the new API is 4X slower than the existing API therefore
> the existing API still should be the primary API for querying audit events.
> And the new API should be used only if sorting is required. Overall the
> server response time of the new API is less than 80 millisecond, compared to
> < 25 milliseconds for the existing API.
>
>
> Thanks,
>
> Deep Singh
>
>
Re: Review Request 73146: ATLAS-4077: replaced print statements in Python files with call to print(), for Python3
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73146/#review222492 --- Ship it! Ship It! - Sarath Subramanian On Jan. 15, 2021, 8:16 p.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73146/ > --- > > (Updated Jan. 15, 2021, 8:16 p.m.) > > > Review request for atlas, Ashutosh Mestry, Deep Singh, Nikhil Bonte, Nixon > Rodrigues, Sarath Subramanian, and Siddharth Singh. > > > Bugs: ATLAS-4077 > https://issues.apache.org/jira/browse/ATLAS-4077 > > > Repository: atlas > > > Description > --- > > Python3 doesn't support print statements; hence replaced it with calls to > print() function > > > Diffs > - > > distro/src/test/python/scripts/TestMetadata.py 251de937c > tools/atlas-index-repair/src/main/resources/repair_index.py bbdc85442 > tools/atlas-migration-exporter/atlas_migration_export.py 591043fcd > > > Diff: https://reviews.apache.org/r/73146/diff/1/ > > > Testing > --- > > - pre-commit tests run: > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/337/ > > > Thanks, > > Madhan Neethiraj > >
Re: Review Request 73119: ATLAS-4094 Sorting functionality on Atlas entity audit api
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73119/#review222480
---
can you include some test coverage for this sorting functionality
- Sarath Subramanian
On Jan. 15, 2021, 9:33 a.m., Deep Singh wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73119/
> ---
>
> (Updated Jan. 15, 2021, 9:33 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath
> Subramanian, and Sidharth Mishra.
>
>
> Bugs: ATLAS-4094
> https://issues.apache.org/jira/browse/ATLAS-4094
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Modified the existing API to take few new parameters. If new parameters are
> provided, the result is genrated by in-memory sort.
> new parameters are offset, sortBy, and sortOrder.
>
> /api/atlas/v2/entity/{entity-guid}/audit?sortBy={user|actor|timestamp}={asc|desc}=10=20
>
> HBase does not support query with sorted results. To support this API
> inmemory sort has to be performed.
> Audit entry can potentially have entire entity dumped into it. Loading entire
> audit entries for an entity can be memory intensive. Therefore we load audit
> entries with limited columns first, perform sort on this light weight list,
> then get the relevant section by removing offsets and reducing to limits.
> With this reduced list we create MultiRowRangeFilter and then again scan the
> table to get all the columns from the table this time.
>
>
> Diffs
> -
>
> intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java
> 083acac73
>
> repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java
> 8a453fd43
>
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java
> 07784d1c4
>
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
> 9fca74470
>
> repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java
> 900df0205
>
> repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java
> ef9e259ea
> webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java 0d6d0c845
>
>
> Diff: https://reviews.apache.org/r/73119/diff/4/
>
>
> Testing
> ---
>
> Manual testing was done.
>
> Did testing on a setup with 1 million audit entries spread across 1000
> entities.
> Existing Rest API took 6-12 milliseconds for preparing the result.
> In-memory sort and double scan approach took 55-75 milliseconds.
> Single Full scan and in-memory approach took 250-300 milliseconds.
>
> As it was expected, the new API is 4X slower than the existing API therefore
> the existing API still should be the primary API for querying audit events.
> And the new API should be used only if sorting is required. Overall the
> server response time of the new API is less than 80 millisecond, compared to
> < 25 milliseconds for the existing API.
>
>
> Thanks,
>
> Deep Singh
>
>
[jira] [Updated] (ATLAS-4099) adls_gen2_directory, Hive Hook : An extra "/" is seen in adls_gen2_directory created by Hook
[
https://issues.apache.org/jira/browse/ATLAS-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4099:
--
Fix Version/s: 2.2.0
3.0.0
> adls_gen2_directory, Hive Hook : An extra "/" is seen in adls_gen2_directory
> created by Hook
>
>
> Key: ATLAS-4099
> URL: https://issues.apache.org/jira/browse/ATLAS-4099
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Sidharth Kumar Mishra
>Assignee: Sidharth Kumar Mishra
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-4099-Removed-slash-from-adls_gen2_directory_2.patch
>
>
> For the query :
> {code:java}
> create external table ext_table(id int) location
> 'abfs://contai...@storageaccount.dfs.core.windows.net/test51' {code}
> adls_gen2_directory created by Hook has QualifiedName :
> *abfs://container@storageaccount/test51/@cm*
> whereas , adls_gen2_directory created by Extractor has QualifiedName:
> *abfs://**container@storageaccount/test51**@cm*
> There is an extra "/" after storage account name which causes issue. Because
> of this , duplicate entities are created.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4099) adls_gen2_directory, Hive Hook : An extra "/" is seen in adls_gen2_directory created by Hook
[
https://issues.apache.org/jira/browse/ATLAS-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4099:
--
Component/s: atlas-core
> adls_gen2_directory, Hive Hook : An extra "/" is seen in adls_gen2_directory
> created by Hook
>
>
> Key: ATLAS-4099
> URL: https://issues.apache.org/jira/browse/ATLAS-4099
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 2.1.0
>Reporter: Sidharth Kumar Mishra
>Assignee: Sidharth Kumar Mishra
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-4099-Removed-slash-from-adls_gen2_directory_2.patch
>
>
> For the query :
> {code:java}
> create external table ext_table(id int) location
> 'abfs://contai...@storageaccount.dfs.core.windows.net/test51' {code}
> adls_gen2_directory created by Hook has QualifiedName :
> *abfs://container@storageaccount/test51/@cm*
> whereas , adls_gen2_directory created by Extractor has QualifiedName:
> *abfs://**container@storageaccount/test51**@cm*
> There is an extra "/" after storage account name which causes issue. Because
> of this , duplicate entities are created.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4099) adls_gen2_directory, Hive Hook : An extra "/" is seen in adls_gen2_directory created by Hook
[
https://issues.apache.org/jira/browse/ATLAS-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4099:
--
Affects Version/s: 2.1.0
> adls_gen2_directory, Hive Hook : An extra "/" is seen in adls_gen2_directory
> created by Hook
>
>
> Key: ATLAS-4099
> URL: https://issues.apache.org/jira/browse/ATLAS-4099
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Sidharth Kumar Mishra
>Assignee: Sidharth Kumar Mishra
>Priority: Major
> Attachments: ATLAS-4099-Removed-slash-from-adls_gen2_directory_2.patch
>
>
> For the query :
> {code:java}
> create external table ext_table(id int) location
> 'abfs://contai...@storageaccount.dfs.core.windows.net/test51' {code}
> adls_gen2_directory created by Hook has QualifiedName :
> *abfs://container@storageaccount/test51/@cm*
> whereas , adls_gen2_directory created by Extractor has QualifiedName:
> *abfs://**container@storageaccount/test51**@cm*
> There is an extra "/" after storage account name which causes issue. Because
> of this , duplicate entities are created.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms and relationship
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73058/#review222468 --- Ship it! Ship It! - Sarath Subramanian On Jan. 13, 2021, 4:31 a.m., Pinal Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73058/ > --- > > (Updated Jan. 13, 2021, 4:31 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, > Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-4057 > https://issues.apache.org/jira/browse/ATLAS-4057 > > > Repository: atlas > > > Description > --- > > Supporting glossary term in advanced search, will allow user to search the > entities which are assigned to particular glossary term. > > > **WorkAround:** > > 'hasTerm' keyword is added to support glossary term. > > User needs to add fully qualifiedName of glossary term. > > > > **Example queries to search with term:** > > hive_table hasTerm "modernTrade@salesGlossary", > > hive_table hasTerm "modernTrade@salesGlossary" and hive_table isA Dimension > > **Example queries to search with relationship:** > > hive_table where db.name = "sales" > > > Diffs > - > > docs/src/documents/Search/SearchAdvanced.md 46be14278 > > repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java > 6c48a82f4 > repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 700da955f > repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java > 36b514e84 > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.g4 > 15d1bb4cb > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java > 142b9cac9 > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.tokens > PRE-CREATION > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 > f1c10600c > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java > 6b33edb53 > > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java > 3139d4374 > > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserListener.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java > a1a727c82 > repository/src/test/java/org/apache/atlas/BasicTestSetup.java d733c1642 > > repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java > 1839e76c7 > repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java > 3bb3b07bf > > repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java > 487cf2778 > > > Diff: https://reviews.apache.org/r/73058/diff/9/ > > > Testing > --- > > Added testcases in DSLQueriesTest.java > Precommit : > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/268/ > > > Thanks, > > Pinal Shah > >
Re: Review Request 73142: ATLAS-4101 : 500 internal server error is thrown instead of 400 malformed request from atlas when the payload is malformed.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73142/#review222460
---
Ship it!
Ship It!
- Sarath Subramanian
On Jan. 13, 2021, 3 a.m., Nixon Rodrigues wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73142/
> ---
>
> (Updated Jan. 13, 2021, 3 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj,
> Nikhil Bonte, Pinal Shah, and Sarath Subramanian.
>
>
> Bugs: ATLAS-4101
> https://issues.apache.org/jira/browse/ATLAS-4101
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Problem: If request json is malformed Atlas server return 500 internal server
> exception instead bad request.
>
> Fix : Fixed http error code for JsonParseException by overriding readFrom
> method of JacksonJsonProvider.
>
>
> Diffs
> -
>
> webapp/src/main/java/org/apache/atlas/web/util/AtlasJsonProvider.java
> efe0486c8
>
>
> Diff: https://reviews.apache.org/r/73142/diff/1/
>
>
> Testing
> ---
>
> Response code is 400 after applying fix.
>
> $$$ curl --header "Accept:application/json" -H "Content-Type:
> application/json" -kv -u admin:admin -X POST
> http://localhost:21000/api/atlas/v2/types/typedefs -d
> '{"classificationDefs":[{"name":"tag_1"},]}'
> Note: Unnecessary use of -X or --request, POST is already inferred.
> * Trying ::1...
> * TCP_NODELAY set
> * Connection failed
> * connect to ::1 port 21000 failed: Connection refused
> * Trying 127.0.0.1...
> * TCP_NODELAY set
> * Connected to localhost (127.0.0.1) port 21000 (#0)
> * Server auth using Basic with user 'admin'
> > POST /api/atlas/v2/types/typedefs HTTP/1.1
> > Host: localhost:21000
> > Authorization: Basic YWRtaW46YWRtaW4=
> > User-Agent: curl/7.54.0
> > Accept:application/json
> > Content-Type: application/json
> > Content-Length: 42
> >
> * upload completely sent off: 42 out of 42 bytes
> < HTTP/1.1 400 Bad Request
> < Date: Wed, 13 Jan 2021 10:55:01 GMT
> < Set-Cookie: ATLASSESSIONID=node01doyctnupiijs15wvr6mvkbxw611.node0; Path=/;
> HttpOnly
> < Expires: Thu, 01 Jan 1970 00:00:00 GMT
> < X-Frame-Options: DENY
> < X-Content-Type-Options: nosniff
> < X-XSS-Protection: 1; mode=block
> < Strict-Transport-Security: max-age=31536000; includeSubDomains
> < Content-Type: application/json;charset=utf-8
> < X-Content-Type-Options: nosniff
> < Content-Security-Policy: default-src 'self'; script-src 'self'
> 'unsafe-inline' 'unsafe-eval' blob: data:; connect-src 'self'; img-src 'self'
> blob: data:; style-src 'self' 'unsafe-inline';font-src 'self' data:
> < Server: Apache Atlas
> < Transfer-Encoding: chunked
> <
> * Connection #0 to host localhost left intact
> {"error":"Unexpected character (']' (code 93)): expected a value\n at
> [Source: (org.eclipse.jetty.server.HttpInputOverHTTP); line: 1, column: 42]\n
> at [Source: (org.eclipse.jetty.server.HttpInputOverHTTP); line: 1, column:
> 41] (through reference chain:
> org.apache.atlas.model.typedef.AtlasTypesDef[\\"classificationDefs\\"])"}
>
>
> Thanks,
>
> Nixon Rodrigues
>
>
Re: Review Request 73076: Deferred Actions Implementation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73076/#review222458 --- repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java Lines 371 (patched) <https://reviews.apache.org/r/73076/#comment311525> line 371-377 can be removed? since this is replaced by pause() webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java Lines 753 (patched) <https://reviews.apache.org/r/73076/#comment311524> API duplicated? line 746/753 - Sarath Subramanian On Jan. 11, 2021, 10:34 p.m., Ashutosh Mestry wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73076/ > --- > > (Updated Jan. 11, 2021, 10:34 p.m.) > > > Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, > Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-3919 > https://issues.apache.org/jira/browse/ATLAS-3919 > > > Repository: atlas > > > Description > --- > > **Approach** > Model: _TaskDef_ > - Definition of task. > - Contains status of the task for querying. > > _AbstractTask_ > - Takes TaskDef as input and calls run. > - The concrete tasks has logic for doing real work. > > _TaskFactory_ > > - Creates tasks given task type. > - Enumerates all the tasks that are supported. > > _TaskManagement_ > > - Component/Service. Constructed after all existing components are > constructed. > - Provides a simple interface for adding tasks. > - Processes tasks that are in pending state. > - Supports HA mode, where the tasks are not executed on passive instance. > > _TaskManagement.Registry_ > > - CRUD operations with TaskDef vertex. > > _TaskManagement.Processor_ > - Provides mechanism for provides single queue for processing tasks in a way > that is non-blocking. > - Uses task type and the associated TaskFactory to construct concrete task. > > Changes to _EntityGraphMapper_ > There are 3 new methods added that use portion of existing classification > propagation logic and create a separate method. > > New class _EntityGraphMapperWithTasks_ acts as a proxy for EntityGraphMapper. > All calls that operate without tasks get channeled to the existing > implementation. The ones involving tasks are modified to use the new task > framework. > > **Concurrent Changes to Classifications** > Via REST APIs and operations from web UI, it is possible to make changes to > the same entity. This situation will be more pronounced in the new setup. > > The existing framework has > _GraphTransactionInterceptor.lockObjectAndReleasePostCommit_. This provides > ReentrantLock on an entity that is identified by the supplied GUID. This will > cause a lock on the entity and all the impacted vertices. This has potential > for adversely impacting scalability. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 > intg/src/main/java/org/apache/atlas/model/tasks/TaskDef.java PRE-CREATION > intg/src/main/java/org/apache/atlas/type/AtlasType.java b30b483e0 > intg/src/main/java/org/apache/atlas/utils/AtlasJson.java abeddf640 > intg/src/test/resources/atlas-application.properties 7e74d5107 > repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java > 86b369fc9 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java > 244072289 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > 2cfcc0bc1 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapperWithTasks.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagateTaskFactory.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationTask.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/tasks/AbstractTask.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/tasks/TaskExecutor.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/tasks/TaskFactory.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java > PRE-C
Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73128/#review222457 --- graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/GraphQueryTest.java Line 50 (original), 50 (patched) <https://reviews.apache.org/r/73128/#comment311522> revert this? repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java Lines 139 (patched) <https://reviews.apache.org/r/73128/#comment311521> refactor line 139-142 to a method getDslQueryExecutor() repository/src/main/java/org/apache/atlas/query/GremlinQuery.java Lines 20 (patched) <https://reviews.apache.org/r/73128/#comment311523> nit: review/remove unused imports repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java Lines 49 (patched) <https://reviews.apache.org/r/73128/#comment311519> SelectClauseInfo => selectClauseInfo repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java Lines 116 (patched) <https://reviews.apache.org/r/73128/#comment311520> 'unique' argument is never used. should the result return only unique entries if this is set? repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java Lines 68 (patched) <https://reviews.apache.org/r/73128/#comment311516> 'end' is never used; consider removing. repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java Lines 80 (patched) <https://reviews.apache.org/r/73128/#comment311517> review and remove TODO? repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java Lines 103 (patched) <https://reviews.apache.org/r/73128/#comment311518> can there be a case where query contains 'offset' and no 'limit' set? - Sarath Subramanian On Jan. 12, 2021, 3:49 p.m., Ashutosh Mestry wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73128/ > --- > > (Updated Jan. 12, 2021, 3:49 p.m.) > > > Review request for atlas, Apoorv Naik, Nikhil Bonte, Nixon Rodrigues, Pinal > Shah, and Sarath Subramanian. > > > Bugs: ATLAS-2932 > https://issues.apache.org/jira/browse/ATLAS-2932 > > > Repository: atlas > > > Description > --- > > **Approach** > General Approach: > - Ensure that Gremlin-based implementation continues to function. This will > ensure backward compatibility and will guarad against regressions or behavior > deviations, if any, are introduced with the new implementation. > - Continue using existing _GremlinQueryComposer_. > - Translate the clauses to _AtlasGraphTraversal_. > - Projects that were earlier handled in Gremlin query are now handled in a > separate Java class. > > Details: > - Modified: _GremlinQueryComposer_: Continue composing queries using existing > claues. Additions alone. > - New: _DSLQueryExecutor_ Instantiates old or new executor based on flag. > - Modifed: _EntityDiscoveryService_ uses the new exectors. > - New _GremlinToTraversalTranslator_: Converts Gremlin clauses to > _AtlasGraphTraversal_. > - Modified: _AtlasGraphTraversal_: Additional methods. > - New: _SelectClauseProjections_: All queries with select operations. > > **Configuration** > _atlas.dsl.executor.traversal_ > - true: Uses the traversal-based implementation for query execution. > - false: Uses the script-engine based implementation for query execution. > > **Credits** > - Apoorv Naik (apoorvnaik): Original implementation. > - Nikhil Bonte (nbonte): Continued on Apoorv's implementation and addressed > short-comings. > > > Diffs > - > > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java > c016f6340 > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphTraversal.java > 881bb1e0f > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java > 0dd573b89 > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphTraversal.java > c33c4f4d0 > > graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/GraphQueryTest.java > 4b0176311 > intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 > > repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java > 01a6c303a > repository/src/main/java/org/apache/atlas/query/AtlasDSL.java b8a744b35 > repository/src/ma
Re: Review Request 73140: ATLAS-4099: Removed slash from adls_gen2_directory
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73140/#review222450 --- Ship it! Ship It! - Sarath Subramanian On Jan. 12, 2021, 1:47 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73140/ > --- > > (Updated Jan. 12, 2021, 1:47 p.m.) > > > Review request for atlas, Ashutosh Mestry, Deep Singh, Madhan Neethiraj, > Radhika Kundam, and Sarath Subramanian. > > > Bugs: ATLAS-4099 > https://issues.apache.org/jira/browse/ATLAS-4099 > > > Repository: atlas > > > Description > --- > > ATLAS-4099: Removed slash from adls_gen2_directory > > > Diffs > - > > common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java > 0e4b507ff > > > Diff: https://reviews.apache.org/r/73140/diff/1/ > > > Testing > --- > > Already tested manually and precommit will be run > > > Thanks, > > Sidharth Mishra > >
Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms and relationship
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73058/#review222445 --- docs/src/documents/Search/SearchAdvanced.md Lines 228 (patched) <https://reviews.apache.org/r/73058/#comment311500> can we add a new section and examples for relationship search? - Sarath Subramanian On Jan. 12, 2021, 3:54 a.m., Pinal Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73058/ > --- > > (Updated Jan. 12, 2021, 3:54 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, > Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-4057 > https://issues.apache.org/jira/browse/ATLAS-4057 > > > Repository: atlas > > > Description > --- > > Supporting glossary term in advanced search, will allow user to search the > entities which are assigned to particular glossary term. > > > **WorkAround:** > > 'hasTerm' keyword is added to support glossary term. > > User needs to add fully qualifiedName of glossary term. > > > > **Example queries to search with term:** > > hive_table hasTerm "modernTrade@salesGlossary", > > hive_table hasTerm "modernTrade@salesGlossary" and hive_table isA Dimension > > **Example queries to search with relationship:** > > hive_table where db.name = "sales" > > > Diffs > - > > docs/src/documents/Search/SearchAdvanced.md 46be14278 > > repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java > 6c48a82f4 > repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 700da955f > repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java > 36b514e84 > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.g4 > 15d1bb4cb > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java > 142b9cac9 > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.tokens > PRE-CREATION > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 > f1c10600c > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java > 6b33edb53 > > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java > 3139d4374 > > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserListener.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java > a1a727c82 > repository/src/test/java/org/apache/atlas/BasicTestSetup.java d733c1642 > > repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java > 1839e76c7 > repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java > 3bb3b07bf > > repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java > 487cf2778 > > > Diff: https://reviews.apache.org/r/73058/diff/8/ > > > Testing > --- > > Added testcases in DSLQueriesTest.java > Precommit : > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/268/ > > > Thanks, > > Pinal Shah > >
Re: Review Request 73058: ATLAS-4057 : DSL Search : Support glossary terms and relationship
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73058/#review222439 --- Fix it, then Ship it! repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java Lines 28 (patched) <https://reviews.apache.org/r/73058/#comment311496> nit: review unused imports and avoid wilcard imports repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java Line 7 (original), 6 (patched) <https://reviews.apache.org/r/73058/#comment311495> avoid wildcard imports and review and remove unused imports - Sarath Subramanian On Dec. 14, 2020, 1:56 a.m., Pinal Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73058/ > --- > > (Updated Dec. 14, 2020, 1:56 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, > Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-4057 > https://issues.apache.org/jira/browse/ATLAS-4057 > > > Repository: atlas > > > Description > --- > > Supporting glossary term in advanced search, will allow user to search the > entities which are assigned to particular glossary term. > > > **WorkAround:** > > 'hasTerm' keyword is added to support glossary term. > > User needs to add fully qualifiedName of glossary term. > > > > **Example queries to search with term:** > > hive_table hasTerm "modernTrade@salesGlossary", > > hive_table hasTerm "modernTrade@salesGlossary" and hive_table isA Dimension > > **Example queries to search with relationship:** > > hive_table where db.name = "sales" > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java > 6c48a82f4 > repository/src/main/java/org/apache/atlas/query/DSLVisitor.java 700da955f > repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java > 249381059 > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.g4 > 15d1bb4cb > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLLexer.java > 142b9cac9 > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.g4 > f1c10600c > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParser.java > 6b33edb53 > > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserBaseVisitor.java > 3139d4374 > > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserListener.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/query/antlr4/AtlasDSLParserVisitor.java > a1a727c82 > repository/src/test/java/org/apache/atlas/BasicTestSetup.java d733c1642 > > repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java > 1839e76c7 > repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java > 3bb3b07bf > > repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java > 487cf2778 > > > Diff: https://reviews.apache.org/r/73058/diff/6/ > > > Testing > --- > > Added testcases in DSLQueriesTest.java > Precommit : > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/268/ > > > Thanks, > > Pinal Shah > >
Re: Review Request 73122: ATLAS-4095 : DSL Search : __isIncomplete System Attribute returns incorrect results or throws error
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73122/#review222435 --- Ship it! Ship It! - Sarath Subramanian On Jan. 7, 2021, 2:25 a.m., Pinal Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73122/ > --- > > (Updated Jan. 7, 2021, 2:25 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, > Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-4095 > https://issues.apache.org/jira/browse/ATLAS-4095 > > > Repository: atlas > > > Description > --- > > **Issue:** With DSL search having 'isIncomplete' System attribute in where > clause, it returns empty results. > > **Example:** "Table where __isIncomplete = false" > > **Workaround:** Need to map boolean value to integer value before querying. > Also need to handle check for the presence of attribute. > > > Diffs > - > > repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java > 249381059 > repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java > 7b03d118b > > > Diff: https://reviews.apache.org/r/73122/diff/2/ > > > Testing > --- > > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/306/ > > > Thanks, > > Pinal Shah > >
[jira] [Updated] (ATLAS-4093) "NullPointerException" is thrown when mandatory attribute is missing endDef-name
[
https://issues.apache.org/jira/browse/ATLAS-4093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4093:
--
Component/s: atlas-core
> "NullPointerException" is thrown when mandatory attribute is missing
> endDef-name
>
>
> Key: ATLAS-4093
> URL: https://issues.apache.org/jira/browse/ATLAS-4093
> Project: Atlas
> Issue Type: Improvement
> Components: atlas-core
>Affects Versions: 2.1.0
>Reporter: chaitali borole
>Assignee: chaitali borole
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
>
> { "relationshipDefs": [
> { "category": "RELATIONSHIP","name": "create_rel_1",
> "relationshipCategory": "ASSOCIATION","endDef1": {"type":
> "rel_type_1"},"endDef2": {"type": "rel_type_2"
> }
> },
> { "category": "RELATIONSHIP","name": "create_rel_2",
> "relationshipCategory": "ASSOCIATION","endDef1": {"type":
> "rel_type_1"},"endDef2": {"type": "rel_type_2"
> }
> }
> ]
> }
> Exception :
> 18:01:23,265 ERROR - [etp238564722-17 -
> ad9e1d0a-2302-40fb-91b0-077f7ddd8f9c:] ~ graph rollback due to exception
> (GraphTransactionInterceptor:192)java.lang.NullPointerException at
> org.apache.atlas.type.AtlasRelationshipType.addRelationshipEdgeDirection(AtlasRelationshipType.java:171)
> at
> org.apache.atlas.type.AtlasRelationshipType.resolveReferencesPhase2(AtlasRelationshipType.java:141)
> at
> org.apache.atlas.type.AtlasTypeRegistry$AtlasTransientTypeRegistry.resolveReferences(AtlasTypeRegistry.java:447)
> at
> org.apache.atlas.type.AtlasTypeRegistry$AtlasTransientTypeRegistry.addTypes(AtlasTypeRegistry.java:517)
> at
> org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStore.tryTypeCreation(AtlasTypeDefGraphStore.java:1193)
> at
> org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStore.createTypesDef(AtlasTypeDefGraphStore.java:395)
> at
> org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStore$$FastClassBySpringCGLIB$$5226c80b.invoke()
> at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> at
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:737)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
> at
> org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:107)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> at
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:672)
> at
> org.apache.atlas.repository.store.graph.v2.AtlasTypeDefGraphStoreV2$$EnhancerBySpringCGLIB$$377d7b16.createTypesDef()
> at
> org.apache.atlas.web.rest.TypesREST.createAtlasTypeDefs(TypesREST.java:381)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
