Re: Review Request 75211: ATLAS-4894: [UI] 'Exclude SubTypes' and 'Exclude Sub-classifications' filter should be removed from 'Tables' tab of hive_db entity details page
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/75211/#review226938 --- Ship it! Ship It! - Pinal Shah On Sept. 18, 2024, 1:58 p.m., Brijesh Bhalala wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/75211/ > --- > > (Updated Sept. 18, 2024, 1:58 p.m.) > > > Review request for atlas, Farhan Khan, Jayendra Parab, Madhan Neethiraj, and > Prasad Pawar. > > > Bugs: ATLAS-4894 > https://issues.apache.org/jira/browse/ATLAS-4894 > > > Repository: atlas > > > Description > --- > > In UI, When we visit the 'Tables' tab section of a hive_db entity (or similar > a entity type), the two filters 'Exclude SubTypes' and 'Exclude > Sub-classifications' (see attached screenshot) are visible, but do not have > any functionality in this page because these only work in the context of > Basic Search. There are no properties being passed to these filters when user > visits the entity details of the page. > > Also from a functional perspective, it doesn't make sense for classifications > or types to be filtered in an entity details page. > > It is better that they should be removed for consistency, as these filters > are already removed from the 'Schema' tab of hive_table entities. > > This fix also include name validation for glossary, term and category. > > > Diffs > - > > dashboardv2/public/js/templates/search/SearchResultLayoutView_tmpl.html > f8cfad616 > dashboardv2/public/js/views/detail_page/DetailPageLayoutView.js 3f62d970f > dashboardv2/public/js/views/glossary/TermPropertiestLayoutView.js e2bb72d20 > dashboardv2/public/js/views/search/RelationSearchResultLayoutView.js > 61176b4f6 > dashboardv2/public/js/views/search/SearchResultLayoutView.js c9e57ad73 > dashboardv3/public/js/templates/search/SearchResultLayoutView_tmpl.html > 6aa3fe185 > dashboardv3/public/js/views/detail_page/DetailPageLayoutView.js 854908da3 > dashboardv3/public/js/views/glossary/TermPropertiestLayoutView.js e2bb72d20 > dashboardv3/public/js/views/search/RelationSearchResultLayoutView.js > f277c4e5f > dashboardv3/public/js/views/search/SearchResultLayoutView.js 16db56623 > intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 77a6fd8c3 > repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java > 87fc2cd49 > repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java > 007b2f24e > > > Diff: https://reviews.apache.org/r/75211/diff/1/ > > > Testing > --- > > Build the Atlas with below command :- > mvn clean package -Pdist,embedded-hbase-solr -DskipDocs -Drat.skip=true > > Valdiated the below scenarios:- > 1)profile table in entity detail page. > 2)CRUD operation for glossary, term and category > > PC has been passed:- > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1694/ > > > Thanks, > > Brijesh Bhalala > >
Re: [DRAFT] [REPORT] Apache Atlas - September 2024
Looks good to me. Thanks Madhan for putting this together. On Thu, Sep 12, 2024 at 8:59 AM Sarath Subramanian wrote: > Thanks Madhan for the draft report. > > +1 > > > On Sep 11, 2024, at 2:51 PM, Madhan Neethiraj wrote: > > > > Atlas community, > > > > > > > > Please review the following draft report for this quarter. > > > > > > > > Thanks, > > > > Madhan > > > > > > > > > > > > > > > > > > > > ## Description: > > > > The mission of Apache Atlas is the creation and maintenance of software > related > > > > to a scalable and extensible set of core foundational governance > services - > > > > enabling enterprises to effectively and efficiently meet their compliance > > > > requirements > > > > > > > > ## Project Status: > > > > Current project status: Ongoing, with low activity > > > > Issues for the board: none > > > > > > > > ## Membership Data: > > > > Apache Atlas was founded 2017-06-20 (7 years ago) > > > > There are currently 48 committers and 32 PMC members in this project. > > > > The Committer-to-PMC ratio is 3:2. > > > > > > > > Community changes, past quarter: > > > > - No new PMC members. Last addition was Ashutosh Mestry on 2019-04-16. > > > > - No new committers. Last addition was Umesh Padashetty on 2023-06-14. > > > > > > > > ## Project Activity: > > > > - fixes and improvements in import functionality > > > > - performance improvement - utility to analyze hook notifications > > > > - improvements in notification processing > > > > - UI fixes in handling of deferred tasks > > > > - updated versions of dependent libraries > > > > > > > > ## Community Health: > > > > - dev@atlas.apache.org had a 21% decrease in traffic in the past > quarter (235 emails compared to 296): > > > > - 10 commits in the past quarter, a decrease from 23 commits the > previous quarter > > > > - 4 code contributors in the past quarter, a decrease from 7 in the > previous quarter > > > > - 2 new contributors were added in the past quarter - Prashant Satam and > Chandra Kanth Peravelli > > > > > > > > ## Recent releases: > > > > - Apache Atlas 2.3.0 was released on 2022-12-06. > > > > - Apache Atlas 2.2.0 was released on 2021-08-17. > > > > - Apache Atlas 2.1.0 was released on 2020-07-15. > > > > > > > > > > >
Re: Review Request 75188: ATLAS-4802 : Atlas 'updateTime' parameter is not updated when business metadata, labels is edited.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/75188/#review226894
---
Ship it!
Ship It!
- Pinal Shah
On Aug. 24, 2024, 9:14 a.m., chaitali wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/75188/
> ---
>
> (Updated Aug. 24, 2024, 9:14 a.m.)
>
>
> Review request for atlas, Jayendra Parab and Pinal Shah.
>
>
> Bugs: ATLAS-4802
> https://issues.apache.org/jira/browse/ATLAS-4802
>
>
> Repository: atlas
>
>
> Description
> ---
>
> When updating the business metadata and Labels on an atlas entity, the
> 'updateTime' system attribute is not changed.
>
>
> Updates to other aspects, such as technical properties and user defined
> attributes does update the timestamp, but not the business metadata and
> labels.
>
>
> Diffs
> -
>
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
> 77ab99a96
>
>
> Diff: https://reviews.apache.org/r/75188/diff/2/
>
>
> Testing
> ---
>
> update time Without patch
> before business metadata update
>
>
> "createTime": 1703221384530,
> "updateTime": 1703221386989,
>
>
> after business metadata update
>
>
> "createTime": 1703221384530,
> "updateTime": 1703221386989,
>
>
> after applying PATCH
>
>
> before business metadata update
> "createTime": 1704190056433,
> "updateTime": 1704190059208,
>
>
> after business metadata update
> "createTime": 1704190056433,
> "updateTime": 1704190523323,
>
>
> Entire json for GET api on entity before applying patch :
>
>
> http://localhost:21000/api/atlas/v2/entity/guid/dc1baa88-3a07-4577-9b6b-48982253aa3d
>
>
> {
> "referredEntities": {
> "a1a27e35-aa0d-472d-8b94-2405d8001a84": {
> "typeName": "Column",
> "attributes": {
> "qualifiedName": "Sales.customer_dim.address@cl1",
> "name": "address",
> "comment": "customer address"
> },
> "guid": "a1a27e35-aa0d-472d-8b94-2405d8001a84",
> "isIncomplete": false,
> "status": "ACTIVE",
> "createdBy": "admin",
> "updatedBy": "admin",
> "createTime": 1704190056433,
> "updateTime": 1704190056433,
> "version": 0,
> "classifications": [
> {
> "typeName": "PII",
> "entityGuid": "a1a27e35-aa0d-472d-8b94-2405d8001a84",
> "entityStatus": "ACTIVE",
> "propagate": true,
> "removePropagationsOnEntityDelete": false
> }
> ],
> "labels": []
> },
> "834ba687-c8f1-4a48-a6a0-ef3ddeadf7ee": {
> "typeName": "Column",
> "attributes": {
> "qualifiedName": "Sales.customer_dim.name@cl1",
> "name": "name",
> "comment": "customer name"
> },
> "guid": "834ba687-c8f1-4a48-a6a0-ef3ddeadf7ee",
> "isIncomplete": false,
> "status": "ACTIVE",
> "createdBy": "admin",
> "updatedBy": "admin",
> "createTime": 1704190056433,
> "updateTime": 1704190056433,
> "version": 0,
> "classifications": [
> {
> "typeName": "PII",
> "entityGuid": "834ba687-c8f1-4a48-a6a0-ef3ddeadf7ee",
> "entityStatus": "ACTIVE",
> "propagate": true,
> "
Re: Review Request 75150: ATLAS-4892 : Export/Import APIs: Unable to import shell entities
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/75150/#review226887 --- Ship it! Ship It! - Pinal Shah On Aug. 23, 2024, 1:47 p.m., Priyanshi Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/75150/ > --- > > (Updated Aug. 23, 2024, 1:47 p.m.) > > > Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, and > Sheetal Shah. > > > Bugs: ATLAS-4892 > https://issues.apache.org/jira/browse/ATLAS-4892 > > > Repository: atlas > > > Description > --- > > Problem: > > During import, if shell entity is present in the importing file, import is > getting failed. > > Solution: > > There is mandatory attribute check present which was not allowing to create > shell entity and the import was getting failed due to this implementation. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 60e57a379 > intg/src/main/java/org/apache/atlas/type/AtlasStructType.java ffbe9e7d8 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityGraphDiscoveryV2.java > b51951fe2 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java > 656c9d14b > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > 77ab99a96 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/IDBasedEntityResolver.java > 2f37eac6b > > > Diff: https://reviews.apache.org/r/75150/diff/6/ > > > Testing > --- > > Tested below scenarios : > > 1. Created shell entity through beeline it is working fine. > 2. Tried with export file which has hive_column as shell entity it is also > working fine. > 3. Had one export file which has shell entity for spark_column_lineage and it > worked fine. > 4. Tried running same exported file which has same shell entity again . > 5. Incremental export file should have entity (shell entity in the earlier > export) which is now resolved to actual entity. > 6. Ran export file without having shell entity. > 7. Created entity through API. > > > Thanks, > > Priyanshi Shah > >
[jira] [Commented] (ATLAS-4884) approximateCount confused
[
https://issues.apache.org/jira/browse/ATLAS-4884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17863122#comment-17863122
]
Pinal Shah commented on ATLAS-4884:
---
[~zhangminguang]
# While using `__customAttribute` in basic search filter, approximateCount is
incorrect, it is a bug, i have created ticket ATLAS-4885, will fix in a day
# For your usecase regarding custom entity type and filtering it with its
attribute, it is possible to search through DSL. but filtering through
`__customAttributes`(UserDefined Attributes) is not supported in DSL
> approximateCount confused
> -
>
> Key: ATLAS-4884
> URL: https://issues.apache.org/jira/browse/ATLAS-4884
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 2.1.0
>Reporter: MinGuang Zhang
>Priority: Major
> Attachments: image-2024-06-26-11-58-18-440.png,
> image-2024-06-26-11-58-50-435.png
>
>
> atlas should return search result total exactly when user post some search
> Conditions, like query string and filter. Is using approximateCount as
> total? but approximateCount make me confused.
> a general scene： i want the total of search result for page i one request.
> but now i should recall api to add limit (eg: 10+) .
> the bad way: to reduce paramaters for get all and return hurge data for
> calculate.
>
> {*}Problem 1{*}:
> the response of post "/search/basic" api.
> see picture1 , i'm not add filter and set search text to find the hive_table
> contains my custom attributes, the approximateCount is correct。 by the way,
> is DSL support find __customAttribute?? how to do?
> picture 1:
> !image-2024-06-26-11-58-18-440.png!
>
> see picture2, i'm add filter and not set search text, the approximateCount
> is incorrect。 i see the correct total is 11 one by one.
> picture2
> !image-2024-06-26-11-58-50-435.png!
> also i can't using the entities size for work when the limit paramters
> changed.
> so how to search use defined attribute for work? use filter or fullText
> search? and is a bug? or need some total variable？
>
> {*}Problem 2{*}:
> Use-defined propertiies can help business . can support dsl search it ? any
> discuss will help me.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (ATLAS-4885) Basic Search : not using index-search for Userdefined attributes
Pinal Shah created ATLAS-4885:
-
Summary: Basic Search : not using index-search for Userdefined
attributes
Key: ATLAS-4885
URL: https://issues.apache.org/jira/browse/ATLAS-4885
Project: Atlas
Issue Type: Bug
Components: atlas-core
Reporter: Pinal Shah
Assignee: Pinal Shah
While searching through basic search having filter for userdefined attribute,
getting below warning log
{code:java}
not using index-search for attribute '__customAttributes'; might cause poor
performance (SearchProcessor:337)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: Review Request 75007: ATLAS-4866: Export/Import: Skip adding processed guids in audit while importing huge zip
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/75007/
---
(Updated June 25, 2024, 5:15 a.m.)
Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj,
and Radhika Kundam.
Changes
---
addressed review comment
Bugs: ATLAS-4866
https://issues.apache.org/jira/browse/ATLAS-4866
Repository: atlas
Description
---
**Background:**
Atlas uses HBase as its store for audit repository.
After import, atlas store the audit entity with the import information along
with all the processed entitiy guids.
**Issue: **
When large sized export zipped file is imported, import gives below error,
internally import gets succesful, but fail creating audit.
*{"errorCode":"ATLAS-500-00-001","errorMessage":"org.janusgraph.core.JanusGraphException:
Could not commit transaction due to exception during
persistence","errorCause":"Could not commit transaction due to exception during
persistence"}*
When size of the entity is greater than "hbase.client.keyvalue.maxsize"
property then audit entity creation fails with exception.
*Caused by:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
action: org.apache.hadoop.hbase.DoNotRetryIOException:
Cell[\x00\x00\x00\x00\x00\x00\x00\x00\x01\x05\xCC\xBB/l:\x00\x06\x18r\xB0\xBE\xFDH\xA00a11ed186467-ve0214-halxg-cloudera-com\xB2\x00\x00\x00\x00\x00\x0D\xB6Y/1715730740890001/Put/vlen=23826488/seqid=0]
with size 23826581 exceeds limit of 10485760 bytes
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.checkCellSizeLimit(RSRpcServices.java:906)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:992)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:927)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:892)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2855)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45961)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:139)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349)
: 1 time, servers with issues: ve0214.halxg.cloudera.com,22101,1715690875185
at
org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:50)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.getErrors(AsyncRequestFutureImpl.java:1228)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:434)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:422)
at org.janusgraph.diskstorage.hbase2.HTable2_0.batch(HTable2_0.java:51)
**Solution: **
Hence in this case, storing processed entities guids is skipped while creating
ExportImportAuditEntry when size of entity goes beyond the value (in bytes) of
below property
atlas.hbase.client.keyvalue.maxsize
Diffs (updated)
-
repository/src/main/java/org/apache/atlas/repository/impexp/ExportImportAuditService.java
3afa17301
Diff: https://reviews.apache.org/r/75007/diff/2/
Changes: https://reviews.apache.org/r/75007/diff/1-2/
Testing
---
manually verified through below api, processed entity guids is not stored
/api/atlas/admin/expimp/audit?userName=admin&operation=IMPORT
Precommit:
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1649/console
Thanks,
Pinal Shah
Re: Review Request 75054: ATLAS-4882: Export/Import: Export exits with 'Found 0 entities'
> On June 18, 2024, 8:04 a.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/util/AtlasGremlin3QueryProvider.java
> > Line 41 (original), 41 (patched)
> > <https://reviews.apache.org/r/75054/diff/1/?file=2289122#file2289122line41>
> >
> > Is removing ".has('__guid')" not needed in line #41, #42, #45?
Above queries are used in VertexExtractor and below is the condition of
extractor strategy
return (atlasEntityDef == null ||
atlasEntityDef.getRelationshipAttributeDefs().size() == 0)
? extractors.get(VERTEX_BASED_EXTRACT)
: extractors.get(RELATION_BASED_EXTRACT);
I was not able to verify it, since couldn't get the usecase of the
vertexExtractor Strategy
- Pinal
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/75054/#review226559
-------
On June 18, 2024, 7:54 a.m., Pinal Shah wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/75054/
> ---
>
> (Updated June 18, 2024, 7:54 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj,
> and Radhika PC.
>
>
> Bugs: ATLAS-4882
> https://issues.apache.org/jira/browse/ATLAS-4882
>
>
> Repository: atlas
>
>
> Description
> ---
>
> please check the jira description
> https://issues.apache.org/jira/browse/ATLAS-4882
>
>
> Diffs
> -
>
>
> repository/src/main/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequest.java
> d01d6775a
>
> repository/src/main/java/org/apache/atlas/util/AtlasGremlin3QueryProvider.java
> 5b10c353e
>
>
> Diff: https://reviews.apache.org/r/75054/diff/1/
>
>
> Testing
> ---
>
> Manually verified
>
>
> Thanks,
>
> Pinal Shah
>
>
[jira] [Updated] (ATLAS-4882) Export/Import: Export exits with "Found 0 entities"
[
https://issues.apache.org/jira/browse/ATLAS-4882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pinal Shah updated ATLAS-4882:
--
Description:
*Issue:*
Export during ingestion fails giving Found 0 entities in the logs
Ingestion meaning Atlas is consuming messages
*Steps to Repro:*
* Make sure backend has above 1M entities
* Start creating tables under `db1@cm`
* Start export for `db1@cm`
*
{code:java}
curl -v -X POST -u admin:admin -H "Content-Type: application/json"
"http://<>/api/atlas/admin/export" -d
'{"itemsToExport":[{"typeName":"hive_db","uniqueAttributes": { "qualifiedName":
"db1@cm" }}],"options":{"fetchType":"full","replicatedTo":"cm"}} >
export1.zip{code}
* It fails after sometime.
*When is the issue seen?*
It occurs when there is huge amount of data in backend and Atlas is consuming
messages linked to entity of which export is running
*Analysis to find Root cause:*
* when there is huge amount of data in backend, export FAILS
* when there is huge amount of data in backend but less tables under it, then
also export FAILS
* if background consumption stops, export PASS
* if consumption is of different entities then requested in export, export PASS
* export query to find starting object uses below query, where has clause to
check property is expensive
{code:java}
g.V().has('_typeName','hive_db').has('Referenceable.qualifiedName','db6@cm').has('__guid').values('__guid'){code}
- has('__guid') queries solr [(35x_t <> null)]:vertex_index
- below is the timetaken in the solr logs
{code:java}
2024-06-14 02:38:56.218 INFO (qtp1158676965-19) [c:vertex_index s:shard1
r:core_node2 x:vertex_index_shard1_replica_n1] o.a.s.c.S.Request
[vertex_index_shard1_replica_n1] webapp=/solr path=/select
params={q=:&stateVer=vertex_index:12&fl=id&start=0&fq=35x_t:*+&rows=50&wt=javabin&version=2}
hits=1681928 status=0 QTime=4227
2024-06-14 02:40:23.945 INFO (qtp1158676965-16) [c:vertex_index s:shard1
r:core_node2 x:vertex_index_shard1_replica_n1] o.a.s.c.S.Request
[vertex_index_shard1_replica_n1] webapp=/solr path=/select
params={q=:&stateVer=vertex_index:12&fl=id&start=50&fq=35x_t:*+&rows=50&wt=javabin&version=2}
hits=1682086 status=0 QTime=787
2024-06-14 02:41:37.703 INFO (qtp1158676965-14) [c:vertex_index s:shard1
r:core_node2 x:vertex_index_shard1_replica_n1] o.a.s.c.S.Request
[vertex_index_shard1_replica_n1] webapp=/solr path=/select
params={q=:&stateVer=vertex_index:12&fl=id&start=100&fq=35x_t:*+&rows=50&wt=javabin&version=2}
hits=1682216 status=0 QTime=1962
2024-06-14 02:42:20.715 INFO (qtp1158676965-20) [c:vertex_index s:shard1
r:core_node2 x:vertex_index_shard1_replica_n1] o.a.s.c.S.Request
[vertex_index_shard1_replica_n1] webapp=/solr path=/select
params={q=:&stateVer=vertex_index:12&fl=id&start=150&fq=35x_t:*+&rows=50&wt=javabin&version=2}
hits=1682363 status=0 QTime=4465 {code}
- ran same query through gremlin shell while ingestion is happening it doesn't
fail
- time taken for above gremlin query in code when ingestion :
214825ms
- time takem for above gremlin query in gremlin shell when ingestion : 104641ms
- time taken for above gremlin query when no ingestion
: 181682ms
Still Root cause is unknown
*WorkAround:*
- Remove .has('__guid') clause from below, it is very quick and issue is not
reproducible.
{code:java}
g.V().has('_typeName','hive_db').has('Referenceable.qualifiedName','db6@cm').has('__guid').values('__guid'){code}
*Tests:*
* upgrded tinkerpop and janusgraph version but didn't help
* invalid property doesn't throw any exception or not existence of property
was:
*Issue:*
Export during ingestion fails giving Found 0 entities in the logs
Ingestion meaning Atlas is consuming messages
*When is the issue seen?*
It occurs when there is huge amount of data in backend and Atlas is consuming
messages linked to entity of which export is running
*Analysis to find Root cause:*
* when there is huge amount of data in backend, export FAILS
* when there is huge amount of data in backend but less tables under it, then
also export FAILS
* if background consumption stops, export PASS
* if consumption is of different entities then requested in export, export PASS
* export query to find starting object uses below query, where has clause to
check property is expensive
{code:java}
g.V().has('_typeName','hive_db').has('Referenceable.qualifiedName','db6@cm').h
Review Request 75054: ATLAS-4882: Export/Import: Export exits with 'Found 0 entities'
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/75054/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Radhika PC. Bugs: ATLAS-4882 https://issues.apache.org/jira/browse/ATLAS-4882 Repository: atlas Description --- please check the jira description https://issues.apache.org/jira/browse/ATLAS-4882 Diffs - repository/src/main/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequest.java d01d6775a repository/src/main/java/org/apache/atlas/util/AtlasGremlin3QueryProvider.java 5b10c353e Diff: https://reviews.apache.org/r/75054/diff/1/ Testing --- Manually verified Thanks, Pinal Shah
[jira] [Updated] (ATLAS-4882) Export/Import: Export exits with "Found 0 entities"
[
https://issues.apache.org/jira/browse/ATLAS-4882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pinal Shah updated ATLAS-4882:
--
Description:
*Issue:*
Export during ingestion fails giving Found 0 entities in the logs
Ingestion meaning Atlas is consuming messages
*When is the issue seen?*
It occurs when there is huge amount of data in backend and Atlas is consuming
messages linked to entity of which export is running
*Analysis to find Root cause:*
* when there is huge amount of data in backend, export FAILS
* when there is huge amount of data in backend but less tables under it, then
also export FAILS
* if background consumption stops, export PASS
* if consumption is of different entities then requested in export, export PASS
* export query to find starting object uses below query, where has clause to
check property is expensive
{code:java}
g.V().has('_typeName','hive_db').has('Referenceable.qualifiedName','db6@cm').has('guid').values('_guid'){code}
- has('__guid') queries solr [(35x_t <> null)]:vertex_index
- below is the timetaken in the solr logs
{code:java}
2024-06-14 02:38:56.218 INFO (qtp1158676965-19) [c:vertex_index s:shard1
r:core_node2 x:vertex_index_shard1_replica_n1] o.a.s.c.S.Request
[vertex_index_shard1_replica_n1] webapp=/solr path=/select
params={q=:&stateVer=vertex_index:12&fl=id&start=0&fq=35x_t:*+&rows=50&wt=javabin&version=2}
hits=1681928 status=0 QTime=4227
2024-06-14 02:40:23.945 INFO (qtp1158676965-16) [c:vertex_index s:shard1
r:core_node2 x:vertex_index_shard1_replica_n1] o.a.s.c.S.Request
[vertex_index_shard1_replica_n1] webapp=/solr path=/select
params={q=:&stateVer=vertex_index:12&fl=id&start=50&fq=35x_t:*+&rows=50&wt=javabin&version=2}
hits=1682086 status=0 QTime=787
2024-06-14 02:41:37.703 INFO (qtp1158676965-14) [c:vertex_index s:shard1
r:core_node2 x:vertex_index_shard1_replica_n1] o.a.s.c.S.Request
[vertex_index_shard1_replica_n1] webapp=/solr path=/select
params={q=:&stateVer=vertex_index:12&fl=id&start=100&fq=35x_t:*+&rows=50&wt=javabin&version=2}
hits=1682216 status=0 QTime=1962
2024-06-14 02:42:20.715 INFO (qtp1158676965-20) [c:vertex_index s:shard1
r:core_node2 x:vertex_index_shard1_replica_n1] o.a.s.c.S.Request
[vertex_index_shard1_replica_n1] webapp=/solr path=/select
params={q=:&stateVer=vertex_index:12&fl=id&start=150&fq=35x_t:*+&rows=50&wt=javabin&version=2}
hits=1682363 status=0 QTime=4465 {code}
- ran same query through gremlin shell while ingestion is happening it doesn't
fail
- time taken for above gremlin query in code when ingestion :
214825ms
- time takem for above gremlin query in gremlin shell when ingestion : 104641ms
- time taken for above gremlin query when no ingestion
: 181682ms
Still Root cause is unknown
*WorkAround:*
- Remove .has('__guid') clause from below, it is very quick and issue is not
reproducible.
{code:java}
g.V().has('_typeName','hive_db').has('Referenceable.qualifiedName','db6@cm').has('guid').values('_guid'){code}
was:
*Issue:*
Export during ingestion fails giving Found 0 entities in the logs
Ingestion meaning Atlas is consuming messages
*When is the issue seen?*
It occurs when there is huge amount of data in backend and Atlas is consuming
messages linked to entity of which export is running
*Analysis to find Root cause:*
* when there is huge amount of data in backend, export FAILS
* when there is huge amount of data in backend but less tables under it, then
also export FAILS
* if background consumption stops, export PASS
* if consumption is of different entities then requested in export, export PASS
* export query to find starting object uses below query, where has clause to
check property is expensive
{code:java}
g.V().has('_typeName','hive_db').has('Referenceable.qualifiedName','db6@cm').has('guid').values('_guid'){code}
- has('__guid') queries solr [(35x_t <> null)]:vertex_index
- below is the timetaken in the solr logs
{code:java}
2024-06-14 02:38:56.218 INFO (qtp1158676965-19) [c:vertex_index s:shard1
r:core_node2 x:vertex_index_shard1_replica_n1] o.a.s.c.S.Request
[vertex_index_shard1_replica_n1] webapp=/solr path=/select
params={q=:&stateVer=vertex_index:12&fl=id&start=0&fq=35x_t:*+&rows=50&wt=javabin&version=2}
hits=1681928 status=0 QTime=4227
2024-06-14 02:40:23.945 INFO (qtp1158676965-16) [c:vertex_index s:shard1
r:core_node2 x:vertex_index_shard1_replica_n1] o.a.s.c.S.Request
[vertex_index_shard1_replica_n1] webapp=/solr path=/select
params
[jira] [Updated] (ATLAS-4882) Export/Import: Export exits with "Found 0 entities"
[
https://issues.apache.org/jira/browse/ATLAS-4882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pinal Shah updated ATLAS-4882:
--
Description:
*Issue:*
Export during ingestion fails giving Found 0 entities in the logs
Ingestion meaning Atlas is consuming messages
*When is the issue seen?*
It occurs when there is huge amount of data in backend and Atlas is consuming
messages linked to entity of which export is running
*Analysis to find Root cause:*
* when there is huge amount of data in backend, export FAILS
* when there is huge amount of data in backend but less tables under it, then
also export FAILS
* if background consumption stops, export PASS
* if consumption is of different entities then requested in export, export PASS
* export query to find starting object uses below query, where has clause to
check property is expensive
{code:java}
g.V().has('_typeName','hive_db').has('Referenceable.qualifiedName','db6@cm').has('guid').values('_guid'){code}
- has('__guid') queries solr [(35x_t <> null)]:vertex_index
- below is the timetaken in the solr logs
{code:java}
2024-06-14 02:38:56.218 INFO (qtp1158676965-19) [c:vertex_index s:shard1
r:core_node2 x:vertex_index_shard1_replica_n1] o.a.s.c.S.Request
[vertex_index_shard1_replica_n1] webapp=/solr path=/select
params={q=:&stateVer=vertex_index:12&fl=id&start=0&fq=35x_t:*+&rows=50&wt=javabin&version=2}
hits=1681928 status=0 QTime=4227
2024-06-14 02:40:23.945 INFO (qtp1158676965-16) [c:vertex_index s:shard1
r:core_node2 x:vertex_index_shard1_replica_n1] o.a.s.c.S.Request
[vertex_index_shard1_replica_n1] webapp=/solr path=/select
params={q=:&stateVer=vertex_index:12&fl=id&start=50&fq=35x_t:*+&rows=50&wt=javabin&version=2}
hits=1682086 status=0 QTime=787
2024-06-14 02:41:37.703 INFO (qtp1158676965-14) [c:vertex_index s:shard1
r:core_node2 x:vertex_index_shard1_replica_n1] o.a.s.c.S.Request
[vertex_index_shard1_replica_n1] webapp=/solr path=/select
params={q=:&stateVer=vertex_index:12&fl=id&start=100&fq=35x_t:*+&rows=50&wt=javabin&version=2}
hits=1682216 status=0 QTime=1962
2024-06-14 02:42:20.715 INFO (qtp1158676965-20) [c:vertex_index s:shard1
r:core_node2 x:vertex_index_shard1_replica_n1] o.a.s.c.S.Request
[vertex_index_shard1_replica_n1] webapp=/solr path=/select
params={q=:&stateVer=vertex_index:12&fl=id&start=150&fq=35x_t:*+&rows=50&wt=javabin&version=2}
hits=1682363 status=0 QTime=4465 {code}
- ran same query through gremlin shell while ingestion is happening it doesn't
fail
- time taken for above gremlin query in code when ingestion :
214825ms
- time takem for above gremlin query in gremlin shell when ingestion : 104641ms
- time taken for above gremlin query when no ingestion
: 181682ms
Still Root cause is unknown
*WorkAround:*
- Remove .has('__guid') clause from below, it is very quick
{code:java}
g.V().has('_typeName','hive_db').has('Referenceable.qualifiedName','db6@cm').has('guid').values('_guid'){code}
was:
*Issue:*
Export during ingestion fails giving Found 0 entities in the logs
Ingestion meaning Atlas is consuming messages
*When is the issue seen?*
It occurs when there is huge amount of data in backend and Atlas is consuming
messages linked to entity of which export is running
*Analysis to find Root cause:*
* when there is huge amount of data in backend, export FAILS
* when there is huge amount of data in backend but less tables under it, then
also export FAILS
* if background consumption stops, export PASS
* if consumption is of different entities then requested in export, export PASS
* export query to find starting object uses below query, where has clause to
check property is expensive
g.V().has('__typeName','hive_db').has('Referenceable.qualifiedName','db6@cm').has('__guid').values('__guid')
- has('__guid') queries [(35x_t <> null)]:vertex_index , checked timetaken in
the solr logs
2024-06-14 02:38:56.218 INFO (qtp1158676965-19) [c:vertex_index s:shard1
r:core_node2 x:vertex_index_shard1_replica_n1] o.a.s.c.S.Request
[vertex_index_shard1_replica_n1] webapp=/solr path=/select
params=\{q=*:*&_stateVer_=vertex_index:12&fl=id&start=0&fq=35x_t:*+&rows=50&wt=javabin&version=2}
hits=1681928 status=0 QTime=4227
2024-06-14 02:40:23.945 INFO (qtp1158676965-16) [c:vertex_index s:shard1
r:core_node2 x:vertex_index_shard1_replica_n1] o.a.s.c.S.Request
[vertex_index_shard1_replica_n1] webapp=/solr path=/select
params=\{q=*:*&_stateVer_=vertex_index:12&fl=id&sta
[jira] [Updated] (ATLAS-4882) Export/Import: Export exits with "Found 0 entities"
[
https://issues.apache.org/jira/browse/ATLAS-4882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pinal Shah updated ATLAS-4882:
--
Description:
*Issue:*
Export during ingestion fails giving Found 0 entities in the logs
Ingestion meaning Atlas is consuming messages
*When is the issue seen?*
It occurs when there is huge amount of data in backend and Atlas is consuming
messages linked to entity of which export is running
*Analysis to find Root cause:*
* when there is huge amount of data in backend, export FAILS
* when there is huge amount of data in backend but less tables under it, then
also export FAILS
* if background consumption stops, export PASS
* if consumption is of different entities then requested in export, export PASS
* export query to find starting object uses below query, where has clause to
check property is expensive
{code:java}
g.V().has('_typeName','hive_db').has('Referenceable.qualifiedName','db6@cm').has('__guid').values('__guid'){code}
- has('__guid') queries solr [(35x_t <> null)]:vertex_index
- below is the timetaken in the solr logs
{code:java}
2024-06-14 02:38:56.218 INFO (qtp1158676965-19) [c:vertex_index s:shard1
r:core_node2 x:vertex_index_shard1_replica_n1] o.a.s.c.S.Request
[vertex_index_shard1_replica_n1] webapp=/solr path=/select
params={q=:&stateVer=vertex_index:12&fl=id&start=0&fq=35x_t:*+&rows=50&wt=javabin&version=2}
hits=1681928 status=0 QTime=4227
2024-06-14 02:40:23.945 INFO (qtp1158676965-16) [c:vertex_index s:shard1
r:core_node2 x:vertex_index_shard1_replica_n1] o.a.s.c.S.Request
[vertex_index_shard1_replica_n1] webapp=/solr path=/select
params={q=:&stateVer=vertex_index:12&fl=id&start=50&fq=35x_t:*+&rows=50&wt=javabin&version=2}
hits=1682086 status=0 QTime=787
2024-06-14 02:41:37.703 INFO (qtp1158676965-14) [c:vertex_index s:shard1
r:core_node2 x:vertex_index_shard1_replica_n1] o.a.s.c.S.Request
[vertex_index_shard1_replica_n1] webapp=/solr path=/select
params={q=:&stateVer=vertex_index:12&fl=id&start=100&fq=35x_t:*+&rows=50&wt=javabin&version=2}
hits=1682216 status=0 QTime=1962
2024-06-14 02:42:20.715 INFO (qtp1158676965-20) [c:vertex_index s:shard1
r:core_node2 x:vertex_index_shard1_replica_n1] o.a.s.c.S.Request
[vertex_index_shard1_replica_n1] webapp=/solr path=/select
params={q=:&stateVer=vertex_index:12&fl=id&start=150&fq=35x_t:*+&rows=50&wt=javabin&version=2}
hits=1682363 status=0 QTime=4465 {code}
- ran same query through gremlin shell while ingestion is happening it doesn't
fail
- time taken for above gremlin query in code when ingestion :
214825ms
- time takem for above gremlin query in gremlin shell when ingestion : 104641ms
- time taken for above gremlin query when no ingestion
: 181682ms
Still Root cause is unknown
*WorkAround:*
- Remove .has('__guid') clause from below, it is very quick and issue is not
reproducible.
{code:java}
g.V().has('_typeName','hive_db').has('Referenceable.qualifiedName','db6@cm').has('__guid').values('__guid'){code}
*Tests:*
* upgrded tinkerpop and janusgraph version but didn't help
* invalid property doesn't throw any exception or not existence of property
was:
*Issue:*
Export during ingestion fails giving Found 0 entities in the logs
Ingestion meaning Atlas is consuming messages
*When is the issue seen?*
It occurs when there is huge amount of data in backend and Atlas is consuming
messages linked to entity of which export is running
*Analysis to find Root cause:*
* when there is huge amount of data in backend, export FAILS
* when there is huge amount of data in backend but less tables under it, then
also export FAILS
* if background consumption stops, export PASS
* if consumption is of different entities then requested in export, export PASS
* export query to find starting object uses below query, where has clause to
check property is expensive
{code:java}
g.V().has('_typeName','hive_db').has('Referenceable.qualifiedName','db6@cm').has('guid').values('_guid'){code}
- has('__guid') queries solr [(35x_t <> null)]:vertex_index
- below is the timetaken in the solr logs
{code:java}
2024-06-14 02:38:56.218 INFO (qtp1158676965-19) [c:vertex_index s:shard1
r:core_node2 x:vertex_index_shard1_replica_n1] o.a.s.c.S.Request
[vertex_index_shard1_replica_n1] webapp=/solr path=/select
params={q=:&stateVer=vertex_index:12&fl=id&start=0&fq=35x_t:*+&rows=50&wt=javabin&version=2}
hits=1681928 status=0 QTime=4227
2024-06-14 02:40:23.945 INFO (qtp1158676965-16
[jira] [Created] (ATLAS-4882) Export/Import: Export exits with "Found 0 entities"
Pinal Shah created ATLAS-4882:
-
Summary: Export/Import: Export exits with "Found 0 entities"
Key: ATLAS-4882
URL: https://issues.apache.org/jira/browse/ATLAS-4882
Project: Atlas
Issue Type: Bug
Components: atlas-core
Reporter: Pinal Shah
Assignee: Pinal Shah
*Issue:*
Export during ingestion fails giving Found 0 entities in the logs
Ingestion meaning Atlas is consuming messages
*When is the issue seen?*
It occurs when there is huge amount of data in backend and Atlas is consuming
messages linked to entity of which export is running
*Analysis to find Root cause:*
* when there is huge amount of data in backend, export FAILS
* when there is huge amount of data in backend but less tables under it, then
also export FAILS
* if background consumption stops, export PASS
* if consumption is of different entities then requested in export, export PASS
* export query to find starting object uses below query, where has clause to
check property is expensive
g.V().has('__typeName','hive_db').has('Referenceable.qualifiedName','db6@cm').has('__guid').values('__guid')
- has('__guid') queries [(35x_t <> null)]:vertex_index , checked timetaken in
the solr logs
2024-06-14 02:38:56.218 INFO (qtp1158676965-19) [c:vertex_index s:shard1
r:core_node2 x:vertex_index_shard1_replica_n1] o.a.s.c.S.Request
[vertex_index_shard1_replica_n1] webapp=/solr path=/select
params=\{q=*:*&_stateVer_=vertex_index:12&fl=id&start=0&fq=35x_t:*+&rows=50&wt=javabin&version=2}
hits=1681928 status=0 QTime=4227
2024-06-14 02:40:23.945 INFO (qtp1158676965-16) [c:vertex_index s:shard1
r:core_node2 x:vertex_index_shard1_replica_n1] o.a.s.c.S.Request
[vertex_index_shard1_replica_n1] webapp=/solr path=/select
params=\{q=*:*&_stateVer_=vertex_index:12&fl=id&start=50&fq=35x_t:*+&rows=50&wt=javabin&version=2}
hits=1682086 status=0 QTime=787
2024-06-14 02:41:37.703 INFO (qtp1158676965-14) [c:vertex_index s:shard1
r:core_node2 x:vertex_index_shard1_replica_n1] o.a.s.c.S.Request
[vertex_index_shard1_replica_n1] webapp=/solr path=/select
params=\{q=*:*&_stateVer_=vertex_index:12&fl=id&start=100&fq=35x_t:*+&rows=50&wt=javabin&version=2}
hits=1682216 status=0 QTime=1962
2024-06-14 02:42:20.715 INFO (qtp1158676965-20) [c:vertex_index s:shard1
r:core_node2 x:vertex_index_shard1_replica_n1] o.a.s.c.S.Request
[vertex_index_shard1_replica_n1] webapp=/solr path=/select
params=\{q=*:*&_stateVer_=vertex_index:12&fl=id&start=150&fq=35x_t:*+&rows=50&wt=javabin&version=2}
hits=1682363 status=0 QTime=4465
- ran same query through gremlin shell while ingestion is happening it doesnt
fail
- time taken for above gremlin query in code when ingestion : 214825ms
- time takem for above gremlin query in gremlin shell when ingestion : 104641ms
- time taken for above gremlin query when no ingestion : 181682ms
WorkAround
- Remove .has('__guid') clause from below, it is very quick
g.V().has('__typeName','hive_db').has('Referenceable.qualifiedName','db6@cm').has('__guid').values('__guid')
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: Review Request 75041: ATLAS-4878: utility to analyze hook notifications
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/75041/#review226558 --- Ship it! Ship It! - Pinal Shah On June 13, 2024, 12:19 a.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/75041/ > --- > > (Updated June 13, 2024, 12:19 a.m.) > > > Review request for atlas, Ashutosh Mestry, chaitali, Disha Talreja, Jayendra > Parab, Mandar Ambawane, Pinal Shah, Radhika Kundam, and Sarath Subramanian. > > > Bugs: ATLAS-4878 > https://issues.apache.org/jira/browse/ATLAS-4878 > > > Repository: atlas > > > Description > --- > > This utility analyzes hook notification messages from a file and reports the > details listed in README. This utility will be useful in troubleshooting. > > > Diffs > - > > distro/pom.xml 874b944f2 > distro/src/main/assemblies/notification-analyzer.xml PRE-CREATION > pom.xml 2f9aedbd2 > tools/notification-analyzer/README PRE-CREATION > tools/notification-analyzer/pom.xml PRE-CREATION > tools/notification-analyzer/scripts/notification-analyzer.sh PRE-CREATION > > tools/notification-analyzer/src/main/java/org/apache/atlas/tools/NotificationAnalyzer.java > PRE-CREATION > tools/notification-analyzer/src/main/resources/atlas-application.properties > PRE-CREATION > tools/notification-analyzer/src/main/resources/atlas-log4j.xml PRE-CREATION > > > Diff: https://reviews.apache.org/r/75041/diff/3/ > > > Testing > --- > > - setup and ran the utility using the instructions in README > > > Thanks, > > Madhan Neethiraj > >
Re: [DRAFT] [REPORT] Apache Atlas - June 2024
Looks Good +1 Thanks Madhan for putting this together On Fri, Jun 14, 2024 at 7:42 AM Sarath Subramanian wrote: > Thanks for the draft report Madhan. Looks good +1 > > > Thanks, > Sarath > > > On Jun 13, 2024, at 5:23 PM, Madhan Neethiraj wrote: > > > > Atlas community, > > > > Please review the following draft report for this quarter. > > > > Thanks, > > Madhan > > > > > > ## Description: > > The mission of Apache Atlas is the creation and maintenance of software > related > > to a scalable and extensible set of core foundational governance > services - > > enabling enterprises to effectively and efficiently meet their compliance > > requirements > > > > ## Project Status: > > Current project status: Ongoing, with low activity > > Issues for the board: there are no issues requiring board attention > > > > ## Membership Data: > > Apache Atlas was founded 2017-06-20 (7 years ago) > > There are currently 48 committers and 32 PMC members in this project. > > The Committer-to-PMC ratio is 3:2. > > > > Community changes, past quarter: > > - No new PMC members. Last addition was Ashutosh Mestry on 2019-04-16. > > - No new committers. Last addition was Umesh Padashetty on 2023-06-14. > > > > ## Project Activity: > > - improvements in Export/Import functionality > > - updates to support ignore patterns in all Atlas hooks > > - added support for Chinese characters in Atlas entity attributes > > - Docker setup update to address HBase failures > > - updated versions of dependent libraries JanusGraph and Tinkerpop > > > > ## Community Health: > > - dev@atlas.apache.org had a 166% increase in traffic in the past > quarter (312 > > emails compared to 117) > > - 23 commits in the past quarter (64% increase) > > - 7 code contributors in the past quarter (40% increase) > > - 4 PRs opened on GitHub, past quarter (100% increase) > > - 2 PRs closed on GitHub, past quarter (100% increase) > > > > ## Recent releases: > > - Apache Atlas 2.3.0 was released on 2022-12-06 > > - Apache Atlas 2.2.0 was released on 2021-08-17 > > - Apache Atlas 2.1.0 was released on 2020-07-15 > > > > > > > > >
Review Request 75007: ATLAS-4866: Export/Import: Skip adding processed guids in audit while importing huge zip
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/75007/
---
Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Radhika Kundam.
Bugs: ATLAS-4866
https://issues.apache.org/jira/browse/ATLAS-4866
Repository: atlas
Description
---
**Background:**
Atlas uses HBase as its store for audit repository.
After import, atlas store the audit entity with the import information along
with all the processed entitiy guids.
**Issue: **
When large sized export zipped file is imported, import gives below error,
internally import gets succesful, but fail creating audit.
*{"errorCode":"ATLAS-500-00-001","errorMessage":"org.janusgraph.core.JanusGraphException:
Could not commit transaction due to exception during
persistence","errorCause":"Could not commit transaction due to exception during
persistence"}*
When size of the entity is greater than "hbase.client.keyvalue.maxsize"
property then audit entity creation fails with exception.
*Caused by:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
action: org.apache.hadoop.hbase.DoNotRetryIOException:
Cell[\x00\x00\x00\x00\x00\x00\x00\x00\x01\x05\xCC\xBB/l:\x00\x06\x18r\xB0\xBE\xFDH\xA00a11ed186467-ve0214-halxg-cloudera-com\xB2\x00\x00\x00\x00\x00\x0D\xB6Y/1715730740890001/Put/vlen=23826488/seqid=0]
with size 23826581 exceeds limit of 10485760 bytes
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.checkCellSizeLimit(RSRpcServices.java:906)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:992)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:927)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:892)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2855)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45961)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:139)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349)
: 1 time, servers with issues: ve0214.halxg.cloudera.com,22101,1715690875185
at
org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:50)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.getErrors(AsyncRequestFutureImpl.java:1228)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:434)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:422)
at org.janusgraph.diskstorage.hbase2.HTable2_0.batch(HTable2_0.java:51)
**Solution: **
Hence in this case, storing processed entities guids is skipped while creating
ExportImportAuditEntry when size of entity goes beyond the value (in bytes) of
below property
atlas.hbase.client.keyvalue.maxsize
Diffs
-
repository/src/main/java/org/apache/atlas/repository/impexp/ExportImportAuditService.java
3afa17301
Diff: https://reviews.apache.org/r/75007/diff/1/
Testing
---
manually verified through below api, processed entity guids is not stored
/api/atlas/admin/expimp/audit?userName=admin&operation=IMPORT
Thanks,
Pinal Shah
[jira] [Updated] (ATLAS-4866) Export/Import: Skip adding processed guids in audit while importing huge zip
[
https://issues.apache.org/jira/browse/ATLAS-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pinal Shah updated ATLAS-4866:
--
Description:
*Background:*
Atlas uses HBase as its store for audit repository.
After import, atlas store the audit entity with the import information along
with all the processed entitiy guids.
*Issue:*
When large sized export zipped file is imported, import gives below error,
internally import gets succesful, but fail creating audit.
{code:java}
{"errorCode":"ATLAS-500-00-001","errorMessage":"org.janusgraph.core.JanusGraphException:
Could not commit transaction due to exception during
persistence","errorCause":"Could not commit transaction due to exception during
persistence"}
{code}
When size of the entity is greater than "hbase.client.keyvalue.maxsize"
property then audit entity creation fails with exception.
{code:java}
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 1 action: org.apache.hadoop.hbase.DoNotRetryIOException:
Cell[\x00\x00\x00\x00\x00\x00\x00\x00\x01\x05\xCC\xBB/l:\x00\x06\x18r\xB0\xBE\xFDH\xA00a11ed186467-ve0214-halxg-cloudera-com\xB2\x00\x00\x00\x00\x00\x0D\xB6Y/1715730740890001/Put/vlen=23826488/seqid=0]
with size 23826581 exceeds limit of 10485760 bytes
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.checkCellSizeLimit(RSRpcServices.java:906)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:992)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:927)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:892)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2855)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45961)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:139)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349)
: 1 time, servers with issues: ve0214.halxg.cloudera.com,22101,1715690875185
at
org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:50)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.getErrors(AsyncRequestFutureImpl.java:1228)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:434)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:422)
at org.janusgraph.diskstorage.hbase2.HTable2_0.batch(HTable2_0.java:51)
{code}
*Solution:*
Hence in this case, storing processed entities guids is skipped while creating
ExportImportAuditEntry when size of entity goes beyond the value stored in
bytes below
{code:java}
atlas.hbase.client.keyvalue.maxsize{code}
was:
*Background:*
Atlas uses HBase as its store for audit repository.
After import, atlas store the audit entity with the import information along
with all the processed entitiy guids.
*Issue:*
When large sized export zipped file is imported, import gives below error,
internally import gets succesful, but fail creating audit.
_{"errorCode":"ATLAS-500-00-001","errorMessage":"org.janusgraph.core.JanusGraphException:
Could not commit transaction due to exception during
persistence","errorCause":"Could not commit transaction due to exception during
persistence"}_
When size of the entity is greater than "hbase.client.keyvalue.maxsize"
property then audit entity creation fails with exception.
{code:java}
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 1 action: org.apache.hadoop.hbase.DoNotRetryIOException:
Cell[\x00\x00\x00\x00\x00\x00\x00\x00\x01\x05\xCC\xBB/l:\x00\x06\x18r\xB0\xBE\xFDH\xA00a11ed186467-ve0214-halxg-cloudera-com\xB2\x00\x00\x00\x00\x00\x0D\xB6Y/1715730740890001/Put/vlen=23826488/seqid=0]
with size 23826581 exceeds limit of 10485760 bytes
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.checkCellSizeLimit(RSRpcServices.java:906)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:992)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:927)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:892)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2855)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45961)
[jira] [Updated] (ATLAS-4866) Export/Import: Skip adding processed guids in audit while importing huge zip
[
https://issues.apache.org/jira/browse/ATLAS-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pinal Shah updated ATLAS-4866:
--
Description:
*Background:*
Atlas uses HBase as its store for audit repository.
After import, atlas store the audit entity with the import information along
with all the processed entitiy guids.
*Issue:*
When large sized export zipped file is imported, import gives below error,
internally import gets succesful, but fail creating audit.
_{"errorCode":"ATLAS-500-00-001","errorMessage":"org.janusgraph.core.JanusGraphException:
Could not commit transaction due to exception during
persistence","errorCause":"Could not commit transaction due to exception during
persistence"}_
When size of the entity is greater than "hbase.client.keyvalue.maxsize"
property then audit entity creation fails with exception.
{code:java}
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 1 action: org.apache.hadoop.hbase.DoNotRetryIOException:
Cell[\x00\x00\x00\x00\x00\x00\x00\x00\x01\x05\xCC\xBB/l:\x00\x06\x18r\xB0\xBE\xFDH\xA00a11ed186467-ve0214-halxg-cloudera-com\xB2\x00\x00\x00\x00\x00\x0D\xB6Y/1715730740890001/Put/vlen=23826488/seqid=0]
with size 23826581 exceeds limit of 10485760 bytes
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.checkCellSizeLimit(RSRpcServices.java:906)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:992)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:927)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:892)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2855)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45961)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:139)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349)
: 1 time, servers with issues: ve0214.halxg.cloudera.com,22101,1715690875185
at
org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:50)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.getErrors(AsyncRequestFutureImpl.java:1228)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:434)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:422)
at org.janusgraph.diskstorage.hbase2.HTable2_0.batch(HTable2_0.java:51)
{code}
*Solution:*
Hence in this case, storing processed entities guids is skipped while creating
ExportImportAuditEntry when size of entity goes beyond the value stored in
bytes below
{code:java}
atlas.hbase.client.keyvalue.maxsize{code}
was:
*Background:*
Atlas uses HBase as its store for audit repository.
After import, atlas store the audit entity with the import information along
with all the processed entitiy guids.
*Issue:*
When size of the entity is greater than "hbase.client.keyvalue.maxsize"
property then audit entity creation fails with exception.
{code:java}
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 1 action: org.apache.hadoop.hbase.DoNotRetryIOException:
Cell[\x00\x00\x00\x00\x00\x00\x00\x00\x01\x05\xCC\xBB/l:\x00\x06\x18r\xB0\xBE\xFDH\xA00a11ed186467-ve0214-halxg-cloudera-com\xB2\x00\x00\x00\x00\x00\x0D\xB6Y/1715730740890001/Put/vlen=23826488/seqid=0]
with size 23826581 exceeds limit of 10485760 bytes
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.checkCellSizeLimit(RSRpcServices.java:906)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:992)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:927)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:892)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2855)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45961)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:139)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349)
: 1 time, servers with issues: ve0214.halxg.cloudera.com,22101,1715690875185
at
org.apac
Re: Review Request 74996: ATLAS-4864 : BasicSearch: Search with combination of terms and freetext, works incorrectly
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74996/#review226481 --- Ship it! Ship It! - Pinal Shah On May 21, 2024, 9:26 a.m., Darshil shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74996/ > --- > > (Updated May 21, 2024, 9:26 a.m.) > > > Review request for atlas and Pinal Shah. > > > Bugs: ATLAS-4864 > https://issues.apache.org/jira/browse/ATLAS-4864 > > > Repository: atlas > > > Description > --- > > Problem: > > When we perform a basic search, using a term with a query filter, it does not > produce the correct output. For example, if we use the filter Term as Term1 > and the query as Joe, the resulting output will be inaccurate. > > > Diffs > - > > repository/src/main/java/org/apache/atlas/discovery/SearchContext.java > 9a9a5512a > > repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java > b8a507eac > > repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java > fb94aaedc > > > Diff: https://reviews.apache.org/r/74996/diff/3/ > > > Testing > --- > > Test cases have been added, and manual testing has been completed. > In the test case, we can filter using term0 and provide felt as the query, > which corresponds to the owner in the entity. This should give us the desired > output. > > > Thanks, > > Darshil shah > >
[jira] [Updated] (ATLAS-4866) Export/Import: Skip adding processed guids in audit while importing huge zip
[
https://issues.apache.org/jira/browse/ATLAS-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pinal Shah updated ATLAS-4866:
--
Description:
*Background:*
Atlas uses HBase as its store for audit repository.
After import, atlas store the audit entity with the import information along
with all the processed entitiy guids.
*Issue:*
When size of the entity is greater than "hbase.client.keyvalue.maxsize"
property then audit entity creation fails with exception.
{code:java}
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 1 action: org.apache.hadoop.hbase.DoNotRetryIOException:
Cell[\x00\x00\x00\x00\x00\x00\x00\x00\x01\x05\xCC\xBB/l:\x00\x06\x18r\xB0\xBE\xFDH\xA00a11ed186467-ve0214-halxg-cloudera-com\xB2\x00\x00\x00\x00\x00\x0D\xB6Y/1715730740890001/Put/vlen=23826488/seqid=0]
with size 23826581 exceeds limit of 10485760 bytes
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.checkCellSizeLimit(RSRpcServices.java:906)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:992)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:927)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:892)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2855)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45961)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:139)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349)
: 1 time, servers with issues: ve0214.halxg.cloudera.com,22101,1715690875185
at
org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:50)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.getErrors(AsyncRequestFutureImpl.java:1228)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:434)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:422)
at org.janusgraph.diskstorage.hbase2.HTable2_0.batch(HTable2_0.java:51)
{code}
*Solution:*
Hence in this case, storing processed entities guids is skipped while creating
ExportImportAuditEntry when size of entity goes beyond the value stored in
bytes below
{code:java}
atlas.hbase.client.keyvalue.maxsize{code}
was:
Atlas uses HBase as its store for audit repository.
After import, atlas store the audit entity with the import information along
with all the processed entitiy guids.
When size of the entity is greater than "hbase.client.keyvalue.maxsize"
property then audit entity creation fails with exception.
{code:java}
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 1 action: org.apache.hadoop.hbase.DoNotRetryIOException:
Cell[\x00\x00\x00\x00\x00\x00\x00\x00\x01\x05\xCC\xBB/l:\x00\x06\x18r\xB0\xBE\xFDH\xA00a11ed186467-ve0214-halxg-cloudera-com\xB2\x00\x00\x00\x00\x00\x0D\xB6Y/1715730740890001/Put/vlen=23826488/seqid=0]
with size 23826581 exceeds limit of 10485760 bytes
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.checkCellSizeLimit(RSRpcServices.java:906)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:992)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:927)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:892)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2855)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45961)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:139)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349)
: 1 time, servers with issues: ve0214.halxg.cloudera.com,22101,1715690875185
at
org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:50)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.getErrors(AsyncRequestFutureImpl.java:1228)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:434)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:422)
at org.janusgraph.diskstorage.hbase2.HTable2_0.batch(HTable2_0.java:51)
{code}
Hence in this case, storing processed entities guids is skipped while creating
ExportImportAuditEntry
&
[jira] [Created] (ATLAS-4866) Export/Import: Skip adding processed guids in audit while importing huge zip
Pinal Shah created ATLAS-4866:
-
Summary: Export/Import: Skip adding processed guids in audit while
importing huge zip
Key: ATLAS-4866
URL: https://issues.apache.org/jira/browse/ATLAS-4866
Project: Atlas
Issue Type: Bug
Components: atlas-core
Reporter: Pinal Shah
Assignee: Pinal Shah
Atlas uses HBase as its store for audit repository.
After import, atlas store the audit entity with the import information along
with all the processed entitiy guids.
When size of the entity is greater than "hbase.client.keyvalue.maxsize"
property then audit entity creation fails with exception.
{code:java}
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 1 action: org.apache.hadoop.hbase.DoNotRetryIOException:
Cell[\x00\x00\x00\x00\x00\x00\x00\x00\x01\x05\xCC\xBB/l:\x00\x06\x18r\xB0\xBE\xFDH\xA00a11ed186467-ve0214-halxg-cloudera-com\xB2\x00\x00\x00\x00\x00\x0D\xB6Y/1715730740890001/Put/vlen=23826488/seqid=0]
with size 23826581 exceeds limit of 10485760 bytes
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.checkCellSizeLimit(RSRpcServices.java:906)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:992)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:927)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:892)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2855)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45961)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:139)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349)
: 1 time, servers with issues: ve0214.halxg.cloudera.com,22101,1715690875185
at
org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:50)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.getErrors(AsyncRequestFutureImpl.java:1228)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:434)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:422)
at org.janusgraph.diskstorage.hbase2.HTable2_0.batch(HTable2_0.java:51)
{code}
Hence in this case, storing processed entities guids is skipped while creating
ExportImportAuditEntry
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (ATLAS-4865) Documentation: Add features, improvement which are missing in the doc
Pinal Shah created ATLAS-4865: - Summary: Documentation: Add features, improvement which are missing in the doc Key: ATLAS-4865 URL: https://issues.apache.org/jira/browse/ATLAS-4865 Project: Atlas Issue Type: Task Components: atlas-webui Affects Versions: 2.3.0 Reporter: Pinal Shah Assignee: Pinal Shah -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74996: ATLAS-4864 : BasicSearch: Search with combination of terms and freetext, works incorrectly
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74996/#review226466 --- repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java Line 107 (original) <https://reviews.apache.org/r/74996/#comment314669> retain blank line repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java Line 114 (original) <https://reviews.apache.org/r/74996/#comment314668> retain blank line repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java Lines 121 (patched) <https://reviews.apache.org/r/74996/#comment314667> add some more data and use those for different test cases - Pinal Shah On May 17, 2024, 11:24 a.m., Darshil shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74996/ > --- > > (Updated May 17, 2024, 11:24 a.m.) > > > Review request for atlas and Pinal Shah. > > > Bugs: ATLAS-4864 > https://issues.apache.org/jira/browse/ATLAS-4864 > > > Repository: atlas > > > Description > --- > > Problem: > > When we perform a basic search, using a term with a query filter, it does not > produce the correct output. For example, if we use the filter Term as Term1 > and the query as Joe, the resulting output will be inaccurate. > > > Diffs > - > > repository/src/main/java/org/apache/atlas/discovery/SearchContext.java > 9a9a5512a > > repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java > b8a507eac > > repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java > fb94aaedc > > > Diff: https://reviews.apache.org/r/74996/diff/2/ > > > Testing > --- > > Test cases have been added, and manual testing has been completed. > In the test case, we can filter using term0 and provide felt as the query, > which corresponds to the owner in the entity. This should give us the desired > output. > > > File Attachments > > > ATLAS-4864.patch > > https://reviews.apache.org/media/uploaded/files/2024/05/17/b9b0b781-28eb-4d18-9cc0-8d01aa35cf3f__ATLAS-4864.patch > ATLAS-4864.patch > > https://reviews.apache.org/media/uploaded/files/2024/05/17/962f256b-bd3a-44eb-b154-c67dd6cd5c6c__ATLAS-4864.patch > > > Thanks, > > Darshil shah > >
[jira] [Created] (ATLAS-4864) BasicSearch: Search with combination of terms and freetext, works incorrectly
Pinal Shah created ATLAS-4864: - Summary: BasicSearch: Search with combination of terms and freetext, works incorrectly Key: ATLAS-4864 URL: https://issues.apache.org/jira/browse/ATLAS-4864 Project: Atlas Issue Type: Bug Components: atlas-core Reporter: Pinal Shah Assignee: Pinal Shah Attachments: Screen Shot 2020-06-08 at 4.03.35 PM (1).png Filter of freetext query doen't apply on the term result. !Screen Shot 2020-06-08 at 4.03.35 PM (1).png|width=744,height=303! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (ATLAS-4863) NPE while deleting BusinessMetadata
Pinal Shah created ATLAS-4863:
-
Summary: NPE while deleting BusinessMetadata
Key: ATLAS-4863
URL: https://issues.apache.org/jira/browse/ATLAS-4863
Project: Atlas
Issue Type: Bug
Components: atlas-core
Reporter: Pinal Shah
Assignee: Pinal Shah
If businessMetadata is created without adding any applicable types, we get
NullPointerException when we try to delete that businessmetadata.
Example:
{code:java}
{
"category": "BUSINESS_METADATA",
"guid": "678dc8e1-3897-4b43-94f2-929ea984e8d9",
"createdBy": "h0650206",
"updatedBy": "h0650206",
"createTime": 1714471319783,
"updateTime": 1714731363238,
"version": 3,
"name": "test_create_bussiness_metadata",
"description": "test_create_bussiness_metadata",
"typeVersion": "1.1",
"attributeDefs": [
{
"name": "test",
"typeName": "string",
"isOptional": true,
"cardinality": "SINGLE",
"valuesMinCount": 0,
"valuesMaxCount": 1,
"isUnique": false,
"isIndexable": true,
"includeInNotification": false,
"searchWeight": 0,
"options": {
"applicableEntityTypes": "[]",
"maxStrLength": "50"
}
}
]
} {code}
If above businessMetadata type is deleted, we get below exception
{code:java}
java.lang.NullPointerException
at java.base/java.util.Objects.requireNonNull(Objects.java:221)
at java.base/java.lang.String.join(String.java:2440)
at
org.apache.atlas.repository.store.graph.v2.AtlasBusinessMetadataDefStoreV2.isBusinessAttributePresent(AtlasBusinessMetadataDefStoreV2.java:425)
at
org.apache.atlas.repository.store.graph.v2.AtlasBusinessMetadataDefStoreV2.checkBusinessMetadataRef(AtlasBusinessMetadataDefStoreV2.java:408)
at
org.apache.atlas.repository.store.graph.v2.AtlasBusinessMetadataDefStoreV2.preDeleteByGuid(AtlasBusinessMetadataDefStoreV2.java:342)
at
org.apache.atlas.repository.store.graph.v2.AtlasAbstractDefStoreV2.deleteByGuid(AtlasAbstractDefStoreV2.java:153)
at
org.apache.atlas.repository.store.graph.v2.AtlasBusinessMetadataDefStoreV2.deleteByGuid(AtlasBusinessMetadataDefStoreV2.java:55)
at
org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStore.deleteTypesDef(AtlasTypeDefGraphStore.java:669)
at
org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStore.deleteTypeByName(AtlasTypeDefGraphStore.java:714)
at
org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStore$$FastClassBySpringCGLIB$$5226c80b.invoke()
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: Review Request 74978: ATLAS-4861: Export/Import: add flag to skip updating replicated attributes
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74978/
---
(Updated May 14, 2024, 4:52 a.m.)
Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj,
Nikhil Bonte, and Nixon Rodrigues.
Bugs: ATLAS-4861
https://issues.apache.org/jira/browse/ATLAS-4861
Repository: atlas
Description
---
**Replicated Attributes:**
Users want knowledge of how the entities have landed into Atlas instance, if
they were created via hook ingestion or whether they were imported from another
Atlas instance.
This is addressed by 2 new attributes that are now part of Referenceable entity
type viz. replicatedFrom and replicatedTo.
**Current Behaviour:**
At the end of export/import, the entities which are exported/imported are
updated with the replicationTo/replicationFrom respectively with the value of
AtlasServer object
Hence updating each and every entity takes much time.
**Improvement:**
Add flag to skip such updates.
**Example request for Export:**
curl -v -X POST -u admin:admin -H "Content-Type: application/json"
"http://<>/api/atlas/admin/export" -d
'{"itemsToExport":[{"typeName":"hive_db","uniqueAttributes":{"qualifiedName":"bsadp@hpchdd2"}}],"options":{"fetchType":"full","replicatedTo":"cm",**"skipUpdateReplicationAttr":"true"**},"fetchTypeOptionValue":"full","skipLineageOptionValue":false,"changeTokenFromOptions":0}}'
> test_noupdate.zip
**Example import-options.json for Import:**
{
"options": {
"replicatedFrom":"cm",
"skipUpdateReplicationAttr":true
}
}
Diffs
-
intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportRequest.java
878b1d8bc
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java
cbc1aa938
repository/src/main/java/org/apache/atlas/repository/impexp/AuditsWriter.java
c4de0ed27
Diff: https://reviews.apache.org/r/74978/diff/1/
Testing
---
Tests(~3000 entity) w/o updateswith updates
Export1 11s 25s
Export2 12s 27s
Thanks,
Pinal Shah
[jira] [Created] (ATLAS-4861) Export/Import: add flag to skip updating replicated attributes
Pinal Shah created ATLAS-4861: - Summary: Export/Import: add flag to skip updating replicated attributes Key: ATLAS-4861 URL: https://issues.apache.org/jira/browse/ATLAS-4861 Project: Atlas Issue Type: Improvement Components: atlas-core Reporter: Pinal Shah Assignee: Pinal Shah *Replicated Attributes:* Users want knowledge of how the entities have landed into Atlas instance, if they were created via hook ingestion or whether they were imported from another Atlas instance. This is addressed by 2 new attributes that are now part of _Referenceable_ entity type viz. _replicatedFrom_ and _replicatedTo_. *Current Behaviour:* At the end of export/import, the entities which are exported/imported are updated with the replicationTo/replicationFrom respectively with the value of AtlasServer object Hence updating each and every entity takes much time. *Improvement:* Add flag to skip such updates. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74925: ATLAS-4225 : Support for Chinese character in entity data.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74925/#review226411 --- Ship it! Ship It! - Pinal Shah On May 3, 2024, 5:18 a.m., chaitali wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74925/ > --- > > (Updated May 3, 2024, 5:18 a.m.) > > > Review request for atlas, Ashutosh Mestry, Deep Singh, Jayendra Parab, Pinal > Shah, and Sheetal Shah. > > > Bugs: ATLAS-4225 > https://issues.apache.org/jira/browse/ATLAS-4225 > > > Repository: atlas > > > Description > --- > > Updated latest patch with synchronized code > > Creating this review request to track and close > https://reviews.apache.org/r/73256/diff/3#1 as the patch here is not in sync > with latest code > > > Diffs > - > > intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 591b31612 > intg/src/test/java/org/apache/atlas/type/TestAtlasStructType.java f117fb359 > > repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java > 9be6517e9 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > 01b88eb2f > repository/src/test/java/org/apache/atlas/BasicTestSetup.java 593f8c19b > > repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java > 282a3fd4a > > repository/src/test/java/org/apache/atlas/discovery/ClassificationSearchProcessorTest.java > 121dca9f5 > > repository/src/test/java/org/apache/atlas/discovery/EntitySearchProcessorTest.java > 3f9e74d11 > > repository/src/test/java/org/apache/atlas/discovery/FreeTextSearchProcessorTest.java > d7825a0b2 > > repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java > 1489b2760 > > > Diff: https://reviews.apache.org/r/74925/diff/8/ > > > Testing > --- > > Testcases are added in the patch > PC is successful : > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1621/ > > > Thanks, > > chaitali > >
Re: Review Request 74925: ATLAS-4225 : Support for Chinese character in entity data.
> On April 23, 2024, 11:22 a.m., Pinal Shah wrote:
> > repository/src/main/java/org/apache/atlas/discovery/SearchContext.java
> > Lines 123 (patched)
> > <https://reviews.apache.org/r/74925/diff/3/?file=2287121#file2287121line123>
> >
> > please review the conditions again, for the below cases
> >
> > query = abc@123* should be query = abc@123
> > query = abc@123 should be query = abc@123
> > query = abc should be query = abc*
> > query = abc* should be query = abc*
> >
> >
> > if (StringUtils.isNotEmpty(query) && query.endsWith("*")) {
> > query = query.substring(0, query.length()-1);
> >
> > }
> >
> > if (StringUtils.isNotEmpty(query) &&
> > !AtlasStructType.AtlasAttribute.hastokenizeChar(query)) {
> > query = query + "*";
> > searchParameters.setQuery(query);
> > }
query = abc@123'*' should be query = abc@123
query = abc@123 should be query = abc@123
query = abc'*' should be query = abc'*'
query = abc should be query = abc'*'
- Pinal
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74925/#review226393
---
On April 23, 2024, 8:04 a.m., chaitali wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74925/
> ---
>
> (Updated April 23, 2024, 8:04 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Deep Singh, Jayendra Parab, Pinal
> Shah, and Sheetal Shah.
>
>
> Bugs: ATLAS-4225
> https://issues.apache.org/jira/browse/ATLAS-4225
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Updated latest patch with synchronized code
>
> Creating this review request to track and close
> https://reviews.apache.org/r/73256/diff/3#1 as the patch here is not in sync
> with latest code
>
>
> Diffs
> -
>
> intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 591b31612
> repository/src/main/java/org/apache/atlas/discovery/SearchContext.java
> 9a9a5512a
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
> 01b88eb2f
> repository/src/test/java/org/apache/atlas/BasicTestSetup.java 593f8c19b
>
> repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
> 282a3fd4a
>
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
> 1489b2760
>
>
> Diff: https://reviews.apache.org/r/74925/diff/3/
>
>
> Testing
> ---
>
> Testcases are added in the patch
> PC is successful :
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1561/consoleFull
>
>
> Thanks,
>
> chaitali
>
>
Re: Review Request 74925: ATLAS-4225 : Support for Chinese character in entity data.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74925/#review226393
---
intg/src/main/java/org/apache/atlas/type/AtlasStructType.java
Lines 1083 (patched)
<https://reviews.apache.org/r/74925/#comment314604>
please another method for #1083 to #1090
private static boolean hasCJKChar(String value, int i) {
}
repository/src/main/java/org/apache/atlas/discovery/SearchContext.java
Lines 120 (patched)
<https://reviews.apache.org/r/74925/#comment314606>
please move this piece of code above
#233
if (needFullTextProcessor()) {
repository/src/main/java/org/apache/atlas/discovery/SearchContext.java
Lines 121 (patched)
<https://reviews.apache.org/r/74925/#comment314607>
This will be checked twice, one here and other in
EntityDiscoveryService.java
public AtlasQuickSearchResult quickSearch(QuickSearchParameters
quickSearchParameters) throws AtlasBaseException {
String query = quickSearchParameters.getQuery();
if (StringUtils.isNotEmpty(query) &&
!AtlasStructType.AtlasAttribute.hastokenizeChar(query)) {
query = query + "*";
}
quickSearchParameters.setQuery(query);
please remove from here
repository/src/main/java/org/apache/atlas/discovery/SearchContext.java
Lines 123 (patched)
<https://reviews.apache.org/r/74925/#comment314605>
please review the conditions again, for the below cases
query = abc@123* should be query = abc@123
query = abc@123 should be query = abc@123
query = abc should be query = abc*
query = abc* should be query = abc*
if (StringUtils.isNotEmpty(query) && query.endsWith("*")) {
query = query.substring(0, query.length()-1);
}
if (StringUtils.isNotEmpty(query) &&
!AtlasStructType.AtlasAttribute.hastokenizeChar(query)) {
query = query + "*";
searchParameters.setQuery(query);
}
- Pinal Shah
On April 23, 2024, 8:04 a.m., chaitali wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74925/
> ---
>
> (Updated April 23, 2024, 8:04 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Deep Singh, Jayendra Parab, Pinal
> Shah, and Sheetal Shah.
>
>
> Bugs: ATLAS-4225
> https://issues.apache.org/jira/browse/ATLAS-4225
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Updated latest patch with synchronized code
>
> Creating this review request to track and close
> https://reviews.apache.org/r/73256/diff/3#1 as the patch here is not in sync
> with latest code
>
>
> Diffs
> -
>
> intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 591b31612
> repository/src/main/java/org/apache/atlas/discovery/SearchContext.java
> 9a9a5512a
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
> 01b88eb2f
> repository/src/test/java/org/apache/atlas/BasicTestSetup.java 593f8c19b
>
> repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
> 282a3fd4a
>
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
> 1489b2760
>
>
> Diff: https://reviews.apache.org/r/74925/diff/3/
>
>
> Testing
> ---
>
> Testcases are added in the patch
> PC is successful :
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1561/consoleFull
>
>
> Thanks,
>
> chaitali
>
>
Re: Review Request 74956: ATLAS-4847 : Export/Import : Atlas export fails and throws NullPointerException.
> On April 12, 2024, 7:38 a.m., Pinal Shah wrote: > > Ship It! Glossary Updates will be handled in other jira - Pinal --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74956/#review226388 --- On April 12, 2024, 7:22 a.m., Priyanshi Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74956/ > --- > > (Updated April 12, 2024, 7:22 a.m.) > > > Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, and > Sheetal Shah. > > > Bugs: ATLAS-4847 > https://issues.apache.org/jira/browse/ATLAS-4847 > > > Repository: atlas > > > Description > --- > > Steps to reproduce: > > 1. Create a hive_table entity and assign a term to it. > 2. Perform 1st incremental export of that entity without using changeMarker. > 3. Without any modification on that entity perform 2nd consecutive > incremental export with changeMarker from previous export. > > Result is it that it throws NullPointerException and export is failed. > > Problem: > > When an entity which has termed assigned was exported with changeMarker value > greater than value of lastModified time of that entity it resulted in > throwing NullPointerException. > > Solution: > > 1. NullPointerException occured because it was trying to fetch referred > entities of AtlasGlossary Object while performing export operation and as > AtlasGlossary didn't have any referred entities it gave raise this exception. > 2. This problem is solved by handling NullPointer before trying to fetch > referred entities. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java > 4615c6c2f > > > Diff: https://reviews.apache.org/r/74956/diff/1/ > > > Testing > --- > > Manual testing has been done for the same. > > PC link: > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1608/ > > > Thanks, > > Priyanshi Shah > >
Re: Review Request 74956: ATLAS-4847 : Export/Import : Atlas export fails and throws NullPointerException.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74956/#review226388 --- Ship it! Ship It! - Pinal Shah On April 12, 2024, 7:22 a.m., Priyanshi Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74956/ > --- > > (Updated April 12, 2024, 7:22 a.m.) > > > Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, and > Sheetal Shah. > > > Bugs: ATLAS-4847 > https://issues.apache.org/jira/browse/ATLAS-4847 > > > Repository: atlas > > > Description > --- > > Steps to reproduce: > > 1. Create a hive_table entity and assign a term to it. > 2. Perform 1st incremental export of that entity without using changeMarker. > 3. Without any modification on that entity perform 2nd consecutive > incremental export with changeMarker from previous export. > > Result is it that it throws NullPointerException and export is failed. > > Problem: > > When an entity which has termed assigned was exported with changeMarker value > greater than value of lastModified time of that entity it resulted in > throwing NullPointerException. > > Solution: > > 1. NullPointerException occured because it was trying to fetch referred > entities of AtlasGlossary Object while performing export operation and as > AtlasGlossary didn't have any referred entities it gave raise this exception. > 2. This problem is solved by handling NullPointer before trying to fetch > referred entities. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java > 4615c6c2f > > > Diff: https://reviews.apache.org/r/74956/diff/1/ > > > Testing > --- > > Manual testing has been done for the same. > > PC link: > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1608/ > > > Thanks, > > Priyanshi Shah > >
Re: Review Request 74951: ATLAS-4845 : Atlas Import is failing with fetchType: Incremental if there are no changes between two consecutive runs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74951/#review226385 --- Ship it! Ship It! - Pinal Shah On April 9, 2024, 1:14 p.m., Priyanshi Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74951/ > --- > > (Updated April 9, 2024, 1:14 p.m.) > > > Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, and > Sheetal Shah. > > > Bugs: ATLAS-4845 > https://issues.apache.org/jira/browse/ATLAS-4845 > > > Repository: atlas > > > Description > --- > > Steps: > > 1. First export and import with changeMarker “0” - worked as expected. > 2. Try to run 2nd export + import without any changes in atlas, 2nd export > also worked and gave differentchangeMarker in output as expected. > > Expectation: > > a) Empty zip should be created if no entities is fetched. > > Problem: > > Export zip file was created with three json files(atlas-typesdef.json, > atlas-export-order.json and atlas-export-info.json) even though no entities > was fetched and hence, while importing these zip files the import was getting > failed. > > Solution: > > 1. The flag "omitZipResponseForEmptyExport" has been created which will > indicate whether to create an empty zip or not when no entities are fetched > during export operation. This flag is introduced to support backward > compatibility for export operation. The default value of this flag is false. > 2. If the "omitZipResponseForEmptyExport" flag is set to true and if no > entities are fetched during export, an empty zip file is created with status > code 204. Otherwise, if the flag value is false zip file is created with 3 > json files (atlas-typesdef.json, atlas-export-order.json and > atlas-export-info.json) with status code as 200. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportRequest.java > b03b386c0 > repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java > 5cec39dd6 > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > 3aa24 > > > Diff: https://reviews.apache.org/r/74951/diff/2/ > > > Testing > --- > > Manual testing for the same has been performed. > > PC is also successfull. > > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1603/ > > > Thanks, > > Priyanshi Shah > >
Re: Review Request 74939: ATLAS-4842 : Export/Import: fetchType as "incremental" does full export instead of connected
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74939/#review226381 --- Ship it! Ship It! - Pinal Shah On April 5, 2024, 8:41 a.m., Priyanshi Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74939/ > --- > > (Updated April 5, 2024, 8:41 a.m.) > > > Review request for atlas, Jayendra Parab, Pinal Shah, and Sheetal Shah. > > > Bugs: ATLAS-4842 > https://issues.apache.org/jira/browse/ATLAS-4842 > > > Repository: atlas > > > Description > --- > > Steps: > > create two hive_tables in same database: > > 1. create table hive1(id int); > 2. create table hive1_ctas as select * from hive1; > 3. create table hive2(name string); > 4. Perform incremental export of hive1 with changeMarker as 0 and skipLineage > value as false. > 5. Import the zip. > > Expectations: > > a) Expectation is only the requested hive_table hive1 and its related > entities should be exported. > > Problem: > > hive2 is also imported which is an independently created entity and not > related to hive1. > > Solution: > > 1. Earlier, the implementation for fetchType as INCREMENTAL was getting > satisfied only if the value of skipLineage was "true" otherwise it was > implementing the default fetchType i.e FULL fetch and hence, all the > independently created entities in same db was also getting fetched. > > 2. Now, the problem is fixed by adding additional condition which will handle > INCREMENTAL fetchType even if the value for skipLineage is false. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/impexp/EntitiesExtractor.java > da5cf37c4 > > repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java > 65d7a1872 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java > 0e3955dcd > > repository/src/test/resources/json/stocksDB-Entities/typedef-new-classification-T3.json > PRE-CREATION > > repository/src/test/resources/json/stocksDB-Entities/typesdef-new-classification-T2.json > PRE-CREATION > > > Diff: https://reviews.apache.org/r/74939/diff/6/ > > > Testing > --- > > Manual testing has been done for the same. > > PC Link : > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1594/ > > > Thanks, > > Priyanshi Shah > >
Re: Review Request 74939: ATLAS-4842 : Export/Import: fetchType as "incremental" does full export instead of connected
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74939/#review226380
---
repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java
Lines 500 (patched)
<https://reviews.apache.org/r/74939/#comment314602>
retain original method and create new one, since it is a public method
public boolean isHiveTableIncrementalSkipLineage() {
return isHiveTableIncremental;
}
public boolean isHiveTableIncremental {
return isHiveTableIncremental;
}
- Pinal Shah
On April 4, 2024, 1:16 p.m., Priyanshi Shah wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74939/
> ---
>
> (Updated April 4, 2024, 1:16 p.m.)
>
>
> Review request for atlas, Jayendra Parab, Pinal Shah, and Sheetal Shah.
>
>
> Bugs: ATLAS-4842
> https://issues.apache.org/jira/browse/ATLAS-4842
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Steps:
>
> create two hive_tables in same database:
>
> 1. create table hive1(id int);
> 2. create table hive1_ctas as select * from hive1;
> 3. create table hive2(name string);
> 4. Perform incremental export of hive1 with changeMarker as 0 and skipLineage
> value as false.
> 5. Import the zip.
>
> Expectations:
>
> a) Expectation is only the requested hive_table hive1 and its related
> entities should be exported.
>
> Problem:
>
> hive2 is also imported which is an independently created entity and not
> related to hive1.
>
> Solution:
>
> 1. Earlier, the implementation for fetchType as INCREMENTAL was getting
> satisfied only if the value of skipLineage was "true" otherwise it was
> implementing the default fetchType i.e FULL fetch and hence, all the
> independently created entities in same db was also getting fetched.
>
> 2. Now, the problem is fixed by adding additional condition which will handle
> INCREMENTAL fetchType even if the value for skipLineage is false.
>
>
> Diffs
> -
>
>
> repository/src/main/java/org/apache/atlas/repository/impexp/EntitiesExtractor.java
> da5cf37c4
>
> repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java
> 65d7a1872
>
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java
> 0e3955dcd
>
> repository/src/test/resources/json/stocksDB-Entities/typedef-new-classification-T3.json
> PRE-CREATION
>
> repository/src/test/resources/json/stocksDB-Entities/typesdef-new-classification-T2.json
> PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/74939/diff/5/
>
>
> Testing
> ---
>
> Manual testing has been done for the same.
>
> PC Link :
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1594/
>
>
> Thanks,
>
> Priyanshi Shah
>
>
Re: Review Request 74787: ATLAS-4817 : Change compression from 'GZ' to 'SNAPPY' in Atlas HBase tables
> On April 3, 2024, 8:23 a.m., Pinal Shah wrote: > > Can we add the tests performed in the description box > > Also Does SNAPPY took 4hr 11min for 18k messages ? less compared to GZ or > > it is mistake Editing above: time taken is more compared to GZ - Pinal --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74787/#review226371 --- On April 3, 2024, 4:19 a.m., Vinayak Marraiya wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74787/ > --- > > (Updated April 3, 2024, 4:19 a.m.) > > > Review request for atlas, Jayendra Parab and Pinal Shah. > > > Repository: atlas > > > Description > --- > > Currently Atlas is using Compression Algorithm as 'GZ' chaning it to 'SNAPPY' > > > Diffs > - > > > graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseStoreManager.java > f98fa7b76 > > repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java > b7f0dd5e2 > > > Diff: https://reviews.apache.org/r/74787/diff/2/ > > > Testing > --- > > 1) Unit tests >New tests added. > >Volume testing >+--+--+-+ >| Description | Message | Duration| >| | Count| | >+--+--+-+ >| Compression | | | >| Alogrithm| 18000 | 1 hr 55 min | >| as "GZ" | | | >+--+--+-+ >| Compression | | | >| Alogrithm| 18000 | 4 hr 11 min | >| as "SNAPPY" | | | >+--+--+-+ > > 2) Precommit - > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1535/console > > > Thanks, > > Vinayak Marraiya > >
Re: Review Request 74787: ATLAS-4817 : Change compression from 'GZ' to 'SNAPPY' in Atlas HBase tables
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74787/#review226371 --- Ship it! Can we add the tests performed in the description box Also Does SNAPPY took 4hr 11min for 18k messages ? less compared to GZ or it is mistake - Pinal Shah On April 3, 2024, 4:19 a.m., Vinayak Marraiya wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74787/ > --- > > (Updated April 3, 2024, 4:19 a.m.) > > > Review request for atlas, Jayendra Parab and Pinal Shah. > > > Repository: atlas > > > Description > --- > > Currently Atlas is using Compression Algorithm as 'GZ' chaning it to 'SNAPPY' > > > Diffs > - > > > graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseStoreManager.java > f98fa7b76 > > repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java > b7f0dd5e2 > > > Diff: https://reviews.apache.org/r/74787/diff/2/ > > > Testing > --- > > 1) Unit tests >New tests added. > >Volume testing >+--+--+-+ >| Description | Message | Duration| >| | Count| | >+--+--+-+ >| Compression | | | >| Alogrithm| 18000 | 1 hr 55 min | >| as "GZ" | | | >+--+--+-+ >| Compression | | | >| Alogrithm| 18000 | 4 hr 11 min | >| as "SNAPPY" | | | >+--+--+-+ > > 2) Precommit - > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1535/console > > > Thanks, > > Vinayak Marraiya > >
Re: Review Request 74922: ATLAS-4827 : Clean Atlas application log
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74922/#review226364 --- Ship it! Ship It! - Pinal Shah On April 3, 2024, 4:19 a.m., Vinayak Marraiya wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74922/ > --- > > (Updated April 3, 2024, 4:19 a.m.) > > > Review request for atlas, Jayendra Parab and Pinal Shah. > > > Repository: atlas > > > Description > --- > > Cleaning unwanted error logs from Atlas application log > > > Diffs > - > > webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java > e19beccb1 > > > Diff: https://reviews.apache.org/r/74922/diff/1/ > > > Testing > --- > > 1)Went through the logs and verify that ERROR log are no more visible after > making it as TRACE log > 2)Precommit - > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1587/console > > > Thanks, > > Vinayak Marraiya > >
Re: Review Request 74947: ATLAS-4838 : Export/Import : changeMarker is not set to entity's lastupdatetime or its closer timestamp value
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74947/#review226361 --- Ship it! Ship It! - Pinal Shah On March 28, 2024, 11:20 a.m., Priyanshi Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74947/ > --- > > (Updated March 28, 2024, 11:20 a.m.) > > > Review request for atlas, Jayendra Parab, Mandar Ambawane, Pinal Shah, and > Sheetal Shah. > > > Bugs: ATLAS-4838 > https://issues.apache.org/jira/browse/ATLAS-4838 > > > Repository: atlas > > > Description > --- > > Steps: > > When an hive table entity is exported using fetchType incremental with > changeMarker 0 , after exporting , the changeMarker in the export response is > not set to recent timestamp. > > Problem: > > The value of changeMarker was set on the basis of the earliest request > recorded on the server. There was some unused pending requests contexts which > were not cleared after its usage and hence, changeMarker the value was > changeMarker was wrongly set. > > Solution: > > Have cleared all the unused pending requests which was found. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java > e72a87713 > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > 0025276bb > webapp/src/main/java/org/apache/atlas/web/service/EmbeddedServer.java > 4c3a1c569 > webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java > ea74d21ad > > > Diff: https://reviews.apache.org/r/74947/diff/2/ > > > Testing > --- > > Performed multiple export operations with incremental fetch type, the change > marker values were updated properly on each export operation. > > PC link: > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1578/ > > > Thanks, > > Priyanshi Shah > >
Re: Review Request 74851: ATLAS-4822 : Relationship Search feature should be configurable
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74851/#review226329 --- Ship it! Ship It! - Pinal Shah On March 14, 2024, 4:05 p.m., Vinayak Marraiya wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74851/ > --- > > (Updated March 14, 2024, 4:05 p.m.) > > > Review request for atlas, Jayendra Parab, Pinal Shah, and Sheetal Shah. > > > Repository: atlas > > > Description > --- > > Relationship search is enabled (for backend and UI), by default and the > value for atlas.relationship.search.enabled is true. > Relationship search will be disabled from backend and UI, if the value for > atlas.relationship.search.enabled is false. > > > Diffs > - > > dashboardv2/public/css/scss/override.scss 0c5f8bc0d > dashboardv2/public/js/main.js f0cea7999 > dashboardv2/public/js/templates/search/SearchLayoutView_tmpl.html f049fa420 > dashboardv2/public/js/utils/Globals.js 431a24b38 > dashboardv2/public/js/views/search/SearchLayoutView.js c59a5d83e > dashboardv3/public/js/main.js 7423c88c3 > dashboardv3/public/js/utils/Globals.js 410d82881 > dashboardv3/public/js/views/search/SearchFilterBrowseLayoutView.js > c81a74fd8 > dashboardv3/public/js/views/search/tree/CustomFilterTreeLayoutView.js > ea7185b35 > intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 090889e1c > > repository/src/main/java/org/apache/atlas/repository/patches/RelationshipTypeNamePatch.java > c3a9106fa > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > 27ec59d37 > webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java d1d19075f > > > Diff: https://reviews.apache.org/r/74851/diff/10/ > > > Testing > --- > > Tested following: > > 1. Checked whether default behaviour (relationship search APIs are disabled > in UI) works. > 2. Checked whether following APIs are disabled when property is set to false. > GET/POST /v2/search/relations > GET /v2/search/relationship > 3. Checked whether current basic search and dsl search functionalities work > even when relationship search is disabled. > 4. Tested disabling and enabling of property > > > Precommit - > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1543/console > > > Thanks, > > Vinayak Marraiya > >
Re: Review Request 74851: ATLAS-4822 : Relationship Search feature should be configurable
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74851/#review226328 --- webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java Lines 549 (patched) <https://reviews.apache.org/r/74851/#comment314571> Remove the check from here, this api is not for relationship edge search. - Pinal Shah On March 13, 2024, 5:09 p.m., Vinayak Marraiya wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74851/ > --- > > (Updated March 13, 2024, 5:09 p.m.) > > > Review request for atlas, Jayendra Parab, Pinal Shah, and Sheetal Shah. > > > Repository: atlas > > > Description > --- > > Relationship search is enabled (for backend and UI), by default and the > value for atlas.relationship.search.enabled is true. > Relationship search will be disabled from backend and UI, if the value for > atlas.relationship.search.enabled is false. > > > Diffs > - > > dashboardv2/public/css/scss/override.scss 0c5f8bc0d > dashboardv2/public/js/main.js f0cea7999 > dashboardv2/public/js/templates/search/SearchLayoutView_tmpl.html f049fa420 > dashboardv2/public/js/utils/Globals.js 431a24b38 > dashboardv2/public/js/views/search/SearchLayoutView.js c59a5d83e > dashboardv3/public/js/main.js 7423c88c3 > dashboardv3/public/js/utils/Globals.js 410d82881 > dashboardv3/public/js/views/search/SearchFilterBrowseLayoutView.js > c81a74fd8 > dashboardv3/public/js/views/search/tree/CustomFilterTreeLayoutView.js > ea7185b35 > intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 090889e1c > > repository/src/main/java/org/apache/atlas/repository/patches/RelationshipTypeNamePatch.java > c3a9106fa > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > 4d59fa3d6 > webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java d1d19075f > > > Diff: https://reviews.apache.org/r/74851/diff/9/ > > > Testing > --- > > Tested following: > > 1. Checked whether default behaviour (relationship search APIs are disabled > in UI) works. > 2. Checked whether following APIs are disabled when property is set to false. > GET/POST /v2/search/relations > GET /v2/search/relationship > 3. Checked whether current basic search and dsl search functionalities work > even when relationship search is disabled. > 4. Tested disabling and enabling of property > > > Precommit - > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1543/console > > > Thanks, > > Vinayak Marraiya > >
Re: Review Request 74902: ATLAS-4826: Provide Liveness and Readyness probes in Atlas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74902/#review226327 --- Ship it! Ship It! - Pinal Shah On Feb. 22, 2024, 11:26 p.m., Disha Talreja wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74902/ > --- > > (Updated Feb. 22, 2024, 11:26 p.m.) > > > Review request for atlas, Jayendra Parab, Pinal Shah, and Radhika Kundam. > > > Bugs: ATLAS-4826 > https://issues.apache.org/jira/browse/ATLAS-4826 > > > Repository: atlas > > > Description > --- > > In order to monitor the application health and be able to restart it if it is > unhealthy we need to check the: > 1) Liveness - if Atlas is live and running as expected > 2) Readyness - if Atlas is live and ready to accept client requests > > > Diffs > - > > repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java > beb90e67b > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > 4d59fa3d6 > webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java > 5b16ba149 > > > Diff: https://reviews.apache.org/r/74902/diff/2/ > > > Testing > --- > > PC Build: > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1551/ > > > Thanks, > > Disha Talreja > >
Re: Review Request 74902: ATLAS-4826: Provide Liveness and Readyness probes in Atlas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74902/#review226234 --- webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java Lines 1148 (patched) <https://reviews.apache.org/r/74902/#comment314476> Will this work for a single instance/ HA is not enabled Wouldn't it show Ready even if patch application is in progress? - Pinal Shah On Feb. 22, 2024, 11:26 p.m., Disha Talreja wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74902/ > --- > > (Updated Feb. 22, 2024, 11:26 p.m.) > > > Review request for atlas, Jayendra Parab, Pinal Shah, and Radhika Kundam. > > > Bugs: ATLAS-4826 > https://issues.apache.org/jira/browse/ATLAS-4826 > > > Repository: atlas > > > Description > --- > > In order to monitor the application health and be able to restart it if it is > unhealthy we need to check the: > 1) Liveness - if Atlas is live and running as expected > 2) Readyness - if Atlas is live and ready to accept client requests > > > Diffs > - > > repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java > beb90e67b > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > 4d59fa3d6 > webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java > 5b16ba149 > > > Diff: https://reviews.apache.org/r/74902/diff/2/ > > > Testing > --- > > PC Build: > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1551/ > > > Thanks, > > Disha Talreja > >
Re: Review Request 74815: ATLAS-4765 : Atlas - Upgrade Spring Security to 5.8.5
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74815/#review226095 --- Ship it! Ship It! - Pinal Shah On Jan. 4, 2024, 9 a.m., Vinayak Marraiya wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74815/ > --- > > (Updated Jan. 4, 2024, 9 a.m.) > > > Review request for atlas, Jayendra Parab and Pinal Shah. > > > Bugs: ATLAS-4765 > https://issues.apache.org/jira/browse/ATLAS-4765 > > > Repository: atlas > > > Description > --- > > Currently Atlas is using Spring Security version 5.7.5 upgrading it to 5.8.5 > > > Diffs > - > > pom.xml c55bec763 > > > Diff: https://reviews.apache.org/r/74815/diff/1/ > > > Testing > --- > > 1)mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip > Atlas server runnning > 2)Quick started was passed successfully > > > Precommit - > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1537/console > > > Thanks, > > Vinayak Marraiya > >
Re: Review Request 74713: ATLAS-4803 : Optimize Edge fetch
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74713/#review226010
---
Ship it!
Ship It!
- Pinal Shah
On Nov. 28, 2023, 3:26 a.m., Paresh Devalia wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74713/
> ---
>
> (Updated Nov. 28, 2023, 3:26 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Mandar Ambawane,
> Pinal Shah, Sheetal Shah, and Sidharth Mishra.
>
>
> Bugs: ATLAS-4803
> https://issues.apache.org/jira/browse/ATLAS-4803
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Kafka lag was not decreasing for ATLAS_HOOK topics, create Entity API was
> taking 50-60 sec per request.
>
> Hive_table typename count was 10mn record.
>
> Impala_lineage_column typename count was 26mn count.
>
> Able to reproduce the issue.
>
> Metrics
>
>
> This difference exists because earlier even fromVertex did not have any
> edges, the search would iterate through all the edges of the toVertex and
> timeConsume was high.
>
>
> Before: "getRelationshipEdge":{"count":10,"timeTaken":5}
> After removing if condition for toVertex.hasEdge:
> "getRelationshipEdge":{"count":10,"timeTaken":80}
>
>
> Diffs
> -
>
>
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
> 0dd573b89
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java
> ef0313e02
>
>
> Diff: https://reviews.apache.org/r/74713/diff/1/
>
>
> Testing
> ---
>
> What was the relationship type?
> __hive_db.table, __hive_table.columns
>
> What entity type was identified and tested , meaning which entity type of
> vertex took time to find edges?
> Impala_column_lineage, impala_process, hive_table, hive_column
>
> What was the count of the edges corresponding to that entity type?
> Hive_column = 28m
> Impala_column_lineage = 24m
>
> Timing before and after
> Before: "getRelationshipEdge":{"count":10,"timeTaken":5}
> After removing if condition for toVertex.hasEdge:
> "getRelationshipEdge":{"count":10,"timeTaken":80}
>
>
> Volume testing
> Initiate kafka dump and lag started decreasing.
>
>
> Thanks,
>
> Paresh Devalia
>
>
Re: Review Request 74713: ATLAS-4803 : Optimize Edge fetch
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74713/#review226009
---
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
Lines 579 (patched)
<https://reviews.apache.org/r/74713/#comment314363>
not required
- Pinal Shah
On Nov. 22, 2023, 5:50 a.m., Paresh Devalia wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74713/
> ---
>
> (Updated Nov. 22, 2023, 5:50 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Mandar Ambawane,
> Pinal Shah, Sheetal Shah, and Sidharth Mishra.
>
>
> Bugs: ATLAS-4803
> https://issues.apache.org/jira/browse/ATLAS-4803
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Kafka lag was not decreasing for ATLAS_HOOK topics, create Entity API was
> taking 50-60 sec per request.
>
> Hive_table typename count was 10mn record.
>
> Impala_lineage_column typename count was 26mn count.
>
> Able to reproduce the issue.
>
> Metrics
>
>
> This difference exists because earlier even fromVertex did not have any
> edges, the search would iterate through all the edges of the toVertex and
> timeConsume was high.
>
>
> Before: "getRelationshipEdge":{"count":10,"timeTaken":5}
> After removing if condition for toVertex.hasEdge:
> "getRelationshipEdge":{"count":10,"timeTaken":80}
>
>
> Diffs
> -
>
>
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
> 0dd573b89
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java
> ef0313e02
>
>
> Diff: https://reviews.apache.org/r/74713/diff/1/
>
>
> Testing
> ---
>
> Volume testing
> Initiate kafka dump and lag started decreasing.
>
>
> Thanks,
>
> Paresh Devalia
>
>
Re: Review Request 74724: ATLAS-4809 : Atlas - Exclude amqp-client from janusgraph-core
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74724/#review225996 --- Ship it! Ship It! - Pinal Shah On Nov. 23, 2023, 7:29 a.m., Sheetal Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74724/ > --- > > (Updated Nov. 23, 2023, 7:29 a.m.) > > > Review request for atlas, Jayendra Parab and Pinal Shah. > > > Bugs: ATLAS-4809 > https://issues.apache.org/jira/browse/ATLAS-4809 > > > Repository: atlas > > > Description > --- > > Upgrade amqp-client to 5.18.0+ > > > Diffs > - > > graphdb/janus-hbase2/pom.xml c2a2e74f0 > tools/atlas-index-repair/pom.xml ab55f4c7f > > > Diff: https://reviews.apache.org/r/74724/diff/2/ > > > Testing > --- > > 1)mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip > Atlas server runnning > 2)Quick started was passed successfully > > > Precommit - > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1513/ > This shows failures which are unrelated to the changes in this patch > > > Thanks, > > Sheetal Shah > >
[jira] [Resolved] (ATLAS-4807) Fix IT test related to Labels
[ https://issues.apache.org/jira/browse/ATLAS-4807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pinal Shah resolved ATLAS-4807. --- Resolution: Fixed > Fix IT test related to Labels > - > > Key: ATLAS-4807 > URL: https://issues.apache.org/jira/browse/ATLAS-4807 > Project: Atlas > Issue Type: Bug > Components: atlas-core > Reporter: Pinal Shah > Assignee: Pinal Shah >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > testSetLabelsByTypeName(org.apache.atlas.web.integration.EntityV2JerseyResourceIT) > test is failing -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4807) Fix IT test related to Labels
[ https://issues.apache.org/jira/browse/ATLAS-4807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pinal Shah updated ATLAS-4807: -- Fix Version/s: 3.0.0 2.3.0 > Fix IT test related to Labels > - > > Key: ATLAS-4807 > URL: https://issues.apache.org/jira/browse/ATLAS-4807 > Project: Atlas > Issue Type: Bug > Components: atlas-core > Reporter: Pinal Shah > Assignee: Pinal Shah >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > testSetLabelsByTypeName(org.apache.atlas.web.integration.EntityV2JerseyResourceIT) > test is failing -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74712: ATLAS-4807: Fix IT test related to Labels
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74712/ --- (Updated Nov. 1, 2023, 9:59 a.m.) Review request for atlas and Jayendra Parab. Bugs: ATLAS-4807 https://issues.apache.org/jira/browse/ATLAS-4807 Repository: atlas Description --- testSetLabelsByTypeName(org.apache.atlas.web.integration.EntityV2JerseyResourceIT) test is failing because of ATLAS-4785 Diffs - webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java 00f0aab1a Diff: https://reviews.apache.org/r/74712/diff/1/ Testing (updated) --- PC run: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1496/ Thanks, Pinal Shah
Review Request 74712: ATLAS-4807: Fix IT test related to Labels
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74712/ --- Review request for atlas and Jayendra Parab. Bugs: ATLAS-4807 https://issues.apache.org/jira/browse/ATLAS-4807 Repository: atlas Description --- testSetLabelsByTypeName(org.apache.atlas.web.integration.EntityV2JerseyResourceIT) test is failing because of ATLAS-4785 Diffs - webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java 00f0aab1a Diff: https://reviews.apache.org/r/74712/diff/1/ Testing --- PC run: Thanks, Pinal Shah
[jira] [Created] (ATLAS-4807) Fix IT test related to Labels
Pinal Shah created ATLAS-4807: - Summary: Fix IT test related to Labels Key: ATLAS-4807 URL: https://issues.apache.org/jira/browse/ATLAS-4807 Project: Atlas Issue Type: Bug Components: atlas-core Reporter: Pinal Shah Assignee: Pinal Shah testSetLabelsByTypeName(org.apache.atlas.web.integration.EntityV2JerseyResourceIT) test is failing -- This message was sent by Atlassian Jira (v8.20.10#820010)
Review Request 74592: ATLAS-4796: Support Opensearch
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74592/ --- Review request for atlas, Jayendra Parab, Radhika Kundam, and Sidharth Mishra. Bugs: ATLAS-4796 https://issues.apache.org/jira/browse/ATLAS-4796 Repository: atlas Description --- **Improvement** : Adding support to opensearch will help users in place of Elasticsearch. **Aproach** : Janusgraph doesn't have support for opensearch as the indexing backend. Hence supporting it, by adding janusgraph elasticsearch dependency and opensearch rest client dependency **Opensearch docker run command** : docker run -d -p 9200:9200 -p 9300:9300 -e "DISABLE_SECURITY_PLUGIN=true" -e "discovery.type=single-node" -e "compatibility.override_main_response_version=true" opensearchproject/opensearch:1.3.0 **Atlas Configurations** : when tested with Opensearch docker atlas.graph.index.search.hostname=localhost atlas.graph.index.search.elasticsearch.client-only=true atlas.graph.index.search.elasticsearch.http.auth.basic.username=admin atlas.graph.index.search.elasticsearch.http.auth.basic.password=admin atlas.graph.index.search.elasticsearch.http.auth.type=BASIC **Note: ** - Quick search/freetext search will not be supported. - Need to check dynamic reindexing Diffs - graphdb/janus/pom.xml 80fe82bfd graphdb/janus/src/main/java/org/janusgraph/diskstorage/es/ElasticSearchSetup.java PRE-CREATION graphdb/janus/src/main/java/org/janusgraph/diskstorage/es/rest/RestClientSetup.java PRE-CREATION graphdb/janus/src/main/java/org/janusgraph/diskstorage/es/rest/RestOpenSearchClient.java PRE-CREATION graphdb/janus/src/main/java/org/janusgraph/diskstorage/es/rest/util/BasicAuthHttpClientConfigCallback.java PRE-CREATION graphdb/janus/src/main/java/org/janusgraph/diskstorage/es/rest/util/ConnectionKeepAliveConfigCallback.java PRE-CREATION graphdb/janus/src/main/java/org/janusgraph/diskstorage/es/rest/util/RestClientAuthenticator.java PRE-CREATION graphdb/janus/src/main/java/org/janusgraph/diskstorage/es/rest/util/SSLConfigurationCallback.java PRE-CREATION intg/src/main/java/org/apache/atlas/ApplicationProperties.java f83dff699 pom.xml fc9def1b4 Diff: https://reviews.apache.org/r/74592/diff/1/ Testing --- Manually tested with Opensearch docker 1.3.0 **How to Test Locally** - As embedded-hbase-elasticsearch profile is not been created - Add above mentioned configurations in atlas-application properties - Manually build Atlas - Make sure hbase is up and running - Make sure opensearch is up and running - Create vertex_index, edge_index collections - Start Atlas. Thanks, Pinal Shah
[jira] [Created] (ATLAS-4796) Support Opensearch
Pinal Shah created ATLAS-4796: - Summary: Support Opensearch Key: ATLAS-4796 URL: https://issues.apache.org/jira/browse/ATLAS-4796 Project: Atlas Issue Type: New Feature Components: atlas-core Reporter: Pinal Shah Assignee: Pinal Shah Adding support to opensearch will help users in place of Elasticsearch. But, janusgraph doesn't have support for opensearch as the indexing backend. Hence supporting it, by adding janusgraph elasticsearch dependency and opensearch rest client dependency -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74545: ATLAS-4783: Fix NoSuchElementException when running different indexing backend than Solr
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74545/#review225659 --- Ship it! Ship It! - Pinal Shah On Aug. 17, 2023, 8:49 a.m., Szymon Orzechowski wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74545/ > --- > > (Updated Aug. 17, 2023, 8:49 a.m.) > > > Review request for atlas, Madhan Neethiraj, Pinal Shah, Prasad Pawar, and > Radhika Kundam. > > > Bugs: ATLAS-4783 > https://issues.apache.org/jira/browse/ATLAS-4783 > > > Repository: atlas > > > Description > --- > > Elasticsearch is in tech preview as an alternative to Solr as indexing > backend. However current Atlas code causes NoSuchElementException when > property atlas.graph.index.search.solr.wait-searcher is not set. This however > shouldn't be a requirement when not using Solr. This patch fixes that. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/ApplicationProperties.java f83dff699 > > > Diff: https://reviews.apache.org/r/74545/diff/1/ > > > Testing > --- > > Ran mvn clean test and tested on my cluster with Elasticsearch > > > Thanks, > > Szymon Orzechowski > >
Re: Review Request 73249: ATLAS-4221 : Solr collection names should be configurable
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73249/#review225658 --- distro/src/bin/atlas_config.py Lines 737 (patched) <https://reviews.apache.org/r/73249/#comment314123> get_solr_collection_name distro/src/conf/atlas-application.properties Lines 289 (patched) <https://reviews.apache.org/r/73249/#comment314122> #289 to #292 duplicate configs distro/src/conf/atlas-application.properties Lines 290 (patched) <https://reviews.apache.org/r/73249/#comment314124> Can you modify configs like atlas.index.name.vertex.index= intg/src/main/java/org/apache/atlas/AtlasConfiguration.java Lines 82 (patched) <https://reviews.apache.org/r/73249/#comment314125> What error do we get, if after deploying Atlas, collection names are changed? - Pinal Shah On Aug. 9, 2023, 6:38 a.m., chaitali wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73249/ > --- > > (Updated Aug. 9, 2023, 6:38 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, > Pinal Shah, and Sarath Subramanian. > > > Bugs: ATLAS-4221 > https://issues.apache.org/jira/browse/ATLAS-4221 > > > Repository: atlas > > > Description > --- > > Currently in Atlas, the collection names are not configurable > > We need to provide functionality where in names of the solr collections can > be customized using property configuration > > > Diffs > - > > common/src/main/java/org/apache/atlas/repository/Constants.java c84e1b2d0 > distro/src/bin/atlas_config.py 31e6fd043 > distro/src/bin/atlas_start.py ffbe16271 > distro/src/conf/atlas-application.properties b5734d7a8 > distro/src/test/python/scripts/TestMetadata.py b813f927f > intg/src/main/java/org/apache/atlas/AtlasConfiguration.java e8c7a15ea > repository/src/test/java/org/apache/atlas/TestSolrConfigurableNames.java > PRE-CREATION > > tools/atlas-index-repair/src/main/java/org/apache/atlas/tools/RepairIndex.java > 8a8e8cafc > > > Diff: https://reviews.apache.org/r/73249/diff/10/ > > > Testing > --- > > mvn clean install > mvn clean package -Pdist,embedded-hbase-solr > atlas server up and running > Collections created with entities linked > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1439/ > > > Thanks, > > chaitali > >
[jira] [Created] (ATLAS-4778) BusinessMetadata string attribute value persists as prefix and suffix
Pinal Shah created ATLAS-4778: - Summary: BusinessMetadata string attribute value persists as prefix and suffix Key: ATLAS-4778 URL: https://issues.apache.org/jira/browse/ATLAS-4778 Project: Atlas Issue Type: Bug Components: atlas-webui Reporter: Pinal Shah -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (ATLAS-4776) Basic Search : not null userDefined Attributes return incorrect results
Pinal Shah created ATLAS-4776:
-
Summary: Basic Search : not null userDefined Attributes return
incorrect results
Key: ATLAS-4776
URL: https://issues.apache.org/jira/browse/ATLAS-4776
Project: Atlas
Issue Type: Bug
Components: atlas-core
Reporter: Pinal Shah
{*}Issue{*}: using not null operator with userDefined attributes, returns some
entities which doesn't have userDefined attirbutes.
{*}Reproduce{*}: Assign one userDefined attribute to X entity, Remove all the
userDefined attribute assigned to X entity. Search with notNull userDefined
attributes. Actual: Lists X entity
Expected: shouldn't list X entity, as all the attributes are removed.
{*}Cause{*}: As the userDefined attributes are stored in json format, when we
remove all the attributes, it removes all the list but it leaves *'{}'* blank
json, which matches the nonNull field.
{*}Fix{*}: While removing the userDefined, check the Empty list, and remove the
json object.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: Review Request 74513: ATLAS-4762 : Basic search with classification as notNull/isNull returns wrong entities
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74513/
---
(Updated July 14, 2023, 7:27 a.m.)
Review request for atlas, Jayendra Parab, Madhan Neethiraj, Mandar Ambawane,
Radhika Kundam, and Sidharth Mishra.
Bugs: ATLAS-4762
https://issues.apache.org/jira/browse/ATLAS-4762
Repository: atlas
Description
---
**Issue:** When the classifications are added and then removed, the
'_classificationNames' field is left with the empty string, hence onselecting
null/notNull operators it returns wrong entities
**Cause:** It is because of Solr version upgrade, untill 8.4.1, Solr supported
non empty string.
**Payload:** {
'queryType': 'BASIC',
'searchParameters': {
'typeName': '_ALL_ENTITY_TYPES',
'excludeDeletedEntities': True,
'includeClassificationAttributes': False,
'includeSubTypes': True,
'includeSubClassifications': True,
'excludeHeaderAttributes': False,
'limit': 1000,
'offset': 0,
'entityFilters': {
'condition': 'AND',
'criterion': [{
'attributeName': '__classificationNames',
'operator': 'notNull',
'attributeValue': ''
}]
},
'attributes': []
}
**Fix:** For IndexQuery : ["" TO *] works to get nonEmpty field entities
For Inmemory Predicates: Used NonEmptyPredicate
Diffs
-
intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java
1de1f122b
repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
49091dcf3
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasBusinessMetadataDefStoreV2.java
ece1a231e
repository/src/test/java/org/apache/atlas/discovery/EntitySearchProcessorTest.java
3f9e74d11
Diff: https://reviews.apache.org/r/74513/diff/3/
Testing
---
Added unittest case
Thanks,
Pinal Shah
[jira] [Assigned] (ATLAS-4762) Basic search with classification as notNull/isNull returns wrong entities
[ https://issues.apache.org/jira/browse/ATLAS-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pinal Shah reassigned ATLAS-4762: - Assignee: Pinal Shah > Basic search with classification as notNull/isNull returns wrong entities > - > > Key: ATLAS-4762 > URL: https://issues.apache.org/jira/browse/ATLAS-4762 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Abhishek Pal > Assignee: Pinal Shah >Priority: Major > > When searching for entities using basic search and filtering by entities > where classification is not null we get entities which are not having any > classification. > This is also true for the case where we are searching for entities with null > classifications > and when using quick search as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74341: ATLAS-4733 : Download Basic and DSL search results
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74341/#review225383
---
Ship it!
Ship It!
- Pinal Shah
On April 14, 2023, 8 a.m., Mandar Ambawane wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74341/
> ---
>
> (Updated April 14, 2023, 8 a.m.)
>
>
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Pinal Shah,
> Radhika Kundam, Sheetal Shah, and Sidharth Mishra.
>
>
> Bugs: ATLAS-4733
> https://issues.apache.org/jira/browse/ATLAS-4733
>
>
> Repository: atlas
>
>
> Description
> ---
>
> This ticket tracks the Server side changes required for the Search result
> download task.
>
> From server side, each download request is considered as a separate thread
> which will run in the background and will generate the csv file according to
> the inputs provided.
>
> The generated files will be stored in the specific location (which is
> configurable).
>
> A scheduler will be run to delete the old files. In this case, the scheduler
> interval and file expiry time is also configurable.
>
>
> Diffs
> -
>
> intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 608342433
>
> intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResultDownloadStatus.java
> PRE-CREATION
>
> repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java
> d94110004
>
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
> 582d97542
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java
> PRE-CREATION
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTaskFactory.java
> PRE-CREATION
> repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java
> 5b4bf71cc
> repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 6f770edb1
> webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java a6ca04f3c
>
>
> Diff: https://reviews.apache.org/r/74341/diff/8/
>
>
> Testing
> ---
>
> PreCommit:
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1338/consoleFull
>
> Request to generate csv file with default columns seen on UI:
>
> curl -u : --request POST
> ":/api/atlas/v2/search/basic/download/create_file" --header
> 'Content-Type: application/json' --data-raw
> '{"searchParameters":{"excludeDeletedEntities":true,"includeSubClassifications":true,"includeSubTypes":true,"includeClassificationAttributes":true,"entityFilters":null,"tagFilters":null,"attributes":[],"limit":25,"offset":0,"typeName":"_ALL_ENTITY_TYPES","classification":null,"termName":null},"attributeLabelMap":{}}'
>
>
> Request to generate csv file with additional columns selected from UI:
>
> curl -u : --request POST
> ":/api/atlas/v2/search/basic/download/create_file" --header
> 'Content-Type: application/json' --data-raw
> '{"searchParameters":{"excludeDeletedEntities":true,"includeSubClassifications":true,"includeSubTypes":true,"includeClassificationAttributes":true,"entityFilters":null,"tagFilters":null,"attributes":["__guid","__state"],"limit":25,"offset":0,"typeName":"_ALL_ENTITY_TYPES","classification":null,"termName":null},"attributeLabelMap":{"Guid":"__guid","Status":"__state"}}'
>
>
> Thanks,
>
> Mandar Ambawane
>
>
Re: Review Request 74341: ATLAS-4733 : Download Basic and DSL search results
> On April 13, 2023, 10:37 a.m., Pinal Shah wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java
> > Lines 166 (patched)
> > <https://reviews.apache.org/r/74341/diff/6/?file=2275775#file2275775line166>
> >
> > please explain the need of this?
is it needed for Header Names?
- Pinal
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74341/#review225376
---
On April 12, 2023, 6:21 a.m., Mandar Ambawane wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74341/
> ---
>
> (Updated April 12, 2023, 6:21 a.m.)
>
>
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Pinal Shah,
> Radhika Kundam, Sheetal Shah, and Sidharth Mishra.
>
>
> Bugs: ATLAS-4733
> https://issues.apache.org/jira/browse/ATLAS-4733
>
>
> Repository: atlas
>
>
> Description
> ---
>
> This ticket tracks the Server side changes required for the Search result
> download task.
>
> From server side, each download request is considered as a separate thread
> which will run in the background and will generate the csv file according to
> the inputs provided.
>
> The generated files will be stored in the specific location (which is
> configurable).
>
> A scheduler will be run to delete the old files. In this case, the scheduler
> interval and file expiry time is also configurable.
>
>
> Diffs
> -
>
> intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 608342433
>
> intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResultDownloadStatus.java
> PRE-CREATION
>
> repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java
> d94110004
>
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
> 582d97542
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java
> PRE-CREATION
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTaskFactory.java
> PRE-CREATION
> repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java
> 5b4bf71cc
> repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 6f770edb1
> webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java a6ca04f3c
>
>
> Diff: https://reviews.apache.org/r/74341/diff/6/
>
>
> Testing
> ---
>
> PreCommit:
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1338/consoleFull
>
> Request to generate csv file with default columns seen on UI:
>
> curl -u : --request POST
> ":/api/atlas/v2/search/basic/download/create_file" --header
> 'Content-Type: application/json' --data-raw
> '{"searchParameters":{"excludeDeletedEntities":true,"includeSubClassifications":true,"includeSubTypes":true,"includeClassificationAttributes":true,"entityFilters":null,"tagFilters":null,"attributes":[],"limit":25,"offset":0,"typeName":"_ALL_ENTITY_TYPES","classification":null,"termName":null},"attributeLabelMap":{}}'
>
>
> Request to generate csv file with additional columns selected from UI:
>
> curl -u : --request POST
> ":/api/atlas/v2/search/basic/download/create_file" --header
> 'Content-Type: application/json' --data-raw
> '{"searchParameters":{"excludeDeletedEntities":true,"includeSubClassifications":true,"includeSubTypes":true,"includeClassificationAttributes":true,"entityFilters":null,"tagFilters":null,"attributes":["__guid","__state"],"limit":25,"offset":0,"typeName":"_ALL_ENTITY_TYPES","classification":null,"termName":null},"attributeLabelMap":{"Guid":"__guid","Status":"__state"}}'
>
>
> Thanks,
>
> Mandar Ambawane
>
>
Re: Review Request 74341: ATLAS-4733 : Download Basic and DSL search results
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74341/#review225376
---
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java
Lines 156 (patched)
<https://reviews.apache.org/r/74341/#comment314020>
Please format the line of code for all the changes of this patch like:
String query = (String) parameters.get(QUERY_KEY);
String typeName = (String) parameters.get(TYPE_NAME_KEY);
String classification = (String) parameters.get(CLASSIFICATION_KEY);
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java
Lines 166 (patched)
<https://reviews.apache.org/r/74341/#comment314019>
please explain the need of this?
- Pinal Shah
On April 12, 2023, 6:21 a.m., Mandar Ambawane wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74341/
> ---
>
> (Updated April 12, 2023, 6:21 a.m.)
>
>
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Pinal Shah,
> Radhika Kundam, Sheetal Shah, and Sidharth Mishra.
>
>
> Bugs: ATLAS-4733
> https://issues.apache.org/jira/browse/ATLAS-4733
>
>
> Repository: atlas
>
>
> Description
> ---
>
> This ticket tracks the Server side changes required for the Search result
> download task.
>
> From server side, each download request is considered as a separate thread
> which will run in the background and will generate the csv file according to
> the inputs provided.
>
> The generated files will be stored in the specific location (which is
> configurable).
>
> A scheduler will be run to delete the old files. In this case, the scheduler
> interval and file expiry time is also configurable.
>
>
> Diffs
> -
>
> intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 608342433
>
> intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResultDownloadStatus.java
> PRE-CREATION
>
> repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java
> d94110004
>
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
> 582d97542
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java
> PRE-CREATION
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTaskFactory.java
> PRE-CREATION
> repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java
> 5b4bf71cc
> repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 6f770edb1
> webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java a6ca04f3c
>
>
> Diff: https://reviews.apache.org/r/74341/diff/6/
>
>
> Testing
> ---
>
> PreCommit:
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1338/consoleFull
>
> Request to generate csv file with default columns seen on UI:
>
> curl -u : --request POST
> ":/api/atlas/v2/search/basic/download/create_file" --header
> 'Content-Type: application/json' --data-raw
> '{"searchParameters":{"excludeDeletedEntities":true,"includeSubClassifications":true,"includeSubTypes":true,"includeClassificationAttributes":true,"entityFilters":null,"tagFilters":null,"attributes":[],"limit":25,"offset":0,"typeName":"_ALL_ENTITY_TYPES","classification":null,"termName":null},"attributeLabelMap":{}}'
>
>
> Request to generate csv file with additional columns selected from UI:
>
> curl -u : --request POST
> ":/api/atlas/v2/search/basic/download/create_file" --header
> 'Content-Type: application/json' --data-raw
> '{"searchParameters":{"excludeDeletedEntities":true,"includeSubClassifications":true,"includeSubTypes":true,"includeClassificationAttributes":true,"entityFilters":null,"tagFilters":null,"attributes":["__guid","__state"],"limit":25,"offset":0,"typeName":"_ALL_ENTITY_TYPES","classification":null,"termName":null},"attributeLabelMap":{"Guid":"__guid","Status":"__state"}}'
>
>
> Thanks,
>
> Mandar Ambawane
>
>
Re: Review Request 74341: ATLAS-4733 : Download Basic and DSL search results
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74341/#review225375
---
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java
Lines 76 (patched)
<https://reviews.apache.org/r/74341/#comment314016>
Please use AtlasConfiguration.SEARCH_MAX_LIMIT
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java
Lines 87 (patched)
<https://reviews.apache.org/r/74341/#comment314017>
Is this check needed?
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java
Lines 151 (patched)
<https://reviews.apache.org/r/74341/#comment314018>
We should use QueryParams.getNormalizedParams()
- Pinal Shah
On April 12, 2023, 6:21 a.m., Mandar Ambawane wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74341/
> ---
>
> (Updated April 12, 2023, 6:21 a.m.)
>
>
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Pinal Shah,
> Radhika Kundam, Sheetal Shah, and Sidharth Mishra.
>
>
> Bugs: ATLAS-4733
> https://issues.apache.org/jira/browse/ATLAS-4733
>
>
> Repository: atlas
>
>
> Description
> ---
>
> This ticket tracks the Server side changes required for the Search result
> download task.
>
> From server side, each download request is considered as a separate thread
> which will run in the background and will generate the csv file according to
> the inputs provided.
>
> The generated files will be stored in the specific location (which is
> configurable).
>
> A scheduler will be run to delete the old files. In this case, the scheduler
> interval and file expiry time is also configurable.
>
>
> Diffs
> -
>
> intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 608342433
>
> intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResultDownloadStatus.java
> PRE-CREATION
>
> repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java
> d94110004
>
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
> 582d97542
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTask.java
> PRE-CREATION
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/searchdownload/SearchResultDownloadTaskFactory.java
> PRE-CREATION
> repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java
> 5b4bf71cc
> repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 6f770edb1
> webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java a6ca04f3c
>
>
> Diff: https://reviews.apache.org/r/74341/diff/6/
>
>
> Testing
> ---
>
> PreCommit:
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1338/consoleFull
>
> Request to generate csv file with default columns seen on UI:
>
> curl -u : --request POST
> ":/api/atlas/v2/search/basic/download/create_file" --header
> 'Content-Type: application/json' --data-raw
> '{"searchParameters":{"excludeDeletedEntities":true,"includeSubClassifications":true,"includeSubTypes":true,"includeClassificationAttributes":true,"entityFilters":null,"tagFilters":null,"attributes":[],"limit":25,"offset":0,"typeName":"_ALL_ENTITY_TYPES","classification":null,"termName":null},"attributeLabelMap":{}}'
>
>
> Request to generate csv file with additional columns selected from UI:
>
> curl -u : --request POST
> ":/api/atlas/v2/search/basic/download/create_file" --header
> 'Content-Type: application/json' --data-raw
> '{"searchParameters":{"excludeDeletedEntities":true,"includeSubClassifications":true,"includeSubTypes":true,"includeClassificationAttributes":true,"entityFilters":null,"tagFilters":null,"attributes":["__guid","__state"],"limit":25,"offset":0,"typeName":"_ALL_ENTITY_TYPES","classification":null,"termName":null},"attributeLabelMap":{"Guid":"__guid","Status":"__state"}}'
>
>
> Thanks,
>
> Mandar Ambawane
>
>
Re: [jira] [Commented] (ATLAS-4742) Everytime changeover happens ,Atlas is taking almost an hour to switch over from PASSIVE to ACTIVE in HA mode.
yes and please incorporate - https://issues.apache.org/jira/browse/ATLAS-4447 as well along with - https://issues.apache.org/jira/browse/ATLAS-4358 Thanks Pinal Shah On Thu, Apr 6, 2023 at 4:58 PM Nixon Rodrigues (Jira) wrote: > > [ > https://issues.apache.org/jira/browse/ATLAS-4742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17709320#comment-17709320 > ] > > Nixon Rodrigues commented on ATLAS-4742: > > > may be hitting this issue - > https://issues.apache.org/jira/browse/ATLAS-4358 > > > Everytime changeover happens ,Atlas is taking almost an hour to switch > over from PASSIVE to ACTIVE in HA mode. > > > -- > > > > Key: ATLAS-4742 > > URL: https://issues.apache.org/jira/browse/ATLAS-4742 > > Project: Atlas > > Issue Type: Bug > >Reporter: Jagadesh Kiran N > >Priority: Major > > Attachments: application.log, image001.png > > > > > > We have tried setting up Apache atlas(2.2.0) in High availability mode > in Kubernetes with separate pods for two instance. > > > > When we simulate a failure on active instance, the failover from passive > instance from active instance is taking over an hour to complete the > transition. This happens when we have created around 300 entities whereas > when we simulated the same failover on fresh install of Atlas, it completed > the transition in 15 minutes. > > > > This is the log before the failover for the passive instance: > > The logs after the failover is in the attached file. This is the > failover during a fresh install. It is doing the similar process but taking > more time when there are more entities created in Atlas. > > Warning : [main:] ~ Query requires iterating over all vertices [(__ > patch.id = JAVA_PATCH__006)]. For better performance, use indexes > (StandardJanusGraphTx$3:1357) > > > > Observation : Everytime switchover happens its recreating repetitively > every-time internal typedef's because of which switchover time is more than > 1 hr > > Please suggest how to go about the same , > > > > -- > This message was sent by Atlassian Jira > (v8.20.10#820010) >
[jira] [Updated] (ATLAS-4727) admin/audits , admin/purge APIs fail with '[__AtlasAuditEntry.startTime] is not indexed in the targeted index [vertex_index]'
[
https://issues.apache.org/jira/browse/ATLAS-4727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pinal Shah updated ATLAS-4727:
--
Fix Version/s: 3.0.0
> admin/audits , admin/purge APIs fail with '[__AtlasAuditEntry.startTime] is
> not indexed in the targeted index [vertex_index]'
> -
>
> Key: ATLAS-4727
> URL: https://issues.apache.org/jira/browse/ATLAS-4727
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Pinal Shah
>Assignee: Pinal Shah
>Priority: Major
> Fix For: 3.0.0, 2.4.0
>
>
> Issue:
> admin/audits , admin/purge calls throw 500 internal server error.
> Error:
> {code:java}
> java.lang.IllegalArgumentException: The used key
> [__AtlasAuditEntry.startTime] is not indexed in the targeted index
> [vertex_index] at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:435)
> at
> org.janusgraph.graphdb.database.IndexSerializer.getOrders(IndexSerializer.java:711)
> at
> org.janusgraph.graphdb.database.IndexSerializer.executeQuery(IndexSerializer.java:726){code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (ATLAS-4727) admin/audits , admin/purge APIs fail with '[__AtlasAuditEntry.startTime] is not indexed in the targeted index [vertex_index]'
[
https://issues.apache.org/jira/browse/ATLAS-4727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pinal Shah updated ATLAS-4727:
--
Fix Version/s: 2.4.0
> admin/audits , admin/purge APIs fail with '[__AtlasAuditEntry.startTime] is
> not indexed in the targeted index [vertex_index]'
> -
>
> Key: ATLAS-4727
> URL: https://issues.apache.org/jira/browse/ATLAS-4727
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Pinal Shah
>Assignee: Pinal Shah
>Priority: Major
> Fix For: 2.4.0
>
>
> Issue:
> admin/audits , admin/purge calls throw 500 internal server error.
> Error:
> {code:java}
> java.lang.IllegalArgumentException: The used key
> [__AtlasAuditEntry.startTime] is not indexed in the targeted index
> [vertex_index] at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:435)
> at
> org.janusgraph.graphdb.database.IndexSerializer.getOrders(IndexSerializer.java:711)
> at
> org.janusgraph.graphdb.database.IndexSerializer.executeQuery(IndexSerializer.java:726){code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: Review Request 74283: ATLAS-4727 : admin/audits , admin/purge APIs fail with '[__AtlasAuditEntry.startTime] is not indexed in the targeted index [vertex_index]'
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74283/ --- (Updated Jan. 23, 2023, 2:07 a.m.) Review request for atlas, Jayendra Parab, Mandar Ambawane, and Sidharth Mishra. Bugs: ATLAS-4727 https://issues.apache.org/jira/browse/ATLAS-4727 Repository: atlas Description --- **Issue:** admin/audits,admin/purge calls throw 500 internal server error. **Error:** java.lang.IllegalArgumentException: The used key [__AtlasAuditEntry.startTime] is not indexed in the targeted index [vertex_index] at com.google.common.base.Preconditions.checkArgument(Preconditions.java:435) at org.janusgraph.graphdb.database.IndexSerializer.getOrders(IndexSerializer.java:711) at org.janusgraph.graphdb.database.IndexSerializer.executeQuery(IndexSerializer.java:726) **Cause:** The attributes of AtlasAuditEntry type where not getting indexed in the Solr, because the entity of AtlasAuditEntry gets created before the attributes gets indexed. Diffs - repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java 8447ae2b4 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasTypeDefGraphStoreV2.java 5da6fdefa repository/src/test/java/org/apache/atlas/TestModules.java a0a635464 Diff: https://reviews.apache.org/r/74283/diff/3/ Testing --- Tested manually Precommit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1313/ Thanks, Pinal Shah
[jira] [Created] (ATLAS-4727) admin/audits , admin/purge APIs fail with '[__AtlasAuditEntry.startTime] is not indexed in the targeted index [vertex_index]'
Pinal Shah created ATLAS-4727:
-
Summary: admin/audits , admin/purge APIs fail with
'[__AtlasAuditEntry.startTime] is not indexed in the targeted index
[vertex_index]'
Key: ATLAS-4727
URL: https://issues.apache.org/jira/browse/ATLAS-4727
Project: Atlas
Issue Type: Bug
Components: atlas-core
Reporter: Pinal Shah
Assignee: Pinal Shah
Issue:
admin/audits , admin/purge calls throw 500 internal server error.
Error:
{code:java}
java.lang.IllegalArgumentException: The used key [__AtlasAuditEntry.startTime]
is not indexed in the targeted index [vertex_index] at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:435)
at
org.janusgraph.graphdb.database.IndexSerializer.getOrders(IndexSerializer.java:711)
at
org.janusgraph.graphdb.database.IndexSerializer.executeQuery(IndexSerializer.java:726){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: Review Request 74195: ATLAS-3403 : Term assigned to historical entity is not visible in Search results page.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74195/#review224893
---
Ship it!
Ship It!
- Pinal Shah
On Nov. 17, 2022, 1:47 p.m., chaitali wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74195/
> ---
>
> (Updated Nov. 17, 2022, 1:47 p.m.)
>
>
> Review request for atlas, Jayendra Parab and Pinal Shah.
>
>
> Bugs: ATLAS-3403
> https://issues.apache.org/jira/browse/ATLAS-3403
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Let us consider that a classification and a term has been assigned to an
> entity.
>
> If that entity is deleted, and then when the "Show Historical Entities"
> filter is enabled on the search results page for that entity, the
> classification is displayed but not the term.
>
> This patch handles above issue
>
>
> Diffs
> -
>
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
> 0ed418330
> repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java
> 27717c313
>
>
> Diff: https://reviews.apache.org/r/74195/diff/3/
>
>
> Testing
> ---
>
> mvn clean install
>
> Entities ,classification,terms,glossaries getting created successfully
>
>
> PC successful
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1284/console
>
>
> Created an Entity of type hdfs_path as "abc" added classification "class1"
> and assigned a term as well "term1"
> then deleted the entity when we search this entity using
> http://localhost:21000/api/atlas/v2/search/basic
>
> without the fix we cant see "term1" in response under "meanings" tab because
> there is a check in the code where if edge is deleted we then term will not
> be assigned to the deleted entity hence we couldnt see it under "show
> historical data"
>
> Below json for output after fixing issue we can see term getting added in
> MeaningNames and meanings:
>
> http://localhost:21000/api/atlas/v2/search/basic
>
>
> {"queryType":"BASIC","searchParameters":{"typeName":"hdfs_path","excludeDeletedEntities":false,"includeClassificationAttributes":true,"includeSubTypes":true,"includeSubClassifications":true,"excludeHeaderAttributes":false,"limit":25,"offset":0,"attributes":[]},"entities":[{"typeName":"hdfs_path","attributes":{"owner":"","createTime":166784580,"qualifiedName":"abc","name":"abc","description":""},"guid":"22f365a3-5d7c-49d0-b2dc-324adf77d3fe","status":"DELETED","displayText":"abc","classificationNames":["class1"],"classifications":[{"typeName":"class1","entityGuid":"22f365a3-5d7c-49d0-b2dc-324adf77d3fe","entityStatus":"DELETED","propagate":true,"validityPeriods":[],"removePropagationsOnEntityDelete":false}],"meaningNames":["term1"],"meanings":[{"termGuid":"c983b9b9-5b0a-409b-9a30-c0328cd1b3ad","relationGuid":"29ddea1a-adbf-4e48-9c85-72b8a97e10ac","displayText":"term1","confidence":0,"qualifiedName":"term1@gloss1"}],"isIncomplete":false,"labels":[]}],"approximateCount":1}
>
>
> Thanks,
>
> chaitali
>
>
Re: Review Request 74195: ATLAS-3403 : Term assigned to historical entity is not visible in Search results page.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74195/#review224866
---
Ship it!
Ship It!
- Pinal Shah
On Nov. 16, 2022, 5:59 a.m., chaitali wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74195/
> ---
>
> (Updated Nov. 16, 2022, 5:59 a.m.)
>
>
> Review request for atlas, Jayendra Parab and Pinal Shah.
>
>
> Bugs: ATLAS-3403
> https://issues.apache.org/jira/browse/ATLAS-3403
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Let us consider that a classification and a term has been assigned to an
> entity.
>
> If that entity is deleted, and then when the "Show Historical Entities"
> filter is enabled on the search results page for that entity, the
> classification is displayed but not the term.
>
> This patch handles above issue
>
>
> Diffs
> -
>
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
> 0ed418330
>
>
> Diff: https://reviews.apache.org/r/74195/diff/1/
>
>
> Testing
> ---
>
> mvn clean install
>
> Entities ,classification,terms,glossaries getting created successfully
>
>
> PC successful
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1284/console
>
>
> Created an Entity of type hdfs_path as "abc" added classification "class1"
> and assigned a term as well "term1"
> then deleted the entity when we search this entity using
> http://localhost:21000/api/atlas/v2/search/basic
>
> without the fix we cant see "term1" in response under "meanings" tab because
> there is a check in the code where if edge is deleted we then term will not
> be assigned to the deleted entity hence we couldnt see it under "show
> historical data"
>
> Below json for output after fixing issue we can see term getting added in
> MeaningNames and meanings:
>
> http://localhost:21000/api/atlas/v2/search/basic
>
>
> {"queryType":"BASIC","searchParameters":{"typeName":"hdfs_path","excludeDeletedEntities":false,"includeClassificationAttributes":true,"includeSubTypes":true,"includeSubClassifications":true,"excludeHeaderAttributes":false,"limit":25,"offset":0,"attributes":[]},"entities":[{"typeName":"hdfs_path","attributes":{"owner":"","createTime":166784580,"qualifiedName":"abc","name":"abc","description":""},"guid":"22f365a3-5d7c-49d0-b2dc-324adf77d3fe","status":"DELETED","displayText":"abc","classificationNames":["class1"],"classifications":[{"typeName":"class1","entityGuid":"22f365a3-5d7c-49d0-b2dc-324adf77d3fe","entityStatus":"DELETED","propagate":true,"validityPeriods":[],"removePropagationsOnEntityDelete":false}],"meaningNames":["term1"],"meanings":[{"termGuid":"c983b9b9-5b0a-409b-9a30-c0328cd1b3ad","relationGuid":"29ddea1a-adbf-4e48-9c85-72b8a97e10ac","displayText":"term1","confidence":0,"qualifiedName":"term1@gloss1"}],"isIncomplete":false,"labels":[]}],"approximateCount":1}
>
>
> Thanks,
>
> chaitali
>
>
Re: Review Request 74180: ATLAS-4691 - Discrepancy in the atlas debug metrics between the active and the not active servers
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74180/#review224826
---
Ship it!
Ship It!
- Pinal Shah
On Oct. 24, 2022, 8:05 a.m., Jayendra Parab wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74180/
> ---
>
> (Updated Oct. 24, 2022, 8:05 a.m.)
>
>
> Review request for atlas, Mandar Ambawane, Pinal Shah, Radhika Kundam, and
> Sidharth Mishra.
>
>
> Bugs: ATLAS-4691
> https://issues.apache.org/jira/browse/ATLAS-4691
>
>
> Repository: atlas
>
>
> Description
> ---
>
> In HA setup, there is difference in response when debug metrics API
> ("admin/debug/metrics") is accessed on active server and passive server.
> The reason for this is debug metrics are maintained in-memory in the active
> server. When "admin/debug/metrics" API call is accessed on the passive
> server, the request isn't redirected to the active server like it does for
> the other APIs, hence the passive node returns empty metrics.
> Adding "admin/debug/metrics" to the list of adminUriNotFiltered, so that the
> API call gets redirected to active server.
>
>
> Diffs
> -
>
> webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java
> a1d30ba87
>
>
> Diff: https://reviews.apache.org/r/74180/diff/1/
>
>
> Testing
> ---
>
> Tested manually on a HA setup, with this change, when the
> "admin/debug/metrics" is sent to passive server, the request is redirected to
> active server and correct response is returned
>
>
> Thanks,
>
> Jayendra Parab
>
>
[jira] [Updated] (ATLAS-4681) Relationship Search : SortBy is not working
[ https://issues.apache.org/jira/browse/ATLAS-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pinal Shah updated ATLAS-4681: -- Fix Version/s: 3.0.0 2.3.0 > Relationship Search : SortBy is not working > --- > > Key: ATLAS-4681 > URL: https://issues.apache.org/jira/browse/ATLAS-4681 > Project: Atlas > Issue Type: Bug > Components: atlas-core > Reporter: Pinal Shah > Assignee: Pinal Shah >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > Currently Relationship Search supports filtering results. There is a sortBy > and sortOrder param that is supposed to sort according to the arguments > provided. > However currently it is return a random order of results even though the > filter is applied. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4681) Relationship Search : SortBy is not working
[ https://issues.apache.org/jira/browse/ATLAS-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pinal Shah updated ATLAS-4681: -- Component/s: atlas-core > Relationship Search : SortBy is not working > --- > > Key: ATLAS-4681 > URL: https://issues.apache.org/jira/browse/ATLAS-4681 > Project: Atlas > Issue Type: Bug > Components: atlas-core > Reporter: Pinal Shah > Assignee: Pinal Shah >Priority: Major > > Currently Relationship Search supports filtering results. There is a sortBy > and sortOrder param that is supposed to sort according to the arguments > provided. > However currently it is return a random order of results even though the > filter is applied. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (ATLAS-4689) UI: Basic Search: Invalid attributes passed in request
Pinal Shah created ATLAS-4689: - Summary: UI: Basic Search: Invalid attributes passed in request Key: ATLAS-4689 URL: https://issues.apache.org/jira/browse/ATLAS-4689 Project: Atlas Issue Type: Bug Components: atlas-webui Reporter: Pinal Shah Steps to Reproduce: Basic Search: # Search with typeName hive_db, attribute filter clusterName. # Search with typeName hive_table The request sent will have the 'clusterName' in 'attributes' field. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4685) Advanced Search : Not Equals operator doesn't work for String type
[ https://issues.apache.org/jira/browse/ATLAS-4685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pinal Shah updated ATLAS-4685: -- Fix Version/s: 3.0.0 2.3.0 > Advanced Search : Not Equals operator doesn't work for String type > -- > > Key: ATLAS-4685 > URL: https://issues.apache.org/jira/browse/ATLAS-4685 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Pinal Shah >Assignee: Pinal Shah >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > from hive_table where name != 'authors' select name, owner > If use the "!=" operator in where clause, it is not working as expected and > it still returns the row which has the value 'authors' -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74163: ATLAS-4681: Relationship Search : SortBy is not working
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74163/ --- (Updated Oct. 12, 2022, 11:27 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Nixon Rodrigues. Bugs: ATLAS-4681 https://issues.apache.org/jira/browse/ATLAS-4681 Repository: atlas Description --- **Issue:** Currently Relationship Search supports filtering results. There is a sortBy and sortOrder param that is supposed to sort according to the arguments provided. However currently it is return a random order of results even though the filter is applied. Diffs - repository/src/main/java/org/apache/atlas/discovery/RelationshipSearchProcessor.java e4c07988d repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 7b36c6c75 repository/src/test/java/org/apache/atlas/discovery/RelationshipSearchProcessorTest.java 3b2b59863 Diff: https://reviews.apache.org/r/74163/diff/1/ Testing (updated) --- Testcase added Precommit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1253/console Thanks, Pinal Shah
[jira] [Updated] (ATLAS-4531) _ALL_ENTITY_TYPES showing same basic search results even when offset is changed
[ https://issues.apache.org/jira/browse/ATLAS-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pinal Shah updated ATLAS-4531: -- Component/s: atlas-core > _ALL_ENTITY_TYPES showing same basic search results even when offset is > changed > --- > > Key: ATLAS-4531 > URL: https://issues.apache.org/jira/browse/ATLAS-4531 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Rahul Kurup >Assignee: Pinal Shah >Priority: Minor > Fix For: 3.0.0, 2.3.0 > > > Whenever we fire a basic search for _ALL_ENTITY_TYPES, the same set of > results are always fetched even if you change the offset. > For ex: > If these are the results when you select page limit as 2 and offset as 0 > (there are total 59 results): > x1 > x2 > Then on increasing the offset to 3 you still get the above results x1 and x2. > This only occurs for _ALL_ENTITY_TYPES. If you search for specific data types > like hive_table and others the behaviour is normal. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4671) Basic Search : Exclude Header attributes of entities from the response
[ https://issues.apache.org/jira/browse/ATLAS-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pinal Shah updated ATLAS-4671: -- Fix Version/s: 3.0.0 > Basic Search : Exclude Header attributes of entities from the response > -- > > Key: ATLAS-4671 > URL: https://issues.apache.org/jira/browse/ATLAS-4671 > Project: Atlas > Issue Type: Improvement > Components: atlas-core > Reporter: Pinal Shah > Assignee: Pinal Shah >Priority: Major > Labels: BasicSearch > Fix For: 3.0.0, 2.3.0 > > > *Issue* : Basic search has AtlasEntityHeader of each entity in the response. > AtlasEntityHeader has many attributes including classification and terms. > hence for each attribute, it will request janusgraph to get all the header > attributes. > *Approach* : To overcome, we can add a flag to exclude other attributes and > add only selected attributes from 'attributes' field in the response -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4531) _ALL_ENTITY_TYPES showing same basic search results even when offset is changed
[ https://issues.apache.org/jira/browse/ATLAS-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pinal Shah updated ATLAS-4531: -- Fix Version/s: 3.0.0 2.3.0 > _ALL_ENTITY_TYPES showing same basic search results even when offset is > changed > --- > > Key: ATLAS-4531 > URL: https://issues.apache.org/jira/browse/ATLAS-4531 > Project: Atlas > Issue Type: Bug >Reporter: Rahul Kurup > Assignee: Pinal Shah >Priority: Minor > Fix For: 3.0.0, 2.3.0 > > > Whenever we fire a basic search for _ALL_ENTITY_TYPES, the same set of > results are always fetched even if you change the offset. > For ex: > If these are the results when you select page limit as 2 and offset as 0 > (there are total 59 results): > x1 > x2 > Then on increasing the offset to 3 you still get the above results x1 and x2. > This only occurs for _ALL_ENTITY_TYPES. If you search for specific data types > like hive_table and others the behaviour is normal. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Review Request 74163: ATLAS-4681: Relationship Search : SortBy is not working
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74163/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Nixon Rodrigues. Bugs: ATLAS-4681 https://issues.apache.org/jira/browse/ATLAS-4681 Repository: atlas Description --- **Issue:** Currently Relationship Search supports filtering results. There is a sortBy and sortOrder param that is supposed to sort according to the arguments provided. However currently it is return a random order of results even though the filter is applied. Diffs - repository/src/main/java/org/apache/atlas/discovery/RelationshipSearchProcessor.java e4c07988d repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 7b36c6c75 repository/src/test/java/org/apache/atlas/discovery/RelationshipSearchProcessorTest.java 3b2b59863 Diff: https://reviews.apache.org/r/74163/diff/1/ Testing --- Testcase added Thanks, Pinal Shah
Re: Review Request 74158: ATLAS-4685: Advanced Search : Not Equals operator doesn't work for String type
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74158/ --- (Updated Oct. 11, 2022, 3:38 a.m.) Review request for atlas, Ashutosh Mestry, Deep Singh, and Jayendra Parab. Bugs: ATLAS-4685 https://issues.apache.org/jira/browse/ATLAS-4685 Repository: atlas Description --- **Issue:** If use "!=" operator in where clause, it is not working as expected and it still returns the row which has the value 'authors' **Example:** from hive_table where name != 'authors' select name, owner Diffs - repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java bc3930250 repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 1d3e245c5 repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 4002a32b1 repository/src/test/java/org/apache/atlas/query/TraversalComposerTest.java 359ac7045 Diff: https://reviews.apache.org/r/74158/diff/1/ Testing (updated) --- Added unit testcases Precommit : https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1251/ Thanks, Pinal Shah
[jira] [Created] (ATLAS-4685) Advanced Search : Not Equals operator doesn't work for String type
Pinal Shah created ATLAS-4685: - Summary: Advanced Search : Not Equals operator doesn't work for String type Key: ATLAS-4685 URL: https://issues.apache.org/jira/browse/ATLAS-4685 Project: Atlas Issue Type: Bug Components: atlas-core Reporter: Pinal Shah Assignee: Pinal Shah from hive_table where name != 'authors' select name, owner If use the "!=" operator in where clause, it is not working as expected and it still returns the row which has the value 'authors' -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74133: ATLAS-4678 : Restrict Relationship Typedef creation if relationshipCategory value is not provided
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74133/#review224768 --- Ship it! Ship It! - Pinal Shah On Oct. 6, 2022, 9:42 a.m., Mandar Ambawane wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74133/ > --- > > (Updated Oct. 6, 2022, 9:42 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, > Pinal Shah, Radhika Kundam, Sarath Subramanian, Sheetal Shah, and Sidharth > Mishra. > > > Bugs: ATLAS-4678 > https://issues.apache.org/jira/browse/ATLAS-4678 > > > Repository: atlas > > > Description > --- > > When Relationship typedef is created without "relationshipCategory", the > default "relationshipCategory" provided as "ASSOCIATION". > For "relationshipCategory" "ASSOCIATION", it is required that both enddefs > should have "isContainer" value as "false". > > while creating the Relationship type, Whenever user does not provide the > "relationshipCategory" value and put one of the "isContainer" value as "true". > Because of this, Relationship type gets created with "relationshipCategory" = > "ASSOCIATION" and one of the Enddefs with "isContainer" = "true" > > When atlas service is restarted, it throws and exception for such > Relationship type. and due to this, entire TypeRegistry does not get > populated. > So even after the Atlas service comes up, user cannot see any Typedefs. > > At the time of create operation, this patch checks Relationship type which > has no "relationshipCategory", it also checks for the "isContainer" values > for both the Enddefs. > If both "isContainer" values are false, then only it sets the default > "relationshipCategory" as "ASSOCIATION" otherwise it throws an exception. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/type/AtlasRelationshipType.java > 5b1737b70 > > > Diff: https://reviews.apache.org/r/74133/diff/3/ > > > Testing > --- > > > Thanks, > > Mandar Ambawane > >
Re: Review Request 74133: ATLAS-4678 : Restrict Relationship Typedef creation if relationshipCategory value is not provided
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74133/#review224767 --- intg/src/main/java/org/apache/atlas/type/AtlasRelationshipType.java Line 315 (original), 315 (patched) <https://reviews.apache.org/r/74133/#comment313570> Can you add an comment stating default relationshipCategory is ASSOCIATION - Pinal Shah On Oct. 6, 2022, 9:42 a.m., Mandar Ambawane wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74133/ > --- > > (Updated Oct. 6, 2022, 9:42 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, > Pinal Shah, Radhika Kundam, Sarath Subramanian, Sheetal Shah, and Sidharth > Mishra. > > > Bugs: ATLAS-4678 > https://issues.apache.org/jira/browse/ATLAS-4678 > > > Repository: atlas > > > Description > --- > > When Relationship typedef is created without "relationshipCategory", the > default "relationshipCategory" provided as "ASSOCIATION". > For "relationshipCategory" "ASSOCIATION", it is required that both enddefs > should have "isContainer" value as "false". > > while creating the Relationship type, Whenever user does not provide the > "relationshipCategory" value and put one of the "isContainer" value as "true". > Because of this, Relationship type gets created with "relationshipCategory" = > "ASSOCIATION" and one of the Enddefs with "isContainer" = "true" > > When atlas service is restarted, it throws and exception for such > Relationship type. and due to this, entire TypeRegistry does not get > populated. > So even after the Atlas service comes up, user cannot see any Typedefs. > > At the time of create operation, this patch checks Relationship type which > has no "relationshipCategory", it also checks for the "isContainer" values > for both the Enddefs. > If both "isContainer" values are false, then only it sets the default > "relationshipCategory" as "ASSOCIATION" otherwise it throws an exception. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/type/AtlasRelationshipType.java > 5b1737b70 > > > Diff: https://reviews.apache.org/r/74133/diff/3/ > > > Testing > --- > > > Thanks, > > Mandar Ambawane > >
Re: Review Request 74101: ATLAS-4668 : When classification is created with multiple super types having same attributes , Atlas doesn't throw an exception
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74101/#review224763 --- Ship it! Ship It! - Pinal Shah On Oct. 6, 2022, 6:19 a.m., Mandar Ambawane wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74101/ > --- > > (Updated Oct. 6, 2022, 6:19 a.m.) > > > Review request for atlas, Jayendra Parab, madhan, Pinal Shah, Radhika Kundam, > Sheetal Shah, and Sidharth Mishra. > > > Bugs: ATLAS-4668 > https://issues.apache.org/jira/browse/ATLAS-4668 > > > Repository: atlas > > > Description > --- > > When an entity type is created with multiple super types having same > attribute name, Atlas throws exception. > > Ex: > > if there are 2 types: > > type1 - attr1, attr2 > > type2- attr1, attr2 > > Both types cant be added to super type to type3 since both type1 and type2 > have same attribute - attr1 > > This patch will check for the exact scenario in case of Classification > Typedef as well. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java > 5b86b808b > intg/src/test/java/org/apache/atlas/type/TestAtlasTypeRegistry.java > 85d041bdb > > > Diff: https://reviews.apache.org/r/74101/diff/4/ > > > Testing > --- > > PreCommit: > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1220/console > > > Thanks, > > Mandar Ambawane > >
Re: Review Request 74133: ATLAS-4678 : Restrict Relationship Typedef creation if relationshipCategory value is not provided
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74133/#review224730 --- repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipDefStoreV2.java Lines 477 (patched) <https://reviews.apache.org/r/74133/#comment313516> Can we not add this validation in AtlasRelationshipType.validateAtlasRelationshipDef() ? - Pinal Shah On Sept. 21, 2022, 6:32 a.m., Mandar Ambawane wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74133/ > --- > > (Updated Sept. 21, 2022, 6:32 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, > Pinal Shah, Radhika Kundam, Sarath Subramanian, Sheetal Shah, and Sidharth > Mishra. > > > Bugs: ATLAS-4678 > https://issues.apache.org/jira/browse/ATLAS-4678 > > > Repository: atlas > > > Description > --- > > When Relationship typedef is created without "relationshipCategory", the > default "relationshipCategory" provided as "ASSOCIATION". > For "relationshipCategory" "ASSOCIATION", it is required that both enddefs > should have "isContainer" value as "false". > > while creating the Relationship type, Whenever user does not provide the > "relationshipCategory" value and put one of the "isContainer" value as "true". > Because of this, Relationship type gets created with "relationshipCategory" = > "ASSOCIATION" and one of the Enddefs with "isContainer" = "true" > > When atlas service is restarted, it throws and exception for such > Relationship type. and due to this, entire TypeRegistry does not get > populated. > So even after the Atlas service comes up, user cannot see any Typedefs. > > At the time of create operation, this patch checks Relationship type which > has no "relationshipCategory", it also checks for the "isContainer" values > for both the Enddefs. > If both "isContainer" values are false, then only it sets the default > "relationshipCategory" as "ASSOCIATION" otherwise it throws an exception. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 608342433 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipDefStoreV2.java > 7bc4b28c3 > > > Diff: https://reviews.apache.org/r/74133/diff/1/ > > > Testing > --- > > > Thanks, > > Mandar Ambawane > >
Re: Review Request 74101: ATLAS-4668 : When classification is created with multiple super types having same attributes , Atlas doesn't throw an exception
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74101/#review224729 --- intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java Line 559 (original), 561 (patched) <https://reviews.apache.org/r/74101/#comment313515> Can you format this to align below line AtlasType type = typeRegistry.getType(attributeDef.getTypeName()); String attributeName = attributeDef.getName(); intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java Line 560 (original), 568 (patched) <https://reviews.apache.org/r/74101/#comment313514> Can you modify ? attributeDef.getName() -> attributeName - Pinal Shah On Sept. 20, 2022, 10:12 a.m., Mandar Ambawane wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74101/ > --- > > (Updated Sept. 20, 2022, 10:12 a.m.) > > > Review request for atlas, Jayendra Parab, madhan, Pinal Shah, Radhika Kundam, > Sheetal Shah, and Sidharth Mishra. > > > Bugs: ATLAS-4668 > https://issues.apache.org/jira/browse/ATLAS-4668 > > > Repository: atlas > > > Description > --- > > When an entity type is created with multiple super types having same > attribute name, Atlas throws exception. > > Ex: > > if there are 2 types: > > type1 - attr1, attr2 > > type2- attr1, attr2 > > Both types cant be added to super type to type3 since both type1 and type2 > have same attribute - attr1 > > This patch will check for the exact scenario in case of Classification > Typedef as well. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/type/AtlasClassificationType.java > 5b86b808b > intg/src/test/java/org/apache/atlas/type/TestAtlasTypeRegistry.java > 85d041bdb > > > Diff: https://reviews.apache.org/r/74101/diff/3/ > > > Testing > --- > > PreCommit: > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1220/console > > > Thanks, > > Mandar Ambawane > >
Re: Review Request 73623: ATLAS-4442 : Support Relationship search
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73623/
---
(Updated Sept. 30, 2022, 10:58 a.m.)
Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath
Subramanian.
Bugs: ATLAS-4442
https://issues.apache.org/jira/browse/ATLAS-4442
Repository: atlas
Description
---
**Current Working :**
Atlas is entirely entity-based – it is not possible to search for relationships.
**Improvement :**
Get all relationships (edges) for particular relationship type, having some
criteria on its properties
**Example :**
Get all the relations matching
type = rel_type and edge_property1 = edge_value1
**Request Payload :**
{"relationshipName" : "rel_type",
"relationshipFilters": { "condition": "OR", "criterion": [
{"attributeName": "edge_property1","operator": "eq",
"attributeValue": "edge_value1" }
] },
"limit":25,
"offset":0 }
**Limitation**
- Search on existing relationship search
- scrubSearchResults
- System attributes are not included.
Diffs
-
common/src/main/java/org/apache/atlas/repository/Constants.java 7cd67a04c
docs/src/documents/Search/SearchRelationship.md PRE-CREATION
docs/src/resources/images/twiki/relationship_search_model.png PRE-CREATION
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
2edca64d7
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusIndexQuery.java
ff3fd4a78
intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java
ce0f84b6e
intg/src/main/java/org/apache/atlas/model/discovery/RelationshipSearchParameters.java
PRE-CREATION
intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java
8e68d0e82
intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationshipHeader.java
ad3b98ed4
intg/src/main/java/org/apache/atlas/model/profile/AtlasUserSavedSearch.java
36ab3b11a
repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java
0f564c200
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
4b113dbef
repository/src/main/java/org/apache/atlas/discovery/GraphIndexQueryBuilder.java
1cd8786be
repository/src/main/java/org/apache/atlas/discovery/RelationshipSearchProcessor.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/discovery/SearchContext.java
b8976e079
repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
f69dc4215
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
9924b2e4c
repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java
44cd8efff
repository/src/main/java/org/apache/atlas/repository/patches/EdgePatchProcessor.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/patches/RelationshipTypeNamePatch.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java
89e9422cd
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
2fce123f8
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java
a0fd71f9a
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
a8fe5a762
repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
9c2975645
repository/src/test/java/org/apache/atlas/BasicTestSetup.java 9b0578fee
repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
ecb398a44
repository/src/test/java/org/apache/atlas/discovery/RelationshipSearchProcessorTest.java
PRE-CREATION
repository/src/test/resources/relationship_search_def.json PRE-CREATION
test-tools/src/main/resources/solr/core-template/solrconfig.xml fa38e72b6
webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 0b044d4cf
Diff: https://reviews.apache.org/r/73623/diff/7/
Testing (updated)
---
added unit test cases
Precommit :
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1236/
Thanks,
Pinal Shah
[jira] [Created] (ATLAS-4681) Relationship Search : SortBy is not working
Pinal Shah created ATLAS-4681: - Summary: Relationship Search : SortBy is not working Key: ATLAS-4681 URL: https://issues.apache.org/jira/browse/ATLAS-4681 Project: Atlas Issue Type: Bug Reporter: Pinal Shah Assignee: Pinal Shah Currently Relationship Search supports filtering results. There is a sortBy and sortOrder param that is supposed to sort according to the arguments provided. However currently it is return a random order of results even though the filter is applied. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 73623: ATLAS-4442 : Support Relationship search
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73623/
---
(Updated Sept. 27, 2022, 11:31 a.m.)
Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Sarath
Subramanian.
Changes
---
rebase patch
Bugs: ATLAS-4442
https://issues.apache.org/jira/browse/ATLAS-4442
Repository: atlas
Description
---
**Current Working :**
Atlas is entirely entity-based – it is not possible to search for relationships.
**Improvement :**
Get all relationships (edges) for particular relationship type, having some
criteria on its properties
**Example :**
Get all the relations matching
type = rel_type and edge_property1 = edge_value1
**Request Payload :**
{"relationshipName" : "rel_type",
"relationshipFilters": { "condition": "OR", "criterion": [
{"attributeName": "edge_property1","operator": "eq",
"attributeValue": "edge_value1" }
] },
"limit":25,
"offset":0 }
**Limitation**
- Search on existing relationship search
- scrubSearchResults
- System attributes are not included.
Diffs (updated)
-
common/src/main/java/org/apache/atlas/repository/Constants.java 7cd67a04c
docs/src/documents/Search/SearchRelationship.md PRE-CREATION
docs/src/resources/images/twiki/relationship_search_model.png PRE-CREATION
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
2edca64d7
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusIndexQuery.java
ff3fd4a78
intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java
ce0f84b6e
intg/src/main/java/org/apache/atlas/model/discovery/RelationshipSearchParameters.java
PRE-CREATION
intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java
8e68d0e82
intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationshipHeader.java
ad3b98ed4
intg/src/main/java/org/apache/atlas/model/profile/AtlasUserSavedSearch.java
36ab3b11a
repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java
0f564c200
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
4b113dbef
repository/src/main/java/org/apache/atlas/discovery/GraphIndexQueryBuilder.java
1cd8786be
repository/src/main/java/org/apache/atlas/discovery/RelationshipSearchProcessor.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/discovery/SearchContext.java
b8976e079
repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
f69dc4215
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
9924b2e4c
repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java
44cd8efff
repository/src/main/java/org/apache/atlas/repository/patches/EdgePatchProcessor.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/patches/RelationshipTypeNamePatch.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java
89e9422cd
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
2fce123f8
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java
a0fd71f9a
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
a8fe5a762
repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
9c2975645
repository/src/test/java/org/apache/atlas/BasicTestSetup.java 9b0578fee
repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
ecb398a44
repository/src/test/java/org/apache/atlas/discovery/RelationshipSearchProcessorTest.java
PRE-CREATION
repository/src/test/resources/relationship_search_def.json PRE-CREATION
test-tools/src/main/resources/solr/core-template/solrconfig.xml fa38e72b6
webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 0b044d4cf
Diff: https://reviews.apache.org/r/73623/diff/6/
Changes: https://reviews.apache.org/r/73623/diff/5-6/
Testing
---
added unit test cases
Thanks,
Pinal Shah
Re: [DRAFT] [REPORT] Apache Atlas - September 2022
Thanks Madhan for putting this together. +1 Thanks, Pinal Shah On Tue, Sep 20, 2022 at 6:08 PM Nikhil Bonte wrote: > Thanks for putting this together Madhan. > > +1 > > Regards > Nikhil P. Bonte > > On Mon, Sep 19, 2022 at 11:45 AM Chaitali Borole < > chaitali.bor...@freestoneinfotech.com> wrote: > > > Thanks Madhan, looks good to me. > > +1 > > > > Thanks, > > Chaitali > > > > On Wed, Sep 14, 2022 at 8:45 AM Madhan Neethiraj > > wrote: > > > > > Atlas community, > > > > > > > > > > > > Please review the following draft report for this quarter. > > > > > > > > > > > > Thanks, > > > > > > Madhan > > > > > > > > > > > > ## Description: > > > > > > The mission of Apache Atlas is the creation and maintenance of software > > > related > > > > > > to a scalable and extensible set of core foundational governance > > services - > > > > > > enabling enterprises to effectively and efficiently meet their > compliance > > > > > > requirements > > > > > > > > > > > > ## Issues: > > > > > > There are no issues requiring board attention. > > > > > > > > > > > > ## Membership Data: > > > > > > Apache Atlas was founded 2017-06-20 (5 years ago) There are currently > 47 > > > > > > committers and 32 PMC members in this project. The Committer-to-PMC > ratio > > > is > > > > > > roughly 3:2. > > > > > > > > > > > > Community changes, past quarter: > > > > > > - No new PMC members. Last addition was Ashutosh Mestry on 2019-04-16. > > > > > > - Mandar Ambawane was added as committer on 2022-09-12 > > > > > > - Prasad Pawar was added as committer on 2022-09-12 > > > > > > - 4 new Atlas contributors in last quarter: > > > > > > - Abhishek Pal, Kundan Kumar Jha, Patrik Marton, Beakal Lemeneh > > > > > > > > > > > > ## Project Activity: > > > > > > - fixes in search API handling of pagination > > > > > > - updates to support authenticated Cassandra cluster as audit > repository > > > > > > - performance improvements in Atlas lineage API and UI > > > > > > - removed Kafka core dependency in Atlas webapp > > > > > > - improvements in import-hive utility to delete entities for deleted > > > objects > > > > > > in Hive > > > > > > - multiple UI fixes and improvements > > > > > > - dependent component version updates: Spring Framework 5.3.21, Spring > > > > > > Security 5.7.2, Netty Project 4.1.78.Final, POI 5.2.2, JUnit 4.13.2 > > > > > > - started a thread on Atlas 2.4.0 release. Looking to complete release > by > > > > > > October 2022 > > > > > > > > > > > > > > > > > > ## Community Health: > > > > > > - dev@atlas.apache.org had a 22% decrease in traffic in the past > quarter > > > (680 > > > > > > emails compared to 868) > > > > > > - u...@atlas.apache.org had a 37% decrease in traffic in the past > > quarter > > > (12 > > > > > > emails compared to 19) > > > > > > - 48 issues opened in JIRA, past quarter (-12% change) > > > > > > - 29 issues closed in JIRA, past quarter (38% increase) > > > > > > - 51 commits in the past quarter (-15% change) > > > > > > - 13 code contributors in the past quarter (8% increase) > > > > > > - 16 PRs opened on GitHub, past quarter (-40% change) > > > > > > - 7 PRs closed on GitHub, past quarter (no change) > > > > > > > > > > > > ## Most Recent releases: > > > > > > 2.2.0 was released on 2021-08-17 > > > > > > 2.1.0 was released on 2020-07-15 > > > > > > 0.8.4 was released on 2019-06-21 > > > > > > > > >
Re: Review Request 74109: ATLAS-4671 : Basic Search : Exclude Header attributes of entities from the response
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74109/
---
(Updated Sept. 21, 2022, 9:48 a.m.)
Review request for atlas, Jayendra Parab, Radhika Kundam, and Sidharth Mishra.
Changes
---
addressed review comments
Bugs: ATLAS-4671
https://issues.apache.org/jira/browse/ATLAS-4671
Repository: atlas
Description (updated)
---
**Issue** : Basic search has AtlasEntityHeader of each entity in the response.
AtlasEntityHeader has many attributes including classification and terms. hence
for each attribute, it will request janusgraph
**Approach** : To overcome, we can add a flag to exclude other attributes and
add only selected attributes from 'attributes' field in the response
**Working** : This improvement can be seen, if user has **entityTypes**,
**excludeHeaderAttributes=true** and **valid entity attributes (not
relationship) in 'attributes' field** , in the request payload.
**Request**: {
"excludeDeletedEntities": true,
"includeSubClassifications": true,
"includeSubTypes": true,
"includeClassificationAttributes": true,
"limit": 25,
"offset": 0,
"typeName": "hdfs_path",
"attributes": ["path", "name"],
"excludeHeaderAttributes": "true"
}
**Response**: {
"queryType": "BASIC",
"searchParameters": {
"typeName": "hdfs_path",
"excludeDeletedEntities": true,
"includeClassificationAttributes": true,
"includeSubTypes": true,
"includeSubClassifications": true,
"limit": 25,
"offset": 0,
"attributes": ["path", "name"]
},
"attributes": {
"name": ["path", "name"],
"values": [
["/data/warehouse/customer", "customer"],
["/data/warehouse/sales", "sales"]
]
},
"approximateCount": 2
}
Diffs (updated)
-
intg/src/main/java/org/apache/atlas/model/discovery/QuickSearchParameters.java
79f5aae0d
intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java
78fb4a48f
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
8fbc22fa0
repository/src/main/java/org/apache/atlas/discovery/SearchContext.java
01954d07e
repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
fbc739652
Diff: https://reviews.apache.org/r/74109/diff/6/
Changes: https://reviews.apache.org/r/74109/diff/5-6/
Testing
---
Unit testcases added
Precommit :
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1215/
Performance Readings:
Basic Search on 20K entities, limit 1000
Thanks,
Pinal Shah
[jira] [Resolved] (ATLAS-4531) _ALL_ENTITY_TYPES showing same basic search results even when offset is changed
[ https://issues.apache.org/jira/browse/ATLAS-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pinal Shah resolved ATLAS-4531. --- Resolution: Fixed > _ALL_ENTITY_TYPES showing same basic search results even when offset is > changed > --- > > Key: ATLAS-4531 > URL: https://issues.apache.org/jira/browse/ATLAS-4531 > Project: Atlas > Issue Type: Bug >Reporter: Rahul Kurup > Assignee: Pinal Shah >Priority: Minor > > Whenever we fire a basic search for _ALL_ENTITY_TYPES, the same set of > results are always fetched even if you change the offset. > For ex: > If these are the results when you select page limit as 2 and offset as 0 > (there are total 59 results): > x1 > x2 > Then on increasing the offset to 3 you still get the above results x1 and x2. > This only occurs for _ALL_ENTITY_TYPES. If you search for specific data types > like hive_table and others the behaviour is normal. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (ATLAS-4531) _ALL_ENTITY_TYPES showing same basic search results even when offset is changed
[ https://issues.apache.org/jira/browse/ATLAS-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17607499#comment-17607499 ] Pinal Shah commented on ATLAS-4531: --- Review board : https://reviews.apache.org/r/74017/ > _ALL_ENTITY_TYPES showing same basic search results even when offset is > changed > --- > > Key: ATLAS-4531 > URL: https://issues.apache.org/jira/browse/ATLAS-4531 > Project: Atlas > Issue Type: Bug >Reporter: Rahul Kurup >Assignee: Pinal Shah >Priority: Minor > > Whenever we fire a basic search for _ALL_ENTITY_TYPES, the same set of > results are always fetched even if you change the offset. > For ex: > If these are the results when you select page limit as 2 and offset as 0 > (there are total 59 results): > x1 > x2 > Then on increasing the offset to 3 you still get the above results x1 and x2. > This only occurs for _ALL_ENTITY_TYPES. If you search for specific data types > like hive_table and others the behaviour is normal. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74109: ATLAS-4671 : Basic Search : Exclude Header attributes of entities from the response
> On Sept. 14, 2022, 11:11 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
> > Lines 478 (patched)
> > <https://reviews.apache.org/r/74109/diff/4/?file=2269974#file2269974line478>
> >
> > Use of HashSet here will eliminate duplicate rows. Is this by choice?
> > Wouldn't this result in incorrect handling of pagination (offset)?
These elimination happens after handling pagination and getting the resultSet.
but yes, you are correct, from user end it will be confusing, that they have
requested for 10, and have got less.
> On Sept. 14, 2022, 11:11 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
> > Lines 507 (patched)
> > <https://reviews.apache.org/r/74109/diff/4/?file=2269974#file2269974line507>
> >
> > return here would result in skipping of scrubSearchResults() - which is
> > incorrect. Please review and update.
> >
> > Instead of introducing #471 - #509, consider updating existing for loop
> > below at #549, like:
> >
> > for (AtlasVertex atlasVertex : resultList) {
> > if (searchContext.excludeHeaderAttributes()) {
> > // code from #475 - #506
> > } else {
> > // existing code
> > ...
> > }
> > }
Correct, but we are skipping many attributes, i.e classifications and also we
don't have 'entities' (we just have the 'attributes') in AtlasSearchResult
I thought to align this with DSL search, Select Clause where AtlasSearchResult
has only 'attributes' field.
If we add support to scrubSearchResult
- we need to create 'entities' from what information
- we need to get 'guid' from the vertex
- we need to get 'classifications'
- we need to add 'referredEntities'
getting classifications and referredEntities will reduce the performance which
we are expecting with this improvement.
I will try implementing this once.
Any suugestions?
> On Sept. 14, 2022, 11:11 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
> > Line 1356 (original), 1356 (patched)
> > <https://reviews.apache.org/r/74109/diff/4/?file=2269976#file2269976line1356>
> >
> > Consider retaining getVertexAttribute() as private, based on comment in
> > EntityDiscoveryService.java #494.
comment on #494,
By entityRetriever.getVertexAttribute();
do you mean entityRetriever.getVertexAttribute(vertex, attribute); ?
We dont have any existing method, other than this which returns the value of
attribute.
And to access this in EntityDiscoveryService, we need to make this as public
- Pinal
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74109/#review224671
---
On Sept. 14, 2022, 12:11 p.m., Pinal Shah wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74109/
> ---
>
> (Updated Sept. 14, 2022, 12:11 p.m.)
>
>
> Review request for atlas, Jayendra Parab, Radhika Kundam, and Sidharth Mishra.
>
>
> Bugs: ATLAS-4671
> https://issues.apache.org/jira/browse/ATLAS-4671
>
>
> Repository: atlas
>
>
> Description
> ---
>
> **Issue** : Basic search has AtlasEntityHeader of each entity in the response.
> AtlasEntityHeader has many attributes including classification and terms.
> hence for each attribute, it will request janusgraph
>
> **Approach** : To overcome, we can add a flag to exclude other attributes and
> add only selected attributes from 'attributes' field in the response
>
> **Request**: {
> "excludeDeletedEntities": true,
> "includeSubClassifications": true,
> "includeSubTypes": true,
> "includeClassificationAttributes": true,
> "limit": 25,
> "offset": 0,
> "typeName": "hdfs_path",
> "attributes": ["path", "name"],
> "excludeHeaderAttributes": "true"
>}
>
> **Response**: {
> "queryType": "BASIC",
> "searchParameters": {
> "typeName": "hdfs_path",
> "excludeDeletedEntities": true,
>
Re: Review Request 74109: ATLAS-4671 : Basic Search : Exclude Header attributes of entities from the response
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74109/
---
(Updated Sept. 14, 2022, 12:11 p.m.)
Review request for atlas, Jayendra Parab, Radhika Kundam, and Sidharth Mishra.
Changes
---
addressed review comments (supported referred attributes)
Bugs: ATLAS-4671
https://issues.apache.org/jira/browse/ATLAS-4671
Repository: atlas
Description
---
**Issue** : Basic search has AtlasEntityHeader of each entity in the response.
AtlasEntityHeader has many attributes including classification and terms. hence
for each attribute, it will request janusgraph
**Approach** : To overcome, we can add a flag to exclude other attributes and
add only selected attributes from 'attributes' field in the response
**Request**: {
"excludeDeletedEntities": true,
"includeSubClassifications": true,
"includeSubTypes": true,
"includeClassificationAttributes": true,
"limit": 25,
"offset": 0,
"typeName": "hdfs_path",
"attributes": ["path", "name"],
"excludeHeaderAttributes": "true"
}
**Response**: {
"queryType": "BASIC",
"searchParameters": {
"typeName": "hdfs_path",
"excludeDeletedEntities": true,
"includeClassificationAttributes": true,
"includeSubTypes": true,
"includeSubClassifications": true,
"limit": 25,
"offset": 0,
"attributes": ["path", "name"]
},
"attributes": {
"name": ["path", "name"],
"values": [
["/data/warehouse/customer", "customer"],
["/data/warehouse/sales", "sales"]
]
},
"approximateCount": 2
}
Diffs (updated)
-
intg/src/main/java/org/apache/atlas/model/discovery/QuickSearchParameters.java
79f5aae0d
intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java
78fb4a48f
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
8fbc22fa0
repository/src/main/java/org/apache/atlas/discovery/SearchContext.java
01954d07e
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
a8fe5a762
repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
fbc739652
Diff: https://reviews.apache.org/r/74109/diff/4/
Changes: https://reviews.apache.org/r/74109/diff/3-4/
Testing
---
Unit testcases added
Precommit :
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1215/
Performance Readings:
Basic Search on 20K entities, limit 1000
Thanks,
Pinal Shah
Re: Review Request 74109: ATLAS-4671 : Basic Search : Exclude Header attributes of entities from the response
> On Sept. 13, 2022, 4:44 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/discovery/SearchContext.java
> > Lines 373 (patched)
> > <https://reviews.apache.org/r/74109/diff/3/?file=2269964#file2269964line373>
> >
> > It is not clear why entity-ref attributes are not allowed when
> > excludeHeaderAttributes is true. Is it for faster query response i.e., to
> > avoid look up? Given the attribute is explicitly asked for, the caller
> > should know the performance impact. Why not allow this case?
>
> Pinal Shah wrote:
> Whenever "excludeHeaderAttributes":"true", response is bit different
>
> Example:
> request: "attributes":["qualifiedName"], "excludeHeaderAttributes":"true"
> response: {
> "entities": [],
> "attributes": {
> "name": ["qualifiedName"],
> "values": [
> ["sales@sys"],
> ["customer@sys"]
> ]
> }
> }
>
> List entities, will be blank
> AttributeSearchResult attributes, populate attribute values here.
>
>
> Now case when
> request: "attributes":["qualifiedName","db"],
> "excludeHeaderAttributes":"true"
> We need to fit below "db" value json, to AttributeSearchResult
> "db": {
> "guid": "6e44a899-320c-4026-a66a-d67a76ef7c41",
> "typeName": "hive_db",
> "uniqueAttributes": {
> "qualifiedName": "sys@cl1"
> }
>}
>
> Madhan Neethiraj wrote:
> AtlasSearchResult.values is of type List>, hence should be
> able to handle obj-ref values as well - right?
yes Madhan
Thanks for reviewing.i will add this support of referred attributes
- Pinal
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74109/#review224662
---
On Sept. 14, 2022, 5:38 a.m., Pinal Shah wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74109/
> ---
>
> (Updated Sept. 14, 2022, 5:38 a.m.)
>
>
> Review request for atlas, Jayendra Parab, Radhika Kundam, and Sidharth Mishra.
>
>
> Bugs: ATLAS-4671
> https://issues.apache.org/jira/browse/ATLAS-4671
>
>
> Repository: atlas
>
>
> Description
> ---
>
> **Issue** : Basic search has AtlasEntityHeader of each entity in the response.
> AtlasEntityHeader has many attributes including classification and terms.
> hence for each attribute, it will request janusgraph
>
> **Approach** : To overcome, we can add a flag to exclude other attributes and
> add only selected attributes from 'attributes' field in the response
>
> **Request**: {
> "excludeDeletedEntities": true,
> "includeSubClassifications": true,
> "includeSubTypes": true,
> "includeClassificationAttributes": true,
> "limit": 25,
> "offset": 0,
> "typeName": "hdfs_path",
> "attributes": ["path", "name"],
> "excludeHeaderAttributes": "true"
>}
>
> **Response**: {
> "queryType": "BASIC",
> "searchParameters": {
> "typeName": "hdfs_path",
> "excludeDeletedEntities": true,
> "includeClassificationAttributes": true,
> "includeSubTypes": true,
> "includeSubClassifications": true,
> "limit": 25,
> "offset": 0,
> "attributes": ["path", "name"]
> },
> "attributes": {
> "name": ["path", "name"],
> "values": [
> ["/data/warehouse/customer", "customer"],
> ["/data/warehouse/sales", "sales"]
> ]
> },
> "approximateCount": 2
> }
>
>
> Diffs
> -
>
> intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 608342433
>
> intg/src/main/java/org/apache/atlas/model/discovery/QuickSearchParameters.java
> 79f5aae0d
> intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java
> 78fb4a48f
>
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
> 8fbc22fa0
> repository/src/main/java/org/apache/atlas/discovery/SearchContext.java
> 01954d07e
>
> repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
> fbc739652
>
>
> Diff: https://reviews.apache.org/r/74109/diff/3/
>
>
> Testing
> ---
>
> Unit testcases added
> Precommit :
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1215/
>
> Performance Readings:
> Basic Search on 20K entities, limit 1000
>
>
> Thanks,
>
> Pinal Shah
>
>
Re: Review Request 74109: ATLAS-4671 : Basic Search : Exclude Header attributes of entities from the response
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74109/
---
(Updated Sept. 14, 2022, 5:38 a.m.)
Review request for atlas, Jayendra Parab, Radhika Kundam, and Sidharth Mishra.
Bugs: ATLAS-4671
https://issues.apache.org/jira/browse/ATLAS-4671
Repository: atlas
Description (updated)
---
**Issue** : Basic search has AtlasEntityHeader of each entity in the response.
AtlasEntityHeader has many attributes including classification and terms. hence
for each attribute, it will request janusgraph
**Approach** : To overcome, we can add a flag to exclude other attributes and
add only selected attributes from 'attributes' field in the response
**Request**: {
"excludeDeletedEntities": true,
"includeSubClassifications": true,
"includeSubTypes": true,
"includeClassificationAttributes": true,
"limit": 25,
"offset": 0,
"typeName": "hdfs_path",
"attributes": ["path", "name"],
"excludeHeaderAttributes": "true"
}
**Response**: {
"queryType": "BASIC",
"searchParameters": {
"typeName": "hdfs_path",
"excludeDeletedEntities": true,
"includeClassificationAttributes": true,
"includeSubTypes": true,
"includeSubClassifications": true,
"limit": 25,
"offset": 0,
"attributes": ["path", "name"]
},
"attributes": {
"name": ["path", "name"],
"values": [
["/data/warehouse/customer", "customer"],
["/data/warehouse/sales", "sales"]
]
},
"approximateCount": 2
}
Diffs
-
intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 608342433
intg/src/main/java/org/apache/atlas/model/discovery/QuickSearchParameters.java
79f5aae0d
intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java
78fb4a48f
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
8fbc22fa0
repository/src/main/java/org/apache/atlas/discovery/SearchContext.java
01954d07e
repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java
fbc739652
Diff: https://reviews.apache.org/r/74109/diff/3/
Testing (updated)
---
Unit testcases added
Precommit :
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1215/
Performance Readings:
Basic Search on 20K entities, limit 1000
Thanks,
Pinal Shah
