[jira] [Commented] (ATLAS-4399) Restore deleted entity
[ https://issues.apache.org/jira/browse/ATLAS-4399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17403250#comment-17403250 ] Sarath Subramanian commented on ATLAS-4399: --- [~josephz], Atlas is a governance tool and its key to store the metadata history of deleted entity as well. Currently there is no way to activate deleted entity in Atlas. If you reactive deleted entity, the metadata information of previous deleted entity is overriden and the history is lost. An entity in Atlas can have only one ACTIVE state and any number of DELETED entity states for the same qualifiedName. > Restore deleted entity > -- > > Key: ATLAS-4399 > URL: https://issues.apache.org/jira/browse/ATLAS-4399 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Zhang Joseph >Priority: Major > Labels: Restore > > Hi All, > I have met this situation that when I delete one entity and entity status > changed from "ACTIVE" to "DELETED", then I want to re-active the entity with > same qualifiedName, but I got two entities with same qualifiedName, one with > status ACTIVE, another with status DELETED. instead of restoring the old one. > Here is my questions: > # Can I create one entity with initial status "DELETED"? > # Do we have approach to restore deleted entity? I have read some clues > about OMRS ways but with no sample, and not find any code in atlas source > code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 73529: ATLAS-4383: Update Atlas website for 2.2.0 release
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73529/#review223356 --- Ship it! Ship It! - Sarath Subramanian On Aug. 18, 2021, 2:44 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73529/ > --- > > (Updated Aug. 18, 2021, 2:44 p.m.) > > > Review request for atlas, Ashutosh Mestry, Disha Talreja, Radhika Kundam, and > Sarath Subramanian. > > > Bugs: ATLAS-4383 > https://issues.apache.org/jira/browse/ATLAS-4383 > > > Repository: atlas > > > Description > --- > > ATLAS-4383: Update Atlas website for 2.2.0 release > > > Diffs > - > > README.txt d8096f31e > docs/docz-lib/config/menu.js d83518e3e > docs/src/documents/Downloads/Downloads.md d1ab04e72 > docs/src/documents/Overview.md 824ec3c67 > docs/src/documents/Setup/InstallationInstruction.md 89ecdc533 > docs/src/documents/Whats-New/WhatsNew-2.2.md PRE-CREATION > docs/src/site/site.xml b841dd439 > > > Diff: https://reviews.apache.org/r/73529/diff/2/ > > > Testing > --- > > Manually checked the md files and formatting. > Precommit - > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/800/ > > > Thanks, > > Sidharth Mishra > >
Re: Review Request 73524: ATLAS-4394: Build error due to wrong version of atllass-buildtools at pom.xml
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73524/#review223350 --- Ship it! Ship It! - Sarath Subramanian On Aug. 17, 2021, 4:54 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73524/ > --- > > (Updated Aug. 17, 2021, 4:54 p.m.) > > > Review request for atlas, Ashutosh Mestry, Disha Talreja, Jayendra Parab, > Nixon Rodrigues, Radhika Kundam, and Sarath Subramanian. > > > Bugs: ATLAS-4394 > https://issues.apache.org/jira/browse/ATLAS-4394 > > > Repository: atlas > > > Description > --- > > ATLAS-4394: Fixed Apache Parent POM artifact and atlas-buildtools version > > > Diffs > - > > pom.xml cf0eb34a7 > > > Diff: https://reviews.apache.org/r/73524/diff/1/ > > > Testing > --- > > Manual successful Build and Precomit - > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/799/ > > > Thanks, > > Sidharth Mishra > >
Re: [VOTE] Release Apache Atlas version 2.2.0 - rc1
+1 for Apache Atlas 2.2.0 rc1
Validated the following:
* Downloaded source and validated MD5 and SHA512 hash
* Validated signature from release manager ("Sidharth Mishra <
sidmis...@apache.org>")
* Verified source doesn't contain any binaries
* No errors in application log startup.
* Build the source successfully using embedded-hbase-solr profile
* Started Atlas and ran quick start to create sample types and entities
* Validated metrics UI works with all entity types and count
* Validated Quick search, Basic search and DSL search, Glossary
* Admin Page - TypeSystem UI, Admin audits
* Tag propagation use-cases
Thanks,
Sarath
On Mon, Aug 9, 2021 at 3:55 PM Sidharth Mishra wrote:
> Atlas team,
>
> Apache Atlas 2.2.0 rc1, with the following fixes since rc0, is now
> available for vote within the dev community.
>
> ATLAS-4390 - Changed Atlas branch to release-2.2.0-rc1
> ATLAS-4391 - NoSuchFileException for {atlas.log.dir}/atlas-metrics.out
> ATLAS-4343 - [Entity Audits] 'Propagated Classification Added'
> Timestamp is < 'Entity
> Created' Timestamp
> ATLAS-4387 - UI: Propagated Classifications shows wrong count for
> entity type on Entity detail
> page
> ATLAS-4386 - Relationship search: Sorting does not work when
> relationship label is passed
>
> Links to the release artifacts are given below. Please review and vote.
>
> The vote will be open for at least 72 hours or until necessary votes
> are reached.
>
> [ ] +1 Approve
>
> [ ] +0 No opinion
>
> [ ] -1 Disapprove (and reason why)
>
> Thanks,
> Sid
>
> List of all issues addressed in this release:
>
> https://issues.apache.org/jira/issues/?jql=project%3DATLAS%20AND%20status%3DResolved%20AND%20fixVersion%3D2.2.0%20ORDER%20BY%20key%20DESC
>
> Git tag for the release:
> https://github.com/apache/atlas/tree/release-2.2.0-rc1
>
> Sources for the release:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz
>
> Source release verification:
>
> PGP Signature:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz.asc
>
> SHA512 Hash:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz.sha512
>
> MD5 Hash:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz.md5
>
>
> Keys to verify the signature of the release artifacts are available
> at: https://dist.apache.org/repos/dist/release/atlas/KEYS
>
>
> New features/enhancements:
>
> - Introduced Deferred Action - Tag propagation will be handled as
> a background process (AtlasTask)
>
> - Added Re-indexing as JAVA_PATCH
>
> - Created JavaPatch to add new super types to existing entities
>
> - Supported for Business Metadata in Atlas Export API
>
> - Supported HA for admin/task API
>
> - Enhanced export service to export Terms
>
> - Added new models and addition attributes to Atlas for Apache
> Flink, GCP and ADLS Gen 2
>
> - Added support to Configure HiveServer2 Hook to send Lineage-only
> Messages
>
> - Improved and enhanced Bulk Glossary Import supporting import
> with relations
>
> - Dependency between Atlas and Kafka Zookeeper removed
>
> - Improved performance of GraphHelper's guid and status getter methods
>
> - Added new privilege - "admin-audits" for Atlas Admin Audits
> authorization
>
> - Basic search Improvements - search text with special characters,
> optimize pagination
>
> - Updated free-text search processor to support Elasticsearch
>
> - Refactored and enhanced Atlas Python Client supporting Python 2.7
>
> - DSL Search enhancements - Support glossary terms and
> relationship, can handle null attribute values, Java Tinkerpop
> Traversals instead of GremlinScriptEngine, support search by
> classification and its attribute, Caching mechanism for translated dsl
> queries
>
> - Enhanced Export/Import Service to conditionally Support
> Simultaneous Operations
>
> - Enhanced NotificationHookConsumer for Large Message Processing
>
> - Enhanced Atlas authorization for Add/Update/Remove
> classification on entities
>
> - Provided option to add mandatory attribute to existing entity
> definition
>
> - Fixed Atlas UI to load faster in case of large number of
> classifications & entities
>
> - Updated versions of dependent libraries/components: JanusGraph,
> elasticsearch, JQuery, Http core, Http Client, slf4j, log4j, ant,
> gremlin, Solr, groovy, netty, Kafka
>
Re: Review Request 73503: ATLAS-4386 : Relationship search: Sorting does not work when passed relationship label
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73503/#review223318 --- Ship it! Ship It! - Sarath Subramanian On Aug. 4, 2021, 7:55 a.m., Pinal Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73503/ > --- > > (Updated Aug. 4, 2021, 7:55 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, > and Sarath Subramanian. > > > Bugs: ATLAS-4386 > https://issues.apache.org/jira/browse/ATLAS-4386 > > > Repository: atlas > > > Description > --- > > **Problem:** > > Relationship Search is not able to sort when relationship label is passed as > a queryParam instead of attribute. > > **Example:** > > api/atlas/v2/search/relationship?limit=25&offset=0&guid=cca5c29d-307f-4a9b-ac6c-79f41a08a90b&relation=__hive_table.columns&sortBy=name&sortOrder=DESCENDING > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java > f2290c672 > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java > ff7fd150e > > repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java > a9fbd435a > > > Diff: https://reviews.apache.org/r/73503/diff/1/ > > > Testing > --- > > added test cases > > > Thanks, > > Pinal Shah > >
Re: Planning for Apache Atlas 2.2.0 release
Thanks for volunteering to release Apache Atlas 2.2 +1 Thanks, Sarath On Tue, Jul 27, 2021 at 12:11 PM Sidharth Mishra wrote: > Dear Atlas community, > > Since Apache Atlas 2.1 was released, there are many new features, > improvements and fixes. Here are few features/enhancements since last > release, Apache Atlas 2.1: > > - Introduced Deferred Action - Tag propagation will be handled as > a background process (AtlasTask) > > - Added Re-indexing as JAVA_PATCH > > - Created JavaPatch to add new super types to existing entities > > - Supported for Business Metadata in Atlas Export API > > - Supported HA for admin/task API > > - Enhanced export service to export Terms > > - Added new models and addition attributes to Atlas for Apache > Flink, GCP and ADLS Gen 2 > > - Added support to Configure HiveServer2 Hook to send Lineage-only > Messages > > - Improved and enhanced Bulk Glossary Import supporting import > with relations > > - Dependency between Atlas and Kafka Zookeeper removed > > - Improved performance of GraphHelper's guid and status getter methods > > - Added new privilege - "admin-audits" for Atlas Admin Audits > authorization > > - Basic search Improvements - search text with special characters, > optimize pagination > > - Updated free-text search processor to support Elasticsearch > > - Refactored and enhanced Atlas Python Client supporting Python 2.7 > > - DSL Search enhancements - Support glossary terms and > relationship, can handle null attribute values, Java Tinkerpop > Traversals instead of GremlinScriptEngine, support search by > classification and its attribute, Caching mechanism for translated dsl > queries > > - Enhanced Export/Import Service to conditionally Support > Simultaneous Operations > > - Enhanced NotificationHookConsumer for Large Message Processing > > - Enhanced Atlas authorization for Add/Update/Remove > classification on entities > > - Provided option to add mandatory attribute to existing entity > definition > > - Fixed Atlas UI to load faster in case of large number of > classifications & entities > > - Updated versions of dependent libraries/components: JanusGraph, > elasticsearch, JQuery, Http core, Http Client, slf4j, log4j, ant, > gremlin, Solr, groovy, netty, Kafka > > There are around 336 plus commits on branch 2.0 and 205 resolved > issues with fix version 2.2.0 after 2.1.0 release. > With these significant improvements in place, it is time for the next > release of Apache Atlas! > > I propose to release Apache Atlas 2.2 by early next month. Please > review and send your comments. > > Regards, > Sidharth >
Re: Review Request 73430: ATLAS-4340: Set Solr wait-searcher property to false by default to make Solr commits async
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73430/ --- (Updated July 27, 2021, 11:32 a.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, and Pinal Shah. Bugs: ATLAS-4340 https://issues.apache.org/jira/browse/ATLAS-4340 Repository: atlas Description --- In Atlas when a transaction is committed, the entries are committed to HBase (primary storage) and Solr (indexing storage). A transaction is rolled-back if the primary storage commit fails, on the other hand when the secondary commit fails (solr), the transaction is not-rolled back and logged as warning and it is recommended to use reindex to repair the missing index documents. This behavior is due to the fact that the primary storage is the source of truth and indexes can be rebuild. In Janusgraph, there is a property for Solr to make solr commits async. This is set to true in Atlas making every commit to wait until the solr commit is successful. This will have a negative impact on performance and is recommended to be false by default. Property: index.[X].solr.wait-searcher When mutating - wait for the index to reflect new mutations before returning. This can have a negative impact on performance. This Jira is about setting the default value for above property to FALSE and can be overridden if need arises. Diffs (updated) - addons/falcon-bridge/src/test/resources/atlas-application.properties 898b69c99 addons/hbase-bridge/src/test/resources/atlas-application.properties 898b69c99 addons/kafka-bridge/src/test/resources/atlas-application.properties 91fd8b092 authorization/src/test/resources/atlas-application.properties 2e02678a6 distro/pom.xml d84f5e7b1 distro/src/bin/atlas_config.py 493a34ad8 distro/src/bin/atlas_start.py 7cf35a92a distro/src/test/python/scripts/TestMetadata.py 662fbddba graphdb/janus/src/test/resources/atlas-application.properties a355234e9 intg/src/main/java/org/apache/atlas/ApplicationProperties.java bf97ab146 intg/src/test/resources/atlas-application.properties 50ce01e70 webapp/src/test/resources/atlas-application.properties 1d45e78f3 Diff: https://reviews.apache.org/r/73430/diff/2/ Changes: https://reviews.apache.org/r/73430/diff/1-2/ Testing --- 1. Precommit Test: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/688/ 2. Build Atlas with embedded Hbase/Solr profile and validated basic sanity tests - running quick start, basic search, tag propagation 3. Performance Test details: Run with default settings - Solr wait-searcher property enabled - true (without patch) -- Start Time : Tue Jun 15 22:26:58 PDT 2021 End Time : Fri Jun 18 02:32:34 PDT 2021 Messages Processed : 91,225 Time Taken : 52 hours 5 mins Rate : ~ 29.2 messages/minute Run with disabled Solr wait-searcher property (will improve solr commit time making it async) - with patch --- Start Time : Mon Jun 14 13:30:04 PDT 2021 End Time : Tue Jun 15 17:23:56 PDT 2021 Messages Processed : 91,225 Time Taken : 27 hours 54 mins Rate : ~ 54.5 messages/minute We see almost 50% perf imrpovement with this change. Thanks, Sarath Subramanian
[jira] [Updated] (ATLAS-4364) Update duplicate Java Patch Id of ProcessNamePatch
[ https://issues.apache.org/jira/browse/ATLAS-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4364: -- Fix Version/s: 2.2.0 3.0.0 > Update duplicate Java Patch Id of ProcessNamePatch > -- > > Key: ATLAS-4364 > URL: https://issues.apache.org/jira/browse/ATLAS-4364 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Fix For: 3.0.0, 2.2.0 > > > Java Patch - ProcessNamePatch has patch_id which is already used for other > Java patch. > Need to update patch_id for ProcessNamePatch. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4348) Atlas-Kafka Hook : When a producer publishes messages to multiple topics, the latest relationship is marked ACTIVE , rest are marked DELETED
[
https://issues.apache.org/jira/browse/ATLAS-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4348:
--
Fix Version/s: 2.2.0
3.0.0
> Atlas-Kafka Hook : When a producer publishes messages to multiple topics, the
> latest relationship is marked ACTIVE , rest are marked DELETED
>
>
> Key: ATLAS-4348
> URL: https://issues.apache.org/jira/browse/ATLAS-4348
> Project: Atlas
> Issue Type: Task
> Components: atlas-core
>Affects Versions: 2.1.0
>Reporter: Radhika Kundam
>Assignee: Radhika Kundam
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments:
> 0001-ATLAS-4348-Atlas-Kafka-Hook-appendRelationshipsOnPar.patch
>
>
> *Steps to reproduce the issue*:
> Created 2 topics : test1, test2
> Using console consumer produced messages to test1 and test2.
> Relationship between console-producer-lineage and test1 is ACTIVE now.
> Relationship between console-producer-lineage and test1 is set to DELETED
> when messages are published to test2.
> *Root cause of issue:*
> When messages published on topic, Atlas hook receives partial_update request
> with only one topic as output and by default partial update deletes
> relationships between previous topics by considering the only topic(which is
> in partial_update request) as the active.
> *Proposed Solution:*
> Introducing typedef option "appendRelationshipsOnPartialUpdate" which will
> hold list of relationship attribute names. While Atlas startup, it will
> resolve entity type definition and all the specified relationship attributes
> definitions will be updated with typedef property "isAppendOnPartialUpdate"
> only for that entity type.
> With this solution, we can keep all the previous relationship entries active
> along with latest relationship entry for the specific entity by providing
> typedef options as below.
> "typeDefOptions": { "appendRelationshipsOnPartialUpdate": "[\"inputs\",
> \"outputs\"]"}
>
> This solution can be used for any entity type to consider all the existing
> relationship entries as active along with new entries from partial_update.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Review Request 73456: ATLAS-4355: UI: Spelling mistake in message after editing Busines Metadata value
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73456/#review223221 --- Ship it! Ship It! - Sarath Subramanian On July 12, 2021, 11:56 p.m., Umesh Padashetty wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73456/ > --- > > (Updated July 12, 2021, 11:56 p.m.) > > > Review request for atlas and Prasad Pawar. > > > Bugs: ATLAS-4355 > https://issues.apache.org/jira/browse/ATLAS-4355 > > > Repository: atlas > > > Description > --- > > "One or more Business Metadada attributess were updated successfully" > > should be > > "One or more Business Metadata attributes were updated successfully" > > > Diffs > - > > > dashboardv2/public/js/views/business_metadata/CreateBusinessMetadataLayoutView.js > 4957002a7 > dashboardv2/public/js/views/entity/EntityBusinessMetaDataView.js 3e61f3a8a > > dashboardv3/public/js/views/business_metadata/CreateBusinessMetadataLayoutView.js > 4957002a7 > dashboardv3/public/js/views/entity/EntityBusinessMetaDataView.js 0735cf47d > > > Diff: https://reviews.apache.org/r/73456/diff/1/ > > > Testing > --- > > > Thanks, > > Umesh Padashetty > >
[jira] [Updated] (ATLAS-4354) atlas import-hive.sh fails with java.lang.AbstractMethodError: Receiver class com.sun.jersey.api.uri.UriBuilderImpl does not define or inherit an implementation of the re
[ https://issues.apache.org/jira/browse/ATLAS-4354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4354: -- Fix Version/s: 2.2.0 3.0.0 > atlas import-hive.sh fails with java.lang.AbstractMethodError: Receiver class > com.sun.jersey.api.uri.UriBuilderImpl does not define or inherit an > implementation of the resolved method abstract uri > > > Key: ATLAS-4354 > URL: https://issues.apache.org/jira/browse/ATLAS-4354 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Fix For: 3.0.0, 2.2.0 > > > While importing hive getting below error. > atlas import-hive.sh fails with java.lang.AbstractMethodError: Receiver class > com.sun.jersey.api.uri.UriBuilderImpl does not define or inherit an > implementation of the resolved method abstract uri > 2021-07-09 23:28:51,803|INFO|MainThread|machine.py:186 - > run()||GUID=78391325-5071-4406-99bd-99c1947a742f|at > javax.ws.rs.core.UriBuilder.fromUri(UriBuilder.java:119) > 2021-07-09 23:28:51,803|INFO|MainThread|machine.py:186 - > run()||GUID=78391325-5071-4406-99bd-99c1947a742f|at > org.apache.atlas.AtlasBaseClient.getAddressIfActive(AtlasBaseClient.java:633) > 2021-07-09 23:28:51,803|INFO|MainThread|machine.py:186 - > run()||GUID=78391325-5071-4406-99bd-99c1947a742f|at > org.apache.atlas.AtlasBaseClient.selectActiveServerAddress(AtlasBaseClient.java:617) > 2021-07-09 23:28:51,803|INFO|MainThread|machine.py:186 - > run()||GUID=78391325-5071-4406-99bd-99c1947a742f|at > org.apache.atlas.AtlasBaseClient.determineActiveServiceURL(AtlasBaseClient.java:323) > 2021-07-09 23:28:51,804|INFO|MainThread|machine.py:186 - > run()||GUID=78391325-5071-4406-99bd-99c1947a742f|at > org.apache.atlas.AtlasBaseClient.initializeState(AtlasBaseClient.java:480) > 2021-07-09 23:28:51,804|INFO|MainThread|machine.py:186 - > run()||GUID=78391325-5071-4406-99bd-99c1947a742f|at > org.apache.atlas.AtlasBaseClient.initializeState(AtlasBaseClient.java:468) > 2021-07-09 23:28:51,804|INFO|MainThread|machine.py:186 - > run()||GUID=78391325-5071-4406-99bd-99c1947a742f|at > org.apache.atlas.AtlasBaseClient.(AtlasBaseClient.java:143) > 2021-07-09 23:28:51,804|INFO|MainThread|machine.py:186 - > run()||GUID=78391325-5071-4406-99bd-99c1947a742f|at > org.apache.atlas.AtlasClientV2.(AtlasClientV2.java:146) > 2021-07-09 23:28:51,804|INFO|MainThread|machine.py:186 - > run()||GUID=78391325-5071-4406-99bd-99c1947a742f|at > org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:158) > 2021-07-09 23:28:51,841|INFO|MainThread|machine.py:186 - > run()||GUID=78391325-5071-4406-99bd-99c1947a742f| -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 73439: ATLAS-4347 : DSL Search : Caching Mechanism for translated dsl queries
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73439/#review223207 --- Ship it! Ship It! - Sarath Subramanian On July 6, 2021, 9:28 p.m., Pinal Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73439/ > --- > > (Updated July 6, 2021, 9:28 p.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, > and Sarath Subramanian. > > > Bugs: ATLAS-4347 > https://issues.apache.org/jira/browse/ATLAS-4347 > > > Repository: atlas > > > Description > --- > > **Approach** > > - DSL Query is converted to Tinkerpop Traversals. > - Caching traversal with respective to query saves time of translation. > - Whenever same query with same limit and offset is fired again, it will pick > the translated query from cache. > > **Config** > > - atlas.dsl.cached.translator when 'true' uses caching mechanism (Default: > true) > > > Diffs > - > > intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 9ef848752 > > repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java > 724cb8244 > > > Diff: https://reviews.apache.org/r/73439/diff/2/ > > > Testing > --- > > Precommit : > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/718/ > > > Thanks, > > Pinal Shah > >
Re: Review Request 73445: ATLAS-4350: Atlas - Jetty Version disclosure in http response
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73445/#review223204 --- Ship it! Ship It! - Sarath Subramanian On July 8, 2021, 3:52 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73445/ > --- > > (Updated July 8, 2021, 3:52 p.m.) > > > Review request for atlas, Ashutosh Mestry, Radhika Kundam, and Sarath > Subramanian. > > > Bugs: ATLAS-4350 > https://issues.apache.org/jira/browse/ATLAS-4350 > > > Repository: atlas > > > Description > --- > > ATLAS-4350: Atlas - Jetty Version disclosure in http response > > > Diffs > - > > webapp/src/main/java/org/apache/atlas/web/service/EmbeddedServer.java > 61aa3134c > webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java > 7a045749f > > > Diff: https://reviews.apache.org/r/73445/diff/1/ > > > Testing > --- > > Manually tested and confirmed that the version is not present as part for > response header: > > Before Change - > > Response Header: > > HTTP/1.1 200 OK > Date: Thu, 08 Jul 2021 22:49:45 GMT > Last-Modified: Tue, 06 Jul 2021 12:06:15 GMT > Content-Type: text/css > Accept-Ranges: bytes > Content-Length: 53485 > Server: Jetty(9.4.39.v20210325) > > After Change - > > > Response Header: > > HTTP/1.1 200 OK > Date: Thu, 08 Jul 2021 22:47:57 GMT > Last-Modified: Wed, 07 Jul 2021 16:51:30 GMT > Content-Type: text/css > Accept-Ranges: bytes > Content-Length: 53485 > > > Thanks, > > Sidharth Mishra > >
Re: Review Request 73439: ATLAS-4347 : DSL Search : Caching Mechanism for translated dsl queries
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73439/#review223198 --- repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java Lines 169 (patched) <https://reviews.apache.org/r/73439/#comment312299> only the query syntax should be cached so that it can be reused not the values Your strategy to cache will generate 5 cache entry for same query pattern, which si not useful: ive_column where table.name = \"sales_fact_daily_mv1\" hive_column where table.name = \"sales_fact_daily_mv2\" hive_column where table.name = \"sales_fact_daily_mv3\" hive_column where table.name = \"sales_fact_daily_mv4\" hive_column where table.name = \"sales_fact_daily_mv5\" - Sarath Subramanian On July 6, 2021, 9:28 p.m., Pinal Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73439/ > --- > > (Updated July 6, 2021, 9:28 p.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, > and Sarath Subramanian. > > > Bugs: ATLAS-4347 > https://issues.apache.org/jira/browse/ATLAS-4347 > > > Repository: atlas > > > Description > --- > > **Approach** > > - DSL Query is converted to Tinkerpop Traversals. > - Caching traversal with respective to query saves time of translation. > - Whenever same query with same limit and offset is fired again, it will pick > the translated query from cache. > > **Config** > > - atlas.dsl.cached.translator when 'true' uses caching mechanism (Default: > true) > > > Diffs > - > > intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 9ef848752 > > repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java > 724cb8244 > > > Diff: https://reviews.apache.org/r/73439/diff/1/ > > > Testing > --- > > Precommit : > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/718/ > > > Thanks, > > Pinal Shah > >
[jira] [Updated] (ATLAS-4339) Atlas should support skip temporary tables using config property
[ https://issues.apache.org/jira/browse/ATLAS-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4339: -- Fix Version/s: 2.2.0 3.0.0 > Atlas should support skip temporary tables using config property > > > Key: ATLAS-4339 > URL: https://issues.apache.org/jira/browse/ATLAS-4339 > Project: Atlas > Issue Type: Task >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Fix For: 3.0.0, 2.2.0 > > > Atlas should skip/allow temporary tables based on user configuration. > At present Atlas skips temporary tables by default. But it should be > configurable so that if user wants Atlas to collect temporary tables they > should be able to configure the option. > Introducing "atlas.hook.hive.skip.temp.tables" config property at Hive Hook > to handle this feature. With this change, by default atlas will skip > temporary tables as earlier. By providing false flag to skip.temp.tables > property, Atlas will collect temporary tables. > *atlas.hook.hive.skip.temp.tables=false* -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4339) Atlas should support skip temporary tables using config property
[ https://issues.apache.org/jira/browse/ATLAS-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4339: -- Labels: hive-hooks (was: ) > Atlas should support skip temporary tables using config property > > > Key: ATLAS-4339 > URL: https://issues.apache.org/jira/browse/ATLAS-4339 > Project: Atlas > Issue Type: Task > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Labels: hive-hooks > Fix For: 3.0.0, 2.2.0 > > > Atlas should skip/allow temporary tables based on user configuration. > At present Atlas skips temporary tables by default. But it should be > configurable so that if user wants Atlas to collect temporary tables they > should be able to configure the option. > Introducing "atlas.hook.hive.skip.temp.tables" config property at Hive Hook > to handle this feature. With this change, by default atlas will skip > temporary tables as earlier. By providing false flag to skip.temp.tables > property, Atlas will collect temporary tables. > *atlas.hook.hive.skip.temp.tables=false* -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4339) Atlas should support skip temporary tables using config property
[ https://issues.apache.org/jira/browse/ATLAS-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4339: -- Affects Version/s: 2.1.0 > Atlas should support skip temporary tables using config property > > > Key: ATLAS-4339 > URL: https://issues.apache.org/jira/browse/ATLAS-4339 > Project: Atlas > Issue Type: Task >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > > Atlas should skip/allow temporary tables based on user configuration. > At present Atlas skips temporary tables by default. But it should be > configurable so that if user wants Atlas to collect temporary tables they > should be able to configure the option. > Introducing "atlas.hook.hive.skip.temp.tables" config property at Hive Hook > to handle this feature. With this change, by default atlas will skip > temporary tables as earlier. By providing false flag to skip.temp.tables > property, Atlas will collect temporary tables. > *atlas.hook.hive.skip.temp.tables=false* -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4339) Atlas should support skip temporary tables using config property
[ https://issues.apache.org/jira/browse/ATLAS-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4339: -- Component/s: atlas-core > Atlas should support skip temporary tables using config property > > > Key: ATLAS-4339 > URL: https://issues.apache.org/jira/browse/ATLAS-4339 > Project: Atlas > Issue Type: Task > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Fix For: 3.0.0, 2.2.0 > > > Atlas should skip/allow temporary tables based on user configuration. > At present Atlas skips temporary tables by default. But it should be > configurable so that if user wants Atlas to collect temporary tables they > should be able to configure the option. > Introducing "atlas.hook.hive.skip.temp.tables" config property at Hive Hook > to handle this feature. With this change, by default atlas will skip > temporary tables as earlier. By providing false flag to skip.temp.tables > property, Atlas will collect temporary tables. > *atlas.hook.hive.skip.temp.tables=false* -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ATLAS-4339) Atlas should support skip temporary tables using config property
[ https://issues.apache.org/jira/browse/ATLAS-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian resolved ATLAS-4339. --- Resolution: Fixed > Atlas should support skip temporary tables using config property > > > Key: ATLAS-4339 > URL: https://issues.apache.org/jira/browse/ATLAS-4339 > Project: Atlas > Issue Type: Task > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Labels: hive-hooks > Fix For: 3.0.0, 2.2.0 > > > Atlas should skip/allow temporary tables based on user configuration. > At present Atlas skips temporary tables by default. But it should be > configurable so that if user wants Atlas to collect temporary tables they > should be able to configure the option. > Introducing "atlas.hook.hive.skip.temp.tables" config property at Hive Hook > to handle this feature. With this change, by default atlas will skip > temporary tables as earlier. By providing false flag to skip.temp.tables > property, Atlas will collect temporary tables. > *atlas.hook.hive.skip.temp.tables=false* -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 73431: ATLAS-4341 : Regression: DSL query with like operator and regex expression does not work
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73431/#review223182 --- Ship it! Ship It! - Sarath Subramanian On June 22, 2021, 10:32 a.m., Pinal Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73431/ > --- > > (Updated June 22, 2021, 10:32 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, > and Sarath Subramanian. > > > Bugs: ATLAS-4341 > https://issues.apache.org/jira/browse/ATLAS-4341 > > > Repository: atlas > > > Description > --- > > **Issue:** > > hdfs_path having 5 entities with suffix 0 to 5 > > /api/atlas/v2/search/dsl?typeName=hdfs_path&query=name%20like%20%22hdfs_path_vzxtp_*%22 > **Finds the result** > > But > > /api/atlas/v2/search/dsl?typeName=hdfs_path&query=name like > "hdfs_path_vzxtp_[0-9]" > **Returns empty** > > > Diffs > - > > repository/src/main/java/org/apache/atlas/query/IdentifierHelper.java > d2906ea40 > repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java > 3404dc6e2 > > repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java > 6220c23b9 > > > Diff: https://reviews.apache.org/r/73431/diff/3/ > > > Testing > --- > > Manually verified > > Precommit : > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/693/ > (In Progress) > > > Thanks, > > Pinal Shah > >
Review Request 73430: ATLAS-4340: Set Solr wait-searcher property to false by default to make Solr commits async
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73430/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, and Pinal Shah. Bugs: ATLAS-4340 https://issues.apache.org/jira/browse/ATLAS-4340 Repository: atlas Description --- In Atlas when a transaction is committed, the entries are committed to HBase (primary storage) and Solr (indexing storage). A transaction is rolled-back if the primary storage commit fails, on the other hand when the secondary commit fails (solr), the transaction is not-rolled back and logged as warning and it is recommended to use reindex to repair the missing index documents. This behavior is due to the fact that the primary storage is the source of truth and indexes can be rebuild. In Janusgraph, there is a property for Solr to make solr commits async. This is set to true in Atlas making every commit to wait until the solr commit is successful. This will have a negative impact on performance and is recommended to be false by default. Property: index.[X].solr.wait-searcher When mutating - wait for the index to reflect new mutations before returning. This can have a negative impact on performance. This Jira is about setting the default value for above property to FALSE and can be overridden if need arises. Diffs - intg/src/main/java/org/apache/atlas/ApplicationProperties.java bf97ab146 repository/src/test/java/org/apache/atlas/BasicTestSetup.java a1d7b62a5 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java a9fbd435a repository/src/test/java/org/apache/atlas/discovery/ClassificationSearchProcessorTest.java 121dca9f5 repository/src/test/java/org/apache/atlas/discovery/EntitySearchProcessorTest.java 764568915 repository/src/test/java/org/apache/atlas/repository/store/graph/v2/EntityCorrelationStoreTest.java a3be5f4ed webapp/src/test/java/org/apache/atlas/web/adapters/TypeDefsRESTTest.java 962345b22 Diff: https://reviews.apache.org/r/73430/diff/1/ Testing --- 1. Precommit Test: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/688/ 2. Build Atlas with embedded Hbase/Solr profile and validated basic sanity tests - running quick start, basic search, tag propagation 3. Performance Test details: Run with default settings - Solr wait-searcher property enabled - true (without patch) -- Start Time : Tue Jun 15 22:26:58 PDT 2021 End Time : Fri Jun 18 02:32:34 PDT 2021 Messages Processed : 91,225 Time Taken : 52 hours 5 mins Rate : ~ 29.2 messages/minute Run with disabled Solr wait-searcher property (will improve solr commit time making it async) - with patch --- Start Time : Mon Jun 14 13:30:04 PDT 2021 End Time : Tue Jun 15 17:23:56 PDT 2021 Messages Processed : 91,225 Time Taken : 27 hours 54 mins Rate : ~ 54.5 messages/minute We see almost 50% perf imrpovement with this change. Thanks, Sarath Subramanian
[jira] [Updated] (ATLAS-4340) Set Solr wait-searcher property to false by default to make Solr commits async
[ https://issues.apache.org/jira/browse/ATLAS-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4340: -- Attachment: ATLAS-4340-001.patch > Set Solr wait-searcher property to false by default to make Solr commits async > -- > > Key: ATLAS-4340 > URL: https://issues.apache.org/jira/browse/ATLAS-4340 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.1.0 > Reporter: Sarath Subramanian > Assignee: Sarath Subramanian >Priority: Major > Labels: perfomance, solr > Fix For: 3.0.0, 2.2.0 > > Attachments: ATLAS-4340-001.patch > > > In Atlas when a transaction is committed, the entries are committed to HBase > (primary storage) and Solr (indexing storage). A transaction is rolled-back > if the primary storage commit fails, on the other hand when the secondary > commit fails (solr), the transaction is not-rolled back and logged as warning > and it is recommended to use reindex to repair the missing index documents. > This behavior is due to the fact that the primary storage is the source of > truth and indexes can be rebuild. > In Janusgraph, there is a property for Solr to make solr commits async. This > is set to *true* in Atlas making every commit to wait until the solr commit > is successful. This will have a negative impact on performance and is > recommended to be false by default. > Property: *index.[X].solr.wait-searcher* > |When mutating - wait for the index to reflect new mutations before > returning. This can have a negative impact on performance.| > > This Jira is about setting the default value for above property to FALSE and > can be overridden if need arises. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-4340) Set Solr wait-searcher property to false by default to make Solr commits async
Sarath Subramanian created ATLAS-4340: - Summary: Set Solr wait-searcher property to false by default to make Solr commits async Key: ATLAS-4340 URL: https://issues.apache.org/jira/browse/ATLAS-4340 Project: Atlas Issue Type: Improvement Components: atlas-core Affects Versions: 2.1.0 Reporter: Sarath Subramanian Assignee: Sarath Subramanian Fix For: 3.0.0, 2.2.0 In Atlas when a transaction is committed, the entries are committed to HBase (primary storage) and Solr (indexing storage). A transaction is rolled-back if the primary storage commit fails, on the other hand when the secondary commit fails (solr), the transaction is not-rolled back and logged as warning and it is recommended to use reindex to repair the missing index documents. This behavior is due to the fact that the primary storage is the source of truth and indexes can be rebuild. In Janusgraph, there is a property for Solr to make solr commits async. This is set to *true* in Atlas making every commit to wait until the solr commit is successful. This will have a negative impact on performance and is recommended to be false by default. Property: *index.[X].solr.wait-searcher* |When mutating - wait for the index to reflect new mutations before returning. This can have a negative impact on performance.| This Jira is about setting the default value for above property to FALSE and can be overridden if need arises. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 73426: ATLAS-4290 [Atlas: Glossary Term Bulk Import] There is not much info available in logs importing terms in bulk
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73426/#review223168 --- Ship it! Ship It! - Sarath Subramanian On June 16, 2021, 5:36 a.m., Mandar Ambawane wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73426/ > --- > > (Updated June 16, 2021, 5:36 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, > and Sarath Subramanian. > > > Bugs: ATLAS-4290 > https://issues.apache.org/jira/browse/ATLAS-4290 > > > Repository: atlas > > > Description > --- > > Added logging statement > > > Diffs > - > > repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java > 9c84598 > > > Diff: https://reviews.apache.org/r/73426/diff/1/ > > > Testing > --- > > > Thanks, > > Mandar Ambawane > >
Re: Review Request 73394: ATLAS-4317 : Upgrade curator jar version to 4.3.0
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73394/#review223167 --- Ship it! Ship It! - Sarath Subramanian On June 1, 2021, 9:06 a.m., chaitali wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73394/ > --- > > (Updated June 1, 2021, 9:06 a.m.) > > > Review request for atlas, Jayendra Parab, Nixon Rodrigues, and Sarath > Subramanian. > > > Bugs: ATLAS-4317 > https://issues.apache.org/jira/browse/ATLAS-4317 > > > Repository: atlas > > > Description > --- > > Upgrade curator jar version to 4.3.0 > > > Diffs > - > > pom.xml 91c23ee04 > > > Diff: https://reviews.apache.org/r/73394/diff/1/ > > > Testing > --- > > Atlas server running successfully > mvn clean install > mvn clean package -Pdist,embedded-hbase-solr -DskipTests -Drat.skip > > > Thanks, > > chaitali > >
Re: Review Request 73404: ATLAS-4324: FS entity created for load data inpath is created as shell entity - Part 2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73404/#review223135 --- Ship it! Ship It! - Sarath Subramanian On June 8, 2021, 4:07 p.m., Ashutosh Mestry wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73404/ > --- > > (Updated June 8, 2021, 4:07 p.m.) > > > Review request for atlas, Mandar Ambawane, Radhika Kundam, and Sarath > Subramanian. > > > Bugs: ATLAS-4324 > https://issues.apache.org/jira/browse/ATLAS-4324 > > > Repository: atlas > > > Description > --- > > **Approach** > Updates: > - Simplified approach. > - Maintained abstraction. > - Updated unit tests. > > > Diffs > - > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java > 94ef22577 > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilter.java > 737c637bf > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java > 9163c4709 > > addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilterTest.java > 4dde1dc2f > addons/hive-bridge/src/test/resources/json/hs2-create-db-v2.json 881ee105e > > addons/hive-bridge/src/test/resources/json/hs2-create-db-with-no-pathentities-to-retain-v2.json > 42553b5cf > > addons/hive-bridge/src/test/resources/json/hs2-create-db-with-no-pathentities-to-retain.json > a5b810f72 > addons/hive-bridge/src/test/resources/json/hs2-create-process-v2.json > 8e55b7282 > addons/hive-bridge/src/test/resources/json/hs2-create-table-v2.json > 801918e04 > addons/hive-bridge/src/test/resources/json/hs2-load-inpath-v2.json > dd31aa073 > > addons/hive-bridge/src/test/resources/json/hs2-load-inpath-with-no-pathentities-to-retain-v2.json > 7f90d1986 > > addons/hive-bridge/src/test/resources/json/hs2-load-inpath-with-no-pathentities-to-retain.json > dd31aa073 > addons/hive-bridge/src/test/resources/json/hs2-load-inpath.json dd31aa073 > addons/hive-bridge/src/test/resources/json/hs2-table-rename-v2.json > f2b2bf7a8 > > > Diff: https://reviews.apache.org/r/73404/diff/2/ > > > Testing > --- > > **Unit tests** > - Additional unit test. > > **PC Build** > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/617/ > > > File Attachments > > > Screen Shot 2021-06-08 at 4.05.26 PM.png > > https://reviews.apache.org/media/uploaded/files/2021/06/08/b15b33fc-007d-4ee3-8862-64750af16432__Screen_Shot_2021-06-08_at_4.05.26_PM.png > > > Thanks, > > Ashutosh Mestry > >
Review Request 73409: ATLAS-4330: Add Kafka topics lag information on metrics and log
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73409/
---
Review request for atlas, Ashutosh Mestry, Deep Singh, Jayendra Parab, Madhan
Neethiraj, Nikhil Bonte, Nixon Rodrigues, Radhika Kundam, and Sidharth Mishra.
Bugs: ATLAS-4330
https://issues.apache.org/jira/browse/ATLAS-4330
Repository: atlas
Description
---
In metrics page, in addition to other Notification details. Lag information of
kafka consumer topics will be useful to check the current backed up messages in
Atlas:
The solution involves using the kafka admin client APIs to get log end offset
of a topic for the consumer ('atlas')
Config property:
atlas.notification.consumer.topic.report.frequency.seconds=60 (default value
3600secs or 1 hour)
Diffs
-
common/src/main/java/org/apache/atlas/utils/KafkaUtils.java 167442259
dashboardv2/public/js/templates/site/Statistics_Topic_Offset_table_tmpl.html
b4c8d8677
dashboardv2/public/js/views/site/Statistics.js 32a66691c
dashboardv3/public/js/templates/site/Statistics_Topic_Offset_table_tmpl.html
b4c8d8677
dashboardv3/public/js/utils/Enums.js ee80313c2
dashboardv3/public/js/views/site/Statistics.js 9eb714d45
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java
96dc5856a
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java
af3727df4
notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java
1fb9f9989
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
1b486e528
repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java
beb90e67b
repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java
bea8eb8b5
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
5643af95c
Diff: https://reviews.apache.org/r/73409/diff/1/
Testing
---
Precommit:
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/624/console
Manually verified lag information is updated correctly on metrics page and
periodically printed on log
Thanks,
Sarath Subramanian
[jira] [Updated] (ATLAS-4330) Add Kafka topics lag information on metrics and log
[
https://issues.apache.org/jira/browse/ATLAS-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4330:
--
Labels: kafka metrics topic-lag (was: )
> Add Kafka topics lag information on metrics and log
> ---
>
> Key: ATLAS-4330
> URL: https://issues.apache.org/jira/browse/ATLAS-4330
> Project: Atlas
> Issue Type: Improvement
> Components: atlas-core
>Affects Versions: 2.1.0
> Reporter: Sarath Subramanian
> Assignee: Sarath Subramanian
>Priority: Major
> Labels: kafka, metrics, topic-lag
> Fix For: 3.0.0, 2.2.0
>
> Attachments: image-2021-06-07-15-03-28-166.png
>
>
> In metrics page, in addition to other Notification details. Lag information
> of kafka consumer topics will be useful to check the current backed up
> messages in Atlas:
> !image-2021-06-07-15-03-28-166.png!
> It will also be useful to print the lag information periodically in atlas
> application.log on a frequency that can be made configurable (print lag every
> 1 hour, 5min or so).
> *atlas.notification.consumer.topic.report.frequency.seconds=60*
> {code:java}
> 2021-06-07 21:56:06,121 INFO - NotificationProcessingStats:
> topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0
> 2021-06-07 21:56:09,105 INFO - NotificationProcessingStats:
> topic=ATLAS_HOOK-0, currentOffset=1118, endOffset=1118, lag=0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4330) Add Kafka topics lag information on metrics and log
[
https://issues.apache.org/jira/browse/ATLAS-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4330:
--
Description:
In metrics page, in addition to other Notification details. Lag information of
kafka consumer topics will be useful to check the current backed up messages in
Atlas:
!image-2021-06-07-15-03-28-166.png!
It will also be useful to print the lag information periodically in atlas
application.log on a frequency that can be made configurable (print lag every 1
hour, 5min or so).
*atlas.notification.consumer.topic.report.frequency.seconds=60*
{code:java}
2021-06-07 21:56:06,121 INFO - NotificationProcessingStats:
topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0
2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: topic=ATLAS_HOOK-0,
currentOffset=1118, endOffset=1118, lag=0
{code}
was:
In metrics page, in addition to other Notification details. Lag information of
kafka consumer topics will be useful to check the current backup up messages in
Atlas:
!image-2021-06-07-15-03-28-166.png!
It will also be useful to print the lag information periodically in atlas
application.log on a frequency that can be made configurable (print lag every 1
hour, 5min or so).
*atlas.notification.consumer.topic.report.frequency.seconds=60*
{code:java}
2021-06-07 21:56:06,121 INFO - NotificationProcessingStats:
topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0
2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: topic=ATLAS_HOOK-0,
currentOffset=1118, endOffset=1118, lag=0
{code}
> Add Kafka topics lag information on metrics and log
> ---
>
> Key: ATLAS-4330
> URL: https://issues.apache.org/jira/browse/ATLAS-4330
> Project: Atlas
> Issue Type: Improvement
> Components: atlas-core
>Affects Versions: 2.1.0
> Reporter: Sarath Subramanian
> Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: image-2021-06-07-15-03-28-166.png
>
>
> In metrics page, in addition to other Notification details. Lag information
> of kafka consumer topics will be useful to check the current backed up
> messages in Atlas:
> !image-2021-06-07-15-03-28-166.png!
> It will also be useful to print the lag information periodically in atlas
> application.log on a frequency that can be made configurable (print lag every
> 1 hour, 5min or so).
> *atlas.notification.consumer.topic.report.frequency.seconds=60*
> {code:java}
> 2021-06-07 21:56:06,121 INFO - NotificationProcessingStats:
> topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0
> 2021-06-07 21:56:09,105 INFO - NotificationProcessingStats:
> topic=ATLAS_HOOK-0, currentOffset=1118, endOffset=1118, lag=0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4330) Add Kafka topics lag information on metrics and log
[
https://issues.apache.org/jira/browse/ATLAS-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4330:
--
Description:
In metrics page, in addition to other Notification details. Lag information of
kafka consumer topics will be useful to check the current backup up messages in
Atlas:
!image-2021-06-07-15-03-28-166.png!
It will also be useful to print the lag information periodically in atlas
application.log on a frequency that can be made configurable (print lag every 1
hour, 5min or so).
*atlas.notification.consumer.topic.report.frequency.seconds=60*
{code:java}
2021-06-07 21:56:06,121 INFO - NotificationProcessingStats:
topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0
2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: topic=ATLAS_HOOK-0,
currentOffset=1118, endOffset=1118, lag=0
{code}
was:
In metrics page, in addition to other Notification details. Lag information of
kafka consumer topics will be useful to check the current backup up messages in
Atlas:
!image-2021-06-07-15-03-28-166.png!
It will also be useful to print the lag information periodically in atlas
application.log on a frequency that can be made configurable (print lag every 1
hour, 5min or so).
*atlas.notification.consumer.topic.report.frequency.seconds=60*
{code:java}
2021-06-07 21:56:06,121 INFO - NotificationProcessingStats:
topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0
2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: topic=ATLAS_HOOK-0,
currentOffset=1118, endOffset=1118, lag=0
{code}
> Add Kafka topics lag information on metrics and log
> ---
>
> Key: ATLAS-4330
> URL: https://issues.apache.org/jira/browse/ATLAS-4330
> Project: Atlas
> Issue Type: Improvement
> Components: atlas-core
>Affects Versions: 2.1.0
> Reporter: Sarath Subramanian
> Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: image-2021-06-07-15-03-28-166.png
>
>
> In metrics page, in addition to other Notification details. Lag information
> of kafka consumer topics will be useful to check the current backup up
> messages in Atlas:
> !image-2021-06-07-15-03-28-166.png!
> It will also be useful to print the lag information periodically in atlas
> application.log on a frequency that can be made configurable (print lag every
> 1 hour, 5min or so).
> *atlas.notification.consumer.topic.report.frequency.seconds=60*
> {code:java}
> 2021-06-07 21:56:06,121 INFO - NotificationProcessingStats:
> topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0
> 2021-06-07 21:56:09,105 INFO - NotificationProcessingStats:
> topic=ATLAS_HOOK-0, currentOffset=1118, endOffset=1118, lag=0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4330) Add Kafka topics lag information on metrics and log
[
https://issues.apache.org/jira/browse/ATLAS-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4330:
--
Description:
In metrics page, in addition to other Notification details. Lag information of
kafka consumer topics will be useful to check the current backup up messages in
Atlas:
!image-2021-06-07-15-03-28-166.png!
It will also be useful to print the lag information periodically in atlas
application.log on a frequency that can be made configurable (print lag every 1
hour, 5min or so).
*atlas.notification.consumer.topic.report.frequency.seconds=60*
{code:java}
2021-06-07 21:56:06,121 INFO - NotificationProcessingStats:
topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0
2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: topic=ATLAS_HOOK-0,
currentOffset=1118, endOffset=1118, lag=0
{code}
was:
In metrics page, in addition to other Notification details. Lag information of
kafka consumer topics will be useful to check the current backup up messages in
Atlas:
!image-2021-06-07-15-03-28-166.png!
> Add Kafka topics lag information on metrics and log
> ---
>
> Key: ATLAS-4330
> URL: https://issues.apache.org/jira/browse/ATLAS-4330
> Project: Atlas
> Issue Type: Improvement
> Components: atlas-core
>Affects Versions: 2.1.0
> Reporter: Sarath Subramanian
> Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: image-2021-06-07-15-03-28-166.png
>
>
> In metrics page, in addition to other Notification details. Lag information
> of kafka consumer topics will be useful to check the current backup up
> messages in Atlas:
> !image-2021-06-07-15-03-28-166.png!
> It will also be useful to print the lag information periodically in atlas
> application.log on a frequency that can be made configurable (print lag every
> 1 hour, 5min or so).
>
> *atlas.notification.consumer.topic.report.frequency.seconds=60*
>
>
> {code:java}
> 2021-06-07 21:56:06,121 INFO - NotificationProcessingStats:
> topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0
> 2021-06-07 21:56:09,105 INFO - NotificationProcessingStats:
> topic=ATLAS_HOOK-0, currentOffset=1118, endOffset=1118, lag=0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ATLAS-4330) Add Kafka topics lag information on metrics and log
Sarath Subramanian created ATLAS-4330: - Summary: Add Kafka topics lag information on metrics and log Key: ATLAS-4330 URL: https://issues.apache.org/jira/browse/ATLAS-4330 Project: Atlas Issue Type: Improvement Components: atlas-core Affects Versions: 2.1.0 Reporter: Sarath Subramanian Assignee: Sarath Subramanian Fix For: 3.0.0, 2.2.0 Attachments: image-2021-06-07-15-03-28-166.png In metrics page, in addition to other Notification details. Lag information of kafka consumer topics will be useful to check the current backup up messages in Atlas: !image-2021-06-07-15-03-28-166.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-4329) Update Kafka version to 2.5
[ https://issues.apache.org/jira/browse/ATLAS-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358752#comment-17358752 ] Sarath Subramanian commented on ATLAS-4329: --- *Precommit:* [https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/620/] *Manual testing:* Build Atlas with embedded HBase and Solr profile and was able to start Atlas and validate basic functionality > Update Kafka version to 2.5 > > > Key: ATLAS-4329 > URL: https://issues.apache.org/jira/browse/ATLAS-4329 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 > Reporter: Sarath Subramanian > Assignee: Sarath Subramanian >Priority: Major > Labels: kafka, pom > Fix For: 3.0.0, 2.2.0 > > Attachments: ATLAS-4329-Update-Kafka-version-to-2.5.patch > > > Atlas uses the following kafka versions for producer and consumer: > +*Current:*+ > * 2.0.0 > * 2.11 > +*New:*+ > * 2.5.0 > * 2.12 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4329) Update Kafka version to 2.5
[ https://issues.apache.org/jira/browse/ATLAS-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4329: -- Attachment: ATLAS-4329-Update-Kafka-version-to-2.5.patch > Update Kafka version to 2.5 > > > Key: ATLAS-4329 > URL: https://issues.apache.org/jira/browse/ATLAS-4329 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 > Reporter: Sarath Subramanian > Assignee: Sarath Subramanian >Priority: Major > Labels: kafka, pom > Fix For: 3.0.0, 2.2.0 > > Attachments: ATLAS-4329-Update-Kafka-version-to-2.5.patch > > > Atlas uses the following kafka versions for producer and consumer: > +*Current:*+ > * 2.0.0 > * 2.11 > +*New:*+ > * 2.5.0 > * 2.12 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4329) Update Kafka version to 2.5
[ https://issues.apache.org/jira/browse/ATLAS-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4329: -- Labels: kafka pom (was: ) > Update Kafka version to 2.5 > > > Key: ATLAS-4329 > URL: https://issues.apache.org/jira/browse/ATLAS-4329 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 > Reporter: Sarath Subramanian > Assignee: Sarath Subramanian >Priority: Major > Labels: kafka, pom > Fix For: 3.0.0, 2.2.0 > > > Atlas uses the following kafka versions for producer and consumer: > +*Current:*+ > * 2.0.0 > * 2.11 > +*New:*+ > * 2.5.0 > * 2.12 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-4329) Update Kafka version to 2.5
Sarath Subramanian created ATLAS-4329: - Summary: Update Kafka version to 2.5 Key: ATLAS-4329 URL: https://issues.apache.org/jira/browse/ATLAS-4329 Project: Atlas Issue Type: Bug Components: atlas-core Affects Versions: 2.1.0 Reporter: Sarath Subramanian Assignee: Sarath Subramanian Fix For: 3.0.0, 2.2.0 Atlas uses the following kafka versions for producer and consumer: +*Current:*+ * 2.0.0 * 2.11 +*New:*+ * 2.5.0 * 2.12 -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 73404: ATLAS-4324: FS entity created for load data inpath is created as shell entity - Part 2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73404/#review223103 --- addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilter.java Lines 62 (patched) <https://reviews.apache.org/r/73404/#comment312234> If config property is set, user should be mindful of including all the 6 path types as well, else shell entities will be created. The types specified from config should only append to the default types. Without defaultTypes all path lineage will be broken. addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java Lines 182 (patched) <https://reviews.apache.org/r/73404/#comment312235> we may not need separate instanceof check for AtlasRelatedObjectId. Lines 188-192 should suffice, since guid belongs to parent - AtlasObjectId. Consider removing 182-187 addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java Line 201 (original), 191 (patched) <https://reviews.apache.org/r/73404/#comment312236> This will also set all path entity types guid to null. If guid is null (non-negative guid is not specified). IdBasedResolver will be ignored and UniqueAttributeBasedResolver will not be able to fetch the entity and result in shell entity creation. The earlier check to ignore path entity types would avoid this. Please review. - Sarath Subramanian On June 3, 2021, 2:52 p.m., Ashutosh Mestry wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73404/ > --- > > (Updated June 3, 2021, 2:52 p.m.) > > > Review request for atlas, Mandar Ambawane, Radhika Kundam, and Sarath > Subramanian. > > > Bugs: ATLAS-4324 > https://issues.apache.org/jira/browse/ATLAS-4324 > > > Repository: atlas > > > Description > --- > > **Approach** > Updates: > - Simplified approach. > - Maintained abstraction. > - Updated unit tests. > > > Diffs > - > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilter.java > 737c637bf > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java > 9163c4709 > > addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilterTest.java > 4dde1dc2f > > addons/hive-bridge/src/test/resources/json/hs2-create-db-with-no-pathentities-to-retain-v2.json > 42553b5cf > > addons/hive-bridge/src/test/resources/json/hs2-create-db-with-no-pathentities-to-retain.json > a5b810f72 > addons/hive-bridge/src/test/resources/json/hs2-load-inpath-v2.json > dd31aa073 > > addons/hive-bridge/src/test/resources/json/hs2-load-inpath-with-no-pathentities-to-retain-v2.json > 7f90d1986 > > addons/hive-bridge/src/test/resources/json/hs2-load-inpath-with-no-pathentities-to-retain.json > dd31aa073 > addons/hive-bridge/src/test/resources/json/hs2-load-inpath.json dd31aa073 > > > Diff: https://reviews.apache.org/r/73404/diff/1/ > > > Testing > --- > > **Unit tests** > - Additional unit test. > > > Thanks, > > Ashutosh Mestry > >
[jira] [Updated] (ATLAS-4324) FS entity created for load data inpath is created as shell entity
[ https://issues.apache.org/jira/browse/ATLAS-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4324: -- Labels: hive-hooks (was: ) > FS entity created for load data inpath is created as shell entity > - > > Key: ATLAS-4324 > URL: https://issues.apache.org/jira/browse/ATLAS-4324 > Project: Atlas > Issue Type: Bug > Components: hive-integration >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Labels: hive-hooks > Fix For: 3.0.0, 2.2.0 > > Attachments: load_data_shell_entity.png > > > beeline > > create external table default.hive_table_cloud_load_data_in_path_123 > (student_roll int, student_name string, student_dob date) ROW FORMAT > DELIMITED FIELDS TERMINATED BY ' ' STORED AS TEXTFILE location > 'hdfs://ns1/tmp/hive_table_cloud_load_data_in_path_123' > > load data inpath 'hdfs://ns1/tmp/data123.txt' into table > hive_table_cloud_load_data_in_path_123; > > Creates 'hdfs://ns1/tmp/data123.txt' as shell entity. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ATLAS-4324) FS entity created for load data inpath is created as shell entity
[ https://issues.apache.org/jira/browse/ATLAS-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian resolved ATLAS-4324. --- Resolution: Fixed > FS entity created for load data inpath is created as shell entity > - > > Key: ATLAS-4324 > URL: https://issues.apache.org/jira/browse/ATLAS-4324 > Project: Atlas > Issue Type: Bug > Components: hive-integration >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Labels: hive-hooks > Fix For: 3.0.0, 2.2.0 > > Attachments: load_data_shell_entity.png > > > beeline > > create external table default.hive_table_cloud_load_data_in_path_123 > (student_roll int, student_name string, student_dob date) ROW FORMAT > DELIMITED FIELDS TERMINATED BY ' ' STORED AS TEXTFILE location > 'hdfs://ns1/tmp/hive_table_cloud_load_data_in_path_123' > > load data inpath 'hdfs://ns1/tmp/data123.txt' into table > hive_table_cloud_load_data_in_path_123; > > Creates 'hdfs://ns1/tmp/data123.txt' as shell entity. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4324) FS entity created for load data inpath is created as shell entity
[ https://issues.apache.org/jira/browse/ATLAS-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4324: -- Component/s: hive-integration > FS entity created for load data inpath is created as shell entity > - > > Key: ATLAS-4324 > URL: https://issues.apache.org/jira/browse/ATLAS-4324 > Project: Atlas > Issue Type: Bug > Components: hive-integration >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: load_data_shell_entity.png > > > beeline > > create external table default.hive_table_cloud_load_data_in_path_123 > (student_roll int, student_name string, student_dob date) ROW FORMAT > DELIMITED FIELDS TERMINATED BY ' ' STORED AS TEXTFILE location > 'hdfs://ns1/tmp/hive_table_cloud_load_data_in_path_123' > > load data inpath 'hdfs://ns1/tmp/data123.txt' into table > hive_table_cloud_load_data_in_path_123; > > Creates 'hdfs://ns1/tmp/data123.txt' as shell entity. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4324) FS entity created for load data inpath is created as shell entity
[ https://issues.apache.org/jira/browse/ATLAS-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4324: -- Fix Version/s: 2.2.0 3.0.0 > FS entity created for load data inpath is created as shell entity > - > > Key: ATLAS-4324 > URL: https://issues.apache.org/jira/browse/ATLAS-4324 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: load_data_shell_entity.png > > > beeline > > create external table default.hive_table_cloud_load_data_in_path_123 > (student_roll int, student_name string, student_dob date) ROW FORMAT > DELIMITED FIELDS TERMINATED BY ' ' STORED AS TEXTFILE location > 'hdfs://ns1/tmp/hive_table_cloud_load_data_in_path_123' > > load data inpath 'hdfs://ns1/tmp/data123.txt' into table > hive_table_cloud_load_data_in_path_123; > > Creates 'hdfs://ns1/tmp/data123.txt' as shell entity. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4324) FS entity created for load data inpath is created as shell entity
[ https://issues.apache.org/jira/browse/ATLAS-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4324: -- Affects Version/s: 2.1.0 > FS entity created for load data inpath is created as shell entity > - > > Key: ATLAS-4324 > URL: https://issues.apache.org/jira/browse/ATLAS-4324 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Attachments: load_data_shell_entity.png > > > beeline > > create external table default.hive_table_cloud_load_data_in_path_123 > (student_roll int, student_name string, student_dob date) ROW FORMAT > DELIMITED FIELDS TERMINATED BY ' ' STORED AS TEXTFILE location > 'hdfs://ns1/tmp/hive_table_cloud_load_data_in_path_123' > > load data inpath 'hdfs://ns1/tmp/data123.txt' into table > hive_table_cloud_load_data_in_path_123; > > Creates 'hdfs://ns1/tmp/data123.txt' as shell entity. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4301) Handle Test Case Failure on Pre-commit environment
[
https://issues.apache.org/jira/browse/ATLAS-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4301:
--
Labels: hive-hooks integration-tests (was: )
> Handle Test Case Failure on Pre-commit environment
> --
>
> Key: ATLAS-4301
> URL: https://issues.apache.org/jira/browse/ATLAS-4301
> Project: Atlas
> Issue Type: Test
> Components: hive-integration
>Affects Versions: 2.1.0
>Reporter: Mandar Ambawane
>Assignee: Mandar Ambawane
>Priority: Major
> Labels: hive-hooks, integration-tests
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-4301.patch
>
>
> Getting following error on Pre-commit build due to change in the file path
> hive-bridge/target/logs/application.log
> {code:java}
> Wrong FS:
> pfile:/home/jenkins/jenkins-agent/workspace/Atlas/PreCommit-ATLAS-Build-Test/addons/hive-bridge/target/partition-data-{dir},
> expected: file:///
> {code}
>
> Also,
> There is Enum "Type" in the "Entity"
> {code:java}
> Class: org.apache.hadoop.hive.ql.hooks.Entity
> Enum: Type{code}
> Enum "Type" has one constant "LOCAL_DIR"
> Due to recent changes, we need to provide support for this constant
> "LOCAL_DIR" in Testing Environment.
> Without which following issues occuring on Testing Environment:
> # While creating "hive_process" entity, The "outputs" attribute is not
> getting set (Which is of type "hdfs_path").
> # While setting the "qualifiedName" of "hive_process" entity, File path is
> not getting appended.
> This causing Failure of some Test cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4301) Handle Test Case Failure on Pre-commit environment
[
https://issues.apache.org/jira/browse/ATLAS-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4301:
--
Component/s: hive-integration
> Handle Test Case Failure on Pre-commit environment
> --
>
> Key: ATLAS-4301
> URL: https://issues.apache.org/jira/browse/ATLAS-4301
> Project: Atlas
> Issue Type: Test
> Components: hive-integration
>Affects Versions: 2.1.0
>Reporter: Mandar Ambawane
>Assignee: Mandar Ambawane
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-4301.patch
>
>
> Getting following error on Pre-commit build due to change in the file path
> hive-bridge/target/logs/application.log
> {code:java}
> Wrong FS:
> pfile:/home/jenkins/jenkins-agent/workspace/Atlas/PreCommit-ATLAS-Build-Test/addons/hive-bridge/target/partition-data-{dir},
> expected: file:///
> {code}
>
> Also,
> There is Enum "Type" in the "Entity"
> {code:java}
> Class: org.apache.hadoop.hive.ql.hooks.Entity
> Enum: Type{code}
> Enum "Type" has one constant "LOCAL_DIR"
> Due to recent changes, we need to provide support for this constant
> "LOCAL_DIR" in Testing Environment.
> Without which following issues occuring on Testing Environment:
> # While creating "hive_process" entity, The "outputs" attribute is not
> getting set (Which is of type "hdfs_path").
> # While setting the "qualifiedName" of "hive_process" entity, File path is
> not getting appended.
> This causing Failure of some Test cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4301) Handle Test Case Failure on Pre-commit environment
[
https://issues.apache.org/jira/browse/ATLAS-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4301:
--
Issue Type: Test (was: Improvement)
> Handle Test Case Failure on Pre-commit environment
> --
>
> Key: ATLAS-4301
> URL: https://issues.apache.org/jira/browse/ATLAS-4301
> Project: Atlas
> Issue Type: Test
>Affects Versions: 2.1.0
>Reporter: Mandar Ambawane
>Assignee: Mandar Ambawane
>Priority: Major
> Attachments: ATLAS-4301.patch
>
>
> Getting following error on Pre-commit build due to change in the file path
> hive-bridge/target/logs/application.log
> {code:java}
> Wrong FS:
> pfile:/home/jenkins/jenkins-agent/workspace/Atlas/PreCommit-ATLAS-Build-Test/addons/hive-bridge/target/partition-data-{dir},
> expected: file:///
> {code}
>
> Also,
> There is Enum "Type" in the "Entity"
> {code:java}
> Class: org.apache.hadoop.hive.ql.hooks.Entity
> Enum: Type{code}
> Enum "Type" has one constant "LOCAL_DIR"
> Due to recent changes, we need to provide support for this constant
> "LOCAL_DIR" in Testing Environment.
> Without which following issues occuring on Testing Environment:
> # While creating "hive_process" entity, The "outputs" attribute is not
> getting set (Which is of type "hdfs_path").
> # While setting the "qualifiedName" of "hive_process" entity, File path is
> not getting appended.
> This causing Failure of some Test cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4301) Handle Test Case Failure on Pre-commit environment
[
https://issues.apache.org/jira/browse/ATLAS-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4301:
--
Affects Version/s: 2.1.0
> Handle Test Case Failure on Pre-commit environment
> --
>
> Key: ATLAS-4301
> URL: https://issues.apache.org/jira/browse/ATLAS-4301
> Project: Atlas
> Issue Type: Improvement
>Affects Versions: 2.1.0
>Reporter: Mandar Ambawane
>Assignee: Mandar Ambawane
>Priority: Major
> Attachments: ATLAS-4301.patch
>
>
> Getting following error on Pre-commit build due to change in the file path
> hive-bridge/target/logs/application.log
> {code:java}
> Wrong FS:
> pfile:/home/jenkins/jenkins-agent/workspace/Atlas/PreCommit-ATLAS-Build-Test/addons/hive-bridge/target/partition-data-{dir},
> expected: file:///
> {code}
>
> Also,
> There is Enum "Type" in the "Entity"
> {code:java}
> Class: org.apache.hadoop.hive.ql.hooks.Entity
> Enum: Type{code}
> Enum "Type" has one constant "LOCAL_DIR"
> Due to recent changes, we need to provide support for this constant
> "LOCAL_DIR" in Testing Environment.
> Without which following issues occuring on Testing Environment:
> # While creating "hive_process" entity, The "outputs" attribute is not
> getting set (Which is of type "hdfs_path").
> # While setting the "qualifiedName" of "hive_process" entity, File path is
> not getting appended.
> This causing Failure of some Test cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4301) Handle Test Case Failure on Pre-commit environment
[
https://issues.apache.org/jira/browse/ATLAS-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4301:
--
Fix Version/s: 2.2.0
3.0.0
> Handle Test Case Failure on Pre-commit environment
> --
>
> Key: ATLAS-4301
> URL: https://issues.apache.org/jira/browse/ATLAS-4301
> Project: Atlas
> Issue Type: Test
>Affects Versions: 2.1.0
>Reporter: Mandar Ambawane
>Assignee: Mandar Ambawane
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-4301.patch
>
>
> Getting following error on Pre-commit build due to change in the file path
> hive-bridge/target/logs/application.log
> {code:java}
> Wrong FS:
> pfile:/home/jenkins/jenkins-agent/workspace/Atlas/PreCommit-ATLAS-Build-Test/addons/hive-bridge/target/partition-data-{dir},
> expected: file:///
> {code}
>
> Also,
> There is Enum "Type" in the "Entity"
> {code:java}
> Class: org.apache.hadoop.hive.ql.hooks.Entity
> Enum: Type{code}
> Enum "Type" has one constant "LOCAL_DIR"
> Due to recent changes, we need to provide support for this constant
> "LOCAL_DIR" in Testing Environment.
> Without which following issues occuring on Testing Environment:
> # While creating "hive_process" entity, The "outputs" attribute is not
> getting set (Which is of type "hdfs_path").
> # While setting the "qualifiedName" of "hive_process" entity, File path is
> not getting appended.
> This causing Failure of some Test cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Review Request 73402: ATLAS-4301 Handle Test Case Failure on Pre-commit environment
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73402/#review223096
---
Ship it!
good work! thanks for fixing the flaky tests :)
- Sarath Subramanian
On June 3, 2021, 7:23 a.m., Mandar Ambawane wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73402/
> ---
>
> (Updated June 3, 2021, 7:23 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues,
> and Sarath Subramanian.
>
>
> Bugs: ATLAS-4301
> https://issues.apache.org/jira/browse/ATLAS-4301
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Getting following error on Pre-commit build due to change in the file path
>
> hive-bridge/target/logs/application.log
>
> Wrong FS:
> pfile:/home/jenkins/jenkins-agent/workspace/Atlas/PreCommit-ATLAS-Build-Test/addons/hive-bridge/target/partition-data-{dir},
> expected: file:///
>
>
> Also,
>
> There is Enum "Type" in the "Entity"
>
> Class: org.apache.hadoop.hive.ql.hooks.Entity
> Enum: Type
> Enum "Type" has one constant "LOCAL_DIR"
>
> Due to recent changes, we need to provide support for this constant
> "LOCAL_DIR" in Testing Environment.
>
> Without which following issues occuring on Testing Environment:
>
> 1. While creating "hive_process" entity, The "outputs" attribute is not
> getting set (Which is of type "hdfs_path").
>
> 2. While setting the "qualifiedName" of "hive_process" entity, File path is
> not getting appended.
>
> This causing Failure of some Test cases.
>
>
> Diffs
> -
>
>
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
> 21c0ae3
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java
> 7a11ec3
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java
> 52684e7
>
>
> Diff: https://reviews.apache.org/r/73402/diff/1/
>
>
> Testing
> ---
>
> PreCommit:
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/612/consoleFull
>
>
> Thanks,
>
> Mandar Ambawane
>
>
Re: Review Request 73376: ATLAS-4307: Atlas bulk clarification REST should be enhanced to associate classification to bulk entities with uniq attribute and type
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73376/#review223054
---
Ship it!
Ship It!
- Sarath Subramanian
On May 26, 2021, 11:34 a.m., Sidharth Mishra wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73376/
> ---
>
> (Updated May 26, 2021, 11:34 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Deep Singh, Radhika Kundam, and
> Sarath Subramanian.
>
>
> Bugs: ATLAS-4307
> https://issues.apache.org/jira/browse/ATLAS-4307
>
>
> Repository: atlas
>
>
> Description
> ---
>
> ATLAS-4307: Atlas bulk clarification REST should be enhanced to associate
> classification to bulk entities with uniq attribute and type
>
>
> Diffs
> -
>
>
> intg/src/main/java/org/apache/atlas/model/instance/ClassificationAssociateRequest.java
> 3f946ef7a
> webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java ec893f25c
>
>
> Diff: https://reviews.apache.org/r/73376/diff/2/
>
>
> Testing
> ---
>
> Manually tested. With new changes the three options will be:
>
> Option 1 (Only with type name and list of uniq attributes)
>
> curl --location --request POST
> 'http://atlas-host:31000/api/atlas/v2/entity/bulk/classification?skipFailedEntities=True'
> \
> --header 'Authorization: Basic YWRtaW46YWRtaW4xMjM=' \
> --header 'Content-Type: application/json' \
> --data-raw '{
> "classification" : {
> "propagate" : false,
> "attributes" : {
> "type" : "false"
> },
> "typeName" : "TEST_TAG_TABLE"
> },
> "entitiesUniqAttributes" : [
> {"qualifiedName": "db1.table43@cm"},
> {"qualifiedName": "db1.table7@cm"},
> {"qualifiedName": "db1.table55@cm"}],
> "entityTypeName" : "hive_table"
> }'
>
> Option 2 (Only with entity Guids)
> curl --location --request POST
> 'http://atlas-host:31000/api/atlas/v2/entity/bulk/classification?skipFailedEntities=True'
> \
> --header 'Authorization: Basic YWRtaW46YWRtaW4xMjM=' \
> --header 'Content-Type: application/json' \
> --data-raw '{
> "classification" : {
> "propagate" : false,
> "attributes" : {
> "type" : "false"
> },
> "typeName" : "TEST_TAG_TABLE"
> },
> "entityGuids" : ["f7288e81-46e4-4c88-b6c7-da092d6b6e35",
> "1793c6df-53a6-481b-a75c-c705f98d6900"],
> "entityTypeName" : "hive_table"
> }'
>
> Option 3 (With both entity GUIDs and type Name with list of uniq attributes)
> curl --location --request POST
> 'http://atlas-host:31000/api/atlas/v2/entity/bulk/classification?skipFailedEntities=True'
> \
> --header 'Authorization: Basic YWRtaW46YWRtaW4xMjM=' \
> --header 'Content-Type: application/json' \
> --data-raw '{
> "classification" : {
> "propagate" : false,
> "attributes" : {
> "type" : "false"
> },
> "typeName" : "TEST_TAG_TABLE"
> },
> "entitiesUniqAttributes" : [
> {"qualifiedName": "db1.table43@cm"},
> {"qualifiedName": "db1.table8@cm"},
> {"qualifiedName": "db1.table55@cm"}],
> "entityGuids" : [
> "6d6ab082-9a62-4aea-97b7-2b7337173631",
> "9a0d3275-eb42-4229-ac51-6db1ddf3ccc6"],
> "entityTypeName" : "hive_table"
> }'
>
>
> Thanks,
>
> Sidharth Mishra
>
>
[jira] [Commented] (ATLAS-4310) NPE seen for CLASSIFICATION_PROPAGATION_DELETE Operation
[
https://issues.apache.org/jira/browse/ATLAS-4310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17351942#comment-17351942
]
Sarath Subramanian commented on ATLAS-4310:
---
+1 thanks for the patch [~amestry]
> NPE seen for CLASSIFICATION_PROPAGATION_DELETE Operation
>
>
> Key: ATLAS-4310
> URL: https://issues.apache.org/jira/browse/ATLAS-4310
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: trunk
>Reporter: Ashutosh Mestry
>Assignee: Ashutosh Mestry
>Priority: Major
> Attachments: ATLAS-4310-Handled-NPE-for-DELETE-classification.patch
>
>
> *Steps to Duplicate*
> # Enabled admin tasks
> # Created an hdfs_path entity
> # In a loop for 330 times: (330 times because to generate 1000 audits)
> ## Updated entity ( updated path)
> ## Added tag1
> ## Removed tag1
> Expected results: Classification is removed.
> Actual results: Classification is removed. Logs indicate NPE:
> {code:java}
> at
> org.apache.atlas.repository.store.graph.v2.EntityGraphMapper.deleteClassificationPropagation(EntityGraphMapper.java:2595)
> at
> org.apache.atlas.repository.store.graph.v2.EntityGraphMapper.deleteClassificationPropagation(EntityGraphMapper.java:2595)
> at
> org.apache.atlas.repository.store.graph.v2.EntityGraphMapper$$FastClassBySpringCGLIB$$8e3f1c72.invoke()
> at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> at
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:737)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
> at
> org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:111)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> at
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:672)
> at
> org.apache.atlas.repository.store.graph.v2.EntityGraphMapper$$EnhancerBySpringCGLIB$$96822c39.deleteClassificationPropagation()
> at
> org.apache.atlas.repository.store.graph.v2.tasks.ClassificationPropagationTasks$Delete.run(ClassificationPropagationTasks.java:73)
> at
> org.apache.atlas.repository.store.graph.v2.tasks.ClassificationTask.perform(ClassificationTask.java:95)
> at org.apache.atlas.tasks.AbstractTask.run(AbstractTask.java:33) at
> org.apache.atlas.tasks.TaskExecutor$TaskConsumer.performTask(TaskExecutor.java:150)
> at
> org.apache.atlas.tasks.TaskExecutor$TaskConsumer.run(TaskExecutor.java:109)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)Caused by:
> java.lang.NullPointerException at
> org.apache.atlas.repository.graph.GraphHelper.getTypeName(GraphHelper.java:867)
> at
> org.apache.atlas.repository.store.graph.v2.EntityGraphRetriever.toAtlasClassification(EntityGraphRetriever.java:334)
> at
> org.apache.atlas.repository.store.graph.v2.EntityGraphMapper.deleteClassificationPropagation(EntityGraphMapper.java:2572)
> ... 18 more2021-05-25 11:07:13,553 ERROR - [atlas-task-0-etp651100072-232 -
> ceaa7213-1d14-4006-8f84-d94e56f4e829:] ~ Task:
> c9f7c463-1c5d-4ae9-8232-506fd2c95a28: Error performing task!
> (ClassificationTask:99)org.apache.atlas.exception.AtlasBaseException:
> java.lang.NullPointerException at
> org.apache.atlas.repository.store.graph.v2.EntityGraphMapper.deleteClassificationPropagation(EntityGraphMapper.java:2595)
> at
> org.apache.atlas.repository.store.graph.v2.EntityGraphMapper$$FastClassBySpringCGLIB$$8e3f1c72.invoke()
> at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> at
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:737)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
> at
> org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:111)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> at
Re: Review Request 73376: ATLAS-4307: Atlas bulk clarification REST should be enhanced to associate classification to bulk entities with uniq attribute and type
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73376/#review223047
---
intg/src/main/java/org/apache/atlas/model/instance/ClassificationAssociateRequest.java
Lines 47 (patched)
<https://reviews.apache.org/r/73376/#comment312202>
entitiesUniqAttributes => entitiesUniqueAttributes
webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java
Lines 813 (patched)
<https://reviews.apache.org/r/73376/#comment312203>
entitiesAttributes => entitiesUniqueAttributes
webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java
Line 814 (original), 828 (patched)
<https://reviews.apache.org/r/73376/#comment312205>
why need of linkedlist here and not array list?
webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java
Lines 856 (patched)
<https://reviews.apache.org/r/73376/#comment312206>
this empty/null check may not be needed, we check this in
entitiesStore.addClassification() method. review and remove.
- Sarath Subramanian
On May 25, 2021, 5:19 p.m., Sidharth Mishra wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73376/
> ---
>
> (Updated May 25, 2021, 5:19 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Deep Singh, Radhika Kundam, and
> Sarath Subramanian.
>
>
> Bugs: ATLAS-4307
> https://issues.apache.org/jira/browse/ATLAS-4307
>
>
> Repository: atlas
>
>
> Description
> ---
>
> ATLAS-4307: Atlas bulk clarification REST should be enhanced to associate
> classification to bulk entities with uniq attribute and type
>
>
> Diffs
> -
>
>
> intg/src/main/java/org/apache/atlas/model/instance/ClassificationAssociateRequest.java
> 3f946ef7a
> webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java ec893f25c
>
>
> Diff: https://reviews.apache.org/r/73376/diff/1/
>
>
> Testing
> ---
>
> Manually tested. With new changes the three options will be:
>
> Option 1 (Only with type name and list of uniq attributes)
>
> curl --location --request POST
> 'http://atlas-host:31000/api/atlas/v2/entity/bulk/classification?skipFailedEntities=True'
> \
> --header 'Authorization: Basic YWRtaW46YWRtaW4xMjM=' \
> --header 'Content-Type: application/json' \
> --data-raw '{
> "classification" : {
> "propagate" : false,
> "attributes" : {
> "type" : "false"
> },
> "typeName" : "TEST_TAG_TABLE"
> },
> "entitiesUniqAttributes" : [
> {"qualifiedName": "db1.table43@cm"},
> {"qualifiedName": "db1.table7@cm"},
> {"qualifiedName": "db1.table55@cm"}],
> "entityTypeName" : "hive_table"
> }'
>
> Option 2 (Only with entity Guids)
> curl --location --request POST
> 'http://atlas-host:31000/api/atlas/v2/entity/bulk/classification?skipFailedEntities=True'
> \
> --header 'Authorization: Basic YWRtaW46YWRtaW4xMjM=' \
> --header 'Content-Type: application/json' \
> --data-raw '{
> "classification" : {
> "propagate" : false,
> "attributes" : {
> "type" : "false"
> },
> "typeName" : "TEST_TAG_TABLE"
> },
> "entityGuids" : ["f7288e81-46e4-4c88-b6c7-da092d6b6e35",
> "1793c6df-53a6-481b-a75c-c705f98d6900"],
> "entityTypeName" : "hive_table"
> }'
>
> Option 3 (With both entity GUIDs and type Name with list of uniq attributes)
> curl --location --request POST
> 'http://atlas-host:31000/api/atlas/v2/entity/bulk/classification?skipFailedEntities=True'
> \
> --header 'Authorization: Basic YWRtaW46YWRtaW4xMjM=' \
> --header 'Content-Type: application/json' \
> --data-raw '{
> "classification" : {
> "propagate" : false,
> "attributes" : {
> "type" : "false"
> },
> "typeName" : "TEST_TAG_TABLE"
> },
> "entitiesUniqAttributes" : [
> {"qualifiedName": "db1.table43@cm"},
> {"qualifiedName": "db1.table8@cm"},
> {"qualifiedName": "db1.table55@cm"}],
> "entityGuids" : [
> "6d6ab082-9a62-4aea-97b7-2b7337173631",
> "9a0d3275-eb42-4229-ac51-6db1ddf3ccc6"],
> "entityTypeName" : "hive_table"
> }'
>
>
> Thanks,
>
> Sidharth Mishra
>
>
Re: Review Request 73292: ATLAS-4254 : Basic Search : Optimize pagination
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73292/#review223040 --- Ship it! Ship It! - Sarath Subramanian On May 13, 2021, 8:43 a.m., Pinal Shah wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73292/ > --- > > (Updated May 13, 2021, 8:43 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, > and Sarath Subramanian. > > > Bugs: ATLAS-4254 > https://issues.apache.org/jira/browse/ATLAS-4254 > > > Repository: atlas > > > Description > --- > > **Current Working of Pagination** > > - When user specifies page information i.e offset and limit in the request, > except some condition, offset is reset to 0 > - Query is then executed from 0 offset and limit > - Filter out the results obtained from query > - Generate result set by skipping the results from 0 to the given offset > > **New Working Structure** > > - New Parameter 'marker' (String) has been introduced in SearchParameters > which will have encrypted string as value. > - The marker should be set as '*' for the start of the page > - The response of above request (AtlasSearchResult) will have 'nextMarker' > field which will have encrypted String which is to be passed as value of > 'marker' in the subsequent request > - if marker is set then offset should not be greater than 0 > > > **Perfomance details** > > Took usecase of ranger tagsync: > - On searching tagged entities starting with offset: 0 and limit: 1000 > returned 1 entities in ~ 21s > - On searching tagged entities starting with marker: * and limit: 1000 > returned 1 entities in ~ 10s > (both the number are taken on the fresh start of the server, so as to avoid > caching affecting the numbers) > - Removed 'atlas.graph.index.search.max-result-set-size=150' property from > atlas-application.properties > > - Around 25% improvement > > - Improvement is seen as the page size increases. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java > e1c550ef7 > intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java > 9d2cd4ffc > > repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java > 647ff9ca0 > > repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java > a3ab6e38c > > repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java > 5dcff3b16 > > repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java > 92152fff5 > > repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java > b37d93a83 > repository/src/main/java/org/apache/atlas/discovery/SearchContext.java > aa49121f6 > repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java > f9832c3b2 > > repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java > 45a815886 > > repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java > 027827af2 > > repository/src/test/java/org/apache/atlas/discovery/ClassificationSearchProcessorTest.java > e1ebbfc70 > webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 4c7b6228d > > > Diff: https://reviews.apache.org/r/73292/diff/5/ > > > Testing > --- > > MAnual testing > unit testcases > > > Thanks, > > Pinal Shah > >
Re: Review Request 73329: Correlating Deleted Entities with Lineage
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73329/#review223021 --- Ship it! Ship It! - Sarath Subramanian On May 20, 2021, 3:18 p.m., Ashutosh Mestry wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73329/ > --- > > (Updated May 20, 2021, 3:18 p.m.) > > > Review request for atlas, Radhika Kundam and Sarath Subramanian. > > > Bugs: ATLAS-4152 > https://issues.apache.org/jira/browse/ATLAS-4152 > > > Repository: atlas > > > Description > --- > > **Background** > As part of ATLAS-4204, HS2 notifications send entity-lineage only (provided > the poperty is enabled). > > When Spooling enabled the order of messages can potentially change. The > notification messages coming from HS2 and HMS may not be in the same order as > when they arrived with direct notification. > > Problem: > Consider the sequence of arriving messages: > > This is the sequence of messages for Entity 1 (C = create, U = update, D = > delete, L?x = Lineage of type 'x') > No problem: C1, U1, L1x, L1y, D1 > Problem: C1, U1, D1, L1x, L1y > > This implementation attempts to handle ths problem mentioned above. If the > above case is not handled, it will end up creating shell entities, since > deleted entities are not looked up as part of entity creation. > > **Approach** > Used bounded stream approach where an incoming stream of messages is bounded > with an indicator that it originates from spool. This helps makes localized > decisions on the incoming stream of messages. > > High-level approach: > - Messages when written to the spool are tagged with a timestamp. > - Deleted entities are maintained in a cache. > - Lineage-only message are checked if they refer to a deleted entity. > - If they refer to deleted entity, they are stitched to the one present in > the cache only if it falls within the threshold. > - Using step-climbing approach for locating right entity to stitch lineage to. > > New: _EntityCorrelationsManager_: Uses message timestamp and cached entity > qualifiedName-GUID map. > Modifed: _NotificationHookConsumer_ Uses the new class. > New: _HiveDDLLineagePreprocess_ Uses entity-correlation to link to deleted > entities. > Modified: _SpoolConfiguration_: Added new configuration to pause message > sending after destination is available: > _atlas.hook.spool.pause.before.send.sec_. > In-memory lookup approach changed to persistent loookup. > > > Diffs > - > > common/src/main/java/org/apache/atlas/repository/Constants.java ffcec9743 > > intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java > 810ba97c9 > notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java > f7d9668ec > notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java > 22bd79fdf > notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java > 3d1b3ccf1 > > notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java > 3264e264c > > notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java > edd8ed931 > > notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java > 2d7d19595 > > notification/src/main/java/org/apache/atlas/notification/spool/Publisher.java > 22242c933 > > notification/src/main/java/org/apache/atlas/notification/spool/SpoolConfiguration.java > a9a3a78cc > notification/src/main/java/org/apache/atlas/notification/spool/Spooler.java > 2cacaaadc > > notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java > d7e4959f7 > > notification/src/test/java/org/apache/atlas/notification/spool/AtlasFileSpoolTest.java > 167efbecc > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > cc727c6ba > > repository/src/main/java/org/apache/atlas/repository/store/graph/EntityCorrelationStore.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java > 0a9470815 > > repository/src/test/java/org/apache/atlas/repository/store/graph/v2/EntityCorrelationStoreTest.java > PRE-CREATION > > webapp/src/main/java/org/apache/atlas/notification/EntityCorrelationManager.java > PRE-CREATION > &
[jira] [Updated] (ATLAS-4285) AtlasTasks: Multiple tag propagation tasks running concurrently, task is complete but propagation is not complete
[ https://issues.apache.org/jira/browse/ATLAS-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4285: -- Component/s: atlas-core > AtlasTasks: Multiple tag propagation tasks running concurrently, task is > complete but propagation is not complete > - > > Key: ATLAS-4285 > URL: https://issues.apache.org/jira/browse/ATLAS-4285 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: > ATLAS-4285-Multiple-propagations-with-intersecting-l.patch > > > Created a 500 level linear lineage . (table1 ---> table2 ---> table3 ---> > .. ---> table500) > Added tag1 to table1 > Added tag2 to table2 > Added tag3 to table3 > 3 tasks are created. > task2 got completed and tag2 is associated only to table2 and not propagated > till table500. > After sometime all tasks are completed , but propagation didn't happen -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4285) AtlasTasks: Multiple tag propagation tasks running concurrently, task is complete but propagation is not complete
[ https://issues.apache.org/jira/browse/ATLAS-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4285: -- Fix Version/s: 2.2.0 3.0.0 > AtlasTasks: Multiple tag propagation tasks running concurrently, task is > complete but propagation is not complete > - > > Key: ATLAS-4285 > URL: https://issues.apache.org/jira/browse/ATLAS-4285 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: > ATLAS-4285-Multiple-propagations-with-intersecting-l.patch > > > Created a 500 level linear lineage . (table1 ---> table2 ---> table3 ---> > .. ---> table500) > Added tag1 to table1 > Added tag2 to table2 > Added tag3 to table3 > 3 tasks are created. > task2 got completed and tag2 is associated only to table2 and not propagated > till table500. > After sometime all tasks are completed , but propagation didn't happen -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4285) AtlasTasks: Multiple tag propagation tasks running concurrently, task is complete but propagation is not complete
[ https://issues.apache.org/jira/browse/ATLAS-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4285: -- Labels: deferred-actions tagpropagation (was: ) > AtlasTasks: Multiple tag propagation tasks running concurrently, task is > complete but propagation is not complete > - > > Key: ATLAS-4285 > URL: https://issues.apache.org/jira/browse/ATLAS-4285 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Labels: deferred-actions, tagpropagation > Fix For: 3.0.0, 2.2.0 > > Attachments: > ATLAS-4285-Multiple-propagations-with-intersecting-l.patch > > > Created a 500 level linear lineage . (table1 ---> table2 ---> table3 ---> > .. ---> table500) > Added tag1 to table1 > Added tag2 to table2 > Added tag3 to table3 > 3 tasks are created. > task2 got completed and tag2 is associated only to table2 and not propagated > till table500. > After sometime all tasks are completed , but propagation didn't happen -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4285) AtlasTasks: Multiple tag propagation tasks running concurrently, task is complete but propagation is not complete
[ https://issues.apache.org/jira/browse/ATLAS-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4285: -- Affects Version/s: 2.1.0 > AtlasTasks: Multiple tag propagation tasks running concurrently, task is > complete but propagation is not complete > - > > Key: ATLAS-4285 > URL: https://issues.apache.org/jira/browse/ATLAS-4285 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Attachments: > ATLAS-4285-Multiple-propagations-with-intersecting-l.patch > > > Created a 500 level linear lineage . (table1 ---> table2 ---> table3 ---> > .. ---> table500) > Added tag1 to table1 > Added tag2 to table2 > Added tag3 to table3 > 3 tasks are created. > task2 got completed and tag2 is associated only to table2 and not propagated > till table500. > After sometime all tasks are completed , but propagation didn't happen -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 73356: Improve Tag Propagation in Entities Involved in Several Lineages
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73356/#review223020 --- Ship it! Ship It! - Sarath Subramanian On May 14, 2021, 9:48 a.m., Ashutosh Mestry wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73356/ > --- > > (Updated May 14, 2021, 9:48 a.m.) > > > Review request for atlas, Jayendra Parab, Nikhil Bonte, and Sarath > Subramanian. > > > Bugs: ATLAS-4285 > https://issues.apache.org/jira/browse/ATLAS-4285 > > > Repository: atlas > > > Description > --- > > **Steps to Duplicate** > _Setup_: > - Create a linear lineage using chained CTAS queries via Hive or REST APIs > E.g. > ``` > create view abcd_vw_1 as select * from abc; > create view abcd_vw_2 as select * from abcd_vw_1; > create view abcd_vw_3 as select * from abcd_vw_2; > > . > . > . > > ``` > This should result in a long lineage. > - Create 3 classifications, say C1, C2, C3. > - Assign C1 to abcd_vw_1, C2 to abcd_vw_2 and C3 to abcd_vw_3. > > Expected result: abcd_vw_n should have all the 3 classifications propagated. > > **Aproach** > Modify _propagateClassification_ to lock all the impacted vertices. Requests > attempting to modify the same entity will thus be blocked. > > **Additional Information** > - Discovered object creation sequence change for _TaskMangement_ and > _TaskFactoryRegistry_. This causes pending tasks not to be scheduled. > - Removed _ClassificationPropagationTasks.Update_ since it is not used. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > 7984a34a4 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagateTaskFactory.java > 8a81dc952 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java > f86cbc722 > repository/src/main/java/org/apache/atlas/tasks/TaskFactoryRegistry.java > 38f2cc9cd > repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java > 2756504da > > repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationWithTasksTest.java > 84aefc9ef > > > Diff: https://reviews.apache.org/r/73356/diff/1/ > > > Testing > --- > > **Manual tests** > Used the scenario described above. See the attached file used to generate > lineage. > > > Thanks, > > Ashutosh Mestry > >
Re: Review Request 73329: Correlating Deleted Entities with Lineage
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73329/#review223017 --- common/src/main/java/org/apache/atlas/repository/Constants.java Lines 138 (patched) <https://reviews.apache.org/r/73329/#comment312171> "deletedMessageTimestamp" => entityDeletedTimestamp DELETED_MESSAGE_TIMESTAMP_PROPERTY_KEY => ENTITY_DELETED_TIMESTAMP_PROPERTY_KEY notification/src/main/java/org/apache/atlas/notification/spool/Publisher.java Lines 216 (patched) <https://reviews.apache.org/r/73329/#comment312175> consider adding a comment - explaining the reason for adding sleep for hs2 messages in spool repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java Lines 377 (patched) <https://reviews.apache.org/r/73329/#comment312172> reorder the garph query conditions for better perf: graph.query().has(STATE_PROPERTY_KEY, Status.DELETED.name()) .has(Constants.DELETED_MESSAGE_TIMESTAMP_PROPERTY_KEY, GREATER_THAN, timestamp) .has(Constants.QUALIFIED_NAME, qualifiedName) .orderBy(Constants.DELETED_MESSAGE_TIMESTAMP_PROPERTY_KEY, ASC); webapp/src/main/java/org/apache/atlas/notification/preprocessor/HiveDbDDLPreprocessor.java Lines 38 (patched) <https://reviews.apache.org/r/73329/#comment312173> dbRelationship => dbObject webapp/src/main/java/org/apache/atlas/notification/preprocessor/HiveTableDDLPreprocessor.java Lines 38 (patched) <https://reviews.apache.org/r/73329/#comment312174> tableObject - Sarath Subramanian On May 19, 2021, 9 p.m., Ashutosh Mestry wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73329/ > --- > > (Updated May 19, 2021, 9 p.m.) > > > Review request for atlas, Radhika Kundam and Sarath Subramanian. > > > Bugs: ATLAS-4152 > https://issues.apache.org/jira/browse/ATLAS-4152 > > > Repository: atlas > > > Description > --- > > **Background** > As part of ATLAS-4204, HS2 notifications send entity-lineage only (provided > the poperty is enabled). > > When Spooling enabled the order of messages can potentially change. The > notification messages coming from HS2 and HMS may not be in the same order as > when they arrived with direct notification. > > Problem: > Consider the sequence of arriving messages: > > This is the sequence of messages for Entity 1 (C = create, U = update, D = > delete, L?x = Lineage of type 'x') > No problem: C1, U1, L1x, L1y, D1 > Problem: C1, U1, D1, L1x, L1y > > This implementation attempts to handle ths problem mentioned above. If the > above case is not handled, it will end up creating shell entities, since > deleted entities are not looked up as part of entity creation. > > **Approach** > Used bounded stream approach where an incoming stream of messages is bounded > with an indicator that it originates from spool. This helps makes localized > decisions on the incoming stream of messages. > > High-level approach: > - Messages when written to the spool are tagged with a timestamp. > - Deleted entities are maintained in a cache. > - Lineage-only message are checked if they refer to a deleted entity. > - If they refer to deleted entity, they are stitched to the one present in > the cache only if it falls within the threshold. > - Using step-climbing approach for locating right entity to stitch lineage to. > > New: _EntityCorrelationsManager_: Uses message timestamp and cached entity > qualifiedName-GUID map. > Modifed: _NotificationHookConsumer_ Uses the new class. > New: _HiveDDLLineagePreprocess_ Uses entity-correlation to link to deleted > entities. > Modified: _SpoolConfiguration_: Added new configuration to pause message > sending after destination is available: > _atlas.hook.spool.pause.before.send.sec_. > In-memory lookup approach changed to persistent loookup. > > > Diffs > - > > common/src/main/java/org/apache/atlas/repository/Constants.java ffcec9743 > > intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java > 810ba97c9 > notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java > f7d9668ec > notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java > 22bd79fdf > notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java > 3d1b3ccf1 > > notification/src/main/java/org/apache/atlas
[jira] [Updated] (ATLAS-4288) [Atlas: Glossary Term Bulk Import] Will all the data populated, while performing bulk import, PreferredToTerms relationship alone is not created
[
https://issues.apache.org/jira/browse/ATLAS-4288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4288:
--
Labels: bulk-glossary-import (was: )
> [Atlas: Glossary Term Bulk Import] Will all the data populated, while
> performing bulk import, PreferredToTerms relationship alone is not created
>
>
> Key: ATLAS-4288
> URL: https://issues.apache.org/jira/browse/ATLAS-4288
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 2.1.0
>Reporter: Dharshana M Krishnamoorthy
>Assignee: Sidharth Kumar Mishra
>Priority: Major
> Labels: bulk-glossary-import
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-4288.patch, image-2021-05-17-16-43-31-487.png
>
>
> Consider the following input, here all the relations are established except
> the preferredToTerms (term_2)
> {code:java}
> GlossaryName, TermName, ShortDescription, LongDescription, Examples,
> Abbreviation, Usage, AdditionalAttributes, TranslationTerms, ValidValuesFor,
> Synonyms, ReplacedBy, ValidValues, ReplacementTerms, SeeAlso,
> TranslatedTerms, IsA, Antonyms, Classifies, PreferredToTerms, PreferredTerms
> dharshmk_11,term_1,"short desc","long description", "Example", "G1", "Usage",
> "glossary:100%","glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2"
> dharshmk_11,term_2,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%""glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2",
> dharshmk_11,term_3,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%",,,"glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2",,
> dharshmk_11,term_4,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%",,"glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2",,,
> dharshmk_11,term_5,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%","glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2"
> dharshmk_11,term_6,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%""glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2",
> dharshmk_11,term_7,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%",,,"glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2",,
> dharshmk_11,term_8,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%",,"glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2",,,
> dharshmk_11,term_9,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%","glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2"
> dharshmk_11,term_10,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%""glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2",
> dharshmk_11,term_11,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%",,,"glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2",,
> dharshmk_11,term_12,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%",,"glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2",,,
> dharshmk_11,term_13,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%&
[jira] [Updated] (ATLAS-4288) [Atlas: Glossary Term Bulk Import] Will all the data populated, while performing bulk import, PreferredToTerms relationship alone is not created
[
https://issues.apache.org/jira/browse/ATLAS-4288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4288:
--
Affects Version/s: 2.1.0
> [Atlas: Glossary Term Bulk Import] Will all the data populated, while
> performing bulk import, PreferredToTerms relationship alone is not created
>
>
> Key: ATLAS-4288
> URL: https://issues.apache.org/jira/browse/ATLAS-4288
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 2.1.0
>Reporter: Dharshana M Krishnamoorthy
>Assignee: Sidharth Kumar Mishra
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-4288.patch, image-2021-05-17-16-43-31-487.png
>
>
> Consider the following input, here all the relations are established except
> the preferredToTerms (term_2)
> {code:java}
> GlossaryName, TermName, ShortDescription, LongDescription, Examples,
> Abbreviation, Usage, AdditionalAttributes, TranslationTerms, ValidValuesFor,
> Synonyms, ReplacedBy, ValidValues, ReplacementTerms, SeeAlso,
> TranslatedTerms, IsA, Antonyms, Classifies, PreferredToTerms, PreferredTerms
> dharshmk_11,term_1,"short desc","long description", "Example", "G1", "Usage",
> "glossary:100%","glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2"
> dharshmk_11,term_2,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%""glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2",
> dharshmk_11,term_3,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%",,,"glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2",,
> dharshmk_11,term_4,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%",,"glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2",,,
> dharshmk_11,term_5,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%","glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2"
> dharshmk_11,term_6,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%""glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2",
> dharshmk_11,term_7,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%",,,"glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2",,
> dharshmk_11,term_8,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%",,"glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2",,,
> dharshmk_11,term_9,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%","glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2"
> dharshmk_11,term_10,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%""glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2",
> dharshmk_11,term_11,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%",,,"glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2",,
> dharshmk_11,term_12,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%",,"glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2",,,
> dharshmk_11,term_13,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%","glossaryBulkImport_1:termBulkImport_1|glossaryBulkImpo
[jira] [Updated] (ATLAS-4288) [Atlas: Glossary Term Bulk Import] Will all the data populated, while performing bulk import, PreferredToTerms relationship alone is not created
[
https://issues.apache.org/jira/browse/ATLAS-4288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4288:
--
Fix Version/s: 2.2.0
3.0.0
> [Atlas: Glossary Term Bulk Import] Will all the data populated, while
> performing bulk import, PreferredToTerms relationship alone is not created
>
>
> Key: ATLAS-4288
> URL: https://issues.apache.org/jira/browse/ATLAS-4288
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Dharshana M Krishnamoorthy
>Assignee: Sidharth Kumar Mishra
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-4288.patch, image-2021-05-17-16-43-31-487.png
>
>
> Consider the following input, here all the relations are established except
> the preferredToTerms (term_2)
> {code:java}
> GlossaryName, TermName, ShortDescription, LongDescription, Examples,
> Abbreviation, Usage, AdditionalAttributes, TranslationTerms, ValidValuesFor,
> Synonyms, ReplacedBy, ValidValues, ReplacementTerms, SeeAlso,
> TranslatedTerms, IsA, Antonyms, Classifies, PreferredToTerms, PreferredTerms
> dharshmk_11,term_1,"short desc","long description", "Example", "G1", "Usage",
> "glossary:100%","glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2"
> dharshmk_11,term_2,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%""glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2",
> dharshmk_11,term_3,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%",,,"glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2",,
> dharshmk_11,term_4,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%",,"glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2",,,
> dharshmk_11,term_5,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%","glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2"
> dharshmk_11,term_6,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%""glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2",
> dharshmk_11,term_7,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%",,,"glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2",,
> dharshmk_11,term_8,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%",,"glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2",,,
> dharshmk_11,term_9,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%","glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2"
> dharshmk_11,term_10,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%""glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2",
> dharshmk_11,term_11,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%",,,"glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2",,
> dharshmk_11,term_12,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%",,"glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_2:termBulkImport_2",,,
> dharshmk_11,term_13,"short desc","long description", "Example", "G1",
> "Usage",
> "glossary:100%","glossaryBulkImport_1:termBulkImport_1|glossaryBulkImport_
Re: Review Request 73329: Correlating Deleted Entities with Lineage
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73329/#review223005 --- webapp/src/main/java/org/apache/atlas/notification/EntityCorrelationManager.java Lines 93 (patched) <https://reviews.apache.org/r/73329/#comment312160> this checks for the first entry that is less than the spooledTimestamp. We should be fetching the entry(ts) having close proximity to spooledTimestamp. CACHE: --- QName | Guid | --- T1@cl1 | [7:00: guid1], [7:40: guid2] --- 6:50 - CTAS (T5) FROM T1 (guid1) 7:20 - CTAS (T6) FROM T1 (guid2) - Sarath Subramanian On May 17, 2021, 10:39 p.m., Ashutosh Mestry wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73329/ > --- > > (Updated May 17, 2021, 10:39 p.m.) > > > Review request for atlas, Radhika Kundam and Sarath Subramanian. > > > Bugs: ATLAS-4152 > https://issues.apache.org/jira/browse/ATLAS-4152 > > > Repository: atlas > > > Description > --- > > **Background** > As part of ATLAS-4204, HS2 notifications send entity-lineage only (provided > the poperty is enabled). > > When Spooling enabled the order of messages can potentially change. The > notification messages coming from HS2 and HMS may not be in the same order as > when they arrived with direct notification. > > Problem: > Consider the sequence of arriving messages: > > This is the sequence of messages for Entity 1 (C = create, U = update, D = > delete, L?x = Lineage of type 'x') > No problem: C1, U1, L1x, L1y, D1 > Problem: C1, U1, D1, L1x, L1y > > This implementation attempts to handle ths problem mentioned above. If the > above case is not handled, it will end up creating shell entities, since > deleted entities are not looked up as part of entity creation. > > **Approach** > Used bounded stream approach where an incoming stream of messages is bounded > with an indicator that it originates from spool. This helps makes localized > decisions on the incoming stream of messages. > > High-level approach: > - Messages when written to the spool are tagged with a timestamp. > - Deleted entities are maintained in a cache. > - Lineage-only message are checked if they refer to a deleted entity. > - If they refer to deleted entity, they are stitched to the one present in > the cache only if it falls within the threshold. > - Using step-climbing approach for locating right entity to stitch lineage to. > > New: _EntityCorrelationsManager_: Uses message timestamp and cached entity > qualifiedName-GUID map. > Modifed: _NotificationHookConsumer_ Uses the new class. > New: _HiveDDLLineagePreprocess_ Uses entity-correlation to link to deleted > entities. > Modified: _SpoolConfiguration_: Added new configuration to pause message > sending after destination is available: > _atlas.hook.spool.pause.before.send.sec_. > > > Diffs > - > > > intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java > 810ba97c9 > notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144 > notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java > f7d9668ec > notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java > 22bd79fdf > notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java > 3d1b3ccf1 > > notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java > 3264e264c > > notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java > edd8ed931 > > notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java > 2d7d19595 > > notification/src/main/java/org/apache/atlas/notification/spool/Publisher.java > 22242c933 > > notification/src/main/java/org/apache/atlas/notification/spool/SpoolConfiguration.java > a9a3a78cc > notification/src/main/java/org/apache/atlas/notification/spool/Spooler.java > 2cacaaadc > > notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java > d7e4959f7 > > notification/src/test/java/org/apache/atlas/notification/spool/AtlasFileSpoolTest.java > 167efbecc > > webapp/src/main/java/org/apache/atlas/notification/Ent
Re: Review Request 73358: ATLAS-4288: [Atlas: Glossary Term Bulk Import] Will all the data populated, while performing bulk import, PreferredToTerms relationship alone is not created
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73358/#review223004
---
Fix it, then Ship it!
repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
Lines 729 (patched)
<https://reviews.apache.org/r/73358/#comment312159>
"Invalid relationship specified for Term. Term cannot have a relationship
with self"
- Sarath Subramanian
On May 17, 2021, 10:45 p.m., Sidharth Mishra wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73358/
> ---
>
> (Updated May 17, 2021, 10:45 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Deep Singh, Radhika Kundam, and
> Sarath Subramanian.
>
>
> Bugs: ATLAS-4288
> https://issues.apache.org/jira/browse/ATLAS-4288
>
>
> Repository: atlas
>
>
> Description
> ---
>
> ATLAS-4288: Added code to create PreferredToTerms relationships when it
> exists alone
>
>
> Diffs
> -
>
> intg/src/main/java/org/apache/atlas/model/glossary/AtlasGlossaryTerm.java
> ee306c6b0
> repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
> fa02c8eb9
>
>
> Diff: https://reviews.apache.org/r/73358/diff/1/
>
>
> Testing
> ---
>
> Manually tested. Changed code to take care of preferredToTerms relation when
> it exists without any other relations. Below it has also self-reference so
> now its throwing error for self reference and then creating the remaining
> relations successfully. Check below error messages and response json.
>
>
> Response json:
>
> {
> "failedImportInfoList": [
>
> { "parentObjectName": "dharshmk_11", "childObjectName": "term_1",
> "importStatus": "FAILED", "remarks": "Invalid Term relationship: Term cant
> have a relationship with self" }
> ,
>
> { "parentObjectName": "dharshmk_11", "childObjectName": "term_2",
> "importStatus": "FAILED", "remarks": "Invalid Term relationship: Term cant
> have a relationship with self" }
> ,
>
> { "parentObjectName": "dharshmk_11", "childObjectName": "term_3",
> "importStatus": "FAILED", "remarks": "Invalid Term relationship: Term cant
> have a relationship with self" }
> ,
>
> { "parentObjectName": "dharshmk_11", "childObjectName": "term_4",
> "importStatus": "FAILED", "remarks": "Invalid Term relationship: Term cant
> have a relationship with self" }
> ,
>
> { "parentObjectName": "dharshmk_11", "childObjectName": "term_5",
> "importStatus": "FAILED", "remarks": "Invalid Term relationship: Term cant
> have a relationship with self" }
> ,
>
> { "parentObjectName": "dharshmk_11", "childObjectName": "term_6",
> "importStatus": "FAILED", "remarks": "Invalid Term relationship: Term cant
> have a relationship with self" }
> ,
>
> { "parentObjectName": "dharshmk_11", "childObjectName": "term_7",
> "importStatus": "FAILED", "remarks": "Invalid Term relationship: Term cant
> have a relationship with self" }
> ,
>
> { "parentObjectName": "dharshmk_11", "childObjectName": "term_8",
> "importStatus": "FAILED", "remarks": "Invalid Term relationship: Term cant
> have a relationship with self" }
> ,
>
> { "parentObjectName": "dharshmk_11", "childObjectName": "term_9",
> "importStatus": "FAILED", "remarks": "Invalid Term relationship: Term cant
> have a relationship with self" }
> ,
>
> { "parentObjectName": "dharshmk_11", "childObjectName": "term_10",
> "importStatus": "FAILED", "remarks": "Invalid Term relationship: Term cant
> have a relationship with self" }
> ,
>
> { "parentObjectName": "dharshmk_11", "childObjectName": "term_11",
&g
Re: Review Request 73329: Correlating Deleted Entities with Lineage
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73329/#review223002
---
webapp/src/main/java/org/apache/atlas/notification/EntityCorrelationManager.java
Lines 76 (patched)
<https://reviews.apache.org/r/73329/#comment312157>
we don't need to reverse iterate the list, since we are iterating the
entire list:
consider maintaining a minValue - the timestamp in cache should be the
closest to the spooled timestamp: consider using the below method for better
readability
```
public String getGuidForDeletedEntity(String qualifiedName, long
spooledMsgTimestamp) {
if (!this.entitiesDeletedByDelete.containsKey(qualifiedName) ||
spooledMsgTimestamp <= 0) {
return null;
}
String ret= null;
List> timestampGuidPairs =
this.entitiesDeletedByDelete.get(qualifiedName);
long minTimestamp =
Long.MAX_VALUE;
for (TypesUtil.Pair tsGuidPair : timestampGuidPairs) {
String entityGuid= tsGuidPair.right;
long entityDeleteTimestamp = tsGuidPair.left;
long timestampDifference = Math.abs(entityDeleteTimestamp -
spooledMsgTimestamp);
if (timestampDifference < minTimestamp) {
minTimestamp = timestampDifference;
ret = entityGuid;
}
}
return ret;
}
```
- Sarath Subramanian
On May 10, 2021, 10:22 p.m., Ashutosh Mestry wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73329/
> ---
>
> (Updated May 10, 2021, 10:22 p.m.)
>
>
> Review request for atlas, Radhika Kundam and Sarath Subramanian.
>
>
> Bugs: ATLAS-4152
> https://issues.apache.org/jira/browse/ATLAS-4152
>
>
> Repository: atlas
>
>
> Description
> ---
>
> **Background**
> As part of ATLAS-4204, HS2 notifications send entity-lineage only (provided
> the poperty is enabled).
>
> When Spooling enabled the order of messages can potentially change. The
> notification messages coming from HS2 and HMS may not be in the same order as
> when they arrived with direct notification.
>
> Problem:
> Consider the sequence of arriving messages:
>
> This is the sequence of messages for Entity 1 (C = create, U = update, D =
> delete, L?x = Lineage of type 'x')
> No problem: C1, U1, L1x, L1y, D1
> Problem: C1, U1, D1, L1x, L1y
>
> This implementation attempts to handle ths problem mentioned above. If the
> above case is not handled, it will end up creating shell entities, since
> deleted entities are not looked up as part of entity creation.
>
> **Approach**
> Used bounded stream approach where an incoming stream of messages is bounded
> with an indicator that it originates from spool. This helps makes localized
> decisions on the incoming stream of messages.
>
> High-level approach:
> - Messages when written to the spool are tagged with a timestamp.
> - Deleted entities are maintained in a cache.
> - Lineage-only message are checked if they refer to a deleted entity.
> - If they refer to deleted entity, they are stitched to the one present in
> the cache only if it falls within the threshold.
> - Using step-climbing approach for locating right entity to stitch lineage to.
>
> New: _EntityCorrelationsManager_: Uses message timestamp and cached entity
> qualifiedName-GUID map.
> Modifed: _NotificationHookConsumer_ Uses the new class.
> New: _HiveDDLLineagePreprocess_ Uses entity-correlation to link to deleted
> entities.
> Modified: _SpoolConfiguration_: Added new configuration to pause message
> sending after destination is available:
> _atlas.hook.spool.pause.before.send.sec_.
>
>
> Diffs
> -
>
>
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java
> 810ba97c9
> notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 9162ac144
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java
> f7d9668ec
> notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java
> 22bd79fdf
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java
> 3d1b3ccf1
>
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java
>
[jira] [Updated] (ATLAS-4274) [Atlas: Glossary] Non matching relation are created via bulk import
[
https://issues.apache.org/jira/browse/ATLAS-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4274:
--
Fix Version/s: 2.2.0
3.0.0
> [Atlas: Glossary] Non matching relation are created via bulk import
> ---
>
> Key: ATLAS-4274
> URL: https://issues.apache.org/jira/browse/ATLAS-4274
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 2.1.0
>Reporter: Dharshana M Krishnamoorthy
>Assignee: Sidharth Kumar Mishra
>Priority: Major
> Labels: bulk-glossary-import, glossary
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-4274.patch, Screenshot 2021-05-04 at 3.31.00
> PM.png, Screenshot 2021-05-04 at 3.34.03 PM.png, Screenshot 2021-05-04 at
> 3.34.21 PM.png, Screenshot 2021-05-04 at 3.34.36 PM.png
>
>
> The related terms provided in the input does not match the relation created
> via import
> {code:java}
> GlossaryName, TermName, ShortDescription, LongDescription, Examples,
> Abbreviation, Usage, AdditionalAttributes, TranslationTerms, ValidValuesFor,
> Synonyms, ReplacedBy, ValidValues, ReplacementTerms, SeeAlso,
> TranslatedTerms, IsA, Antonyms, Classifies, PreferredToTerms, PreferredTerms
> a_glossary_1,term_1,,,"a_glossary_1:term_2"
> a_glossary_1,term_2,"a_glossary_1:term_3",,
> a_glossary_1,term_3,,"a_glossary_1:term_1", {code}
> !Screenshot 2021-05-04 at 3.31.00 PM.png|width=1973,height=127!
> !Screenshot 2021-05-04 at 3.34.03 PM.png|width=1038,height=578!
> !Screenshot 2021-05-04 at 3.34.21 PM.png|width=1005,height=563!
> !Screenshot 2021-05-04 at 3.34.36 PM.png|width=541,height=303!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4274) [Atlas: Glossary] Non matching relation are created via bulk import
[
https://issues.apache.org/jira/browse/ATLAS-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4274:
--
Labels: bulk-glossary-import glossary (was: )
> [Atlas: Glossary] Non matching relation are created via bulk import
> ---
>
> Key: ATLAS-4274
> URL: https://issues.apache.org/jira/browse/ATLAS-4274
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 2.1.0
>Reporter: Dharshana M Krishnamoorthy
>Assignee: Sidharth Kumar Mishra
>Priority: Major
> Labels: bulk-glossary-import, glossary
> Attachments: ATLAS-4274.patch, Screenshot 2021-05-04 at 3.31.00
> PM.png, Screenshot 2021-05-04 at 3.34.03 PM.png, Screenshot 2021-05-04 at
> 3.34.21 PM.png, Screenshot 2021-05-04 at 3.34.36 PM.png
>
>
> The related terms provided in the input does not match the relation created
> via import
> {code:java}
> GlossaryName, TermName, ShortDescription, LongDescription, Examples,
> Abbreviation, Usage, AdditionalAttributes, TranslationTerms, ValidValuesFor,
> Synonyms, ReplacedBy, ValidValues, ReplacementTerms, SeeAlso,
> TranslatedTerms, IsA, Antonyms, Classifies, PreferredToTerms, PreferredTerms
> a_glossary_1,term_1,,,"a_glossary_1:term_2"
> a_glossary_1,term_2,"a_glossary_1:term_3",,
> a_glossary_1,term_3,,"a_glossary_1:term_1", {code}
> !Screenshot 2021-05-04 at 3.31.00 PM.png|width=1973,height=127!
> !Screenshot 2021-05-04 at 3.34.03 PM.png|width=1038,height=578!
> !Screenshot 2021-05-04 at 3.34.21 PM.png|width=1005,height=563!
> !Screenshot 2021-05-04 at 3.34.36 PM.png|width=541,height=303!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4274) [Atlas: Glossary] Non matching relation are created via bulk import
[
https://issues.apache.org/jira/browse/ATLAS-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4274:
--
Component/s: atlas-core
> [Atlas: Glossary] Non matching relation are created via bulk import
> ---
>
> Key: ATLAS-4274
> URL: https://issues.apache.org/jira/browse/ATLAS-4274
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 2.1.0
>Reporter: Dharshana M Krishnamoorthy
>Assignee: Sidharth Kumar Mishra
>Priority: Major
> Attachments: ATLAS-4274.patch, Screenshot 2021-05-04 at 3.31.00
> PM.png, Screenshot 2021-05-04 at 3.34.03 PM.png, Screenshot 2021-05-04 at
> 3.34.21 PM.png, Screenshot 2021-05-04 at 3.34.36 PM.png
>
>
> The related terms provided in the input does not match the relation created
> via import
> {code:java}
> GlossaryName, TermName, ShortDescription, LongDescription, Examples,
> Abbreviation, Usage, AdditionalAttributes, TranslationTerms, ValidValuesFor,
> Synonyms, ReplacedBy, ValidValues, ReplacementTerms, SeeAlso,
> TranslatedTerms, IsA, Antonyms, Classifies, PreferredToTerms, PreferredTerms
> a_glossary_1,term_1,,,"a_glossary_1:term_2"
> a_glossary_1,term_2,"a_glossary_1:term_3",,
> a_glossary_1,term_3,,"a_glossary_1:term_1", {code}
> !Screenshot 2021-05-04 at 3.31.00 PM.png|width=1973,height=127!
> !Screenshot 2021-05-04 at 3.34.03 PM.png|width=1038,height=578!
> !Screenshot 2021-05-04 at 3.34.21 PM.png|width=1005,height=563!
> !Screenshot 2021-05-04 at 3.34.36 PM.png|width=541,height=303!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4274) [Atlas: Glossary] Non matching relation are created via bulk import
[
https://issues.apache.org/jira/browse/ATLAS-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4274:
--
Affects Version/s: 2.1.0
> [Atlas: Glossary] Non matching relation are created via bulk import
> ---
>
> Key: ATLAS-4274
> URL: https://issues.apache.org/jira/browse/ATLAS-4274
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Dharshana M Krishnamoorthy
>Assignee: Sidharth Kumar Mishra
>Priority: Major
> Attachments: ATLAS-4274.patch, Screenshot 2021-05-04 at 3.31.00
> PM.png, Screenshot 2021-05-04 at 3.34.03 PM.png, Screenshot 2021-05-04 at
> 3.34.21 PM.png, Screenshot 2021-05-04 at 3.34.36 PM.png
>
>
> The related terms provided in the input does not match the relation created
> via import
> {code:java}
> GlossaryName, TermName, ShortDescription, LongDescription, Examples,
> Abbreviation, Usage, AdditionalAttributes, TranslationTerms, ValidValuesFor,
> Synonyms, ReplacedBy, ValidValues, ReplacementTerms, SeeAlso,
> TranslatedTerms, IsA, Antonyms, Classifies, PreferredToTerms, PreferredTerms
> a_glossary_1,term_1,,,"a_glossary_1:term_2"
> a_glossary_1,term_2,"a_glossary_1:term_3",,
> a_glossary_1,term_3,,"a_glossary_1:term_1", {code}
> !Screenshot 2021-05-04 at 3.31.00 PM.png|width=1973,height=127!
> !Screenshot 2021-05-04 at 3.34.03 PM.png|width=1038,height=578!
> !Screenshot 2021-05-04 at 3.34.21 PM.png|width=1005,height=563!
> !Screenshot 2021-05-04 at 3.34.36 PM.png|width=541,height=303!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Review Request 73340: ATLAS-4274: Non matching relation are created via bulk import
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73340/#review222984 --- Ship it! Ship It! - Sarath Subramanian On May 12, 2021, 1 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73340/ > --- > > (Updated May 12, 2021, 1 p.m.) > > > Review request for atlas, Ashutosh Mestry and Sarath Subramanian. > > > Bugs: ATLAS-4274 > https://issues.apache.org/jira/browse/ATLAS-4274 > > > Repository: atlas > > > Description > --- > > ATLAS-4274: Added code to load some terms from graph DB if needed during > glossary import > > > Diffs > - > > repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java > 73217ded3 > repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java > 553d3d0a6 > > > Diff: https://reviews.apache.org/r/73340/diff/3/ > > > Testing > --- > > Manually tested and confirmed its working as expected. > org.apache.atlas.glossary.GlossaryServiceTest UTs are passing. > > > Thanks, > > Sidharth Mishra > >
[jira] [Updated] (ATLAS-4284) Pruned tables are ignored
[ https://issues.apache.org/jira/browse/ATLAS-4284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4284: -- Affects Version/s: 2.1.0 > Pruned tables are ignored > - > > Key: ATLAS-4284 > URL: https://issues.apache.org/jira/browse/ATLAS-4284 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > > Settings in Atlas Server: > atlas.notification.consumer.preprocess.hive_table.prune.pattern=db2.* > No config changes for prune/update made in hive. > With the above settings , created db2.table1. The table is ignored instead of > getting pruned. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4284) Pruned tables are ignored
[ https://issues.apache.org/jira/browse/ATLAS-4284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4284: -- Labels: hive-hooks (was: ) > Pruned tables are ignored > - > > Key: ATLAS-4284 > URL: https://issues.apache.org/jira/browse/ATLAS-4284 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Labels: hive-hooks > Fix For: 3.0.0, 2.2.0 > > > Settings in Atlas Server: > atlas.notification.consumer.preprocess.hive_table.prune.pattern=db2.* > No config changes for prune/update made in hive. > With the above settings , created db2.table1. The table is ignored instead of > getting pruned. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4284) Pruned tables are ignored
[ https://issues.apache.org/jira/browse/ATLAS-4284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4284: -- Fix Version/s: 2.2.0 3.0.0 > Pruned tables are ignored > - > > Key: ATLAS-4284 > URL: https://issues.apache.org/jira/browse/ATLAS-4284 > Project: Atlas > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Fix For: 3.0.0, 2.2.0 > > > Settings in Atlas Server: > atlas.notification.consumer.preprocess.hive_table.prune.pattern=db2.* > No config changes for prune/update made in hive. > With the above settings , created db2.table1. The table is ignored instead of > getting pruned. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4284) Pruned tables are ignored
[ https://issues.apache.org/jira/browse/ATLAS-4284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4284: -- Component/s: atlas-core > Pruned tables are ignored > - > > Key: ATLAS-4284 > URL: https://issues.apache.org/jira/browse/ATLAS-4284 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Fix For: 3.0.0, 2.2.0 > > > Settings in Atlas Server: > atlas.notification.consumer.preprocess.hive_table.prune.pattern=db2.* > No config changes for prune/update made in hive. > With the above settings , created db2.table1. The table is ignored instead of > getting pruned. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4164) [Atlas: Spooling] Tables created after spooling are created before the spooled tables when there is multiple frequent restart in kafka brokers
[
https://issues.apache.org/jira/browse/ATLAS-4164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4164:
--
Labels: spool (was: )
> [Atlas: Spooling] Tables created after spooling are created before the
> spooled tables when there is multiple frequent restart in kafka brokers
> --
>
> Key: ATLAS-4164
> URL: https://issues.apache.org/jira/browse/ATLAS-4164
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 2.1.0
>Reporter: Dharshana M Krishnamoorthy
>Assignee: Ashutosh Mestry
>Priority: Major
> Labels: spool
> Fix For: 3.0.0
>
> Attachments: ATLAS-4164-Spooling-Status.patch
>
>
> Scenario:
> * Stop kafka broker
> * Create a few (20) tables save the prefix (abc_table_1, abc_table_2, ...
> abc_table_n)
> * Make sure the data is spooled
> * Start kafka and create a few more tables (xyz_table_1, xyz_table_2, ...
> xyz_table_n)
> * Wait for 5 mins for the tables to reflect in atlas
> In this case we expect all the abc_table_* to be created before xyz_table_1,
> meaning all the spooled tables are created before the tables that are created
> after spooling.
>
> Observation:
> createTime of some spooled tables is greater than the create time of the
> xyz_table_1
>
> Sample data:
> createTime for tables that are spooled:
> {code:java}
> [1613573518284, 1613573531470, 1613573531861, 1613573529446, 1613573543253,
> 1613573525390, 1613573525950, 1613573517796, 1613573518284, 1613573522629,
> 1613573513524, 1613573524856, 1613573518992, 1613573519477, 1613573519947,
> 1613573521737, 1613573514066, 1613573514555, 1613573515065,
> 1613573515605]{code}
> createTime for tables that are created after spooling:
> {code:java}
> [1613573540582, 1613573541300, 1613573551691, 1613573552628, 1613573553356,
> 1613573555478, 1613573556275, 1613573556940, 1613573557763, 1613573558659,
> 1613573560673, 1613573561363, 1613573562310, 1613573563096, 1613573564004,
> 1613573566533, 1613573567602, 1613573568439, 1613573569379, 1613573570202]
> {code}
> We expect all spooled tables to have createTime smaller than the tables
> created after spooling.
> But *1613573543253 (Spooled tabled create time) is greater than 1613573540582
> (table created after spooling)*
> which means, the table created after spooling is created before spooled table
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (ATLAS-4164) [Atlas: Spooling] Tables created after spooling are created before the spooled tables when there is multiple frequent restart in kafka brokers
[
https://issues.apache.org/jira/browse/ATLAS-4164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian reassigned ATLAS-4164:
-
Assignee: Radhika Kundam (was: Ashutosh Mestry)
> [Atlas: Spooling] Tables created after spooling are created before the
> spooled tables when there is multiple frequent restart in kafka brokers
> --
>
> Key: ATLAS-4164
> URL: https://issues.apache.org/jira/browse/ATLAS-4164
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 2.1.0
>Reporter: Dharshana M Krishnamoorthy
>Assignee: Radhika Kundam
>Priority: Major
> Labels: spool
> Fix For: 3.0.0
>
> Attachments: ATLAS-4164-Spooling-Status.patch
>
>
> Scenario:
> * Stop kafka broker
> * Create a few (20) tables save the prefix (abc_table_1, abc_table_2, ...
> abc_table_n)
> * Make sure the data is spooled
> * Start kafka and create a few more tables (xyz_table_1, xyz_table_2, ...
> xyz_table_n)
> * Wait for 5 mins for the tables to reflect in atlas
> In this case we expect all the abc_table_* to be created before xyz_table_1,
> meaning all the spooled tables are created before the tables that are created
> after spooling.
>
> Observation:
> createTime of some spooled tables is greater than the create time of the
> xyz_table_1
>
> Sample data:
> createTime for tables that are spooled:
> {code:java}
> [1613573518284, 1613573531470, 1613573531861, 1613573529446, 1613573543253,
> 1613573525390, 1613573525950, 1613573517796, 1613573518284, 1613573522629,
> 1613573513524, 1613573524856, 1613573518992, 1613573519477, 1613573519947,
> 1613573521737, 1613573514066, 1613573514555, 1613573515065,
> 1613573515605]{code}
> createTime for tables that are created after spooling:
> {code:java}
> [1613573540582, 1613573541300, 1613573551691, 1613573552628, 1613573553356,
> 1613573555478, 1613573556275, 1613573556940, 1613573557763, 1613573558659,
> 1613573560673, 1613573561363, 1613573562310, 1613573563096, 1613573564004,
> 1613573566533, 1613573567602, 1613573568439, 1613573569379, 1613573570202]
> {code}
> We expect all spooled tables to have createTime smaller than the tables
> created after spooling.
> But *1613573543253 (Spooled tabled create time) is greater than 1613573540582
> (table created after spooling)*
> which means, the table created after spooling is created before spooled table
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4164) [Atlas: Spooling] Tables created after spooling are created before the spooled tables when there is multiple frequent restart in kafka brokers
[
https://issues.apache.org/jira/browse/ATLAS-4164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4164:
--
Affects Version/s: (was: 3.0.0)
> [Atlas: Spooling] Tables created after spooling are created before the
> spooled tables when there is multiple frequent restart in kafka brokers
> --
>
> Key: ATLAS-4164
> URL: https://issues.apache.org/jira/browse/ATLAS-4164
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 2.1.0
>Reporter: Dharshana M Krishnamoorthy
>Assignee: Ashutosh Mestry
>Priority: Major
> Attachments: ATLAS-4164-Spooling-Status.patch
>
>
> Scenario:
> * Stop kafka broker
> * Create a few (20) tables save the prefix (abc_table_1, abc_table_2, ...
> abc_table_n)
> * Make sure the data is spooled
> * Start kafka and create a few more tables (xyz_table_1, xyz_table_2, ...
> xyz_table_n)
> * Wait for 5 mins for the tables to reflect in atlas
> In this case we expect all the abc_table_* to be created before xyz_table_1,
> meaning all the spooled tables are created before the tables that are created
> after spooling.
>
> Observation:
> createTime of some spooled tables is greater than the create time of the
> xyz_table_1
>
> Sample data:
> createTime for tables that are spooled:
> {code:java}
> [1613573518284, 1613573531470, 1613573531861, 1613573529446, 1613573543253,
> 1613573525390, 1613573525950, 1613573517796, 1613573518284, 1613573522629,
> 1613573513524, 1613573524856, 1613573518992, 1613573519477, 1613573519947,
> 1613573521737, 1613573514066, 1613573514555, 1613573515065,
> 1613573515605]{code}
> createTime for tables that are created after spooling:
> {code:java}
> [1613573540582, 1613573541300, 1613573551691, 1613573552628, 1613573553356,
> 1613573555478, 1613573556275, 1613573556940, 1613573557763, 1613573558659,
> 1613573560673, 1613573561363, 1613573562310, 1613573563096, 1613573564004,
> 1613573566533, 1613573567602, 1613573568439, 1613573569379, 1613573570202]
> {code}
> We expect all spooled tables to have createTime smaller than the tables
> created after spooling.
> But *1613573543253 (Spooled tabled create time) is greater than 1613573540582
> (table created after spooling)*
> which means, the table created after spooling is created before spooled table
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4164) [Atlas: Spooling] Tables created after spooling are created before the spooled tables when there is multiple frequent restart in kafka brokers
[
https://issues.apache.org/jira/browse/ATLAS-4164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4164:
--
Affects Version/s: 3.0.0
2.1.0
> [Atlas: Spooling] Tables created after spooling are created before the
> spooled tables when there is multiple frequent restart in kafka brokers
> --
>
> Key: ATLAS-4164
> URL: https://issues.apache.org/jira/browse/ATLAS-4164
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 2.1.0, 3.0.0
>Reporter: Dharshana M Krishnamoorthy
>Assignee: Ashutosh Mestry
>Priority: Major
> Attachments: ATLAS-4164-Spooling-Status.patch
>
>
> Scenario:
> * Stop kafka broker
> * Create a few (20) tables save the prefix (abc_table_1, abc_table_2, ...
> abc_table_n)
> * Make sure the data is spooled
> * Start kafka and create a few more tables (xyz_table_1, xyz_table_2, ...
> xyz_table_n)
> * Wait for 5 mins for the tables to reflect in atlas
> In this case we expect all the abc_table_* to be created before xyz_table_1,
> meaning all the spooled tables are created before the tables that are created
> after spooling.
>
> Observation:
> createTime of some spooled tables is greater than the create time of the
> xyz_table_1
>
> Sample data:
> createTime for tables that are spooled:
> {code:java}
> [1613573518284, 1613573531470, 1613573531861, 1613573529446, 1613573543253,
> 1613573525390, 1613573525950, 1613573517796, 1613573518284, 1613573522629,
> 1613573513524, 1613573524856, 1613573518992, 1613573519477, 1613573519947,
> 1613573521737, 1613573514066, 1613573514555, 1613573515065,
> 1613573515605]{code}
> createTime for tables that are created after spooling:
> {code:java}
> [1613573540582, 1613573541300, 1613573551691, 1613573552628, 1613573553356,
> 1613573555478, 1613573556275, 1613573556940, 1613573557763, 1613573558659,
> 1613573560673, 1613573561363, 1613573562310, 1613573563096, 1613573564004,
> 1613573566533, 1613573567602, 1613573568439, 1613573569379, 1613573570202]
> {code}
> We expect all spooled tables to have createTime smaller than the tables
> created after spooling.
> But *1613573543253 (Spooled tabled create time) is greater than 1613573540582
> (table created after spooling)*
> which means, the table created after spooling is created before spooled table
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4164) [Atlas: Spooling] Tables created after spooling are created before the spooled tables when there is multiple frequent restart in kafka brokers
[
https://issues.apache.org/jira/browse/ATLAS-4164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4164:
--
Fix Version/s: 3.0.0
> [Atlas: Spooling] Tables created after spooling are created before the
> spooled tables when there is multiple frequent restart in kafka brokers
> --
>
> Key: ATLAS-4164
> URL: https://issues.apache.org/jira/browse/ATLAS-4164
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 2.1.0
>Reporter: Dharshana M Krishnamoorthy
>Assignee: Ashutosh Mestry
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: ATLAS-4164-Spooling-Status.patch
>
>
> Scenario:
> * Stop kafka broker
> * Create a few (20) tables save the prefix (abc_table_1, abc_table_2, ...
> abc_table_n)
> * Make sure the data is spooled
> * Start kafka and create a few more tables (xyz_table_1, xyz_table_2, ...
> xyz_table_n)
> * Wait for 5 mins for the tables to reflect in atlas
> In this case we expect all the abc_table_* to be created before xyz_table_1,
> meaning all the spooled tables are created before the tables that are created
> after spooling.
>
> Observation:
> createTime of some spooled tables is greater than the create time of the
> xyz_table_1
>
> Sample data:
> createTime for tables that are spooled:
> {code:java}
> [1613573518284, 1613573531470, 1613573531861, 1613573529446, 1613573543253,
> 1613573525390, 1613573525950, 1613573517796, 1613573518284, 1613573522629,
> 1613573513524, 1613573524856, 1613573518992, 1613573519477, 1613573519947,
> 1613573521737, 1613573514066, 1613573514555, 1613573515065,
> 1613573515605]{code}
> createTime for tables that are created after spooling:
> {code:java}
> [1613573540582, 1613573541300, 1613573551691, 1613573552628, 1613573553356,
> 1613573555478, 1613573556275, 1613573556940, 1613573557763, 1613573558659,
> 1613573560673, 1613573561363, 1613573562310, 1613573563096, 1613573564004,
> 1613573566533, 1613573567602, 1613573568439, 1613573569379, 1613573570202]
> {code}
> We expect all spooled tables to have createTime smaller than the tables
> created after spooling.
> But *1613573543253 (Spooled tabled create time) is greater than 1613573540582
> (table created after spooling)*
> which means, the table created after spooling is created before spooled table
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4183) web.xml requires external dtd resource
[ https://issues.apache.org/jira/browse/ATLAS-4183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4183: -- Fix Version/s: 2.2.0 3.0.0 > web.xml requires external dtd resource > -- > > Key: ATLAS-4183 > URL: https://issues.apache.org/jira/browse/ATLAS-4183 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Deep Singh >Assignee: Deep Singh >Priority: Major > Fix For: 3.0.0, 2.2.0 > > > We have Servlet 2.3 deployment descriptor (web.xml), which follows J2EE 1.3 > DTDs schema. > Therefore it requires dtd resource hosted at > http://java.sun.com/dtd/web-app_2_3.dtd > In setups that are behind firewalls with limited access to external > resources, the Atlas server fails to come up without the external dtd > resource. > This can be fixed by hosting dtd resource locally or upgrading the deployment > descriptor. > We can upgrade it to Servlet 2.5 deployment descriptor which is Java EE 5 XML > schema and it does not require external dtd resource for validation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4183) web.xml requires external dtd resource
[ https://issues.apache.org/jira/browse/ATLAS-4183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4183: -- Affects Version/s: 3.0.0 2.1.0 > web.xml requires external dtd resource > -- > > Key: ATLAS-4183 > URL: https://issues.apache.org/jira/browse/ATLAS-4183 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0, 3.0.0 >Reporter: Deep Singh >Assignee: Deep Singh >Priority: Major > > We have Servlet 2.3 deployment descriptor (web.xml), which follows J2EE 1.3 > DTDs schema. > Therefore it requires dtd resource hosted at > http://java.sun.com/dtd/web-app_2_3.dtd > In setups that are behind firewalls with limited access to external > resources, the Atlas server fails to come up without the external dtd > resource. > This can be fixed by hosting dtd resource locally or upgrading the deployment > descriptor. > We can upgrade it to Servlet 2.5 deployment descriptor which is Java EE 5 XML > schema and it does not require external dtd resource for validation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4183) web.xml requires external dtd resource
[ https://issues.apache.org/jira/browse/ATLAS-4183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4183: -- Affects Version/s: (was: 3.0.0) > web.xml requires external dtd resource > -- > > Key: ATLAS-4183 > URL: https://issues.apache.org/jira/browse/ATLAS-4183 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Deep Singh >Assignee: Deep Singh >Priority: Major > > We have Servlet 2.3 deployment descriptor (web.xml), which follows J2EE 1.3 > DTDs schema. > Therefore it requires dtd resource hosted at > http://java.sun.com/dtd/web-app_2_3.dtd > In setups that are behind firewalls with limited access to external > resources, the Atlas server fails to come up without the external dtd > resource. > This can be fixed by hosting dtd resource locally or upgrading the deployment > descriptor. > We can upgrade it to Servlet 2.5 deployment descriptor which is Java EE 5 XML > schema and it does not require external dtd resource for validation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 73317: ATLAS-3812: Add schema for Apache Flink
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73317/#review222974 --- addons/models/1000-Hadoop/1110-flink_model.json Lines 2 (patched) <https://reviews.apache.org/r/73317/#comment312146> remove empty entity types - enumDefs, structDefs, classificationDefs, relationshipDefs addons/models/1000-Hadoop/1110-flink_model.json Lines 47 (patched) <https://reviews.apache.org/r/73317/#comment312145> parent supertype already contains inputs/outputs as relationship attribute. This will possibly conflict with those attribute. Consider renaming inputs/outputs - Sarath Subramanian On April 30, 2021, 12:39 a.m., Josh Yeh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73317/ > --- > > (Updated April 30, 2021, 12:39 a.m.) > > > Review request for atlas, Ashutosh Mestry and Sarath Subramanian. > > > Repository: atlas > > > Description > --- > > ATLAS-3812: Add schema for Apache Flink > > > Diffs > - > > addons/models/1000-Hadoop/1110-flink_model.json PRE-CREATION > > > Diff: https://reviews.apache.org/r/73317/diff/1/ > > > Testing > --- > > Tested on my CDH cluster with flink example wordcount streaming job. > > > File Attachments > > > Screen Shot 2021-04-30 at 12.38.27 AM.png > > https://reviews.apache.org/media/uploaded/files/2021/04/30/2ff59445-7237-4be1-a1a0-b6db70bc7827__Screen_Shot_2021-04-30_at_12.38.27_AM.png > > > Thanks, > > Josh Yeh > >
Re: Review Request 73340: ATLAS-4274: Non matching relation are created via bulk import
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73340/#review222973 --- repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java Lines 1164 (patched) <https://reviews.apache.org/r/73340/#comment312141> updateGlossaryTermIfNeeded => updateGlossaryTermRelations repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java Line 69 (original), 69 (patched) <https://reviews.apache.org/r/73340/#comment312142> glossaryNameGuidCacheForImport => glossaryNameGuidCache glossaryTermOrderDuringImport => glossaryTermOrderCache repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java Lines 71 (patched) <https://reviews.apache.org/r/73340/#comment312143> glossaryTermNeedLoadForRelationsUpdate => glossaryTermQNameGuidCache repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java Lines 665 (patched) <https://reviews.apache.org/r/73340/#comment312139> glossaryTermToBeUpdated => updatedGlossaryTerm repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java Lines 668 (patched) <https://reviews.apache.org/r/73340/#comment312140> glossaryTermAtGraphDB => glossaryTermFromDB repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java Lines 842 (patched) <https://reviews.apache.org/r/73340/#comment312144> probably longest method name seen in Atlas code :) consider renaming to: cacheRelatedTermQNameGuid() - Sarath Subramanian On May 10, 2021, 12:52 p.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73340/ > --- > > (Updated May 10, 2021, 12:52 p.m.) > > > Review request for atlas, Ashutosh Mestry and Sarath Subramanian. > > > Bugs: ATLAS-4274 > https://issues.apache.org/jira/browse/ATLAS-4274 > > > Repository: atlas > > > Description > --- > > ATLAS-4274: Added code to load some terms from graph DB if needed during > glossary import > > > Diffs > - > > repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java > 73217ded3 > repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java > 553d3d0a6 > > > Diff: https://reviews.apache.org/r/73340/diff/2/ > > > Testing > --- > > Manually tested and confirmed its working as expected. > org.apache.atlas.glossary.GlossaryServiceTest UTs are passing. > > > Thanks, > > Sidharth Mishra > >
[jira] [Updated] (ATLAS-4278) Deferred Actions : Deleting a tag after disassociating it from a table which propagated fails
[ https://issues.apache.org/jira/browse/ATLAS-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4278: -- Fix Version/s: 2.2.0 3.0.0 > Deferred Actions : Deleting a tag after disassociating it from a table which > propagated fails > - > > Key: ATLAS-4278 > URL: https://issues.apache.org/jira/browse/ATLAS-4278 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Jayendra Parab >Assignee: Jayendra Parab >Priority: Major > Labels: deferred-actions > Fix For: 3.0.0, 2.2.0 > > > # Create a lineage : table1 ---> process1 > table2 > # Add tag tag1 to table1. > # Wait for task to complete, it propagates to table2 > # Disassociate tag from table1 > # Wait for task to complete , it is removed from all tables and process. > # Attempt to delete the tag. > # Tag deletion fails with `tag1` has references , though the tag is not > associated to any entity. > If the tag was added to entity without propagate flag set to False , and then > disassociated and deleted , it is deleted successfully. > This issue is seen only with deferred actions enabled. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4278) Deferred Actions : Deleting a tag after disassociating it from a table which propagated fails
[ https://issues.apache.org/jira/browse/ATLAS-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4278: -- Component/s: atlas-core > Deferred Actions : Deleting a tag after disassociating it from a table which > propagated fails > - > > Key: ATLAS-4278 > URL: https://issues.apache.org/jira/browse/ATLAS-4278 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Jayendra Parab >Assignee: Jayendra Parab >Priority: Major > > # Create a lineage : table1 ---> process1 > table2 > # Add tag tag1 to table1. > # Wait for task to complete, it propagates to table2 > # Disassociate tag from table1 > # Wait for task to complete , it is removed from all tables and process. > # Attempt to delete the tag. > # Tag deletion fails with `tag1` has references , though the tag is not > associated to any entity. > If the tag was added to entity without propagate flag set to False , and then > disassociated and deleted , it is deleted successfully. > This issue is seen only with deferred actions enabled. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4278) Deferred Actions : Deleting a tag after disassociating it from a table which propagated fails
[ https://issues.apache.org/jira/browse/ATLAS-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4278: -- Labels: deferred-actions (was: ) > Deferred Actions : Deleting a tag after disassociating it from a table which > propagated fails > - > > Key: ATLAS-4278 > URL: https://issues.apache.org/jira/browse/ATLAS-4278 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Jayendra Parab >Assignee: Jayendra Parab >Priority: Major > Labels: deferred-actions > > # Create a lineage : table1 ---> process1 > table2 > # Add tag tag1 to table1. > # Wait for task to complete, it propagates to table2 > # Disassociate tag from table1 > # Wait for task to complete , it is removed from all tables and process. > # Attempt to delete the tag. > # Tag deletion fails with `tag1` has references , though the tag is not > associated to any entity. > If the tag was added to entity without propagate flag set to False , and then > disassociated and deleted , it is deleted successfully. > This issue is seen only with deferred actions enabled. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 73341: ATLAS-4278 - Deferred Actions : Deleting a tag after disassociating it from a table which propagated fails
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73341/#review222972 --- Ship it! Ship It! - Sarath Subramanian On May 9, 2021, 6:09 a.m., Jayendra Parab wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73341/ > --- > > (Updated May 9, 2021, 6:09 a.m.) > > > Review request for atlas, Ashutosh Mestry, Nikhil Bonte, Nixon Rodrigues, > Pinal Shah, and Sarath Subramanian. > > > Bugs: ATLAS-4278 > https://issues.apache.org/jira/browse/ATLAS-4278 > > > Repository: atlas > > > Description > --- > > When deferred action is enabled, even after dissociating the tag from all > entities the classification vertex created while associating the tag still > exists. > The error given "tag1 has references" is because of the classification vertex > created is still present in graphdb. > > When deferred action is disable, at the time of dissociating the tag, the > classification vertex is deleted from DeleteHandlerV1.deleteEdgeReference. > But when the deferred action is enabled, the classification vertex isn't > deleted because when DeleteHandlerV1.deleteEdgeReference is invoked, that > instance the classification vertex has references to the propagated entities. > In the background thread, which executes the dissociating of tags from > propagated entities, the classification vertex isn't deleted. > To resolve this issue, the classification vertex is deleted once the tag is > disassoociated from propagated entities. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > 02ba5ef8d > > > Diff: https://reviews.apache.org/r/73341/diff/1/ > > > Testing > --- > > With deferred action enabled: > Test 1: > - Associated a tag to an entity table1 which has lineage like table1 -> > process1 -> table2 > - Waited for tag propagation to complete > - Once tag is propagation is complete, deleted the tag1 from table1 > - Once the tag is removed from all the entities in the lineage above, deleted > the tag > - The tag is deleted properly > > Test 2: > - Associated a tag to an entity table1 which has lineage like table1 -> > process1 -> table2 > - Waited for tag propagation to complete > - Once tag is propagation is complete, deleted the tag1 from table1 > - Associated the same tag to an entity table3 which has lineage like table3 > -> process1 -> table4 > - Once the tag is removed from all the entities in the lineage above, deleted > the tag > - The tag is deleted properly > > > Thanks, > > Jayendra Parab > >
Re: Review Request 73340: ATLAS-4274: Non matching relation are created via bulk import
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73340/#review222958 --- intg/src/main/java/org/apache/atlas/model/glossary/AtlasGlossaryTerm.java Lines 428 (patched) <https://reviews.apache.org/r/73340/#comment312131> avoid adding copy logic here to model class (AtlasGlossaryTerm) consider moving to GlossaryTermUtils - Sarath Subramanian On May 8, 2021, 9:58 a.m., Sidharth Mishra wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73340/ > --- > > (Updated May 8, 2021, 9:58 a.m.) > > > Review request for atlas, Ashutosh Mestry and Sarath Subramanian. > > > Bugs: ATLAS-4274 > https://issues.apache.org/jira/browse/ATLAS-4274 > > > Repository: atlas > > > Description > --- > > ATLAS-4274: Added code to load some terms from graph DB if needed during > glossary import > > > Diffs > - > > intg/src/main/java/org/apache/atlas/model/glossary/AtlasGlossaryTerm.java > ee306c6b0 > repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java > 73217ded3 > repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java > 553d3d0a6 > > > Diff: https://reviews.apache.org/r/73340/diff/1/ > > > Testing > --- > > Manually tested and confirmed its working as expected. > org.apache.atlas.glossary.GlossaryServiceTest UTs are passing. > > > Thanks, > > Sidharth Mishra > >
[jira] [Resolved] (ATLAS-4106) Adding Debug metrics to atlas.
[ https://issues.apache.org/jira/browse/ATLAS-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian resolved ATLAS-4106. --- Resolution: Fixed > Adding Debug metrics to atlas. > -- > > Key: ATLAS-4106 > URL: https://issues.apache.org/jira/browse/ATLAS-4106 > Project: Atlas > Issue Type: New Feature > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Mayank Jain >Assignee: Mayank Jain >Priority: Major > Fix For: 3.0.0, 2.2.0 > > > This is a new feature added to atlas to create a ease for keeping track of > certain highly used api's in atlas , so at the time of evaluating the > performance of certain API we could have our data ready. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4106) Adding Debug metrics to atlas.
[ https://issues.apache.org/jira/browse/ATLAS-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4106: -- Component/s: atlas-core > Adding Debug metrics to atlas. > -- > > Key: ATLAS-4106 > URL: https://issues.apache.org/jira/browse/ATLAS-4106 > Project: Atlas > Issue Type: New Feature > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Mayank Jain >Assignee: Mayank Jain >Priority: Major > Fix For: 3.0.0, 2.2.0 > > > This is a new feature added to atlas to create a ease for keeping track of > certain highly used api's in atlas , so at the time of evaluating the > performance of certain API we could have our data ready. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4106) Adding Debug metrics to atlas.
[ https://issues.apache.org/jira/browse/ATLAS-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4106: -- Affects Version/s: 2.1.0 > Adding Debug metrics to atlas. > -- > > Key: ATLAS-4106 > URL: https://issues.apache.org/jira/browse/ATLAS-4106 > Project: Atlas > Issue Type: New Feature >Affects Versions: 2.1.0 >Reporter: Mayank Jain >Assignee: Mayank Jain >Priority: Major > > This is a new feature added to atlas to create a ease for keeping track of > certain highly used api's in atlas , so at the time of evaluating the > performance of certain API we could have our data ready. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4106) Adding Debug metrics to atlas.
[ https://issues.apache.org/jira/browse/ATLAS-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4106: -- Fix Version/s: 2.2.0 3.0.0 > Adding Debug metrics to atlas. > -- > > Key: ATLAS-4106 > URL: https://issues.apache.org/jira/browse/ATLAS-4106 > Project: Atlas > Issue Type: New Feature >Affects Versions: 2.1.0 >Reporter: Mayank Jain >Assignee: Mayank Jain >Priority: Major > Fix For: 3.0.0, 2.2.0 > > > This is a new feature added to atlas to create a ease for keeping track of > certain highly used api's in atlas , so at the time of evaluating the > performance of certain API we could have our data ready. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 73316: ATLAS-4269: Deferred Actions : When a tag is propagated from an entity via 2 processes , blocking 1 process removes tag propagated from another process
> On April 29, 2021, 6:16 p.m., Ashutosh Mestry wrote: > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java > > Lines 602 (patched) > > <https://reviews.apache.org/r/73316/diff/1/?file=2249611#file2249611line602> > > > > listContainsVertex -> findByVertexId, pass vertexId as parameter, that > > will address earlier comment. updated to use map - Sarath --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73316/#review222917 --- On April 29, 2021, 9:01 p.m., Sarath Subramanian wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73316/ > --- > > (Updated April 29, 2021, 9:01 p.m.) > > > Review request for atlas, Ashutosh Mestry, Deep Singh, Madhan Neethiraj, > Radhika Kundam, and Sidharth Mishra. > > > Bugs: ATLAS-4269 > https://issues.apache.org/jira/browse/ATLAS-4269 > > > Repository: atlas > > > Description > --- > > Issue > = > 1.Create a lineage as attached in the screenshot (see JIRA) tag propagated > from a source using 2 processes to a target > > 2. Add a tag to t3 , now the tag is propagated to t4 via both process1 and > process2. > > 3. Now block the propagation from t3 to process1. > > 4. Expectation is that , tag will be propagated via process2 to t4. But the > propagated tag is removed from t4 > > > Solution: > = > > Recently tag propagation logic was updated to use queue implementation. When > adding impacted vertices to the result added a check to see if vertex already > present. So no duplicates present. > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java > 9abcf646c > > > Diff: https://reviews.apache.org/r/73316/diff/2/ > > > Testing > --- > > Precommit: > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/532/console > > Manually validated in circular lineage > > > Thanks, > > Sarath Subramanian > >
Re: Review Request 73316: ATLAS-4269: Deferred Actions : When a tag is propagated from an entity via 2 processes , blocking 1 process removes tag propagated from another process
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73316/ --- (Updated April 29, 2021, 9:01 p.m.) Review request for atlas, Ashutosh Mestry, Deep Singh, Madhan Neethiraj, Radhika Kundam, and Sidharth Mishra. Changes --- updated implementation to use map to avoid search on collection on every result addition. Bugs: ATLAS-4269 https://issues.apache.org/jira/browse/ATLAS-4269 Repository: atlas Description --- Issue = 1.Create a lineage as attached in the screenshot (see JIRA) tag propagated from a source using 2 processes to a target 2. Add a tag to t3 , now the tag is propagated to t4 via both process1 and process2. 3. Now block the propagation from t3 to process1. 4. Expectation is that , tag will be propagated via process2 to t4. But the propagated tag is removed from t4 Solution: = Recently tag propagation logic was updated to use queue implementation. When adding impacted vertices to the result added a check to see if vertex already present. So no duplicates present. Diffs (updated) - repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 9abcf646c Diff: https://reviews.apache.org/r/73316/diff/2/ Changes: https://reviews.apache.org/r/73316/diff/1-2/ Testing --- Precommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/532/console Manually validated in circular lineage Thanks, Sarath Subramanian
[jira] [Updated] (ATLAS-4269) Deferred Actions : When a tag is propagated from an entity via 2 processes , blocking 1 process removes tag propagated from another process
[ https://issues.apache.org/jira/browse/ATLAS-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4269: -- Attachment: default...@cm.png > Deferred Actions : When a tag is propagated from an entity via 2 processes , > blocking 1 process removes tag propagated from another process > --- > > Key: ATLAS-4269 > URL: https://issues.apache.org/jira/browse/ATLAS-4269 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 > Reporter: Sarath Subramanian >Assignee: Sarath Subramanian >Priority: Major > Attachments: default...@cm.png > > > 1.Create a lineage as attached in the screenshot (i.e) tag propagated from a > source using 2 processes to a target > 2. Add a tag to t3 , now the tag is propagated to t4 via both process1 and > process2. > 3. Now block the propagation from t3 to process1. > 4. Expectation is that , tag will be propagated via process2 to t4. But the > propagated tag is removed from t4 > !default...@cm.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4269) Deferred Actions : When a tag is propagated from an entity via 2 processes , blocking 1 process removes tag propagated from another process
[ https://issues.apache.org/jira/browse/ATLAS-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4269: -- Labels: tagpropagation (was: ) > Deferred Actions : When a tag is propagated from an entity via 2 processes , > blocking 1 process removes tag propagated from another process > --- > > Key: ATLAS-4269 > URL: https://issues.apache.org/jira/browse/ATLAS-4269 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 > Reporter: Sarath Subramanian >Assignee: Sarath Subramanian >Priority: Major > Labels: tagpropagation > Fix For: 3.0.0, 2.2.0 > > Attachments: default...@cm.png > > > 1.Create a lineage as attached in the screenshot (i.e) tag propagated from a > source using 2 processes to a target > 2. Add a tag to t3 , now the tag is propagated to t4 via both process1 and > process2. > 3. Now block the propagation from t3 to process1. > 4. Expectation is that , tag will be propagated via process2 to t4. But the > propagated tag is removed from t4 > !default...@cm.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-4269) Deferred Actions : When a tag is propagated from an entity via 2 processes , blocking 1 process removes tag propagated from another process
Sarath Subramanian created ATLAS-4269: - Summary: Deferred Actions : When a tag is propagated from an entity via 2 processes , blocking 1 process removes tag propagated from another process Key: ATLAS-4269 URL: https://issues.apache.org/jira/browse/ATLAS-4269 Project: Atlas Issue Type: Bug Components: atlas-core Affects Versions: 2.1.0 Reporter: Sarath Subramanian Assignee: Sarath Subramanian Attachments: default...@cm.png 1.Create a lineage as attached in the screenshot (i.e) tag propagated from a source using 2 processes to a target 2. Add a tag to t3 , now the tag is propagated to t4 via both process1 and process2. 3. Now block the propagation from t3 to process1. 4. Expectation is that , tag will be propagated via process2 to t4. But the propagated tag is removed from t4 !default...@cm.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4269) Deferred Actions : When a tag is propagated from an entity via 2 processes , blocking 1 process removes tag propagated from another process
[ https://issues.apache.org/jira/browse/ATLAS-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4269: -- Fix Version/s: 2.2.0 3.0.0 > Deferred Actions : When a tag is propagated from an entity via 2 processes , > blocking 1 process removes tag propagated from another process > --- > > Key: ATLAS-4269 > URL: https://issues.apache.org/jira/browse/ATLAS-4269 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.1.0 > Reporter: Sarath Subramanian >Assignee: Sarath Subramanian >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: default...@cm.png > > > 1.Create a lineage as attached in the screenshot (i.e) tag propagated from a > source using 2 processes to a target > 2. Add a tag to t3 , now the tag is propagated to t4 via both process1 and > process2. > 3. Now block the propagation from t3 to process1. > 4. Expectation is that , tag will be propagated via process2 to t4. But the > propagated tag is removed from t4 > !default...@cm.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
Review Request 73316: ATLAS-4269: Deferred Actions : When a tag is propagated from an entity via 2 processes , blocking 1 process removes tag propagated from another process
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73316/ --- Review request for atlas, Ashutosh Mestry, Deep Singh, Madhan Neethiraj, Radhika Kundam, and Sidharth Mishra. Bugs: ATLAS-4269 https://issues.apache.org/jira/browse/ATLAS-4269 Repository: atlas Description --- Issue = 1.Create a lineage as attached in the screenshot (see JIRA) tag propagated from a source using 2 processes to a target 2. Add a tag to t3 , now the tag is propagated to t4 via both process1 and process2. 3. Now block the propagation from t3 to process1. 4. Expectation is that , tag will be propagated via process2 to t4. But the propagated tag is removed from t4 Solution: = Recently tag propagation logic was updated to use queue implementation. When adding impacted vertices to the result added a check to see if vertex already present. So no duplicates present. Diffs - repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 9abcf646c Diff: https://reviews.apache.org/r/73316/diff/1/ Testing --- Precommit: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/532/console Manually validated in circular lineage Thanks, Sarath Subramanian
