Re: Review Request 74960: ATLAS-4853: Upgrade Netty to 4.1.108.Final due to CVE-2024-29025
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74960/ --- (Updated April 23, 2024, 3:52 a.m.) Review request for atlas, Jayendra Parab and Radhika Kundam. Summary (updated) - ATLAS-4853: Upgrade Netty to 4.1.108.Final due to CVE-2024-29025 Bugs: ATLAS-4853 https://issues.apache.org/jira/browse/ATLAS-4853 Repository: atlas Description --- Upgrade netty to 4.1.108.Final Diffs - pom.xml 6e6724275 Diff: https://reviews.apache.org/r/74960/diff/1/ Testing --- mvn clean -DskipTests install mvn clean -DskipTests package -Pdist,embedded-hbase-solr Thanks, Disha Talreja
Review Request 74960: ATLAS-4853: Upgrade Netty to 4.1.108.Final due to CVE-2024-29025 Edit Add comment Assign More Submit Patch Start Progress
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74960/ --- Review request for atlas, Jayendra Parab and Radhika Kundam. Bugs: ATLAS-4853 https://issues.apache.org/jira/browse/ATLAS-4853 Repository: atlas Description --- Upgrade netty to 4.1.108.Final Diffs - pom.xml 6e6724275 Diff: https://reviews.apache.org/r/74960/diff/1/ Testing --- mvn clean -DskipTests install mvn clean -DskipTests package -Pdist,embedded-hbase-solr Thanks, Disha Talreja
[jira] [Created] (ATLAS-4853) Upgrade Netty to 4.1.108.Final due to CVE-2024-29025
Disha Talreja created ATLAS-4853: Summary: Upgrade Netty to 4.1.108.Final due to CVE-2024-29025 Key: ATLAS-4853 URL: https://issues.apache.org/jira/browse/ATLAS-4853 Project: Atlas Issue Type: Task Components: atlas-core Reporter: Disha Talreja Assignee: Disha Talreja Upgrade Netty to 4.1.108.Final due to CVE-2024-29025 Netty is an asynchronous event-driven network application framework for rapid development of maintainable high performance protocol servers & clients. The `HttpPostRequestDecoder` can be tricked to accumulate data. While the decoder can store items on the disk if configured so, there are no limits to the number of fields the form can have, an attacher can send a chunked post consisting of many small fields that will be accumulated in the `bodyListHttpData` list. The decoder cumulates bytes in the `undecodedChunk` buffer until it can decode a field, this field can cumulate data without limits. This vulnerability is fixed in 4.1.108.Final. [https://nvd.nist.gov/vuln/detail/CVE-2024-29025] [https://github.com/advisories/GHSA-5jpm-x58v-624v] [https://ossindex.sonatype.org/vulnerability/CVE-2024-29025] -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74902: ATLAS-4826: Provide Liveness and Readyness probes in Atlas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74902/ --- (Updated Feb. 22, 2024, 11:26 p.m.) Review request for atlas, Jayendra Parab, Pinal Shah, and Radhika Kundam. Changes --- As the previous patch was specific to HBase & Solr, this change accommodates the use of any storage backend & index stores for Atlas Health Check. Bugs: ATLAS-4826 https://issues.apache.org/jira/browse/ATLAS-4826 Repository: atlas Description --- In order to monitor the application health and be able to restart it if it is unhealthy we need to check the: 1) Liveness - if Atlas is live and running as expected 2) Readyness - if Atlas is live and ready to accept client requests Diffs (updated) - repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java beb90e67b webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 4d59fa3d6 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 5b16ba149 Diff: https://reviews.apache.org/r/74902/diff/2/ Changes: https://reviews.apache.org/r/74902/diff/1-2/ Testing --- Thanks, Disha Talreja
Review Request 74902: ATLAS-4826: Provide Liveness and Readyness probes in Atlas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74902/ --- Review request for atlas, Jayendra Parab, Pinal Shah, and Radhika Kundam. Bugs: ATLAS-4826 https://issues.apache.org/jira/browse/ATLAS-4826 Repository: atlas Description --- In order to monitor the application health and be able to restart it if it is unhealthy we need to check the: 1) Liveness - if Atlas is live and running as expected 2) Readyness - if Atlas is live and ready to accept client requests Diffs - graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/GraphDatabase.java 83f57d139 graphdb/janus/pom.xml 80fe82bfd graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 115b681cc webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 4d59fa3d6 webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 5b16ba149 Diff: https://reviews.apache.org/r/74902/diff/1/ Testing --- Thanks, Disha Talreja
[jira] [Created] (ATLAS-4826) Provide Liveness and Readyness probes in Atlas
Disha Talreja created ATLAS-4826: Summary: Provide Liveness and Readyness probes in Atlas Key: ATLAS-4826 URL: https://issues.apache.org/jira/browse/ATLAS-4826 Project: Atlas Issue Type: Improvement Components: atlas-core Reporter: Disha Talreja Assignee: Disha Talreja In order to monitor the application health and be able to restart it if it is unhealthy we need to check the: 1) Liveness - if Atlas is live and running as expected 2) Readyness - if Atlas is live and ready to accept client requests -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74707: ATLAS-4806: Upgrade netty to 4.1.100.Final due to CVE-2023-44487
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74707/ --- (Updated Feb. 8, 2024, 5:32 p.m.) Review request for atlas, Jayendra Parab and Radhika Kundam. Bugs: ATLAS-4806 https://issues.apache.org/jira/browse/ATLAS-4806 Repository: atlas Description --- Upgraded netty version to 4.1.100.Final Diffs - pom.xml 0e21c7f31 Diff: https://reviews.apache.org/r/74707/diff/2/ Testing (updated) --- Verified Manually PC Build: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1547/ Thanks, Disha Talreja
Review Request 74707: ATLAS-4806: Upgrade netty to 4.1.100.Final due to CVE-2023-44487
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74707/ --- Review request for atlas, Jayendra Parab, Radhika Kundam, and Sidharth Mishra. Bugs: ATLAS-4806 https://issues.apache.org/jira/browse/ATLAS-4806 Repository: atlas Description --- Upgraded netty version to 4.1.100.Final Diffs - pom.xml f76d0ea04 Diff: https://reviews.apache.org/r/74707/diff/1/ Testing --- Verified Manually Thanks, Disha Talreja
[jira] [Created] (ATLAS-4806) Upgrade netty to 4.1.100.Final due to CVE-2023-44487
Disha Talreja created ATLAS-4806: Summary: Upgrade netty to 4.1.100.Final due to CVE-2023-44487 Key: ATLAS-4806 URL: https://issues.apache.org/jira/browse/ATLAS-4806 Project: Atlas Issue Type: Task Components: atlas-core Reporter: Disha Talreja Assignee: Disha Talreja CVE-2023-44487 The HTTP/2 protocol allows a denial of service (server resource consumption) because request cancellation can reset many streams quickly, as exploited in the wild in August through October 2023. *Base Score:* [7.5 HIGH|https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator?name=CVE-2023-44487=AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H=3.1=NIST] There is a known exploit for this vulnerability, so we need to prioritise this despite it being a High severity CVE and not a critical. [https://github.com/netty/netty/security/advisories/GHSA-xpw8-rcwv-8f8p] h4. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74676: ATLAS-4801: Atlas - Upgrade Okio to 3.4.0 due to CVE-2023-3635
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74676/ --- (Updated Oct. 27, 2023, 5:50 p.m.) Review request for atlas, Jayendra Parab, Radhika Kundam, and Sidharth Mishra. Bugs: ATLAS-4801 https://issues.apache.org/jira/browse/ATLAS-4801 Repository: atlas Description --- GzipSource does not handle an exception that might be raised when parsing a malformed gzip buffer. This may lead to denial of service of the Okio client when handling a crafted GZIP archive, by using the GzipSource class. CVSSv3 Score:- 7.5(High) https://nvd.nist.gov/vuln/detail/CVE-2023-3635 Diffs - pom.xml f76d0ea04 Diff: https://reviews.apache.org/r/74676/diff/1/ Testing (updated) --- Verified Manually PC Build: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1485/ [One test failed which is not related to this change - (EntityV2JerseyResourceIT.testSetLabelsByTypeName:986)] Thanks, Disha Talreja
Review Request 74676: ATLAS-4801: Atlas - Upgrade Okio to 3.4.0 due to CVE-2023-3635
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74676/ --- Review request for atlas, Jayendra Parab, Radhika Kundam, and Sidharth Mishra. Bugs: ATLAS-4801 https://issues.apache.org/jira/browse/ATLAS-4801 Repository: atlas Description --- GzipSource does not handle an exception that might be raised when parsing a malformed gzip buffer. This may lead to denial of service of the Okio client when handling a crafted GZIP archive, by using the GzipSource class. CVSSv3 Score:- 7.5(High) https://nvd.nist.gov/vuln/detail/CVE-2023-3635 Diffs - pom.xml f76d0ea04 Diff: https://reviews.apache.org/r/74676/diff/1/ Testing --- Verified Manually Thanks, Disha Talreja
Review Request 74664: ATLAS-4800: Atlas - Upgrade Apache Ivy to 2.5.2 due to CVE-2022-46751
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74664/ --- Review request for atlas, Jayendra Parab, Mehul Parikh, and Radhika Kundam. Bugs: ATLAS-4800 https://issues.apache.org/jira/browse/ATLAS-4800 Repository: atlas Description --- Upgrade Apache Ivy to 2.5.2 due to CVE-2022-46751 Diffs - pom.xml f76d0ea04 Diff: https://reviews.apache.org/r/74664/diff/1/ Testing --- Verified locally Thanks, Disha Talreja
[jira] [Created] (ATLAS-4801) Atlas - Upgrade Okio to 3.4.0 due to CVE-2023-3635
Disha Talreja created ATLAS-4801: Summary: Atlas - Upgrade Okio to 3.4.0 due to CVE-2023-3635 Key: ATLAS-4801 URL: https://issues.apache.org/jira/browse/ATLAS-4801 Project: Atlas Issue Type: Task Components: atlas-core Reporter: Disha Talreja Assignee: Disha Talreja GzipSource does not handle an exception that might be raised when parsing a malformed gzip buffer. This may lead to denial of service of the Okio client when handling a crafted GZIP archive, by using the GzipSource class. CVSSv3 Score:- 7.5(High) [https://nvd.nist.gov/vuln/detail/CVE-2023-3635] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4800) Atlas - Upgrade Apache Ivy to 2.5.2 due to CVE-2022-46751
[ https://issues.apache.org/jira/browse/ATLAS-4800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Disha Talreja updated ATLAS-4800: - Description: Upgrade Apache Ivy to 2.5.2 due to CVE-2022-46751 Improper Restriction of XML External Entity Reference, XML Injection (aka Blind XPath Injection) vulnerability in Apache Software Foundation Apache Ivy.This issue affects any version of Apache Ivy prior to 2.5.2. When Apache Ivy prior to 2.5.2 parses XML files - either its own configuration, Ivy files or Apache Maven POMs - it will allow downloading external document type definitions and expand any entity references contained therein when used. This can be used to exfiltrate data, access resources only the machine running Ivy has access to or disturb the execution of Ivy in different ways. CVSSv3 Score:- 8.2(High) [https://nvd.nist.gov/vuln/detail/CVE-2022-46751] > Atlas - Upgrade Apache Ivy to 2.5.2 due to CVE-2022-46751 > - > > Key: ATLAS-4800 > URL: https://issues.apache.org/jira/browse/ATLAS-4800 > Project: Atlas > Issue Type: Task > Components: atlas-core > Reporter: Disha Talreja > Assignee: Disha Talreja >Priority: Major > > Upgrade Apache Ivy to 2.5.2 due to CVE-2022-46751 > Improper Restriction of XML External Entity Reference, XML Injection (aka > Blind XPath Injection) vulnerability in Apache Software Foundation Apache > Ivy.This issue affects any version of Apache Ivy prior to 2.5.2. When Apache > Ivy prior to 2.5.2 parses XML files - either its own configuration, Ivy files > or Apache Maven POMs - it will allow downloading external document type > definitions and expand any entity references contained therein when used. > This can be used to exfiltrate data, access resources only the machine > running Ivy has access to or disturb the execution of Ivy in different ways. > CVSSv3 Score:- 8.2(High) > [https://nvd.nist.gov/vuln/detail/CVE-2022-46751] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (ATLAS-4800) Atlas - Upgrade Apache Ivy to 2.5.2 due to CVE-2022-46751
Disha Talreja created ATLAS-4800: Summary: Atlas - Upgrade Apache Ivy to 2.5.2 due to CVE-2022-46751 Key: ATLAS-4800 URL: https://issues.apache.org/jira/browse/ATLAS-4800 Project: Atlas Issue Type: Task Components: atlas-core Reporter: Disha Talreja Assignee: Disha Talreja -- This message was sent by Atlassian Jira (v8.20.10#820010)
Review Request 74544: ATLAS-4565: Remove log4j 1.x dependencies due to EOL
/classification-updater/src/main/resources/atlas-log4j.xml 9658edf4e tools/classification-updater/src/main/resources/atlas-log4j2.xml PRE-CREATION tools/classification-updater/src/main/resources/update-classifications.sh c11d5e19a webapp/pom.xml 7d2d4c952 webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java e476ba28a Diff: https://reviews.apache.org/r/74544/diff/1/ Testing --- 1. Local Build Successful. 2. Atlas server running. 3. Patched a cluster Thanks, Disha Talreja
Re: Review Request 74390: ATLAS-4739: Log4j migration - Replace DailyRollingFileAppender with RollingFileAppender
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74390/ --- (Updated April 11, 2023, 4 p.m.) Review request for atlas. Bugs: ATLAS-4739 https://issues.apache.org/jira/browse/ATLAS-4739 Repository: atlas Description --- Log4j 1.x provides a class org.apache.log4j.DailyRollingFileAppender which is no longer supported by Log4j 2.x. Hence, DailyRollingFileAppender needs to be replaced by a custom rolling file appender supported by log4j2. Diffs - notification/src/main/java/org/apache/atlas/hook/FailedMessagesLogger.java 5488c1c2c notification/src/main/java/org/apache/atlas/notification/LogConfigUtils.java ea9c1cecc Diff: https://reviews.apache.org/r/74390/diff/1/ Testing (updated) --- mvn clean -DskipTests install mvn clean -DskipTests package -Pdist,embedded-hbase-solr Atlas server running. PC build: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1340/ Thanks, Disha Talreja
Review Request 74390: ATLAS-4739: Log4j migration - Replace DailyRollingFileAppender with RollingFileAppender
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74390/ --- Review request for atlas. Bugs: ATLAS-4739 https://issues.apache.org/jira/browse/ATLAS-4739 Repository: atlas Description --- Log4j 1.x provides a class org.apache.log4j.DailyRollingFileAppender which is no longer supported by Log4j 2.x. Hence, DailyRollingFileAppender needs to be replaced by a custom rolling file appender supported by log4j2. Diffs - notification/src/main/java/org/apache/atlas/hook/FailedMessagesLogger.java 5488c1c2c notification/src/main/java/org/apache/atlas/notification/LogConfigUtils.java ea9c1cecc Diff: https://reviews.apache.org/r/74390/diff/1/ Testing --- mvn clean -DskipTests install mvn clean -DskipTests package -Pdist,embedded-hbase-solr Atlas server running. Thanks, Disha Talreja
[jira] [Created] (ATLAS-4739) Log4j migration: Replace DailyRollingFileAppender with RollingFileAppender
Disha Talreja created ATLAS-4739:
Summary: Log4j migration: Replace DailyRollingFileAppender with
RollingFileAppender
Key: ATLAS-4739
URL: https://issues.apache.org/jira/browse/ATLAS-4739
Project: Atlas
Issue Type: Sub-task
Components: atlas-core
Reporter: Disha Talreja
Assignee: Disha Talreja
Log4j 1.x provides a class
[{{org.apache.log4j.DailyRollingFileAppender}}|https://logging.apache.org/log4j/1.2/apidocs/]
which is no longer supported by Log4j 2.x. Hence, DailyRollingFileAppender
needs to be replaced by a custom rolling file appender supported by log4j2.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (ATLAS-4674) Regression: Classification tagging is not sending appropriate notification to ATLAS_ENTITIES
[
https://issues.apache.org/jira/browse/ATLAS-4674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Disha Talreja updated ATLAS-4674:
-
Attachment: ATLAS-4674.patch
> Regression: Classification tagging is not sending appropriate notification to
> ATLAS_ENTITIES
>
>
> Key: ATLAS-4674
> URL: https://issues.apache.org/jira/browse/ATLAS-4674
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
> Reporter: Disha Talreja
> Assignee: Disha Talreja
>Priority: Major
> Attachments: ATLAS-4674.patch
>
>
> When we add/delete a classification from an entity, we send notification to
> ATLAS_ENTITIES so ranger tag sync works appropriately to facilitate Tag-based
> policy. This functionally seems to be broken and is a regression.
> On further investigation, it is seen that the issue is caused by deferred
> action for classification propagation as the right notifications are sent
> when deferred action is disabled.
> Considering the following example:
> 1. Create two table as shown below at Hive:
> =
> create table parent_table(ssn varchar(100));
> create table child_table as select * from parent_table;
> 2. Created two classification: "PII" and "sensitive"
>
> 3. Tagged PII to parent_table and notification received at ATLAS_ENTITIES
> topic:
> ---
> {code:java}
> {
> "source": {},
> "version":
> { "version": "1.0.0", "versionParts": [ 1 ] }
> ,
> "msgCompressionKind": "NONE",
> "msgSplitIdx": 1,
> "msgSplitCount": 1,
> "msgSourceIP": "172.27.40.5",
> "msgCreatedBy": "",
> "msgCreationTime": 1662660308744,
> "spooled": false,
> "message": {
> "type": "ENTITY_NOTIFICATION_V2",
> "entity": {
> "typeName": "hive_table",
> "attributes":
> { "owner": "admin", "createTime": 1662602695000,
> "qualifiedName": "default.parent_table@cm", "name": "parent_table"
> }
> ,
> "guid": "04ff79d7-b1ff-4354-ba75-a35b28d2f53f",
> "status": "ACTIVE",
> "displayText": "parent_table",
> "classificationNames": [
> "PII" ],
> "classifications": [
>
> { "typeName": "PII", "entityGuid":
> "04ff79d7-b1ff-4354-ba75-a35b28d2f53f", "entityStatus": "ACTIVE",
> "propagate": true, "validityPeriods": [],
> "removePropagationsOnEntityDelete": false }
> ],
> "isIncomplete": false },
> "operationType": "CLASSIFICATION_ADD",
> "eventTime": 1662660307360
> }
> }
> {
> "source": {},
> "version":
> { "version": "1.0.0", "versionParts": [ 1 ] }
> ,
> "msgCompressionKind": "NONE",
> "msgSplitIdx": 1,
> "msgSplitCount": 1,
> "msgSourceIP": "172.27.40.5",
> "msgCreatedBy": "",
> "msgCreationTime": 1662660309419,
> "spooled": false,
> "message": {
> "type": "ENTITY_NOTIFICATION_V2",
> "entity": {
> "typeName": "hive_process",
> "attributes":
> { "qualifiedName": "default.child_table@cm:1662602726000",
> "clusterName": "cm", "name": "default.child_table@cm:1662602726000"
> }
> ,
> "guid": "0e180d04-79ed-4cab-bbc3-76af4b876a5b",
> "status": "ACTIVE",
> "displayText": "default.child_table@cm:1662602726000",
> "classificationNames": [
> "PII" ],
> "classifications": [
>
> {
[jira] [Updated] (ATLAS-4674) Regression: Classification tagging is not sending appropriate notification to ATLAS_ENTITIES
[
https://issues.apache.org/jira/browse/ATLAS-4674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Disha Talreja updated ATLAS-4674:
-
Attachment: (was: ATLAS-4674.patch)
> Regression: Classification tagging is not sending appropriate notification to
> ATLAS_ENTITIES
>
>
> Key: ATLAS-4674
> URL: https://issues.apache.org/jira/browse/ATLAS-4674
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
> Reporter: Disha Talreja
> Assignee: Disha Talreja
>Priority: Major
> Attachments: ATLAS-4674.patch
>
>
> When we add/delete a classification from an entity, we send notification to
> ATLAS_ENTITIES so ranger tag sync works appropriately to facilitate Tag-based
> policy. This functionally seems to be broken and is a regression.
> On further investigation, it is seen that the issue is caused by deferred
> action for classification propagation as the right notifications are sent
> when deferred action is disabled.
> Considering the following example:
> 1. Create two table as shown below at Hive:
> =
> create table parent_table(ssn varchar(100));
> create table child_table as select * from parent_table;
> 2. Created two classification: "PII" and "sensitive"
>
> 3. Tagged PII to parent_table and notification received at ATLAS_ENTITIES
> topic:
> ---
> {code:java}
> {
> "source": {},
> "version":
> { "version": "1.0.0", "versionParts": [ 1 ] }
> ,
> "msgCompressionKind": "NONE",
> "msgSplitIdx": 1,
> "msgSplitCount": 1,
> "msgSourceIP": "172.27.40.5",
> "msgCreatedBy": "",
> "msgCreationTime": 1662660308744,
> "spooled": false,
> "message": {
> "type": "ENTITY_NOTIFICATION_V2",
> "entity": {
> "typeName": "hive_table",
> "attributes":
> { "owner": "admin", "createTime": 1662602695000,
> "qualifiedName": "default.parent_table@cm", "name": "parent_table"
> }
> ,
> "guid": "04ff79d7-b1ff-4354-ba75-a35b28d2f53f",
> "status": "ACTIVE",
> "displayText": "parent_table",
> "classificationNames": [
> "PII" ],
> "classifications": [
>
> { "typeName": "PII", "entityGuid":
> "04ff79d7-b1ff-4354-ba75-a35b28d2f53f", "entityStatus": "ACTIVE",
> "propagate": true, "validityPeriods": [],
> "removePropagationsOnEntityDelete": false }
> ],
> "isIncomplete": false },
> "operationType": "CLASSIFICATION_ADD",
> "eventTime": 1662660307360
> }
> }
> {
> "source": {},
> "version":
> { "version": "1.0.0", "versionParts": [ 1 ] }
> ,
> "msgCompressionKind": "NONE",
> "msgSplitIdx": 1,
> "msgSplitCount": 1,
> "msgSourceIP": "172.27.40.5",
> "msgCreatedBy": "",
> "msgCreationTime": 1662660309419,
> "spooled": false,
> "message": {
> "type": "ENTITY_NOTIFICATION_V2",
> "entity": {
> "typeName": "hive_process",
> "attributes":
> { "qualifiedName": "default.child_table@cm:1662602726000",
> "clusterName": "cm", "name": "default.child_table@cm:1662602726000"
> }
> ,
> "guid": "0e180d04-79ed-4cab-bbc3-76af4b876a5b",
> "status": "ACTIVE",
> "displayText": "default.child_table@cm:1662602726000",
> "classificationNames": [
> "PII" ],
> "classifications": [
>
> {
Re: Review Request 74176: ATLAS-4674 - Regression: Classification tagging is not sending appropriate notification to ATLAS_ENTITIES
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74176/ --- (Updated Oct. 18, 2022, 7:57 p.m.) Review request for atlas and Jayendra Parab. Bugs: ATLAS-4674 https://issues.apache.org/jira/browse/ATLAS-4674 Repository: atlas Description --- When we add/delete a classification from an entity, we send notification to ATLAS_ENTITIES so ranger tag sync works appropriately to facilitate Tag-based policy. This functionally seems to be broken and is a regression. On further investigation, it is seen that the issue is caused by deferred action for classification propagation as the right notifications are sent when deferred action is disabled. Entity details for the notification messages are fetched from the entity cache which is not updated and lacks the new classification mapping. Hence, this patch clears the cache so that the entity is newly mapped from it's vertex and contains all the information. Diffs (updated) - repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationTask.java 74bed571f Diff: https://reviews.apache.org/r/74176/diff/2/ Changes: https://reviews.apache.org/r/74176/diff/1-2/ Testing --- mvn clean -DskipTests install mvn clean -DskipTests package -Pdist,embedded-hbase-solr Atlas server running. Classification add, Classification delete: Verified appropriate messages being received on ATLAS_ENTITIES. Thanks, Disha Talreja
[jira] [Updated] (ATLAS-4674) Regression: Classification tagging is not sending appropriate notification to ATLAS_ENTITIES
[
https://issues.apache.org/jira/browse/ATLAS-4674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Disha Talreja updated ATLAS-4674:
-
Attachment: (was: ATLAS-4674.patch)
> Regression: Classification tagging is not sending appropriate notification to
> ATLAS_ENTITIES
>
>
> Key: ATLAS-4674
> URL: https://issues.apache.org/jira/browse/ATLAS-4674
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
> Reporter: Disha Talreja
> Assignee: Disha Talreja
>Priority: Major
> Attachments: ATLAS-4674.patch
>
>
> When we add/delete a classification from an entity, we send notification to
> ATLAS_ENTITIES so ranger tag sync works appropriately to facilitate Tag-based
> policy. This functionally seems to be broken and is a regression.
> On further investigation, it is seen that the issue is caused by deferred
> action for classification propagation as the right notifications are sent
> when deferred action is disabled.
> Considering the following example:
> 1. Create two table as shown below at Hive:
> =
> create table parent_table(ssn varchar(100));
> create table child_table as select * from parent_table;
> 2. Created two classification: "PII" and "sensitive"
>
> 3. Tagged PII to parent_table and notification received at ATLAS_ENTITIES
> topic:
> ---
> {code:java}
> {
> "source": {},
> "version":
> { "version": "1.0.0", "versionParts": [ 1 ] }
> ,
> "msgCompressionKind": "NONE",
> "msgSplitIdx": 1,
> "msgSplitCount": 1,
> "msgSourceIP": "172.27.40.5",
> "msgCreatedBy": "",
> "msgCreationTime": 1662660308744,
> "spooled": false,
> "message": {
> "type": "ENTITY_NOTIFICATION_V2",
> "entity": {
> "typeName": "hive_table",
> "attributes":
> { "owner": "admin", "createTime": 1662602695000,
> "qualifiedName": "default.parent_table@cm", "name": "parent_table"
> }
> ,
> "guid": "04ff79d7-b1ff-4354-ba75-a35b28d2f53f",
> "status": "ACTIVE",
> "displayText": "parent_table",
> "classificationNames": [
> "PII" ],
> "classifications": [
>
> { "typeName": "PII", "entityGuid":
> "04ff79d7-b1ff-4354-ba75-a35b28d2f53f", "entityStatus": "ACTIVE",
> "propagate": true, "validityPeriods": [],
> "removePropagationsOnEntityDelete": false }
> ],
> "isIncomplete": false },
> "operationType": "CLASSIFICATION_ADD",
> "eventTime": 1662660307360
> }
> }
> {
> "source": {},
> "version":
> { "version": "1.0.0", "versionParts": [ 1 ] }
> ,
> "msgCompressionKind": "NONE",
> "msgSplitIdx": 1,
> "msgSplitCount": 1,
> "msgSourceIP": "172.27.40.5",
> "msgCreatedBy": "",
> "msgCreationTime": 1662660309419,
> "spooled": false,
> "message": {
> "type": "ENTITY_NOTIFICATION_V2",
> "entity": {
> "typeName": "hive_process",
> "attributes":
> { "qualifiedName": "default.child_table@cm:1662602726000",
> "clusterName": "cm", "name": "default.child_table@cm:1662602726000"
> }
> ,
> "guid": "0e180d04-79ed-4cab-bbc3-76af4b876a5b",
> "status": "ACTIVE",
> "displayText": "default.child_table@cm:1662602726000",
> "classificationNames": [
> "PII" ],
> "classifications": [
>
> {
Review Request 74176: ATLAS-4674 - Regression: Classification tagging is not sending appropriate notification to ATLAS_ENTITIES
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74176/ --- Review request for atlas and Jayendra Parab. Bugs: ATLAS-4674 https://issues.apache.org/jira/browse/ATLAS-4674 Repository: atlas Description --- When we add/delete a classification from an entity, we send notification to ATLAS_ENTITIES so ranger tag sync works appropriately to facilitate Tag-based policy. This functionally seems to be broken and is a regression. On further investigation, it is seen that the issue is caused by deferred action for classification propagation as the right notifications are sent when deferred action is disabled. Entity details for the notification messages are fetched from the entity cache which is not updated and lacks the new classification mapping. Hence, this patch clears the cache so that the entity is newly mapped from it's vertex and contains all the information. Diffs - repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationTask.java 74bed571f Diff: https://reviews.apache.org/r/74176/diff/1/ Testing --- mvn clean -DskipTests install mvn clean -DskipTests package -Pdist,embedded-hbase-solr Atlas server running. Classification add, Classification delete: Verified appropriate messages being received on ATLAS_ENTITIES. Thanks, Disha Talreja
[jira] [Updated] (ATLAS-4674) Regression: Classification tagging is not sending appropriate notification to ATLAS_ENTITIES
[
https://issues.apache.org/jira/browse/ATLAS-4674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Disha Talreja updated ATLAS-4674:
-
Attachment: ATLAS-4674.patch
> Regression: Classification tagging is not sending appropriate notification to
> ATLAS_ENTITIES
>
>
> Key: ATLAS-4674
> URL: https://issues.apache.org/jira/browse/ATLAS-4674
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
> Reporter: Disha Talreja
> Assignee: Disha Talreja
>Priority: Major
> Attachments: ATLAS-4674.patch
>
>
> When we add/delete a classification from an entity, we send notification to
> ATLAS_ENTITIES so ranger tag sync works appropriately to facilitate Tag-based
> policy. This functionally seems to be broken and is a regression.
> On further investigation, it is seen that the issue is caused by deferred
> action for classification propagation as the right notifications are sent
> when deferred action is disabled.
> Considering the following example:
> 1. Create two table as shown below at Hive:
> =
> create table parent_table(ssn varchar(100));
> create table child_table as select * from parent_table;
> 2. Created two classification: "PII" and "sensitive"
>
> 3. Tagged PII to parent_table and notification received at ATLAS_ENTITIES
> topic:
> ---
> {code:java}
> {
> "source": {},
> "version":
> { "version": "1.0.0", "versionParts": [ 1 ] }
> ,
> "msgCompressionKind": "NONE",
> "msgSplitIdx": 1,
> "msgSplitCount": 1,
> "msgSourceIP": "172.27.40.5",
> "msgCreatedBy": "",
> "msgCreationTime": 1662660308744,
> "spooled": false,
> "message": {
> "type": "ENTITY_NOTIFICATION_V2",
> "entity": {
> "typeName": "hive_table",
> "attributes":
> { "owner": "admin", "createTime": 1662602695000,
> "qualifiedName": "default.parent_table@cm", "name": "parent_table"
> }
> ,
> "guid": "04ff79d7-b1ff-4354-ba75-a35b28d2f53f",
> "status": "ACTIVE",
> "displayText": "parent_table",
> "classificationNames": [
> "PII" ],
> "classifications": [
>
> { "typeName": "PII", "entityGuid":
> "04ff79d7-b1ff-4354-ba75-a35b28d2f53f", "entityStatus": "ACTIVE",
> "propagate": true, "validityPeriods": [],
> "removePropagationsOnEntityDelete": false }
> ],
> "isIncomplete": false },
> "operationType": "CLASSIFICATION_ADD",
> "eventTime": 1662660307360
> }
> }
> {
> "source": {},
> "version":
> { "version": "1.0.0", "versionParts": [ 1 ] }
> ,
> "msgCompressionKind": "NONE",
> "msgSplitIdx": 1,
> "msgSplitCount": 1,
> "msgSourceIP": "172.27.40.5",
> "msgCreatedBy": "",
> "msgCreationTime": 1662660309419,
> "spooled": false,
> "message": {
> "type": "ENTITY_NOTIFICATION_V2",
> "entity": {
> "typeName": "hive_process",
> "attributes":
> { "qualifiedName": "default.child_table@cm:1662602726000",
> "clusterName": "cm", "name": "default.child_table@cm:1662602726000"
> }
> ,
> "guid": "0e180d04-79ed-4cab-bbc3-76af4b876a5b",
> "status": "ACTIVE",
> "displayText": "default.child_table@cm:1662602726000",
> "classificationNames": [
> "PII" ],
> "classifications": [
>
> {
[jira] [Updated] (ATLAS-4674) Regression: Classification tagging is not sending appropriate notification to ATLAS_ENTITIES
[
https://issues.apache.org/jira/browse/ATLAS-4674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Disha Talreja updated ATLAS-4674:
-
Attachment: ATLAS-4674.patch
> Regression: Classification tagging is not sending appropriate notification to
> ATLAS_ENTITIES
>
>
> Key: ATLAS-4674
> URL: https://issues.apache.org/jira/browse/ATLAS-4674
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
> Reporter: Disha Talreja
> Assignee: Disha Talreja
>Priority: Major
> Attachments: ATLAS-4674.patch
>
>
> When we add/delete a classification from an entity, we send notification to
> ATLAS_ENTITIES so ranger tag sync works appropriately to facilitate Tag-based
> policy. This functionally seems to be broken and is a regression.
> On further investigation, it is seen that the issue is caused by deferred
> action for classification propagation as the right notifications are sent
> when deferred action is disabled.
> Considering the following example:
> 1. Create two table as shown below at Hive:
> =
> create table parent_table(ssn varchar(100));
> create table child_table as select * from parent_table;
> 2. Created two classification: "PII" and "sensitive"
>
> 3. Tagged PII to parent_table and notification received at ATLAS_ENTITIES
> topic:
> ---
> {code:java}
> {
> "source": {},
> "version":
> { "version": "1.0.0", "versionParts": [ 1 ] }
> ,
> "msgCompressionKind": "NONE",
> "msgSplitIdx": 1,
> "msgSplitCount": 1,
> "msgSourceIP": "172.27.40.5",
> "msgCreatedBy": "",
> "msgCreationTime": 1662660308744,
> "spooled": false,
> "message": {
> "type": "ENTITY_NOTIFICATION_V2",
> "entity": {
> "typeName": "hive_table",
> "attributes":
> { "owner": "admin", "createTime": 1662602695000,
> "qualifiedName": "default.parent_table@cm", "name": "parent_table"
> }
> ,
> "guid": "04ff79d7-b1ff-4354-ba75-a35b28d2f53f",
> "status": "ACTIVE",
> "displayText": "parent_table",
> "classificationNames": [
> "PII" ],
> "classifications": [
>
> { "typeName": "PII", "entityGuid":
> "04ff79d7-b1ff-4354-ba75-a35b28d2f53f", "entityStatus": "ACTIVE",
> "propagate": true, "validityPeriods": [],
> "removePropagationsOnEntityDelete": false }
> ],
> "isIncomplete": false },
> "operationType": "CLASSIFICATION_ADD",
> "eventTime": 1662660307360
> }
> }
> {
> "source": {},
> "version":
> { "version": "1.0.0", "versionParts": [ 1 ] }
> ,
> "msgCompressionKind": "NONE",
> "msgSplitIdx": 1,
> "msgSplitCount": 1,
> "msgSourceIP": "172.27.40.5",
> "msgCreatedBy": "",
> "msgCreationTime": 1662660309419,
> "spooled": false,
> "message": {
> "type": "ENTITY_NOTIFICATION_V2",
> "entity": {
> "typeName": "hive_process",
> "attributes":
> { "qualifiedName": "default.child_table@cm:1662602726000",
> "clusterName": "cm", "name": "default.child_table@cm:1662602726000"
> }
> ,
> "guid": "0e180d04-79ed-4cab-bbc3-76af4b876a5b",
> "status": "ACTIVE",
> "displayText": "default.child_table@cm:1662602726000",
> "classificationNames": [
> "PII" ],
> "classifications": [
>
> {
[jira] [Updated] (ATLAS-4674) Regression: Classification tagging is not sending appropriate notification to ATLAS_ENTITIES
[
https://issues.apache.org/jira/browse/ATLAS-4674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Disha Talreja updated ATLAS-4674:
-
Attachment: (was: ATLAS-4674.patch)
> Regression: Classification tagging is not sending appropriate notification to
> ATLAS_ENTITIES
>
>
> Key: ATLAS-4674
> URL: https://issues.apache.org/jira/browse/ATLAS-4674
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
> Reporter: Disha Talreja
> Assignee: Disha Talreja
>Priority: Major
> Attachments: ATLAS-4674.patch
>
>
> When we add/delete a classification from an entity, we send notification to
> ATLAS_ENTITIES so ranger tag sync works appropriately to facilitate Tag-based
> policy. This functionally seems to be broken and is a regression.
> On further investigation, it is seen that the issue is caused by deferred
> action for classification propagation as the right notifications are sent
> when deferred action is disabled.
> Considering the following example:
> 1. Create two table as shown below at Hive:
> =
> create table parent_table(ssn varchar(100));
> create table child_table as select * from parent_table;
> 2. Created two classification: "PII" and "sensitive"
>
> 3. Tagged PII to parent_table and notification received at ATLAS_ENTITIES
> topic:
> ---
> {code:java}
> {
> "source": {},
> "version":
> { "version": "1.0.0", "versionParts": [ 1 ] }
> ,
> "msgCompressionKind": "NONE",
> "msgSplitIdx": 1,
> "msgSplitCount": 1,
> "msgSourceIP": "172.27.40.5",
> "msgCreatedBy": "",
> "msgCreationTime": 1662660308744,
> "spooled": false,
> "message": {
> "type": "ENTITY_NOTIFICATION_V2",
> "entity": {
> "typeName": "hive_table",
> "attributes":
> { "owner": "admin", "createTime": 1662602695000,
> "qualifiedName": "default.parent_table@cm", "name": "parent_table"
> }
> ,
> "guid": "04ff79d7-b1ff-4354-ba75-a35b28d2f53f",
> "status": "ACTIVE",
> "displayText": "parent_table",
> "classificationNames": [
> "PII" ],
> "classifications": [
>
> { "typeName": "PII", "entityGuid":
> "04ff79d7-b1ff-4354-ba75-a35b28d2f53f", "entityStatus": "ACTIVE",
> "propagate": true, "validityPeriods": [],
> "removePropagationsOnEntityDelete": false }
> ],
> "isIncomplete": false },
> "operationType": "CLASSIFICATION_ADD",
> "eventTime": 1662660307360
> }
> }
> {
> "source": {},
> "version":
> { "version": "1.0.0", "versionParts": [ 1 ] }
> ,
> "msgCompressionKind": "NONE",
> "msgSplitIdx": 1,
> "msgSplitCount": 1,
> "msgSourceIP": "172.27.40.5",
> "msgCreatedBy": "",
> "msgCreationTime": 1662660309419,
> "spooled": false,
> "message": {
> "type": "ENTITY_NOTIFICATION_V2",
> "entity": {
> "typeName": "hive_process",
> "attributes":
> { "qualifiedName": "default.child_table@cm:1662602726000",
> "clusterName": "cm", "name": "default.child_table@cm:1662602726000"
> }
> ,
> "guid": "0e180d04-79ed-4cab-bbc3-76af4b876a5b",
> "status": "ACTIVE",
> "displayText": "default.child_table@cm:1662602726000",
> "classificationNames": [
> "PII" ],
> "classifications": [
>
> {
Review Request 74166: ATLAS-4674 : Regression: Classification tagging is not sending appropriate notification to ATLAS_ENTITIES
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74166/ --- Review request for atlas, Jayendra Parab and Sidharth Mishra. Bugs: ATLAS-4674 https://issues.apache.org/jira/browse/ATLAS-4674 Repository: atlas Description --- When we add/delete a classification from an entity, we send notification to ATLAS_ENTITIES so ranger tag sync works appropriately to facilitate Tag-based policy. This functionally seems to be broken and is a regression. On further investigation, it is seen that the issue is caused by deferred action for classification propagation as the right notifications are sent when deferred action is disabled. Entity details for the notification messages are fetched from the entity cache which is not updated and lacks the new classification mapping. This patch updates the cache with the most recent entity information after a classification is attached/removed. Diffs - repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java 8d306c745 Diff: https://reviews.apache.org/r/74166/diff/1/ Testing --- mvn clean -DskipTests install mvn clean -DskipTests package -Pdist,embedded-hbase-solr Atlas server running. Classification add, Classification delete: Verified appropriate messages being received on ATLAS_ENTITIES. Thanks, Disha Talreja
[jira] [Updated] (ATLAS-4674) Regression: Classification tagging is not sending appropriate notification to ATLAS_ENTITIES
[
https://issues.apache.org/jira/browse/ATLAS-4674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Disha Talreja updated ATLAS-4674:
-
Description:
When we add/delete a classification from an entity, we send notification to
ATLAS_ENTITIES so ranger tag sync works appropriately to facilitate Tag-based
policy. This functionally seems to be broken and is a regression.
On further investigation, it is seen that the issue is caused by deferred
action for classification propagation as the right notifications are sent when
deferred action is disabled.
Considering the following example:
1. Create two table as shown below at Hive:
=
create table parent_table(ssn varchar(100));
create table child_table as select * from parent_table;
2. Created two classification: "PII" and "sensitive"
3. Tagged PII to parent_table and notification received at ATLAS_ENTITIES topic:
---
{code:java}
{
"source": {},
"version":
{ "version": "1.0.0", "versionParts": [ 1 ] }
,
"msgCompressionKind": "NONE",
"msgSplitIdx": 1,
"msgSplitCount": 1,
"msgSourceIP": "172.27.40.5",
"msgCreatedBy": "",
"msgCreationTime": 1662660308744,
"spooled": false,
"message": {
"type": "ENTITY_NOTIFICATION_V2",
"entity": {
"typeName": "hive_table",
"attributes":
{ "owner": "admin", "createTime": 1662602695000,
"qualifiedName": "default.parent_table@cm", "name": "parent_table"
}
,
"guid": "04ff79d7-b1ff-4354-ba75-a35b28d2f53f",
"status": "ACTIVE",
"displayText": "parent_table",
"classificationNames": [
"PII" ],
"classifications": [
{ "typeName": "PII", "entityGuid":
"04ff79d7-b1ff-4354-ba75-a35b28d2f53f", "entityStatus": "ACTIVE",
"propagate": true, "validityPeriods": [],
"removePropagationsOnEntityDelete": false }
],
"isIncomplete": false },
"operationType": "CLASSIFICATION_ADD",
"eventTime": 1662660307360
}
}
{
"source": {},
"version":
{ "version": "1.0.0", "versionParts": [ 1 ] }
,
"msgCompressionKind": "NONE",
"msgSplitIdx": 1,
"msgSplitCount": 1,
"msgSourceIP": "172.27.40.5",
"msgCreatedBy": "",
"msgCreationTime": 1662660309419,
"spooled": false,
"message": {
"type": "ENTITY_NOTIFICATION_V2",
"entity": {
"typeName": "hive_process",
"attributes":
{ "qualifiedName": "default.child_table@cm:1662602726000",
"clusterName": "cm", "name": "default.child_table@cm:1662602726000"
}
,
"guid": "0e180d04-79ed-4cab-bbc3-76af4b876a5b",
"status": "ACTIVE",
"displayText": "default.child_table@cm:1662602726000",
"classificationNames": [
"PII" ],
"classifications": [
{ "typeName": "PII", "entityGuid":
"04ff79d7-b1ff-4354-ba75-a35b28d2f53f", "entityStatus": "ACTIVE",
"propagate": true, "validityPeriods": [],
"removePropagationsOnEntityDelete": false }
],
"isIncomplete": false },
"operationType": "CLASSIFICATION_ADD",
"eventTime": 1662660308935
}
}
{
"source": {},
"version":
{ "version": "1.0.0", "versionParts": [ 1 ] }
,
"msgCompressionKind": "NONE",
"msgSplitIdx": 1,
"msgSplitCount": 1,
"msgSourceIP": "172.27.40.5",
"msgCreatedBy": "",
"msgCreationTime": 1662660309419,
"spooled": false,
"message": {
"type": "ENTITY_NOTIFICATION_V2",
"entit
[jira] [Created] (ATLAS-4674) Regression: Classification tagging is not sending appropriate notification to ATLAS_ENTITIES
Disha Talreja created ATLAS-4674:
Summary: Regression: Classification tagging is not sending
appropriate notification to ATLAS_ENTITIES
Key: ATLAS-4674
URL: https://issues.apache.org/jira/browse/ATLAS-4674
Project: Atlas
Issue Type: Bug
Components: atlas-core
Reporter: Disha Talreja
Assignee: Disha Talreja
When we add/delete a classification from an entity, we send notification to
ATLAS_ENTITIES so ranger tag sync works appropriately to facilitate Tag-based
policy. This functionally seems to be broken and is a regression.
On further investigation, it is seen that the issue is caused by deferred
action for classification propagation as the right notifications are sent when
deferred action is disabled.
Considering the following exampe:
1. Create two table as shown below at Hive:
=
create table parent_table(ssn varchar(100));
create table child_table as select * from parent_table;
2. Created two classification: "PII" and "sensitive"
3. Tagged PII to parent_table and notification received at ATLAS_ENTITIES topic:
---
{
"source": {},
"version": {
"version": "1.0.0",
"versionParts": [
1
]
},
"msgCompressionKind": "NONE",
"msgSplitIdx": 1,
"msgSplitCount": 1,
"msgSourceIP": "172.27.40.5",
"msgCreatedBy": "",
"msgCreationTime": 1662660308744,
"spooled": false,
"message": {
"type": "ENTITY_NOTIFICATION_V2",
"entity": {
"typeName": "hive_table",
"attributes": {
"owner": "admin",
"createTime": 1662602695000,
"qualifiedName": "default.parent_table@cm",
"name": "parent_table" },
"guid": "04ff79d7-b1ff-4354-ba75-a35b28d2f53f",
"status": "ACTIVE",
"displayText": "parent_table",
"classificationNames": [
"PII" ],
"classifications": [
{
"typeName": "PII",
"entityGuid": "04ff79d7-b1ff-4354-ba75-a35b28d2f53f",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false }
],
"isIncomplete": false },
"operationType": "CLASSIFICATION_ADD",
"eventTime": 1662660307360
}
}
{
"source": {},
"version": {
"version": "1.0.0",
"versionParts": [
1
]
},
"msgCompressionKind": "NONE",
"msgSplitIdx": 1,
"msgSplitCount": 1,
"msgSourceIP": "172.27.40.5",
"msgCreatedBy": "",
"msgCreationTime": 1662660309419,
"spooled": false,
"message": {
"type": "ENTITY_NOTIFICATION_V2",
"entity": {
"typeName": "hive_process",
"attributes": {
"qualifiedName": "default.child_table@cm:1662602726000",
"clusterName": "cm",
"name": "default.child_table@cm:1662602726000" },
"guid": "0e180d04-79ed-4cab-bbc3-76af4b876a5b",
"status": "ACTIVE",
"displayText": "default.child_table@cm:1662602726000",
"classificationNames": [
"PII" ],
"classifications": [
{
"typeName": "PII",
"entityGuid": "04ff79d7-b1ff-4354-ba75-a35b28d2f53f",
"entityStatus": "ACTIVE",
"propagate": true,
"validityPeriods": [],
"removePropagationsOnEntityDelete": false }
],
"isIncomplete": false },
"operationType": "CLASSIFICATION_ADD",
"eventTime": 1662660308935
}
}
{
"source": {},
"version": {
"version": "1.0.0",
"versionParts": [
1
]
},
"msgCompressionKind": "NONE",
"msgSplitIdx": 1,
"msgSplitCount": 1,
"msgSourceIP": "172.27.40.5",
"msgCrea
Re: Review Request 73956: ATLAS-4569: Kafka importer should use only kafka-clients from Kafka libs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73956/#review224352 --- Ship it! Ship It! - Disha Talreja On April 22, 2022, 6:46 a.m., Andras Katona wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73956/ > --- > > (Updated April 22, 2022, 6:46 a.m.) > > > Review request for atlas, Dharshana MK, Disha Talreja, madhan, and Madhan > Neethiraj. > > > Bugs: ATLAS-4569 > https://issues.apache.org/jira/browse/ATLAS-4569 > > > Repository: atlas > > > Description > --- > > Putting kafka-clients and slf4j-api to classpath from kafka/libs. Other > dependencies of kafka-clients are compression libs, which are not used, hence > not required. > > > Diffs > - > > addons/kafka-bridge/src/bin/import-kafka.sh fbc16f0d0 > > > Diff: https://reviews.apache.org/r/73956/diff/1/ > > > Testing > --- > > manually on a cluster > > > Thanks, > > Andras Katona > >
[jira] [Assigned] (ATLAS-4569) Import kafka is failing with NoSuchMethod error
[
https://issues.apache.org/jira/browse/ATLAS-4569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Disha Talreja reassigned ATLAS-4569:
Assignee: Disha Talreja
> Import kafka is failing with NoSuchMethod error
> ---
>
> Key: ATLAS-4569
> URL: https://issues.apache.org/jira/browse/ATLAS-4569
> Project: Atlas
> Issue Type: Bug
>Reporter: Dharshana M Krishnamoorthy
> Assignee: Disha Talreja
>Priority: Major
>
> Unable to perform kafka import as it is failing with the following exception.
> Log from console
> {code:java}
> >>>>> /opt/cloudera/parcels/CDH/lib/atlas
> Using Kafka configuration directory [/etc/kafka/conf]
> Log file for import is /var/log/atlas/import-kafka.log
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.23291269/jars/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.23291269/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Exception in thread "main" java.lang.NoSuchMethodError:
> org.apache.log4j.helpers.OptionConverter.convertLevel(Ljava/lang/String;Lorg/apache/logging/log4j/Level;)Lorg/apache/logging/log4j/Level;
> at
> org.apache.log4j.xml.XmlConfiguration.parseLevel(XmlConfiguration.java:646)
> at
> org.apache.log4j.xml.XmlConfiguration.lambda$parseChildrenOfLoggerElement$4(XmlConfiguration.java:563)
> at
> org.apache.log4j.xml.XmlConfiguration.forEachElement(XmlConfiguration.java:762)
> at
> org.apache.log4j.xml.XmlConfiguration.parseChildrenOfLoggerElement(XmlConfiguration.java:548)
> at
> org.apache.log4j.xml.XmlConfiguration.parseCategory(XmlConfiguration.java:530)
> at
> org.apache.log4j.xml.XmlConfiguration.lambda$parse$6(XmlConfiguration.java:721)
> at
> org.apache.log4j.xml.XmlConfiguration.forEachElement(XmlConfiguration.java:762)
> at org.apache.log4j.xml.XmlConfiguration.parse(XmlConfiguration.java:717)
> at
> org.apache.log4j.xml.XmlConfiguration.doConfigure(XmlConfiguration.java:166)
> at
> org.apache.log4j.xml.XmlConfiguration.doConfigure(XmlConfiguration.java:141)
> at
> org.apache.log4j.config.Log4j1Configuration.initialize(Log4j1Configuration.java:60)
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:293)
> at
> org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:626)
> at
> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:699)
> at
> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:716)
> at
> org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:270)
> at
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:155)
> at
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)
> at org.apache.logging.log4j.LogManager.getContext(LogManager.java:196)
> at
> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:137)
> at
> org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:55)
> at
> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:47)
> at
> org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:33)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:363)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:388)
> at org.apache.atlas.kafka.bridge.KafkaBridge.(KafkaBridge.java:56)
> Failed to import Kafka Data Model!!! {code}
> Issue is seen on 7.1.8 build: *23291269*
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (ATLAS-4565) Remove log4j 1.x dependencies due to EOL
Disha Talreja created ATLAS-4565: Summary: Remove log4j 1.x dependencies due to EOL Key: ATLAS-4565 URL: https://issues.apache.org/jira/browse/ATLAS-4565 Project: Atlas Issue Type: Task Components: atlas-core Reporter: Disha Talreja Assignee: Disha Talreja Atlas - Remove log4j 1.x dependencies due to EOL Log4j 1.x went EOL in August 2015 - so it has been >5 years. [https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces] -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: Review Request 73637: ATLAS-4370: Persist Metrics for user to retrieve Metrics info at a past timestamp
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73637/ --- (Updated Dec. 2, 2021, 5:36 p.m.) Review request for atlas, Ashutosh Mestry, Radhika Kundam, Sarath Subramanian, and Sidharth Mishra. Bugs: ATLAS-4370 https://issues.apache.org/jira/browse/ATLAS-4370 Repository: atlas Description --- Rebased patch #73483 (https://reviews.apache.org/r/73483/) against the newest master branch. Diffs - addons/models/-Area0/0010-base_model.json 769d8859f intg/src/main/java/org/apache/atlas/AtlasConfiguration.java b63fab7cf intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 7d092616d intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetricsMapToChart.java PRE-CREATION intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetricsStat.java PRE-CREATION intg/src/main/java/org/apache/atlas/utils/AtlasEntityUtil.java 1e78e2550 repository/src/main/java/org/apache/atlas/repository/ogm/metrics/AtlasMetricsStatDTO.java PRE-CREATION repository/src/main/java/org/apache/atlas/services/MetricsService.java ceb252822 repository/src/test/java/org/apache/atlas/TestModules.java 8dda20825 repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 040592184 test-tools/src/main/resources/solr/core-template/solrconfig.xml 3341e7199 webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 0580f7f33 Diff: https://reviews.apache.org/r/73637/diff/7/ Testing (updated) --- 1) Tested locally. 2) Patched a cluster. PC build: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/997/console Thanks, Disha Talreja
Re: Review Request 73637: ATLAS-4370: Persist Metrics for user to retrieve Metrics info at a past timestamp
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73637/ --- (Updated Nov. 15, 2021, 4:06 p.m.) Review request for atlas, Ashutosh Mestry, Radhika Kundam, Sarath Subramanian, and Sidharth Mishra. Changes --- 1. Removed duplicate "__AtlasMetricsStat" 2. Single entry and exit debug logs in "scheduleSaveAndDeleteMetrics()" 3. Moved private methods after public Bugs: ATLAS-4370 https://issues.apache.org/jira/browse/ATLAS-4370 Repository: atlas Description --- Rebased patch #73483 (https://reviews.apache.org/r/73483/) against the newest master branch. Diffs (updated) - addons/models/-Area0/0010-base_model.json 769d8859f intg/src/main/java/org/apache/atlas/AtlasConfiguration.java b63fab7cf intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 7d092616d intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetricsMapToChart.java PRE-CREATION intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetricsStat.java PRE-CREATION intg/src/main/java/org/apache/atlas/utils/AtlasEntityUtil.java 1e78e2550 repository/src/main/java/org/apache/atlas/repository/ogm/metrics/AtlasMetricsStatDTO.java PRE-CREATION repository/src/main/java/org/apache/atlas/services/MetricsService.java ceb252822 repository/src/test/java/org/apache/atlas/TestModules.java 8dda20825 repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 040592184 test-tools/src/main/resources/solr/core-template/solrconfig.xml 3341e7199 webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 0580f7f33 Diff: https://reviews.apache.org/r/73637/diff/7/ Changes: https://reviews.apache.org/r/73637/diff/6-7/ Testing --- 1) Tested locally. 2) Patched a cluster. PC build: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/880/ Thanks, Disha Talreja
Re: Review Request 73637: ATLAS-4370: Persist Metrics for user to retrieve Metrics info at a past timestamp
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73637/ --- (Updated Nov. 9, 2021, 7:54 p.m.) Review request for atlas, Ashutosh Mestry, Radhika Kundam, Sarath Subramanian, and Sidharth Mishra. Bugs: ATLAS-4370 https://issues.apache.org/jira/browse/ATLAS-4370 Repository: atlas Description --- Rebased patch #73483 (https://reviews.apache.org/r/73483/) against the newest master branch. Diffs (updated) - addons/models/-Area0/0010-base_model.json 769d8859f intg/src/main/java/org/apache/atlas/AtlasConfiguration.java b63fab7cf intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 7d092616d intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetricsMapToChart.java PRE-CREATION intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetricsStat.java PRE-CREATION intg/src/main/java/org/apache/atlas/utils/AtlasEntityUtil.java 1e78e2550 repository/src/main/java/org/apache/atlas/repository/ogm/metrics/AtlasMetricsStatDTO.java PRE-CREATION repository/src/main/java/org/apache/atlas/services/MetricsService.java ceb252822 repository/src/test/java/org/apache/atlas/TestModules.java 8dda20825 repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 040592184 test-tools/src/main/resources/solr/core-template/solrconfig.xml 3341e7199 webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 0580f7f33 Diff: https://reviews.apache.org/r/73637/diff/6/ Changes: https://reviews.apache.org/r/73637/diff/5-6/ Testing --- 1) Tested locally. 2) Patched a cluster. PC build: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/880/ Thanks, Disha Talreja
Re: Review Request 73691: ATLAS-4431: Random NPE when retrieving tasks (“part2 - additional null check”)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73691/ --- (Updated Nov. 8, 2021, 11:33 p.m.) Review request for atlas, Ashutosh Mestry, Radhika Kundam, Sarath Subramanian, and Sidharth Mishra. Changes --- Removed the logger because it didn't carry much information. Bugs: ATLAS-4431 https://issues.apache.org/jira/browse/ATLAS-4431 Repository: atlas Description --- System tests fail randomly because retrieving tasks using the "admin/tasks" endpoint sometimes returns NPE. This patch includes a few additional null checks to avoid NPE. Diffs (updated) - repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 32e0ad91b Diff: https://reviews.apache.org/r/73691/diff/3/ Changes: https://reviews.apache.org/r/73691/diff/2-3/ Testing --- Local Build Successful. Thanks, Disha Talreja
Re: Review Request 73691: ATLAS-4431: Random NPE when retrieving tasks (“part2 - additional null check”)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73691/ --- (Updated Nov. 8, 2021, 9:52 p.m.) Review request for atlas, Ashutosh Mestry, Radhika Kundam, Sarath Subramanian, and Sidharth Mishra. Changes --- Added a log statement Bugs: ATLAS-4431 https://issues.apache.org/jira/browse/ATLAS-4431 Repository: atlas Description --- System tests fail randomly because retrieving tasks using the "admin/tasks" endpoint sometimes returns NPE. This patch includes a few additional null checks to avoid NPE. Diffs (updated) - repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 32e0ad91b Diff: https://reviews.apache.org/r/73691/diff/2/ Changes: https://reviews.apache.org/r/73691/diff/1-2/ Testing --- Local Build Successful. Thanks, Disha Talreja
[jira] [Reopened] (ATLAS-4431) Random NPE when retrieving tasks
[
https://issues.apache.org/jira/browse/ATLAS-4431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Disha Talreja reopened ATLAS-4431:
--
> Random NPE when retrieving tasks
>
>
> Key: ATLAS-4431
> URL: https://issues.apache.org/jira/browse/ATLAS-4431
> Project: Atlas
> Issue Type: Bug
> Components: atlas-intg
>Affects Versions: 2.2.0
> Reporter: Disha Talreja
> Assignee: Disha Talreja
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: ATLAS-4431.patch
>
>
> System tests fail randomly because retrieving tasks using the "admin/tasks"
> endpoint sometimes returns HTTP 500.
> Atlas logs:
> {noformat}
> 2021-09-06 13:51:28,938 INFO - [etp985324122-24:] ~ Logged into Atlas as =
> hrt_qa, by proxyUser = null
> (AtlasAuthenticationFilter$KerberosFilterChainWrapper:724)
> 2021-09-06 13:51:28,939 INFO -
> [etp985324122-24:knox:POST/api/atlas/v2/entity/bulk/classification] ~ Request
> from authenticated user: knox,
> URL=/api/atlas/v2/entity/bulk/classification?doAs=hrt_qa
> (AtlasAuthenticationFilter$KerberosFilterChainWrapper:739)
> 2021-09-06 13:51:28,969 INFO - [etp985324122-24 -
> 78a12f6b-7472-4600-a35d-bfe660358712:knox:POST/api/atlas/v2/entity/bulk/classification]
> ~ updateClassificationText: tag_iuanl: tag_iuanl date Mon Sep 06 13:51:28
> UTC 2021 hive_principal_type USER boolean true string XXJid byte -58 double
> 9.475147275250526E37 arr_string fEOKO short -29141 arr_int 781216025 float
> -3.09E38 int 1369026004 long -2808157562510626816 (EntityGraphMapper:2732)
> 2021-09-06 13:51:29,173 INFO - [etp985324122-24 -
> 78a12f6b-7472-4600-a35d-bfe660358712:knox:POST/api/atlas/v2/entity/bulk/classification]
> ~
> {"type":"CLASSIFICATION_PROPAGATION_ADD","guid":"a4ed45a8-cc15-41dd-9340-ebaca95b07f1","createdBy":"hrt_qa","createdTime":1630936288947,"updatedTime":1630936288947,"parameters":{"relationshipGuid":null,"entityGuid":"5c89916f-9dbc-4be8-afd5-9e7659bbb745","classificationVertexId":"44486696"},"attemptCount":0,"status":"PENDING"}
> (TaskExecutor$TaskLogger:170)
> 2021-09-06 13:51:29,190 INFO - [atlas-task-0-etp985324122-24 -
> c6aa9190-8454-43a4-a862-1f61393994a6:] ~ updateClassificationText: tag_iuanl:
> tag_iuanl date Mon Sep 06 13:51:28 UTC 2021 hive_principal_type USER boolean
> true string XXJid byte -58 double 9.475147275250526E37 arr_string fEOKO short
> -29141 arr_int 781216025 float -3.09E38 int 1369026004 long
> -2808157562510626816 (EntityGraphMapper:2732)
> 2021-09-06 13:51:29,193 INFO - [atlas-task-0-etp985324122-24 -
> c6aa9190-8454-43a4-a862-1f61393994a6:] ~ updateClassificationText: tag_iuanl:
> tag_iuanl date Mon Sep 06 13:51:28 UTC 2021 hive_principal_type USER boolean
> true string XXJid byte -58 double 9.475147275250526E37 arr_string fEOKO short
> -29141 arr_int 781216025 float -3.09E38 int 1369026004 long
> -2808157562510626816 (EntityGraphMapper:2732)
> 2021-09-06 13:51:29,366 ERROR - [etp985324122-594:] ~ URL not supported in HA
> mode: /api/atlas/admin/tasks (ActiveServerFilter:121)
> 2021-09-06 13:51:29,366 INFO - [etp985324122-594:] ~ Logged into Atlas as =
> hrt_qa, by proxyUser = null
> (AtlasAuthenticationFilter$KerberosFilterChainWrapper:724)
> 2021-09-06 13:51:29,366 INFO -
> [etp985324122-594:knox:GET/api/atlas/admin/tasks] ~ Request from
> authenticated user: knox, URL=/api/atlas/admin/tasks?doAs=hrt_qa
> (AtlasAuthenticationFilter$KerberosFilterChainWrapper:739)
> 2021-09-06 13:51:29,375 ERROR - [etp985324122-594 -
> 73d22020-c612-4a00-8f47-b2ebd149e976:knox:GET/api/atlas/admin/tasks] ~ Error
> handling a request: f1318c48aa5440a9 (ExceptionMapperUtil:32)
> java.lang.NullPointerException
> at
> org.apache.atlas.tasks.TaskRegistry.toAtlasTask(TaskRegistry.java:188)
> at org.apache.atlas.tasks.TaskRegistry.getAll(TaskRegistry.java:155)
> at
> org.apache.atlas.tasks.TaskManagement.getAll(TaskManagement.java:120)
> at
> org.apache.atlas.web.resources.AdminResource.getTaskStatus(AdminResource.java:772)
> at jdk.internal.reflect.GeneratedMethodAccessor324.invoke(Unknown
> Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethod
Review Request 73691: ATLAS-4431: Random NPE when retrieving tasks (“part2 - additional null check”)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73691/ --- Review request for atlas, Ashutosh Mestry, Radhika Kundam, Sarath Subramanian, and Sidharth Mishra. Bugs: ATLAS-4431 https://issues.apache.org/jira/browse/ATLAS-4431 Repository: atlas Description --- System tests fail randomly because retrieving tasks using the "admin/tasks" endpoint sometimes returns NPE. This patch includes a few additional null checks to avoid NPE. Diffs - repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 32e0ad91b Diff: https://reviews.apache.org/r/73691/diff/1/ Testing --- Local Build Successful. Thanks, Disha Talreja
Re: Review Request 73637: ATLAS-4370: Persist Metrics for user to retrieve Metrics info at a past timestamp
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73637/ --- (Updated Nov. 2, 2021, 8:35 p.m.) Review request for atlas, Ashutosh Mestry, Radhika Kundam, Sarath Subramanian, and Sidharth Mishra. Changes --- Few changes based on review comments. Bugs: ATLAS-4370 https://issues.apache.org/jira/browse/ATLAS-4370 Repository: atlas Description --- Rebased patch #73483 (https://reviews.apache.org/r/73483/) against the newest master branch. Diffs (updated) - addons/models/-Area0/0010-base_model.json 769d8859f intg/src/main/java/org/apache/atlas/AtlasConfiguration.java b63fab7cf intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 7d092616d intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetricsMapToChart.java PRE-CREATION intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetricsStat.java PRE-CREATION intg/src/main/java/org/apache/atlas/utils/AtlasEntityUtil.java 1e78e2550 repository/src/main/java/org/apache/atlas/repository/ogm/metrics/AtlasMetricsStatDTO.java PRE-CREATION repository/src/main/java/org/apache/atlas/services/MetricsService.java ceb252822 repository/src/test/java/org/apache/atlas/TestModules.java 8dda20825 repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 040592184 test-tools/src/main/resources/solr/core-template/solrconfig.xml 3341e7199 webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 135b94b36 Diff: https://reviews.apache.org/r/73637/diff/5/ Changes: https://reviews.apache.org/r/73637/diff/4-5/ Testing --- 1) Tested locally. 2) Patched a cluster. PC build: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/880/ Thanks, Disha Talreja
Re: Review Request 73637: ATLAS-4370: Persist Metrics for user to retrieve Metrics info at a past timestamp
> On Oct. 29, 2021, 7:09 p.m., Radhika Kundam wrote: > > repository/src/main/java/org/apache/atlas/services/MetricsService.java > > Lines 441 (patched) > > <https://reviews.apache.org/r/73637/diff/4/?file=2254350#file2254350line442> > > > > Why do we need to map to AtlasMetricsStat as already allMetrics is List > > of AtlasMetricsStat allMetrics is a list of AtlasMetricsStat but this method also includes 'typeNames' and it needs to be passed to the appropriate constructor. > On Oct. 29, 2021, 7:09 p.m., Radhika Kundam wrote: > > repository/src/main/java/org/apache/atlas/services/MetricsService.java > > Lines 442 (patched) > > <https://reviews.apache.org/r/73637/diff/4/?file=2254350#file2254350line443> > > > > Instead of getting entire metricstats and then setting metrics to null > > here you can send minInfo flas true at Line#436 I had done that before but setting minInfo to true in the beginning fails with a NPE because constructor makes the following call "AtlasEntityUtil.metricsToTypeData(metrics, listOfTypeNames, typeData)" and this requires metrics. Hence, had to get entire metricsstats and then set metrics to null. - Disha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73637/#review223697 --- On Oct. 26, 2021, 10:49 p.m., Disha Talreja wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73637/ > --- > > (Updated Oct. 26, 2021, 10:49 p.m.) > > > Review request for atlas, Ashutosh Mestry, Radhika Kundam, Sarath > Subramanian, and Sidharth Mishra. > > > Bugs: ATLAS-4370 > https://issues.apache.org/jira/browse/ATLAS-4370 > > > Repository: atlas > > > Description > --- > > Rebased patch #73483 (https://reviews.apache.org/r/73483/) against the newest > master branch. > > > Diffs > - > > addons/models/-Area0/0010-base_model.json 769d8859f > intg/src/main/java/org/apache/atlas/AtlasConfiguration.java b63fab7cf > intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 2febff469 > > intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetricsMapToChart.java > PRE-CREATION > intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetricsStat.java > PRE-CREATION > intg/src/main/java/org/apache/atlas/utils/AtlasEntityUtil.java 1e78e2550 > > repository/src/main/java/org/apache/atlas/repository/ogm/metrics/AtlasMetricsStatDTO.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/services/MetricsService.java > ceb252822 > repository/src/test/java/org/apache/atlas/TestModules.java 8dda20825 > repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java > 040592184 > test-tools/src/main/resources/solr/core-template/solrconfig.xml 3341e7199 > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > 135b94b36 > > > Diff: https://reviews.apache.org/r/73637/diff/4/ > > > Testing > --- > > 1) Tested locally. > 2) Patched a cluster. > > PC build: > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/880/ > > > Thanks, > > Disha Talreja > >
Re: Review Request 73637: ATLAS-4370: Persist Metrics for user to retrieve Metrics info at a past timestamp
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73637/ --- (Updated Oct. 26, 2021, 10:49 p.m.) Review request for atlas, Ashutosh Mestry, Sarath Subramanian, and Sidharth Mishra. Changes --- Rebased against the most recent changes. Bugs: ATLAS-4370 https://issues.apache.org/jira/browse/ATLAS-4370 Repository: atlas Description --- Rebased patch #73483 (https://reviews.apache.org/r/73483/) against the newest master branch. Diffs (updated) - addons/models/-Area0/0010-base_model.json 769d8859f intg/src/main/java/org/apache/atlas/AtlasConfiguration.java b63fab7cf intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 2febff469 intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetricsMapToChart.java PRE-CREATION intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetricsStat.java PRE-CREATION intg/src/main/java/org/apache/atlas/utils/AtlasEntityUtil.java 1e78e2550 repository/src/main/java/org/apache/atlas/repository/ogm/metrics/AtlasMetricsStatDTO.java PRE-CREATION repository/src/main/java/org/apache/atlas/services/MetricsService.java ceb252822 repository/src/test/java/org/apache/atlas/TestModules.java 8dda20825 repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 040592184 test-tools/src/main/resources/solr/core-template/solrconfig.xml 3341e7199 webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 135b94b36 Diff: https://reviews.apache.org/r/73637/diff/4/ Changes: https://reviews.apache.org/r/73637/diff/3-4/ Testing --- 1) Tested locally. 2) Patched a cluster. PC build: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/880/ Thanks, Disha Talreja
[jira] [Assigned] (ATLAS-4370) Persist Metrics for user to retrieve Metrics info at a past timestamp
[ https://issues.apache.org/jira/browse/ATLAS-4370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Disha Talreja reassigned ATLAS-4370: Assignee: Disha Talreja > Persist Metrics for user to retrieve Metrics info at a past timestamp > - > > Key: ATLAS-4370 > URL: https://issues.apache.org/jira/browse/ATLAS-4370 > Project: Atlas > Issue Type: New Feature > Components: atlas-intg, atlas-webui >Affects Versions: 2.1.0 >Reporter: JP > Assignee: Disha Talreja >Priority: Major > > # Create new typeDef of MetricsStat to persist Metrics entity. > # Create MetricsStat class to keep Metrics' info about collectionTime, > metricsId and ttl; create MetricsStatsDTO class to transfer states between > MetricsStat and AtlasEntity when it is saved or retrieved from JanusGraph. > # Create MetricsStatMin to represent the minimal information about persisted > MetricsStat. > # Create job scheduler with configurable metrics.persist.schedule (default > as 7 days) & metrics.ttl.hours (default as 1 hr interval) to save a metrics > entity to db and to sweep through all metrics entities and clear the ones > whose time-to-live (ttl) has past. > # Update MetricsService to implement REST APIs in AdminResource to realize > api calls of: > ** Get all persisted metricsStats, with boolean variable to choose its > return type as a list of metricsStats with specific details, or with minimal > information; > ** Get a specific metricsStat by collectionTime > # Update MetricsService to include helper methods of: > ** Save a metricsStat to db > ** Get or delete a persisted metricsStat by collectionTime > ** Purge metricsStats whose ttl has past -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 73637: ATLAS-4370: Persist Metrics for user to retrieve Metrics info at a past timestamp
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73637/ --- (Updated Oct. 23, 2021, 6:20 a.m.) Review request for atlas, Ashutosh Mestry, Sarath Subramanian, and Sidharth Mishra. Changes --- 1. Changes based on review comments 2. Merged AtlasMetricsBaseObject & AtlasMetricsMinInfo into AtlasMetricsStat class Bugs: ATLAS-4370 https://issues.apache.org/jira/browse/ATLAS-4370 Repository: atlas Description --- Rebased patch #73483 (https://reviews.apache.org/r/73483/) against the newest master branch. Diffs (updated) - addons/models/-Area0/0010-base_model.json 769d8859f intg/src/main/java/org/apache/atlas/AtlasConfiguration.java b63fab7cf intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 2febff469 intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetricsMapToChart.java PRE-CREATION intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetricsStat.java PRE-CREATION intg/src/main/java/org/apache/atlas/utils/AtlasEntityUtil.java 1e78e2550 repository/src/main/java/org/apache/atlas/repository/ogm/metrics/AtlasMetricsStatDTO.java PRE-CREATION repository/src/main/java/org/apache/atlas/services/MetricsService.java ceb252822 repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java beb90e67b repository/src/test/java/org/apache/atlas/TestModules.java 8dda20825 repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 040592184 test-tools/src/main/resources/solr/core-template/solrconfig.xml 3341e7199 webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 135b94b36 Diff: https://reviews.apache.org/r/73637/diff/3/ Changes: https://reviews.apache.org/r/73637/diff/2-3/ Testing --- 1) Tested locally. 2) Patched a cluster. PC build: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/880/ Thanks, Disha Talreja
[jira] [Commented] (ATLAS-4458) Commons-Logging Exclusion Causes Startup Problems
[ https://issues.apache.org/jira/browse/ATLAS-4458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17431444#comment-17431444 ] Disha Talreja commented on ATLAS-4458: -- +1 > Commons-Logging Exclusion Causes Startup Problems > - > > Key: ATLAS-4458 > URL: https://issues.apache.org/jira/browse/ATLAS-4458 > Project: Atlas > Issue Type: Bug >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Attachments: ATLAS-4458-Commons-logging-reference-fix.patch > > > Earlier commit of ATLAS-4351, caused _commons-logging*.jar_ to be excluded. > This causes startup problem due to missing dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Review Request 73637: ATLAS-4370: Persist Metrics for user to retrieve Metrics info at a past timestamp
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73637/ --- Review request for atlas, Ashutosh Mestry, Sarath Subramanian, and Sidharth Mishra. Bugs: ATLAS-4370 https://issues.apache.org/jira/browse/ATLAS-4370 Repository: atlas Description --- Rebased patch #73483 (https://reviews.apache.org/r/73483/) against the newest master branch. Diffs - addons/models/-Area0/0010-base_model.json 769d8859f intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 20f8f7343 intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 2febff469 intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetricsBaseObject.java PRE-CREATION intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetricsMapToChart.java PRE-CREATION intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetricsMinInfo.java PRE-CREATION intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetricsStat.java PRE-CREATION intg/src/main/java/org/apache/atlas/utils/AtlasEntityUtil.java 1e78e2550 repository/src/main/java/org/apache/atlas/repository/ogm/metrics/AtlasMetricsStatDTO.java PRE-CREATION repository/src/main/java/org/apache/atlas/services/MetricsService.java ceb252822 repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java beb90e67b repository/src/test/java/org/apache/atlas/TestModules.java 8dda20825 repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 040592184 test-tools/src/main/resources/solr/core-template/solrconfig.xml e7956cb36 webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 135b94b36 webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java.orig PRE-CREATION Diff: https://reviews.apache.org/r/73637/diff/1/ Testing --- 1) Tested locally. 2) Patched a cluster. Thanks, Disha Talreja
Review Request 73628: ATLAS-4444: Atlas - Upgrade commons-io to 2.11.0
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73628/ --- Review request for atlas, Ashutosh Mestry and Sarath Subramanian. Repository: atlas Description --- This patch upgrades commons-io 2.8 to 2.11.0 Diffs - pom.xml 47768e9fa Diff: https://reviews.apache.org/r/73628/diff/1/ Testing --- 1. Tested locally 2. Patched a cluster Thanks, Disha Talreja
[jira] [Updated] (ATLAS-4444) Atlas - Upgrade commons-io to 2.11.0
[ https://issues.apache.org/jira/browse/ATLAS-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Disha Talreja updated ATLAS-: - Description: Atlas - Upgrade commons-io to 2.11.0 > Atlas - Upgrade commons-io to 2.11.0 > > > Key: ATLAS- > URL: https://issues.apache.org/jira/browse/ATLAS- > Project: Atlas > Issue Type: Task > Components: atlas-core > Reporter: Disha Talreja > Assignee: Disha Talreja >Priority: Major > Fix For: 2.3.0 > > > Atlas - Upgrade commons-io to 2.11.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ATLAS-4444) Atlas - Upgrade commons-io to 2.11.0
[ https://issues.apache.org/jira/browse/ATLAS-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Disha Talreja reassigned ATLAS-: Assignee: Disha Talreja > Atlas - Upgrade commons-io to 2.11.0 > > > Key: ATLAS- > URL: https://issues.apache.org/jira/browse/ATLAS- > Project: Atlas > Issue Type: Task > Components: atlas-core > Reporter: Disha Talreja > Assignee: Disha Talreja >Priority: Major > Fix For: 2.3.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-4444) Atlas - Upgrade commons-io to 2.11.0
Disha Talreja created ATLAS-: Summary: Atlas - Upgrade commons-io to 2.11.0 Key: ATLAS- URL: https://issues.apache.org/jira/browse/ATLAS- Project: Atlas Issue Type: Task Components: atlas-core Reporter: Disha Talreja Fix For: 2.3.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-4440) Upgrade Atlas's Kafka dependency to 2.8
[ https://issues.apache.org/jira/browse/ATLAS-4440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424000#comment-17424000 ] Disha Talreja commented on ATLAS-4440: -- +1 for the patch. > Upgrade Atlas's Kafka dependency to 2.8 > --- > > Key: ATLAS-4440 > URL: https://issues.apache.org/jira/browse/ATLAS-4440 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Reporter: Viktor Somogyi-Vass >Assignee: Viktor Somogyi-Vass >Priority: Major > Attachments: 0001-ATLAS-4440-Update-Kafka-dependency-to-2.8.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
Review Request 73616: ATLAS-4421: [Atlas: Hive Import] When import-hive is run with incorrect input the information is not conveyed to the user
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73616/ --- Review request for atlas, Ashutosh Mestry and Sarath Subramanian. Repository: atlas Description --- Even if a non-existent file name/ database name/ table name is passed to the import-hive script, hive meta data gets imported successfully. This gives the user an impression that the import is successful while nothing has actually happened. With this patch, the program gets terminated for any incorrect argument passed and the issue is logged in the log file. Diffs - addons/hive-bridge/src/bin/import-hive.sh fde486f28 addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 08471b480 Diff: https://reviews.apache.org/r/73616/diff/1/ Testing --- Patched in a cluster. Thanks, Disha Talreja
Review Request 73593: ATLAS-4431: Random NPE when retrieving tasks
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73593/ --- Review request for atlas, Ashutosh Mestry, Sarath Subramanian, and Sidharth Mishra. Repository: atlas Description --- System tests fail randomly because retrieving tasks using the "admin/tasks" endpoint sometimes returns HTTP 500. This patch includes a few null checks to avoid NPE. Diffs - repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java fae8a4fed Diff: https://reviews.apache.org/r/73593/diff/1/ Testing --- Local Build Successful. Thanks, Disha Talreja
[jira] [Created] (ATLAS-4431) Random NPE when retrieving tasks
Disha Talreja created ATLAS-4431:
Summary: Random NPE when retrieving tasks
Key: ATLAS-4431
URL: https://issues.apache.org/jira/browse/ATLAS-4431
Project: Atlas
Issue Type: Bug
Components: atlas-intg
Reporter: Disha Talreja
System tests fail randomly because retrieving tasks using the "admin/tasks"
endpoint sometimes returns HTTP 500.
Atlas logs:
{noformat}
2021-09-06 13:51:28,938 INFO - [etp985324122-24:] ~ Logged into Atlas as =
hrt_qa, by proxyUser = null
(AtlasAuthenticationFilter$KerberosFilterChainWrapper:724)
2021-09-06 13:51:28,939 INFO -
[etp985324122-24:knox:POST/api/atlas/v2/entity/bulk/classification] ~ Request
from authenticated user: knox,
URL=/api/atlas/v2/entity/bulk/classification?doAs=hrt_qa
(AtlasAuthenticationFilter$KerberosFilterChainWrapper:739)
2021-09-06 13:51:28,969 INFO - [etp985324122-24 -
78a12f6b-7472-4600-a35d-bfe660358712:knox:POST/api/atlas/v2/entity/bulk/classification]
~ updateClassificationText: tag_iuanl: tag_iuanl date Mon Sep 06 13:51:28 UTC
2021 hive_principal_type USER boolean true string XXJid byte -58 double
9.475147275250526E37 arr_string fEOKO short -29141 arr_int 781216025 float
-3.09E38 int 1369026004 long -2808157562510626816 (EntityGraphMapper:2732)
2021-09-06 13:51:29,173 INFO - [etp985324122-24 -
78a12f6b-7472-4600-a35d-bfe660358712:knox:POST/api/atlas/v2/entity/bulk/classification]
~
{"type":"CLASSIFICATION_PROPAGATION_ADD","guid":"a4ed45a8-cc15-41dd-9340-ebaca95b07f1","createdBy":"hrt_qa","createdTime":1630936288947,"updatedTime":1630936288947,"parameters":{"relationshipGuid":null,"entityGuid":"5c89916f-9dbc-4be8-afd5-9e7659bbb745","classificationVertexId":"44486696"},"attemptCount":0,"status":"PENDING"}
(TaskExecutor$TaskLogger:170)
2021-09-06 13:51:29,190 INFO - [atlas-task-0-etp985324122-24 -
c6aa9190-8454-43a4-a862-1f61393994a6:] ~ updateClassificationText: tag_iuanl:
tag_iuanl date Mon Sep 06 13:51:28 UTC 2021 hive_principal_type USER boolean
true string XXJid byte -58 double 9.475147275250526E37 arr_string fEOKO short
-29141 arr_int 781216025 float -3.09E38 int 1369026004 long
-2808157562510626816 (EntityGraphMapper:2732)
2021-09-06 13:51:29,193 INFO - [atlas-task-0-etp985324122-24 -
c6aa9190-8454-43a4-a862-1f61393994a6:] ~ updateClassificationText: tag_iuanl:
tag_iuanl date Mon Sep 06 13:51:28 UTC 2021 hive_principal_type USER boolean
true string XXJid byte -58 double 9.475147275250526E37 arr_string fEOKO short
-29141 arr_int 781216025 float -3.09E38 int 1369026004 long
-2808157562510626816 (EntityGraphMapper:2732)
2021-09-06 13:51:29,366 ERROR - [etp985324122-594:] ~ URL not supported in HA
mode: /api/atlas/admin/tasks (ActiveServerFilter:121)
2021-09-06 13:51:29,366 INFO - [etp985324122-594:] ~ Logged into Atlas as =
hrt_qa, by proxyUser = null
(AtlasAuthenticationFilter$KerberosFilterChainWrapper:724)
2021-09-06 13:51:29,366 INFO -
[etp985324122-594:knox:GET/api/atlas/admin/tasks] ~ Request from authenticated
user: knox, URL=/api/atlas/admin/tasks?doAs=hrt_qa
(AtlasAuthenticationFilter$KerberosFilterChainWrapper:739)
2021-09-06 13:51:29,375 ERROR - [etp985324122-594 -
73d22020-c612-4a00-8f47-b2ebd149e976:knox:GET/api/atlas/admin/tasks] ~ Error
handling a request: f1318c48aa5440a9 (ExceptionMapperUtil:32)
java.lang.NullPointerException
at
org.apache.atlas.tasks.TaskRegistry.toAtlasTask(TaskRegistry.java:188)
at org.apache.atlas.tasks.TaskRegistry.getAll(TaskRegistry.java:155)
at org.apache.atlas.tasks.TaskManagement.getAll(TaskManagement.java:120)
at
org.apache.atlas.web.resources.AdminResource.getTaskStatus(AdminResource.java:772)
at jdk.internal.reflect.GeneratedMethodAccessor324.invoke(Unknown
Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
...
at java.base/java.lang.Thread.run(Thread.java:834)
2021-09-06 13:51:29,410 INFO - [atlas-task-0-etp985324122-24 -
c6aa9190-8454-43a4-a862-1
[jira] [Assigned] (ATLAS-4431) Random NPE when retrieving tasks
[
https://issues.apache.org/jira/browse/ATLAS-4431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Disha Talreja reassigned ATLAS-4431:
Assignee: Disha Talreja
> Random NPE when retrieving tasks
>
>
> Key: ATLAS-4431
> URL: https://issues.apache.org/jira/browse/ATLAS-4431
> Project: Atlas
> Issue Type: Bug
> Components: atlas-intg
> Reporter: Disha Talreja
> Assignee: Disha Talreja
>Priority: Major
>
> System tests fail randomly because retrieving tasks using the "admin/tasks"
> endpoint sometimes returns HTTP 500.
> Atlas logs:
> {noformat}
> 2021-09-06 13:51:28,938 INFO - [etp985324122-24:] ~ Logged into Atlas as =
> hrt_qa, by proxyUser = null
> (AtlasAuthenticationFilter$KerberosFilterChainWrapper:724)
> 2021-09-06 13:51:28,939 INFO -
> [etp985324122-24:knox:POST/api/atlas/v2/entity/bulk/classification] ~ Request
> from authenticated user: knox,
> URL=/api/atlas/v2/entity/bulk/classification?doAs=hrt_qa
> (AtlasAuthenticationFilter$KerberosFilterChainWrapper:739)
> 2021-09-06 13:51:28,969 INFO - [etp985324122-24 -
> 78a12f6b-7472-4600-a35d-bfe660358712:knox:POST/api/atlas/v2/entity/bulk/classification]
> ~ updateClassificationText: tag_iuanl: tag_iuanl date Mon Sep 06 13:51:28
> UTC 2021 hive_principal_type USER boolean true string XXJid byte -58 double
> 9.475147275250526E37 arr_string fEOKO short -29141 arr_int 781216025 float
> -3.09E38 int 1369026004 long -2808157562510626816 (EntityGraphMapper:2732)
> 2021-09-06 13:51:29,173 INFO - [etp985324122-24 -
> 78a12f6b-7472-4600-a35d-bfe660358712:knox:POST/api/atlas/v2/entity/bulk/classification]
> ~
> {"type":"CLASSIFICATION_PROPAGATION_ADD","guid":"a4ed45a8-cc15-41dd-9340-ebaca95b07f1","createdBy":"hrt_qa","createdTime":1630936288947,"updatedTime":1630936288947,"parameters":{"relationshipGuid":null,"entityGuid":"5c89916f-9dbc-4be8-afd5-9e7659bbb745","classificationVertexId":"44486696"},"attemptCount":0,"status":"PENDING"}
> (TaskExecutor$TaskLogger:170)
> 2021-09-06 13:51:29,190 INFO - [atlas-task-0-etp985324122-24 -
> c6aa9190-8454-43a4-a862-1f61393994a6:] ~ updateClassificationText: tag_iuanl:
> tag_iuanl date Mon Sep 06 13:51:28 UTC 2021 hive_principal_type USER boolean
> true string XXJid byte -58 double 9.475147275250526E37 arr_string fEOKO short
> -29141 arr_int 781216025 float -3.09E38 int 1369026004 long
> -2808157562510626816 (EntityGraphMapper:2732)
> 2021-09-06 13:51:29,193 INFO - [atlas-task-0-etp985324122-24 -
> c6aa9190-8454-43a4-a862-1f61393994a6:] ~ updateClassificationText: tag_iuanl:
> tag_iuanl date Mon Sep 06 13:51:28 UTC 2021 hive_principal_type USER boolean
> true string XXJid byte -58 double 9.475147275250526E37 arr_string fEOKO short
> -29141 arr_int 781216025 float -3.09E38 int 1369026004 long
> -2808157562510626816 (EntityGraphMapper:2732)
> 2021-09-06 13:51:29,366 ERROR - [etp985324122-594:] ~ URL not supported in HA
> mode: /api/atlas/admin/tasks (ActiveServerFilter:121)
> 2021-09-06 13:51:29,366 INFO - [etp985324122-594:] ~ Logged into Atlas as =
> hrt_qa, by proxyUser = null
> (AtlasAuthenticationFilter$KerberosFilterChainWrapper:724)
> 2021-09-06 13:51:29,366 INFO -
> [etp985324122-594:knox:GET/api/atlas/admin/tasks] ~ Request from
> authenticated user: knox, URL=/api/atlas/admin/tasks?doAs=hrt_qa
> (AtlasAuthenticationFilter$KerberosFilterChainWrapper:739)
> 2021-09-06 13:51:29,375 ERROR - [etp985324122-594 -
> 73d22020-c612-4a00-8f47-b2ebd149e976:knox:GET/api/atlas/admin/tasks] ~ Error
> handling a request: f1318c48aa5440a9 (ExceptionMapperUtil:32)
> java.lang.NullPointerException
> at
> org.apache.atlas.tasks.TaskRegistry.toAtlasTask(TaskRegistry.java:188)
> at org.apache.atlas.tasks.TaskRegistry.getAll(TaskRegistry.java:155)
> at
> org.apache.atlas.tasks.TaskManagement.getAll(TaskManagement.java:120)
> at
> org.apache.atlas.web.resources.AdminResource.getTaskStatus(AdminResource.java:772)
> at jdk.internal.reflect.GeneratedMethodAccessor324.invoke(Unknown
> Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
[jira] [Assigned] (ATLAS-4419) [Atlas: Hive Import] Suggestion suggests entity that is deleted
[ https://issues.apache.org/jira/browse/ATLAS-4419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Disha Talreja reassigned ATLAS-4419: Assignee: Disha Talreja > [Atlas: Hive Import] Suggestion suggests entity that is deleted > --- > > Key: ATLAS-4419 > URL: https://issues.apache.org/jira/browse/ATLAS-4419 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Dharshana M Krishnamoorthy > Assignee: Disha Talreja >Priority: Major > > Scenario: > # Create a database: say dharsh_db > # Create few tables : say table_1, table_2, table_3 > # Disable the hive hook > # Delete table table_1 from dharsh_db > # Run import hive command as > "/opt/cloudera/parcels/CDH/lib/atlas/hook-bin/import-hive.sh > -deleteNonExisting" > # Now search with string "dharsh_db" > Expectation: > Suggestion should suggest entities that are still active in atlas "table_2, > table_3" > Observation: > Suggestions suggests table1 of dharsh_db, which is now a deleted entity > [^Screen Recording 2021-09-09 at 4.28.34 PM.mov]w -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ATLAS-4395) Ignore/Prune configs : When the patterns are provided in upper case or mixed case , the configs don't work
[ https://issues.apache.org/jira/browse/ATLAS-4395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Disha Talreja resolved ATLAS-4395. -- Resolution: Workaround > Ignore/Prune configs : When the patterns are provided in upper case or mixed > case , the configs don't work > -- > > Key: ATLAS-4395 > URL: https://issues.apache.org/jira/browse/ATLAS-4395 > Project: Atlas > Issue Type: Bug > Reporter: Disha Talreja > Assignee: Disha Talreja >Priority: Major > Labels: doc > > In Atlas there is a known limitation on Hive Hook processing table names in > lower case letters. > When temporary/staging tables are created in uppercase/mixed case and if the > configs are provided as same, then ignore/prune configs don't work . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ATLAS-4421) [Atlas: Hive Import] When import-hive is run with incorrect input the information is not conveyed to the user
[
https://issues.apache.org/jira/browse/ATLAS-4421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Disha Talreja reassigned ATLAS-4421:
Assignee: Disha Talreja
> [Atlas: Hive Import] When import-hive is run with incorrect input the
> information is not conveyed to the user
> -
>
> Key: ATLAS-4421
> URL: https://issues.apache.org/jira/browse/ATLAS-4421
> Project: Atlas
> Issue Type: Bug
>Reporter: Dharshana M Krishnamoorthy
> Assignee: Disha Talreja
>Priority: Major
>
> {code:java}
> [root@quasar-cxwzxp-3 bin]#
> /opt/cloudera/parcels/CDH/lib/atlas/hook-bin/import-hive.sh -f /tmp/file2.txt
> ...
> Log file for import is /var/log/atlas/import-hive.log
> ...
> Hive Meta Data imported successfully!!! {code}
> In the above example, file */tmp/file2.txt* does not exists.
> The log file is not created in the expected location
> /var/log/atlas/import-hive.log which is tracked by Jira**
> Here, this gives the user an impression that the import is success while
> nothing has actually happened.
> It would be good to convey that no data was imported in such cases
> This is true even when the database name/pattern provided to -d or table
> name/pattern provided to to -t are incorrect
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (ATLAS-4369) import-hive.sh does not write logs to /var/log/atlas/import-hive.log file, instead write into ${atlas.log.file} under the directory "${atlas.log.dir}"
[
https://issues.apache.org/jira/browse/ATLAS-4369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Disha Talreja closed ATLAS-4369.
> import-hive.sh does not write logs to /var/log/atlas/import-hive.log file,
> instead write into ${atlas.log.file} under the directory "${atlas.log.dir}"
> ---
>
> Key: ATLAS-4369
> URL: https://issues.apache.org/jira/browse/ATLAS-4369
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Umesh Padashetty
>Assignee: Disha Talreja
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: ATLAS-4369.patch
>
>
> When the import scripts like
> * import-kafka.sh
> * import-hive.sh
> * import-hbase.sh
> are run, the corresponding logs are written into
> * /var/log/atlas/import-kafka.log
> * /var/log/atlas/import-hive.log
> * /var/log/atlas/import-hbase.log
> /var/log/atlas/import-kafka.log and /var/log/atlas/import-hbase.log logs are
> correctly written. But when import-hive.sh is ran,
> /var/log/atlas/import-hive.log is not created or written to
> {code:java}
> [root@ccycloud-1 atlas]# ls -ltr
> total 10988
> drwxr-xr-x 2 cstm_atlas cstm_atlas 4096 Jul 16 06:55 jstacks
> drwxr-xr-x 4 root root 4096 Jul 16 06:55 audit
> -rw-r--r-- 1 cstm_atlas cstm_atlas0 Jul 16 06:55 metric.log
> -rw-r--r-- 1 cstm_atlas cstm_atlas0 Jul 16 06:55 failed.log
> drwxr-xr-x 2 cstm_atlas cstm_atlas 4096 Jul 18 03:52 support-bundle
> -rw-r--r-- 1 root root 21573 Jul 22 09:56 import-kafka.log
> -rw-r--r-- 1 root root 159459 Jul 22 09:58 import-hbase.log
> -rw-r--r-- 1 cstm_atlas cstm_atlas19429 Jul 22 10:12
> gc-worker.log.0.current
> -rw-r--r-- 1 cstm_atlas cstm_atlas 179081 Jul 22 10:22 audit.log
> -rw-r--r-- 1 cstm_atlas cstm_atlas 10840903 Jul 22 10:23 application.log
> {code}
> whereas it creates a directory with the name ${atlas.log.dir} and a file with
> the name ${atlas.log.file} under it
> {code:java}
> [root@ccycloud-1 hook-bin]# ls -ltr
> total 44
> -rwxr-xr-x 1 root root 3892 Jul 14 13:09 import-kafka.sh
> -rwxr-xr-x 1 root root 4547 Jul 14 13:09 import-hive.sh
> -rwxr-xr-x 1 root root 4341 Jul 14 13:09 import-hbase.sh
> drwxr-sr-x 2 root root 4096 Jul 22 10:23 ${atlas.log.dir}
> -rw-r--r-- 1 root root 20094 Jul 22 10:23 derby.log {code}
> The import-hive.sh logs are written into ${atlas.log.file}
> {code:java}
> [root@ccycloud-1 ${atlas.log.dir}]# ls -ltra
> total 48
> drwxr-sr-x 3 root root 4096 Jul 22 10:23 ..
> drwxr-sr-x 2 root root 4096 Jul 22 10:23 .
> -rw-r--r-- 1 root root 40960 Jul 22 10:23 ${atlas.log.file} {code}
> The import-hive.sh logs should be written into /var/log/atlas/import-hive.log
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (ATLAS-4401) Upgrade commons-io to 2.8.0
[ https://issues.apache.org/jira/browse/ATLAS-4401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Disha Talreja closed ATLAS-4401. > Upgrade commons-io to 2.8.0 > --- > > Key: ATLAS-4401 > URL: https://issues.apache.org/jira/browse/ATLAS-4401 > Project: Atlas > Issue Type: Task > Components: atlas-core > Reporter: Disha Talreja > Assignee: Disha Talreja >Priority: Major > Attachments: ATLAS-4401.patch > > > Upgrade commons-io 2.6 to 2.8.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (ATLAS-4368) Upgrade elasticsearch version to 6.8.15
[ https://issues.apache.org/jira/browse/ATLAS-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Disha Talreja closed ATLAS-4368. > Upgrade elasticsearch version to 6.8.15 > --- > > Key: ATLAS-4368 > URL: https://issues.apache.org/jira/browse/ATLAS-4368 > Project: Atlas > Issue Type: Task > Reporter: Disha Talreja > Assignee: Disha Talreja >Priority: Minor > Attachments: ATLAS-4368.patch > > > Atlas is pulling in Elasticsearch 6.8.14. ES dependency should be removed or > upgraded to 6.8.15. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (ATLAS-4372) Upgrade nimbus-jose-jwt to 9.8.1
[ https://issues.apache.org/jira/browse/ATLAS-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Disha Talreja closed ATLAS-4372. > Upgrade nimbus-jose-jwt to 9.8.1 > > > Key: ATLAS-4372 > URL: https://issues.apache.org/jira/browse/ATLAS-4372 > Project: Atlas > Issue Type: Task > Reporter: Disha Talreja > Assignee: Disha Talreja >Priority: Minor > Attachments: ATLAS-4372.patch > > > Atlas is currently pulling in nimbus-jose-jwt 5.4. Upgrate it to 9.8.1 or > later. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (ATLAS-4420) [Atlas: Hive Import] The hive import logs are not generated
[
https://issues.apache.org/jira/browse/ATLAS-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Disha Talreja closed ATLAS-4420.
> [Atlas: Hive Import] The hive import logs are not generated
> ---
>
> Key: ATLAS-4420
> URL: https://issues.apache.org/jira/browse/ATLAS-4420
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Dharshana M Krishnamoorthy
> Assignee: Disha Talreja
>Priority: Major
>
> {code:java}
> [root@quasar-cxwzxp-3 bin]#
> /opt/cloudera/parcels/CDH/lib/atlas/hook-bin/import-hive.sh -f /tmp/file2.txt
> Using Hive configuration directory [/etc/hive/conf]
> ...
> Log file for import is /var/log/atlas/import-hive.log
> SLF4J: Class path contains multiple SLF4J bindings.
> ...
> Hive Meta Data imported successfully!!! {code}
> As seen above, the import command says that import log file will be generated
> in */var/log/atlas/import-hive.log* location
> But no such file is created
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (ATLAS-4402) Upgrade junit to 4.13.1
[ https://issues.apache.org/jira/browse/ATLAS-4402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Disha Talreja closed ATLAS-4402. > Upgrade junit to 4.13.1 > --- > > Key: ATLAS-4402 > URL: https://issues.apache.org/jira/browse/ATLAS-4402 > Project: Atlas > Issue Type: Task > Components: atlas-core >Affects Versions: 2.2.0 > Reporter: Disha Talreja > Assignee: Disha Talreja >Priority: Major > Fix For: 2.3.0 > > Attachments: ATLAS-4402.patch > > > Upgrade junit 4.13 to 4.13.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (ATLAS-4406) Ignore configs: A hive table DDL entity is created for ignored hive tables
[ https://issues.apache.org/jira/browse/ATLAS-4406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Disha Talreja closed ATLAS-4406. > Ignore configs: A hive table DDL entity is created for ignored hive tables > -- > > Key: ATLAS-4406 > URL: https://issues.apache.org/jira/browse/ATLAS-4406 > Project: Atlas > Issue Type: Bug > Components: atlas-core > Reporter: Disha Talreja > Assignee: Disha Talreja >Priority: Critical > Attachments: ATLAS-4406.patch, Ignore configs - Atlas server.png, > Ignored Hive Table-DDL entity.png > > > When hive tables are ignored, a hive table DDL entity is created in Atlas for > the same. A shell entity is also created for the hive table associated with > the DDL operation. > A DDL entity should not be created for ignored tables. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4369) import-hive.sh does not write logs to /var/log/atlas/import-hive.log file, instead write into ${atlas.log.file} under the directory "${atlas.log.dir}"
[
https://issues.apache.org/jira/browse/ATLAS-4369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Disha Talreja updated ATLAS-4369:
-
Attachment: ATLAS-4369.patch
> import-hive.sh does not write logs to /var/log/atlas/import-hive.log file,
> instead write into ${atlas.log.file} under the directory "${atlas.log.dir}"
> ---
>
> Key: ATLAS-4369
> URL: https://issues.apache.org/jira/browse/ATLAS-4369
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Umesh Padashetty
>Assignee: Disha Talreja
>Priority: Major
> Attachments: ATLAS-4369.patch
>
>
> When the import scripts like
> * import-kafka.sh
> * import-hive.sh
> * import-hbase.sh
> are run, the corresponding logs are written into
> * /var/log/atlas/import-kafka.log
> * /var/log/atlas/import-hive.log
> * /var/log/atlas/import-hbase.log
> /var/log/atlas/import-kafka.log and /var/log/atlas/import-hbase.log logs are
> correctly written. But when import-hive.sh is ran,
> /var/log/atlas/import-hive.log is not created or written to
> {code:java}
> [root@ccycloud-1 atlas]# ls -ltr
> total 10988
> drwxr-xr-x 2 cstm_atlas cstm_atlas 4096 Jul 16 06:55 jstacks
> drwxr-xr-x 4 root root 4096 Jul 16 06:55 audit
> -rw-r--r-- 1 cstm_atlas cstm_atlas0 Jul 16 06:55 metric.log
> -rw-r--r-- 1 cstm_atlas cstm_atlas0 Jul 16 06:55 failed.log
> drwxr-xr-x 2 cstm_atlas cstm_atlas 4096 Jul 18 03:52 support-bundle
> -rw-r--r-- 1 root root 21573 Jul 22 09:56 import-kafka.log
> -rw-r--r-- 1 root root 159459 Jul 22 09:58 import-hbase.log
> -rw-r--r-- 1 cstm_atlas cstm_atlas19429 Jul 22 10:12
> gc-worker.log.0.current
> -rw-r--r-- 1 cstm_atlas cstm_atlas 179081 Jul 22 10:22 audit.log
> -rw-r--r-- 1 cstm_atlas cstm_atlas 10840903 Jul 22 10:23 application.log
> {code}
> whereas it creates a directory with the name ${atlas.log.dir} and a file with
> the name ${atlas.log.file} under it
> {code:java}
> [root@ccycloud-1 hook-bin]# ls -ltr
> total 44
> -rwxr-xr-x 1 root root 3892 Jul 14 13:09 import-kafka.sh
> -rwxr-xr-x 1 root root 4547 Jul 14 13:09 import-hive.sh
> -rwxr-xr-x 1 root root 4341 Jul 14 13:09 import-hbase.sh
> drwxr-sr-x 2 root root 4096 Jul 22 10:23 ${atlas.log.dir}
> -rw-r--r-- 1 root root 20094 Jul 22 10:23 derby.log {code}
> The import-hive.sh logs are written into ${atlas.log.file}
> {code:java}
> [root@ccycloud-1 ${atlas.log.dir}]# ls -ltra
> total 48
> drwxr-sr-x 3 root root 4096 Jul 22 10:23 ..
> drwxr-sr-x 2 root root 4096 Jul 22 10:23 .
> -rw-r--r-- 1 root root 40960 Jul 22 10:23 ${atlas.log.file} {code}
> The import-hive.sh logs should be written into /var/log/atlas/import-hive.log
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4369) import-hive.sh does not write logs to /var/log/atlas/import-hive.log file, instead write into ${atlas.log.file} under the directory "${atlas.log.dir}"
[
https://issues.apache.org/jira/browse/ATLAS-4369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Disha Talreja updated ATLAS-4369:
-
Attachment: (was: ATLAS-4369.patch)
> import-hive.sh does not write logs to /var/log/atlas/import-hive.log file,
> instead write into ${atlas.log.file} under the directory "${atlas.log.dir}"
> ---
>
> Key: ATLAS-4369
> URL: https://issues.apache.org/jira/browse/ATLAS-4369
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Umesh Padashetty
>Assignee: Disha Talreja
>Priority: Major
>
> When the import scripts like
> * import-kafka.sh
> * import-hive.sh
> * import-hbase.sh
> are run, the corresponding logs are written into
> * /var/log/atlas/import-kafka.log
> * /var/log/atlas/import-hive.log
> * /var/log/atlas/import-hbase.log
> /var/log/atlas/import-kafka.log and /var/log/atlas/import-hbase.log logs are
> correctly written. But when import-hive.sh is ran,
> /var/log/atlas/import-hive.log is not created or written to
> {code:java}
> [root@ccycloud-1 atlas]# ls -ltr
> total 10988
> drwxr-xr-x 2 cstm_atlas cstm_atlas 4096 Jul 16 06:55 jstacks
> drwxr-xr-x 4 root root 4096 Jul 16 06:55 audit
> -rw-r--r-- 1 cstm_atlas cstm_atlas0 Jul 16 06:55 metric.log
> -rw-r--r-- 1 cstm_atlas cstm_atlas0 Jul 16 06:55 failed.log
> drwxr-xr-x 2 cstm_atlas cstm_atlas 4096 Jul 18 03:52 support-bundle
> -rw-r--r-- 1 root root 21573 Jul 22 09:56 import-kafka.log
> -rw-r--r-- 1 root root 159459 Jul 22 09:58 import-hbase.log
> -rw-r--r-- 1 cstm_atlas cstm_atlas19429 Jul 22 10:12
> gc-worker.log.0.current
> -rw-r--r-- 1 cstm_atlas cstm_atlas 179081 Jul 22 10:22 audit.log
> -rw-r--r-- 1 cstm_atlas cstm_atlas 10840903 Jul 22 10:23 application.log
> {code}
> whereas it creates a directory with the name ${atlas.log.dir} and a file with
> the name ${atlas.log.file} under it
> {code:java}
> [root@ccycloud-1 hook-bin]# ls -ltr
> total 44
> -rwxr-xr-x 1 root root 3892 Jul 14 13:09 import-kafka.sh
> -rwxr-xr-x 1 root root 4547 Jul 14 13:09 import-hive.sh
> -rwxr-xr-x 1 root root 4341 Jul 14 13:09 import-hbase.sh
> drwxr-sr-x 2 root root 4096 Jul 22 10:23 ${atlas.log.dir}
> -rw-r--r-- 1 root root 20094 Jul 22 10:23 derby.log {code}
> The import-hive.sh logs are written into ${atlas.log.file}
> {code:java}
> [root@ccycloud-1 ${atlas.log.dir}]# ls -ltra
> total 48
> drwxr-sr-x 3 root root 4096 Jul 22 10:23 ..
> drwxr-sr-x 2 root root 4096 Jul 22 10:23 .
> -rw-r--r-- 1 root root 40960 Jul 22 10:23 ${atlas.log.file} {code}
> The import-hive.sh logs should be written into /var/log/atlas/import-hive.log
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Review Request 73580: ATLAS-4369: import-hive.sh does not write logs to /var/log/atlas/import-hive.log file, instead write into ${atlas.log.file} under the directory "${atlas.log.dir}"
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73580/
---
(Updated Sept. 13, 2021, 5:49 p.m.)
Review request for atlas, Ashutosh Mestry, Sarath Subramanian, and Sidharth
Mishra.
Repository: atlas
Description
---
Due to the conflicting log4j jars from hive conf, import-hive.sh did not write
logs to /var/log/atlas/import-hive.log and wrote to
${atlas.log.dir}/${atlas.log.file} instead. With this patch, the logs are
written to the correct/desired location (/var/log/atlas/import-hive.log).
Diffs (updated)
-
addons/hive-bridge/src/bin/import-hive.sh 693a79bd0
Diff: https://reviews.apache.org/r/73580/diff/2/
Changes: https://reviews.apache.org/r/73580/diff/1-2/
Testing
---
Patched a cluster
Thanks,
Disha Talreja
Review Request 73580: ATLAS-4369: import-hive.sh does not write logs to /var/log/atlas/import-hive.log file, instead write into ${atlas.log.file} under the directory "${atlas.log.dir}"
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73580/
---
Review request for atlas, Ashutosh Mestry, Sarath Subramanian, and Sidharth
Mishra.
Repository: atlas
Description
---
Due to the conflicting log4j jars from hive conf, import-hive.sh did not write
logs to /var/log/atlas/import-hive.log and wrote to
${atlas.log.dir}/${atlas.log.file} instead. With this patch, the logs are
written to the correct/desired location (/var/log/atlas/import-hive.log).
Diffs
-
addons/hive-bridge/src/bin/import-hive.sh 693a79bd0
Diff: https://reviews.apache.org/r/73580/diff/1/
Testing
---
Patched a cluster
Thanks,
Disha Talreja
[jira] [Resolved] (ATLAS-4420) [Atlas: Hive Import] The hive import logs are not generated
[
https://issues.apache.org/jira/browse/ATLAS-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Disha Talreja resolved ATLAS-4420.
--
Resolution: Duplicate
> [Atlas: Hive Import] The hive import logs are not generated
> ---
>
> Key: ATLAS-4420
> URL: https://issues.apache.org/jira/browse/ATLAS-4420
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Dharshana M Krishnamoorthy
> Assignee: Disha Talreja
>Priority: Major
>
> {code:java}
> [root@quasar-cxwzxp-3 bin]#
> /opt/cloudera/parcels/CDH/lib/atlas/hook-bin/import-hive.sh -f /tmp/file2.txt
> Using Hive configuration directory [/etc/hive/conf]
> ...
> Log file for import is /var/log/atlas/import-hive.log
> SLF4J: Class path contains multiple SLF4J bindings.
> ...
> Hive Meta Data imported successfully!!! {code}
> As seen above, the import command says that import log file will be generated
> in */var/log/atlas/import-hive.log* location
> But no such file is created
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (ATLAS-4420) [Atlas: Hive Import] The hive import logs are not generated
[
https://issues.apache.org/jira/browse/ATLAS-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Disha Talreja reassigned ATLAS-4420:
Assignee: Disha Talreja
> [Atlas: Hive Import] The hive import logs are not generated
> ---
>
> Key: ATLAS-4420
> URL: https://issues.apache.org/jira/browse/ATLAS-4420
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Dharshana M Krishnamoorthy
> Assignee: Disha Talreja
>Priority: Major
>
> {code:java}
> [root@quasar-cxwzxp-3 bin]#
> /opt/cloudera/parcels/CDH/lib/atlas/hook-bin/import-hive.sh -f /tmp/file2.txt
> Using Hive configuration directory [/etc/hive/conf]
> ...
> Log file for import is /var/log/atlas/import-hive.log
> SLF4J: Class path contains multiple SLF4J bindings.
> ...
> Hive Meta Data imported successfully!!! {code}
> As seen above, the import command says that import log file will be generated
> in */var/log/atlas/import-hive.log* location
> But no such file is created
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-4395) Ignore/Prune configs : When the patterns are provided in upper case or mixed case , the configs don't work
[ https://issues.apache.org/jira/browse/ATLAS-4395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Disha Talreja updated ATLAS-4395: - Labels: doc (was: ) > Ignore/Prune configs : When the patterns are provided in upper case or mixed > case , the configs don't work > -- > > Key: ATLAS-4395 > URL: https://issues.apache.org/jira/browse/ATLAS-4395 > Project: Atlas > Issue Type: Bug > Reporter: Disha Talreja > Assignee: Disha Talreja >Priority: Major > Labels: doc > > In Atlas there is a known limitation on Hive Hook processing table names in > lower case letters. > When temporary/staging tables are created in uppercase/mixed case and if the > configs are provided as same, then ignore/prune configs don't work . -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 73552: ATLAS-4406: Ignore configs:- A hive table DDL entity is created for ignored hive tables
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73552/ --- (Updated Aug. 27, 2021, 3:54 p.m.) Review request for atlas, Ashutosh Mestry, Radhika Kundam, Sarath Subramanian, and Sidharth Mishra. Bugs: ATLAS-4406 https://issues.apache.org/jira/browse/ATLAS-4406 Repository: atlas Description (updated) --- When hive tables are ignored, a hive table DDL entity is created in Atlas for the same. A shell entity is also created for the hive table associated with the DDL operation. A DDL entity should not be created for ignored tables and this patch avoids the creation of a DDL entity for ignored tables. Ignore configs used for Atlas server: atlas.notification.consumer.preprocess.hive_table.ignore.pattern=finance..*_tmp.* Ignore configs used for Hive Metastore server: atlas.hook.hive.hive_table.ignore.pattern=finance..*_tmp.*.* Diffs - webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java bf6a623fe Diff: https://reviews.apache.org/r/73552/diff/1/ Testing --- mvn clean -DskipTests install mvn clean -DskipTests package -Pdist,embedded-hbase-solr Atlas server running. Thanks, Disha Talreja
Review Request 73552: ATLAS-4406: Ignore configs:- A hive table DDL entity is created for ignored hive tables
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73552/ --- Review request for atlas, Ashutosh Mestry, Radhika Kundam, Sarath Subramanian, and Sidharth Mishra. Bugs: ATLAS-4406 https://issues.apache.org/jira/browse/ATLAS-4406 Repository: atlas Description --- When hive tables are ignored, a hive table DDL entity is created in Atlas for the same. A shell entity is also created for the hive table associated with the DDL operation. A DDL entity should not be created for ignored tables and this patch avoids the creation of a DDL entity for ignored tables. Diffs - webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java bf6a623fe Diff: https://reviews.apache.org/r/73552/diff/1/ Testing --- mvn clean -DskipTests install mvn clean -DskipTests package -Pdist,embedded-hbase-solr Atlas server running. Thanks, Disha Talreja
[jira] [Created] (ATLAS-4406) Ignore configs: A hive table DDL entity is created for ignored hive tables
Disha Talreja created ATLAS-4406: Summary: Ignore configs: A hive table DDL entity is created for ignored hive tables Key: ATLAS-4406 URL: https://issues.apache.org/jira/browse/ATLAS-4406 Project: Atlas Issue Type: Bug Components: atlas-core Reporter: Disha Talreja Assignee: Disha Talreja Attachments: Ignore configs - Atlas server.png, Ignored Hive Table-DDL entity.png -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4406) Ignore configs: A hive table DDL entity is created for ignored hive tables
[ https://issues.apache.org/jira/browse/ATLAS-4406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Disha Talreja updated ATLAS-4406: - Description: When hive tables are ignored, a hive table DDL entity is created in Atlas for the same. A shell entity is also created for the hive table associated with the DDL operation. A DDL entity should not be created for ignored tables. > Ignore configs: A hive table DDL entity is created for ignored hive tables > -- > > Key: ATLAS-4406 > URL: https://issues.apache.org/jira/browse/ATLAS-4406 > Project: Atlas > Issue Type: Bug > Components: atlas-core > Reporter: Disha Talreja > Assignee: Disha Talreja >Priority: Critical > Attachments: Ignore configs - Atlas server.png, Ignored Hive > Table-DDL entity.png > > > When hive tables are ignored, a hive table DDL entity is created in Atlas for > the same. A shell entity is also created for the hive table associated with > the DDL operation. > A DDL entity should not be created for ignored tables. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4402) Upgrade junit to 4.13.1
[ https://issues.apache.org/jira/browse/ATLAS-4402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Disha Talreja updated ATLAS-4402: - Fix Version/s: 2.3.0 Affects Version/s: 2.2.0 > Upgrade junit to 4.13.1 > --- > > Key: ATLAS-4402 > URL: https://issues.apache.org/jira/browse/ATLAS-4402 > Project: Atlas > Issue Type: Task > Components: atlas-core >Affects Versions: 2.2.0 > Reporter: Disha Talreja > Assignee: Disha Talreja >Priority: Major > Fix For: 2.3.0 > > Attachments: ATLAS-4402.patch > > > Upgrade junit 4.13 to 4.13.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
Review Request 73548: ATLAS-4402: Upgrade junit to 4.13.1
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73548/ --- Review request for atlas, Ashutosh Mestry, Radhika Kundam, Sarath Subramanian, and Sidharth Mishra. Bugs: ATLAS-4402 https://issues.apache.org/jira/browse/ATLAS-4402 Repository: atlas Description --- This patch upgrades junit version 4.13 to 4.13.1 Diffs - pom.xml 8ae338cf2 Diff: https://reviews.apache.org/r/73548/diff/1/ Testing --- mvn clean -DskipTests install mvn clean -DskipTests package -Pdist,embedded-hbase-solr Atlas server running. Thanks, Disha Talreja
[jira] [Updated] (ATLAS-4402) Upgrade junit to 4.13.1
[ https://issues.apache.org/jira/browse/ATLAS-4402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Disha Talreja updated ATLAS-4402: - Summary: Upgrade junit to 4.13.1 (was: Upgrade to junit 4.13.1) > Upgrade junit to 4.13.1 > --- > > Key: ATLAS-4402 > URL: https://issues.apache.org/jira/browse/ATLAS-4402 > Project: Atlas > Issue Type: Task > Components: atlas-core > Reporter: Disha Talreja > Assignee: Disha Talreja >Priority: Major > > Upgrade junit 4.13 to 4.13.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
Review Request 73547: ATLAS-4401: Upgrade commons-io to 2.8.0
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73547/ --- Review request for atlas, Ashutosh Mestry, Radhika Kundam, Sarath Subramanian, and Sidharth Mishra. Bugs: ATLAS-4401 https://issues.apache.org/jira/browse/ATLAS-4401 Repository: atlas Description --- This patch upgrades commons-io 2.6 to 2.8.0 Diffs - pom.xml 8ae338cf2 Diff: https://reviews.apache.org/r/73547/diff/1/ Testing --- mvn clean -DskipTests install mvn clean -DskipTests package -Pdist,embedded-hbase-solr Atlas server running. Thanks, Disha Talreja
[jira] [Updated] (ATLAS-4401) Upgrade commons-io to 2.8.0
[ https://issues.apache.org/jira/browse/ATLAS-4401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Disha Talreja updated ATLAS-4401: - Description: Upgrade commons-io 2.6 to 2.8.0 (was: Upgrade commons-io 2.6 to 2.8) Summary: Upgrade commons-io to 2.8.0 (was: Upgrade commons-io to 2.8) > Upgrade commons-io to 2.8.0 > --- > > Key: ATLAS-4401 > URL: https://issues.apache.org/jira/browse/ATLAS-4401 > Project: Atlas > Issue Type: Task > Components: atlas-core > Reporter: Disha Talreja > Assignee: Disha Talreja >Priority: Major > > Upgrade commons-io 2.6 to 2.8.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-4402) Upgrade to junit 4.13.1
Disha Talreja created ATLAS-4402: Summary: Upgrade to junit 4.13.1 Key: ATLAS-4402 URL: https://issues.apache.org/jira/browse/ATLAS-4402 Project: Atlas Issue Type: Task Components: atlas-core Reporter: Disha Talreja Assignee: Disha Talreja Upgrade junit 4.13 to 4.13.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ATLAS-4401) Upgrade commons-io to 2.8
[ https://issues.apache.org/jira/browse/ATLAS-4401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Disha Talreja reassigned ATLAS-4401: Assignee: Disha Talreja > Upgrade commons-io to 2.8 > - > > Key: ATLAS-4401 > URL: https://issues.apache.org/jira/browse/ATLAS-4401 > Project: Atlas > Issue Type: Task > Components: atlas-core > Reporter: Disha Talreja > Assignee: Disha Talreja >Priority: Major > > Upgrade commons-io 2.6 to 2.8 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-4401) Upgrade commons-io to 2.8
Disha Talreja created ATLAS-4401: Summary: Upgrade commons-io to 2.8 Key: ATLAS-4401 URL: https://issues.apache.org/jira/browse/ATLAS-4401 Project: Atlas Issue Type: Task Components: atlas-core Reporter: Disha Talreja Upgrade commons-io 2.6 to 2.8 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-4395) Ignore/Prune configs : When the patterns are provided in upper case or mixed case , the configs don't work
Disha Talreja created ATLAS-4395: Summary: Ignore/Prune configs : When the patterns are provided in upper case or mixed case , the configs don't work Key: ATLAS-4395 URL: https://issues.apache.org/jira/browse/ATLAS-4395 Project: Atlas Issue Type: Bug Reporter: Disha Talreja In Atlas there is a known limitation on Hive Hook processing table names in lower case letters. When temporary/staging tables are created in uppercase/mixed case and if the configs are provided as same, then ignore/prune configs don't work . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ATLAS-4395) Ignore/Prune configs : When the patterns are provided in upper case or mixed case , the configs don't work
[ https://issues.apache.org/jira/browse/ATLAS-4395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Disha Talreja reassigned ATLAS-4395: Assignee: Disha Talreja > Ignore/Prune configs : When the patterns are provided in upper case or mixed > case , the configs don't work > -- > > Key: ATLAS-4395 > URL: https://issues.apache.org/jira/browse/ATLAS-4395 > Project: Atlas > Issue Type: Bug > Reporter: Disha Talreja > Assignee: Disha Talreja >Priority: Major > > In Atlas there is a known limitation on Hive Hook processing table names in > lower case letters. > When temporary/staging tables are created in uppercase/mixed case and if the > configs are provided as same, then ignore/prune configs don't work . -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [VOTE] Release Apache Atlas version 2.2.0 - rc1
Hi Sidharth,
+1 for Apache Atlas v2.2.0 rc1.
Verified the following:
- Clean Build and setup Atlas with embedded-hbase-solr
- Created entities, glossaries, terms & classifications on Atlas UI
- Verified Entity Audit data
- Verified entity search with REST API
- Verified Basic Search and Advanced Search
Thanks,
Disha
On Mon, Aug 9, 2021 at 6:55 PM Sidharth Mishra wrote:
> Atlas team,
>
> Apache Atlas 2.2.0 rc1, with the following fixes since rc0, is now
> available for vote within the dev community.
>
> ATLAS-4390 - Changed Atlas branch to release-2.2.0-rc1
> ATLAS-4391 - NoSuchFileException for {atlas.log.dir}/atlas-metrics.out
> ATLAS-4343 - [Entity Audits] 'Propagated Classification Added'
> Timestamp is < 'Entity
> Created' Timestamp
> ATLAS-4387 - UI: Propagated Classifications shows wrong count for
> entity type on Entity detail
> page
> ATLAS-4386 - Relationship search: Sorting does not work when
> relationship label is passed
>
> Links to the release artifacts are given below. Please review and vote.
>
> The vote will be open for at least 72 hours or until necessary votes
> are reached.
>
> [ ] +1 Approve
>
> [ ] +0 No opinion
>
> [ ] -1 Disapprove (and reason why)
>
> Thanks,
> Sid
>
> List of all issues addressed in this release:
>
> https://issues.apache.org/jira/issues/?jql=project%3DATLAS%20AND%20status%3DResolved%20AND%20fixVersion%3D2.2.0%20ORDER%20BY%20key%20DESC
>
> Git tag for the release:
> https://github.com/apache/atlas/tree/release-2.2.0-rc1
>
> Sources for the release:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz
>
> Source release verification:
>
> PGP Signature:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz.asc
>
> SHA512 Hash:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz.sha512
>
> MD5 Hash:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc1/apache-atlas-2.2.0-sources.tar.gz.md5
>
>
> Keys to verify the signature of the release artifacts are available
> at: https://dist.apache.org/repos/dist/release/atlas/KEYS
>
>
> New features/enhancements:
>
> - Introduced Deferred Action - Tag propagation will be handled as
> a background process (AtlasTask)
>
> - Added Re-indexing as JAVA_PATCH
>
> - Created JavaPatch to add new super types to existing entities
>
> - Supported for Business Metadata in Atlas Export API
>
> - Supported HA for admin/task API
>
> - Enhanced export service to export Terms
>
> - Added new models and addition attributes to Atlas for Apache
> Flink, GCP and ADLS Gen 2
>
> - Added support to Configure HiveServer2 Hook to send Lineage-only
> Messages
>
> - Improved and enhanced Bulk Glossary Import supporting import
> with relations
>
> - Dependency between Atlas and Kafka Zookeeper removed
>
> - Improved performance of GraphHelper's guid and status getter methods
>
> - Added new privilege - "admin-audits" for Atlas Admin Audits
> authorization
>
> - Basic search Improvements - search text with special characters,
> optimize pagination
>
> - Updated free-text search processor to support Elasticsearch
>
> - Refactored and enhanced Atlas Python Client supporting Python 2.7
>
> - DSL Search enhancements - Support glossary terms and
> relationship, can handle null attribute values, Java Tinkerpop
> Traversals instead of GremlinScriptEngine, support search by
> classification and its attribute, Caching mechanism for translated dsl
> queries
>
> - Enhanced Export/Import Service to conditionally Support
> Simultaneous Operations
>
> - Enhanced NotificationHookConsumer for Large Message Processing
>
> - Enhanced Atlas authorization for Add/Update/Remove
> classification on entities
>
> - Provided option to add mandatory attribute to existing entity
> definition
>
> - Fixed Atlas UI to load faster in case of large number of
> classifications & entities
>
> - Updated versions of dependent libraries/components: JanusGraph,
> elasticsearch, JQuery, Http core, Http Client, slf4j, log4j, ant,
> gremlin, Solr, groovy, netty, Kafka
>
[jira] [Assigned] (ATLAS-4369) import-hive.sh does not write logs to /var/log/atlas/import-hive.log file, instead write into ${atlas.log.file} under the directory "${atlas.log.dir}"
[
https://issues.apache.org/jira/browse/ATLAS-4369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Disha Talreja reassigned ATLAS-4369:
Assignee: Disha Talreja
> import-hive.sh does not write logs to /var/log/atlas/import-hive.log file,
> instead write into ${atlas.log.file} under the directory "${atlas.log.dir}"
> ---
>
> Key: ATLAS-4369
> URL: https://issues.apache.org/jira/browse/ATLAS-4369
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Umesh Padashetty
>Assignee: Disha Talreja
>Priority: Major
>
> When the import scripts like
> * import-kafka.sh
> * import-hive.sh
> * import-hbase.sh
> are run, the corresponding logs are written into
> * /var/log/atlas/import-kafka.log
> * /var/log/atlas/import-hive.log
> * /var/log/atlas/import-hbase.log
> /var/log/atlas/import-kafka.log and /var/log/atlas/import-hbase.log logs are
> correctly written. But when import-hive.sh is ran,
> /var/log/atlas/import-hive.log is not created or written to
> {code:java}
> [root@ccycloud-1 atlas]# ls -ltr
> total 10988
> drwxr-xr-x 2 cstm_atlas cstm_atlas 4096 Jul 16 06:55 jstacks
> drwxr-xr-x 4 root root 4096 Jul 16 06:55 audit
> -rw-r--r-- 1 cstm_atlas cstm_atlas0 Jul 16 06:55 metric.log
> -rw-r--r-- 1 cstm_atlas cstm_atlas0 Jul 16 06:55 failed.log
> drwxr-xr-x 2 cstm_atlas cstm_atlas 4096 Jul 18 03:52 support-bundle
> -rw-r--r-- 1 root root 21573 Jul 22 09:56 import-kafka.log
> -rw-r--r-- 1 root root 159459 Jul 22 09:58 import-hbase.log
> -rw-r--r-- 1 cstm_atlas cstm_atlas19429 Jul 22 10:12
> gc-worker.log.0.current
> -rw-r--r-- 1 cstm_atlas cstm_atlas 179081 Jul 22 10:22 audit.log
> -rw-r--r-- 1 cstm_atlas cstm_atlas 10840903 Jul 22 10:23 application.log
> {code}
> whereas it creates a directory with the name ${atlas.log.dir} and a file with
> the name ${atlas.log.file} under it
> {code:java}
> [root@ccycloud-1 hook-bin]# ls -ltr
> total 44
> -rwxr-xr-x 1 root root 3892 Jul 14 13:09 import-kafka.sh
> -rwxr-xr-x 1 root root 4547 Jul 14 13:09 import-hive.sh
> -rwxr-xr-x 1 root root 4341 Jul 14 13:09 import-hbase.sh
> drwxr-sr-x 2 root root 4096 Jul 22 10:23 ${atlas.log.dir}
> -rw-r--r-- 1 root root 20094 Jul 22 10:23 derby.log {code}
> The import-hive.sh logs are written into ${atlas.log.file}
> {code:java}
> [root@ccycloud-1 ${atlas.log.dir}]# ls -ltra
> total 48
> drwxr-sr-x 3 root root 4096 Jul 22 10:23 ..
> drwxr-sr-x 2 root root 4096 Jul 22 10:23 .
> -rw-r--r-- 1 root root 40960 Jul 22 10:23 ${atlas.log.file} {code}
> The import-hive.sh logs should be written into /var/log/atlas/import-hive.log
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (ATLAS-4372) Upgrade nimbus-jose-jwt to 9.8.1
[ https://issues.apache.org/jira/browse/ATLAS-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Disha Talreja reassigned ATLAS-4372: Assignee: Disha Talreja > Upgrade nimbus-jose-jwt to 9.8.1 > > > Key: ATLAS-4372 > URL: https://issues.apache.org/jira/browse/ATLAS-4372 > Project: Atlas > Issue Type: Task > Reporter: Disha Talreja > Assignee: Disha Talreja >Priority: Minor > Attachments: ATLAS-4372.patch > > > Atlas is currently pulling in nimbus-jose-jwt 5.4. Upgrate it to 9.8.1 or > later. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ATLAS-4368) Upgrade elasticsearch version to 6.8.15
[ https://issues.apache.org/jira/browse/ATLAS-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Disha Talreja reassigned ATLAS-4368: Assignee: Disha Talreja > Upgrade elasticsearch version to 6.8.15 > --- > > Key: ATLAS-4368 > URL: https://issues.apache.org/jira/browse/ATLAS-4368 > Project: Atlas > Issue Type: Task > Reporter: Disha Talreja > Assignee: Disha Talreja >Priority: Minor > Attachments: ATLAS-4368.patch > > > Atlas is pulling in Elasticsearch 6.8.14. ES dependency should be removed or > upgraded to 6.8.15. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Request for Contributor Access
AtlasDev, This is to follow up on my previous email. I would like to be added as a contributor to Apache Atlas, my Apache Jira username is dishatalreja. Please let me know if you need any other information from my end. Thanks, Disha Talreja On Fri, Jul 23, 2021 at 2:47 PM disha talreja wrote: > AtlasDev, > > Please add me as a contributor to Apache Atlas. > Apache Jira Username: dishatalreja > > Thanks and Regards, > Disha Talreja > [image: dishatalreja.com] <https://www.dishatalreja.com/> > >
Re: Planning for Apache Atlas 2.2.0 release
Thanks for volunteering, Sidharth. +1 for the Apache Atlas 2.2.0 release. Regards, Disha On Tue, Jul 27, 2021 at 3:11 PM Sidharth Mishra wrote: > Dear Atlas community, > > Since Apache Atlas 2.1 was released, there are many new features, > improvements and fixes. Here are few features/enhancements since last > release, Apache Atlas 2.1: > > - Introduced Deferred Action - Tag propagation will be handled as > a background process (AtlasTask) > > - Added Re-indexing as JAVA_PATCH > > - Created JavaPatch to add new super types to existing entities > > - Supported for Business Metadata in Atlas Export API > > - Supported HA for admin/task API > > - Enhanced export service to export Terms > > - Added new models and addition attributes to Atlas for Apache > Flink, GCP and ADLS Gen 2 > > - Added support to Configure HiveServer2 Hook to send Lineage-only > Messages > > - Improved and enhanced Bulk Glossary Import supporting import > with relations > > - Dependency between Atlas and Kafka Zookeeper removed > > - Improved performance of GraphHelper's guid and status getter methods > > - Added new privilege - "admin-audits" for Atlas Admin Audits > authorization > > - Basic search Improvements - search text with special characters, > optimize pagination > > - Updated free-text search processor to support Elasticsearch > > - Refactored and enhanced Atlas Python Client supporting Python 2.7 > > - DSL Search enhancements - Support glossary terms and > relationship, can handle null attribute values, Java Tinkerpop > Traversals instead of GremlinScriptEngine, support search by > classification and its attribute, Caching mechanism for translated dsl > queries > > - Enhanced Export/Import Service to conditionally Support > Simultaneous Operations > > - Enhanced NotificationHookConsumer for Large Message Processing > > - Enhanced Atlas authorization for Add/Update/Remove > classification on entities > > - Provided option to add mandatory attribute to existing entity > definition > > - Fixed Atlas UI to load faster in case of large number of > classifications & entities > > - Updated versions of dependent libraries/components: JanusGraph, > elasticsearch, JQuery, Http core, Http Client, slf4j, log4j, ant, > gremlin, Solr, groovy, netty, Kafka > > There are around 336 plus commits on branch 2.0 and 205 resolved > issues with fix version 2.2.0 after 2.1.0 release. > With these significant improvements in place, it is time for the next > release of Apache Atlas! > > I propose to release Apache Atlas 2.2 by early next month. Please > review and send your comments. > > Regards, > Sidharth >
Review Request 73492: ATLAS-4372: Upgrade nimbus-jose-jwt to 9.8.1
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73492/ --- Review request for atlas, Radhika Kundam and Sarath Subramanian. Repository: atlas Description --- Atlas is currently pulling in nimbus-jose-jwt 5.4. This patch upgrades it to 9.8.1. Diffs - webapp/pom.xml ccbcc3141 Diff: https://reviews.apache.org/r/73492/diff/1/ Testing --- mvn clean -DskipTests install mvn clean -DskipTests package -Pdist,embedded-hbase-solr Atlas server running. Thanks, Disha Talreja
Re: Review Request 73478: ATLAS-4368: Upgrade elasticsearch version to 6.8.15
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73478/ --- (Updated July 27, 2021, 8:45 p.m.) Review request for atlas, Ashutosh Mestry, Radhika Kundam, Sarath Subramanian, and Sidharth Mishra. Bugs: ATLAS-4368 https://issues.apache.org/jira/browse/ATLAS-4368 Repository: atlas Description --- Currently Atlas is pulling in Elasticsearch 6.8.14 this patch upgrades Elasticsearch version to 6.8.15 Diffs - pom.xml 19db43146 Diff: https://reviews.apache.org/r/73478/diff/1/ Testing (updated) --- mvn clean -DskipTests install mvn clean -DskipTests package -Pdist,embedded-hbase-solr Atlas server running. PC Build: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/767/ Thanks, Disha Talreja
[jira] [Created] (ATLAS-4372) Upgrade nimbus-jose-jwt to 9.8.1
Disha Talreja created ATLAS-4372: Summary: Upgrade nimbus-jose-jwt to 9.8.1 Key: ATLAS-4372 URL: https://issues.apache.org/jira/browse/ATLAS-4372 Project: Atlas Issue Type: Task Reporter: Disha Talreja Atlas is currently pulling in nimbus-jose-jwt 5.4. Upgrate it to 9.8.1 or later. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4368) Upgrade elasticsearch version to 6.8.15
[ https://issues.apache.org/jira/browse/ATLAS-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Disha Talreja updated ATLAS-4368: - Description: Atlas is pulling in Elasticsearch 6.8.14. ES dependency should be removed or upgraded to 6.8.15. > Upgrade elasticsearch version to 6.8.15 > --- > > Key: ATLAS-4368 > URL: https://issues.apache.org/jira/browse/ATLAS-4368 > Project: Atlas > Issue Type: Task > Reporter: Disha Talreja >Priority: Minor > > Atlas is pulling in Elasticsearch 6.8.14. ES dependency should be removed or > upgraded to 6.8.15. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Request for Contributor Access
AtlasDev, Please add me as a contributor to Apache Atlas. Apache Jira Username: dishatalreja Thanks and Regards, Disha Talreja [image: dishatalreja.com] <https://www.dishatalreja.com/>
Review Request 73478: ATLAS-4368: Upgrade elasticsearch version to 6.8.15
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73478/ --- Review request for atlas, Ashutosh Mestry, Radhika Kundam, Sarath Subramanian, and Sidharth Mishra. Bugs: ATLAS-4368 https://issues.apache.org/jira/browse/ATLAS-4368 Repository: atlas Description --- Currently Atlas is pulling in Elasticsearch 6.8.14 this patch upgrades Elasticsearch version to 6.8.15 Diffs - pom.xml 19db43146 Diff: https://reviews.apache.org/r/73478/diff/1/ Testing --- mvn clean -DskipTests install mvn clean -DskipTests package -Pdist,embedded-hbase-solr Atlas server running. Thanks, Disha Talreja
[jira] [Created] (ATLAS-4368) Upgrade elasticsearch version to 6.8.15
Disha Talreja created ATLAS-4368: Summary: Upgrade elasticsearch version to 6.8.15 Key: ATLAS-4368 URL: https://issues.apache.org/jira/browse/ATLAS-4368 Project: Atlas Issue Type: Task Reporter: Disha Talreja -- This message was sent by Atlassian Jira (v8.3.4#803005)
