Review Request 74204: ATLAS-3820: fix import-kafka.sh failures
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74204/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nikhil Bonte, Nixon Rodrigues, Pinal Shah, Radhika Kundam, Ramesh Mani, Sarath Subramanian, Sharmadha Sainath, Sidharth Mishra, and Umesh Padashetty. Bugs: ATLAS-3820 https://issues.apache.org/jira/browse/ATLAS-3820 Repository: atlas Description --- - added missing libraries to Kafka bridge package - updated docker setup to include Kafka bridge Diffs - addons/kafka-bridge/pom.xml 9ce22a757 addons/kafka-bridge/src/bin/import-kafka.sh ada3b4ae2 addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java 8be2fca3a dev-support/atlas-docker/.dockerignore d4a3db26b dev-support/atlas-docker/Dockerfile.atlas-kafka e06d62845 dev-support/atlas-docker/scripts/atlas-build.sh e53ebbba9 dev-support/atlas-docker/scripts/atlas-kafka-application.properties PRE-CREATION Diff: https://reviews.apache.org/r/74204/diff/1/ Testing --- - verified that import-kafka.sh sucessfully added Kafka topics to Atlas Thanks, Madhan Neethiraj
[jira] [Assigned] (ATLAS-3820) run import-kafka.sh throws Expection
[ https://issues.apache.org/jira/browse/ATLAS-3820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj reassigned ATLAS-3820: --- Assignee: Madhan Neethiraj > run import-kafka.sh throws Expection > > > Key: ATLAS-3820 > URL: https://issues.apache.org/jira/browse/ATLAS-3820 > Project: Atlas > Issue Type: Bug > Components: kafka-integration >Affects Versions: 1.0.0 > Environment: Kafka version: 1.1.0 >Reporter: fa >Assignee: Madhan Neethiraj >Priority: Major > > maven build command: mvn clean -DskipTests package -Pdist,embedded-hbase-solr > I run `bin/atlas_stat.py`, and I can see the web-ui. > Then I run > apache-atlas-1.0.0-kafka-hook/apache-atlas-kafka-hook-1.0.0/hook-bin/import-kafka.sh. > stack trace > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/commons/lang/exception/NestableRuntimeExceptionException in thread > "main" java.lang.NoClassDefFoundError: > org/apache/commons/lang/exception/NestableRuntimeException at > java.lang.ClassLoader.defineClass1(Native Method) at > java.lang.ClassLoader.defineClass(ClassLoader.java:763) at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at > java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at > java.net.URLClassLoader.access$100(URLClassLoader.java:74) at > java.net.URLClassLoader$1.run(URLClassLoader.java:369) at > java.net.URLClassLoader$1.run(URLClassLoader.java:363) at > java.security.AccessController.doPrivileged(Native Method) at > java.net.URLClassLoader.findClass(URLClassLoader.java:362) at > java.lang.ClassLoader.loadClass(ClassLoader.java:424) at > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at > java.lang.ClassLoader.loadClass(ClassLoader.java:357) at > java.lang.ClassLoader.defineClass1(Native Method) at > java.lang.ClassLoader.defineClass(ClassLoader.java:763) at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at > java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at > java.net.URLClassLoader.access$100(URLClassLoader.java:74) at > java.net.URLClassLoader$1.run(URLClassLoader.java:369) at > java.net.URLClassLoader$1.run(URLClassLoader.java:363) at > java.security.AccessController.doPrivileged(Native Method) at > java.net.URLClassLoader.findClass(URLClassLoader.java:362) at > java.lang.ClassLoader.loadClass(ClassLoader.java:424) at > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at > java.lang.ClassLoader.loadClass(ClassLoader.java:357) at > org.apache.atlas.kafka.bridge.KafkaBridge.main(KafkaBridge.java:100)Caused > by: java.lang.ClassNotFoundException: > org.apache.commons.lang.exception.NestableRuntimeException at > java.net.URLClassLoader.findClass(URLClassLoader.java:382) at > java.lang.ClassLoader.loadClass(ClassLoader.java:424) at > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at > java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 25 more -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74195: ATLAS-3403 : Term assigned to historical entity is not visible in Search results page.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74195/
---
(Updated Nov. 16, 2022, 5:59 a.m.)
Review request for atlas, Jayendra Parab and Pinal Shah.
Bugs: ATLAS-3403
https://issues.apache.org/jira/browse/ATLAS-3403
Repository: atlas
Description
---
Let us consider that a classification and a term has been assigned to an
entity.
If that entity is deleted, and then when the "Show Historical Entities" filter
is enabled on the search results page for that entity, the classification is
displayed but not the term.
This patch handles above issue
Diffs
-
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
0ed418330
Diff: https://reviews.apache.org/r/74195/diff/1/
Testing (updated)
---
mvn clean install
Entities ,classification,terms,glossaries getting created successfully
PC successful
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1284/console
Created an Entity of type hdfs_path as "abc" added classification "class1" and
assigned a term as well "term1"
then deleted the entity when we search this entity using
http://localhost:21000/api/atlas/v2/search/basic
without the fix we cant see "term1" in response under "meanings" tab because
there is a check in the code where if edge is deleted we then term will not be
assigned to the deleted entity hence we couldnt see it under "show historical
data"
Below json for output after fixing issue we can see term getting added in
MeaningNames and meanings:
http://localhost:21000/api/atlas/v2/search/basic
{"queryType":"BASIC","searchParameters":{"typeName":"hdfs_path","excludeDeletedEntities":false,"includeClassificationAttributes":true,"includeSubTypes":true,"includeSubClassifications":true,"excludeHeaderAttributes":false,"limit":25,"offset":0,"attributes":[]},"entities":[{"typeName":"hdfs_path","attributes":{"owner":"","createTime":166784580,"qualifiedName":"abc","name":"abc","description":""},"guid":"22f365a3-5d7c-49d0-b2dc-324adf77d3fe","status":"DELETED","displayText":"abc","classificationNames":["class1"],"classifications":[{"typeName":"class1","entityGuid":"22f365a3-5d7c-49d0-b2dc-324adf77d3fe","entityStatus":"DELETED","propagate":true,"validityPeriods":[],"removePropagationsOnEntityDelete":false}],"meaningNames":["term1"],"meanings":[{"termGuid":"c983b9b9-5b0a-409b-9a30-c0328cd1b3ad","relationGuid":"29ddea1a-adbf-4e48-9c85-72b8a97e10ac","displayText":"term1","confidence":0,"qualifiedName":"term1@gloss1"}],"isIncomplete":false,"labels":[]}],"approximateCount":1}
Thanks,
chaitali
Re: Review Request 74195: ATLAS-3403 : Term assigned to historical entity is not visible in Search results page.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74195/
---
(Updated Nov. 16, 2022, 5:56 a.m.)
Review request for atlas, Jayendra Parab and Pinal Shah.
Bugs: ATLAS-3403
https://issues.apache.org/jira/browse/ATLAS-3403
Repository: atlas
Description
---
Let us consider that a classification and a term has been assigned to an
entity.
If that entity is deleted, and then when the "Show Historical Entities" filter
is enabled on the search results page for that entity, the classification is
displayed but not the term.
This patch handles above issue
Diffs
-
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
0ed418330
Diff: https://reviews.apache.org/r/74195/diff/1/
Testing (updated)
---
mvn clean install
Sanity testing
PC successful
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1284/console
Created an Entity of type hdfs_path as "abc" added classification "class1" and
assigned a term as well "term1"
then deleted the entity when we search this entity using
http://localhost:21000/api/atlas/v2/search/basic
without the fix we cant see "term1" in response under "meanings" tab because
there is a check in the code where if edge is deleted we then term will not be
assigned to the deleted entity hence we couldnt see it under "show historical
data"
Below json for output after fixing issue we can see term getting added in
MeaningNames and meanings:
http://localhost:21000/api/atlas/v2/search/basic
{"queryType":"BASIC","searchParameters":{"typeName":"hdfs_path","excludeDeletedEntities":false,"includeClassificationAttributes":true,"includeSubTypes":true,"includeSubClassifications":true,"excludeHeaderAttributes":false,"limit":25,"offset":0,"attributes":[]},"entities":[{"typeName":"hdfs_path","attributes":{"owner":"","createTime":166784580,"qualifiedName":"abc","name":"abc","description":""},"guid":"22f365a3-5d7c-49d0-b2dc-324adf77d3fe","status":"DELETED","displayText":"abc","classificationNames":["class1"],"classifications":[{"typeName":"class1","entityGuid":"22f365a3-5d7c-49d0-b2dc-324adf77d3fe","entityStatus":"DELETED","propagate":true,"validityPeriods":[],"removePropagationsOnEntityDelete":false}],"meaningNames":["term1"],"meanings":[{"termGuid":"c983b9b9-5b0a-409b-9a30-c0328cd1b3ad","relationGuid":"29ddea1a-adbf-4e48-9c85-72b8a97e10ac","displayText":"term1","confidence":0,"qualifiedName":"term1@gloss1"}],"isIncomplete":false,"labels":[]}],"approximateCount":1}
Thanks,
chaitali
Review Request 74203: ATLAS-4241: fix import-hbase errors
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74203/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nikhil Bonte, Nixon Rodrigues, Pinal Shah, Radhika Kundam, Ramesh Mani, Sarath Subramanian, and Sidharth Mishra. Bugs: ATLAS-4241 https://issues.apache.org/jira/browse/ATLAS-4241 Repository: atlas Description --- - fixed classpath for import-hbase to exclude couple of libraries present in HBase lib directory - added following libraries in HBase plugin impl: commons-collections, commons-logging, jackson-jaxrs-json-provider, jackson-jaxrs-base, jersey-bundle Diffs - addons/hbase-bridge/pom.xml f51a7d6f1 addons/hbase-bridge/src/bin/import-hbase.sh a343036fa Diff: https://reviews.apache.org/r/74203/diff/1/ Testing --- verified the following using Docker setup - HBase hook successfully reports changes to Atlas server - import-hbase utility successfully imports namespaces and tables to Atlas server Thanks, Madhan Neethiraj
[jira] [Assigned] (ATLAS-4241) import-hbase.sh get map error
[ https://issues.apache.org/jira/browse/ATLAS-4241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj reassigned ATLAS-4241: --- Assignee: Madhan Neethiraj (was: Radhika Kundam) > import-hbase.sh get map error > - > > Key: ATLAS-4241 > URL: https://issues.apache.org/jira/browse/ATLAS-4241 > Project: Atlas > Issue Type: Bug > Components: hbase-integration >Affects Versions: 2.1.0 >Reporter: junsionzhang >Assignee: Madhan Neethiraj >Priority: Major > > when run import-hbase.sh ,errors is as follows: > 2021-04-12 14:34:25,865 ERROR - [main:] ~ ImportHBaseEntities failed > (HBaseBridge:197) > org.apache.atlas.AtlasException: Failed to load application properties > at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:147) > at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:100) > at org.apache.atlas.hbase.bridge.HBaseBridge.main(HBaseBridge.java:139) > Caused by: org.apache.commons.configuration.ConversionException: > 'atlas.graph.index.search.solr.wait-searcher' doesn't map to a List object: > true, a java.lang.Boolean > at > org.apache.commons.configuration.AbstractConfiguration.getList(AbstractConfiguration.java:1144) > at > org.apache.commons.configuration.AbstractConfiguration.getList(AbstractConfiguration.java:1109) > at > org.apache.commons.configuration.AbstractConfiguration.interpolatedConfiguration(AbstractConfiguration.java:1274) > at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:142) > ... 2 more > > It seems the common-configuration does not work correctly . the hbasebridge > use common-configuration 1.10 ,but the version also exists. could somebody > upgrade from 1.x to 2.x > > https://commons.apache.org/proper/commons-configuration/userguide/upgradeto2_0.html -- This message was sent by Atlassian Jira (v8.20.10#820010)
Review Request 74202: ATLAS-4713: docker setup updates to be in sync with component versions in project pom.xml
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74202/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, Mandar Ambawane, Nikhil Bonte, Nixon Rodrigues, Pinal Shah, Radhika Kundam, Sarath Subramanian, and Sidharth Mishra. Bugs: ATLAS-4713 https://issues.apache.org/jira/browse/ATLAS-4713 Repository: atlas Description --- - updated version of Kafka, Hive-hadoop - fixed incorrect Atlas URL in hook configuration setup Diffs - dev-support/atlas-docker/.env 19175c2ed dev-support/atlas-docker/scripts/atlas-hbase-application.properties 8c5b9e088 dev-support/atlas-docker/scripts/atlas-hive-application.properties 8c5b9e088 Diff: https://reviews.apache.org/r/74202/diff/1/ Testing --- - verified that Atlas docker setup builds successfully - performed sanity testing of Atlas, Hive hook, HBase hook Thanks, Madhan Neethiraj
[jira] [Created] (ATLAS-4713) docker setup - update component versions
Madhan Neethiraj created ATLAS-4713: --- Summary: docker setup - update component versions Key: ATLAS-4713 URL: https://issues.apache.org/jira/browse/ATLAS-4713 Project: Atlas Issue Type: Bug Components: atlas-docker Reporter: Madhan Neethiraj Assignee: Madhan Neethiraj Docker setup needs updates to be in sync with component versions in project pom.xml file. For example: * ATLAS_VERSION=2.2.0 should be 2.3.0-SNAPSHOT * KAFKA_VERSION=2.5.0 should be 2.8.1 * HIVE_HADOOP_VERSION=3.1.0 should be 3.1.1 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Review Request 74201: ATLAS-4445: fix import-hive.sh failure due to NoClassDefFoundError: javax/ws/rs/core/Link$Builder
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74201/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, keval bhatt, Nikhil Bonte, Nixon Rodrigues, Pinal Shah, Radhika Kundam, Ramesh Mani, Sarath Subramanian, and Sidharth Mishra. Bugs: ATLAS-4445 https://issues.apache.org/jira/browse/ATLAS-4445 Repository: atlas Description --- - updated hive version from 3.1.0 to 3.1.2 - removed packaging jsr311-api in Hive hook Diffs - addons/hive-bridge/pom.xml 856e5d95d addons/hive-bridge/src/bin/import-hive.sh ebe6976f0 pom.xml ca221fe29 Diff: https://reviews.apache.org/r/74201/diff/1/ Testing --- - verified import-hive.sh in docker setup - all existing tests pass successfully - pre-commit tests run: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1286/ Thanks, Madhan Neethiraj
[jira] [Assigned] (ATLAS-4445) unable to run import-hive.sh successfully
[
https://issues.apache.org/jira/browse/ATLAS-4445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj reassigned ATLAS-4445:
---
Assignee: Madhan Neethiraj
> unable to run import-hive.sh successfully
> -
>
> Key: ATLAS-4445
> URL: https://issues.apache.org/jira/browse/ATLAS-4445
> Project: Atlas
> Issue Type: Bug
> Components: hive-integration
>Affects Versions: 2.2.0
> Environment: RHEL 7.x
>Reporter: Tim Hunter
>Assignee: Madhan Neethiraj
>Priority: Major
>
> Same configuration this worked in 2.1.0, unable to figure out why it doesn't
> in 2.2.0
>
> {{$ ./import-hive.sh $ ./import-hive.sh Using Hive configuration directory
> [/usr/local/hive/conf]Log file for import is
> /var/log/atlas/import-hive.logSLF4J: Class path contains multiple SLF4J
> bindings.SLF4J: Found binding in
> [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> Found binding in
> [jar:file:/home/ec2-user/hadoop-3.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J:
> See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]2021-10-05T12:57:32,810 INFO
> [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - delete non existing
> flag : false 2021-10-05T12:57:32,841 INFO [main]
> org.apache.atlas.ApplicationProperties - Looking for
> atlas-application.properties in classpath2021-10-05T12:57:32,843 INFO [main]
> org.apache.atlas.ApplicationProperties - Loading atlas-application.properties
> from
> file:/opt/atlas/apache-atlas-2.2.0/conf/atlas-application.properties2021-10-05T12:57:32,903
> INFO [main] org.apache.atlas.ApplicationProperties - Using graphdb backend
> 'janus'2021-10-05T12:57:32,903 INFO [main]
> org.apache.atlas.ApplicationProperties - Using storage backend
> 'hbase2'2021-10-05T12:57:32,903 INFO [main]
> org.apache.atlas.ApplicationProperties - Using index backend
> 'solr'2021-10-05T12:57:32,903 INFO [main]
> org.apache.atlas.ApplicationProperties - Atlas is running in MODE:
> PROD.2021-10-05T12:57:32,904 INFO [main]
> org.apache.atlas.ApplicationProperties - Setting solr.wait-searcher property
> 'false'2021-10-05T12:57:32,904 INFO [main]
> org.apache.atlas.ApplicationProperties - Setting index.search.map-name
> property 'false'2021-10-05T12:57:32,904 INFO [main]
> org.apache.atlas.ApplicationProperties - Setting
> atlas.graph.index.search.max-result-set-size = 1502021-10-05T12:57:32,904
> INFO [main] org.apache.atlas.ApplicationProperties - Setting
> atlas.graph.index.search.solr.wait-searcher = false2021-10-05T12:57:32,905
> INFO [main] org.apache.atlas.ApplicationProperties - Property (set to
> default) atlas.graph.cache.db-cache = true2021-10-05T12:57:32,905 INFO [main]
> org.apache.atlas.ApplicationProperties - Property (set to default)
> atlas.graph.cache.db-cache-clean-wait = 202021-10-05T12:57:32,905 INFO [main]
> org.apache.atlas.ApplicationProperties - Property (set to default)
> atlas.graph.cache.db-cache-size = 0.52021-10-05T12:57:32,905 INFO [main]
> org.apache.atlas.ApplicationProperties - Property (set to default)
> atlas.graph.cache.tx-cache-size = 150002021-10-05T12:57:32,905 INFO [main]
> org.apache.atlas.ApplicationProperties - Property (set to default)
> atlas.graph.cache.tx-dirty-size = 120Enter username for atlas :- adminEnter
> password for atlas :- Exception in thread "main"
> java.lang.NoClassDefFoundError: javax/ws/rs/core/Link$Builder at
> java.lang.Class.getDeclaredConstructors0(Native Method) at
> java.lang.Class.privateGetDeclaredConstructors(Class.java:2671) at
> java.lang.Class.getConstructor0(Class.java:3075) at
> java.lang.Class.newInstance(Class.java:412) at
> javax.ws.rs.ext.FactoryFinder.newInstance(FactoryFinder.java:65) at
> javax.ws.rs.ext.FactoryFinder.find(FactoryFinder.java:117) at
> javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:105) at
> javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:91) at
> javax.ws.rs.core.MediaType.(MediaType.java:44) at
> com.sun.jersey.core.header.MediaTypes.(MediaTypes.java:65) at
> com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:182)
> at
> com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:175)
> at
> com.sun.jersey.core.spi.factory.MessageBodyFactory.init(MessageBodyFactory.java:162)
> at com.sun.jersey.api.client.Client.init(Client.java:343) at
> com.sun.jersey.api.client.Client.access$000(Client.java:119) at
> com.sun.jersey.api.client.Client$1.f(Client.java:192) at
> com.sun.jersey.api.client.Client$1.f(Client.java:188) at
>
[jira] [Updated] (ATLAS-3403) Term assigned to historical entity is not visible in Search results page.
[ https://issues.apache.org/jira/browse/ATLAS-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chaitali borole updated ATLAS-3403: --- Fix Version/s: 3.0.0 > Term assigned to historical entity is not visible in Search results page. > - > > Key: ATLAS-3403 > URL: https://issues.apache.org/jira/browse/ATLAS-3403 > Project: Atlas > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Rahul Kurup >Assignee: chaitali borole >Priority: Minor > Fix For: 3.0.0 > > > Let us consider that a classification and a term has been assigned to an > entity. > If that entity is deleted, and then when the "Show Historical Entities" > filter is enabled on the search results page for that entity, the > classification is displayed but not the term. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-3403) Term assigned to historical entity is not visible in Search results page.
[ https://issues.apache.org/jira/browse/ATLAS-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chaitali borole updated ATLAS-3403: --- Affects Version/s: 3.0.0 > Term assigned to historical entity is not visible in Search results page. > - > > Key: ATLAS-3403 > URL: https://issues.apache.org/jira/browse/ATLAS-3403 > Project: Atlas > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Rahul Kurup >Assignee: chaitali borole >Priority: Minor > > Let us consider that a classification and a term has been assigned to an > entity. > If that entity is deleted, and then when the "Show Historical Entities" > filter is enabled on the search results page for that entity, the > classification is displayed but not the term. -- This message was sent by Atlassian Jira (v8.20.10#820010)
