Re: Review Request 74077: Refactor Atlas webapp module to remove Kafka core dependency
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74077/#review224593 --- Ship it! Ship It! - Ashutosh Mestry On Aug. 2, 2022, 1:21 p.m., Patrik Márton wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74077/ > --- > > (Updated Aug. 2, 2022, 1:21 p.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Nixon > Rodrigues. > > > Bugs: ATLAS-4619 > https://issues.apache.org/jira/browse/ATLAS-4619 > > > Repository: atlas > > > Description > --- > > Goal is to break the strong coupling between Atlas components and Kafka. > These dependencies include using server side libraries of Kafka (this couples > the Scala version and other non-public interfaces of Kafka). Any code using > server side libraries of Kafka should be refactored. > The webapp module uses ShutdownAbleThread from the core kafka library. With > this commit, it is changed to Thread, as the try-catch-finally block inside > the run() method should solve the issue with closing the consumer. > > > Diffs > - > > > webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java > 49c504f9f > > webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java > fdfc2560d > > > Diff: https://reviews.apache.org/r/74077/diff/2/ > > > Testing > --- > > - Unit tests are green > - Atlas server is healthy on manually provisoned cluster > - Tested the functionality with Hive Hook, no ConcurrentModificationException > was present in the logs after stopping Atlas server. > > > Thanks, > > Patrik Márton > >
[jira] [Created] (ATLAS-4654) 'atlas.graph.index.search.max-result-set-size' doesn't map to a List object: 150
Omar Tougui created ATLAS-4654:
--
Summary: 'atlas.graph.index.search.max-result-set-size' doesn't
map to a List object: 150
Key: ATLAS-4654
URL: https://issues.apache.org/jira/browse/ATLAS-4654
Project: Atlas
Issue Type: Bug
Components: atlas-intg
Affects Versions: 2.3.0
Reporter: Omar Tougui
when I run the following sqoop command :
{{sqoop import --connect jdbc:postgresql://server/db --username user -P --table
tab --hive-import --create-hive-table}}
{{}}
{{I get this error:}}
{{}}
{{{}22/08/01 16:44:51 INFO atlas.ApplicationProperties: Setting
atlas.graph.index.search.max-result-set-size = 150
22/08/01 16:44:51 INFO atlas.ApplicationProperties: Setting
atlas.graph.index.search.solr.wait-searcher = false
22/08/01 16:44:51 INFO atlas.ApplicationProperties: Property (set to default)
atlas.graph.cache.db-cache = true
22/08/01 16:44:51 INFO atlas.ApplicationProperties: Property (set to default)
atlas.graph.cache.db-cache-clean-wait = 20
22/08/01 16:44:51 INFO atlas.ApplicationProperties: Property (set to default)
atlas.graph.cache.db-cache-size = 0.5
22/08/01 16:44:51 INFO atlas.ApplicationProperties: Property (set to default)
atlas.graph.cache.tx-cache-size = 15000
22/08/01 16:44:51 INFO atlas.ApplicationProperties: Property (set to default)
atlas.graph.cache.tx-dirty-size = 120
22/08/01 16:44:51 INFO hook.AtlasHook: Failed to load application properties
org.apache.atlas.AtlasException: Failed to load application properties
at
org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:155)
at
org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:108)
at org.apache.atlas.hook.AtlasHook.(AtlasHook.java:82)
at org.apache.atlas.sqoop.hook.SqoopHook.(SqoopHook.java:86)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at
org.apache.sqoop.mapreduce.PublishJobData.publishJobData(PublishJobData.java:46)
at
org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:284)
at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:692)
at
org.apache.sqoop.manager.PostgresqlManager.importTable(PostgresqlManager.java:127)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:520)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Caused by: org.apache.commons.configuration.ConversionException:
'atlas.graph.index.search.max-result-set-size' doesn't map to a List object:
150, a java.lang.Integer
at
org.apache.commons.configuration.AbstractConfiguration.getList(AbstractConfiguration.java:1144)
at
org.apache.commons.configuration.AbstractConfiguration.getList(AbstractConfiguration.java:1109)
at
org.apache.commons.configuration.AbstractConfiguration.interpolatedConfiguration(AbstractConfiguration.java:1274)
at
org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:150)
... 17 more
Exception in thread "main" java.lang.ExceptionInInitializerError
at org.apache.atlas.sqoop.hook.SqoopHook.(SqoopHook.java:86)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at
org.apache.sqoop.mapreduce.PublishJobData.publishJobData(PublishJobData.java:46)
at
org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:284)
at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:692)
at
org.apache.sqoop.manager.PostgresqlManager.importTable(PostgresqlManager.java:127)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:520)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Caused by: java.lang.NullPointerException
at org.apache.atlas.hook.AtlasHook.(AtlasHook.java:87)
... 15 more{}}}{{{}{}}}{{{}{}}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: Review Request 74077: Refactor Atlas webapp module to remove Kafka core dependency
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74077/ --- (Updated Aug. 2, 2022, 1:21 p.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Nixon Rodrigues. Changes --- Updated diff with some refactoring and increased timeout for Kafka consumer shutdown. Bugs: ATLAS-4619 https://issues.apache.org/jira/browse/ATLAS-4619 Repository: atlas Description --- Goal is to break the strong coupling between Atlas components and Kafka. These dependencies include using server side libraries of Kafka (this couples the Scala version and other non-public interfaces of Kafka). Any code using server side libraries of Kafka should be refactored. The webapp module uses ShutdownAbleThread from the core kafka library. With this commit, it is changed to Thread, as the try-catch-finally block inside the run() method should solve the issue with closing the consumer. Diffs (updated) - webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 49c504f9f webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java fdfc2560d Diff: https://reviews.apache.org/r/74077/diff/2/ Changes: https://reviews.apache.org/r/74077/diff/1-2/ Testing --- - Unit tests are green - Atlas server is healthy on manually provisoned cluster - Tested the functionality with Hive Hook, no ConcurrentModificationException was present in the logs after stopping Atlas server. Thanks, Patrik Márton
[jira] [Updated] (ATLAS-4653) Upgrade Hadoop to 3.3.3
[ https://issues.apache.org/jira/browse/ATLAS-4653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chaitali borole updated ATLAS-4653: --- Description: currently Hadoop is using 3.3.0 and 3.0.3 This patch also upgrades jackson-databind to 2.13.3 and commons-beanutils to 1.9.4 was:currently Hadoop is using 3.3.0 and 3.0.3 > Upgrade Hadoop to 3.3.3 > --- > > Key: ATLAS-4653 > URL: https://issues.apache.org/jira/browse/ATLAS-4653 > Project: Atlas > Issue Type: Improvement >Reporter: chaitali borole >Assignee: chaitali borole >Priority: Major > > currently Hadoop is using 3.3.0 and 3.0.3 > This patch also upgrades jackson-databind to 2.13.3 and commons-beanutils to > 1.9.4 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Review Request 74079: ATLAS-4653 : Upgrade Hadoop to 3.3.3
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74079/ --- Review request for atlas, Jayendra Parab and Pinal Shah. Bugs: ATLAS-4653 https://issues.apache.org/jira/browse/ATLAS-4653 Repository: atlas Description --- currently Hadoop is using 3.3.0 and 3.0.3 This patch also upgrades jackson-databind to 2.13.3 and commons-beanutils to 1.9.4 Diffs - addons/hbase-bridge/pom.xml a6ed51421 addons/hbase-testing-util/pom.xml 982e9c85e addons/hive-bridge/pom.xml 8c3636e5d addons/sqoop-bridge/pom.xml 021e93f56 addons/storm-bridge/pom.xml e8106afce notification/pom.xml aaf11c7a4 pom.xml 1e32e45f3 test-tools/pom.xml 991fa14c9 webapp/pom.xml 37eeaf4bf Diff: https://reviews.apache.org/r/74079/diff/1/ Testing --- mvn clean -DskipTests package -Pdist,embedded-hbase-solr mvn clean install -Drat.skip atlas server running successfully https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1175/consoleFull Thanks, chaitali
[jira] [Created] (ATLAS-4653) Upgrade Hadoop to 3.3.3
chaitali borole created ATLAS-4653: -- Summary: Upgrade Hadoop to 3.3.3 Key: ATLAS-4653 URL: https://issues.apache.org/jira/browse/ATLAS-4653 Project: Atlas Issue Type: Improvement Reporter: chaitali borole Assignee: chaitali borole currently Hadoop is using 3.3.0 and 3.0.3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74074: ATLAS-4650 : Upgrade Apache Commons BeanUtils to 1.9.4
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74074/ --- (Updated Aug. 2, 2022, 11:09 a.m.) Review request for atlas, Jayendra Parab and Pinal Shah. Bugs: ATLAS-4650 https://issues.apache.org/jira/browse/ATLAS-4650 Repository: atlas Description --- Currently Commons BeanUtils version is 1.9.2,1.9.3,1.8.3 Diffs (updated) - addons/hbase-bridge/pom.xml a6ed51421 addons/hbase-testing-util/pom.xml 982e9c85e addons/hive-bridge/pom.xml 8c3636e5d addons/sqoop-bridge/pom.xml 021e93f56 addons/storm-bridge/pom.xml e8106afce notification/pom.xml aaf11c7a4 pom.xml 1e32e45f3 test-tools/pom.xml 991fa14c9 webapp/pom.xml 37eeaf4bf Diff: https://reviews.apache.org/r/74074/diff/2/ Changes: https://reviews.apache.org/r/74074/diff/1-2/ Testing (updated) --- mvn clean -DskipTests package -Pdist,embedded-hbase-solr mvn clean install -Drat.skip atlas server running successfully https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1175/consoleFull Thanks, chaitali
Review Request 74077: Refactor Atlas webapp module to remove Kafka core dependency
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74077/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, and Nixon Rodrigues. Bugs: ATLAS-4619 https://issues.apache.org/jira/browse/ATLAS-4619 Repository: atlas Description --- Goal is to break the strong coupling between Atlas components and Kafka. These dependencies include using server side libraries of Kafka (this couples the Scala version and other non-public interfaces of Kafka). Any code using server side libraries of Kafka should be refactored. The webapp module uses ShutdownAbleThread from the core kafka library. With this commit, it is changed to Thread, as the try-catch-finally block inside the run() method should solve the issue with closing the consumer. Diffs - webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 49c504f9f Diff: https://reviews.apache.org/r/74077/diff/1/ Testing --- - Unit tests are green - Atlas server is healthy on manually provisoned cluster - Tested the functionality with Hive Hook, no ConcurrentModificationException was present in the logs after stopping Atlas server. Thanks, Patrik Márton
[jira] [Assigned] (ATLAS-4619) Refactor Atlas webapp module to remove Kafka core dependency
[ https://issues.apache.org/jira/browse/ATLAS-4619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrik Márton reassigned ATLAS-4619: Assignee: Patrik Márton > Refactor Atlas webapp module to remove Kafka core dependency > > > Key: ATLAS-4619 > URL: https://issues.apache.org/jira/browse/ATLAS-4619 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Patrik Márton >Assignee: Patrik Márton >Priority: Major > > Goal is to break the strong coupling between Atlas components and Kafka. > These dependencies include using server side libraries of Kafka (this couples > the Scala version and other non-public interfaces of Kafka). Any code using > server side libraries of Kafka should be refactored. > Since atlas webapp module uses the ShutdownableThread from core kafka, it > should be refactored in a way to eliminate this dependency. > https://github.com/apache/atlas/blob/master/webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java#L526 -- This message was sent by Atlassian Jira (v8.20.10#820010)
