[
https://issues.apache.org/jira/browse/ATLAS-5408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18116637#comment-18116637
]
ASF subversion and git services commented on ATLAS-5408:
--------------------------------------------------------
Commit 9c8d3436fa96aa0d84a5de897a5362bfaff12310 in atlas's branch
refs/heads/atlas-2.6 from Nixon Rodrigues
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=9c8d3436f ]
ATLAS-5408: Fix hook import classpath and logging issues in docker envs (#761)
* ATLAS-5408: Fix hook import classpath and logging in docker/minimal envs
HBase: ship commons-configuration 1.x in atlas-hbase-plugin-impl because
import-hbase.sh excludes it from HBase lib while Hadoop still needs it when
HADOOP_HOME/classpath is absent.
Logging: create /var/log/atlas in atlas-base and mkdir -p ATLAS_LOG_DIR in
import-hbase, import-hive, and import-kafka so logback can open import logs.
Kafka: stop putting broker config and log4j jars on the import classpath to
avoid broker log4j.properties writing to /server.log; default KAFKA_CONF to
$KAFKA_HOME/config.
> Hook import scripts fail or emit errors in minimal/docker environments (HBase
> NoClassDefFoundError, Kafka log4j writing to /, missing /var/log/atlas)
> -----------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ATLAS-5408
> URL: https://issues.apache.org/jira/browse/ATLAS-5408
> Project: Atlas
> Issue Type: Bug
> Components: atlas-docker
> Reporter: Nixon Rodrigues
> Assignee: Nixon Rodrigues
> Priority: Major
> Fix For: 3.0.0, 2.6.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Running {{{}import-hbase.sh{}}}, {{{}import-hive.sh{}}}, and
> {{import-kafka.sh}} against the dev-support Docker stack (and similar minimal
> installs without full Hadoop/Kafka broker layout) hits three related
> classpath/logging problems. All three scripts default logs to
> {{/var/log/atlas}} and assemble classpaths from hook plugin impl + cluster
> libs; base images and script behavior did not align.
> ----
> h4. 1) HBase import — {{NoClassDefFoundError:
> org/apache/commons/configuration/Configuration}}
> Symptom
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/commons/configuration/Configuration
> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem...
> at org.apache.atlas.hbase.bridge.HBaseBridge.main
>
> Steps to reproduce
> # Start atlas-docker HBase container (HBase only, no {{{}HADOOP_HOME{}}}).
> # Run {{{}/opt/apache-atlas-hbase-hook/hook-bin/import-hbase.sh{}}}.
> Root cause
> * {{import-hbase.sh}} excludes {{commons-configuration-1.*}} from HBase
> {{lib/}} to avoid clashing with Atlas {{{}commons-configuration2{}}}.
> * Hadoop metrics/UGI still require commons-configuration 1.x.
> * With no {{{}hadoop classpath{}}}, nothing supplies that class.
> Fix
> * Bundle {{commons-configuration}} 1.10 in {{atlas-hbase-plugin-impl}}
> ({{{}addons/hbase-bridge/pom.xml{}}}).
> * Document in {{import-hbase.sh}} why 1.x is in the plugin impl.
> Local commit: {{34a0c82bb}} (message references ATLAS-5404)
> ----
> h4. 2) Hook import logs — logback cannot create
> {{/var/log/atlas/import-*.log}}
> Symptom
> ERROR ... RollingFileAppender[FILE] - Failed to create parent directories for
> [/var/log/atlas/import-hbase.log]
> FileNotFoundException: /var/log/atlas/import-hbase.log (No such file or
> directory)
>
> (Same pattern for {{{}import-hive.log{}}}, {{{}import-kafka.log{}}}.)
> Steps to reproduce
> # Run any hook import script in atlas-base-derived images (HBase/Hive/Kafka)
> without pre-creating the log dir.
> Root cause
> * Scripts default {{{}ATLAS_LOG_DIR=/var/log/atlas{}}}.
> * {{Dockerfile.atlas}} creates that path; {{Dockerfile.atlas-base}} did not
> (hook containers inherit base only).
> * Scripts did not {{mkdir -p}} before starting the JVM.
> Fix
> * {{{}dev-support/atlas-docker/Dockerfile.atlas-base{}}}: create
> {{{}/var/log/atlas{}}}, {{{}root:hadoop{}}}, mode {{{}775{}}}.
> * {{{}import-hbase.sh{}}}, {{{}import-hive.sh{}}}, {{{}import-kafka.sh{}}}:
> {{mkdir -p "${ATLAS_LOG_DIR}"}} after default is set.
> Local commit: {{144e3bc61}} (message references ATLAS-5407)
> ----
> h4. 3) Kafka import — log4j errors for {{{}/server.log{}}},
> {{{}/kafka-request.log{}}}, etc.
> Symptom
> # Custom Kafka bridge #
> log4j:ERROR setFile(null,true) call failed.
> java.io.FileNotFoundException: /server.log (Permission denied)
> ...
> /kafka-request.log, /controller.log, /log-cleaner.log, /state-change.log
>
> Steps to reproduce
> # In Kafka docker container, run
> {{{}/opt/apache-atlas-kafka-hook/hook-bin/import-kafka.sh{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
