Nixon Rodrigues created ATLAS-5408:
--------------------------------------
Summary: Hook import scripts fail or emit errors in minimal/docker
environments (HBase NoClassDefFoundError, Kafka log4j writing to /, missing
/var/log/atlas)
Key: ATLAS-5408
URL: https://issues.apache.org/jira/browse/ATLAS-5408
Project: Atlas
Issue Type: Bug
Components: atlas-docker
Reporter: Nixon Rodrigues
Fix For: 3.0.0, 2.6.0
Running {{{}import-hbase.sh{}}}, {{{}import-hive.sh{}}}, and
{{import-kafka.sh}} against the dev-support Docker stack (and similar minimal
installs without full Hadoop/Kafka broker layout) hits three related
classpath/logging problems. All three scripts default logs to
{{/var/log/atlas}} and assemble classpaths from hook plugin impl + cluster
libs; base images and script behavior did not align.
----
h4. 1) HBase import — {{NoClassDefFoundError:
org/apache/commons/configuration/Configuration}}
Symptom
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/commons/configuration/Configuration
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem...
at org.apache.atlas.hbase.bridge.HBaseBridge.main
Steps to reproduce
# Start atlas-docker HBase container (HBase only, no {{{}HADOOP_HOME{}}}).
# Run {{{}/opt/apache-atlas-hbase-hook/hook-bin/import-hbase.sh{}}}.
Root cause
* {{import-hbase.sh}} excludes {{commons-configuration-1.*}} from HBase
{{lib/}} to avoid clashing with Atlas {{{}commons-configuration2{}}}.
* Hadoop metrics/UGI still require commons-configuration 1.x.
* With no {{{}hadoop classpath{}}}, nothing supplies that class.
Fix
* Bundle {{commons-configuration}} 1.10 in {{atlas-hbase-plugin-impl}}
({{{}addons/hbase-bridge/pom.xml{}}}).
* Document in {{import-hbase.sh}} why 1.x is in the plugin impl.
Local commit: {{34a0c82bb}} (message references ATLAS-5404)
----
h4. 2) Hook import logs — logback cannot create {{/var/log/atlas/import-*.log}}
Symptom
ERROR ... RollingFileAppender[FILE] - Failed to create parent directories for
[/var/log/atlas/import-hbase.log]
FileNotFoundException: /var/log/atlas/import-hbase.log (No such file or
directory)
(Same pattern for {{{}import-hive.log{}}}, {{{}import-kafka.log{}}}.)
Steps to reproduce
# Run any hook import script in atlas-base-derived images (HBase/Hive/Kafka)
without pre-creating the log dir.
Root cause
* Scripts default {{{}ATLAS_LOG_DIR=/var/log/atlas{}}}.
* {{Dockerfile.atlas}} creates that path; {{Dockerfile.atlas-base}} did not
(hook containers inherit base only).
* Scripts did not {{mkdir -p}} before starting the JVM.
Fix
* {{{}dev-support/atlas-docker/Dockerfile.atlas-base{}}}: create
{{{}/var/log/atlas{}}}, {{{}root:hadoop{}}}, mode {{{}775{}}}.
* {{{}import-hbase.sh{}}}, {{{}import-hive.sh{}}}, {{{}import-kafka.sh{}}}:
{{mkdir -p "${ATLAS_LOG_DIR}"}} after default is set.
Local commit: {{144e3bc61}} (message references ATLAS-5407)
----
h4. 3) Kafka import — log4j errors for {{{}/server.log{}}},
{{{}/kafka-request.log{}}}, etc.
Symptom
# Custom Kafka bridge #
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /server.log (Permission denied)
...
/kafka-request.log, /controller.log, /log-cleaner.log, /state-change.log
Steps to reproduce
# In Kafka docker container, run
{{{}/opt/apache-atlas-kafka-hook/hook-bin/import-kafka.sh{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
