[
https://issues.apache.org/jira/browse/IMPALA-10057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17834601#comment-17834601
]
ASF subversion and git services commented on IMPALA-10057:
----------------------------------------------------------
Commit 54faec0523d3049d6a84c9027d4cde5268a62d15 in impala's branch
refs/heads/branch-3.4.2 from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=54faec052 ]
IMPALA-10057: Fix log spew by using jars in the classpath
Some tests saw log spew that causes the INFO log files to
be filled with output like this:
E0903 02:25:39.453887 12060 TransactionKeepalive.java:137] Unexpected exception
thrown
Java exception follows:
java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError:
org/apache/impala/common/TransactionKeepalive$HeartbeatContext
at
org.apache.impala.common.TransactionKeepalive$DaemonThread.run(TransactionKeepalive.java:114)
at java.lang.Thread.run(Thread.java:748)
...
It turns out that the catalogd/impalad use a CLASSPATH in
tests that refers to fe/target/classes. The maven command
that runs frontend tests recompiles these classes and
causes the files in fe/target/classes to be deleted and
recreated. There are race conditions where this causes
the symptoms above.
This changes the CLASSPATH to use the frontend jars, which
are not impacted by the machinations on fe/target/classes.
To find the appropriate jar, set-classpath.sh needs to
know the Impala version. This adds IMPALA_VERSION in
bin/impala-config.sh to provide an easy to use
environment variable.
To make the versioning more uniform, this modifies
bin/save-version.sh to use this environment variable.
It also adds a check to make sure that the Java pom.xml
files use the same version as the environment variable.
It fails the build if the Java pom.xml files do not
match.
Testing:
- Ran core jobs
- Checked the log file sizes on jobs
- Changed a Java pom.xml's version and verified that
bin/validate-java-pom-versions.sh fails
Merge conflicts:
- Change version string "4.1.0" to "3.4.2".
Change-Id: Id35544e446c5bf283c322d3fe2e7ad475cfa12eb
Reviewed-on: http://gerrit.cloudera.org:8080/18415
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Michael Smith <[email protected]>
Reviewed-by: Quanlong Huang <[email protected]>
Reviewed-on: http://gerrit.cloudera.org:8080/18879
Reviewed-by: Joe McDonnell <[email protected]>
Tested-by: Quanlong Huang <[email protected]>
Reviewed-on: http://gerrit.cloudera.org:8080/21129
Reviewed-by: Zihao Ye <[email protected]>
> TransactionKeepalive NoClassDefFoundError floods logs during JDBC_TEST/FE_TEST
> ------------------------------------------------------------------------------
>
> Key: IMPALA-10057
> URL: https://issues.apache.org/jira/browse/IMPALA-10057
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 4.0.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
> Labels: flaky
> Fix For: Impala 4.2.0, Impala 4.1.1
>
>
> For the both the normal tests and the docker-based tests, the Impala logs
> generated during the FE_TEST/JDBC_TEST can be huge:
>
> {noformat}
> $ du -c -h fe_test/ee_tests
> 4.0K fe_test/ee_tests/minidumps/statestored
> 4.0K fe_test/ee_tests/minidumps/impalad
> 4.0K fe_test/ee_tests/minidumps/catalogd
> 16K fe_test/ee_tests/minidumps
> 352K fe_test/ee_tests/profiles
> 81G fe_test/ee_tests
> 81G total{noformat}
> Creating a tarball of these logs takes 10 minutes. The Impalad/catalogd logs
> are filled with this error over and over:
> {noformat}
> E0903 02:25:39.453887 12060 TransactionKeepalive.java:137] Unexpected
> exception thrown
> Java exception follows:
> java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError:
> org/apache/impala/common/TransactionKeepalive$HeartbeatContext
> at
> org.apache.impala.common.TransactionKeepalive$DaemonThread.run(TransactionKeepalive.java:114)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/impala/common/TransactionKeepalive$HeartbeatContext
> ... 2 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.impala.common.TransactionKeepalive$HeartbeatContext
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 2 more{noformat}
> Two interesting points:
> # The frontend/jdbc tests are passing, so all of these errors in the impalad
> logs are not impacting tests.
> # These errors aren't concurrently with any of the other tests (ee tests,
> custom cluster tests, etc).
> This is happening on normal core runs (including the GVO job that does
> FE_TEST/JDBC_TEST) on both Ubuntu and Centos 7. It is also happening on
> docker-based tests. A theory is that FE_TEST/JDBC_TEST have an Impala cluster
> running and then invoke maven to run the tests. Maven could manipulate jars
> while Impala is running. Maybe there is a race-condition or conflict when
> manipulating those jars that could cause the NoClassDefFoundError. It makes
> no sense for Impala not to be able to find
> TransactionKeepalive$HeartbeatContext.
> When it happens, it is in a tight loop, printing the message more than once
> per millisecond. It fills the ERROR, WARNING, and INFO logs with that
> message, sometimes for multiple Impalads and/or catalogd.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
