[
https://issues.apache.org/jira/browse/IMPALA-10057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell updated IMPALA-10057:
-----------------------------------
Description:
For the both the normal tests and the docker-based tests, the Impala logs
generated during the FE_TEST/JDBC_TEST can be huge:
{noformat}
$ du -c -h fe_test/ee_tests
4.0K fe_test/ee_tests/minidumps/statestored
4.0K fe_test/ee_tests/minidumps/impalad
4.0K fe_test/ee_tests/minidumps/catalogd
16K fe_test/ee_tests/minidumps
352K fe_test/ee_tests/profiles
81G fe_test/ee_tests
81G total{noformat}
Creating a tarball of these logs takes 10 minutes. The Impalad/catalogd logs
are filled with this error over and over:
{noformat}
E0903 02:25:39.453887 12060 TransactionKeepalive.java:137] Unexpected exception
thrown
Java exception follows:
java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError:
org/apache/impala/common/TransactionKeepalive$HeartbeatContext
at
org.apache.impala.common.TransactionKeepalive$DaemonThread.run(TransactionKeepalive.java:114)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError:
org/apache/impala/common/TransactionKeepalive$HeartbeatContext
... 2 more
Caused by: java.lang.ClassNotFoundException:
org.apache.impala.common.TransactionKeepalive$HeartbeatContext
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 2 more{noformat}
Two interesting points:
# The frontend/jdbc tests are passing, so all of these errors in the impalad
logs are not impacting tests.
# These errors aren't concurrently with any of the other tests (ee tests,
custom cluster tests, etc).
This is happening on normal core runs (including the GVO job that does
FE_TEST/JDBC_TEST) on both Ubuntu and Centos 7. It is also happening on
docker-based tests. A theory is that FE_TEST/JDBC_TEST have an Impala cluster
running and then invoke maven to run the tests. Maven could manipulate jars
while Impala is running. Maybe there is a race-condition or conflict when
manipulating those jars that could cause the NoClassDefFoundError. It makes no
sense for Impala not to be able to find TransactionKeepalive$HeartbeatContext.
When it happens, it is in a tight loop, printing the message more than once per
millisecond. It fills the ERROR, WARNING, and INFO logs with that message,
sometimes for multiple Impalads and/or catalogd.
was:
For the docker-based tests, the Impala logs generated during the FE_TEST are
huge:
{noformat}
$ du -c -h fe_test/ee_tests
4.0K fe_test/ee_tests/minidumps/statestored
4.0K fe_test/ee_tests/minidumps/impalad
4.0K fe_test/ee_tests/minidumps/catalogd
16K fe_test/ee_tests/minidumps
352K fe_test/ee_tests/profiles
81G fe_test/ee_tests
81G total{noformat}
Creating a tarball of these logs takes 10 minutes. The Impalad/catalogd logs
are filled with this error over and over:
{noformat}
E0805 06:08:45.485440 11219 TransactionKeepalive.java:137] Unexpected exception
thrown
Java exception follows:
java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError:
at
org.apache.impala.common.TransactionKeepalive$DaemonThread.run(TransactionKeepalive.java:114)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError:
... 2 more{noformat}
Two interesting points:
# The frontend tests are passing, so all of these errors in the impalad logs
are not impacting tests.
# These errors aren't happening in any of the other tests (ee tests, custom
cluster tests, etc). These errors are not seen outside the docker-based tests.
A theory is that FE_TEST runs mvn to build and run the frontend tests. If there
were some bad interaction of mvn with the docker filesystem in manipulating the
~/.m2 directory, that may cause problems. One thing to try may be to copy the
.m2 directory to make sure it is in the top docker layer (similar to what we do
with kudu wal files).
> TransactionKeepalive NoClassDefFoundError floods logs during JDBC_TEST/FE_TEST
> ------------------------------------------------------------------------------
>
> Key: IMPALA-10057
> URL: https://issues.apache.org/jira/browse/IMPALA-10057
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 4.0
> Reporter: Joe McDonnell
> Priority: Major
>
> For the both the normal tests and the docker-based tests, the Impala logs
> generated during the FE_TEST/JDBC_TEST can be huge:
>
> {noformat}
> $ du -c -h fe_test/ee_tests
> 4.0K fe_test/ee_tests/minidumps/statestored
> 4.0K fe_test/ee_tests/minidumps/impalad
> 4.0K fe_test/ee_tests/minidumps/catalogd
> 16K fe_test/ee_tests/minidumps
> 352K fe_test/ee_tests/profiles
> 81G fe_test/ee_tests
> 81G total{noformat}
> Creating a tarball of these logs takes 10 minutes. The Impalad/catalogd logs
> are filled with this error over and over:
> {noformat}
> E0903 02:25:39.453887 12060 TransactionKeepalive.java:137] Unexpected
> exception thrown
> Java exception follows:
> java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError:
> org/apache/impala/common/TransactionKeepalive$HeartbeatContext
> at
> org.apache.impala.common.TransactionKeepalive$DaemonThread.run(TransactionKeepalive.java:114)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/impala/common/TransactionKeepalive$HeartbeatContext
> ... 2 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.impala.common.TransactionKeepalive$HeartbeatContext
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 2 more{noformat}
> Two interesting points:
> # The frontend/jdbc tests are passing, so all of these errors in the impalad
> logs are not impacting tests.
> # These errors aren't concurrently with any of the other tests (ee tests,
> custom cluster tests, etc).
> This is happening on normal core runs (including the GVO job that does
> FE_TEST/JDBC_TEST) on both Ubuntu and Centos 7. It is also happening on
> docker-based tests. A theory is that FE_TEST/JDBC_TEST have an Impala cluster
> running and then invoke maven to run the tests. Maven could manipulate jars
> while Impala is running. Maybe there is a race-condition or conflict when
> manipulating those jars that could cause the NoClassDefFoundError. It makes
> no sense for Impala not to be able to find
> TransactionKeepalive$HeartbeatContext.
> When it happens, it is in a tight loop, printing the message more than once
> per millisecond. It fills the ERROR, WARNING, and INFO logs with that
> message, sometimes for multiple Impalads and/or catalogd.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
