[jira] [Commented] (IMPALA-7069) Java UDF tests can trigger a crash in Java ClassLoader
[
https://issues.apache.org/jira/browse/IMPALA-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542106#comment-16542106
]
Vuk Ercegovac commented on IMPALA-7069:
---
yup, I think they had the same root cause. closing this... lets open up a new
Jira in the event that we see more of these types of crashes.
> Java UDF tests can trigger a crash in Java ClassLoader
> --
>
> Key: IMPALA-7069
> URL: https://issues.apache.org/jira/browse/IMPALA-7069
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Taras Bobrovytsky
>Priority: Blocker
> Labels: broken-build, crash, flaky
> Attachments: hs_err_pid22764.log, hs_err_pid29246.log,
> hs_err_pid8975.log, hs_err_pid9694.log
>
>
> I hit this crash on a GVO, but was able to reproduce it on master on my
> desktop.
> Repro steps:
> {code}
> git checkout c1362afb9a072e49df470d9068d44cdbdf5cdec5
> ./buildall.sh -debug -noclean -notests -skiptests -ninja
> start-impala-cluster.py
> while impala-py.test tests/query_test/test_udfs.py -k 'hive or java or jar'
> -n4 --verbose; do date; done
> {code}
> I generally hit the crash within a hour of looping the test.
> {noformat}
> Stack: [0x7fa04791f000,0x7fa04812], sp=0x7fa04811aff0, free
> space=8175k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> V [libjvm.so+0x8a8107]
> V [libjvm.so+0x96cf5f]
> v ~RuntimeStub::_complete_monitor_locking_Java
> J 2758 C2
> java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object;
> (362 bytes) @ 0x7fa0c73637d4 [0x7fa0c7362d00+0xad4]
> J 2311 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122
> bytes) @ 0x7fa0c70a09a8 [0x7fa0c70a08e0+0xc8]
> J 3953 C2
> java.net.FactoryURLClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;
> (40 bytes) @ 0x7fa0c71ce0f0 [0x7fa0c71ce0a0+0x50]
> J 2987 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (7
> bytes) @ 0x7fa0c72ddb64 [0x7fa0c72ddb20+0x44]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x661ec4]
> V [libjvm.so+0x662523]
> V [libjvm.so+0x9e398d]
> V [libjvm.so+0x9e2326]
> V [libjvm.so+0x9e2b50]
> V [libjvm.so+0x42c099]
> V [libjvm.so+0x9dc786]
> V [libjvm.so+0x6a5edf]
> V [libjvm.so+0x6a70cb] JVM_DefineClass+0xbb
> V [libjvm.so+0xa31ea5]
> V [libjvm.so+0xa37ea7]
> J 4842
> sun.misc.Unsafe.defineClass(Ljava/lang/String;[BIILjava/lang/ClassLoader;Ljava/security/ProtectionDomain;)Ljava/lang/Class;
> (0 bytes) @ 0x7fa0c7af120b [0x7fa0c7af1100+0x10b]
> J 13229 C2 sun.reflect.MethodAccessorGenerator$1.run()Ljava/lang/Object; (5
> bytes) @ 0x7fa0c8cf2a74 [0x7fa0c8cf2940+0x134]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6b5949] JVM_DoPrivileged+0x429
> J 1035
> java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;
> (0 bytes) @ 0x7fa0c7220c7f [0x7fa0c7220bc0+0xbf]
> J 20421 C2
> sun.reflect.MethodAccessorGenerator.generate(Ljava/lang/Class;Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;IZZLjava/lang/Class;)Lsun/reflect/MagicAccessorImpl;
> (762 bytes) @ 0x7fa0c89bb848 [0x7fa0c89b9da0+0x1aa8]
> J 4163 C2
> sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (104 bytes) @ 0x7fa0c789cca8 [0x7fa0c789c8c0+0x3e8]
> J 2379 C2 org.apache.impala.hive.executor.UdfExecutor.evaluate()V (396 bytes)
> @ 0x7fa0c711c638 [0x7fa0c711c400+0x238]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6822d7]
> V [libjvm.so+0x6862c9]
> C [impalad+0x2a004fa] JNIEnv_::CallNonvirtualVoidMethodA(_jobject*,
> _jclass*, _jmethodID*, jvalue const*)+0x40
> C [impalad+0x29fe4ff]
> impala::HiveUdfCall::Evaluate(impala::ScalarExprEvaluator*, impala::TupleRow
> const*) const+0x44b
> C [impalad+0x29ffde9]
> impala::HiveUdfCall::GetSmallIntVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0xbb
> C [impalad+0x2a0948a]
> impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&,
> impala::TupleRow const*)+0x14c
> C [impalad+0x2a48eb1]
> impala::ScalarFnCall::EvaluateNonConstantChildren(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x9d
> C [impalad+0x2a4abba] impala_udf::BooleanVal
> impala::ScalarFnCall::InterpretEval(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x18c
> C [impalad+0x2a4907d]
> impala::ScalarFnCall::GetBooleanVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x179
> C [impalad+0x2a09c7f]
>
[jira] [Commented] (IMPALA-7069) Java UDF tests can trigger a crash in Java ClassLoader
[
https://issues.apache.org/jira/browse/IMPALA-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542097#comment-16542097
]
Tim Armstrong commented on IMPALA-7069:
---
[~vukercegovac] [~tarasbob] I don't think we've seen this for a while. Do we
think this could have been fixed by IMPALA-7150
> Java UDF tests can trigger a crash in Java ClassLoader
> --
>
> Key: IMPALA-7069
> URL: https://issues.apache.org/jira/browse/IMPALA-7069
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Taras Bobrovytsky
>Priority: Blocker
> Labels: broken-build, crash, flaky
> Attachments: hs_err_pid22764.log, hs_err_pid29246.log,
> hs_err_pid8975.log, hs_err_pid9694.log
>
>
> I hit this crash on a GVO, but was able to reproduce it on master on my
> desktop.
> Repro steps:
> {code}
> git checkout c1362afb9a072e49df470d9068d44cdbdf5cdec5
> ./buildall.sh -debug -noclean -notests -skiptests -ninja
> start-impala-cluster.py
> while impala-py.test tests/query_test/test_udfs.py -k 'hive or java or jar'
> -n4 --verbose; do date; done
> {code}
> I generally hit the crash within a hour of looping the test.
> {noformat}
> Stack: [0x7fa04791f000,0x7fa04812], sp=0x7fa04811aff0, free
> space=8175k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> V [libjvm.so+0x8a8107]
> V [libjvm.so+0x96cf5f]
> v ~RuntimeStub::_complete_monitor_locking_Java
> J 2758 C2
> java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object;
> (362 bytes) @ 0x7fa0c73637d4 [0x7fa0c7362d00+0xad4]
> J 2311 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122
> bytes) @ 0x7fa0c70a09a8 [0x7fa0c70a08e0+0xc8]
> J 3953 C2
> java.net.FactoryURLClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;
> (40 bytes) @ 0x7fa0c71ce0f0 [0x7fa0c71ce0a0+0x50]
> J 2987 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (7
> bytes) @ 0x7fa0c72ddb64 [0x7fa0c72ddb20+0x44]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x661ec4]
> V [libjvm.so+0x662523]
> V [libjvm.so+0x9e398d]
> V [libjvm.so+0x9e2326]
> V [libjvm.so+0x9e2b50]
> V [libjvm.so+0x42c099]
> V [libjvm.so+0x9dc786]
> V [libjvm.so+0x6a5edf]
> V [libjvm.so+0x6a70cb] JVM_DefineClass+0xbb
> V [libjvm.so+0xa31ea5]
> V [libjvm.so+0xa37ea7]
> J 4842
> sun.misc.Unsafe.defineClass(Ljava/lang/String;[BIILjava/lang/ClassLoader;Ljava/security/ProtectionDomain;)Ljava/lang/Class;
> (0 bytes) @ 0x7fa0c7af120b [0x7fa0c7af1100+0x10b]
> J 13229 C2 sun.reflect.MethodAccessorGenerator$1.run()Ljava/lang/Object; (5
> bytes) @ 0x7fa0c8cf2a74 [0x7fa0c8cf2940+0x134]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6b5949] JVM_DoPrivileged+0x429
> J 1035
> java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;
> (0 bytes) @ 0x7fa0c7220c7f [0x7fa0c7220bc0+0xbf]
> J 20421 C2
> sun.reflect.MethodAccessorGenerator.generate(Ljava/lang/Class;Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;IZZLjava/lang/Class;)Lsun/reflect/MagicAccessorImpl;
> (762 bytes) @ 0x7fa0c89bb848 [0x7fa0c89b9da0+0x1aa8]
> J 4163 C2
> sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (104 bytes) @ 0x7fa0c789cca8 [0x7fa0c789c8c0+0x3e8]
> J 2379 C2 org.apache.impala.hive.executor.UdfExecutor.evaluate()V (396 bytes)
> @ 0x7fa0c711c638 [0x7fa0c711c400+0x238]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6822d7]
> V [libjvm.so+0x6862c9]
> C [impalad+0x2a004fa] JNIEnv_::CallNonvirtualVoidMethodA(_jobject*,
> _jclass*, _jmethodID*, jvalue const*)+0x40
> C [impalad+0x29fe4ff]
> impala::HiveUdfCall::Evaluate(impala::ScalarExprEvaluator*, impala::TupleRow
> const*) const+0x44b
> C [impalad+0x29ffde9]
> impala::HiveUdfCall::GetSmallIntVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0xbb
> C [impalad+0x2a0948a]
> impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&,
> impala::TupleRow const*)+0x14c
> C [impalad+0x2a48eb1]
> impala::ScalarFnCall::EvaluateNonConstantChildren(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x9d
> C [impalad+0x2a4abba] impala_udf::BooleanVal
> impala::ScalarFnCall::InterpretEval(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x18c
> C [impalad+0x2a4907d]
> impala::ScalarFnCall::GetBooleanVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x179
> C [impalad+0x2a09c7f]
>
[jira] [Commented] (IMPALA-7069) Java UDF tests can trigger a crash in Java ClassLoader
[
https://issues.apache.org/jira/browse/IMPALA-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505657#comment-16505657
]
Philip Zeyliger commented on IMPALA-7069:
-
Depending on how we do class loaders, it's entirely possible that we leak Java
memory (as distinct from malloc() memory) too. The easiest way to check that is
to run with some {{JAVA_TOOL_OPTIONS=-verbose:gc}} flags and see what comes
out. (Or configure a GC log file.)
> Java UDF tests can trigger a crash in Java ClassLoader
> --
>
> Key: IMPALA-7069
> URL: https://issues.apache.org/jira/browse/IMPALA-7069
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Taras Bobrovytsky
>Priority: Blocker
> Labels: broken-build, crash, flaky
> Attachments: hs_err_pid22764.log, hs_err_pid29246.log,
> hs_err_pid8975.log, hs_err_pid9694.log
>
>
> I hit this crash on a GVO, but was able to reproduce it on master on my
> desktop.
> Repro steps:
> {code}
> git checkout c1362afb9a072e49df470d9068d44cdbdf5cdec5
> ./buildall.sh -debug -noclean -notests -skiptests -ninja
> start-impala-cluster.py
> while impala-py.test tests/query_test/test_udfs.py -k 'hive or java or jar'
> -n4 --verbose; do date; done
> {code}
> I generally hit the crash within a hour of looping the test.
> {noformat}
> Stack: [0x7fa04791f000,0x7fa04812], sp=0x7fa04811aff0, free
> space=8175k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> V [libjvm.so+0x8a8107]
> V [libjvm.so+0x96cf5f]
> v ~RuntimeStub::_complete_monitor_locking_Java
> J 2758 C2
> java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object;
> (362 bytes) @ 0x7fa0c73637d4 [0x7fa0c7362d00+0xad4]
> J 2311 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122
> bytes) @ 0x7fa0c70a09a8 [0x7fa0c70a08e0+0xc8]
> J 3953 C2
> java.net.FactoryURLClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;
> (40 bytes) @ 0x7fa0c71ce0f0 [0x7fa0c71ce0a0+0x50]
> J 2987 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (7
> bytes) @ 0x7fa0c72ddb64 [0x7fa0c72ddb20+0x44]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x661ec4]
> V [libjvm.so+0x662523]
> V [libjvm.so+0x9e398d]
> V [libjvm.so+0x9e2326]
> V [libjvm.so+0x9e2b50]
> V [libjvm.so+0x42c099]
> V [libjvm.so+0x9dc786]
> V [libjvm.so+0x6a5edf]
> V [libjvm.so+0x6a70cb] JVM_DefineClass+0xbb
> V [libjvm.so+0xa31ea5]
> V [libjvm.so+0xa37ea7]
> J 4842
> sun.misc.Unsafe.defineClass(Ljava/lang/String;[BIILjava/lang/ClassLoader;Ljava/security/ProtectionDomain;)Ljava/lang/Class;
> (0 bytes) @ 0x7fa0c7af120b [0x7fa0c7af1100+0x10b]
> J 13229 C2 sun.reflect.MethodAccessorGenerator$1.run()Ljava/lang/Object; (5
> bytes) @ 0x7fa0c8cf2a74 [0x7fa0c8cf2940+0x134]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6b5949] JVM_DoPrivileged+0x429
> J 1035
> java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;
> (0 bytes) @ 0x7fa0c7220c7f [0x7fa0c7220bc0+0xbf]
> J 20421 C2
> sun.reflect.MethodAccessorGenerator.generate(Ljava/lang/Class;Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;IZZLjava/lang/Class;)Lsun/reflect/MagicAccessorImpl;
> (762 bytes) @ 0x7fa0c89bb848 [0x7fa0c89b9da0+0x1aa8]
> J 4163 C2
> sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (104 bytes) @ 0x7fa0c789cca8 [0x7fa0c789c8c0+0x3e8]
> J 2379 C2 org.apache.impala.hive.executor.UdfExecutor.evaluate()V (396 bytes)
> @ 0x7fa0c711c638 [0x7fa0c711c400+0x238]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6822d7]
> V [libjvm.so+0x6862c9]
> C [impalad+0x2a004fa] JNIEnv_::CallNonvirtualVoidMethodA(_jobject*,
> _jclass*, _jmethodID*, jvalue const*)+0x40
> C [impalad+0x29fe4ff]
> impala::HiveUdfCall::Evaluate(impala::ScalarExprEvaluator*, impala::TupleRow
> const*) const+0x44b
> C [impalad+0x29ffde9]
> impala::HiveUdfCall::GetSmallIntVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0xbb
> C [impalad+0x2a0948a]
> impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&,
> impala::TupleRow const*)+0x14c
> C [impalad+0x2a48eb1]
> impala::ScalarFnCall::EvaluateNonConstantChildren(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x9d
> C [impalad+0x2a4abba] impala_udf::BooleanVal
> impala::ScalarFnCall::InterpretEval(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x18c
> C [impalad+0x2a4907d]
>
[jira] [Commented] (IMPALA-7069) Java UDF tests can trigger a crash in Java ClassLoader
[
https://issues.apache.org/jira/browse/IMPALA-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503918#comment-16503918
]
Taras Bobrovytsky commented on IMPALA-7069:
---
An interesting thing that I noticed is that the crash happens after running the
test in a loop for about 45 minutes. The time between starting the test and
when the crash happens is strangely consistent. I tried this a few times, so it
seems that it's unlikely to be due to randomness.
> Java UDF tests can trigger a crash in Java ClassLoader
> --
>
> Key: IMPALA-7069
> URL: https://issues.apache.org/jira/browse/IMPALA-7069
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Taras Bobrovytsky
>Priority: Blocker
> Labels: broken-build, crash, flaky
> Attachments: hs_err_pid22764.log, hs_err_pid29246.log,
> hs_err_pid8975.log, hs_err_pid9694.log
>
>
> I hit this crash on a GVO, but was able to reproduce it on master on my
> desktop.
> Repro steps:
> {code}
> git checkout c1362afb9a072e49df470d9068d44cdbdf5cdec5
> ./buildall.sh -debug -noclean -notests -skiptests -ninja
> start-impala-cluster.py
> while impala-py.test tests/query_test/test_udfs.py -k 'hive or java or jar'
> -n4 --verbose; do date; done
> {code}
> I generally hit the crash within a hour of looping the test.
> {noformat}
> Stack: [0x7fa04791f000,0x7fa04812], sp=0x7fa04811aff0, free
> space=8175k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> V [libjvm.so+0x8a8107]
> V [libjvm.so+0x96cf5f]
> v ~RuntimeStub::_complete_monitor_locking_Java
> J 2758 C2
> java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object;
> (362 bytes) @ 0x7fa0c73637d4 [0x7fa0c7362d00+0xad4]
> J 2311 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122
> bytes) @ 0x7fa0c70a09a8 [0x7fa0c70a08e0+0xc8]
> J 3953 C2
> java.net.FactoryURLClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;
> (40 bytes) @ 0x7fa0c71ce0f0 [0x7fa0c71ce0a0+0x50]
> J 2987 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (7
> bytes) @ 0x7fa0c72ddb64 [0x7fa0c72ddb20+0x44]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x661ec4]
> V [libjvm.so+0x662523]
> V [libjvm.so+0x9e398d]
> V [libjvm.so+0x9e2326]
> V [libjvm.so+0x9e2b50]
> V [libjvm.so+0x42c099]
> V [libjvm.so+0x9dc786]
> V [libjvm.so+0x6a5edf]
> V [libjvm.so+0x6a70cb] JVM_DefineClass+0xbb
> V [libjvm.so+0xa31ea5]
> V [libjvm.so+0xa37ea7]
> J 4842
> sun.misc.Unsafe.defineClass(Ljava/lang/String;[BIILjava/lang/ClassLoader;Ljava/security/ProtectionDomain;)Ljava/lang/Class;
> (0 bytes) @ 0x7fa0c7af120b [0x7fa0c7af1100+0x10b]
> J 13229 C2 sun.reflect.MethodAccessorGenerator$1.run()Ljava/lang/Object; (5
> bytes) @ 0x7fa0c8cf2a74 [0x7fa0c8cf2940+0x134]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6b5949] JVM_DoPrivileged+0x429
> J 1035
> java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;
> (0 bytes) @ 0x7fa0c7220c7f [0x7fa0c7220bc0+0xbf]
> J 20421 C2
> sun.reflect.MethodAccessorGenerator.generate(Ljava/lang/Class;Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;IZZLjava/lang/Class;)Lsun/reflect/MagicAccessorImpl;
> (762 bytes) @ 0x7fa0c89bb848 [0x7fa0c89b9da0+0x1aa8]
> J 4163 C2
> sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (104 bytes) @ 0x7fa0c789cca8 [0x7fa0c789c8c0+0x3e8]
> J 2379 C2 org.apache.impala.hive.executor.UdfExecutor.evaluate()V (396 bytes)
> @ 0x7fa0c711c638 [0x7fa0c711c400+0x238]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6822d7]
> V [libjvm.so+0x6862c9]
> C [impalad+0x2a004fa] JNIEnv_::CallNonvirtualVoidMethodA(_jobject*,
> _jclass*, _jmethodID*, jvalue const*)+0x40
> C [impalad+0x29fe4ff]
> impala::HiveUdfCall::Evaluate(impala::ScalarExprEvaluator*, impala::TupleRow
> const*) const+0x44b
> C [impalad+0x29ffde9]
> impala::HiveUdfCall::GetSmallIntVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0xbb
> C [impalad+0x2a0948a]
> impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&,
> impala::TupleRow const*)+0x14c
> C [impalad+0x2a48eb1]
> impala::ScalarFnCall::EvaluateNonConstantChildren(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x9d
> C [impalad+0x2a4abba] impala_udf::BooleanVal
> impala::ScalarFnCall::InterpretEval(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x18c
> C
[jira] [Commented] (IMPALA-7069) Java UDF tests can trigger a crash in Java ClassLoader
[
https://issues.apache.org/jira/browse/IMPALA-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503706#comment-16503706
]
Tim Armstrong commented on IMPALA-7069:
---
I wonder if the system memory pressure is somehow causing malloc() to return
null and send the JVM down a weird path or something.
> Java UDF tests can trigger a crash in Java ClassLoader
> --
>
> Key: IMPALA-7069
> URL: https://issues.apache.org/jira/browse/IMPALA-7069
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Taras Bobrovytsky
>Priority: Blocker
> Labels: broken-build, crash, flaky
> Attachments: hs_err_pid22764.log, hs_err_pid29246.log,
> hs_err_pid8975.log, hs_err_pid9694.log
>
>
> I hit this crash on a GVO, but was able to reproduce it on master on my
> desktop.
> Repro steps:
> {code}
> git checkout c1362afb9a072e49df470d9068d44cdbdf5cdec5
> ./buildall.sh -debug -noclean -notests -skiptests -ninja
> start-impala-cluster.py
> while impala-py.test tests/query_test/test_udfs.py -k 'hive or java or jar'
> -n4 --verbose; do date; done
> {code}
> I generally hit the crash within a hour of looping the test.
> {noformat}
> Stack: [0x7fa04791f000,0x7fa04812], sp=0x7fa04811aff0, free
> space=8175k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> V [libjvm.so+0x8a8107]
> V [libjvm.so+0x96cf5f]
> v ~RuntimeStub::_complete_monitor_locking_Java
> J 2758 C2
> java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object;
> (362 bytes) @ 0x7fa0c73637d4 [0x7fa0c7362d00+0xad4]
> J 2311 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122
> bytes) @ 0x7fa0c70a09a8 [0x7fa0c70a08e0+0xc8]
> J 3953 C2
> java.net.FactoryURLClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;
> (40 bytes) @ 0x7fa0c71ce0f0 [0x7fa0c71ce0a0+0x50]
> J 2987 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (7
> bytes) @ 0x7fa0c72ddb64 [0x7fa0c72ddb20+0x44]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x661ec4]
> V [libjvm.so+0x662523]
> V [libjvm.so+0x9e398d]
> V [libjvm.so+0x9e2326]
> V [libjvm.so+0x9e2b50]
> V [libjvm.so+0x42c099]
> V [libjvm.so+0x9dc786]
> V [libjvm.so+0x6a5edf]
> V [libjvm.so+0x6a70cb] JVM_DefineClass+0xbb
> V [libjvm.so+0xa31ea5]
> V [libjvm.so+0xa37ea7]
> J 4842
> sun.misc.Unsafe.defineClass(Ljava/lang/String;[BIILjava/lang/ClassLoader;Ljava/security/ProtectionDomain;)Ljava/lang/Class;
> (0 bytes) @ 0x7fa0c7af120b [0x7fa0c7af1100+0x10b]
> J 13229 C2 sun.reflect.MethodAccessorGenerator$1.run()Ljava/lang/Object; (5
> bytes) @ 0x7fa0c8cf2a74 [0x7fa0c8cf2940+0x134]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6b5949] JVM_DoPrivileged+0x429
> J 1035
> java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;
> (0 bytes) @ 0x7fa0c7220c7f [0x7fa0c7220bc0+0xbf]
> J 20421 C2
> sun.reflect.MethodAccessorGenerator.generate(Ljava/lang/Class;Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;IZZLjava/lang/Class;)Lsun/reflect/MagicAccessorImpl;
> (762 bytes) @ 0x7fa0c89bb848 [0x7fa0c89b9da0+0x1aa8]
> J 4163 C2
> sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (104 bytes) @ 0x7fa0c789cca8 [0x7fa0c789c8c0+0x3e8]
> J 2379 C2 org.apache.impala.hive.executor.UdfExecutor.evaluate()V (396 bytes)
> @ 0x7fa0c711c638 [0x7fa0c711c400+0x238]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6822d7]
> V [libjvm.so+0x6862c9]
> C [impalad+0x2a004fa] JNIEnv_::CallNonvirtualVoidMethodA(_jobject*,
> _jclass*, _jmethodID*, jvalue const*)+0x40
> C [impalad+0x29fe4ff]
> impala::HiveUdfCall::Evaluate(impala::ScalarExprEvaluator*, impala::TupleRow
> const*) const+0x44b
> C [impalad+0x29ffde9]
> impala::HiveUdfCall::GetSmallIntVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0xbb
> C [impalad+0x2a0948a]
> impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&,
> impala::TupleRow const*)+0x14c
> C [impalad+0x2a48eb1]
> impala::ScalarFnCall::EvaluateNonConstantChildren(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x9d
> C [impalad+0x2a4abba] impala_udf::BooleanVal
> impala::ScalarFnCall::InterpretEval(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x18c
> C [impalad+0x2a4907d]
> impala::ScalarFnCall::GetBooleanVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x179
> C [impalad+0x2a09c7f]
>
[jira] [Commented] (IMPALA-7069) Java UDF tests can trigger a crash in Java ClassLoader
[
https://issues.apache.org/jira/browse/IMPALA-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503690#comment-16503690
]
Vuk Ercegovac commented on IMPALA-7069:
---
Just checked that commit.. was 4/28. Oldest change to tests where we increase
the workload that I'm aware of is here:
241031d339a7a78ccc75f5c6c6146028bb9c76db. Would be interesting to see if the
repro shows up prior (either due to the test change or system change).
> Java UDF tests can trigger a crash in Java ClassLoader
> --
>
> Key: IMPALA-7069
> URL: https://issues.apache.org/jira/browse/IMPALA-7069
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Taras Bobrovytsky
>Priority: Blocker
> Labels: broken-build, crash, flaky
> Attachments: hs_err_pid22764.log, hs_err_pid29246.log,
> hs_err_pid8975.log, hs_err_pid9694.log
>
>
> I hit this crash on a GVO, but was able to reproduce it on master on my
> desktop.
> Repro steps:
> {code}
> git checkout c1362afb9a072e49df470d9068d44cdbdf5cdec5
> ./buildall.sh -debug -noclean -notests -skiptests -ninja
> start-impala-cluster.py
> while impala-py.test tests/query_test/test_udfs.py -k 'hive or java or jar'
> -n4 --verbose; do date; done
> {code}
> I generally hit the crash within a hour of looping the test.
> {noformat}
> Stack: [0x7fa04791f000,0x7fa04812], sp=0x7fa04811aff0, free
> space=8175k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> V [libjvm.so+0x8a8107]
> V [libjvm.so+0x96cf5f]
> v ~RuntimeStub::_complete_monitor_locking_Java
> J 2758 C2
> java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object;
> (362 bytes) @ 0x7fa0c73637d4 [0x7fa0c7362d00+0xad4]
> J 2311 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122
> bytes) @ 0x7fa0c70a09a8 [0x7fa0c70a08e0+0xc8]
> J 3953 C2
> java.net.FactoryURLClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;
> (40 bytes) @ 0x7fa0c71ce0f0 [0x7fa0c71ce0a0+0x50]
> J 2987 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (7
> bytes) @ 0x7fa0c72ddb64 [0x7fa0c72ddb20+0x44]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x661ec4]
> V [libjvm.so+0x662523]
> V [libjvm.so+0x9e398d]
> V [libjvm.so+0x9e2326]
> V [libjvm.so+0x9e2b50]
> V [libjvm.so+0x42c099]
> V [libjvm.so+0x9dc786]
> V [libjvm.so+0x6a5edf]
> V [libjvm.so+0x6a70cb] JVM_DefineClass+0xbb
> V [libjvm.so+0xa31ea5]
> V [libjvm.so+0xa37ea7]
> J 4842
> sun.misc.Unsafe.defineClass(Ljava/lang/String;[BIILjava/lang/ClassLoader;Ljava/security/ProtectionDomain;)Ljava/lang/Class;
> (0 bytes) @ 0x7fa0c7af120b [0x7fa0c7af1100+0x10b]
> J 13229 C2 sun.reflect.MethodAccessorGenerator$1.run()Ljava/lang/Object; (5
> bytes) @ 0x7fa0c8cf2a74 [0x7fa0c8cf2940+0x134]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6b5949] JVM_DoPrivileged+0x429
> J 1035
> java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;
> (0 bytes) @ 0x7fa0c7220c7f [0x7fa0c7220bc0+0xbf]
> J 20421 C2
> sun.reflect.MethodAccessorGenerator.generate(Ljava/lang/Class;Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;IZZLjava/lang/Class;)Lsun/reflect/MagicAccessorImpl;
> (762 bytes) @ 0x7fa0c89bb848 [0x7fa0c89b9da0+0x1aa8]
> J 4163 C2
> sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (104 bytes) @ 0x7fa0c789cca8 [0x7fa0c789c8c0+0x3e8]
> J 2379 C2 org.apache.impala.hive.executor.UdfExecutor.evaluate()V (396 bytes)
> @ 0x7fa0c711c638 [0x7fa0c711c400+0x238]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6822d7]
> V [libjvm.so+0x6862c9]
> C [impalad+0x2a004fa] JNIEnv_::CallNonvirtualVoidMethodA(_jobject*,
> _jclass*, _jmethodID*, jvalue const*)+0x40
> C [impalad+0x29fe4ff]
> impala::HiveUdfCall::Evaluate(impala::ScalarExprEvaluator*, impala::TupleRow
> const*) const+0x44b
> C [impalad+0x29ffde9]
> impala::HiveUdfCall::GetSmallIntVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0xbb
> C [impalad+0x2a0948a]
> impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&,
> impala::TupleRow const*)+0x14c
> C [impalad+0x2a48eb1]
> impala::ScalarFnCall::EvaluateNonConstantChildren(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x9d
> C [impalad+0x2a4abba] impala_udf::BooleanVal
> impala::ScalarFnCall::InterpretEval(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x18c
> C [impalad+0x2a4907d]
>
[jira] [Commented] (IMPALA-7069) Java UDF tests can trigger a crash in Java ClassLoader
[
https://issues.apache.org/jira/browse/IMPALA-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503671#comment-16503671
]
Vuk Ercegovac commented on IMPALA-7069:
---
>From scanning build status', I noticed that udf test failures were often
>co-occuring with that test debug spill test. Was not sure if the udf test was
>the culprit or just unlucky since it is one of the longest tests, so has a
>higher chance of running during other crashes. Either way, one thing to try
>would be to reduce the tests that cause the issue reported here and see if it
>can be blamed on the recently (1-2 months) added multi-threaded tests.
> Java UDF tests can trigger a crash in Java ClassLoader
> --
>
> Key: IMPALA-7069
> URL: https://issues.apache.org/jira/browse/IMPALA-7069
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Taras Bobrovytsky
>Priority: Blocker
> Labels: broken-build, crash, flaky
> Attachments: hs_err_pid22764.log, hs_err_pid29246.log,
> hs_err_pid8975.log, hs_err_pid9694.log
>
>
> I hit this crash on a GVO, but was able to reproduce it on master on my
> desktop.
> Repro steps:
> {code}
> git checkout c1362afb9a072e49df470d9068d44cdbdf5cdec5
> ./buildall.sh -debug -noclean -notests -skiptests -ninja
> start-impala-cluster.py
> while impala-py.test tests/query_test/test_udfs.py -k 'hive or java or jar'
> -n4 --verbose; do date; done
> {code}
> I generally hit the crash within a hour of looping the test.
> {noformat}
> Stack: [0x7fa04791f000,0x7fa04812], sp=0x7fa04811aff0, free
> space=8175k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> V [libjvm.so+0x8a8107]
> V [libjvm.so+0x96cf5f]
> v ~RuntimeStub::_complete_monitor_locking_Java
> J 2758 C2
> java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object;
> (362 bytes) @ 0x7fa0c73637d4 [0x7fa0c7362d00+0xad4]
> J 2311 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122
> bytes) @ 0x7fa0c70a09a8 [0x7fa0c70a08e0+0xc8]
> J 3953 C2
> java.net.FactoryURLClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;
> (40 bytes) @ 0x7fa0c71ce0f0 [0x7fa0c71ce0a0+0x50]
> J 2987 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (7
> bytes) @ 0x7fa0c72ddb64 [0x7fa0c72ddb20+0x44]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x661ec4]
> V [libjvm.so+0x662523]
> V [libjvm.so+0x9e398d]
> V [libjvm.so+0x9e2326]
> V [libjvm.so+0x9e2b50]
> V [libjvm.so+0x42c099]
> V [libjvm.so+0x9dc786]
> V [libjvm.so+0x6a5edf]
> V [libjvm.so+0x6a70cb] JVM_DefineClass+0xbb
> V [libjvm.so+0xa31ea5]
> V [libjvm.so+0xa37ea7]
> J 4842
> sun.misc.Unsafe.defineClass(Ljava/lang/String;[BIILjava/lang/ClassLoader;Ljava/security/ProtectionDomain;)Ljava/lang/Class;
> (0 bytes) @ 0x7fa0c7af120b [0x7fa0c7af1100+0x10b]
> J 13229 C2 sun.reflect.MethodAccessorGenerator$1.run()Ljava/lang/Object; (5
> bytes) @ 0x7fa0c8cf2a74 [0x7fa0c8cf2940+0x134]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6b5949] JVM_DoPrivileged+0x429
> J 1035
> java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;
> (0 bytes) @ 0x7fa0c7220c7f [0x7fa0c7220bc0+0xbf]
> J 20421 C2
> sun.reflect.MethodAccessorGenerator.generate(Ljava/lang/Class;Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;IZZLjava/lang/Class;)Lsun/reflect/MagicAccessorImpl;
> (762 bytes) @ 0x7fa0c89bb848 [0x7fa0c89b9da0+0x1aa8]
> J 4163 C2
> sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (104 bytes) @ 0x7fa0c789cca8 [0x7fa0c789c8c0+0x3e8]
> J 2379 C2 org.apache.impala.hive.executor.UdfExecutor.evaluate()V (396 bytes)
> @ 0x7fa0c711c638 [0x7fa0c711c400+0x238]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6822d7]
> V [libjvm.so+0x6862c9]
> C [impalad+0x2a004fa] JNIEnv_::CallNonvirtualVoidMethodA(_jobject*,
> _jclass*, _jmethodID*, jvalue const*)+0x40
> C [impalad+0x29fe4ff]
> impala::HiveUdfCall::Evaluate(impala::ScalarExprEvaluator*, impala::TupleRow
> const*) const+0x44b
> C [impalad+0x29ffde9]
> impala::HiveUdfCall::GetSmallIntVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0xbb
> C [impalad+0x2a0948a]
> impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&,
> impala::TupleRow const*)+0x14c
> C [impalad+0x2a48eb1]
> impala::ScalarFnCall::EvaluateNonConstantChildren(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x9d
> C
[jira] [Commented] (IMPALA-7069) Java UDF tests can trigger a crash in Java ClassLoader
[
https://issues.apache.org/jira/browse/IMPALA-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503637#comment-16503637
]
Tim Armstrong commented on IMPALA-7069:
---
[~tarasbob] that's interesting. I remember [~vukercegovac] mentioned that he
thought the repro rate might have increased because of changes to the UDF tests.
> Java UDF tests can trigger a crash in Java ClassLoader
> --
>
> Key: IMPALA-7069
> URL: https://issues.apache.org/jira/browse/IMPALA-7069
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Taras Bobrovytsky
>Priority: Blocker
> Labels: broken-build, crash, flaky
> Attachments: hs_err_pid22764.log, hs_err_pid29246.log,
> hs_err_pid8975.log, hs_err_pid9694.log
>
>
> I hit this crash on a GVO, but was able to reproduce it on master on my
> desktop.
> Repro steps:
> {code}
> git checkout c1362afb9a072e49df470d9068d44cdbdf5cdec5
> ./buildall.sh -debug -noclean -notests -skiptests -ninja
> start-impala-cluster.py
> while impala-py.test tests/query_test/test_udfs.py -k 'hive or java or jar'
> -n4 --verbose; do date; done
> {code}
> I generally hit the crash within a hour of looping the test.
> {noformat}
> Stack: [0x7fa04791f000,0x7fa04812], sp=0x7fa04811aff0, free
> space=8175k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> V [libjvm.so+0x8a8107]
> V [libjvm.so+0x96cf5f]
> v ~RuntimeStub::_complete_monitor_locking_Java
> J 2758 C2
> java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object;
> (362 bytes) @ 0x7fa0c73637d4 [0x7fa0c7362d00+0xad4]
> J 2311 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122
> bytes) @ 0x7fa0c70a09a8 [0x7fa0c70a08e0+0xc8]
> J 3953 C2
> java.net.FactoryURLClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;
> (40 bytes) @ 0x7fa0c71ce0f0 [0x7fa0c71ce0a0+0x50]
> J 2987 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (7
> bytes) @ 0x7fa0c72ddb64 [0x7fa0c72ddb20+0x44]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x661ec4]
> V [libjvm.so+0x662523]
> V [libjvm.so+0x9e398d]
> V [libjvm.so+0x9e2326]
> V [libjvm.so+0x9e2b50]
> V [libjvm.so+0x42c099]
> V [libjvm.so+0x9dc786]
> V [libjvm.so+0x6a5edf]
> V [libjvm.so+0x6a70cb] JVM_DefineClass+0xbb
> V [libjvm.so+0xa31ea5]
> V [libjvm.so+0xa37ea7]
> J 4842
> sun.misc.Unsafe.defineClass(Ljava/lang/String;[BIILjava/lang/ClassLoader;Ljava/security/ProtectionDomain;)Ljava/lang/Class;
> (0 bytes) @ 0x7fa0c7af120b [0x7fa0c7af1100+0x10b]
> J 13229 C2 sun.reflect.MethodAccessorGenerator$1.run()Ljava/lang/Object; (5
> bytes) @ 0x7fa0c8cf2a74 [0x7fa0c8cf2940+0x134]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6b5949] JVM_DoPrivileged+0x429
> J 1035
> java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;
> (0 bytes) @ 0x7fa0c7220c7f [0x7fa0c7220bc0+0xbf]
> J 20421 C2
> sun.reflect.MethodAccessorGenerator.generate(Ljava/lang/Class;Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;IZZLjava/lang/Class;)Lsun/reflect/MagicAccessorImpl;
> (762 bytes) @ 0x7fa0c89bb848 [0x7fa0c89b9da0+0x1aa8]
> J 4163 C2
> sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (104 bytes) @ 0x7fa0c789cca8 [0x7fa0c789c8c0+0x3e8]
> J 2379 C2 org.apache.impala.hive.executor.UdfExecutor.evaluate()V (396 bytes)
> @ 0x7fa0c711c638 [0x7fa0c711c400+0x238]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6822d7]
> V [libjvm.so+0x6862c9]
> C [impalad+0x2a004fa] JNIEnv_::CallNonvirtualVoidMethodA(_jobject*,
> _jclass*, _jmethodID*, jvalue const*)+0x40
> C [impalad+0x29fe4ff]
> impala::HiveUdfCall::Evaluate(impala::ScalarExprEvaluator*, impala::TupleRow
> const*) const+0x44b
> C [impalad+0x29ffde9]
> impala::HiveUdfCall::GetSmallIntVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0xbb
> C [impalad+0x2a0948a]
> impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&,
> impala::TupleRow const*)+0x14c
> C [impalad+0x2a48eb1]
> impala::ScalarFnCall::EvaluateNonConstantChildren(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x9d
> C [impalad+0x2a4abba] impala_udf::BooleanVal
> impala::ScalarFnCall::InterpretEval(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x18c
> C [impalad+0x2a4907d]
> impala::ScalarFnCall::GetBooleanVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x179
> C
[jira] [Commented] (IMPALA-7069) Java UDF tests can trigger a crash in Java ClassLoader
[
https://issues.apache.org/jira/browse/IMPALA-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502516#comment-16502516
]
Taras Bobrovytsky commented on IMPALA-7069:
---
I was able to reproduce this on a pretty old commit:
d7bba821923b96272c254840facd3d870b1ebe36 Looks this was not introduced very
recently.
> Java UDF tests can trigger a crash in Java ClassLoader
> --
>
> Key: IMPALA-7069
> URL: https://issues.apache.org/jira/browse/IMPALA-7069
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Taras Bobrovytsky
>Priority: Blocker
> Labels: broken-build, crash, flaky
> Attachments: hs_err_pid22764.log, hs_err_pid29246.log,
> hs_err_pid8975.log, hs_err_pid9694.log
>
>
> I hit this crash on a GVO, but was able to reproduce it on master on my
> desktop.
> Repro steps:
> {code}
> git checkout c1362afb9a072e49df470d9068d44cdbdf5cdec5
> ./buildall.sh -debug -noclean -notests -skiptests -ninja
> start-impala-cluster.py
> while impala-py.test tests/query_test/test_udfs.py -k 'hive or java or jar'
> -n4 --verbose; do date; done
> {code}
> I generally hit the crash within a hour of looping the test.
> {noformat}
> Stack: [0x7fa04791f000,0x7fa04812], sp=0x7fa04811aff0, free
> space=8175k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> V [libjvm.so+0x8a8107]
> V [libjvm.so+0x96cf5f]
> v ~RuntimeStub::_complete_monitor_locking_Java
> J 2758 C2
> java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object;
> (362 bytes) @ 0x7fa0c73637d4 [0x7fa0c7362d00+0xad4]
> J 2311 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122
> bytes) @ 0x7fa0c70a09a8 [0x7fa0c70a08e0+0xc8]
> J 3953 C2
> java.net.FactoryURLClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;
> (40 bytes) @ 0x7fa0c71ce0f0 [0x7fa0c71ce0a0+0x50]
> J 2987 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (7
> bytes) @ 0x7fa0c72ddb64 [0x7fa0c72ddb20+0x44]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x661ec4]
> V [libjvm.so+0x662523]
> V [libjvm.so+0x9e398d]
> V [libjvm.so+0x9e2326]
> V [libjvm.so+0x9e2b50]
> V [libjvm.so+0x42c099]
> V [libjvm.so+0x9dc786]
> V [libjvm.so+0x6a5edf]
> V [libjvm.so+0x6a70cb] JVM_DefineClass+0xbb
> V [libjvm.so+0xa31ea5]
> V [libjvm.so+0xa37ea7]
> J 4842
> sun.misc.Unsafe.defineClass(Ljava/lang/String;[BIILjava/lang/ClassLoader;Ljava/security/ProtectionDomain;)Ljava/lang/Class;
> (0 bytes) @ 0x7fa0c7af120b [0x7fa0c7af1100+0x10b]
> J 13229 C2 sun.reflect.MethodAccessorGenerator$1.run()Ljava/lang/Object; (5
> bytes) @ 0x7fa0c8cf2a74 [0x7fa0c8cf2940+0x134]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6b5949] JVM_DoPrivileged+0x429
> J 1035
> java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;
> (0 bytes) @ 0x7fa0c7220c7f [0x7fa0c7220bc0+0xbf]
> J 20421 C2
> sun.reflect.MethodAccessorGenerator.generate(Ljava/lang/Class;Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;IZZLjava/lang/Class;)Lsun/reflect/MagicAccessorImpl;
> (762 bytes) @ 0x7fa0c89bb848 [0x7fa0c89b9da0+0x1aa8]
> J 4163 C2
> sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (104 bytes) @ 0x7fa0c789cca8 [0x7fa0c789c8c0+0x3e8]
> J 2379 C2 org.apache.impala.hive.executor.UdfExecutor.evaluate()V (396 bytes)
> @ 0x7fa0c711c638 [0x7fa0c711c400+0x238]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6822d7]
> V [libjvm.so+0x6862c9]
> C [impalad+0x2a004fa] JNIEnv_::CallNonvirtualVoidMethodA(_jobject*,
> _jclass*, _jmethodID*, jvalue const*)+0x40
> C [impalad+0x29fe4ff]
> impala::HiveUdfCall::Evaluate(impala::ScalarExprEvaluator*, impala::TupleRow
> const*) const+0x44b
> C [impalad+0x29ffde9]
> impala::HiveUdfCall::GetSmallIntVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0xbb
> C [impalad+0x2a0948a]
> impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&,
> impala::TupleRow const*)+0x14c
> C [impalad+0x2a48eb1]
> impala::ScalarFnCall::EvaluateNonConstantChildren(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x9d
> C [impalad+0x2a4abba] impala_udf::BooleanVal
> impala::ScalarFnCall::InterpretEval(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x18c
> C [impalad+0x2a4907d]
> impala::ScalarFnCall::GetBooleanVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x179
> C [impalad+0x2a09c7f]
[jira] [Commented] (IMPALA-7069) Java UDF tests can trigger a crash in Java ClassLoader
[
https://issues.apache.org/jira/browse/IMPALA-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501330#comment-16501330
]
ASF subversion and git services commented on IMPALA-7069:
-
Commit 1693e149b714a0c5bbdd603bc3fc70a03502830c in impala's branch
refs/heads/master from [~tarmstr...@cloudera.com]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=1693e14 ]
Collect hs_err_pid*.log files in logs/
These JVM error dumps are often useful for debugging job failures. We
should put them into the logs/ directory so they can be collected
along with other logs.
This was motivated by IMPALA-7069.
Testing:
Induced a segfault by changing the code, checked that the file ended up
in the logs/ subdirectory.
Change-Id: I753f6acb63cec600432675cfb6e5dd1e1427b5ae
Reviewed-on: http://gerrit.cloudera.org:8080/10594
Reviewed-by: Lars Volker
Tested-by: Impala Public Jenkins
Reviewed-by: Tim Armstrong
> Java UDF tests can trigger a crash in Java ClassLoader
> --
>
> Key: IMPALA-7069
> URL: https://issues.apache.org/jira/browse/IMPALA-7069
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Taras Bobrovytsky
>Priority: Critical
> Labels: crash, flaky
> Attachments: hs_err_pid22764.log, hs_err_pid29246.log,
> hs_err_pid8975.log, hs_err_pid9694.log
>
>
> I hit this crash on a GVO, but was able to reproduce it on master on my
> desktop.
> Repro steps:
> {code}
> git checkout c1362afb9a072e49df470d9068d44cdbdf5cdec5
> ./buildall.sh -debug -noclean -notests -skiptests -ninja
> start-impala-cluster.py
> while impala-py.test tests/query_test/test_udfs.py -k 'hive or java or jar'
> -n4 --verbose; do date; done
> {code}
> I generally hit the crash within a hour of looping the test.
> {noformat}
> Stack: [0x7fa04791f000,0x7fa04812], sp=0x7fa04811aff0, free
> space=8175k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> V [libjvm.so+0x8a8107]
> V [libjvm.so+0x96cf5f]
> v ~RuntimeStub::_complete_monitor_locking_Java
> J 2758 C2
> java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object;
> (362 bytes) @ 0x7fa0c73637d4 [0x7fa0c7362d00+0xad4]
> J 2311 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122
> bytes) @ 0x7fa0c70a09a8 [0x7fa0c70a08e0+0xc8]
> J 3953 C2
> java.net.FactoryURLClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;
> (40 bytes) @ 0x7fa0c71ce0f0 [0x7fa0c71ce0a0+0x50]
> J 2987 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (7
> bytes) @ 0x7fa0c72ddb64 [0x7fa0c72ddb20+0x44]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x661ec4]
> V [libjvm.so+0x662523]
> V [libjvm.so+0x9e398d]
> V [libjvm.so+0x9e2326]
> V [libjvm.so+0x9e2b50]
> V [libjvm.so+0x42c099]
> V [libjvm.so+0x9dc786]
> V [libjvm.so+0x6a5edf]
> V [libjvm.so+0x6a70cb] JVM_DefineClass+0xbb
> V [libjvm.so+0xa31ea5]
> V [libjvm.so+0xa37ea7]
> J 4842
> sun.misc.Unsafe.defineClass(Ljava/lang/String;[BIILjava/lang/ClassLoader;Ljava/security/ProtectionDomain;)Ljava/lang/Class;
> (0 bytes) @ 0x7fa0c7af120b [0x7fa0c7af1100+0x10b]
> J 13229 C2 sun.reflect.MethodAccessorGenerator$1.run()Ljava/lang/Object; (5
> bytes) @ 0x7fa0c8cf2a74 [0x7fa0c8cf2940+0x134]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6b5949] JVM_DoPrivileged+0x429
> J 1035
> java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;
> (0 bytes) @ 0x7fa0c7220c7f [0x7fa0c7220bc0+0xbf]
> J 20421 C2
> sun.reflect.MethodAccessorGenerator.generate(Ljava/lang/Class;Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;IZZLjava/lang/Class;)Lsun/reflect/MagicAccessorImpl;
> (762 bytes) @ 0x7fa0c89bb848 [0x7fa0c89b9da0+0x1aa8]
> J 4163 C2
> sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (104 bytes) @ 0x7fa0c789cca8 [0x7fa0c789c8c0+0x3e8]
> J 2379 C2 org.apache.impala.hive.executor.UdfExecutor.evaluate()V (396 bytes)
> @ 0x7fa0c711c638 [0x7fa0c711c400+0x238]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6822d7]
> V [libjvm.so+0x6862c9]
> C [impalad+0x2a004fa] JNIEnv_::CallNonvirtualVoidMethodA(_jobject*,
> _jclass*, _jmethodID*, jvalue const*)+0x40
> C [impalad+0x29fe4ff]
> impala::HiveUdfCall::Evaluate(impala::ScalarExprEvaluator*, impala::TupleRow
> const*) const+0x44b
> C [impalad+0x29ffde9]
> impala::HiveUdfCall::GetSmallIntVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*)
[jira] [Commented] (IMPALA-7069) Java UDF tests can trigger a crash in Java ClassLoader
[
https://issues.apache.org/jira/browse/IMPALA-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500696#comment-16500696
]
Tim Armstrong commented on IMPALA-7069:
---
I saw another crash with this backtrace:
{noformat}
CORE: ./core.1528115186.21531.impalad
BINARY: ./be/build/latest/service/impalad
Core was generated by
`/data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/bu'.
Program terminated with signal 6, Aborted.
#0 0x0039f74328e5 in raise () from /lib64/libc.so.6
To enable execution of this file add
add-auto-load-safe-path
/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/gcc-4.9.2/lib64/libstdc++.so.6.0.20-gdb.py
line to your configuration file "/var/lib/jenkins/.gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/var/lib/jenkins/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
info "(gdb)Auto-loading safe path"
#0 0x0039f74328e5 in raise () from /lib64/libc.so.6
#1 0x0039f74340c5 in abort () from /lib64/libc.so.6
#2 0x7f04924521a5 in os::abort(bool) () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#3 0x7f04925e2843 in VMError::report_and_die() () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#4 0x7f04925e2cfe in crash_handler(int, siginfo*, void*) () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#5 0x7f0492451692 in os::Linux::chained_handler(int, siginfo*, void*) ()
from /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#6 0x7f04924574cd in JVM_handle_linux_signal () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#7 0x7f049244e4f3 in signalHandler(int, siginfo*, void*) () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#8
#9 0x7f04924413bf in oopDesc::print_on(outputStream*) const () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#10 0x7f0492456e12 in os::print_register_info(outputStream*, void*) () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#11 0x7f04925e0e24 in VMError::report(outputStream*) () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#12 0x7f04925e243a in VMError::report_and_die() () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#13 0x7f0492457562 in JVM_handle_linux_signal () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#14 0x7f049244e4f3 in signalHandler(int, siginfo*, void*) () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#15
#16 0x7f049240f0bd in MutableSpace::object_iterate(ObjectClosure*) () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#17 0x7f04924bc9b0 in PSYoungGen::object_iterate(ObjectClosure*) () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#18 0x7f04924b8647 in PSScavenge::clean_up_failed_promotion() () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#19 0x7f04924b9f25 in PSScavenge::invoke_no_policy() () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#20 0x7f04924ba271 in PSScavenge::invoke() () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#21 0x7f0492472ed0 in ParallelScavengeHeap::failed_mem_allocate(unsigned
long) () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#22 0x7f04925e3af7 in VM_ParallelGCFailedAllocation::doit() () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#23 0x7f04925eb215 in VM_Operation::evaluate() () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#24 0x7f04925e960a in VMThread::evaluate_operation(VM_Operation*) () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#25 0x7f04925e998e in VMThread::loop() () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#26 0x7f04925e9e00 in VMThread::run() () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#27 0x7f04924530c8 in java_start(Thread*) () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#28 0x0039f7807851 in start_thread () from /lib64/libpthread.so.0
#29 0x0039f74e894d in clone () from /lib64/libc.so.6
{noformat}
Some of these tests were running:
{noformat}
05:26:15
query_test/test_udfs.py::TestUdfExecution::test_udf_constant_folding[exec_option:
{'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
'exec_single_node_rows_threshold': 100, 'enable_expr_rewrites': False} |
table_format:
[jira] [Commented] (IMPALA-7069) Java UDF tests can trigger a crash in Java ClassLoader
[
https://issues.apache.org/jira/browse/IMPALA-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489522#comment-16489522
]
Tim Armstrong commented on IMPALA-7069:
---
I saw it on a commit branched off fd7e7c93c5d2ae153784548a2f83f423e85dda43 .
AFAIK we didn't see it before yesterday.
> Java UDF tests can trigger a crash in Java ClassLoader
> --
>
> Key: IMPALA-7069
> URL: https://issues.apache.org/jira/browse/IMPALA-7069
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Vuk Ercegovac
>Priority: Critical
> Labels: crash, flaky
> Attachments: hs_err_pid22764.log, hs_err_pid29246.log,
> hs_err_pid8975.log, hs_err_pid9694.log
>
>
> I hit this crash on a GVO, but was able to reproduce it on master on my
> desktop.
> Repro steps:
> {code}
> git checkout c1362afb9a072e49df470d9068d44cdbdf5cdec5
> ./buildall.sh -debug -noclean -notests -skiptests -ninja
> start-impala-cluster.py
> while impala-py.test tests/query_test/test_udfs.py -k 'hive or java or jar'
> -n4 --verbose; do date; done
> {code}
> I generally hit the crash within a hour of looping the test.
> {noformat}
> Stack: [0x7fa04791f000,0x7fa04812], sp=0x7fa04811aff0, free
> space=8175k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> V [libjvm.so+0x8a8107]
> V [libjvm.so+0x96cf5f]
> v ~RuntimeStub::_complete_monitor_locking_Java
> J 2758 C2
> java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object;
> (362 bytes) @ 0x7fa0c73637d4 [0x7fa0c7362d00+0xad4]
> J 2311 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122
> bytes) @ 0x7fa0c70a09a8 [0x7fa0c70a08e0+0xc8]
> J 3953 C2
> java.net.FactoryURLClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;
> (40 bytes) @ 0x7fa0c71ce0f0 [0x7fa0c71ce0a0+0x50]
> J 2987 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (7
> bytes) @ 0x7fa0c72ddb64 [0x7fa0c72ddb20+0x44]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x661ec4]
> V [libjvm.so+0x662523]
> V [libjvm.so+0x9e398d]
> V [libjvm.so+0x9e2326]
> V [libjvm.so+0x9e2b50]
> V [libjvm.so+0x42c099]
> V [libjvm.so+0x9dc786]
> V [libjvm.so+0x6a5edf]
> V [libjvm.so+0x6a70cb] JVM_DefineClass+0xbb
> V [libjvm.so+0xa31ea5]
> V [libjvm.so+0xa37ea7]
> J 4842
> sun.misc.Unsafe.defineClass(Ljava/lang/String;[BIILjava/lang/ClassLoader;Ljava/security/ProtectionDomain;)Ljava/lang/Class;
> (0 bytes) @ 0x7fa0c7af120b [0x7fa0c7af1100+0x10b]
> J 13229 C2 sun.reflect.MethodAccessorGenerator$1.run()Ljava/lang/Object; (5
> bytes) @ 0x7fa0c8cf2a74 [0x7fa0c8cf2940+0x134]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6b5949] JVM_DoPrivileged+0x429
> J 1035
> java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;
> (0 bytes) @ 0x7fa0c7220c7f [0x7fa0c7220bc0+0xbf]
> J 20421 C2
> sun.reflect.MethodAccessorGenerator.generate(Ljava/lang/Class;Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;IZZLjava/lang/Class;)Lsun/reflect/MagicAccessorImpl;
> (762 bytes) @ 0x7fa0c89bb848 [0x7fa0c89b9da0+0x1aa8]
> J 4163 C2
> sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (104 bytes) @ 0x7fa0c789cca8 [0x7fa0c789c8c0+0x3e8]
> J 2379 C2 org.apache.impala.hive.executor.UdfExecutor.evaluate()V (396 bytes)
> @ 0x7fa0c711c638 [0x7fa0c711c400+0x238]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6822d7]
> V [libjvm.so+0x6862c9]
> C [impalad+0x2a004fa] JNIEnv_::CallNonvirtualVoidMethodA(_jobject*,
> _jclass*, _jmethodID*, jvalue const*)+0x40
> C [impalad+0x29fe4ff]
> impala::HiveUdfCall::Evaluate(impala::ScalarExprEvaluator*, impala::TupleRow
> const*) const+0x44b
> C [impalad+0x29ffde9]
> impala::HiveUdfCall::GetSmallIntVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0xbb
> C [impalad+0x2a0948a]
> impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&,
> impala::TupleRow const*)+0x14c
> C [impalad+0x2a48eb1]
> impala::ScalarFnCall::EvaluateNonConstantChildren(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x9d
> C [impalad+0x2a4abba] impala_udf::BooleanVal
> impala::ScalarFnCall::InterpretEval(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x18c
> C [impalad+0x2a4907d]
> impala::ScalarFnCall::GetBooleanVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x179
> C [impalad+0x2a09c7f]
>
[jira] [Commented] (IMPALA-7069) Java UDF tests can trigger a crash in Java ClassLoader
[
https://issues.apache.org/jira/browse/IMPALA-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489466#comment-16489466
]
Dan Hecht commented on IMPALA-7069:
---
Any idea the earliest githash we first saw this at? Presumably it's a
regression, right?
> Java UDF tests can trigger a crash in Java ClassLoader
> --
>
> Key: IMPALA-7069
> URL: https://issues.apache.org/jira/browse/IMPALA-7069
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Vuk Ercegovac
>Priority: Critical
> Labels: crash, flaky
> Attachments: hs_err_pid22764.log, hs_err_pid29246.log,
> hs_err_pid8975.log, hs_err_pid9694.log
>
>
> I hit this crash on a GVO, but was able to reproduce it on master on my
> desktop.
> Repro steps:
> {code}
> git checkout c1362afb9a072e49df470d9068d44cdbdf5cdec5
> ./buildall.sh -debug -noclean -notests -skiptests -ninja
> start-impala-cluster.py
> while impala-py.test tests/query_test/test_udfs.py -k 'hive or java or jar'
> -n4 --verbose; do date; done
> {code}
> I generally hit the crash within a hour of looping the test.
> {noformat}
> Stack: [0x7fa04791f000,0x7fa04812], sp=0x7fa04811aff0, free
> space=8175k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> V [libjvm.so+0x8a8107]
> V [libjvm.so+0x96cf5f]
> v ~RuntimeStub::_complete_monitor_locking_Java
> J 2758 C2
> java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object;
> (362 bytes) @ 0x7fa0c73637d4 [0x7fa0c7362d00+0xad4]
> J 2311 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122
> bytes) @ 0x7fa0c70a09a8 [0x7fa0c70a08e0+0xc8]
> J 3953 C2
> java.net.FactoryURLClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;
> (40 bytes) @ 0x7fa0c71ce0f0 [0x7fa0c71ce0a0+0x50]
> J 2987 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (7
> bytes) @ 0x7fa0c72ddb64 [0x7fa0c72ddb20+0x44]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x661ec4]
> V [libjvm.so+0x662523]
> V [libjvm.so+0x9e398d]
> V [libjvm.so+0x9e2326]
> V [libjvm.so+0x9e2b50]
> V [libjvm.so+0x42c099]
> V [libjvm.so+0x9dc786]
> V [libjvm.so+0x6a5edf]
> V [libjvm.so+0x6a70cb] JVM_DefineClass+0xbb
> V [libjvm.so+0xa31ea5]
> V [libjvm.so+0xa37ea7]
> J 4842
> sun.misc.Unsafe.defineClass(Ljava/lang/String;[BIILjava/lang/ClassLoader;Ljava/security/ProtectionDomain;)Ljava/lang/Class;
> (0 bytes) @ 0x7fa0c7af120b [0x7fa0c7af1100+0x10b]
> J 13229 C2 sun.reflect.MethodAccessorGenerator$1.run()Ljava/lang/Object; (5
> bytes) @ 0x7fa0c8cf2a74 [0x7fa0c8cf2940+0x134]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6b5949] JVM_DoPrivileged+0x429
> J 1035
> java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;
> (0 bytes) @ 0x7fa0c7220c7f [0x7fa0c7220bc0+0xbf]
> J 20421 C2
> sun.reflect.MethodAccessorGenerator.generate(Ljava/lang/Class;Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;IZZLjava/lang/Class;)Lsun/reflect/MagicAccessorImpl;
> (762 bytes) @ 0x7fa0c89bb848 [0x7fa0c89b9da0+0x1aa8]
> J 4163 C2
> sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (104 bytes) @ 0x7fa0c789cca8 [0x7fa0c789c8c0+0x3e8]
> J 2379 C2 org.apache.impala.hive.executor.UdfExecutor.evaluate()V (396 bytes)
> @ 0x7fa0c711c638 [0x7fa0c711c400+0x238]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6822d7]
> V [libjvm.so+0x6862c9]
> C [impalad+0x2a004fa] JNIEnv_::CallNonvirtualVoidMethodA(_jobject*,
> _jclass*, _jmethodID*, jvalue const*)+0x40
> C [impalad+0x29fe4ff]
> impala::HiveUdfCall::Evaluate(impala::ScalarExprEvaluator*, impala::TupleRow
> const*) const+0x44b
> C [impalad+0x29ffde9]
> impala::HiveUdfCall::GetSmallIntVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0xbb
> C [impalad+0x2a0948a]
> impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&,
> impala::TupleRow const*)+0x14c
> C [impalad+0x2a48eb1]
> impala::ScalarFnCall::EvaluateNonConstantChildren(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x9d
> C [impalad+0x2a4abba] impala_udf::BooleanVal
> impala::ScalarFnCall::InterpretEval(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x18c
> C [impalad+0x2a4907d]
> impala::ScalarFnCall::GetBooleanVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x179
> C [impalad+0x2a09c7f]
> impala::ScalarExprEvaluator::GetBooleanVal(impala::TupleRow*)+0x37
> C
[jira] [Commented] (IMPALA-7069) Java UDF tests can trigger a crash in Java ClassLoader
[
https://issues.apache.org/jira/browse/IMPALA-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489358#comment-16489358
]
Tim Armstrong commented on IMPALA-7069:
---
Assigned to Vuk since he's looked at the UDF loading most recently. I attached
the hs_err_pid files from my repro attempts.
> Java UDF tests can trigger a crash in Java ClassLoader
> --
>
> Key: IMPALA-7069
> URL: https://issues.apache.org/jira/browse/IMPALA-7069
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Vuk Ercegovac
>Priority: Critical
> Labels: crash, flaky
> Attachments: hs_err_pid22764.log, hs_err_pid29246.log,
> hs_err_pid8975.log, hs_err_pid9694.log
>
>
> I hit this crash on a GVO, but was able to reproduce it on master on my
> desktop.
> Repro steps:
> {code}
> git checkout c1362afb9a072e49df470d9068d44cdbdf5cdec5
> ./buildall.sh -debug -noclean -notests -skiptests -ninja
> start-impala-cluster.py
> while impala-py.test tests/query_test/test_udfs.py -k 'hive or java or jar'
> -n4 --verbose; do date; done
> {code}
> I generally hit the crash within a hour of looping the test.
> {noformat}
> Stack: [0x7fa04791f000,0x7fa04812], sp=0x7fa04811aff0, free
> space=8175k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> V [libjvm.so+0x8a8107]
> V [libjvm.so+0x96cf5f]
> v ~RuntimeStub::_complete_monitor_locking_Java
> J 2758 C2
> java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object;
> (362 bytes) @ 0x7fa0c73637d4 [0x7fa0c7362d00+0xad4]
> J 2311 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122
> bytes) @ 0x7fa0c70a09a8 [0x7fa0c70a08e0+0xc8]
> J 3953 C2
> java.net.FactoryURLClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;
> (40 bytes) @ 0x7fa0c71ce0f0 [0x7fa0c71ce0a0+0x50]
> J 2987 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (7
> bytes) @ 0x7fa0c72ddb64 [0x7fa0c72ddb20+0x44]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x661ec4]
> V [libjvm.so+0x662523]
> V [libjvm.so+0x9e398d]
> V [libjvm.so+0x9e2326]
> V [libjvm.so+0x9e2b50]
> V [libjvm.so+0x42c099]
> V [libjvm.so+0x9dc786]
> V [libjvm.so+0x6a5edf]
> V [libjvm.so+0x6a70cb] JVM_DefineClass+0xbb
> V [libjvm.so+0xa31ea5]
> V [libjvm.so+0xa37ea7]
> J 4842
> sun.misc.Unsafe.defineClass(Ljava/lang/String;[BIILjava/lang/ClassLoader;Ljava/security/ProtectionDomain;)Ljava/lang/Class;
> (0 bytes) @ 0x7fa0c7af120b [0x7fa0c7af1100+0x10b]
> J 13229 C2 sun.reflect.MethodAccessorGenerator$1.run()Ljava/lang/Object; (5
> bytes) @ 0x7fa0c8cf2a74 [0x7fa0c8cf2940+0x134]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6b5949] JVM_DoPrivileged+0x429
> J 1035
> java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;
> (0 bytes) @ 0x7fa0c7220c7f [0x7fa0c7220bc0+0xbf]
> J 20421 C2
> sun.reflect.MethodAccessorGenerator.generate(Ljava/lang/Class;Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;IZZLjava/lang/Class;)Lsun/reflect/MagicAccessorImpl;
> (762 bytes) @ 0x7fa0c89bb848 [0x7fa0c89b9da0+0x1aa8]
> J 4163 C2
> sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (104 bytes) @ 0x7fa0c789cca8 [0x7fa0c789c8c0+0x3e8]
> J 2379 C2 org.apache.impala.hive.executor.UdfExecutor.evaluate()V (396 bytes)
> @ 0x7fa0c711c638 [0x7fa0c711c400+0x238]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6822d7]
> V [libjvm.so+0x6862c9]
> C [impalad+0x2a004fa] JNIEnv_::CallNonvirtualVoidMethodA(_jobject*,
> _jclass*, _jmethodID*, jvalue const*)+0x40
> C [impalad+0x29fe4ff]
> impala::HiveUdfCall::Evaluate(impala::ScalarExprEvaluator*, impala::TupleRow
> const*) const+0x44b
> C [impalad+0x29ffde9]
> impala::HiveUdfCall::GetSmallIntVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0xbb
> C [impalad+0x2a0948a]
> impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&,
> impala::TupleRow const*)+0x14c
> C [impalad+0x2a48eb1]
> impala::ScalarFnCall::EvaluateNonConstantChildren(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x9d
> C [impalad+0x2a4abba] impala_udf::BooleanVal
> impala::ScalarFnCall::InterpretEval(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x18c
> C [impalad+0x2a4907d]
> impala::ScalarFnCall::GetBooleanVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x179
> C [impalad+0x2a09c7f]
>
