[
https://issues.apache.org/jira/browse/IMPALA-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16505657#comment-16505657
]
Philip Zeyliger commented on IMPALA-7069:
-----------------------------------------
Depending on how we do class loaders, it's entirely possible that we leak Java
memory (as distinct from malloc() memory) too. The easiest way to check that is
to run with some {{JAVA_TOOL_OPTIONS=-verbose:gc}} flags and see what comes
out. (Or configure a GC log file.)
> Java UDF tests can trigger a crash in Java ClassLoader
> ------------------------------------------------------
>
> Key: IMPALA-7069
> URL: https://issues.apache.org/jira/browse/IMPALA-7069
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 3.1.0
> Reporter: Tim Armstrong
> Assignee: Taras Bobrovytsky
> Priority: Blocker
> Labels: broken-build, crash, flaky
> Attachments: hs_err_pid22764.log, hs_err_pid29246.log,
> hs_err_pid8975.log, hs_err_pid9694.log
>
>
> I hit this crash on a GVO, but was able to reproduce it on master on my
> desktop.
> Repro steps:
> {code}
> git checkout c1362afb9a072e49df470d9068d44cdbdf5cdec5
> ./buildall.sh -debug -noclean -notests -skiptests -ninja
> start-impala-cluster.py
> while impala-py.test tests/query_test/test_udfs.py -k 'hive or java or jar'
> -n4 --verbose; do date; done
> {code}
> I generally hit the crash within a hour of looping the test.
> {noformat}
> Stack: [0x00007fa04791f000,0x00007fa048120000], sp=0x00007fa04811aff0, free
> space=8175k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> V [libjvm.so+0x8a8107]
> V [libjvm.so+0x96cf5f]
> v ~RuntimeStub::_complete_monitor_locking_Java
> J 2758 C2
> java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object;
> (362 bytes) @ 0x00007fa0c73637d4 [0x00007fa0c7362d00+0xad4]
> J 2311 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122
> bytes) @ 0x00007fa0c70a09a8 [0x00007fa0c70a08e0+0xc8]
> J 3953 C2
> java.net.FactoryURLClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;
> (40 bytes) @ 0x00007fa0c71ce0f0 [0x00007fa0c71ce0a0+0x50]
> J 2987 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (7
> bytes) @ 0x00007fa0c72ddb64 [0x00007fa0c72ddb20+0x44]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x661ec4]
> V [libjvm.so+0x662523]
> V [libjvm.so+0x9e398d]
> V [libjvm.so+0x9e2326]
> V [libjvm.so+0x9e2b50]
> V [libjvm.so+0x42c099]
> V [libjvm.so+0x9dc786]
> V [libjvm.so+0x6a5edf]
> V [libjvm.so+0x6a70cb] JVM_DefineClass+0xbb
> V [libjvm.so+0xa31ea5]
> V [libjvm.so+0xa37ea7]
> J 4842
> sun.misc.Unsafe.defineClass(Ljava/lang/String;[BIILjava/lang/ClassLoader;Ljava/security/ProtectionDomain;)Ljava/lang/Class;
> (0 bytes) @ 0x00007fa0c7af120b [0x00007fa0c7af1100+0x10b]
> J 13229 C2 sun.reflect.MethodAccessorGenerator$1.run()Ljava/lang/Object; (5
> bytes) @ 0x00007fa0c8cf2a74 [0x00007fa0c8cf2940+0x134]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6b5949] JVM_DoPrivileged+0x429
> J 1035
> java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;
> (0 bytes) @ 0x00007fa0c7220c7f [0x00007fa0c7220bc0+0xbf]
> J 20421 C2
> sun.reflect.MethodAccessorGenerator.generate(Ljava/lang/Class;Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;IZZLjava/lang/Class;)Lsun/reflect/MagicAccessorImpl;
> (762 bytes) @ 0x00007fa0c89bb848 [0x00007fa0c89b9da0+0x1aa8]
> J 4163 C2
> sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (104 bytes) @ 0x00007fa0c789cca8 [0x00007fa0c789c8c0+0x3e8]
> J 2379 C2 org.apache.impala.hive.executor.UdfExecutor.evaluate()V (396 bytes)
> @ 0x00007fa0c711c638 [0x00007fa0c711c400+0x238]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6822d7]
> V [libjvm.so+0x6862c9]
> C [impalad+0x2a004fa] JNIEnv_::CallNonvirtualVoidMethodA(_jobject*,
> _jclass*, _jmethodID*, jvalue const*)+0x40
> C [impalad+0x29fe4ff]
> impala::HiveUdfCall::Evaluate(impala::ScalarExprEvaluator*, impala::TupleRow
> const*) const+0x44b
> C [impalad+0x29ffde9]
> impala::HiveUdfCall::GetSmallIntVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0xbb
> C [impalad+0x2a0948a]
> impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&,
> impala::TupleRow const*)+0x14c
> C [impalad+0x2a48eb1]
> impala::ScalarFnCall::EvaluateNonConstantChildren(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x9d
> C [impalad+0x2a4abba] impala_udf::BooleanVal
> impala::ScalarFnCall::InterpretEval<impala_udf::BooleanVal>(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x18c
> C [impalad+0x2a4907d]
> impala::ScalarFnCall::GetBooleanVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x179
> C [impalad+0x2a09c7f]
> impala::ScalarExprEvaluator::GetBooleanVal(impala::TupleRow*)+0x37
> C [impalad+0x1b70efb]
> impala::ExecNode::EvalPredicate(impala::ScalarExprEvaluator*,
> impala::TupleRow*)+0x23
> C [impalad+0x1b70efb]
> impala::ExecNode::EvalPredicate(impala::ScalarExprEvaluator*,
> impala::TupleRow*)+0x23
> C [impalad+0x1b6fdf0]
> impala::ExecNode::EvalConjuncts(impala::ScalarExprEvaluator* const*, int,
> impala::TupleRow*)+0x42
> C [impalad+0x1bb60e3]
> impala::HdfsScanner::EvalConjuncts(impala::TupleRow*)+0x4d
> C [impalad+0x1bb08fd]
> impala::HdfsScanner::WriteCompleteTuple(impala::MemPool*,
> impala::FieldLocation*, impala::Tuple*, impala::TupleRow*, impala::Tuple*,
> unsigned char*, unsigned char*)+0x1b1
> C [impalad+0x1bc176d]
> impala::HdfsScanner::WriteAlignedTuples(impala::MemPool*, impala::TupleRow*,
> impala::FieldLocation*, int, int, int, int, bool)+0x1f3
> C [impalad+0x1bf93d9] impala::HdfsTextScanner::WriteFields(int, int,
> impala::MemPool*, impala::TupleRow*)+0x779
> C [impalad+0x1bf48be]
> impala::HdfsTextScanner::ProcessRange(impala::RowBatch*, int*)+0x9d8
> C [impalad+0x1bf55e8]
> impala::HdfsTextScanner::GetNextInternal(impala::RowBatch*)+0x422
> C [impalad+0x1baf676] impala::HdfsScanner::ProcessSplit()+0x1ea
> C [impalad+0x1b8695c]
> impala::HdfsScanNode::ProcessSplit(std::vector<impala::FilterContext,
> std::allocator<impala::FilterContext> > const&, impala::MemPool*,
> impala::io::ScanRange*, long*)+0x77a
> C [impalad+0x1b85cfd] impala::HdfsScanNode::ScannerThread(bool, long)+0x3cd
> {noformat}
> I've only tried to hit this on Ubuntu16.04. It repros on at least two java
> versions: 1.8.0_171-8u171-b11-0ubuntu0.16.04.1-b11 and
> 1.8.0_03-Ubuntu-8u77-b03-3ubuntu3-b03 (I downgraded on my system to see if it
> was the new Java version).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
