[
https://issues.apache.org/jira/browse/IMPALA-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16500696#comment-16500696
]
Tim Armstrong commented on IMPALA-7069:
---------------------------------------
I saw another crash with this backtrace:
{noformat}
CORE: ./core.1528115186.21531.impalad
BINARY: ./be/build/latest/service/impalad
Core was generated by
`/data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/bu'.
Program terminated with signal 6, Aborted.
#0 0x00000039f74328e5 in raise () from /lib64/libc.so.6
To enable execution of this file add
add-auto-load-safe-path
/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/gcc-4.9.2/lib64/libstdc++.so.6.0.20-gdb.py
line to your configuration file "/var/lib/jenkins/.gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/var/lib/jenkins/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
info "(gdb)Auto-loading safe path"
#0 0x00000039f74328e5 in raise () from /lib64/libc.so.6
#1 0x00000039f74340c5 in abort () from /lib64/libc.so.6
#2 0x00007f04924521a5 in os::abort(bool) () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#3 0x00007f04925e2843 in VMError::report_and_die() () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#4 0x00007f04925e2cfe in crash_handler(int, siginfo*, void*) () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#5 0x00007f0492451692 in os::Linux::chained_handler(int, siginfo*, void*) ()
from /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#6 0x00007f04924574cd in JVM_handle_linux_signal () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#7 0x00007f049244e4f3 in signalHandler(int, siginfo*, void*) () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#8 <signal handler called>
#9 0x00007f04924413bf in oopDesc::print_on(outputStream*) const () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#10 0x00007f0492456e12 in os::print_register_info(outputStream*, void*) () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#11 0x00007f04925e0e24 in VMError::report(outputStream*) () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#12 0x00007f04925e243a in VMError::report_and_die() () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#13 0x00007f0492457562 in JVM_handle_linux_signal () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#14 0x00007f049244e4f3 in signalHandler(int, siginfo*, void*) () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#15 <signal handler called>
#16 0x00007f049240f0bd in MutableSpace::object_iterate(ObjectClosure*) () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#17 0x00007f04924bc9b0 in PSYoungGen::object_iterate(ObjectClosure*) () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#18 0x00007f04924b8647 in PSScavenge::clean_up_failed_promotion() () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#19 0x00007f04924b9f25 in PSScavenge::invoke_no_policy() () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#20 0x00007f04924ba271 in PSScavenge::invoke() () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#21 0x00007f0492472ed0 in ParallelScavengeHeap::failed_mem_allocate(unsigned
long) () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#22 0x00007f04925e3af7 in VM_ParallelGCFailedAllocation::doit() () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#23 0x00007f04925eb215 in VM_Operation::evaluate() () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#24 0x00007f04925e960a in VMThread::evaluate_operation(VM_Operation*) () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#25 0x00007f04925e998e in VMThread::loop() () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#26 0x00007f04925e9e00 in VMThread::run() () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#27 0x00007f04924530c8 in java_start(Thread*) () from
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#28 0x00000039f7807851 in start_thread () from /lib64/libpthread.so.0
#29 0x00000039f74e894d in clone () from /lib64/libc.so.6
{noformat}
Some of these tests were running:
{noformat}
05:26:15
query_test/test_udfs.py::TestUdfExecution::test_udf_constant_folding[exec_option:
{'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
'exec_single_node_rows_threshold': 100, 'enable_expr_rewrites': False} |
table_format: text/none]
05:26:15 [gw2] PASSED
query_test/test_udfs.py::TestUdfExecution::test_udf_constant_folding[exec_option:
{'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
'exec_single_node_rows_threshold': 100, 'enable_expr_rewrites': False} |
table_format: text/none]
05:30:49
query_test/test_udfs.py::TestUdfTargeted::test_hive_udfs_missing_jar[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': True, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: text/none]
05:30:49 [gw2] FAILED
query_test/test_udfs.py::TestUdfTargeted::test_hive_udfs_missing_jar[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': True, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: text/none]
05:30:49 [gw1] FAILED
query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_aggs[exec_option:
{'debug_action': '-1:OPEN:[email protected]',
'default_spillable_buffer_size': '256k'} | table_format: parquet/none]
05:30:49
query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_aggs[exec_option:
{'debug_action': '-1:OPEN:[email protected]',
'default_spillable_buffer_size': '256k'} | table_format: parquet/none]
05:30:49
query_test/test_udfs.py::TestUdfTargeted::test_hive_udfs_missing_jar[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: text/none]
05:30:50 [gw0] FAILED
query_test/test_queries.py::TestQueries::test_sort[exec_option: {'batch_size':
0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen':
False, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: seq/none]
05:30:52 query_test/test_queries.py::TestQueries::test_sort[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': True, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: seq/none]
05:30:52 [gw3] FAILED
query_test/test_udfs.py::TestUdfTargeted::test_concurrent_jar_drop_use[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': True, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: text/none]
05:30:58
query_test/test_udfs.py::TestUdfTargeted::test_concurrent_jar_drop_use[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: text/none]
{noformat}
I wasn't able to get the hs_err_pid* file from this job, but I pushed a JIRA to
enable collecting it with other logs. I'm tentatively blaming this issue, but
we can split it out if it turns out ot be something different.
> Java UDF tests can trigger a crash in Java ClassLoader
> ------------------------------------------------------
>
> Key: IMPALA-7069
> URL: https://issues.apache.org/jira/browse/IMPALA-7069
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 3.1.0
> Reporter: Tim Armstrong
> Assignee: Taras Bobrovytsky
> Priority: Critical
> Labels: crash, flaky
> Attachments: hs_err_pid22764.log, hs_err_pid29246.log,
> hs_err_pid8975.log, hs_err_pid9694.log
>
>
> I hit this crash on a GVO, but was able to reproduce it on master on my
> desktop.
> Repro steps:
> {code}
> git checkout c1362afb9a072e49df470d9068d44cdbdf5cdec5
> ./buildall.sh -debug -noclean -notests -skiptests -ninja
> start-impala-cluster.py
> while impala-py.test tests/query_test/test_udfs.py -k 'hive or java or jar'
> -n4 --verbose; do date; done
> {code}
> I generally hit the crash within a hour of looping the test.
> {noformat}
> Stack: [0x00007fa04791f000,0x00007fa048120000], sp=0x00007fa04811aff0, free
> space=8175k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> V [libjvm.so+0x8a8107]
> V [libjvm.so+0x96cf5f]
> v ~RuntimeStub::_complete_monitor_locking_Java
> J 2758 C2
> java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object;
> (362 bytes) @ 0x00007fa0c73637d4 [0x00007fa0c7362d00+0xad4]
> J 2311 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122
> bytes) @ 0x00007fa0c70a09a8 [0x00007fa0c70a08e0+0xc8]
> J 3953 C2
> java.net.FactoryURLClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;
> (40 bytes) @ 0x00007fa0c71ce0f0 [0x00007fa0c71ce0a0+0x50]
> J 2987 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (7
> bytes) @ 0x00007fa0c72ddb64 [0x00007fa0c72ddb20+0x44]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x661ec4]
> V [libjvm.so+0x662523]
> V [libjvm.so+0x9e398d]
> V [libjvm.so+0x9e2326]
> V [libjvm.so+0x9e2b50]
> V [libjvm.so+0x42c099]
> V [libjvm.so+0x9dc786]
> V [libjvm.so+0x6a5edf]
> V [libjvm.so+0x6a70cb] JVM_DefineClass+0xbb
> V [libjvm.so+0xa31ea5]
> V [libjvm.so+0xa37ea7]
> J 4842
> sun.misc.Unsafe.defineClass(Ljava/lang/String;[BIILjava/lang/ClassLoader;Ljava/security/ProtectionDomain;)Ljava/lang/Class;
> (0 bytes) @ 0x00007fa0c7af120b [0x00007fa0c7af1100+0x10b]
> J 13229 C2 sun.reflect.MethodAccessorGenerator$1.run()Ljava/lang/Object; (5
> bytes) @ 0x00007fa0c8cf2a74 [0x00007fa0c8cf2940+0x134]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6b5949] JVM_DoPrivileged+0x429
> J 1035
> java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;
> (0 bytes) @ 0x00007fa0c7220c7f [0x00007fa0c7220bc0+0xbf]
> J 20421 C2
> sun.reflect.MethodAccessorGenerator.generate(Ljava/lang/Class;Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;IZZLjava/lang/Class;)Lsun/reflect/MagicAccessorImpl;
> (762 bytes) @ 0x00007fa0c89bb848 [0x00007fa0c89b9da0+0x1aa8]
> J 4163 C2
> sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (104 bytes) @ 0x00007fa0c789cca8 [0x00007fa0c789c8c0+0x3e8]
> J 2379 C2 org.apache.impala.hive.executor.UdfExecutor.evaluate()V (396 bytes)
> @ 0x00007fa0c711c638 [0x00007fa0c711c400+0x238]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6822d7]
> V [libjvm.so+0x6862c9]
> C [impalad+0x2a004fa] JNIEnv_::CallNonvirtualVoidMethodA(_jobject*,
> _jclass*, _jmethodID*, jvalue const*)+0x40
> C [impalad+0x29fe4ff]
> impala::HiveUdfCall::Evaluate(impala::ScalarExprEvaluator*, impala::TupleRow
> const*) const+0x44b
> C [impalad+0x29ffde9]
> impala::HiveUdfCall::GetSmallIntVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0xbb
> C [impalad+0x2a0948a]
> impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&,
> impala::TupleRow const*)+0x14c
> C [impalad+0x2a48eb1]
> impala::ScalarFnCall::EvaluateNonConstantChildren(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x9d
> C [impalad+0x2a4abba] impala_udf::BooleanVal
> impala::ScalarFnCall::InterpretEval<impala_udf::BooleanVal>(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x18c
> C [impalad+0x2a4907d]
> impala::ScalarFnCall::GetBooleanVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x179
> C [impalad+0x2a09c7f]
> impala::ScalarExprEvaluator::GetBooleanVal(impala::TupleRow*)+0x37
> C [impalad+0x1b70efb]
> impala::ExecNode::EvalPredicate(impala::ScalarExprEvaluator*,
> impala::TupleRow*)+0x23
> C [impalad+0x1b70efb]
> impala::ExecNode::EvalPredicate(impala::ScalarExprEvaluator*,
> impala::TupleRow*)+0x23
> C [impalad+0x1b6fdf0]
> impala::ExecNode::EvalConjuncts(impala::ScalarExprEvaluator* const*, int,
> impala::TupleRow*)+0x42
> C [impalad+0x1bb60e3]
> impala::HdfsScanner::EvalConjuncts(impala::TupleRow*)+0x4d
> C [impalad+0x1bb08fd]
> impala::HdfsScanner::WriteCompleteTuple(impala::MemPool*,
> impala::FieldLocation*, impala::Tuple*, impala::TupleRow*, impala::Tuple*,
> unsigned char*, unsigned char*)+0x1b1
> C [impalad+0x1bc176d]
> impala::HdfsScanner::WriteAlignedTuples(impala::MemPool*, impala::TupleRow*,
> impala::FieldLocation*, int, int, int, int, bool)+0x1f3
> C [impalad+0x1bf93d9] impala::HdfsTextScanner::WriteFields(int, int,
> impala::MemPool*, impala::TupleRow*)+0x779
> C [impalad+0x1bf48be]
> impala::HdfsTextScanner::ProcessRange(impala::RowBatch*, int*)+0x9d8
> C [impalad+0x1bf55e8]
> impala::HdfsTextScanner::GetNextInternal(impala::RowBatch*)+0x422
> C [impalad+0x1baf676] impala::HdfsScanner::ProcessSplit()+0x1ea
> C [impalad+0x1b8695c]
> impala::HdfsScanNode::ProcessSplit(std::vector<impala::FilterContext,
> std::allocator<impala::FilterContext> > const&, impala::MemPool*,
> impala::io::ScanRange*, long*)+0x77a
> C [impalad+0x1b85cfd] impala::HdfsScanNode::ScannerThread(bool, long)+0x3cd
> {noformat}
> I've only tried to hit this on Ubuntu16.04. It repros on at least two java
> versions: 1.8.0_171-8u171-b11-0ubuntu0.16.04.1-b11 and
> 1.8.0_03-Ubuntu-8u77-b03-3ubuntu3-b03 (I downgraded on my system to see if it
> was the new Java version).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
