bharath v created IMPALA-6873:
---------------------------------
Summary: Crash in Expr::GetConstVal() due to NULL dereference
Key: IMPALA-6873
URL: https://issues.apache.org/jira/browse/IMPALA-6873
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 2.9.0, Impala 2.8.0
Reporter: bharath v
Fix For: Impala 2.10.0
Log file crashing frame
{noformat}
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x000000357f88980b, pid=564763, tid=0x00007f7b0386c700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_162-b12) (build
1.8.0_162-b12)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.162-b12 mixed mode linux-amd64
compressed oops)
# Problematic frame:
# C [libc.so.6+0x8980b] memcpy+0x15b
{noformat}
Crashing stack, extracted from core dump
{noformat}
#10 0x00007f4d8eaadbe7 in os::print_location(outputStream*, long, bool) () from
/root/usr/java/latest/jre/lib/amd64/server/libjvm.so
#11 0x00007f4d8eabcaf5 in os::print_register_info(outputStream*, void*) () from
/root/usr/java/latest/jre/lib/amd64/server/libjvm.so
#12 0x00007f4d8ec595a3 in VMError::report(outputStream*) () from
/root/usr/java/latest/jre/lib/amd64/server/libjvm.so
#13 0x00007f4d8ec5ab2a in VMError::report_and_die() () from
/root/usr/java/latest/jre/lib/amd64/server/libjvm.so
#14 0x00007f4d8eabd22f in JVM_handle_linux_signal () from
/root/usr/java/latest/jre/lib/amd64/server/libjvm.so
#15 0x00007f4d8eab3253 in signalHandler(int, siginfo*, void*) () from
/root/usr/java/latest/jre/lib/amd64/server/libjvm.so
#16 <signal handler called>
#17 0x0000003b4d089750 in memcpy () from /lib64/libc.so.6
#18 0x0000000000845578 in impala::Expr::GetConstVal (this=0x7f430831f400,
state=0x7f4cdc91b750, context=0xe331540, const_val=Unhandled dwarf expression
opcode 0xf3
) at /usr/src/debug/impala-2.9.0-cdh5.12.2/be/src/exprs/expr.cc:577
#19 0x00000000008909b9 in impala::ScalarFnCall::Open (this=0x7f430831e600,
state=0x7f4cdc91b750, ctx=0xe331540,
scope=impala_udf::FunctionContext::FRAGMENT_LOCAL)
at /usr/src/debug/impala-2.9.0-cdh5.12.2/be/src/exprs/scalar-fn-call.cc:189
#20 0x000000000084af8c in impala::ExprContext::Open (this=Unhandled dwarf
expression opcode 0xf3
) at /usr/src/debug/impala-2.9.0-cdh5.12.2/be/src/exprs/expr-context.cc:70
#21 0x0000000000ab2a3f in
Java_org_apache_impala_service_FeSupport_NativeEvalExprsWithoutRow
(env=0xcca31f8, caller_class=Unhandled dwarf expression opcode 0xf3
) at /usr/src/debug/impala-2.9.0-cdh5.12.2/be/src/service/fe-support.cc:142
#22 0x00007f4d7b284dad in ?? ()
#23 0x000000059cabbe18 in ?? ()
#24 0x000000059cabfcd8 in ?? ()
#25 0xb395702563a2136b in ?? ()
#26 0x00000000806394b0 in ?? ()
#27 0xb395701200000002 in ?? ()
#28 0x000000059cab8090 in ?? ()
#29 0x00000000802f3c08 in ?? ()
#30 0x000000059beef118 in ?? ()
#31 0x00007f4cdc91bf70 in ?? ()
#32 0x00007f4d7b28033c in ?? ()
#33 0x000000059cab8438 in ?? ()
#34 0x000000008d567eb0 in ?? ()
#35 0x000000059cab8588 in ?? ()
#36 0x000000059cab8308 in ?? ()
#37 0x000000059cab85a0 in ?? ()
#38 0x000000059cab85d0 in ?? ()
#39 0x0000001811aad009 in ?? ()
#40 0x00000008ffffffff in ?? ()
{noformat}
Missing frames are from the JVM and are below (extracted from hs_err_pid file)
{noformat}
J 12167 org.apache.impala.service.FeSupport.NativeEvalExprsWithoutRow([B[B)[B
(0 bytes) @ 0x00007f7bad2e1cf3 [0x00007f7bad2e1c80+0x73]
J 12158 C1
org.apache.impala.service.FeSupport.EvalExprWithoutRow(Lorg/apache/impala/analysis/Expr;Lorg/apache/impala/thrift/TQueryCtx;)Lorg/apache/impala/thrift/TColumnValue;
(170 bytes) @ 0x00007f7bad307bf4 [0x00007f7bad305be0+0x2014]
J 12206 C1
org.apache.impala.service.FeSupport.EvalPredicate(Lorg/apache/impala/analysis/Expr;Lorg/apache/impala/thrift/TQueryCtx;)Z
(60 bytes) @ 0x00007f7bad32daac [0x00007f7bad32d180+0x92c]
J 12207 C1
org.apache.impala.analysis.Analyzer.isTrueWithNullSlots(Lorg/apache/impala/analysis/Expr;)Z
(137 bytes) @ 0x00007f7bad331c54 [0x00007f7bad32fe40+0x1e14]
j
org.apache.impala.planner.HdfsScanNode.computeDictionaryFilterConjuncts(Lorg/apache/impala/analysis/Analyzer;)V+135
j
org.apache.impala.planner.HdfsScanNode.init(Lorg/apache/impala/analysis/Analyzer;)V+22
j
org.apache.impala.planner.SingleNodePlanner.createHdfsScanPlan(Lorg/apache/impala/analysis/TableRef;ZLjava/util/List;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+306
j
org.apache.impala.planner.SingleNodePlanner.createScanNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+143
j
org.apache.impala.planner.SingleNodePlanner.createTableRefNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+14
j
org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(Ljava/util/List;Ljava/util/List;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+41
j
org.apache.impala.planner.SingleNodePlanner.createSelectPlan(Lorg/apache/impala/analysis/SelectStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+203
j
org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+31
j
org.apache.impala.planner.SingleNodePlanner.createInlineViewPlan(Lorg/apache/impala/analysis/Analyzer;Lorg/apache/impala/analysis/InlineViewRef;)Lorg/apache/impala/planner/PlanNode;+208
j
org.apache.impala.planner.SingleNodePlanner.createTableRefNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+107
j
org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(Ljava/util/List;Ljava/util/List;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+41
j
org.apache.impala.planner.SingleNodePlanner.createSelectPlan(Lorg/apache/impala/analysis/SelectStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+203
j
org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+31
j
org.apache.impala.planner.SingleNodePlanner.createUnionPlan(Lorg/apache/impala/analysis/Analyzer;Lorg/apache/impala/analysis/UnionStmt;Ljava/util/List;Lorg/apache/impala/planner/PlanNode;)Lorg/apache/impala/planner/UnionNode;+141
j
org.apache.impala.planner.SingleNodePlanner.createUnionPlan(Lorg/apache/impala/analysis/UnionStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+164
j
org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+144
j
org.apache.impala.planner.SingleNodePlanner.createInlineViewPlan(Lorg/apache/impala/analysis/Analyzer;Lorg/apache/impala/analysis/InlineViewRef;)Lorg/apache/impala/planner/PlanNode;+208
j
org.apache.impala.planner.SingleNodePlanner.createTableRefNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+107
j
org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(Ljava/util/List;Ljava/util/List;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+41
j
org.apache.impala.planner.SingleNodePlanner.createSelectPlan(Lorg/apache/impala/analysis/SelectStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+203
j
org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+31
j
org.apache.impala.planner.SingleNodePlanner.createInlineViewPlan(Lorg/apache/impala/analysis/Analyzer;Lorg/apache/impala/analysis/InlineViewRef;)Lorg/apache/impala/planner/PlanNode;+208
j
org.apache.impala.planner.SingleNodePlanner.createTableRefNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+107
j
org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(Ljava/util/List;Ljava/util/List;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+41
j
org.apache.impala.planner.SingleNodePlanner.createSelectPlan(Lorg/apache/impala/analysis/SelectStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+203
j
org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+31
j
org.apache.impala.planner.SingleNodePlanner.createInlineViewPlan(Lorg/apache/impala/analysis/Analyzer;Lorg/apache/impala/analysis/InlineViewRef;)Lorg/apache/impala/planner/PlanNode;+208
j
org.apache.impala.planner.SingleNodePlanner.createTableRefNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+107
j
org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(Ljava/util/List;Ljava/util/List;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+41
j
org.apache.impala.planner.SingleNodePlanner.createSelectPlan(Lorg/apache/impala/analysis/SelectStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+203
j
org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+31
j
org.apache.impala.planner.SingleNodePlanner.createSingleNodePlan()Lorg/apache/impala/planner/PlanNode;+104
j org.apache.impala.planner.Planner.createPlan()Ljava/util/ArrayList;+25
j
org.apache.impala.service.Frontend.createExecRequest(Lorg/apache/impala/planner/Planner;Ljava/lang/StringBuilder;)Lorg/apache/impala/thrift/TQueryExecRequest;+111
J 12874 C1
org.apache.impala.service.Frontend.createExecRequest(Lorg/apache/impala/thrift/TQueryCtx;Ljava/lang/StringBuilder;)Lorg/apache/impala/thrift/TExecRequest;
(956 bytes) @ 0x00007f7bad587174 [0x00007f7bad583780+0x39f4]
J 13160 C1 org.apache.impala.service.JniFrontend.createExecRequest([B)[B (100
bytes) @ 0x00007f7bad687d7c [0x00007f7bad687760+0x61c]
{noformat}
So the root cause seems to be in the {{memcpy()}} in the following piece of
code in expr.cc
{noformat}
case TYPE_VARCHAR: {
StringVal* sv = reinterpret_cast<StringVal*>(*const_val);
*sv = GetStringVal(context, NULL);
if (sv->len > 0) {
// Make sure the memory is owned by 'context'.
uint8_t* ptr_copy = context->pool_->TryAllocate(sv->len);
if (ptr_copy == NULL) {
return context->pool_->mem_tracker()->MemLimitExceeded(
state, "Could not allocate constant string value", sv->len);
}
memcpy(ptr_copy, sv->ptr, sv->len); <--- CRASH since sv->ptr = NULL
an sv->len > 0
sv->ptr = ptr_copy;
}
break;
}
{noformat}
Few observations:
- The query crashes the coordinator during the query compilation/analysis (as
evident from the JVM stack trace)
- The root cause seems to be due to a malformed {{StringVal}} (ptr = NULL and
len >0) returned by {{GetStringVal}} and it is unclear at this point which
specific function/piece of code is generating that.
- In this particular case, I figured that the ScalarFn in the crashing stack
that is calling {{GetConstVal}} is {{concat()}} and removing it doesn't crash
the coordinator.
- Unable to reproduce it locally on my dev box
- The problematic piece of code memcpy'ing the NULL ptr is introduced by
IMPALA-4302 and removed by IMPALA-4192. Hence only 2.9.0 and 2.10.0 are the
affected versions
Next Steps:
- Avoid the crash by having a stricter is_null check on the output StringVal
- Figure out which possible builtins can generate such StringVals.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
