[
https://issues.apache.org/jira/browse/IMPALA-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
bharath v resolved IMPALA-6873.
-------------------------------
Resolution: Fixed
[~tarmstrong] Yes.
Fixed via:
https://github.com/apache/impala/commit/b38d9826d7ef9bc0ecff548626d30690f935e9c3#diff-7140ed1301fa7a470056719186b1d646
> Crash in Expr::GetConstVal() due to NULL dereference
> ----------------------------------------------------
>
> Key: IMPALA-6873
> URL: https://issues.apache.org/jira/browse/IMPALA-6873
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.8.0, Impala 2.9.0
> Reporter: bharath v
> Priority: Blocker
> Labels: crash
> Fix For: Impala 2.10.0
>
>
> Log file crashing frame
> {noformat}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x000000357f88980b, pid=564763, tid=0x00007f7b0386c700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_162-b12) (build
> 1.8.0_162-b12)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.162-b12 mixed mode
> linux-amd64 compressed oops)
> # Problematic frame:
> # C [libc.so.6+0x8980b] memcpy+0x15b
> {noformat}
> Crashing stack, extracted from core dump
> {noformat}
> #10 0x00007f4d8eaadbe7 in os::print_location(outputStream*, long, bool) ()
> from /root/usr/java/latest/jre/lib/amd64/server/libjvm.so
> #11 0x00007f4d8eabcaf5 in os::print_register_info(outputStream*, void*) ()
> from /root/usr/java/latest/jre/lib/amd64/server/libjvm.so
> #12 0x00007f4d8ec595a3 in VMError::report(outputStream*) () from
> /root/usr/java/latest/jre/lib/amd64/server/libjvm.so
> #13 0x00007f4d8ec5ab2a in VMError::report_and_die() () from
> /root/usr/java/latest/jre/lib/amd64/server/libjvm.so
> #14 0x00007f4d8eabd22f in JVM_handle_linux_signal () from
> /root/usr/java/latest/jre/lib/amd64/server/libjvm.so
> #15 0x00007f4d8eab3253 in signalHandler(int, siginfo*, void*) () from
> /root/usr/java/latest/jre/lib/amd64/server/libjvm.so
> #16 <signal handler called>
> #17 0x0000003b4d089750 in memcpy () from /lib64/libc.so.6
> #18 0x0000000000845578 in impala::Expr::GetConstVal (this=0x7f430831f400,
> state=0x7f4cdc91b750, context=0xe331540, const_val=Unhandled dwarf expression
> opcode 0xf3
> ) at /usr/src/debug/impala-2.9.0-cdh5.12.2/be/src/exprs/expr.cc:577
> #19 0x00000000008909b9 in impala::ScalarFnCall::Open (this=0x7f430831e600,
> state=0x7f4cdc91b750, ctx=0xe331540,
> scope=impala_udf::FunctionContext::FRAGMENT_LOCAL)
> at
> /usr/src/debug/impala-2.9.0-cdh5.12.2/be/src/exprs/scalar-fn-call.cc:189
> #20 0x000000000084af8c in impala::ExprContext::Open (this=Unhandled dwarf
> expression opcode 0xf3
> ) at /usr/src/debug/impala-2.9.0-cdh5.12.2/be/src/exprs/expr-context.cc:70
> #21 0x0000000000ab2a3f in
> Java_org_apache_impala_service_FeSupport_NativeEvalExprsWithoutRow
> (env=0xcca31f8, caller_class=Unhandled dwarf expression opcode 0xf3
> ) at /usr/src/debug/impala-2.9.0-cdh5.12.2/be/src/service/fe-support.cc:142
> #22 0x00007f4d7b284dad in ?? ()
> #23 0x000000059cabbe18 in ?? ()
> #24 0x000000059cabfcd8 in ?? ()
> #25 0xb395702563a2136b in ?? ()
> #26 0x00000000806394b0 in ?? ()
> #27 0xb395701200000002 in ?? ()
> #28 0x000000059cab8090 in ?? ()
> #29 0x00000000802f3c08 in ?? ()
> #30 0x000000059beef118 in ?? ()
> #31 0x00007f4cdc91bf70 in ?? ()
> #32 0x00007f4d7b28033c in ?? ()
> #33 0x000000059cab8438 in ?? ()
> #34 0x000000008d567eb0 in ?? ()
> #35 0x000000059cab8588 in ?? ()
> #36 0x000000059cab8308 in ?? ()
> #37 0x000000059cab85a0 in ?? ()
> #38 0x000000059cab85d0 in ?? ()
> #39 0x0000001811aad009 in ?? ()
> #40 0x00000008ffffffff in ?? ()
> {noformat}
>
> Missing frames are from the JVM and are below (extracted from hs_err_pid file)
> {noformat}
> J 12167
> org.apache.impala.service.FeSupport.NativeEvalExprsWithoutRow([B[B)[B (0
> bytes) @ 0x00007f7bad2e1cf3 [0x00007f7bad2e1c80+0x73]
> J 12158 C1
> org.apache.impala.service.FeSupport.EvalExprWithoutRow(Lorg/apache/impala/analysis/Expr;Lorg/apache/impala/thrift/TQueryCtx;)Lorg/apache/impala/thrift/TColumnValue;
> (170 bytes) @ 0x00007f7bad307bf4 [0x00007f7bad305be0+0x2014]
> J 12206 C1
> org.apache.impala.service.FeSupport.EvalPredicate(Lorg/apache/impala/analysis/Expr;Lorg/apache/impala/thrift/TQueryCtx;)Z
> (60 bytes) @ 0x00007f7bad32daac [0x00007f7bad32d180+0x92c]
> J 12207 C1
> org.apache.impala.analysis.Analyzer.isTrueWithNullSlots(Lorg/apache/impala/analysis/Expr;)Z
> (137 bytes) @ 0x00007f7bad331c54 [0x00007f7bad32fe40+0x1e14]
> j
> org.apache.impala.planner.HdfsScanNode.computeDictionaryFilterConjuncts(Lorg/apache/impala/analysis/Analyzer;)V+135
> j
> org.apache.impala.planner.HdfsScanNode.init(Lorg/apache/impala/analysis/Analyzer;)V+22
> j
> org.apache.impala.planner.SingleNodePlanner.createHdfsScanPlan(Lorg/apache/impala/analysis/TableRef;ZLjava/util/List;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+306
> j
> org.apache.impala.planner.SingleNodePlanner.createScanNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+143
> j
> org.apache.impala.planner.SingleNodePlanner.createTableRefNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+14
> j
> org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(Ljava/util/List;Ljava/util/List;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+41
> j
> org.apache.impala.planner.SingleNodePlanner.createSelectPlan(Lorg/apache/impala/analysis/SelectStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+203
> j
> org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+31
> j
> org.apache.impala.planner.SingleNodePlanner.createInlineViewPlan(Lorg/apache/impala/analysis/Analyzer;Lorg/apache/impala/analysis/InlineViewRef;)Lorg/apache/impala/planner/PlanNode;+208
> j
> org.apache.impala.planner.SingleNodePlanner.createTableRefNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+107
> j
> org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(Ljava/util/List;Ljava/util/List;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+41
> j
> org.apache.impala.planner.SingleNodePlanner.createSelectPlan(Lorg/apache/impala/analysis/SelectStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+203
> j
> org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+31
> j
> org.apache.impala.planner.SingleNodePlanner.createUnionPlan(Lorg/apache/impala/analysis/Analyzer;Lorg/apache/impala/analysis/UnionStmt;Ljava/util/List;Lorg/apache/impala/planner/PlanNode;)Lorg/apache/impala/planner/UnionNode;+141
> j
> org.apache.impala.planner.SingleNodePlanner.createUnionPlan(Lorg/apache/impala/analysis/UnionStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+164
> j
> org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+144
> j
> org.apache.impala.planner.SingleNodePlanner.createInlineViewPlan(Lorg/apache/impala/analysis/Analyzer;Lorg/apache/impala/analysis/InlineViewRef;)Lorg/apache/impala/planner/PlanNode;+208
> j
> org.apache.impala.planner.SingleNodePlanner.createTableRefNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+107
> j
> org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(Ljava/util/List;Ljava/util/List;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+41
> j
> org.apache.impala.planner.SingleNodePlanner.createSelectPlan(Lorg/apache/impala/analysis/SelectStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+203
> j
> org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+31
> j
> org.apache.impala.planner.SingleNodePlanner.createInlineViewPlan(Lorg/apache/impala/analysis/Analyzer;Lorg/apache/impala/analysis/InlineViewRef;)Lorg/apache/impala/planner/PlanNode;+208
> j
> org.apache.impala.planner.SingleNodePlanner.createTableRefNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+107
> j
> org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(Ljava/util/List;Ljava/util/List;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+41
> j
> org.apache.impala.planner.SingleNodePlanner.createSelectPlan(Lorg/apache/impala/analysis/SelectStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+203
> j
> org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+31
> j
> org.apache.impala.planner.SingleNodePlanner.createInlineViewPlan(Lorg/apache/impala/analysis/Analyzer;Lorg/apache/impala/analysis/InlineViewRef;)Lorg/apache/impala/planner/PlanNode;+208
> j
> org.apache.impala.planner.SingleNodePlanner.createTableRefNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+107
> j
> org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(Ljava/util/List;Ljava/util/List;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+41
> j
> org.apache.impala.planner.SingleNodePlanner.createSelectPlan(Lorg/apache/impala/analysis/SelectStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+203
> j
> org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+31
> j
> org.apache.impala.planner.SingleNodePlanner.createSingleNodePlan()Lorg/apache/impala/planner/PlanNode;+104
> j org.apache.impala.planner.Planner.createPlan()Ljava/util/ArrayList;+25
> j
> org.apache.impala.service.Frontend.createExecRequest(Lorg/apache/impala/planner/Planner;Ljava/lang/StringBuilder;)Lorg/apache/impala/thrift/TQueryExecRequest;+111
> J 12874 C1
> org.apache.impala.service.Frontend.createExecRequest(Lorg/apache/impala/thrift/TQueryCtx;Ljava/lang/StringBuilder;)Lorg/apache/impala/thrift/TExecRequest;
> (956 bytes) @ 0x00007f7bad587174 [0x00007f7bad583780+0x39f4]
> J 13160 C1 org.apache.impala.service.JniFrontend.createExecRequest([B)[B (100
> bytes) @ 0x00007f7bad687d7c [0x00007f7bad687760+0x61c]
> {noformat}
> So the root cause seems to be in the {{memcpy()}} in the following piece of
> code in expr.cc
> {noformat}
> case TYPE_VARCHAR: {
> StringVal* sv = reinterpret_cast<StringVal*>(*const_val);
> *sv = GetStringVal(context, NULL);
> if (sv->len > 0) {
> // Make sure the memory is owned by 'context'.
> uint8_t* ptr_copy = context->pool_->TryAllocate(sv->len);
> if (ptr_copy == NULL) {
> return context->pool_->mem_tracker()->MemLimitExceeded(
> state, "Could not allocate constant string value", sv->len);
> }
> memcpy(ptr_copy, sv->ptr, sv->len); <--- CRASH since sv->ptr = NULL
> an sv->len > 0
> sv->ptr = ptr_copy;
> }
> break;
> }
> {noformat}
> Few observations:
> - The query crashes the coordinator during the query compilation/analysis
> (as evident from the JVM stack trace)
> - The root cause seems to be due to a malformed {{StringVal}} (ptr = NULL
> and len >0) returned by {{GetStringVal}} and it is unclear at this point
> which specific function/piece of code is generating that.
> - In this particular case, I figured that the ScalarFn in the crashing stack
> that is calling {{GetConstVal}} is {{concat()}} and removing it doesn't crash
> the coordinator.
> - Unable to reproduce it locally on my dev box
> - The problematic piece of code memcpy'ing the NULL ptr is introduced by
> IMPALA-4302 and removed by IMPALA-4192. Hence only 2.9.0 and 2.10.0 are the
> affected versions
> Next Steps:
> - Avoid the crash by having a stricter is_null check on the output StringVal
> - Figure out which possible builtins can generate such StringVals.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
