Impala Public Jenkins has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/11545 )
Change subject: IMPALA-5031: null ptr errors in C calls in BE tests
......................................................................
IMPALA-5031: null ptr errors in C calls in BE tests
This patch fixes all remaining UBSAN "null pointer passed as argument"
errors in the backend tests. These are undefined behavior according to
"7.1.4 Use of library functions" in the C99 standard (which is
included in C++14 in section [intro.refs]):
If an argument to a function has an invalid value (such as a value
outside the domain of the function, or a pointer outside the
address space of the program, or a null pointer, or a pointer to
non-modifiable storage when the corresponding parameter is not
const-qualified) or a type (after promotion) not expected by a
function with variable number of arguments, the behavior is
undefined.
The interesting parts of the backtraces for the errors fixed in this
patch are below:
exprs/string-functions-ir.cc:311:17: runtime error: null pointer passed as
argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified here
#0 StringFunctions::Replace(impala_udf::FunctionContext*,
impala_udf::StringVal const&, impala_udf::StringVal const&,
impala_udf::StringVal const&) exprs/string-functions-ir.cc:311:5
#1 impala_udf::StringVal
ScalarFnCall::InterpretEval<impala_udf::StringVal>(ScalarExprEvaluator*,
TupleRow const*) const exprs/scalar-fn-call.cc:485:580
#2 ScalarFnCall::GetStringVal(ScalarExprEvaluator*, TupleRow const*) const
exprs/scalar-fn-call.cc:599:44
#3 ScalarExprEvaluator::GetValue(ScalarExpr const&, TupleRow const*)
exprs/scalar-expr-evaluator.cc:299:38
#4 ScalarExprEvaluator::GetValue(TupleRow const*)
exprs/scalar-expr-evaluator.cc:250:10
#5 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor
const&, ScalarExprEvaluator* const*, MemPool*, StringValue**, int*, int*)
runtime/tuple.cc:222:27
#6 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor
const&, vector<ScalarExprEvaluator*> const&, MemPool*, vector<StringValue*>*,
int*) runtime/tuple.h:174:5
#7 UnionNode::MaterializeExprs(vector<ScalarExprEvaluator*> const&,
TupleRow*, unsigned char*, RowBatch*) exec/union-node-ir.cc:29:14
#8 UnionNode::GetNextConst(RuntimeState*, RowBatch*)
exec/union-node.cc:263:5
#9 UnionNode::GetNext(RuntimeState*, RowBatch*, bool*)
exec/union-node.cc:296:45
#10 FragmentInstanceState::ExecInternal()
runtime/fragment-instance-state.cc:310:59
#11 FragmentInstanceState::Exec() runtime/fragment-instance-state.cc:95:14
#12 QueryState::ExecFInstance(FragmentInstanceState*)
runtime/query-state.cc:488:24
#13 QueryState::StartFInstances()::$_0::operator()() const
runtime/query-state.cc:416:35
#20 thread_proxy (exprs/expr-test+0x55ca939)
exprs/string-functions-ir.cc:868:15: runtime error: null pointer passed as
argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified here
#0 StringFunctions::ConcatWs(impala_udf::FunctionContext*,
impala_udf::StringVal const&, int, impala_udf::StringVal const*)
exprs/string-functions-ir.cc:868:3
#1 impala_udf::StringVal
ScalarFnCall::InterpretEval<impala_udf::StringVal>(ScalarExprEvaluator*,
TupleRow const*) const exprs/scalar-fn-call.cc:510:270
#2 ScalarFnCall::GetStringVal(ScalarExprEvaluator*, TupleRow const*) const
exprs/scalar-fn-call.cc:599:44
#3 ScalarExprEvaluator::GetValue(ScalarExpr const&, TupleRow const*)
exprs/scalar-expr-evaluator.cc:299:38
#4 ScalarExprEvaluator::GetValue(TupleRow const*)
exprs/scalar-expr-evaluator.cc:250:10
#5 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor
const&, ScalarExprEvaluator* const*, MemPool*, StringValue**, int*, int*)
runtime/tuple.cc:222:27
#6 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor
const&, vector<ScalarExprEvaluator*> const&, MemPool*, vector<StringValue*>*,
int*) runtime/tuple.h:174:5
#7 UnionNode::MaterializeExprs(vector<ScalarExprEvaluator*> const&,
TupleRow*, unsigned char*, RowBatch*) exec/union-node-ir.cc:29:14
#8 UnionNode::GetNextConst(RuntimeState*, RowBatch*)
exec/union-node.cc:263:5
#9 UnionNode::GetNext(RuntimeState*, RowBatch*, bool*)
exec/union-node.cc:296:45
#10 FragmentInstanceState::ExecInternal()
runtime/fragment-instance-state.cc:310:59
#11 FragmentInstanceState::Exec() runtime/fragment-instance-state.cc:95:14
#12 QueryState::ExecFInstance(FragmentInstanceState*)
runtime/query-state.cc:488:24
#13 QueryState::StartFInstances()::$_0::operator()() const
runtime/query-state.cc:416:35
#20 thread_proxy (exprs/expr-test+0x55ca939)
exprs/string-functions-ir.cc:871:17: runtime error: null pointer passed as
argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified here
#0 StringFunctions::ConcatWs(impala_udf::FunctionContext*,
impala_udf::StringVal const&, int, impala_udf::StringVal const*)
exprs/string-functions-ir.cc:871:5
#1 StringFunctions::Concat(impala_udf::FunctionContext*, int,
impala_udf::StringVal const*) exprs/string-functions-ir.cc:843:10
#2 impala_udf::StringVal
ScalarFnCall::InterpretEval<impala_udf::StringVal>(ScalarExprEvaluator*,
TupleRow const*) const exprs/scalar-fn-call.cc:510:95
#3 ScalarFnCall::GetStringVal(ScalarExprEvaluator*, TupleRow const*) const
exprs/scalar-fn-call.cc:599:44
#4 ScalarExprEvaluator::GetValue(ScalarExpr const&, TupleRow const*)
exprs/scalar-expr-evaluator.cc:299:38
#5 ScalarExprEvaluator::GetValue(TupleRow const*)
exprs/scalar-expr-evaluator.cc:250:10
#6 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor
const&, ScalarExprEvaluator* const*, MemPool*, StringValue**, int*, int*)
runtime/tuple.cc:222:27
#7 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor
const&, vector<ScalarExprEvaluator*> const&, MemPool*, vector<StringValue*>*,
int*) runtime/tuple.h:174:5
#8 UnionNode::MaterializeExprs(vector<ScalarExprEvaluator*> const&,
TupleRow*, unsigned char*, RowBatch*) exec/union-node-ir.cc:29:14
#9 UnionNode::GetNextConst(RuntimeState*, RowBatch*)
exec/union-node.cc:263:5
#10 UnionNode::GetNext(RuntimeState*, RowBatch*, bool*)
exec/union-node.cc:296:45
#11 FragmentInstanceState::ExecInternal()
runtime/fragment-instance-state.cc:310:59
#12 FragmentInstanceState::Exec() runtime/fragment-instance-state.cc:95:14
#13 QueryState::ExecFInstance(FragmentInstanceState*)
runtime/query-state.cc:488:24
#14 QueryState::StartFInstances()::$_0::operator()() const
runtime/query-state.cc:416:35
#21 thread_proxy (exprs/expr-test+0x55ca939)
exprs/string-functions-ir.cc:873:17: runtime error: null pointer passed as
argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified here
#0 StringFunctions::ConcatWs(impala_udf::FunctionContext*,
impala_udf::StringVal const&, int, impala_udf::StringVal const*)
exprs/string-functions-ir.cc:873:5
#1 StringFunctions::Concat(impala_udf::FunctionContext*, int,
impala_udf::StringVal const*) exprs/string-functions-ir.cc:843:10
#2 impala_udf::StringVal
ScalarFnCall::InterpretEval<impala_udf::StringVal>(ScalarExprEvaluator*,
TupleRow const*) const exprs/scalar-fn-call.cc:510:95
#3 ScalarFnCall::GetStringVal(ScalarExprEvaluator*, TupleRow const*) const
exprs/scalar-fn-call.cc:599:44
#4 ScalarExprEvaluator::GetValue(ScalarExpr const&, TupleRow const*)
exprs/scalar-expr-evaluator.cc:299:38
#5 ScalarExprEvaluator::GetValue(TupleRow const*)
exprs/scalar-expr-evaluator.cc:250:10
#6 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor
const&, ScalarExprEvaluator* const*, MemPool*, StringValue**, int*, int*)
runtime/tuple.cc:222:27
#7 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor
const&, vector<ScalarExprEvaluator*> const&, MemPool*, vector<StringValue*>*,
int*) runtime/tuple.h:174:5
#8 UnionNode::MaterializeExprs(vector<ScalarExprEvaluator*> const&,
TupleRow*, unsigned char*, RowBatch*) exec/union-node-ir.cc:29:14
#9 UnionNode::GetNextConst(RuntimeState*, RowBatch*)
exec/union-node.cc:263:5
#10 UnionNode::GetNext(RuntimeState*, RowBatch*, bool*)
exec/union-node.cc:296:45
#11 FragmentInstanceState::ExecInternal()
runtime/fragment-instance-state.cc:310:59
#12 FragmentInstanceState::Exec() runtime/fragment-instance-state.cc:95:14
#13 QueryState::ExecFInstance(FragmentInstanceState*)
runtime/query-state.cc:488:24
#14 QueryState::StartFInstances()::$_0::operator()() const
runtime/query-state.cc:416:35
#21 thread_proxy (exprs/expr-test+0x55ca939)
runtime/raw-value.cc:159:27: runtime error: null pointer passed as argument 2,
which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified here
#0 RawValue::Write(void const*, void*, ColumnType const&, MemPool*)
runtime/raw-value.cc:159:9
#1 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor
const&, ScalarExprEvaluator* const*, MemPool*, StringValue**, int*, int*)
runtime/tuple.cc:225:7
#2 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor
const&, vector<ScalarExprEvaluator*> const&, MemPool*, vector<StringValue*>*,
int*) runtime/tuple.h:174:5
#3 UnionNode::MaterializeExprs(vector<ScalarExprEvaluator*> const&,
TupleRow*, unsigned char*, RowBatch*) exec/union-node-ir.cc:29:14
#4 UnionNode::GetNextConst(RuntimeState*, RowBatch*)
exec/union-node.cc:263:5
#5 UnionNode::GetNext(RuntimeState*, RowBatch*, bool*)
exec/union-node.cc:296:45
#6 FragmentInstanceState::ExecInternal()
runtime/fragment-instance-state.cc:310:59
#7 FragmentInstanceState::Exec() runtime/fragment-instance-state.cc:95:14
#8 QueryState::ExecFInstance(FragmentInstanceState*)
runtime/query-state.cc:488:24
#9 QueryState::StartFInstances()::$_0::operator()() const
runtime/query-state.cc:416:35
#16 thread_proxy (exprs/expr-test+0x55ca939)
udf/udf.cc:521:24: runtime error: null pointer passed as argument 2, which is
declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified here
#0 impala_udf::StringVal::CopyFrom(impala_udf::FunctionContext*, unsigned
char const*, unsigned long) udf/udf.cc:521:5
#1 AnyValUtil::FromBuffer(impala_udf::FunctionContext*, char const*, int)
exprs/anyval-util.h:241:12
#2 StringFunctions::RegexpExtract(impala_udf::FunctionContext*,
impala_udf::StringVal const&, impala_udf::StringVal const&,
impala_udf::BigIntVal const&) exprs/string-functions-ir.cc:726:10
#3 impala_udf::StringVal
ScalarFnCall::InterpretEval<impala_udf::StringVal>(ScalarExprEvaluator*,
TupleRow const*) const exprs/scalar-fn-call.cc:485:580
#4 ScalarFnCall::GetStringVal(ScalarExprEvaluator*, TupleRow const*) const
exprs/scalar-fn-call.cc:599:44
#5 ScalarExprEvaluator::GetValue(ScalarExpr const&, TupleRow const*)
exprs/scalar-expr-evaluator.cc:299:38
#6 ScalarExprEvaluator::GetValue(TupleRow const*)
exprs/scalar-expr-evaluator.cc:250:10
#7 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor
const&, ScalarExprEvaluator* const*, MemPool*, StringValue**, int*, int*)
runtime/tuple.cc:222:27
#8 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor
const&, vector<ScalarExprEvaluator*> const&, MemPool*, vector<StringValue*>*,
int*) runtime/tuple.h:174:5
#9 UnionNode::MaterializeExprs(vector<ScalarExprEvaluator*> const&,
TupleRow*, unsigned char*, RowBatch*) exec/union-node-ir.cc:29:14
#10 UnionNode::GetNextConst(RuntimeState*, RowBatch*)
exec/union-node.cc:263:5
#11 UnionNode::GetNext(RuntimeState*, RowBatch*, bool*)
exec/union-node.cc:296:45
#12 FragmentInstanceState::ExecInternal()
runtime/fragment-instance-state.cc:310:59
#13 FragmentInstanceState::Exec() runtime/fragment-instance-state.cc:95:14
#14 QueryState::ExecFInstance(FragmentInstanceState*)
runtime/query-state.cc:488:24
#15 QueryState::StartFInstances()::$_0::operator()() const
runtime/query-state.cc:416:35
#22 thread_proxy (exprs/expr-test+0x55ca939)
util/coding-util-test.cc:45:10: runtime error: null pointer passed as argument
1, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified here
#0 TestUrl(string const&, string const&, bool) util/coding-util-test.cc:45:3
#1 UrlCodingTest_BlankString_Test::TestBody() util/coding-util-test.cc:88:3
#2 void
testing::internal::HandleExceptionsInMethodIfSupported<testing::Test,
void>(testing::Test*, void (testing::Test::*)(), char const*)
(util/coding-util-test+0x6630f42)
#8 main util/coding-util-test.cc:123:192
util/decompress-test.cc:126:261: runtime error: null pointer passed as argument
1, which is declared to never be null
/usr/include/string.h:66:58: note: nonnull attribute specified here
#0 DecompressorTest::CompressAndDecompress(Codec*, Codec*, long, unsigned
char*) util/decompress-test.cc:126:254
#1 DecompressorTest::RunTest(THdfsCompression::type)
util/decompress-test.cc:84:9
#2 DecompressorTest_Default_Test::TestBody() util/decompress-test.cc:373:3
#3 void
testing::internal::HandleExceptionsInMethodIfSupported<testing::Test,
void>(testing::Test*, void (testing::Test::*)(), char const*)
(util/decompress-test+0x6642bb2)
#9 main util/decompress-test.cc:479:47
util/decompress-test.cc:148:261: runtime error: null pointer passed as argument
1, which is declared to never be null
/usr/include/string.h:66:58: note: nonnull attribute specified here
#0 DecompressorTest::CompressAndDecompress(Codec*, Codec*, long, unsigned
char*) util/decompress-test.cc:148:254
#1 DecompressorTest::RunTest(THdfsCompression::type)
util/decompress-test.cc:84:9
#2 DecompressorTest_Default_Test::TestBody() util/decompress-test.cc:373:3
#3 void
testing::internal::HandleExceptionsInMethodIfSupported<testing::Test,
void>(testing::Test*, void (testing::Test::*)(), char const*)
(util/decompress-test+0x6642bb2)
#9 main util/decompress-test.cc:479:47
util/decompress-test.cc:269:261: runtime error: null pointer passed as argument
1, which is declared to never be null
/usr/include/string.h:66:58: note: nonnull attribute specified here
#0 DecompressorTest::CompressAndDecompressNoOutputAllocated(Codec*, Codec*,
long, unsigned char*) util/decompress-test.cc:269:254
#1 DecompressorTest::RunTest(THdfsCompression::type)
util/decompress-test.cc:71:7
#2 DecompressorTest_LZ4_Test::TestBody() util/decompress-test.cc:381:3
#3 void
testing::internal::HandleExceptionsInMethodIfSupported<testing::Test,
void>(testing::Test*, void (testing::Test::*)(), char const*)
(util/decompress-test+0x6642bb2)
#9 main util/decompress-test.cc:479:47
util/decompress-test.cc:221:329: runtime error: null pointer passed as argument
1, which is declared to never be null
/usr/include/string.h:66:58: note: nonnull attribute specified here
#0 DecompressorTest::StreamingDecompress(Codec*, long, unsigned char*,
long, unsigned char*, bool, long*) util/decompress-test.cc:221:322
#1 DecompressorTest::CompressAndStreamingDecompress(Codec*, Codec*, long,
unsigned char*) util/decompress-test.cc:245:35
#2 DecompressorTest::RunTestStreaming(THdfsCompression::type)
util/decompress-test.cc:104:5
#3 DecompressorTest_Gzip_Test::TestBody() util/decompress-test.cc:386:3
#4 void
testing::internal::HandleExceptionsInMethodIfSupported<testing::Test,
void>(testing::Test*, void (testing::Test::*)(), char const*)
(util/decompress-test+0x6642bb2)
#10 main util/decompress-test.cc:479:47
util/streaming-sampler.h:55:22: runtime error: null pointer passed as argument
2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified here
#0 StreamingSampler<long, 64>::StreamingSampler(int, vector<long> const&)
util/streaming-sampler.h:55:5
#1 RuntimeProfile::TimeSeriesCounter::TimeSeriesCounter(string const&,
TUnit::type, int, vector<long> const&) util/runtime-profile-counters.h:401:53
#2 RuntimeProfile::Update(vector<TRuntimeProfileNode> const&, int*)
util/runtime-profile.cc:310:28
#3 RuntimeProfile::Update(TRuntimeProfileTree const&)
util/runtime-profile.cc:245:3
#4
Coordinator::BackendState::InstanceStats::Update(TFragmentInstanceExecStatus
const&, Coordinator::ExecSummary*, ProgressUpdater*)
runtime/coordinator-backend-state.cc:473:13
#5 Coordinator::BackendState::ApplyExecStatusReport(TReportExecStatusParams
const&, Coordinator::ExecSummary*, ProgressUpdater*)
runtime/coordinator-backend-state.cc:286:21
#6 Coordinator::UpdateBackendExecStatus(TReportExecStatusParams const&)
runtime/coordinator.cc:678:22
#7 ClientRequestState::UpdateBackendExecStatus(TReportExecStatusParams
const&) service/client-request-state.cc:1253:18
#8 ImpalaServer::ReportExecStatus(TReportExecStatusResult&,
TReportExecStatusParams const&) service/impala-server.cc:1343:18
#9 ImpalaInternalService::ReportExecStatus(TReportExecStatusResult&,
TReportExecStatusParams const&) service/impala-internal-service.cc:87:19
#24 thread_proxy (exprs/expr-test+0x55ca939)
Change-Id: I317ccc99549744a26d65f3e07242079faad0355a
Reviewed-on: http://gerrit.cloudera.org:8080/11545
Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
---
M be/src/exprs/string-functions-ir.cc
M be/src/runtime/raw-value.cc
M be/src/udf/udf.cc
M be/src/util/coding-util-test.cc
M be/src/util/decompress-test.cc
M be/src/util/streaming-sampler.h
M be/src/util/ubsan.h
7 files changed, 27 insertions(+), 12 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/11545
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I317ccc99549744a26d65f3e07242079faad0355a
Gerrit-Change-Number: 11545
Gerrit-PatchSet: 3
Gerrit-Owner: Jim Apple <jbapple-imp...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jbapple-imp...@apache.org>
Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>
