[Impala-ASF-CR] IMPALA-5031: NULL is undefined in memcpy and memcmp
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13436 ) Change subject: IMPALA-5031: NULL is undefined in memcpy and memcmp .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3395/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13436 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9362ce6b9ba470ed90e5bd2dc313b66ebd8c6af5 Gerrit-Change-Number: 13436 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sun, 26 May 2019 02:13:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: signed overflow is undefined behavior
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13437 ) Change subject: IMPALA-5031: signed overflow is undefined behavior .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3394/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13437 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73dd6802ec1023275d09a99a2950f3558313fc8e Gerrit-Change-Number: 13437 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sun, 26 May 2019 01:54:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: NULL is undefined in memcpy and memcmp
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13436 ) Change subject: IMPALA-5031: NULL is undefined in memcpy and memcmp .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/13436/2/be/src/runtime/tuple.h File be/src/runtime/tuple.h: http://gerrit.cloudera.org:8080/#/c/13436/2/be/src/runtime/tuple.h@107 PS2, Line 107: Ubsan::MemSet(reinterpret_cast(this) + null_bytes_offset, 0, num_null_bytes); line too long (91 > 90) -- To view, visit http://gerrit.cloudera.org:8080/13436 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9362ce6b9ba470ed90e5bd2dc313b66ebd8c6af5 Gerrit-Change-Number: 13436 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sun, 26 May 2019 01:33:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5031: NULL is undefined in memcpy and memcmp
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13436 to look at the new patch set (#2). Change subject: IMPALA-5031: NULL is undefined in memcpy and memcmp .. IMPALA-5031: NULL is undefined in memcpy and memcmp This patch fixes UBSAN "null pointer passed as argument" errors in the end-to-end tests. These are undefined behavior according to "7.1.4 Use of library functions" in the C99 standard (which is included in C++14 in section [intro.refs]): If an argument to a function has an invalid value (such as a value outside the domain of the function, or a pointer outside the address space of the program, or a null pointer, or a pointer to non-modifiable storage when the corresponding parameter is not const-qualified) or a type (after promotion) not expected by a function with variable number of arguments, the behavior is undefined. The interesting parts of the backtraces are: runtime/sorter.cc:575:18: runtime error: null pointer passed as argument 2, which is declared to never be null /usr/include/string.h:43:45: note: nonnull attribute specified here #0 Sorter::Run::CopyVarLenData(vector const&, unsigned char*) runtime/sorter.cc:575:5 #1 Status Sorter::Run::AddBatchInternal(RowBatch*, int, int*) runtime/sorter.cc:232:11 #2 Sorter::Run::AddInputBatch(RowBatch*, int, int*) runtime/sorter.cc:660:12 #3 Sorter::AddBatchNoSpill(RowBatch*, int, int*) runtime/sorter.cc:882:58 #4 Sorter::AddBatch(RowBatch*) runtime/sorter.cc:862:45 #5 SortNode::SortInput(RuntimeState*) exec/sort-node.cc:177:54 #6 SortNode::Open(RuntimeState*) exec/sort-node.cc:90:43 runtime/tuple.cc:105:25: runtime error: null pointer passed as argument 2, which is declared to never be null /usr/include/string.h:43:45: note: nonnull attribute specified here #0 Tuple::DeepCopyVarlenData(TupleDescriptor const&, MemPool*) runtime/tuple.cc:105:5 #1 Tuple::DeepCopy(Tuple*, TupleDescriptor const&, MemPool*) runtime/tuple.cc:94:35 #2 Tuple::DeepCopy(TupleDescriptor const&, MemPool*) runtime/tuple.cc:85:3 #3 KrpcDataStreamSender::Channel::AddRow(TupleRow*) runtime/krpc-data-stream-sender.cc:509:43 #4 KrpcDataStreamSender::AddRowToChannel(int, TupleRow*) runtime/krpc-data-stream-sender.cc:846 #5 () runtime/tuple.cc:146:19: runtime error: null pointer passed as argument 2, which is declared to never be null /usr/include/string.h:43:45: note: nonnull attribute specified here #0 Tuple::DeepCopyVarlenData(TupleDescriptor const&, char**, int*, bool) runtime/tuple.cc:146:5 #1 Tuple::DeepCopy(TupleDescriptor const&, char**, int*, bool) runtime/tuple.cc:135:35 #2 RowBatch::SerializeInternal(long, FixedSizeHashTable*, vector*, string*) runtime/row-batch.cc:392:14 #3 RowBatch::Serialize(bool, vector*, string*, long*, bool*) runtime/row-batch.cc:290:45 #4 RowBatch::Serialize(OutboundRowBatch*) runtime/row-batch.cc:259:43 #5 KrpcDataStreamSender::SerializeBatch(RowBatch*, OutboundRowBatch*, int) runtime/krpc-data-stream-sender.cc:955:50 #6 KrpcDataStreamSender::Send(RuntimeState*, RowBatch*) runtime/krpc-data-stream-sender.cc:870:45 runtime/tuple.h:106:12: runtime error: null pointer passed as argument 1, which is declared to never be null /usr/include/string.h:62:79: note: nonnull attribute specified here #0 Tuple::ClearNullBits(int, int) runtime/tuple.h:106:5 #1 HdfsScanner::InitTuple(TupleDescriptor const*, Tuple*, Tuple*) exec/hdfs-scanner.h:512:14 #2 HdfsOrcScanner::AssembleCollection(OrcComplexColumnReader const&, int, CollectionValueBuilder*) exec/hdfs-orc-scanner.cc:742:7 #3 OrcCollectionReader::ReadValue(int, Tuple*, MemPool*) exec/orc-column-readers.cc:375:20 #4 OrcStructReader::ReadValue(int, Tuple*, MemPool*) exec/orc-column-readers.cc:322:52 #5 OrcListReader::ReadChildrenValue(int, int, Tuple*, MemPool*) const exec/orc-column-readers.cc:473:52 #6 HdfsOrcScanner::AssembleCollection(OrcComplexColumnReader const&, int, CollectionValueBuilder*) exec/hdfs-orc-scanner.cc:743:60 #7 OrcCollectionReader::ReadValue(int, Tuple*, MemPool*) exec/orc-column-readers.cc:375:20 #8 OrcStructReader::TransferTuple(Tuple*, MemPool*) exec/orc-column-readers.cc:346:52 #9 HdfsOrcScanner::TransferTuples(OrcComplexColumnReader*, RowBatch*) exec/hdfs-orc-scanner.cc:669:58 #10 HdfsOrcScanner::AssembleRows(RowBatch*) exec/hdfs-orc-scanner.cc:629:45 #11 HdfsOrcScanner::GetNextInternal(RowBatch*) exec/hdfs-orc-scanner.cc:507:19 #12 HdfsOrcScanner::ProcessSplit() exec/hdfs-orc-scanner.cc:426:21 #13
[Impala-ASF-CR] IMPALA-5031: signed overflow is undefined behavior
Jim Apple has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13437 Change subject: IMPALA-5031: signed overflow is undefined behavior .. IMPALA-5031: signed overflow is undefined behavior Fix remaining signed overflow undefined behaviors in end-to-end tests. The interesting part of the backtraces: exprs/aggregate-functions-ir.cc:464:25: runtime error: signed integer overflow: 0x5a4728ca063b522c0b728f80 + 0x3c2f7086aed236c807a1b500 cannot be represented in type '__int128' #0 AggregateFunctions::DecimalAvgMerge( impala_udf::FunctionContext*, impala_udf::StringVal const&, impala_udf::StringVal*) exprs/aggregate-functions-ir.cc:464:25 #1 AggFnEvaluator::Update(TupleRow const*, Tuple*, void*) exprs/agg-fn-evaluator.cc:327:7 #2 AggFnEvaluator::Add(TupleRow const*, Tuple*) exprs/agg-fn-evaluator.h:257:3 #3 Aggregator::UpdateTuple(AggFnEvaluator**, Tuple*, TupleRow*, bool) exec/aggregator.cc:167:24 #4 NonGroupingAggregator::AddBatchImpl(RowBatch*) exec/non-grouping-aggregator-ir.cc:27:5 #5 NonGroupingAggregator::AddBatch(RuntimeState*, RowBatch*) exec/non-grouping-aggregator.cc:124:45 #6 AggregationNode::Open(RuntimeState*) exec/aggregation-node.cc:70:57 exprs/aggregate-functions-ir.cc:513:12: runtime error: signed integer overflow: -8282081183197145958 + -4473782455107795527 cannot be represented in type 'long' #0 void AggregateFunctions::SumUpdate(impala_udf::FunctionContext*, impala_udf::BigIntVal const&, impala_udf::BigIntVal*) exprs/aggregate-functions-ir.cc:513:12 #1 AggFnEvaluator::Update(TupleRow const*, Tuple*, void*) exprs/agg-fn-evaluator.cc:327:7 #2 AggFnEvaluator::Add(TupleRow const*, Tuple*) exprs/agg-fn-evaluator.h:257:3 #3 Aggregator::UpdateTuple(AggFnEvaluator**, Tuple*, TupleRow*, bool) exec/aggregator.cc:167:24 #4 NonGroupingAggregator::AddBatchImpl(RowBatch*) exec/non-grouping-aggregator-ir.cc:27:5 #5 NonGroupingAggregator::AddBatch(RuntimeState*, RowBatch*) exec/non-grouping-aggregator.cc:124:45 #6 AggregationNode::Open(RuntimeState*) exec/aggregation-node.cc:70:57 exprs/aggregate-functions-ir.cc:585:14: runtime error: signed integer overflow: 0x5a4728ca063b522c0b728f80 + 0x3c2f7086aed236c807a1b500 cannot be represented in type '__int128' #0 AggregateFunctions::SumDecimalMerge( impala_udf::FunctionContext*, impala_udf::DecimalVal const&, impala_udf::DecimalVal*) exprs/aggregate-functions-ir.cc:585:14 #1 AggFnEvaluator::Update(TupleRow const*, Tuple*, void*) exprs/agg-fn-evaluator.cc:327:7 #2 AggFnEvaluator::Add(TupleRow const*, Tuple*) exprs/agg-fn-evaluator.h:257:3 #3 Aggregator::UpdateTuple(AggFnEvaluator**, Tuple*, TupleRow*, bool) exec/aggregator.cc:167:24 #4 NonGroupingAggregator::AddBatchImpl(RowBatch*) exec/non-grouping-aggregator-ir.cc:27:5 #5 NonGroupingAggregator::AddBatch(RuntimeState*, RowBatch*) exec/non-grouping-aggregator.cc:124:45 #6 AggregationNode::Open(RuntimeState*) exec/aggregation-node.cc:70:57 runtime/decimal-value.inline.h:145:12: runtime error: signed integer overflow: 18 * 0x0785ee10d5da46d900f436a0 cannot be represented in type '__int128' #0 DecimalValue<__int128>::ScaleTo(int, int, int, bool*) const runtime/decimal-value.inline.h:145:12 #1 DecimalOperators::ScaleDecimalValue( impala_udf::FunctionContext*, DecimalValue const&, int, int, int) exprs/decimal-operators-ir.cc:132:41 #2 DecimalOperators::RoundDecimal(impala_udf::FunctionContext*, impala_udf::DecimalVal const&, int, int, int, int, DecimalOperators::DecimalRoundOp const&) exprs/decimal-operators-ir.cc:465:16 #3 DecimalOperators::RoundDecimal(impala_udf::FunctionContext*, impala_udf::DecimalVal const&, DecimalOperators::DecimalRoundOp const&) exprs/decimal-operators-ir.cc:519:10 #4 DecimalOperators::CastToDecimalVal( impala_udf::FunctionContext*, impala_udf::DecimalVal const&) exprs/decimal-operators-ir.cc:529:10 #5 impala_udf::DecimalVal ScalarFnCall::InterpretEval (ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:485:208 #6 ScalarFnCall::GetDecimalVal(ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:618:44 #7 ScalarExprEvaluator::GetValue(ScalarExpr const&, TupleRow const*) exprs/scalar-expr-evaluator.cc:321:27 #8 ScalarExprEvaluator::GetValue(TupleRow const*) exprs/scalar-expr-evaluator.cc:251:10 #9 Java_org_apache_impala_service_FeSupport_NativeEvalExprsWithoutRow service/fe-support.cc:246:26 #10 () runtime/multi-precision.h:116:21: runtime
[Impala-ASF-CR] IMPALA-5031: NULL is undefined in memcpy and memcmp
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13436 ) Change subject: IMPALA-5031: NULL is undefined in memcpy and memcmp .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3393/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13436 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9362ce6b9ba470ed90e5bd2dc313b66ebd8c6af5 Gerrit-Change-Number: 13436 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 25 May 2019 22:32:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: NULL is undefined in memcpy and memcmp
Jim Apple has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13436 Change subject: IMPALA-5031: NULL is undefined in memcpy and memcmp .. IMPALA-5031: NULL is undefined in memcpy and memcmp This patch fixes UBSAN "null pointer passed as argument" errors in the end-to-end tests. These are undefined behavior according to "7.1.4 Use of library functions" in the C99 standard (which is included in C++14 in section [intro.refs]): If an argument to a function has an invalid value (such as a value outside the domain of the function, or a pointer outside the address space of the program, or a null pointer, or a pointer to non-modifiable storage when the corresponding parameter is not const-qualified) or a type (after promotion) not expected by a function with variable number of arguments, the behavior is undefined. The interesting parts of the backtraces are: runtime/sorter.cc:575:18: runtime error: null pointer passed as argument 2, which is declared to never be null /usr/include/string.h:43:45: note: nonnull attribute specified here #0 Sorter::Run::CopyVarLenData(vector const&, unsigned char*) runtime/sorter.cc:575:5 #1 Status Sorter::Run::AddBatchInternal(RowBatch*, int, int*) runtime/sorter.cc:232:11 #2 Sorter::Run::AddInputBatch(RowBatch*, int, int*) runtime/sorter.cc:660:12 #3 Sorter::AddBatchNoSpill(RowBatch*, int, int*) runtime/sorter.cc:882:58 #4 Sorter::AddBatch(RowBatch*) runtime/sorter.cc:862:45 #5 SortNode::SortInput(RuntimeState*) exec/sort-node.cc:177:54 #6 SortNode::Open(RuntimeState*) exec/sort-node.cc:90:43 runtime/tuple.cc:105:25: runtime error: null pointer passed as argument 2, which is declared to never be null /usr/include/string.h:43:45: note: nonnull attribute specified here #0 Tuple::DeepCopyVarlenData(TupleDescriptor const&, MemPool*) runtime/tuple.cc:105:5 #1 Tuple::DeepCopy(Tuple*, TupleDescriptor const&, MemPool*) runtime/tuple.cc:94:35 #2 Tuple::DeepCopy(TupleDescriptor const&, MemPool*) runtime/tuple.cc:85:3 #3 KrpcDataStreamSender::Channel::AddRow(TupleRow*) runtime/krpc-data-stream-sender.cc:509:43 #4 KrpcDataStreamSender::AddRowToChannel(int, TupleRow*) runtime/krpc-data-stream-sender.cc:846 #5 () runtime/tuple.cc:146:19: runtime error: null pointer passed as argument 2, which is declared to never be null /usr/include/string.h:43:45: note: nonnull attribute specified here #0 Tuple::DeepCopyVarlenData(TupleDescriptor const&, char**, int*, bool) runtime/tuple.cc:146:5 #1 Tuple::DeepCopy(TupleDescriptor const&, char**, int*, bool) runtime/tuple.cc:135:35 #2 RowBatch::SerializeInternal(long, FixedSizeHashTable*, vector*, string*) runtime/row-batch.cc:392:14 #3 RowBatch::Serialize(bool, vector*, string*, long*, bool*) runtime/row-batch.cc:290:45 #4 RowBatch::Serialize(OutboundRowBatch*) runtime/row-batch.cc:259:43 #5 KrpcDataStreamSender::SerializeBatch(RowBatch*, OutboundRowBatch*, int) runtime/krpc-data-stream-sender.cc:955:50 #6 KrpcDataStreamSender::Send(RuntimeState*, RowBatch*) runtime/krpc-data-stream-sender.cc:870:45 runtime/tuple.h:106:12: runtime error: null pointer passed as argument 1, which is declared to never be null /usr/include/string.h:62:79: note: nonnull attribute specified here #0 Tuple::ClearNullBits(int, int) runtime/tuple.h:106:5 #1 HdfsScanner::InitTuple(TupleDescriptor const*, Tuple*, Tuple*) exec/hdfs-scanner.h:512:14 #2 HdfsOrcScanner::AssembleCollection(OrcComplexColumnReader const&, int, CollectionValueBuilder*) exec/hdfs-orc-scanner.cc:742:7 #3 OrcCollectionReader::ReadValue(int, Tuple*, MemPool*) exec/orc-column-readers.cc:375:20 #4 OrcStructReader::ReadValue(int, Tuple*, MemPool*) exec/orc-column-readers.cc:322:52 #5 OrcListReader::ReadChildrenValue(int, int, Tuple*, MemPool*) const exec/orc-column-readers.cc:473:52 #6 HdfsOrcScanner::AssembleCollection(OrcComplexColumnReader const&, int, CollectionValueBuilder*) exec/hdfs-orc-scanner.cc:743:60 #7 OrcCollectionReader::ReadValue(int, Tuple*, MemPool*) exec/orc-column-readers.cc:375:20 #8 OrcStructReader::TransferTuple(Tuple*, MemPool*) exec/orc-column-readers.cc:346:52 #9 HdfsOrcScanner::TransferTuples(OrcComplexColumnReader*, RowBatch*) exec/hdfs-orc-scanner.cc:669:58 #10 HdfsOrcScanner::AssembleRows(RowBatch*) exec/hdfs-orc-scanner.cc:629:45 #11 HdfsOrcScanner::GetNextInternal(RowBatch*) exec/hdfs-orc-scanner.cc:507:19 #12 HdfsOrcScanner::ProcessSplit() exec/hdfs-orc-scanner.cc:426:21 #13 HdfsScanNode::ProcessSplit(vector const&, MemPool*, io::ScanRange*,
[Impala-ASF-CR] IMPALA-5031: NULL is undefined in memcpy and memcmp
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13436 ) Change subject: IMPALA-5031: NULL is undefined in memcpy and memcmp .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/13436/1/be/src/runtime/tuple.h File be/src/runtime/tuple.h: http://gerrit.cloudera.org:8080/#/c/13436/1/be/src/runtime/tuple.h@107 PS1, Line 107: Ubsan::MemSet(reinterpret_cast(this) + null_bytes_offset, 0, num_null_bytes); line too long (91 > 90) -- To view, visit http://gerrit.cloudera.org:8080/13436 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9362ce6b9ba470ed90e5bd2dc313b66ebd8c6af5 Gerrit-Change-Number: 13436 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 25 May 2019 21:53:29 + Gerrit-HasComments: Yes
