[jira] [Commented] (ARROW-2023) [C++] Test opening IPC stream reader or file reader on an empty InputStream

2018-02-27 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/ARROW-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378287#comment-16378287
 ] 

ASF GitHub Bot commented on ARROW-2023:
---

xhochy closed pull request #1503: ARROW-2023: [C++] Fix ASAN failure on 
malformed / empty stream input, enable ASAN builds, add more dev docs
URL: https://github.com/apache/arrow/pull/1503
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/cpp/CMakeLists.txt b/cpp/CMakeLists.txt
index b9221414b..8c0e95634 100644
--- a/cpp/CMakeLists.txt
+++ b/cpp/CMakeLists.txt
@@ -291,8 +291,6 @@ include(ThirdpartyToolchain)
 set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${CXX_COMMON_FLAGS}")
 set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${ARROW_CXXFLAGS}")
 
-message(STATUS "CMAKE_CXX_FLAGS: ${CMAKE_CXX_FLAGS}")
-
 if ("${COMPILER_FAMILY}" STREQUAL "clang")
   # Using Clang with ccache causes a bunch of spurious warnings that are
   # purportedly fixed in the next version of ccache. See the following for 
details:
@@ -305,7 +303,7 @@ endif()
 
 # ASAN / TSAN / UBSAN
 if(ARROW_FUZZING)
-set(ARROW_USE_COVERAGE ON)
+  set(ARROW_USE_COVERAGE ON)
 endif()
 include(san-config)
 
@@ -333,6 +331,9 @@ if ("${ARROW_GENERATE_COVERAGE}")
   endif()
 endif()
 
+# CMAKE_CXX_FLAGS now fully assembled
+message(STATUS "CMAKE_CXX_FLAGS: ${CMAKE_CXX_FLAGS}")
+
 # set compile output directory
 string (TOLOWER ${CMAKE_BUILD_TYPE} BUILD_SUBDIR_NAME)
 
diff --git a/cpp/README.md b/cpp/README.md
index 1daf86381..daeeade72 100644
--- a/cpp/README.md
+++ b/cpp/README.md
@@ -99,7 +99,14 @@ and benchmarks or `make runbenchmark` to run only the 
benchmark tests.
 
 Benchmark logs will be placed in the build directory under 
`build/benchmark-logs`.
 
-## Building/Running fuzzers
+### Testing with LLVM AddressSanitizer
+
+To use AddressSanitizer (ASAN) to find bad memory accesses or leaks with LLVM,
+pass `-DARROW_USE_ASAN=ON` when building. You must use clang to compile with
+ASAN, and `ARROW_USE_ASAN` is mutually-exclusive with the valgrind option
+`ARROW_TEST_MEMCHECK`.
+
+### Building/Running fuzzers
 
 Fuzzers can help finding unhandled exceptions and problems with untrusted 
input that
 may lead to crashes, security issues and undefined behavior. They do this by
@@ -128,6 +135,22 @@ stack trace as well as the input data. After a problem was 
found this way, it sh
 be reported and fixed. Usually, the fuzzing process cannot be continued until 
the
 fix is applied, since the fuzzer usually converts to the problem again.
 
+If you build fuzzers with ASAN, you need to set the `ASAN_SYMBOLIZER_PATH`
+environment variable to the absolute path of `llvm-symbolizer`, which is a tool
+that ships with LLVM.
+
+```shell
+export ASAN_SYMBOLIZER_PATH=$(type -p llvm-symbolizer)
+```
+
+Note that some fuzzer builds currently reject paths with a version qualifier
+(like `llvm-sanitizer-5.0`). To overcome this, set an appropriate symlink
+(here, when using LLVM 5.0):
+
+```shell
+ln -sf /usr/bin/llvm-sanitizer-5.0 /usr/bin/llvm-sanitizer
+```
+
 There are some problems that may occur during the compilation process:
 
 - libfuzzer was not distributed with your LLVM: `ld: file not found: 
.../libLLVMFuzzer.a`
diff --git a/cpp/build-support/lsan-suppressions.txt 
b/cpp/build-support/lsan-suppressions.txt
new file mode 100644
index 0..927afb397
--- /dev/null
+++ b/cpp/build-support/lsan-suppressions.txt
@@ -0,0 +1,19 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# False positive from atexit() registration in libc
+leak:*__new_exitfn*
diff --git a/cpp/build-support/run-test.sh b/cpp/build-support/run-test.sh
index b4da4f3f0..656ab7bd3 100755
--- a/cpp/build-support/run-test.sh
+++ b/cpp/build-support/run-test.sh
@@ -88,12 +88,6 @@ function setup_sanitizers() {
   # Set up suppressions for LeakSanitizer
   LSAN_OPTIONS="$LSAN_OPTIONS 
suppressions=$ROOT/build-support/lsan-suppressions.txt"
   export LSAN_OPTIONS
-
-  # Suppressions require

[jira] [Commented] (ARROW-2023) [C++] Test opening IPC stream reader or file reader on an empty InputStream

2018-02-26 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/ARROW-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377663#comment-16377663
 ] 

ASF GitHub Bot commented on ARROW-2023:
---

wesm commented on issue #1503: ARROW-2023: [C++] Fix ASAN failure on malformed 
/ empty stream input, enable ASAN builds, add more dev docs
URL: https://github.com/apache/arrow/pull/1503#issuecomment-368666040
 
 
   appveyor build: https://ci.appveyor.com/project/wesm/arrow/build/1.0.1724


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> [C++] Test opening IPC stream reader or file reader on an empty InputStream
> ---
>
> Key: ARROW-2023
> URL: https://issues.apache.org/jira/browse/ARROW-2023
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: C++
>Reporter: Wes McKinney
>Assignee: Wes McKinney
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.9.0
>
>
> This was reported to segfault in ARROW-1589



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ARROW-2023) [C++] Test opening IPC stream reader or file reader on an empty InputStream

2018-02-26 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/ARROW-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377554#comment-16377554
 ] 

ASF GitHub Bot commented on ARROW-2023:
---

wesm commented on issue #1503: ARROW-2023: [C++] Fix ASAN failure on malformed 
/ empty stream input, enable ASAN builds, add more dev docs
URL: https://github.com/apache/arrow/pull/1503#issuecomment-368645019
 
 
   @xhochy @crepererum this is ready to go. note this only fixes one of the 
fuzzer failures, now it returns a Flatbuffers error:
   
   ```
   ==19598==The signal is caused by a READ memory access.
   #0 0x682c79 in int flatbuffers::ReadScalar(void const*) 
/home/wesm/cpp-toolchain/include/flatbuffers/flatbuffers.h:198:23
   #1 0x682b6f in flatbuffers::Table::GetVTable() const 
/home/wesm/cpp-toolchain/include/flatbuffers/flatbuffers.h:1583:20
   #2 0x682a3a in flatbuffers::Table::GetOptionalFieldOffset(unsigned 
short) const /home/wesm/cpp-toolchain/include/flatbuffers/flatbuffers.h:1590:19
   #3 0x6b0e8e in long flatbuffers::Table::GetField(unsigned short, 
long) const /home/wesm/cpp-toolchain/include/flatbuffers/flatbuffers.h:1599:25
   #4 0x783934 in org::apache::arrow::flatbuf::Message::bodyLength() const 
/home/wesm/code/arrow/cpp/build/src/arrow/ipc/Message_generated.h:294:12
   #5 0x77ced7 in 
arrow::ipc::Message::ReadFrom(std::shared_ptr const&, 
arrow::io::InputStream*, std::unique_ptr*) 
/home/wesm/code/arrow/cpp/build/../src/arrow/ipc/message.cc:140:37
   #6 0x7824d9 in arrow::ipc::ReadMessage(arrow::io::InputStream*, 
std::unique_ptr*) /home/wesm/code/arrow/cpp/build/../src/arrow/ipc/message.cc:236:10
   #7 0x784036 in 
arrow::ipc::InputStreamMessageReader::ReadNextMessage(std::unique_ptr*) 
/home/wesm/code/arrow/cpp/build/../src/arrow/ipc/message.cc:255:12
   #8 0x67abb8 in 
arrow::ipc::ReadMessageAndValidate(arrow::ipc::MessageReader*, 
arrow::ipc::Message::Type, bool, std::unique_ptr*) 
/home/wesm/code/arrow/cpp/build/../src/arrow/ipc/reader.cc:371:29
   #9 0x6b2e82 in 
arrow::ipc::RecordBatchStreamReader::RecordBatchStreamReaderImpl::ReadSchema() 
/home/wesm/code/arrow/cpp/build/../src/arrow/ipc/reader.cc:426:23
   #10 0x67d40f in 
arrow::ipc::RecordBatchStreamReader::RecordBatchStreamReaderImpl::Open(std::unique_ptr) 
/home/wesm/code/arrow/cpp/build/../src/arrow/ipc/reader.cc:407:12
   #11 0x6724d2 in 
arrow::ipc::RecordBatchStreamReader::Open(std::unique_ptr, 
std::shared_ptr*) 
/home/wesm/code/arrow/cpp/build/../src/arrow/ipc/reader.cc:477:36
   #12 0x672dc0 in 
arrow::ipc::RecordBatchStreamReader::Open(arrow::io::InputStream*, 
std::shared_ptr*) 
/home/wesm/code/arrow/cpp/build/../src/arrow/ipc/reader.cc:484:10
   #13 0x651842 in LLVMFuzzerTestOneInput 
/home/wesm/code/arrow/cpp/build/../src/arrow/ipc/ipc-fuzzing-test.cc:31:12
   #14 0x52a0ae in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, 
unsigned long) (/home/wesm/code/arrow/cpp/build/debug/ipc-fuzzing-test+0x52a0ae)
   #15 0x529a94 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned 
long, bool, fuzzer::InputInfo*) 
(/home/wesm/code/arrow/cpp/build/debug/ipc-fuzzing-test+0x529a94)
   #16 0x52ad51 in fuzzer::Fuzzer::MutateAndTestOne() 
(/home/wesm/code/arrow/cpp/build/debug/ipc-fuzzing-test+0x52ad51)
   #17 0x52af17 in fuzzer::Fuzzer::Loop() 
(/home/wesm/code/arrow/cpp/build/debug/ipc-fuzzing-test+0x52af17)
   #18 0x523415 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned 
char const*, unsigned long)) 
(/home/wesm/code/arrow/cpp/build/debug/ipc-fuzzing-test+0x523415)
   #19 0x51e280 in main 
(/home/wesm/code/arrow/cpp/build/debug/ipc-fuzzing-test+0x51e280)
   #20 0x7fdeb9619f44 in __libc_start_main 
/build/eglibc-ripdx6/eglibc-2.19/csu/libc-start.c:287
   #21 0x51e185 in _start 
(/home/wesm/code/arrow/cpp/build/debug/ipc-fuzzing-test+0x51e185)
   ```
   
   I created https://issues.apache.org/jira/browse/ARROW- about adding an 
option to validate input messages


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> [C++] Test opening IPC stream reader or file reader on an empty InputStream
> ---
>
> Key: ARROW-2023
> URL: https://issues.apache.org/jira/browse/ARROW-2023
> Project: Apache Arrow
>  Issue Type: Bug
>

[jira] [Commented] (ARROW-2023) [C++] Test opening IPC stream reader or file reader on an empty InputStream

2018-01-24 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/ARROW-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338411#comment-16338411
 ] 

ASF GitHub Bot commented on ARROW-2023:
---

wesm opened a new pull request #1503: WIP ARROW-2023: [C++] Start stream reader 
tests for malformed inputs
URL: https://github.com/apache/arrow/pull/1503
 
 
   More to do here, see discussion in 
https://issues.apache.org/jira/browse/ARROW-1589


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> [C++] Test opening IPC stream reader or file reader on an empty InputStream
> ---
>
> Key: ARROW-2023
> URL: https://issues.apache.org/jira/browse/ARROW-2023
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: C++
>Reporter: Wes McKinney
>Assignee: Wes McKinney
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.9.0
>
>
> This was reported to segfault in ARROW-1589



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)