[
https://issues.apache.org/jira/browse/ARROW-2222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16432282#comment-16432282
]
ASF GitHub Bot commented on ARROW-2222:
---------------------------------------
crepererum commented on issue #1763: ARROW-2222: handle untrusted inputs (POC)
URL: https://github.com/apache/arrow/pull/1763#issuecomment-380103831
## Environment
### Build Commands
```
env CC=clang CXX=clang++ cmake -DCMAKE_BUILD_TYPE=Release
-DARROW_BUILD_BENCHMARKS=ON -GNinja ..
env CC=clang CXX=clang++ ninja
```
### Conda
```
# Name Version Build Channel
boost 1.66.0 py36_1 conda-forge
boost-cpp 1.66.0 1 conda-forge
bzip2 1.0.6 1 conda-forge
ca-certificates 2018.1.18 0 conda-forge
certifi 2018.1.18 py36_0 conda-forge
clangdev 6.0.0 default_0 conda-forge
cmake 3.11.0 0 conda-forge
curl 7.59.0 1 conda-forge
expat 2.2.5 0 conda-forge
icu 58.2 0 conda-forge
intel-openmp 2018.0.0 8
krb5 1.14.6 0 conda-forge
libcxx 6.0.0 0 conda-forge
libgfortran 3.0.1 h93005f0_2
libiconv 1.15 0 conda-forge
libssh2 1.8.0 2 conda-forge
libuv 1.20.0 0 conda-forge
libxml2 2.9.8 0 conda-forge
llvm-meta 6.0.0 0 conda-forge
llvmdev 6.0.0 default_0 conda-forge
mkl 2018.0.2 1
mkl_fft 1.0.1 py36_1 conda-forge
mkl_random 1.0.1 py36_0 conda-forge
ncurses 5.9 10 conda-forge
ninja 1.8.2 1 conda-forge
numpy 1.14.2 py36ha9ae307_1
openssl 1.0.2n 0 conda-forge
pip 9.0.3 py36_0 conda-forge
python 3.6.5 1 conda-forge
readline 7.0 0 conda-forge
rhash 1.3.4 0 conda-forge
setuptools 39.0.1 py36_0 conda-forge
sqlite 3.20.1 2 conda-forge
tk 8.6.7 0 conda-forge
wheel 0.31.0 py36_0 conda-forge
xz 5.2.3 0 conda-forge
zlib 1.2.11 0 conda-forge
```
## Test Results
### complete PR
```
Run on (4 X 2700 MHz CPU s)
2018-04-10 15:33:15
Benchmark Time
CPU Iterations
----------------------------------------------------------------------------------------
BM_WriteRecordBatch/1/min_time:1.000/real_time 76462 ns 76204
ns 16080 12.7719GB/s
BM_WriteRecordBatch/4/min_time:1.000/real_time 63077 ns 62791
ns 21229 15.482GB/s
BM_WriteRecordBatch/16/min_time:1.000/real_time 56850 ns 56584
ns 24134 17.1778GB/s
BM_WriteRecordBatch/64/min_time:1.000/real_time 58209 ns 58003
ns 22891 16.7768GB/s
BM_WriteRecordBatch/256/min_time:1.000/real_time 105328 ns 104930
ns 12450 9.27162GB/s
BM_WriteRecordBatch/1024/min_time:1.000/real_time 247453 ns 245516
ns 5383 3.94645GB/s
BM_WriteRecordBatch/4k/min_time:1.000/real_time 904931 ns 900426
ns 1384 1105.06MB/s
BM_WriteRecordBatch/8k/min_time:1.000/real_time 1869873 ns 1861388
ns 704 534.796MB/s
BM_ReadRecordBatch/1/min_time:1.000/real_time 2409 ns 2402
ns 584482 405.366GB/s
BM_ReadRecordBatch/4/min_time:1.000/real_time 4919 ns 4888
ns 284403 198.544GB/s
BM_ReadRecordBatch/16/min_time:1.000/real_time 14176 ns 14123
ns 73649 68.8866GB/s
BM_ReadRecordBatch/64/min_time:1.000/real_time 51842 ns 51373
ns 26951 18.8372GB/s
BM_ReadRecordBatch/256/min_time:1.000/real_time 201006 ns 200181
ns 6691 4.85837GB/s
BM_ReadRecordBatch/1024/min_time:1.000/real_time 845037 ns 835158
ns 1228 1.15564GB/s
BM_ReadRecordBatch/4k/min_time:1.000/real_time 3340162 ns 3328564
ns 422 299.387MB/s
BM_ReadRecordBatch/8k/min_time:1.000/real_time 7270398 ns 7240046
ns 195 137.544MB/s
```
### PR w/o flatbuffer validation
```
Run on (4 X 2700 MHz CPU s)
2018-04-10 15:35:33
Benchmark Time
CPU Iterations
----------------------------------------------------------------------------------------
BM_WriteRecordBatch/1/min_time:1.000/real_time 76915 ns 76548
ns 18341 12.6967GB/s
BM_WriteRecordBatch/4/min_time:1.000/real_time 65802 ns 65135
ns 21929 14.841GB/s
BM_WriteRecordBatch/16/min_time:1.000/real_time 56544 ns 56359
ns 24133 17.271GB/s
BM_WriteRecordBatch/64/min_time:1.000/real_time 62888 ns 62205
ns 23517 15.5287GB/s
BM_WriteRecordBatch/256/min_time:1.000/real_time 106749 ns 105842
ns 13172 9.14824GB/s
BM_WriteRecordBatch/1024/min_time:1.000/real_time 255753 ns 253039
ns 5469 3.81838GB/s
BM_WriteRecordBatch/4k/min_time:1.000/real_time 990244 ns 969574
ns 1428 1009.85MB/s
BM_WriteRecordBatch/8k/min_time:1.000/real_time 1881824 ns 1870245
ns 754 531.399MB/s
BM_ReadRecordBatch/1/min_time:1.000/real_time 2408 ns 2393
ns 585709 405.624GB/s
BM_ReadRecordBatch/4/min_time:1.000/real_time 4815 ns 4790
ns 293507 202.816GB/s
BM_ReadRecordBatch/16/min_time:1.000/real_time 14351 ns 14294
ns 95709 68.047GB/s
BM_ReadRecordBatch/64/min_time:1.000/real_time 51902 ns 51532
ns 26885 18.8155GB/s
BM_ReadRecordBatch/256/min_time:1.000/real_time 214495 ns 209335
ns 7029 4.55284GB/s
BM_ReadRecordBatch/1024/min_time:1.000/real_time 817379 ns 809841
ns 1781 1.19475GB/s
BM_ReadRecordBatch/4k/min_time:1.000/real_time 3305725 ns 3296526
ns 401 302.506MB/s
BM_ReadRecordBatch/8k/min_time:1.000/real_time 7218564 ns 7187446
ns 193 138.532MB/s
```
### baseline (`6633cc94c84f9ecc516f03175da56361cf4bd67f`)
```
Run on (4 X 2700 MHz CPU s)
2018-04-10 15:36:57
Benchmark Time
CPU Iterations
----------------------------------------------------------------------------------------
BM_WriteRecordBatch/1/min_time:1.000/real_time 77609 ns 77412
ns 16827 12.5831GB/s
BM_WriteRecordBatch/4/min_time:1.000/real_time 62939 ns 62705
ns 21531 15.516GB/s
BM_WriteRecordBatch/16/min_time:1.000/real_time 58021 ns 57736
ns 22232 16.8311GB/s
BM_WriteRecordBatch/64/min_time:1.000/real_time 58180 ns 57996
ns 24262 16.7851GB/s
BM_WriteRecordBatch/256/min_time:1.000/real_time 109448 ns 108065
ns 11190 8.92262GB/s
BM_WriteRecordBatch/1024/min_time:1.000/real_time 256618 ns 254950
ns 4017 3.80551GB/s
BM_WriteRecordBatch/4k/min_time:1.000/real_time 907773 ns 904273
ns 1553 1101.6MB/s
BM_WriteRecordBatch/8k/min_time:1.000/real_time 1864168 ns 1859038
ns 732 536.432MB/s
BM_ReadRecordBatch/1/min_time:1.000/real_time 2414 ns 2405
ns 579529 404.6GB/s
BM_ReadRecordBatch/4/min_time:1.000/real_time 4828 ns 4807
ns 292168 202.264GB/s
BM_ReadRecordBatch/16/min_time:1.000/real_time 14357 ns 14312
ns 96203 68.0212GB/s
BM_ReadRecordBatch/64/min_time:1.000/real_time 51445 ns 51189
ns 26539 18.9828GB/s
BM_ReadRecordBatch/256/min_time:1.000/real_time 196948 ns 196421
ns 6995 4.95849GB/s
BM_ReadRecordBatch/1024/min_time:1.000/real_time 799547 ns 797232
ns 1773 1.22139GB/s
BM_ReadRecordBatch/4k/min_time:1.000/real_time 3374307 ns 3362444
ns 423 296.357MB/s
BM_ReadRecordBatch/8k/min_time:1.000/real_time 7205776 ns 7186692
ns 195 138.778MB/s
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> [C++] Add option to validate Flatbuffers messages
> -------------------------------------------------
>
> Key: ARROW-2222
> URL: https://issues.apache.org/jira/browse/ARROW-2222
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Wes McKinney
> Assignee: Marco Neumann
> Priority: Major
> Labels: pull-request-available
>
> This is follow up work to ARROW-1589, ARROW-2023, and can be validated by the
> {{ipc-fuzzer-test}}. Users receiving untrusted input streams can prevent
> segfaults this way
> As part of this, we should quantify the overhead associated with message
> validation in regular use
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
