[
https://issues.apache.org/jira/browse/ARROW-6417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16923607#comment-16923607
]
Wes McKinney commented on ARROW-6417:
-------------------------------------
The benchmark results in arrow-builder-benchmark are pretty damning...
master
{code}
---------------------------------------------------------------------------
Benchmark Time CPU Iterations
---------------------------------------------------------------------------
BufferBuilderTinyWrites/real_time 264925407 ns 264925189 ns 2
966.31MB/s
BufferBuilderSmallWrites/real_time 178721490 ns 178720664 ns 4
1.39882GB/s
BufferBuilderLargeWrites/real_time 192722520 ns 192720335 ns 4
1.29027GB/s
BuildBooleanArrayNoNulls 61622618 ns 61620052 ns 11
4.05712GB/s
BuildIntArrayNoNulls 159926782 ns 159919611 ns 4
1.56329GB/s
BuildAdaptiveIntNoNulls 34141484 ns 34141072 ns 20
7.32256GB/s
BuildAdaptiveIntNoNullsScalarAppend 118671966 ns 118669726 ns 6
2.10669GB/s
BuildBinaryArray 646172067 ns 646165509 ns 1
396.183MB/s
BuildChunkedBinaryArray 629538527 ns 629517882 ns 1
406.66MB/s
BuildFixedSizeBinaryArray 319843478 ns 319421997 ns 2
801.448MB/s
BuildDecimalArray 613258571 ns 613249404 ns 1
834.897MB/s
BuildInt64DictionaryArrayRandom 265489567 ns 265479003 ns 3
964.295MB/s
BuildInt64DictionaryArraySequential 256461735 ns 256454103 ns 3
998.229MB/s
BuildInt64DictionaryArraySimilar 436497455 ns 436496161 ns 2
586.489MB/s
BuildStringDictionaryArray 737468427 ns 737429710 ns 1
463.142MB/s
ArrayDataConstructDestruct 38895 ns 38895 ns 18067
{code}
master with the older jemalloc
{code}
---------------------------------------------------------------------------
Benchmark Time CPU Iterations
---------------------------------------------------------------------------
BufferBuilderTinyWrites/real_time 139816022 ns 139814056 ns 5
1.78806GB/s
BufferBuilderSmallWrites/real_time 35215592 ns 35214766 ns 19
7.09912GB/s
BufferBuilderLargeWrites/real_time 32460612 ns 32456001 ns 21
7.66046GB/s
BuildBooleanArrayNoNulls 33690068 ns 33688611 ns 21
7.42091GB/s
BuildIntArrayNoNulls 49988970 ns 49987507 ns 14
5.00125GB/s
BuildAdaptiveIntNoNulls 23878665 ns 23876703 ns 29
10.4705GB/s
BuildAdaptiveIntNoNullsScalarAppend 116140426 ns 116137665 ns 6
2.15262GB/s
BuildBinaryArray 593711307 ns 593699295 ns 1
431.195MB/s
BuildChunkedBinaryArray 538185849 ns 538185876 ns 1
475.672MB/s
BuildFixedSizeBinaryArray 218638403 ns 218631191 ns 3
1.14348GB/s
BuildDecimalArray 294477232 ns 294474155 ns 2
1.69794GB/s
BuildInt64DictionaryArrayRandom 248790745 ns 248788395 ns 3
1028.99MB/s
BuildInt64DictionaryArraySequential 238954386 ns 238949356 ns 3
1071.36MB/s
BuildInt64DictionaryArraySimilar 422484600 ns 422471016 ns 2
605.959MB/s
BuildStringDictionaryArray 716507144 ns 716487471 ns 1
476.68MB/s
ArrayDataConstructDestruct 38406 ns 38406 ns 18229
{code}
So it seems that performance in realloc-heavy workloads is degraded
> [C++][Parquet] Non-dictionary BinaryArray reads from Parquet format have
> slowed down since 0.11.x
> -------------------------------------------------------------------------------------------------
>
> Key: ARROW-6417
> URL: https://issues.apache.org/jira/browse/ARROW-6417
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++, Python
> Reporter: Wes McKinney
> Priority: Major
> Labels: pull-request-available
> Attachments: 20190903_parquet_benchmark.py,
> 20190903_parquet_read_perf.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> In doing some benchmarking, I have found that binary reads seem to be slower
> from Arrow 0.11.1 to master branch. It would be a good idea to do some basic
> profiling to see where we might improve our memory allocation strategy (or
> whatever the bottleneck turns out to be)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
