westonpace commented on PR #13364:
URL: https://github.com/apache/arrow/pull/13364#issuecomment-1159348973
I spent some time playing around with the
`ExecuteScalarExpressionOverhead/complex_expression` benchmark today to see if
this PR made much change there. I didn't see too much change (which might not
be surprising, I don't really know if any was expected).
We still fail to be able to run as efficiently with batches of 10k rows.
* 10k row batches => 6G rows/s
* 100k row batches => 7G rows/s
I'm not as convinced as before that this has anything to do with thread
contention. It may not have been an issue in the past so I may have been too
eager to jump on that particular bandwagon.
* When going from 1 thread to 16 threads there is a significant amount of
"front-end latency" that gets introduced. vTune can find no attributable cause
to this (e.g. not branch resteers, icache misses, or anything else there is a
category for).
* However, the above happens at both 10k rows per batch and 100k rows per
batch and the extra latency appears to be in the execution of the critical
section itself, and not just the overhead parts.
There are however, just plain more instructions. This is obviously
expected, to some degree, when going from 100k batches to 10k batches. I don't
really have a solid way of knowing how much additional overhead is right.
* With 100k batches we spend 96% of the time in the critical section
* With 10k batches we spend 75% of the time in the critical section
The overhead, in terms of instructions, is very widely dispersed. If I had
to pick any particular theme I would say allocation/deallocation. Here are the
top-15 non-critical section calls along with my crude attempt at categorization.
Call | % instructions | Category guess
--- | --- | ---
__GI___libc_malloc | 1.1% | Allocation
std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release | 1.0% |
Deallocation
arrow::compute::ExecuteScalarExpression | 1.0% | self-time / branchy?
__GI_ | 0.9% | Unknown
arrow::compute::detail::(anonymous
namespace)::KernelExecutorImpl<arrow::compute::ScalarKernel>::PrepareOutput |
0.6% | self-time / allocation?
std::vector<arrow::ValueDescr, std::allocator<arrow::ValueDescr>>::~vector |
0.6% | Deallocation
tcache_get | 0.5% | Allocation
arrow::CPUDevice::Instance | 0.5% | Allocation
arrow::util::detail::VariantImpl<arrow::util::Variant<arrow::Datum::Empty,
std::shared_ptr<arrow::Scalar>, std::shared_ptr<arrow::ArrayData>,
std::shared_ptr<arrow::ChunkedArray>, std::shared_ptr<arrow::RecordBatch>,
std::shared_ptr<arrow::Table>>, std::shared_ptr<arrow::Scalar>,
std::shared_ptr<arrow::ArrayData>, std::shared_ptr<arrow::ChunkedArray>,
std::shared_ptr<arrow::RecordBatch>,
std::shared_ptr<arrow::Table>>::copy_to<arrow::util::Variant<arrow::Datum::Empty,
std::shared_ptr<arrow::Scalar>, std::shared_ptr<arrow::ArrayData>,
std::shared_ptr<arrow::ChunkedArray>, std::shared_ptr<arrow::RecordBatch>,
std::shared_ptr<arrow::Table>>> | 0.5% | Allocation?
arrow::BaseMemoryPoolImpl<arrow::memory_pool::internal::JemallocAllocator>::Allocate
| 0.5% | Allocation
arrow::Datum::descr | 0.4% | shared_ptr copy
arrow::compute::detail::(anonymous
namespace)::ScalarExecutor::~ScalarExecutor | 0.4% | Deallocation? (there are
some vectors in here that will get deallocated)
arrow::compute::detail::(anonymous namespace)::ScalarExecutor::Execute |
0.4% | Unknown
arrow::CPUMemoryManager::Make | 0.4% | Allocation
arrow::compute::detail::(anonymous namespace)::ScalarExecutor::ExecuteSpans
| 0.4% | Unknown
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
