uchenily commented on issue #45847:
URL: https://github.com/apache/arrow/issues/45847#issuecomment-2742932993
If a two-phase output is adopted, significant performance improvements can
be achieved in scenarios where the build-side involves large-scale data.
However, I currently cannot provide a reasonable explanation for this,
@zanmato1984 do you have any ideas?
```diff
if (num_output_rows > 0) {
// Materialize (and output whenever buffers get full) hash table
// values according to the generated list of ids.
//
+
RETURN_NOT_OK(local_states_[thread_id].materialize.Flush([&](ExecBatch batch) {
+ return output_batch_callback_(thread_id, std::move(batch));
+ }));
Status status =
local_states_[thread_id].materialize.AppendBuildOnly(
num_output_rows, key_ids_buf.mutable_data(),
payload_ids_buf.mutable_data(),
[&](ExecBatch batch) {
return output_batch_callback_(static_cast<int64_t>(thread_id),
std::move(batch));
});
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
