wenlongbrother commented on code in PR #1311:
URL:
https://github.com/apache/incubator-uniffle/pull/1311#discussion_r1397162479
##########
rust/experimental/server/src/store/memory.rs:
##########
@@ -413,6 +413,18 @@ impl Store for MemoryStore {
last_block_id = -1;
}
}
+
+ // get block_ids filter
+ // In AQE, after executing the sub-QueryStages,
collect the shuffle data size
+ // So if we can filter block, it will improve the
performance of AQE.
+ candidate_blocks = candidate_blocks
Review Comment:
Your consideration is correct.
Integrating filter logic into
read_partial_data_with_max_size_limit_and_task_filter functions is indeed a
better design choice, so that the filter can be applied directly when the data
is read, rather than filtering and then reading.
The advantage of this is that data can be processed more efficiently,
especially when the amount of data is large or the filtering conditions are
complex.
Thank you for your meticulous guidance and patience.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
