AQE (Adaptive Query Execution) is gradually becoming a standard capability of mainstream distributed execution engines and MPP databases, much like vectorized execution did before it. Therefore, if we want Apache Cloudberry to remain competitive and technologically ahead, I believe we should seriously consider adding adaptive query execution capabilities.
Once we introduce this capability, the possibilities extend far beyond the commonly discussed use cases such as dynamic Bloom filters. There are many things we could potentially do: (1) Real-time query execution visibility. We could observe the real-time execution progress of an individual query, rather than having to wait until the query completes before obtaining meaningful execution feedback. (2) Breaking the current fixed interaction model between the QD and QEs. Today, the interaction model is largely static: the QD sends information to the QEs during the dispatch phase, while the QEs primarily send information back to the QD after a slice task has completed. With Anser, the QD and QEs could exchange information dynamically throughout the lifetime of a query. This would fundamentally expand what is possible at the execution layer—for example, allowing us to move beyond traditional tree-shaped execution plans toward more flexible DAG-based even loop-based execution models. (3) Dynamically selecting and adjusting execution strategies. For example: - dynamically applying global encoding to certain columns and then performing aggregation, joins, sorting, and other operations on the encoded values; - dynamically deciding whether data transferred through Motion operators should be compressed; - detecting data skew at runtime and dynamically adjusting the shuffling strategy and its granularity; - automatically cancelling queries that become hung or stalled for unknown reasons; - and potentially many other forms of runtime optimization that are difficult or impossible under the current static execution model. The real value is not AQE itself. Anser provides the execution engine with a runtime feedback loop, allowing the system to observe, reason about, and adapt its behavior while a query is still running. Once that capability exists, many execution strategies that are impossible under a static execution model become possible. Lirong Leonid Borchuk <[email protected]> 于2026年8月21日周五 15:30写道： > Hi all, > > We discussed in one topic (I do not remember where, but it doesn't matter > here) about the future of query evaluation in Cloudberry. And one of the > ideas (from Lirong I think) was about implementing Alibaba approach with > working with runtime query evaluation > https://vldb.org/pvldb/vol16/p3636-wu.pdf > > This idea captured me since then. > > What do you think if I try to implement Anser for Cloudberry? > > There are several solutions in an article - gathering stat, creating > bloomfilters, step-by-step sessions creation for slice processing, switch > from broadcast to redistribute and so on. > > In an overall all ideas are spilling around improving runtime behaviour - > gather statistics and use it to improve query performance or save computing > resources (mainly memory). > > I'd want to start with the quite simple functionality - working with bloom > filters to reduce the number of processed rows. Create them while reading > one side of hash join, propagate to the second side and filter out rows > before we spent CPU for creating hash table and perform hash probes. We > could check how Anser could be implemented in Cloudberry and what is the > overall benefit, what are the overhead costs and check if the whole > approach is applicable. > > WBW, Leonid >
