Just realized dev@ is missing in the receivers.. Adding dev@ back to the loop.
Feel free to pick IMPALA-15253. I plan to work on that after HBO supports all the basic node types, so might still need some time. Regards, Quanlong On Tue, Aug 11, 2026 at 5:18 AM Aleksandr Efimov <[email protected]> wrote: > Thanks for the details! I’ve started working on IMPALA-15236. Do you plan > to work on IMPALA-15253, or should I pick it up after 15236? > > пн, 10 авг. 2026 г. в 18:57, Quanlong Huang <[email protected]>: > >> Hi Aleksandr, >> >> Yeah, we can use the snapshot id to replace the catalog version for >> Iceberg tables. That improves the scenario when numRows are missing - >> catalog version changes might be due to a REFRESH operation and the Iceberg >> snapshot id remains unchanged (historical run can still match). However, >> both catalog versions and snapshot ids are too sensitive, i.e. they could >> change while the data remains the same. numRows is better to be the major >> matching key. When it's missing and the catalog version / snapshot id >> doesn't match, use the total file size as the final matching key. >> >> Note that there is a corner case that a non-iceberg table can be migrated >> to an iceberg table. In such a case, the table name remains unchanged but >> the historical runs don't have Iceberg snapshot ids. The data also remains >> unchanged so ideally historical stats should be reused. We need to take >> care of such cases in using Iceberg snapshot ids. >> >> On the other hand, Iceberg tables always have numRows in either the table >> level or file level. We should use this instead of numRows from HMS. Then >> numRows won't be missing and probably we don't need the snapshot ids. Filed >> IMPALA-15253 for this. >> >> BTW, I've uploaded the initial design of HBO in IMPALA-14596: >> https://docs.google.com/document/d/1kDjJTGxiKafh6HgxVAY6m06KCtaVW1O7pJ1aQc0xOGQ/edit?usp=sharing >> Some content might be stale and the Gerrit patch sets have the latest >> design. >> >> Regards, >> Quanlong >> >> On Sun, Aug 2, 2026 at 7:24 PM Aleksandr Efimov <[email protected]> >> wrote: >> >>> Hi Quanlong, >>> >>> Thanks, that clears things up. I also noticed IMPALA-15234, which seems >>> to cover the bad-cache case. >>> >>> One small question: I couldn’t find a snapshot ID in TScanInputStats, >>> and Iceberg seems to use the regular scan matching path. Is >>> snapshot-specific matching planned, or is reuse currently based only on >>> numRows, catalog version and file size? >>> >>> I’ll take a closer look at 24556 and file a separate JIRA for showing >>> the HBO matching strategy in the profile. >>> >>> Best, >>> Aleksandr >>> >>> пт, 31 июл. 2026 г. в 10:53, Quanlong Huang <[email protected]>: >>> >>>> Hi Aleksandr, >>>> >>>> Thanks for reaching out and for the insightful feedback! Here are the >>>> details regarding your questions: >>>> >>>> > What I could not find was a general rule for when an HBO entry >>>> should be considered stale. As far as I can tell, when input row counts are >>>> available, a match does not require the catalog version to match, and the >>>> Iceberg snapshot ID is not part of the scan key. Is that intentional? >>>> >>>> Our goal with HBO statistics is to enable sharing across similar >>>> queries, not just identical ones. We achieve this by canonicalization >>>> strategies which rely on specific assumptions. For example, the >>>> IGNORE_PARTITION_CONSTANTS strategy assumes that different partitions of >>>> the same table share similar data distributions and predicate >>>> selectivities. This strategy transforms equality predicates like >>>> dt=20260730 and dt=20260731 into a generic dt=<CONST> while preserving >>>> other row-level predicates. As long as the input row counts between >>>> two partition scans match, we assume the output row counts (under the same >>>> row-level predicates) will match as well. >>>> >>>> Here is how we manage the HBO entries: >>>> - If the input row counts differ significantly, both runs are >>>> preserved in the HBO value list. >>>> - If input row counts match within the threshold, the newer run >>>> replaces the older one. >>>> - Eviction: Once an HBO value list hits capacity, the oldest entry is >>>> evicted. Additionally, the underlying HBO KV cache manages its own TTL and >>>> eviction policies. >>>> >>>> Catalog versions and Iceberg snapshot IDs change much more frequently >>>> than underlying data distributions. Treating them as hard invalidation >>>> triggers would be too strict and reduce HBO stats reuse. Instead, we use >>>> them as fallbacks when HMS input row counts (numRows) are unavailable. For >>>> a deeper look at this logic, feel free to check out this patch: >>>> https://gerrit.cloudera.org/c/24556/8/fe/src/main/java/org/apache/impala/service/HistoricalStats.java >>>> . Your review comments are very welcome! >>>> >>>> > More generally, should dropping and recreating a table, changing an >>>> Iceberg snapshot, upgrading Impala, or switching between the Classic and >>>> Calcite planners prevent history from being reused even when the key and >>>> input row count still match? IMPALA-15059 makes the planner case relevant, >>>> but does not seem to define whether both planners should share the same >>>> history. >>>> >>>> While canonicalization assumptions might occasionally fail (e.g., a >>>> table is overwritten with a radically different data distribution), we >>>> expect these edge cases to be rare. In such scenarios, users can either >>>> explicitly disable HBO for specific queries or allow the initial >>>> post-overwrite query to accept a sub-optimal plan; subsequent runs will >>>> quickly collect and benefit from the new HBO statistics. >>>> >>>> For Impala upgrades or switching between the original and Calcite >>>> planners, we intentionally want cardinality history to remain reusable. >>>> Cardinality reflects the underlying data distribution, which is independent >>>> of the query engine / planner. >>>> >>>> > THboStatsType already separates different kinds of statistics. When >>>> memory, CPU, or runtime statistics are added, is the expectation that each >>>> type may also define its own matching conditions? Those values seem more >>>> sensitive than cardinality to query options and cluster conditions. >>>> >>>> Yes, CPU, memory, and runtime statistics are significantly more >>>> sensitive to cluster conditions and query options than simple cardinality. >>>> For these non-cardinality stats types, we plan to narrow the matching scope >>>> by incorporating query options and cluster metadata into the lookup key. >>>> Since this falls outside the scope of our initial milestone, we will >>>> revisit the exact matching criteria during later phases. >>>> >>>> > Separately, would it make sense to expose the selected >>>> canonicalization strategy in the query profile? Today it is available in >>>> debug logs, while the plan only says “(from HBO)”. Having it in the profile >>>> would help investigate surprising estimates and would also be useful for >>>> plan-graph.py and the analyzer work under IMPALA-14953. I do not mean >>>> stabilizing the V2 profile format now, only eventually making the match >>>> provenance visible. >>>> >>>> That's a great suggestion! I think we can add the strategy to the >>>> annotation, e.g. "(from HBO EXPR_REWRITE)". We can also add the HBO hash >>>> key that hits the stats, similar to what the TupleCacheNode shows. Showing >>>> the estimated cardinality if it differs a lot from the HBO one will also be >>>> helpful. Feel free to file JIRAs for your thoughts. >>>> >>>> > If I missed an existing design note or JIRA, please point me to it. >>>> Otherwise, I can file focused follow-up issues once the intended direction >>>> is clear. >>>> >>>> We have an internal design doc with more details but I need some time >>>> to publish it (e.g. remove downstream contents). I'll share it in JIRA. >>>> Some >>>> content of it might be stale, but the Gerrit patches have the most >>>> up-to-date design. >>>> >>>> Best regards, >>>> Quanlong >>>> >>>> On Wed, Jul 29, 2026 at 5:37 PM Aleksandr Efimov <[email protected]> >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> Quanlong, following up on the HBO/profile work around IMPALA-14596. >>>>> After working on IMPALA-15096 and IMPALA-15097, I spent some time looking >>>>> through the merged HBO code and the follow-up JIRAs. >>>>> >>>>> IMPALA-14597 uses input-row similarity for matching HdfsScanNode runs. >>>>> I also saw the source-specific follow-ups: IMPALA-14846 discusses >>>>> additional context for mutable Kudu/HBase tables, and IMPALA-15063 >>>>> proposes >>>>> including runtime filters in the key. >>>>> >>>>> What I could not find was a general rule for when an HBO entry should >>>>> be considered stale. As far as I can tell, when input row counts are >>>>> available, a match does not require the catalog version to match, and the >>>>> Iceberg snapshot ID is not part of the scan key. >>>>> >>>>> Is that intentional? More generally, should dropping and recreating a >>>>> table, changing an Iceberg snapshot, upgrading Impala, or switching >>>>> between >>>>> the Classic and Calcite planners prevent history from being reused even >>>>> when the key and input row count still match? IMPALA-15059 makes the >>>>> planner case relevant, but does not seem to define whether both planners >>>>> should share the same history. >>>>> >>>>> THboStatsType already separates different kinds of statistics. When >>>>> memory, CPU, or runtime statistics are added, is the expectation that each >>>>> type may also define its own matching conditions? Those values seem more >>>>> sensitive than cardinality to query options and cluster conditions. >>>>> >>>>> Separately, would it make sense to expose the selected >>>>> canonicalization strategy in the query profile? Today it is available in >>>>> debug logs, while the plan only says “(from HBO)”. Having it in the >>>>> profile >>>>> would help investigate surprising estimates and would also be useful for >>>>> plan-graph.py and the analyzer work under IMPALA-14953. I do not mean >>>>> stabilizing the V2 profile format now, only eventually making the match >>>>> provenance visible. >>>>> >>>>> If I missed an existing design note or JIRA, please point me to it. >>>>> Otherwise, I can file focused follow-up issues once the intended direction >>>>> is clear. >>>>> >>>>> Best, >>>>> Aleksandr >>>>> >>>>
