bringyou opened a new issue, #70737:
URL: https://github.com/apache/doris/issues/70737

   ### Note before reading
   
   Doris already exposes a per-table switch for this — `auto_analyze_policy` 
(`enable` / `disable` / `base_on_catalog`), added in **#39532** and available 
since 2.1.9 / 3.0.3. This request is **not** asking for a new server-side 
capability; it is asking the adapter to use the existing one on relations it is 
about to drop.
   
   Related: #65993 (dbt-doris tracking issue), #68896 (failed auto-analyze jobs 
restart every cycle).
   
   ### Environment
   
   - dbt-core: 1.12.0
   - dbt-doris: 1.0.0 (`extension/dbt-doris`)
   - Apache Doris: 3.1.4
   - Materializations in use: `table`, `incremental` (`append` and 
`insert_overwrite`)
   
   ### Summary
   
   Every `table` / `incremental` run creates a short-lived physical table 
(`<model>__dbt_tmp`, `<model>__dbt_backup`) via CTAS and drops it seconds 
later. Doris's auto-analyze daemon still holds that relation in its pending job 
queue, so when it picks the job up the table is already gone and the FE logs a 
full stack trace per relation, per run:
   
   ```text
   WARN [StatisticsAutoCollector.collect():103] Failed to analyze table 
ads_sss_series_10m__dbt_tmp
        with columns [ads_sss_series_10m__dbt_tmp:value, 
ads_sss_series_10m__dbt_tmp:metric, ...]
   java.lang.RuntimeException: Table: 
`internal.ds_zimmula_prod.ads_sss_series_10m__dbt_tmp` not exists
       at 
org.apache.doris.statistics.util.StatisticsUtil.findTable(StatisticsUtil.java:421)
       at 
org.apache.doris.statistics.StatisticsAutoCollector.collect(StatisticsAutoCollector.java:103)
       at 
org.apache.doris.statistics.StatisticsAutoCollector.runAfterCatalogReady(StatisticsAutoCollector.java:82)
   ```
   
   The exception is caught per job inside `collect()` and the loop continues, 
so nothing breaks — it is pure log noise. But a 40-model project running hourly 
produces on the order of 1,000 stack traces per day, which drowns `fe.log`.
   
   ### Steps to reproduce
   
   1. `dbt run --select some_incremental_model` — any run that takes the 
`create_table_as(True, tmp_relation, sql)` path (`incremental.sql`, the 
`append` and `insert_overwrite` branches), or the `table` materialization's 
intermediate relation (`table.sql`).
   2. Wait one auto-analyze cycle (`auto_check_statistics_in_minutes`, default 
5 minutes).
   3. `grep "Failed to analyze table .*__dbt_tmp" fe.log`.
   
   ### Root cause
   
   The adapter creates the temp/intermediate relation as a **real, registered 
table** in the target database:
   
   ```jinja
   {% macro doris__create_table_as(temporary, relation, sql) -%}
       ...
       create table {{ table }}
       ...
       {{ doris__properties() }} as {{ doris__table_colume_type(sql) }};
   {%- endmacro %}
   ```
   
   Doris registers the new table for automatic statistics collection; dbt drops 
it before the collector reaches it, and `StatisticsUtil.findTable()` throws. 
The relation lives for a few seconds and is never queried, so collecting 
statistics on it is meaningless in the first place.
   
   ### Proposed change
   
   Apply the existing `auto_analyze_policy = disable` property to short-lived 
relations:
   
   ```sql
   create table db.tbl ( ... )
   properties ("auto_analyze_policy" = "disable")
   as select ...
   ```
   
   Scope — relations created and dropped within the same run:
   
   | call site | relation suffix | dropped after |
   |---|---|---|
   | `incremental.sql`, `create_table_as(True, tmp_relation, sql)` | 
`__dbt_tmp` | insert finishes |
   | `incremental.sql`, full-refresh / backup path | `__dbt_backup` | exchange 
finishes |
   | `table.sql`, intermediate relation | `__dbt_tmp` | exchange finishes |
   
   Suggested implementation: a boolean project var, default `true`, read inside 
`doris__create_table_as` and applied only to the disposable relations. 
`doris__properties()` currently takes no arguments, so it would need an 
optional `extra` dict:
   
   ```jinja
   {% macro doris__properties(extra=none) -%}
     {% set properties = config.get('properties', 
validator=validation.any[dict]) %}
     ...
     {% if extra %}{% do properties.update(extra) %}{% endif %}
   {%- endmacro %}
   ```
   
   ### Caveat the maintainers should decide on
   
   In the `table` materialization the intermediate relation **becomes** the 
final table via `exchange_relation`, so any property set on it is inherited by 
the model's real table — which would silently disable statistics collection for 
every model in the project. Two ways out:
   
   1. Set the property on the disposable relation and explicitly restore it 
after the exchange (`alter table <target> set ("auto_analyze_policy" = 
"enable")`).
   2. Only apply it in the `incremental` path (`__dbt_tmp` / `__dbt_backup`) 
and leave the `table` materialization alone.
   
   Option 2 is smaller and fixes the bulk of the noise; option 1 is complete 
but touches the exchange path. I'm fine with either.
   
   ### Alternative worth checking
   
   If `enable_auto_analyze` is settable at **session** scope (not just `SET 
GLOBAL`), the adapter could set it on the connection that runs the CTAS and 
avoid DDL-level properties entirely. I have not verified that the enqueue path 
honours the session value on 3.1 — please confirm against the FE source before 
relying on it.
   
   ### Acceptance criteria
   
   - `show create table <model>__dbt_tmp` (while it still exists) contains 
`"auto_analyze_policy" = "disable"`.
   - After a full `dbt run`, `fe.log` contains no `Failed to analyze table 
.*__dbt_tmp`.
   - With the var set to `false`, behaviour is unchanged.
   - Model-level `properties` config still takes precedence on the final table.
   
   ### Workaround in the meantime
   
   Server-side, either disable auto-analyze globally (`set global 
enable_auto_analyze = false`, requires ADMIN) or drop the log line at 
collection time (`Failed to analyze table .*__dbt_tmp`).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to