codeant-ai-for-open-source[bot] commented on code in PR #41946:
URL: https://github.com/apache/superset/pull/41946#discussion_r3564126395
##########
superset/connectors/sqla/models.py:
##########
@@ -2032,8 +2039,13 @@ def fetch_metadata(self) -> MetadataResult:
if not any_date_col and new_column.is_temporal:
any_date_col = col["column_name"]
- # add back calculated (virtual) columns
- columns.extend([col for col in old_columns if col.expression])
+ # Add back calculated (virtual) columns, i.e. those that weren't
matched
+ # against `new_columns` above and are thus still present in
+ # `old_columns_by_name`. Columns that were matched are already
appended to
+ # `columns` in the loop above, and re-adding them here (e.g. via
`old_columns`)
+ # would duplicate any synced physical column that also carries a truthy
+ # `expression`, such as Trino's expanded nested `ROW` fields.
+ columns.extend([col for col in old_columns_by_name.values() if
col.expression])
Review Comment:
**Suggestion:** The new add-back filter uses `col.expression` as a proxy for
virtual/calculated columns, but expanded Trino nested physical columns now also
have a non-empty expression. If one of those nested fields disappears from
source metadata (or row expansion is turned off), it remains in
`old_columns_by_name` and gets re-added instead of removed, leaving stale
physical columns in the dataset and causing downstream query failures against
non-existent fields. Restrict this add-back path to true virtual/calculated
columns only (not physical synced columns). [incorrect condition logic]
<details>
<summary><b>Severity Level:</b> Critical 🚨</summary>
```mdx
- ❌ Trino datasets retain dropped nested ROW child columns.
- ❌ Queries reference non-existent nested fields after schema changes.
- ⚠️ Metadata sync falsely reports removal but keeps stale columns.
- ⚠️ Dashboard filters using removed nested fields start failing.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. Create a Trino-backed dataset with `expand_rows` enabled so that nested
ROW fields like
`metadata.uuid` are expanded; `TrinoEngineSpec._expand_columns` at
`superset/db_engine_specs/trino.py:12-58` returns metadata entries where the
inner field
has `"column_name": "metadata.uuid"` and `"expression": '"metadata"."uuid"'`.
2. Sync metadata for this dataset (via the UI’s “Sync columns from source”
or a direct
`SqlaTable.fetch_metadata()` call); `SqlaTable.fetch_metadata` at
`superset/connectors/sqla/models.py:161-241` persists a `TableColumn` for
`metadata.uuid`
whose `.expression` is set to the quoted path.
3. In the Trino database, remove the nested field (e.g. drop `uuid` from the
`metadata`
ROW) so that a subsequent inspection no longer returns `metadata.uuid` in
`external_metadata()` for this table; now `new_columns` from
`fetch_metadata()` no longer
contains `"column_name": "metadata.uuid"`.
4. Trigger another metadata sync (`SqlaTable.fetch_metadata()`); in
`fetch_metadata`,
`old_columns_by_name` still contains the prior `TableColumn` for
`metadata.uuid` with a
non-empty `.expression`, so it is included in the `removed` list at lines
`194-203` but
then re-added by `columns.extend([col for col in
old_columns_by_name.values() if
col.expression])` at line `2048`, leaving `self.columns` containing a stale
physical
column that no longer exists in Trino and causing downstream queries that
use this column
to fail with Trino `COLUMN_NOT_FOUND` errors instead of removing the column
from the
dataset.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=5c40bda552b343a4bb6b3f2a37ccf6db&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
[](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=5c40bda552b343a4bb6b3f2a37ccf6db&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/connectors/sqla/models.py
**Line:** 2048:2048
**Comment:**
*Incorrect Condition Logic: The new add-back filter uses
`col.expression` as a proxy for virtual/calculated columns, but expanded Trino
nested physical columns now also have a non-empty expression. If one of those
nested fields disappears from source metadata (or row expansion is turned off),
it remains in `old_columns_by_name` and gets re-added instead of removed,
leaving stale physical columns in the dataset and causing downstream query
failures against non-existent fields. Restrict this add-back path to true
virtual/calculated columns only (not physical synced columns).
Validate the correctness of the flagged issue. If correct, How can I resolve
this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask
user if the user wants to fix the rest of the comments as well. if said yes,
then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a
href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41946&comment_hash=91747a0d37324583316a2d7c4d71213a3852f5375a1bc969a13c9cdaf313ae47&reaction=like'>👍</a>
| <a
href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41946&comment_hash=91747a0d37324583316a2d7c4d71213a3852f5375a1bc969a13c9cdaf313ae47&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
