bluemalkin opened a new issue, #41906:
URL: https://github.com/apache/superset/issues/41906
### Bug description
## Bug description
When a BigQuery connection uses **Application Default Credentials** (in our
case: GKE Workload Identity — the pod's Kubernetes Service Account is bound to
a Google Service Account that lives in the GKE project but has been granted
read access to BigQuery datasets in a *different* project), adding a physical
dataset fails as soon as a table is selected:
```json
{
"errors": [{
"message": "GET
https://bigquery.googleapis.com/bigquery/v2/projects/<GSA-PROJECT>/datasets/support/tables/api_events?prettyPrint=false:
Not found: Dataset <GSA-PROJECT>:support",
"error_type": "GENERIC_BACKEND_ERROR",
"level": "error"
}]
}
```
The connection URI is `bigquery://<DATA-PROJECT>`, but the request targets
`<GSA-PROJECT>` — the project the ADC credentials belong to (where GKE runs),
not the project where the datasets live.
Notably, the **schema and table dropdowns populate correctly** (those go
through the SQLAlchemy dialect, which honors the URI host); only selecting a
table fails ("Unable to load columns for the selected table"). Enabling
`allow_multi_catalog` and selecting the correct catalog in the dataset panel
does not help.
This worked in 4.x and 5.x with the same connection URI, the same Workload
Identity setup, and the same `sqlalchemy-bigquery` version (1.17.0).
## Root cause
`BigQueryEngineSpec._get_client` builds the raw Google client **without a
`project=` argument** ([bigquery.py L527-L553 @
6.1.0](https://github.com/apache/superset/blob/6.1.0/superset/db_engine_specs/bigquery.py#L527-L553)):
```python
credentials = google.auth.default()[0]
return bigquery.Client(credentials=credentials) # project defaults to the
ADC project
```
so `client.project` is the credentials' project, ignoring the project in the
connection URI (`engine.url.host`).
This bug is present but *latent* in ≤5.0 — nothing in the dataset-creation
flow called `_get_client`. Superset 6.0's partition-metadata feature added
`get_time_partition_column` ([L409-L423 @
6.1.0](https://github.com/apache/superset/blob/6.1.0/superset/db_engine_specs/bigquery.py#L409-L423)),
which is invoked from `get_extra_table_metadata` on every table selection and
passes an **unqualified two-part table ID** to that client:
```python
bq_table = client.get_table(f"{table.schema}.{table.table}")
```
A two-part ID is resolved against `client.project` → wrong project → 404.
Other `_get_client` consumers (`get_catalog_names`, `estimate_query_cost`) are
affected the same way.
## How to reproduce
1. Run Superset ≥6.0 on GKE with Workload Identity; the bound GSA lives in
project **A** and has `bigquery.dataViewer`/`bigquery.jobUser` on datasets in
project **B**.
2. Create a BigQuery connection with URI `bigquery://<project-B>` and no
`credentials_info` (ADC only).
3. **Datasets → New dataset** → pick the connection → schema and table
dropdowns list project B's contents correctly.
4. Select a table → error: `Not found: Dataset <project-A>:<schema>`.
## Expected results
The table metadata call should target the project from the connection URI
(or the selected catalog), consistent with how the SQLAlchemy dialect resolves
it.
## Suggested fix
Pass the engine's project to the client, and/or fully qualify the table
reference:
```python
# _get_client
return bigquery.Client(credentials=credentials, project=engine.url.host or
None)
```
## Workaround we deployed (confirmed working)
Monkeypatch via `superset_config.py` (`FLASK_APP_MUTATOR` to avoid circular
imports at config-load time):
```python
def FLASK_APP_MUTATOR(app):
import google.auth
from google.cloud import bigquery
from google.oauth2 import service_account
from superset.db_engine_specs.bigquery import BigQueryEngineSpec
@classmethod
def _get_client(cls, engine, database):
project = engine.url.host or None
if credentials_info := engine.dialect.credentials_info:
credentials =
service_account.Credentials.from_service_account_info(credentials_info)
return bigquery.Client(credentials=credentials, project=project)
try:
credentials = google.auth.default()[0]
except google.auth.exceptions.DefaultCredentialsError as ex:
from superset.db_engine_specs.exceptions import
SupersetDBAPIConnectionError
raise SupersetDBAPIConnectionError("The database credentials
could not be found.") from ex
return bigquery.Client(credentials=credentials, project=project)
BigQueryEngineSpec._get_client = _get_client
```
After this patch, cross-project dataset creation works again.
## Environment
- Superset: 6.1.0 (official Helm chart 0.17.2); the affected code is
unchanged in `master`
- `sqlalchemy-bigquery`: 1.17.0 (same version worked on Superset 5.0 — not a
driver issue)
- Deployment: GKE, Workload Identity (KSA → GSA), GSA project ≠ dataset
project
- Auth to BigQuery: Application Default Credentials (no `credentials_info`
on the connection)
### Screenshots/recordings
_No response_
### Superset version
6.1.0
### Python version
3.10
### Node version
I don't know
### Browser
Chrome
### Additional context
_No response_
### Checklist
- [x] I have searched Superset docs and Slack and didn't find a solution to
my problem.
- [x] I have searched the GitHub issue tracker and didn't find a similar bug
report.
- [x] I have checked Superset's logs for errors and if I found a relevant
Python stacktrace, I included it here as text in the "additional context"
section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
