will-sargent-dbtlabs opened a new issue, #4660:
URL: https://github.com/apache/arrow-adbc/issues/4660
## Summary
When BigQuery returns **HTTP 409 Already Exists** during job submission, the
Go BigQuery ADBC driver surfaces the error to the caller. The correct behaviour
is to attach to the already-created job (using the driver's own job ID) and
read its results — the same job ran successfully, we just need to reconnect to
it.
This matters whenever job submission is retried after a transient network
timeout: the server created the job on the first attempt, but the client never
received the `200 OK`. A subsequent `query.Run()` with the same job ID hits the
409 path and the driver has no recovery.
## Affected code
`go/adbc/driver/bigquery/record_reader.go` — `runQuery`
```go
job, err := query.Run(ctx)
if err != nil {
return nil, -1, err // ← 409 falls through here unchanged
}
```
## Expected behaviour
On a `*googleapi.Error` with `.Code == 409`, call
`client.JobFromIDLocation(ctx, query.JobID, query.Location)` to attach to the
existing job and continue normally.
## Second issue: job ID is not refreshed per execution
`runQuery` previously only assigned a driver-owned UUID when `query.JobID ==
""`:
```go
if query.JobID == "" {
query.JobID = "adbc-" + uuid.NewString()
}
```
A `*bigquery.Query` is reused for each row when bound parameters are in play
(`queryRecordWithSchemaCallback`). Because `query.JobID` is non-empty after the
first execution, subsequent rows reuse the same ID — which either collides with
the live job or makes the 409 recovery attach to the wrong execution.
The fix is to assign a fresh UUID unconditionally at the start of every
`runQuery` call.
## Proposed fix
```go
func runQuery(ctx context.Context, client *bigquery.Client, query
*bigquery.Query, ...) (...) {
// Fresh ID per execution — safe for bound-parameter reuse.
query.JobID = "adbc-" + uuid.NewString()
job, err := query.Run(ctx)
var apiErr *googleapi.Error
if errors.As(err, &apiErr) && apiErr.Code == http.StatusConflict {
job, err = client.JobFromIDLocation(ctx, query.JobID, query.Location)
}
if err != nil {
return nil, -1, err
}
// ... rest unchanged
}
```
Threading `*bigquery.Client` through `runQuery` / `runPlainQuery` /
`queryRecordWithSchemaCallback` / `newRecordReader` is required because the
recovery call needs the client directly.
A regression test (`TestRunQueryRecoversExistingJobAfterDuplicateInsert`)
uses an `httptest.Server` that always returns 409 on submission, then serves
the job on the subsequent GET, and verifies two consecutive executions each
produce distinct job IDs and both return results.
## Fix in dbt-labs fork
A patch implementing this fix (including the regression test) is available
at:
**dbt-labs/arrow-adbc#153**
Happy to port it upstream as a PR against this repo if that would be helpful.
## Environment
- Go BigQuery ADBC driver (`go/adbc/driver/bigquery`)
- Reproducible via the httptest-based unit test in the patch above (no real
GCP credentials needed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
