Wellbek commented on issue #21061: URL: https://github.com/apache/beam/issues/21061#issuecomment-4460771819
Hey! As my intro to contributing to Apache Beam, I took a look at this older issue to see whether it still relevant. From the traceback, my understanding is that the failure happened when `WriteToBigQuery` was used with: * `method="FILE_LOADS"` * a dynamic `table` callable * and the `BundleBasedDirectRunner` which then failed with: ```python AttributeError: 'str' object has no attribute 'projectId' ``` From the issue description, problem stems from `wait_for_bq_job` expecting a proper `JobReference` object, while a plain string was being passed somewhere in the `WaitForBQJobs` path. To reproduce this, I first created a small GCP setup: * created a new GCP project * created a GCS bucket * created a BigQuery dataset I then wrote a minimal reproducer (analogous to your step outlined) using: * `BundleBasedDirectRunner` * dynamic destinations * `method="FILE_LOADS"` The reproducer writes rows into multiple dynamic BigQuery tables to ensure the copy/load path is triggered. I tested this in a local setup with `python:3.14.3` using a recent Beam SDK cloned at May 15 2026 (Beam 2.73.0 release). The pipeline completed successfully and the rows were written correctly to BQ. I could not reproduce the original `AttributeError`. Just to make sure, I tried some more test cases, specifically * larger numbers of dynamic tables (50) * more rows * smaller `batch_size` * larger payloads (1KB text) * mixed data types (`None`, floats, booleans, unicode text, long strings) All of these also completed successfully without any error. During testing, I also added temporary type checks before calls to `wait_for_bq_job` through the cloned python sdk to verify what was being passed in. In all cases, the value was a proper `JobReference` object and never a plain string. One thing that also stood out while looking through the code history is that the mentioned `WaitForBQJobs` seems to have been removed in Beam 2.42.0 -> 2.43.0 (see https://github.com/apache/beam/compare/v2.42.0...v2.43.0). **I therefore suspect the issue may already have been fixed as part of later refactors.** At this point I'm unable to reproduce the bug, so I wanted to check whether this issue can likely be closed, or whether there is still a known failing scenario I may have missed. Since this is also my first time digging into the Beam codebase, I'd really appreciate any guidance on, 1) whether there are more scenarios I should verify against (e.g., using older beam versions), 2) whether there's a more specific reproducer I should try Happy to run more tests or investigate further if helpful. Cheers -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
