Hi! We use BigQuery a fair amount, both through Beam jobs and elsewhere, both with Python. We often do some templatizations and simple generation of the queries as well.
When not using Beam we use the google.cloud.bigquery library, and when a query fails because there's something wrong with the query itself, I often go to the BigQuery web interface and find my failed query in the recent job history, and that makes it really easy to iterate and debug and fix my query, then once I have my results I can copy my fixes back over to my code. That doesn't work when I call BigQuery from Beam though (we're currently on 2.59.0). I get an error on the console, but it seems like the query never gets executed on BigQuery/it's not available in the job history. The most useful info I get is a line number, but depending on how our queries were constructed, the line number doesn't necessarily help me very much. Is there a way to coerce Beam into executing the query even when it's invalid so that I can do my debugging in the BigQuery web interface? Thanks, -Lina
