tgaddy commented on PR #25352: URL: https://github.com/apache/beam/pull/25352#issuecomment-1448767963
@lostluck I tried adding some integration tests now if you want to take a look. Ran the tests in my own gcp project with the following command (had to increase the timeout to 15 minutes in order to run all the tests) ``` go test -v ./test/integration/io/bigqueryio/... --runner=dataflow --project=<PROJECT> --region=europe-west1 --staging_location=<STAGING_LOCATION> --worker_harness_container_image=apache/beam_go_sdk:latest --bq_dataset=<BQ_DATASET> --timeout=15m ``` I borrowed heavily from the integration tests in `xlang/bigquery` so that one can potentially add more integration tests in the future for `Read` and `Query`. There were a few things I wasn't so sure about (in addition to just normal idiomatic golang stuff I've probably missed): - Whether to add anything in the `integration.go` filters. There's already a `TestBigQueryIO` (from xlang), so perhaps the existing filters take care of it? Or perhaps I should change the name of this test, although `TestBigQueryIO` seems like the most natural name.. - I assumed that dataflow is running in the same project as the one in which the bigquery dataset is located. As far as I could tell, the tests using the dataflow runner run in `apache-beam-testing`, which is also where the testing dataset is. - I wanted to check that the write actually worked and wrote the expected number of rows. Since it's a streaming write I had to look at `streamingBuffer.EstimatedRows`, but given that its the "estimated" rows perhaps it doesn't make sense to expect that value to be exactly the `inputSize`? - Whether I should add `bq_dataset` option to `dataflowValidatesRunner` task in `beam/sdks/go/test/build.gradle` Thanks for taking a look! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
