psud commented on PR #38056: URL: https://github.com/apache/beam/pull/38056#issuecomment-4185000956
Good call out with the integration test @stankiewicz ! Looking at the existing `BigtableReadIT` pattern, there are two options: **Pattern 1:** Pre-create the data in `apache-beam-testing` GCP project (according to the [gradle file](https://github.com/apache/beam/blob/25518f07fe285ebdc530092439b6d7f2558af4dd/sdks/java/io/google-cloud-platform/build.gradle#L178)) which is similar to how [testE2EBigtableRead](https://github.com/apache/beam/blob/4221266ffabcd1070224019711bf832950ddb2c1/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableReadIT.java#L110) relies on a pre-provisioned BigtableReadTest table). **Pattern 2:** Create the table and materialized view as part of the test which is similar to how [testE2EBigtableSegmentRead](https://github.com/apache/beam/blob/4221266ffabcd1070224019711bf832950ddb2c1/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableReadIT.java#L127) works Creating an MV as part of test setup isn't ideal as materialized views can take a few minutes to initialize and populate, so the test would need to poll and wait, making it significantly slower than the existing table-based tests. @mutianf I saw that you last edited a lot of the e2e tests and may have access to the `apache-beam-testing` GCP project. What option do you prefer and if we go for option 1 are you able to assist with seeding the data? I am happy to provide a script that configures a table with a materialized view and then seeds the data. Alternatively if @stankiewicz knows anyone else who could know more about this I'd love some input -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
