reuvenlax commented on PR #38313: URL: https://github.com/apache/beam/pull/38313#issuecomment-5685707988
@damccorm 1. Good question, I'm still chatting with BQ about this. I will say that the current test waits until the table.get() returns the new schema, however that doesn't guarantee that the write-api servers have seen the update. I do see that we generally see updates from write-api about _some_ of the tables, but generally not all. Connection pooling means that updates for many tables are multiplexed on a single RPC, and the schema update comes from the update response. It's possible that this changes timing enough to break the test. It's also possible that there's a bug in BigQuery around schema updates and connection pooling. 2. I'm not sure how to make this test fundamentally not flaky. We can query write api for the schema and wait until we see that updated. However with the default stream, the query might go to any of.a pool of write-api frontend servers, which means that the test might still be flaky. I do have a pending PR that adds a "high consistency" mode to schema updates - i.e. it guarantees that we don't ever miss updates. However this mode is more expensive, and unlikely to be the default mode for many users. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
