mxtymoshyk opened a new pull request, #40843: URL: https://github.com/apache/beam/pull/40843
Fixes #26386. `TableDestination` gets `withLabels(Map)` and `getLabels()`. BigQueryIO applies those labels to the destination table: - STREAMING_INSERTS and STORAGE_WRITE_API set them when they create the table. - FILE_LOADS patches them onto the table after the load job, or after the copy job when the write goes through temp tables. Copy jobs have no label setting, so a patch is the only option. The patch merges with labels already on the table. The labels have to survive encoding of the destination. When the destination type is `TableDestination`, the V1 to V3 coders drop them, so this adds `TableDestinationCoderV4`. It is opt-in so existing streaming pipelines can still be updated in place: - With `to(SerializableFunction)`, call `withTableLabels()`. - With `to(DynamicDestinations)`, return `TableDestinationCoderV4` from `getDestinationCoder()`. Moving a running streaming pipeline to V4 changes how destinations are encoded, so it needs a drain, the same as turning on `withClustering()`. If a destination has labels and its coder would drop them, the write fails with a message that says which option to set. `withTableLabels()` on a single-table write fails when the pipeline is built. Two other places dropped labels and are fixed here: the wrapper that applies `withTimePartitioning()` and `withClustering()` to dynamic destinations, and the temp-table path that reads an existing table's partitioning. Testing: new tests in BigQueryIOWriteTest cover FILE_LOADS, streaming inserts, temp tables on a table that already has labels, and the error cases. TableDestinationCoderV4Test covers round trips and deterministic encoding. Both pass locally. I have not run this against real BigQuery, and the Storage Write API path has no direct test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
