mxtymoshyk opened a new pull request, #40790:
URL: https://github.com/apache/beam/pull/40790
Fixes #27069.
Adds integer-range partitioning to `BigQueryIO.Write` in the Java SDK.
Python already supports it through `additional_bq_parameters`.
## Usage
```java
BigQueryIO.writeTableRows()
.to("project:dataset.table")
.withSchema(schema)
.withRangePartitioning(
new RangePartitioning()
.setField("customer_id")
.setRange(new
RangePartitioning.Range().setStart(0L).setEnd(1000L).setInterval(10L)));
```
With `to(SerializableFunction)` or `to(DynamicDestinations)`, set the
`RangePartitioning` on each returned `TableDestination` and call
`withRangePartitioning()` with no arguments. A custom `DynamicDestinations`
must also return `TableDestinationCoderV4` from `getDestinationCoder()`.
## Update compatibility: drain and restart required
Turning on range partitioning switches the destination coder to a new
`TableDestinationCoderV4`. V4 writes the V3 encoding followed by the range
partitioning, and it can't read state written by V2 or V3. A running pipeline
can't be updated in place to turn this on. Drain it and start a new one.
Pipelines that don't call `withRangePartitioning` keep
`TableDestinationCoderV2` or `V3`, so their encoding is unchanged and in-place
updates keep working. V4 is registered in the Dataflow runner's
`DefaultCoderCloudObjectTranslatorRegistrar`.
## Changes
- `TableDestination` carries an optional range partitioning as a JSON
string, the same way it stores time partitioning and clustering.
- Table creation (`CreateTableHelpers`) and load jobs (`WriteTables`,
`UpdateSchemaDestination`) pass range partitioning to BigQuery. This covers
`FILE_LOADS`, `STREAMING_INSERTS` and the Storage Write API.
- `FILE_LOADS` temp tables copy the range partitioning of an existing
destination table, so copy jobs into it succeed. This is the Java counterpart
of #38833.
- `PrepareWrite` fails with an error if a destination has range partitioning
but the destination coder would drop it. Without the check, the table would be
created without partitioning and no error.
- Setting both `withTimePartitioning` and `withRangePartitioning` fails at
construction, since a BigQuery table can have only one.
- `BigQueryIOTranslation` adds `json_range_partitioning` to the transform
upgrade schema, read only when upgrading from 2.78.0 or later. If this misses
the 2.78.0 cut, I'll move that gate and the `CHANGES.md` entry to 2.79.0.
## Tests
- Unit: `TableDestinationCoderV4Test`, plus new cases in
`BigQueryIOWriteTest` and `BigQueryIOTranslationTest`.
- Integration: `BigQueryRangePartitioningIT` writes to real BigQuery: a
single table, a table function with clustering, the Storage Write API, and temp
tables into an existing range-partitioned table. It runs in
`beam_PreCommit_Java_GCP_IO_Direct_BigQuery`. I haven't run it against my own
project, so that job is its first real run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
