github-actions[bot] commented on code in PR #40622:
URL: https://github.com/apache/beam/pull/40622#discussion_r4289504101
##########
sdks/python/apache_beam/io/gcp/bigquery.py:
##########
@@ -1517,6 +1519,7 @@ def __init__(
- BigQueryDisposition.CREATE_NEVER: fail the write if does not exist.
write_disposition: A string describing what happens if the table has
already some data. Possible values are:
+ - BigQueryDisposition.WRITE_TRUNCATE_DATA: delete table data but
preserve table metadata.
Review Comment:
**🟢 Low**: This line is 98 characters long (and line 2133 is 105
characters), exceeding the 80-character line length limit enforced by lint
checks.
##########
sdks/python/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_messages.py:
##########
@@ -5440,12 +5440,15 @@ class JobConfigurationTableCopy(_messages.Message):
destination table already exists. The following values are supported: *
WRITE_TRUNCATE: If the table already exists, BigQuery overwrites the
table data and uses the schema and table constraints from the source
- table. * WRITE_APPEND: If the table already exists, BigQuery appends the
- data to the table. * WRITE_EMPTY: If the table already exists and
- contains data, a 'duplicate' error is returned in the job result. The
- default value is WRITE_EMPTY. Each action is atomic and only occurs if
- BigQuery is able to complete the job successfully. Creation, truncation
- and append actions occur as one atomic update upon job completion.
+ table. * WRITE_TRUNCATE_DATA: If the table already exists, BigQuery
+ overwrites the table data, but does not use schema and table constraints
+ from the source table.* WRITE_APPEND: If the table already exists,
Review Comment:
**🟡 Medium**: `bigquery_v2_messages.py` is an autogenerated Apitools client
file (`# NOTE: This file is autogenerated and should not be edited by hand.`),
and `writeDisposition` is an unvalidated `_messages.StringField(8)`, so editing
this docstring has no runtime effect. In addition, lines 5445–5450 introduce
trailing whitespace that will fail lint checks (`table.* WRITE_APPEND: `).
Consider reverting manual edits to this file or removing the trailing
whitespace.
##########
sdks/python/apache_beam/io/gcp/bigquery.py:
##########
@@ -606,6 +607,7 @@ def validate_create(disposition):
@staticmethod
def validate_write(disposition):
values = (
+ BigQueryDisposition.WRITE_TRUNCATE_DATA,
Review Comment:
**🔴 Critical**: Allowing `WRITE_TRUNCATE_DATA` here enables it for
`BigQueryBatchFileLoads`, but
`sdks/python/apache_beam/io/gcp/bigquery_file_loads.py` only checks for
`('WRITE_EMPTY', 'WRITE_TRUNCATE')` in `BigQueryBatchFileLoads._load_data`
(line 970) and `('WRITE_TRUNCATE', 'WRITE_APPEND')` in
`UpdateDestinationSchema.process` (line 368).
Without including `'WRITE_TRUNCATE_DATA'` in
`BigQueryBatchFileLoads._load_data`, loads that spill into multiple temporary
tables will not be grouped by destination before `TriggerCopyJobs`. As a
result, parallel copy jobs will each execute with `WRITE_TRUNCATE_DATA` and
overwrite one another's data. Please update both checks in
`bigquery_file_loads.py`.
##########
CHANGES.md:
##########
@@ -66,7 +66,7 @@
* Support for X source added (Java/Python)
([#X](https://github.com/apache/beam/issues/X)).
* (Java) FirestoreIO: added
`RpcQosOptions.Builder.withRampupThrottlingDisabled()`, mirroring the existing
DatastoreIO option, to opt out of the 500/50/5 write ramp-up
([#27170](https://github.com/apache/beam/issues/27170)).
-
+* (Python) The BigQuery IO writer now allows the `WRITE_TRUNCATE_DATA`
write_disposition
Review Comment:
**🟢 Low**: Please preserve the blank line before `## New Features /
Improvements` and include the PR link at the end of the entry.
```suggestion
* (Python) The BigQuery IO writer now allows the `WRITE_TRUNCATE_DATA`
write_disposition ([#40622](https://github.com/apache/beam/pull/40622)).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
