damccorm opened a new pull request, #41060:
URL: https://github.com/apache/beam/pull/41060
### Description
Optimizes three hot paths in
`sdks/python/apache_beam/io/gcp/bigquery_tools.py`:
1. **Avoid per-row Avro block flushes in `AvroRowWriter.tell()`:**
- During BigQuery `FILE_LOADS` with `temp_file_format=AVRO`, the file
writer calls `AvroRowWriter.tell()` after every row to check shard rollover
limits. Previously, `AvroRowWriter.tell()` called `self._avro_writer.flush()`
before `self._file_handle.tell()`, forcing `fastavro` to close, compress, and
write a 1-row Avro block (plus a 16-byte sync marker) on every single record.
- Returning `self._file_handle.tell() + self._avro_writer.io.tell()`
includes buffered bytes without flushing the active Avro block (**16.05x
faster** `AvroRowWriter` throughput and **12.2x smaller** output files on
50,000 rows).
2. **Fast-path JSON-safe scalar rows in `BigQueryWrapper.insert_rows` and
`RowAsDictJsonCoder.decode`:**
- Skips the redundant `fast_json_loads(fast_json_dumps(r))` round-trip in
`BigQueryWrapper.insert_rows` when a row `dict` contains only JSON-safe scalar
values (`str`, `int`, `bool`, `None`) (**11.61x faster** row preparation), and
passes `bytes` directly to `json.loads()` in `RowAsDictJsonCoder.decode`.
3. **Precompile table spec regular expressions in `_split_table_spec`:**
- Hoists `_COMPILED_TABLE_SPEC_RE`, `_COMPILED_PROJECT_RE`, and
`_COMPILED_PROJECT_SEGMENT_RE` to module initialization (**2.34x faster**
`_split_table_spec`).
### Testing
- Verified with `pytest
sdks/python/apache_beam/io/gcp/bigquery_tools_test.py`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
