github-actions[bot] commented on code in PR #41059:
URL: https://github.com/apache/beam/pull/41059#discussion_r4498085205
##########
sdks/python/apache_beam/io/gcp/bigquery.py:
##########
@@ -1522,6 +1523,25 @@ def BigQuerySink(*args, validate=False, **kwargs):
_KNOWN_TABLES = set()
+_PRIMITIVE_SIZEOF_TYPES = (str, int, float, bool, bytes, type(None))
+
+
+def _fast_row_and_insert_id_size(row_and_insert_id):
+ if type(row_and_insert_id) is tuple and len(row_and_insert_id) == 2:
+ row, insert_id = row_and_insert_id
+ if type(row) is dict and type(insert_id) in _PRIMITIVE_SIZEOF_TYPES:
+ getsizeof = sys.getsizeof
+ total = getsizeof(row_and_insert_id) + getsizeof(row)
+ if insert_id is not None:
+ total += getsizeof(insert_id)
+ for k, v in row.items():
+ if not (type(k) in _PRIMITIVE_SIZEOF_TYPES and
+ type(v) in _PRIMITIVE_SIZEOF_TYPES):
+ return get_deep_size(row_and_insert_id)
+ total += getsizeof(k) + (
Review Comment:
**🟡 Medium**: `objsize.get_deep_size` tracks visited object IDs so that
objects referenced multiple times within the same row (e.g., multiple columns
referencing the same string/bytes object, or cached/interned scalars like
`True`/`False`, small `int`s, or `""`) are only counted once. Here, `v` is only
deduplicated against `insert_id`, so if multiple keys in `row` share the same
value reference (especially a large `str` or `bytes` object), `total` will
count `sys.getsizeof(v)` multiple times. This could cause a row to be
overestimated relative to `get_deep_size` and falsely exceed
`_max_insert_payload_size` (routing valid rows to `FAILED_ROWS`).
Consider either falling back to `get_deep_size` when `total >=
max_insert_payload_size` in `_check_row_size` before rejecting a row, or
tracking seen object IDs when needed.
##########
sdks/python/apache_beam/io/gcp/bigquery.py:
##########
@@ -1727,7 +1747,7 @@ def _create_table_if_needed(self, table_reference,
schema=None):
def _check_row_size(self, row_and_insert_id) -> tuple[int, Optional[str]]:
"""Returns error string when the row estimated size is too big"""
- row_byte_size = get_deep_size(row_and_insert_id)
+ row_byte_size = _fast_row_and_insert_id_size(row_and_insert_id)
Review Comment:
**🟡 Medium**: If `_fast_row_and_insert_id_size` overestimates a row due to
shared object references across fields (which `get_deep_size` deduplicates), a
row near the limit could be routed to the DLQ even though
`get_deep_size(row_and_insert_id) < self._max_insert_payload_size`. Re-checking
with `get_deep_size(row_and_insert_id)` inside the `if row_byte_size >=
self._max_insert_payload_size:` branch would preserve exact
backwards-compatible DLQ behavior at zero cost on the normal path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
