damccorm commented on code in PR #41060:
URL: https://github.com/apache/beam/pull/41060#discussion_r4513554688
##########
sdks/python/apache_beam/io/gcp/bigquery_tools.py:
##########
@@ -1562,7 +1570,11 @@ def insert_rows(
str(self.unique_row_id) if not insert_ids else insert_ids[i]
for i, _ in enumerate(rows)
]
+ _json_safe = (str, int, bool, type(None))
rows = [
+ r if (
+ type(r) is dict and
+ all(type(v) in _json_safe for v in r.values())) else
Review Comment:
Fixed in fd808b34037: added `type(k) is str` and 64-bit integer range check
(`-(1 << 63) <= v < (1 << 63)`).
##########
sdks/python/apache_beam/io/gcp/bigquery_tools.py:
##########
@@ -1828,8 +1840,11 @@ def read(self, size=-1):
raise io.UnsupportedOperation("AvroRowWriter is not readable")
def tell(self):
- # Flush the fastavro Writer to the underlying stream, otherwise there isn't
- # a reliable way to determine how many bytes have been written.
+ # Include both bytes already flushed to the underlying stream and bytes
+ # currently buffered in the fastavro Writer, avoiding a costly block flush
+ # (and 16-byte sync marker) after every single row.
+ if hasattr(self._avro_writer, 'io'):
Review Comment:
Fixed in fd808b34037: added an assertion in
`TestAvroRowWriter.test_write_row` verifying `writer.tell() > buf.tell()` while
records are buffered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
