DKER2 commented on code in PR #34603:
URL: https://github.com/apache/beam/pull/34603#discussion_r2068710809
##########
sdks/python/apache_beam/io/avroio.py:
##########
@@ -163,6 +163,8 @@ def __init__(
super().__init__()
self._source = _FastAvroSource(
file_pattern, min_bundle_size, validate=validate)
+ # Disable fastavro's automatic logical type conversion
+ fastavro.read.LOGICAL_READERS.clear()
Review Comment:
Hi guys, sorry for late reply. Thank you for your input. CMIIW, from what I
understand we only support this following datatype, that require data, that is
read by fastavio in sepecific type to be able to convert (E.g. boolean for
`schema_pb2.BOOLEAN`). I cannot find a type that it is compatible with
Timestamp. Image attached is from `schema_pb2.pyi`
> If we decide to pass timsetamp-millis as integers, a more targeted fix
would be to remove the LOGICAL_READER for 'long-timestamp-millis'
Here is the fulll list of `fastavrio==1.9.7` logical type converserion
functions. If we only eliminate LOGICAL_READER for
'long-timestamp-millis', it will still break for another functions. But
agree that we should have it only if as_row=True
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
