claudevdm commented on code in PR #34603:
URL: https://github.com/apache/beam/pull/34603#discussion_r2040265102
##########
sdks/python/apache_beam/io/avroio.py:
##########
@@ -163,6 +163,8 @@ def __init__(
super().__init__()
self._source = _FastAvroSource(
file_pattern, min_bundle_size, validate=validate)
+ # Disable fastavro's automatic logical type conversion
+ fastavro.read.LOGICAL_READERS.clear()
Review Comment:
This will remove all avro LOGICAL_READERS in both the case where someone
outputs as_row, or not meaning whatever the atomic value of the logical type is
will be passed and all other info will be lost..
This might break for people who depend on the current LOGICAL_READERS
behavior when as_row=False. If we decide to pass timsetamp-millis as integers,
a more targeted fix would be to remove the LOGICAL_READER for
'long-timestamp-millis' only if as_row=True.
Also worth considering whether we want to treat the "timestamp-millis" as a
Timestamp like mentioned in
https://github.com/apache/beam/issues/31656#issuecomment-2180164907.
@Abacn do you have an opinion whether the "timestamp-millis" type should be
treated as an integer or a "Timestamp" type?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@beam.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
