github-actions[bot] commented on code in PR #41392:
URL: https://github.com/apache/beam/pull/41392#discussion_r4583252354
##########
sdks/python/apache_beam/yaml/yaml_mapping_test.py:
##########
@@ -541,6 +541,69 @@ def test_extract_windowing_info_duplicate_field(self):
fields: [timestamp]
''')
+ def test_assign_timestamps_iso8601_and_datetime(self):
+ with beam.Pipeline(options=beam.options.pipeline_options.PipelineOptions(
+ pickle_library='cloudpickle')) as p:
+ elements = p | beam.Create([
+ beam.Row(user='alice', event_time='2026-01-01T00:00:10Z'),
+ beam.Row(user='bob', event_time='2026-01-01T02:00:25+02:00'),
+ beam.Row(user='carol', event_time='2026-01-01T00:00:30'),
+ ])
+ result = (
+ elements
+ | 'Assign' >> YamlTransform(
+ '''
+ type: AssignTimestamps
+ config:
+ timestamp: event_time
+ ''')
+ | 'Extract' >> YamlTransform(
+ '''
+ type: ExtractWindowingInfo
+ config:
+ fields: [timestamp]
+ '''))
+ assert_that(
+ result | beam.Map(lambda x: (x.user, x.timestamp)),
+ equal_to([
+ ('alice', Timestamp.from_rfc3339('2026-01-01T00:00:10Z')),
+ ('bob', Timestamp.from_rfc3339('2026-01-01T00:00:25Z')),
+ ('carol', Timestamp.from_rfc3339('2026-01-01T00:00:30Z')),
+ ]))
+
+ def test_assign_timestamps_datetime_callable(self):
Review Comment:
**🔵 Low**: Consider also including a timezone-aware non-UTC
`datetime.datetime` (e.g., `datetime.timezone(datetime.timedelta(hours=2))`) in
this test to cover the `else: ts = ts.astimezone(datetime.timezone.utc)` branch
in `_to_timestamp`.
##########
sdks/python/apache_beam/yaml/yaml_mapping.py:
##########
@@ -871,8 +871,29 @@ def _AssignTimestamps(
"""
timestamp_fn = _as_callable_for_pcoll(pcoll, timestamp, 'timestamp',
language)
T = TypeVar('T')
- return pcoll | beam.Map(lambda x: TimestampedValue(x, timestamp_fn(x))
- ).with_input_types(T).with_output_types(T)
+ return pcoll | beam.Map(
+ lambda x: TimestampedValue(x, _to_timestamp(timestamp_fn(x)))
+ ).with_input_types(T).with_output_types(T)
+
+
+def _to_timestamp(ts):
+ if isinstance(ts, str):
+ if ts and ts[-1] not in 'Zz' and '+' not in ts[10:] and '-' not in ts[10:]:
Review Comment:
**🟠High**: The heuristic here has a couple of issues with timezone-less
ISO-8601 strings:
1. **Precision loss / failure on sub-microsecond fractions**: For a 9-digit
nanosecond string without timezone (e.g. `'2026-01-01T00:00:10.123456789'`),
`datetime.datetime.fromisoformat()` in Python 3.11+ silently truncates the
fractional part to microseconds (`123456`), whereas `Timestamp.from_rfc3339`
preserves nanosecond precision. For 7 or 8 fractional digits (e.g.
`'2026-01-01T00:00:10.1234567'`), `fromisoformat()` raises `ValueError`, falls
through to `Timestamp.from_rfc3339(ts)`, which then fails in
`dateutil.parser.isoparse(ts).astimezone(pytz.UTC)` with `ValueError: Not naive
datetime (tzinfo is already set)` inside `pytz` because `isoparse` returns a
naive datetime.
2. **Compact ISO-8601 dates**: Strings like `'20260101T000010-0500'` have
length < 10 before the offset, or strings like `'2026-01-01'` (length 10) may
fall through to `Timestamp.from_rfc3339('2026-01-01')`, which similarly crashes
in `.astimezone(pytz.UTC)` because `isoparse` produces a naive `datetime`.
Instead of parsing manually with `fromisoformat`, consider fixing
`Timestamp.from_rfc3339` (in `apache_beam/utils/timestamp.py`) to handle naive
datetimes returned by `dateutil.parser.isoparse` (e.g., `dt =
dt.replace(tzinfo=pytz.UTC) if dt.tzinfo is None else
dt.astimezone(pytz.UTC)`), and then simply calling `Timestamp.from_rfc3339(ts)`
for all `str` inputs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]