FrankYang0529 commented on code in PR #74783:
URL: https://github.com/apache/airflow/pull/74783#discussion_r4321303690
##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -2907,7 +2907,15 @@ def process_log_messages_from_subprocess(
if ts := event.get("timestamp"):
# We use msgspec to decode the timestamp as it does it orders of
magnitude quicker than
# datetime.strptime cn
- event["timestamp"] = msgspec.json.decode(f'"{ts}"', type=datetime)
+ try:
+ event["timestamp"] = msgspec.json.decode(f'"{ts}"',
type=datetime)
+ except msgspec.DecodeError:
+ # The subprocess sends the timestamp field, and msgspec cannot
always parse its value. For
+ # example, user code can call self.log.info("...",
timestamp=1697190085).
+ # An uncaught DecodeError would stop the task supervisor
before it reports the
+ # task state, or stop the whole Dag processor during Dag
parsing. Moving the value to
+ # invalid_timestamp lets MaybeTimeStamper add the time the
supervisor received the line.
+ event["invalid_timestamp"] = event.pop("timestamp")
Review Comment:
Thanks. I tried keeping the value in `timestamp` as a string to see what the
log view shows. When the API server reads the task log file, it parses
`timestamp` again with pydantic. Pydantic reads some of these values
differently from msgspec:
- `1697190085` (an epoch number) is read as 2023-10-13T09:41:25Z, so the
line moves above lines that were written before it.
- A value with a newline, or a word such as `yesterday`, fails validation,
and the log view shows the whole JSON line as raw text.
With `invalid_timestamp`, the line keeps its place and gets the time the
supervisor received it. The log view still shows the original value after the
message. I'd like to keep `invalid_timestamp`. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
