jolin1337 commented on code in PR #61777:
URL: https://github.com/apache/airflow/pull/61777#discussion_r2803010076
##########
providers/common/io/src/airflow/providers/common/io/xcom/backend.py:
##########
@@ -117,9 +117,13 @@ def serialize_value( # type: ignore[override]
run_id: str | None = None,
map_index: int | None = None,
) -> bytes | str:
- # We will use this serialized value to write to the object store.
- s_val = json.dumps(value, cls=XComEncoder)
- s_val_encoded = s_val.encode("utf-8")
+ if isinstance(value, bytes):
+ # Store raw bytes as-is
+ s_val_encoded = value
Review Comment:
Fair enough, I will start a discussion :)
You point out that we could do this manually in each task but what is the
point of/use case of this xcom backend in that case? I think it is pretty
convenient that it exists both as a fallback that we do not accidentally store
big files in S3 but also as a reliable feature to standardize the bucket paths
in s3 for all dags that are running in the airflow instance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
