bolkedebruin commented on code in PR #35867:
URL: https://github.com/apache/airflow/pull/35867#discussion_r1408819129
##########
airflow/serialization/serde.py:
##########
@@ -83,6 +91,81 @@ def decode(d: dict[str, Any]) -> tuple[str, int, Any]:
return classname, version, data
+def encrypt(value: P) -> str:
Review Comment:
Sure I understand, but let's not mix several things here. `Serde` is being
used to serialize XCom values since 2.5 and in that way there is nothing new
about it. It was also quite elaborately discussed on the PRs at the time. It
was done without putting things into providers. Arguably, it should be the case
(as I mentioned elsewhere), but it isn't now.
Now some background on this PR:
XCom exposes values to the UI as you can view those values. This means that
everyone having access to the Airflow UI can see what has been shared between
tasks, including potential sensitive values - like access keys, secret keys,
etc. We limit visibility of this information for connections by encrypting
them, here we are doing the same.
Both the datalake table and iceberg serializer already included encryption
(see the change), because they store access credentials in their objects.
ObjectStoragePath can also keep storage options which potentially includes
access key and secret key. Hence, I decided to generalize and standardize the
encryption as it is error prone.
A protocol is required, because `fernet` cannot serialize anything else as
`bytes` and thus loses type information which is required for deserialization.
So, yes certain serializers could be moved to providers. It would require
making the API of `serde` public. That is not the scope of this PR. And yes, a
discussion should happen but I also think that should not hold back this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
