potiuk commented on pull request #20473: URL: https://github.com/apache/airflow/pull/20473#issuecomment-1002275764
> My real point always has been that the `xcom_pull` result should be consistent, and whether an encoding/decoding is used shouldn't matter to the user. Since `enable_xcom_pickling=False` (which uses json) is quite limited in the data types that can be serialized, we chose to uuencode the return value from the `SSHOperator`, which causes the inconsistency. It works, though, as you indicated. I just wonder, why do you think it should be consistent? We do not have auto-discovery of types and format of the returned values in Airflow. Airlfow is task-based not data-based, so basicaly in case you want to pass data between tasks both producer and consumer has to agree on naming conventions and data format used. There is no "universal" discovery. Also even in the case you started with - the "pickling" is deprecated and actually it should be set "per installlation". So it could be that in different installations, different format is used, but it will be "constistent" for that installation. I do not see huge value in providing "consistent" output format really in this case. > However, if "arbitrary binary return value" + "json serialization" is what we want to support, we should wrap json to deal with binary data, instead of forcing `SSHOperator` to do the encode work --- what about other operators that may return binary data? I do not think we need to do it. Again this is something we do not reallu want to support via json serialization. SSH is an outlier here indeed. We deliberately only support "json-serializable" objects in default XCom implementation. And possibility of supporting binary storage is implemented already via cusom XCom backends. See here for a well explained example https://www.astronomer.io/guides/custom-xcom-backends -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
