vemikhaylov commented on PR #29608: URL: https://github.com/apache/airflow/pull/29608#issuecomment-1437501187
@uranusjr thank you for your feedback! > Using JSON for this feels cumbersome to me. I think it does for me as well.. At the same time, it's probably worth mentioning that `--env-vars` and `--task-params` also already use JSON for the command, so it might be not a bad idea to have `--xcom-args` consistent with them. Thinking out loud which alternative options of passing things through we may also have. **1. Simplified format in CLI** (task_id, xcom_key, value) triplets joined with a delimiter (a comma below). ``` --xcom-args get_python_echo_message,return_value,"test xcom arg" manually_push_bash_command,command,"echo \'test bash\'" ``` Probably all the escaping and ad-hoc handling will make things even worse. **2. Values passed through a file** ``` --xcom-args task_args.json ``` Sorry for ".json" - Yeah, like again, the contents of the file should have some structure. We may go with yaml :) What generally doesn't look convenient here with whatever structure of the file - requiring to create a file even for simple scenarios (if passing through the file is the only option). -- > Also we need to find a way to support custom backends that may not store values as JSON. Let me clarify to make sure that I understand it correctly. You mean that generally the `BaseXCom` model allow us to store any binary (`LargeBinary`) `value`'s and not all of them may be passed as JSON and through CLI itself? Probably for that the approach #2 above can be used like: ``` --xcom-args task_args.pickle ``` Though from the usability perspective I wouldn't stick to this option as the only available one. What do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
