dilex42 commented on issue #28199: URL: https://github.com/apache/airflow/issues/28199#issuecomment-1522233528
Hi. I have the same problem and it's kinda frustrating as I think it completely ruins Taskflow idea of presenting DAGs as composition of tasks. As per your [Medium article](https://medium.com/apache-airflow/airflow-2-0-dag-authoring-redesigned-651edc397178) and issue #8059, Custom XCom was designed to allow users to pass large objects between tasks for cleaner code. And it works great when a task returns something. Problem arises later when you try to pass that returned value to another task. Then all of a sudden we face another attempt to write this large object to database for some reason. So in reality whole "pass objects from task to task" concept isn't working and that's why I completely agree that this issue should be classified as a bug. Next, I am wondering what possible solutions could there be: 1. Is it necessary to deserialize XComs before task serialization in db? While I am not an avid user of `rendered template` tab, I understand why some would like to see `templates_dict` rendered. But what's the usage of rendered args and especially rendered xcoms? You can always check them in xcoms section or in you custom storage. And, I believe, that tasks that yet to be run have something like `XComArgs(DecoratedPython)...` in their rendered args. So can't it stay that way in others states? 2. Perhaps you will be open to let users customize what is serializing in each task? Currently it's `['templates_dict', 'op_args', 'op_kwargs']` but maybe we can pass some parameter to change that behaviour and exclude args from certain tasks 3. That being said, I agree that using `orm_deserialize` seems to be good idea as it gives necessary customisation opportunities and kinda why it was invented, if I understand correctly. Am I wrong in hoping that this change is only 1 line of code and should be pretty straightforward or there are a lot of places where this should be changed? If you can point me to specific code in question, I don't mind to try to make a PR. Also for people with same problem, I'm currently using a workaround of explicitly pulling desired XCom inside a task. It's not ideal but it's clean enough and works. Thanks for your time, and sorry if it sounds harsh, I'm just frustrated a little bit😞 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
