potiuk commented on PR #61657: URL: https://github.com/apache/airflow/pull/61657#issuecomment-4043224567
> When I was researching how to write a custom xcom backend on my own, I noticed that the memory usage was very high when using Python operators. I eventually discovered that this was caused by this configuration. I think it should be reminded to the DAG authors that enabling this configuration will increase memory usage, or a log message should be provided to alert them. Otherwise, beginners will always fail to notice these issues. It's only when whatever is returned in a huge value as output - you might just have hit this particular issue. What is your proposal @13129 for those 100.000+ people who rely on this behaviour - do you propose that all such people suddenly just find their Dag stop working and scramble to modify all their 100s of Dags to add this parameter indivdually to every Python Operator or @task annotated function ? Is this is what you propose? Because I am not sure you know - every single @task annotated method uses PythonOperator under the hood - and it meaas that when we change the defaults - all of those Dags stop working. Do you reaslise consequeneces of what you propose? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
