eladkal edited a comment on issue #13898: URL: https://github.com/apache/airflow/issues/13898#issuecomment-768041034
Xcom has limits. In Airflow 2 you can define a custom xcom backend see [documentation](https://airflow.apache.org/docs/apache-airflow/stable/concepts.html?highlight=xcom#custom-xcom-backend), [polidea article](https://www.polidea.com/blog/airflow-2-0-dag-authoring-redesigned/) I don't know your use case however from what you described I'm not sure if xcom are even needed here. Why not simply writing the delta (increment) to a folder in GCS in a known path? For example using `{{ ds }}` for daily runs or `{{ ts_nodash }}` for hourly runs then the loading job can just read the data from this known path without looking in xcoms. In any case since this is not a bug/feature request but more of a troubleshooting question for a specific ETL I'm closing this issue. If you need further assistance please ask on https://stackoverflow.com/ ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
