ashb opened a new issue, #74657: URL: https://github.com/apache/airflow/issues/74657
Affects 3.4.0: beta1 When I landed the 12 PR stack for Task Loops/AIP-111, we all (I and the reviewers) failed to catch that the custom XCom backend is only given the 4-tuple coordinates for the task (dag_id,run_id,task_id, map_index). While this is fine for both metadata (where it uses the requesting TI to look up the current coordinates) and the common.io object backend (where it just so happens to also use a uuidv4 in the name, so it doesn't clash) it does mean a different XComBackend might not do that, and could end up storing the XCom data for each iteration of the loop in the same path, meaning effectively that only the final iteration data is kept. Need to think how to resolve this without breaking existing dags by changing the `XCom.set` interface, and without totally breaking existing XCom backends. One option might be to somehow detect if the xcom backend has been updated, and to fail the dag at parse time when an invalid xcom backend is used with loops. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
