GitHub user richangg closed a discussion: xcom 
pull/session.query(xcom)/connector.exec not able get latest xcom item from 
database

Original Problem:
Databricks occasionally returns a 500 error on the get_run_id status request. 
The plan is to use XCom to retrieve the previous run_id on the second trial and 
check if the first trial is still running. If it is, the run_id will be 
returned when the second trial submits the run.

Issue with the Above Solution:
The current task is configured with 3 retries, so when the first attempt fails, 
the second attempt is triggered.

I applied do_xcom_push on the DatabricksSubmitRunOperator to ensure that the 
run_id is saved in XCom when it is submitted successfully. I also verified via 
query that the latest XCom value is saved. However, when I use xcom_pull, 
session.query(XCom), or connector.exec to retrieve the value, I only get the 
XCom value from the prior date (the last retry of the previous run), not the 
XCom value from the first trial of the current run.

GitHub link: https://github.com/apache/airflow/discussions/43242

----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Reply via email to