GitHub user richangg closed a discussion: xcom pull/session.query(xcom)/connector.exec not able get latest xcom item from database
Original Problem: Databricks occasionally returns a 500 error on the get_run_id status request. The plan is to use XCom to retrieve the previous run_id on the second trial and check if the first trial is still running. If it is, the run_id will be returned when the second trial submits the run. Issue with the Above Solution: The current task is configured with 3 retries, so when the first attempt fails, the second attempt is triggered. I applied do_xcom_push on the DatabricksSubmitRunOperator to ensure that the run_id is saved in XCom when it is submitted successfully. I also verified via query that the latest XCom value is saved. However, when I use xcom_pull, session.query(XCom), or connector.exec to retrieve the value, I only get the XCom value from the prior date (the last retry of the previous run), not the XCom value from the first trial of the current run. GitHub link: https://github.com/apache/airflow/discussions/43242 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
