hkc-8010 commented on PR #66854: URL: https://github.com/apache/airflow/pull/66854#issuecomment-4980960821
@uranusjr I reworked this along the lines we discussed. The set-state endpoint no longer registers asset events under the task_instance lock and no longer drops them on failure. On success it now commits the TI state first (releasing the row lock), then tries the asset write once under a short lock timeout. If it cannot get the locks in time, or fails for any reason, it writes the events to a new asset_event_queue table. The scheduler drains that queue and retries until the events land, parking a row only after a bounded number of failures (with a metric). So the common case still registers inline, contention sheds to the queue, and nothing is silently dropped. I also kept the alias direct-INSERT fix, documented the enqueue-to-registration window in the asset-scheduling docs, and added a significant newsfragment. Validation: new endpoint and scheduler unit tests, plus breeze core-tests on Python 3.10 against both sqlite and postgres. Draining the TI state update itself through the same queue is left out, as we agreed. Would appreciate another look when you get a chance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
