dabla commented on PR #57975: URL: https://github.com/apache/airflow/pull/57975#issuecomment-3666718770
You might be wondering why I came up with this. We’re using YugabyteDB as our Airflow metadata database (yes, I know it’s not officially supported), and recently we ran into performance issues with the API server, including crashes when XCom routes were invoked. Some DAGs using mapped tasks caused spikes in XCom retrieval, which contributed to the problem. This led me to design this solution—to reduce load—even though it didn’t fully solve the root issue. After digging deeper, we discovered that the indices created for the Airflow metadata database rely on Postgres logic, because Airflow (via SQLAlchemy) assumes it’s talking to Postgres, whereas in reality, it’s not. By recreating optimized indices specifically for YugabyteDB, we were finally able to tackle the root cause. Still, I think this PR optimization is valuable and worth considering. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
