Taragolis commented on PR #29580: URL: https://github.com/apache/airflow/pull/29580#issuecomment-1433725441
> Customers don't always write the most optimized DAGs. Conns, Configs and variables are always searched in secrets backend before env variables and meta store. This change will allow customers to only lookup secrets backend when their variables, et al. follow a pattern defined by them. Based on MWAA data, I can confirm that customers are spending hundreds of dollars on their secrets backed, just to use it with Ariflow. That actually just postpone the problem, and might lead to problem with Aurora instances (AFAIK it uses in MWAA as Postgres backend), yeah it would be cheaper for end users, because in this case they would not pay for IOPS on Aurora in case of MWAA, but still can be an issue with backend itself. Is MWAA still support only Secrets Manager by default? Because I guess SSM Parameter store much cheaper rather than AWS SM. You don't have to pay 0.4 cents secrets/per month, and no cost for API call for Standard Parameter with Standard Throughput, I know because personally use SSM since Secrets Backends introduced in Airflow 1.10.10 I do not have any concern about this changes, just want to mention that the main problem, that users do not follow Best Practice, and as result they have a huge or not bills. The huge bill might happen if user turn on DEBUG level in production and write everything to logs in this case CloudWatch pricing might be, well... pretty surprising. > Maybe it's time resurrect this one from @Taragolis ? https://github.com/apache/airflow/pull/23560 Ooooh... My first PR to Airflow repo 🤣 That still in my list, actually in two list, TODO and make wide discussion about Configurations and consistency with Secrets Backend, still no idea how to resolve the problem, that Airflow DB use both Secrets Backend capability and own implementation for obtain Variables and Connections in CLI/API/Webserver -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
