bhirsz commented on PR #24058: URL: https://github.com/apache/airflow/pull/24058#issuecomment-1144473114
> ink it would be an easy lift too, t > I'd love to abstract this into a more general interface, but we'd have to get buy-in from the others who were involved in that AIP first. @o-nikolas I was also one of the people who were involved in the AIP. The communication was done through Mateusz, who then left the company - so he can chim in in the discussion since it's an open source community but the chances are we are on our own :). The general idea was to keep environment setup and running system tests separate - mostly because each provider have different needs (just for an example for running Google tests you need Google infrastructure, for AWS.. etc). At some point part of the old design was "polluted" by special needs of some providers - rather than having a uniform way of doing things. I'm all for moving anything that's common into common utils - so we don't reinvent the wheel ;) And if there are provider specific parts design the interface in a "plugable" way. I like your utils because they cover some of the issues I've also noticed: The common env_id helps to reduce required env id (for example in Google they used dozens upon dozens env vars to ensure uniqness of the name of the resource) but 1) you need to have it defined first 2) env id need to be of the specific format. The reason for 2) is because when we're using this variable in names it needs to be a valid name for the name (for example some names cannot contain - or special signs, or start with numbers). @ferruzzi Good idea with var_priority - it will allow to define a custom way of retrieving variables too (we could start with having only os.env as default and then allow to overwrite it). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
