We (Industry Dive) also use environment variable based configuration to configure our airflow system. Our particular case is that we use docker-compose for configuration management between environments of airflow services we deploy as docker containers, which can easily forward environment variables. We also use the cloud CI tool CircleCI to run automated tests on our DAGs which utilizes environment-variable based customization of the test environment. Lastly, we've also developed custom hooks that are closely coupled with the the airflow environment variable configuration system.
I agree with the idea that it is not ideal that the existing system results in multiple configuration levels that are not clearly exposed. But for 12-factor inspired services or 12-factor people/organizations, configuring via environment variables is a requirement. On Sat, Oct 1, 2016 at 3:09 AM, George Leslie-Waksman < [email protected]> wrote: > I would absolutely prefer to see them in the UI, and there are some > annoying details in the way that some hooks use env vars in non-standard > ways (I'm looking at you S3Hook) or have extra parameters that are hard > (impossible?) to include via the environment. > > It might be nice if the UI pulled connection displays from the same place > that the configuration is pulling them (config / env / etc.) > > There is also the slight wrinkle that we want people to be able to query > our metadata database but we don't necessarily want all of those people to > be able to read connection information or other secrets. I hope the fernet > key is good enough to make this a non-issue but, as I said, some of the > connections behave in funny ways and I certainly haven't had the chance to > dig deep enough to be entirely certain. > > It's sub-ideal and I'll be the first to admit my distaste for having > multiple ways to configure the same thing but, for the time being, it's how > we use Airflow. > > --George Leslie-Waksman > > On Fri, Sep 30, 2016 at 11:47 PM siddharth anand <[email protected]> > wrote: > > > I see. Thanks for the explanation. > > > > Then, wouldn't you like Airflow's connections views to read, edit, and > > delete connection information, whether that connection information is > > stored in the db or in the OS environment? Currently, OS env-provided > > connections are not exposed by the web app! Sure, they can be accessed by > > the hooks, but the connection list/edit views don't expose them. Do you > > find this a short-coming? > > > > -s > > > > On Fri, Sep 30, 2016 at 10:07 PM, George Leslie-Waksman < > > [email protected]> wrote: > > > > > We (Clover Health) currently deploy airflow to a 12-factor style > > > environment (Aptible) and exclusively use environment variables to > define > > > connections. Removal of this feature would require a significant > refactor > > > of how we use Airflow and would likely force us away from the mainline > > code > > > branch. > > > > > > I would be happy to discuss things further, if you need more details. > > > Certainly we could work around, and certainly there are things we don't > > > like about the current system, but we only have so many places that we > > can > > > devote resources at any given point in time. > > > > > > --George Leslie-Waksman (Clover Health) > > > > > > > > > On Fri, Sep 30, 2016 at 9:20 PM siddharth anand <[email protected]> > > wrote: > > > > > > I'm trying to understand the impetus for a feature - the ability to > > define > > > connections using OS environment vars. > > > > > > https://airflow.incubator.apache.org/concepts.html#connections > > > > > > It seems folks use this so that they can set up Airflow connections > using > > > automation (e.g. Ansible, chef, puppet. etc...). > > > > > > I've never used this feature and would prefer to have CLI commands set > up > > > connections (e.g. https://github.com/apache/ > incubator-airflow/pull/1802 > > ). > > > > > > It does not seem to me that these connections are revealed in the > > > connection list view page or imported into the db table. It seems like > > the > > > OS environment serves as a parallel repository for connection > > information, > > > side-by-side the airflow metadata db. I don't like this this "parallel" > > > repository as it is one more source of metadata to keep track of. > > > > > > If anyone is using the connections being set via env_var for any > purpose > > > aside from what I've listed here, please make it known. Else, I would > > like > > > to add this feature to the list of 2.0 deprecated features. > > > > > > In the meantime, I will spend some time to review and push the CLI > > > connection PR forward. We currently already support setting Variables > and > > > Pools via the CLI. It makes sense to manage Connections in the same > way. > > > The CLI can be called via automation toolsets like those listed above. > > > > > > -s > > > > > >
