dstandish commented on issue #15933: URL: https://github.com/apache/airflow/issues/15933#issuecomment-844585293
This came up in part because leveldb hooks depend on plyvel library which requires that the machine has leveldb installed. So on mac if you want to install google provider (e.g. because you want to use some GCP hook) then you also have to install leveldb (which has nothing to do with GCP), and if you do not have leveldb installed, then the google provider install will fail. To resolve this, what we ended up doing was making plyvel an optional dependency of google provider (#15770) so you have to install `...google[leveldb]` to get plyvel (and this will only work if your system has leveldb installed). This also required that we make plyvel a "core" dependency of airflow, since CI requires it in order to fully import the google provider. _(Come to think of it perhaps CI could be amended to do `...provider-x[ALL]` and then we would not have to add it as a core dep...)_ Rather than making leveldb a google provider extra, I think a cleaner solution would be if leveldb was made a distinct provider, either google-leveldb or just leveldb. In the case of leveldb, there's no interaction with GCP hooks -- AFAIK it's completely independent. And this way plyvel would not need to be an extra, and would not need to be a airflow core dep. The issue with leveldb is my main concern because being forced to `brew install` this system dependency is a bad user experience. But I'm not aware if there are other similar problems within the google provider, or other reasons that there may be to separate certain components of the provider -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
