claw89 commented on PR #43446:
URL: https://github.com/apache/airflow/pull/43446#issuecomment-2478385846
## Options for Financial Services API in Airflow
To summarize the discussion, we have outlined the possible directions we
could take with this PR. We'd welcome your comments on any alternative
solutions or additional pros/cons so we can reach a consensus on the best way
forward.
### 1. Existing Google provider
```
📦providers
┣ 📂google
┃ ┣ 📂cloud
┃ ┃ ┣ 📂hooks
┃ ┃ ┃ ┗ 📜financial_services.py
┃ ┃ ┣ 📂operators
┃ ┃ ┃ ┗ 📜financial_services.py
┃ ┣ 📂common
┃ ┃ ┣ 📂hooks
┃ ┃ ┃ ┗ 📜base_google.py
```
Pros
- No need to set up/support a new provider
- Financial Services hook can use google authentication in base_google.py
- Adds no additional dependencies to the google provider package
Cons
- Additional methods in hooks/financial_services.py to access discovery doc
- Adds another service to the already large google provider
### 2. Separate provider with duplicated base code
```
📦providers
┣ 📂google-cloud-ai
┃ ┣ 📂hooks
┃ ┃ ┣ 📜base_google.py *
┃ ┃ ┗ 📜financial_services.py
┃ ┣ 📂operators
┃ ┃ ┗ 📜financial_services.py
┣ 📂google
┃ ┣ 📂cloud
┃ ┣ 📂common
┃ ┃ ┣ 📂hooks
┃ ┃ ┃ ┗ 📜base_google.py *
```
Pros
- Initiates the processes of separating out Cloud AI products from other GCP
products
- Separates Financial Services into new CI dashboard
Cons
- Authentication code in `base_google.py` needs to be duplicated inside
`google-cloud-ai` provider. Possible divergence between
`google-cloud-ai/hooks/base_google.py` and `google/hooks/base_google.py` as
changes are introduced in one of the providers.
### 3. Shared code in separate core provider
```
📦providers
┣ 📂google-core
┃ ┣ 📂hooks
┃ ┃ ┗ 📜base_google.py
┣ 📂google-cloud-ai
┃ ┣ 📂hooks
┃ ┃ ┗ 📜financial_services.py
┃ ┣ 📂operators
┃ ┃ ┗ 📜financial_services.py
┣ 📂google
```
Pros
- No duplication of authentication code; all hooks requiring google
authentication import from `google-core`
- Lightweight `google-core` only depends on core google packages (e.g.,
`google.auth` and `google.api_core`)
Cons
- Unlcear if inter-provider dependency is possible
- Splitting `hooks/base_google.py` out of google provider into new
`google-core` provider; `google-core` becomes a dependency for `google`.
Similarly, `google-core` is a dependency for `google-cloud-ai`. May require
significant refactor of `google` provider
### 4. Delay merge of Financial Services
Until either 1) Financial Services discovery doc is released without need
for developer key or 2) Google provider is split into smaller providers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
