Spasnof commented on issue #8508:
URL: https://github.com/apache/airflow/pull/8508#issuecomment-617505777
To add a bit of background @r-richmond and I ran into this issue today
upgrading from 1.10.2 to 1.10.10 and relying on a `spark2-submit` that was set
in our connection to set the behavior for 150+ operator calls in our
environment.
The "normal" order of operations for most operators (pun not intended)
appears to be:
1. Operator arg values (local to dag)
2. Connection arg values (local to environment)
3. Global parameter values (fallback)
In this case we have the `connection_id` and `spark_binary` args competing
for the same value. And the operator will always win if set and it is set by
default. So we are faced with a couple unpleasant options:
- Thinly wrapping the implementation in our own operator
- Setting the operator value in the 150+ occurrences.
- Monkey patching and linking back to this pr.
I hope this offers some background into the dilemma of having connection
vars overwritten by operator defaults.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
