Anyone who was more involved in deferred operators ?
On Tue, Dec 21, 2021 at 5:14 PM Jarek Potiuk <[email protected]> wrote: > > Hey everyone, > > We had some discussions recently in Slack and GitHub Issues, about a > new approach that apparently we should take when it comes to hooks. > Seems that the "new" approach for current and future Hooks should be > to make them stateless. and not store connection information (or any > other non-serializable (and possibly state) information, because > keeping them "stateful" makes it not possible to fit into the > "Deferrable Operators" approach. > > I "feel" where it is coming from and there are some examples of "true" > deferrable (with remote connections) Operators being developed - > notably: Databricks operators : > https://github.com/apache/airflow/pull/19736. But I have no full > understanding/don't realize the full consequences yet. This might have > an impact on the reviews of committers when it comes to new > operators/hooks. > > So maybe we should discuss it here and let everyone know about this > change of approach (we have many hooks that have state and keep > connections once created). > > Is my understanding correct? Is it about "non-serializable" fields of > Hooks? Or any state fields ? Can someone who already has a deeper > understanding of this issue explain the context to others as well > (maybe an example too where it could go wrong :) ? > > J.
