> That would require some mechanism of declaring prior to task execution what > connections would be used
That’s exactly what I’m proposing in the proposal doc I’m working on (It’s part of also overhauling and re-designing the “Task Execution interface” that also gives us the ability to nicely have support for running tasks in other languages — much more than just BashOperator) This is a bit of a fundamental shift in thinking about task execution in Airflow, but I think it gives us some really nice properties that the project is currently missing. Tl;dr; lets discuss this in my doc when it comes our (next week most likely) please :) -ash > On 13 May 2024, at 18:15, Daniel Standish > <daniel.stand...@astronomer.io.INVALID> wrote: > > re > > As tasks require connection access, I assume connection data will somehow >> be passed as part of the >> metadata to task execution - whether it's part of the executor protocol or >> in some other way (I'm >> not an expert on that part of Airflow). Then, provided it's accessible as >> part of some execution >> context, and not only passed to the task's execute method, OpenLineage >> could utilize it. >> > > It's not strictly necessary that connection info be passed "as part of task > matadata". That would require some mechanism of declaring prior to task > execution what connections would be used. This is a thought that has come > up when thinking about execution of non-python tasks. But it's not > required from a technical perspective by AIP-44 because the > `get_connection` function can be made to be an RPC call so a task could > continue to retrieve connections at runtime.
