This is what we’ve been doing internally, and if it becomes the “official” way
of doing things, then I’m more than happy.
Cheers,
Björn
> On 14. Jan 2019, at 13:38, Ash Berlin-Taylor <[email protected]> wrote:
>
> Operators and hooks are just normal python modules, and don't actually need
> to be defined under `airflow.` to work.
>
> My advice (and something I've been meaning to update the docs to say) is to
> just make your custom hooks/ops a normal class and do
>
> from myco.airflow.operators.my_op import MyOp
>
> So long as the "myco" package is installed on the scheduler and worker nodes
> this will work fine (this is what we do at work)
>
> The plugin mechanism is I think still needed to get Flask views/menu_items
> integrated (as the need to be registered with the app at creation time)?
>
> So: not to namespace packages as I don't think we need it.
>
> -ash
>
>> On 14 Jan 2019, at 11:58, Björn Pollex
>> <[email protected]> wrote:
>>
>> With the recent (awesome) new feature to [load plugins from entry
>> points][1], I’m wondering if this could be taken further, to make handling
>> plugins simpler and less magic. One pain point for me personally has been
>> that, since plugins are loaded by Airflow magic, my IDE cannot give me any
>> help (e.g. syntax completion) when using operators, hooks or similar that
>> are loaded from plugins, because it can’t find the source files based on the
>> package name. This could be fixed by changing the packages `airflow`,
>> `airflow.hooks`, `airflow.operators` etc. into [namespace packages][2]. With
>> such a change, one wouldn’t even need to declare a plugin in order to
>> distribute a new hook or operator, a plain Python package is enough.
>>
>> Furthermore, I wonder if we need the `AirflowPlugin` class at all. Instead,
>> we could have dedicated entry points for views, custom menus, etc., and each
>> package would register relevant classes with those entry points directly.
>> This would remove a lot of magic from Airflow, and instead use established
>> tools to handle the complexity.
>>
>> I believe it would be possible to implement both of these without breaking
>> existing code, so a migration path is available. The main problem is that we
>> would need to move all code that currently resides in the `__init__.py`
>> files of the prospective namespace packages elsewhere (the constituent parts
>> of namespace packages are loaded in a non-deterministic order, so one can’t
>> rely on code any specific `__init__.py` if a namespace package to be
>> executed first, or, in fact, at all), but as far as I can see, everything in
>> there can neatly fit somewhere else.
>>
>> I would love to start working on this, if there is a chance that at least
>> parts of this would find acceptance. What does the Airflow dev community
>> think about this?
>>
>> Cheers,
>>
>> Björn Pollex
>>
>> [1]: https://github.com/apache/airflow/pull/4412
>> [2]: https://packaging.python.org/guides/packaging-namespace-packages/
>>
>
