I think maybe we should simply gather some evidence on what users are using the extensions for. I can put together some evidence from my customers, and maybe others could contribute theirs?
Do others have experience with similar case where people extend Airflow by modifying Airflow's source code? I agree that the more flexible we are the more complex maintenance and backwards compatibility is so it all should be done very, very carefully. J. On Tue, Feb 18, 2020 at 6:13 PM Dan Davydov <ddavy...@twitter.com.invalid> wrote: > I think the kinds of plugins you are talking make sense in some contexts > (e.g. custom views when clicking on a task in the UI, e.g. ability to > visualize the data an ETL job provides) but we should be careful allowing > extensions to more core parts, it will become very hard to change/maintain > the project. From what I've seen the biggest gaps in customization/lowest > hanging fruit is extending the REST API and making it more robust instead > of experimental. > > On Tue, Feb 18, 2020 at 7:33 AM Jarek Potiuk <jarek.pot...@polidea.com> > wrote: > > > I have another discussion to start. We've recently talked to a number of > > customers who are extending airflow. It's often the case that people are > > modifying airflow's source code and later have a hard time with updating > it > > when newer versions of Airflow are released. > > > > This is a common trait - we've heard the same story from at least three > of > > our customers and it was also mentioned today at Slack by one of the > users: > > https://apache-airflow.slack.com/archives/CSS36QQS1/p1582014236100800 > and > > I heard some anecdotal evidence of people doing it at many events I > spoke. > > > > We have a plugin mechanism that allows for some extensibility but I > wonder > > if we should do something more complete. Maybe we could gather from > people > > a list of ways people are extending Airflow currently by modifying it's > > code and maybe we can come up with some "extension points" that we might > > introduce to Airflow to let them add custom functionality they want > rather > > than modifying Airflow's code. > > > > I think many of the extensions do not need to modify Airflow's source > code, > > and there won't be many extensions (maybe even we already have all that > we > > need but people do not know that they can extend airflow without > modifying > > the code. I think it would require a bit more description (and maybe some > > verification) of what internal API of Airflow is for those extensions so > > that we can keep backwards compatibility > > > > Let me know what you think? Is it worth it? Do you see any problems with > > trying to manage that? Maybe that's something we could introduce in 2.0 > (at > > least by better documenting what is "an extension" and providing some > > examples on how Airflow can be extended. > > > > J. > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/> > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
