Hi,
also sharing the opinion that no dedicated UI is needed for this - but
would be very welcoming to share experience and maybe expereince and
starting promt to get going. So if you want to post (somewhere, e.g.
medium) an article about how this is possible, that migth be worthwile
to share.
CFP for the next Airflow summit is over but there might be also other
oppuortunities to share and spread the use w/o needing to integrate into
Airflow codebase. Maybe even on Monthly townhall hosted by Astronomer.
(The taks from 2 years ago might be a bit outdated though they share the
same ground idea - but a lot has evolved since then...)
Jens
On 07.07.25 12:53, Kaxil Naik wrote:
Yup agreed with Jarek. A strong no from my side. We don't want to allow
authoring DAGs from Airflow UI especially just to provide an LLM interface.
On Mon, 7 Jul 2025 at 12:27, Jarek Potiuk <ja...@potiuk.com> wrote:
Also you might take a look at Airflow Summit videos
https://www.youtube.com/playlist?list=PLGudixcDaxY2NIjMYT8t5zA9KJ47wTCkM
->
and look back to 2023. There were at least several talks about using LLMs
to generate Airflow Dags, and our users are doing it already - and I guess
it's quite natural for people to generate the Dags with the help of LLMs
already.
On Mon, Jul 7, 2025 at 8:52 AM Jarek Potiuk <ja...@potiuk.com> wrote:
FYI I added your email directly - because apparently you are not
subscribed to devlist - please do subscribe following the "community" tab
on our website.
I don't want to cut down your wings and excitement, but this is a
deliberate choice that Airflow UI does not allow to author DAGs. This is
a
security feature. And our security model
https://airflow.apache.org/docs/apache-airflow/stable/security/security_model.html
is very clear that "UI users" do not have (and should not have)
capabilities of authoring DAGs (not as Python code - that allows
arbitrary code execution). Maybe (and that is something we might consider
in the future) if there is a declarative way of creating DAGs which does
not allow to provide arbitrary code, we could allow that, but we have not
even settled on the idea of having a single declarative way of creating
Dags.
Also Airflow DAGS are just Python Code placed in a folder. And there is
absolutely nothing stopping you to open your IDE with Claude , Cursor,
Copilot, use the prompt of your choice and ... generate DAGs with LLM.
There is absolutely no need to have a UI for that.- all the IDEs out
there
already have a fantastic LLM integration, with capability of adding
prompt,
using MCP servers (there are even several MCP servers for Airflow created
by the community and we are discussing about creating our own MCP server
https://lists.apache.org/thread/xgd66v6s7zf0xkvy3c7ysqvn4csgmw0 - those
IDEs have code completion, syntax check, allow you to interact with the
Agents and approve/reject proposals when you are using agents to create
your DAGs. They even allow you to use your own models that can be
RAG-ified
based - for example - on the private DAGs your company might have. This
all
works **today**.
I don't think personally there is any benefit of creating a similar
feature in Airflow UI. I can't see any to be honest. Maybe others have a
different opinion or maybe you can explain what benefits you see by
adding
such a "UI feature" to Airflow itself (and also the problem about
security
is extremely important and a huge blocker for the whole idea - until this
is somewhat addressed the whole idea is basically impossible to be
accepted
by the community.
J.
On Mon, Jul 7, 2025 at 8:37 AM Harikrishnan Girikumar <
harikrishnangiriku...@gmail.com> wrote:
Hello Team,
My name is Harikrishnan(Hari), I have an idea/improvement proposal for
Airflow.
LLM-powered feature within Apache Airflow to significantly enhance the
DAG
authoring experience. Users would be able to provide natural language
descriptions or queries and leverage Large Language Models (LLMs) to
automatically generate and modify Airflow DAGs. This aims to democratize
DAG creation, reduce the learning curve for new users, and accelerate
the
development of complex workflows. For example: We can have a UI tab in
Airflow where users can add their respective authentication credentials
for
the LLMs they want to use (OpenAI, Claude or their personal model
serving
link etc.) they can select their AI from drop down and a chat window to
input queries like: Create a DAG to copy my data from S3 to Postgres and
the code generated would be copied to DAG folder. We can restrict the
Prompts to be strictly for DAG generation for initial trial, further
down
the line a RAG feature could be added where a Vectorized version of
Airflow
documentation is used to improve the accuracy of DAG creation.
I am really excited about this feature, this would reduce the learning
curve and improve the interaction for new users. Let me know your
thoughts,
looking forward to hearing from the team.
Regards,
Hari
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org
