Hi All, Had a chat with @Kaxil. And we discussed on use-case first approach rather than opening up to all possibilities, on building and maintaining the MCP server.
A few use-cases which I had in mind were: - Debugging task failures - Schedule sparse across all dags for better resource utilization. - Recommend provider usage in place of native python codes. - Cross-DAG Dependency Analysis - Migration/Refactor Planning We would need an ongoing effort on this, as every other day something new comes up. Could be more optimization, or newer use-cases. We also need to maintain the security aspect of if. Like, where do we recommend running, auth and transport methods, etc. I made a new release today with FastMCP and discovery of tools based on category. After seeing today that FastMCP mount and sending refresh the tools list notification is working (*which it wasn't 3 months back*). Things I have on my roadmap: - HTTP based transport specifically for OAuth currently it is stdio only for it to be installed in a plugin (*airflow-wingman*) as a dependency. - Waiting for search tools to be implemented. - Decide to bake prompts with the server, perhaps. - Very minimal Docs as a resource which tells about recent changes in Airflow behavior (Example: *every LLM till date still tries to write schedule_interval instead of schedule*) @Shahar feel free to take a lead and see if there are things you would like to cherry-pick from it. Thanks, Avi On Fri, May 30, 2025 at 9:57 AM Avi <a...@astronomer.io> wrote: > 💯 agreed > > On Fri, May 30, 2025 at 9:52 AM Kaxil Naik <kaxiln...@gmail.com> wrote: > >> Agreed >> >> On Fri, 30 May 2025 at 15:18, Jarek Potiuk <ja...@potiuk.com> wrote: >> >> > Yeah, I think none of us want to just publish the code and be "done" >> with >> > it ... There is a real work to be done to make MCP much more useful and >> "AI >> > friendly" and the examples you gave Brian are cool, because I think we >> need >> > to (and this is mostly the ask from maintainers for the users to come >> and >> > participate in the design phase of what the MCP can do). >> > It looks like that "backbone" of the MCP and the glue between the REST >> API >> > and MCP can be done in a simple (and easy to maintain) way. But the >> added >> > value is indeed in figuring out what would be useful missing things - >> > starting from what broad use cases we want to address - and whether >> some >> > help for the agent can be described as prompts, better description of >> the >> > API and examples or maybe we need more aggregated, new APIs (maybe >> simply >> > new REST API calls we need) that will allow the agents to reason better >> and >> > faster. All of that is possible. >> > >> > J >> > >> > On Fri, May 30, 2025 at 11:29 AM Kaxil Naik <kaxiln...@gmail.com> >> wrote: >> > >> > > You can easily add as many tools you want: >> > > https://gofastmcp.com/servers/tools >> > > >> > > I would be surprised if there is a thing you can't do with FastMCP >> 2.0+ >> > > that you can do with the MCP Python SDK. >> > > >> > > Like I said earlier: This is a simplistic example :) but the gist is >> we >> > > should be using the newer abstractions which I am happy to comment >> during >> > > the development phase too. Like everything else we need to ensure >> > > maintainability is worth the value we create. >> > > >> > > >> > > >> > > On Fri, 30 May 2025 at 14:48, Kaxil Naik <kaxiln...@gmail.com> wrote: >> > > >> > > > Btw we don’t need to use FastMCP just for create MCP from OpenApi >> spec. >> > > > Many of you mighht already be aware - FastMCP 1.0 was adopted in the >> > > > official mcp python sdk since 1.2 and is recommended high-level >> server >> > > > framework. >> > > > >> > > > Check: >> > > > >> https://github.com/modelcontextprotocol/python-sdk/releases/tag/v1.2.0 >> > > > >> > > > @Bryan Coder: I will be surprised if you can’t do the use-case you >> > > > mentioned with FastMCP - either the one donated to MCP Python SDK or >> > > > FastMCP 2.0 - have you tried that? It isn’t just a wrapper! >> > > > >> > > > On Fri, 30 May 2025 at 13:16, Avi <a...@astronomer.io.invalid> >> wrote: >> > > > >> > > >> Yeah FastMCP is nice, I didn't select fast mcp for this specific >> > reason: >> > > >> - The sheer number of tools that are created using OpenAPI spec >> > doesn't >> > > >> need to be passed to AI every single message. >> > > >> - Instead, we can do a hierarchical tool discovery based on >> > categories. >> > > >> And >> > > >> let AI select a particular category and then get tools only for >> that >> > > >> category. >> > > >> >> > > >> python3 -c " >> > > >> > import json >> > > >> > with open('path/to/openapi.json') as f: >> > > >> > spec = json.load(f) >> > > >> > >> > > >> > tags = {} >> > > >> > for path, methods in spec['paths'].items(): >> > > >> > for method, details in methods.items(): >> > > >> > if 'tags' in details: >> > > >> > for tag in details['tags']: >> > > >> > tags[tag] = tags.get(tag, 0) + 1 >> > > >> > >> > > >> > print('Tags and their counts:') >> > > >> > for tag, count in sorted(tags.items(), key=lambda x: x[1], >> > > >> reverse=True): >> > > >> > print(f'{tag}: {count}') >> > > >> > " >> > > >> Tags and their counts: >> > > >> Task Instance: 19 >> > > >> Asset: 13 >> > > >> Connection: 8 >> > > >> DagRun: 8 >> > > >> Backfill: 7 >> > > >> DAG: 7 >> > > >> Pool: 6 >> > > >> Variable: 6 >> > > >> XCom: 4 >> > > >> Config: 2 >> > > >> Event Log: 2 >> > > >> Import Error: 2 >> > > >> Plugin: 2 >> > > >> Task: 2 >> > > >> DagVersion: 2 >> > > >> Login: 2 >> > > >> DagSource: 1 >> > > >> DagStats: 1 >> > > >> DagReport: 1 >> > > >> DagWarning: 1 >> > > >> Extra Links: 1 >> > > >> Job: 1 >> > > >> Provider: 1 >> > > >> DAG Parsing: 1 >> > > >> Monitor: 1 >> > > >> Version: 1 >> > > >> >> > > >> My last attempt to do a hierarchical discovery with FastMCP didn't >> go >> > as >> > > >> expected. >> > > >> But this could be short term. There is something cooking in the >> model >> > > >> context protocol repo for search of a tool. Ref: >> > > >> >> https://github.com/modelcontextprotocol/modelcontextprotocol/pull/322 >> > > >> >> > > >> I'll give this a try with FastMCP to see if I can get the >> > > >> hierarchical discovery working. >> > > >> >> > > >> - Avi >> > > >> >> > > >> On Fri, May 30, 2025 at 1:33 AM Bryan Corder < >> bryancor...@gmail.com> >> > > >> wrote: >> > > >> >> > > >> > In order to bring value, we might want to think beyond just >> wrapping >> > > the >> > > >> > API. As Kaxil just showed, it's easy to create something with 10 >> > lines >> > > >> of >> > > >> > code and FastMCP. >> > > >> > >> > > >> > However, the Airflow API was made for Airflow operators' >> > consumption, >> > > >> not >> > > >> > necessarily for LLM consumption. When you have an endpoint called >> > > >> "Delete >> > > >> > DAG" with a description "Delete a specific DAG" that's very easy >> for >> > > any >> > > >> > user who has already navigated to the Airflow API spec to >> > understand, >> > > >> but >> > > >> > maybe not the best tool description for an LLM. I think we'd >> want to >> > > >> either >> > > >> > exclude that or add additional context for the LLM to know it's >> > > >> > destructive. >> > > >> > >> > > >> > In addition, LLMs can struggle with tool selection when you give >> it >> > 80 >> > > >> > tools to work with. Things in the middle sometimes get lost in >> the >> > > >> context. >> > > >> > There are ways to customize the FastMCP ( >> > > >> > https://gofastmcp.com/servers/openapi#custom-route-maps) to cut >> > down >> > > >> the >> > > >> > list of options, should you choose. >> > > >> > >> > > >> > However, it may be better to create something more tailored to >> LLMs. >> > > >> > Thinking about the use case of getting LLM assistance with >> > debugging a >> > > >> > failed run, one of the things my teams do is put the "run book" >> for >> > > prod >> > > >> > support in the doc_md notes right with the DAG, so if a file >> never >> > > >> shows up >> > > >> > they know exactly what to do in that situation (potentially, do >> > > >> nothing). >> > > >> > We also include other information like, "xx task can be flaky. If >> > you >> > > >> get >> > > >> > this error, rerunning it will usually resolve it." The goal is >> for >> > any >> > > >> > engineer armed with the stack trace and the run book to be able >> to >> > > solve >> > > >> > any error. My team has all that information right in the UI. To >> get >> > > that >> > > >> > information, the LLM would need to know to hit the DAG Details >> > > endpoint >> > > >> for >> > > >> > one minor attribute amongst several for the doc_md and get the >> > correct >> > > >> dag >> > > >> > id, run id, task id and try number to grab the stack trace from >> the >> > > >> failed >> > > >> > run. It would then need to go elsewhere to find the DAG code to >> > > debug. I >> > > >> > think it would be better to just create a "debug_failed_task" >> tool >> > an >> > > >> LLM >> > > >> > could call from an MCP server that would string those calls >> together >> > > and >> > > >> > serve them up to the LLM on a silver platter. The LLM could focus >> > all >> > > >> its >> > > >> > "reasoning" efforts on solving the problem instead of figuring >> out >> > how >> > > >> to >> > > >> > get the information it needs to even begin. >> > > >> > >> > > >> > Again, if we just want to wrap the API in FastMCP, we can share >> > > Kaxil's >> > > >> 10 >> > > >> > lines of code in a Medium article and be done. I think the real >> > value >> > > >> is in >> > > >> > providing an implementation of a limited set of more complex base >> > > tools >> > > >> > like debug_failed_task (described above), pause_all_active_DAGs >> > > (because >> > > >> > I'm about to upgrade!), describe_DAG (grabs only the description, >> > > >> > dependencies, converts cron schedule to human readable if >> > applicable, >> > > >> etc) >> > > >> > and giving people a way to extend the server. >> > > >> > >> > > >> > The above is tool focused. As Avi pointed out, there are also >> > > resources >> > > >> and >> > > >> > prompts, but I've only personally worked with tools and have >> nothing >> > > to >> > > >> add >> > > >> > there. >> > > >> > >> > > >> > With all the LLM tools quickly advancing on the development side >> > (e.g. >> > > >> code >> > > >> > generation/review), it's great to see the community working on >> > > building >> > > >> > tools to help with the operational side. >> > > >> > >> > > >> > Bryan >> > > >> > >> > > >> > >> > > >> > On Thu, May 29, 2025, 4:50 PM Kaxil Naik <kaxiln...@gmail.com> >> > wrote: >> > > >> > >> > > >> > > One more comment: MCP SDKs have advanced quite a bit and I was >> > able >> > > to >> > > >> > get >> > > >> > > an Airflow MCP Server working with just the following code >> block. >> > I >> > > >> was >> > > >> > > successfully able to pause/unpause a DAG from Claude and other >> MCP >> > > >> client >> > > >> > > as an example. So as much as possible we should utilize higher >> > level >> > > >> > > abstraction like FastMCP which allows creating client from >> OpenAPI >> > > >> spec >> > > >> > > <https://gofastmcp.com/servers/openapi#openapi-integration>: >> > > >> > > >> > > >> > > import os >> > > >> > > >> > > >> > > import httpx >> > > >> > > from fastmcp import FastMCP >> > > >> > > >> > > >> > > token = os.environ.get("AF_ACCESS_TOKEN") >> > > >> > > client = httpx.AsyncClient( >> > > >> > > base_url="http://localhost:28080";, >> > > >> > > headers={"Authorization": f"Bearer {token}"}, >> > > >> > > ) >> > > >> > > >> > > >> > > openapi_spec = httpx.get(" >> http://localhost:28080/openapi.json >> > > >> > ").json() >> > > >> > > >> > > >> > > mcp = FastMCP.from_openapi( >> > > >> > > openapi_spec=openapi_spec, >> > > >> > > client=client, >> > > >> > > name="Airflow 3.0 API Server" >> > > >> > > ) >> > > >> > > >> > > >> > > if __name__ == "__main__": >> > > >> > > mcp.run() >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > On Thu, 29 May 2025 at 20:32, Avi <a...@astronomer.io.invalid> >> > > wrote: >> > > >> > > >> > > >> > > > @Shahar -- Yes. Definitely. Feel free to reachout if you need >> > > >> anything. >> > > >> > > > >> > > >> > > > I totally agree, it to live as a separate repo. >> > > >> > > > >> > > >> > > > - Avi >> > > >> > > > >> > > >> > > > On Thu, May 29, 2025 at 12:50 PM Kaxil Naik < >> > kaxiln...@gmail.com> >> > > >> > wrote: >> > > >> > > > >> > > >> > > > > @Shahar -- Absolutely, I think you are driving it with this >> > > email. >> > > >> > So I >> > > >> > > > > think you can lead it from here and whoever wants to join >> can >> > > >> co-lead >> > > >> > > or >> > > >> > > > > join in development. >> > > >> > > > > >> > > >> > > > > Please feel free to drive :) >> > > >> > > > > >> > > >> > > > > On Thu, 29 May 2025 at 17:07, Aaron Dantley < >> > > >> aarondant...@gmail.com> >> > > >> > > > > wrote: >> > > >> > > > > >> > > >> > > > > > Hey All! >> > > >> > > > > > >> > > >> > > > > > I’d be grateful to be included in the AIP discussions to >> > help >> > > if >> > > >> > > > possible >> > > >> > > > > > too! Like Shahar, I’ve never worked on any of these >> items so >> > > >> it’d >> > > >> > be >> > > >> > > > > great >> > > >> > > > > > to see how work gets assigned and goes through a whole >> > > >> development >> > > >> > > > cycle! >> > > >> > > > > > >> > > >> > > > > > Looking forward to it! >> > > >> > > > > > Aaron >> > > >> > > > > > >> > > >> > > > > > On Thu, May 29, 2025 at 7:32 AM Shahar Epstein < >> > > >> sha...@apache.org> >> > > >> > > > > wrote: >> > > >> > > > > > >> > > >> > > > > > > If it's ok, I would like to lead the AIP effort (or at >> > least >> > > >> > > > co-lead), >> > > >> > > > > as >> > > >> > > > > > > I've never written an AIP before. I could start >> drafting >> > it >> > > >> > during >> > > >> > > > the >> > > >> > > > > > next >> > > >> > > > > > > week. >> > > >> > > > > > > Avi - please let me know if it works for you. >> > > >> > > > > > > >> > > >> > > > > > > >> > > >> > > > > > > Shahar >> > > >> > > > > > > >> > > >> > > > > > > >> > > >> > > > > > > On Thu, May 29, 2025, 13:09 Kaxil Naik < >> > kaxiln...@gmail.com >> > > > >> > > >> > > wrote: >> > > >> > > > > > > >> > > >> > > > > > > > Yes separate repo, please and we would need someone >> to >> > > lead >> > > >> > this >> > > >> > > > > effort >> > > >> > > > > > > on >> > > >> > > > > > > > the proposal & development too. Avi - you are >> probably >> > > well >> > > >> > > > equipped >> > > >> > > > > to >> > > >> > > > > > > > lead it and I am sure more folks like Aaraon would be >> > > eager >> > > >> to >> > > >> > > work >> > > >> > > > > on >> > > >> > > > > > > its >> > > >> > > > > > > > development and on-going maintenance. >> > > >> > > > > > > > >> > > >> > > > > > > > Regards, >> > > >> > > > > > > > Kaxil >> > > >> > > > > > > > >> > > >> > > > > > > > On Thu, 29 May 2025 at 15:25, Jarek Potiuk < >> > > >> ja...@potiuk.com> >> > > >> > > > wrote: >> > > >> > > > > > > > >> > > >> > > > > > > > > Yep. Having MCP is cool and drawing our >> implementation >> > > >> from >> > > >> > > > > > experiences >> > > >> > > > > > > > and >> > > >> > > > > > > > > usage of other MCP servers out there is even cooler >> > > >> > (especially >> > > >> > > > > that >> > > >> > > > > > we >> > > >> > > > > > > > can >> > > >> > > > > > > > > have some insights how people already use them with >> > > >> Airflow) >> > > >> > - >> > > >> > > if >> > > >> > > > > we >> > > >> > > > > > > can >> > > >> > > > > > > > > bring together a few of those, put some nice, >> relevant >> > > >> > Airflow >> > > >> > > > > > prompts. >> > > >> > > > > > > > > Ideally we could have some examples of how MCP can >> be >> > > used >> > > >> > > taken >> > > >> > > > > from >> > > >> > > > > > > > those >> > > >> > > > > > > > > who are using airflow (the debugging example by >> Avi is >> > > >> cool) >> > > >> > > > > > > > > >> > > >> > > > > > > > > I am not sure implementing it as provider is really >> > "the >> > > >> way" >> > > >> > > > > though >> > > >> > > > > > - >> > > >> > > > > > > I >> > > >> > > > > > > > > would rather see `apache-airflow-mcp" separate >> repo - >> > > >> it's so >> > > >> > > > > > different >> > > >> > > > > > > > and >> > > >> > > > > > > > > distinct from airflow it does not really require >> any >> > of >> > > >> > Airflow >> > > >> > > > > > > internals >> > > >> > > > > > > > > and code to be implemented - it makes very little >> > sense >> > > >> to be >> > > >> > > the >> > > >> > > > > > part >> > > >> > > > > > > of >> > > >> > > > > > > > > airflow "workspace" where we would develop it >> together >> > > >> with >> > > >> > > > > airflow - >> > > >> > > > > > > > > because if it will talk over the REST api, all we >> need >> > > is >> > > >> the >> > > >> > > > > > `client` >> > > >> > > > > > > > that >> > > >> > > > > > > > > might be just a dependency. And there is even no >> > reason >> > > >> for >> > > >> > MCP >> > > >> > > > and >> > > >> > > > > > > > airflow >> > > >> > > > > > > > > to be installed and developed together (that's the >> > main >> > > >> > reason >> > > >> > > > why >> > > >> > > > > we >> > > >> > > > > > > > want >> > > >> > > > > > > > > providers to be kept in monorepo. >> > > >> > > > > > > > > >> > > >> > > > > > > > > J. >> > > >> > > > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > > On Thu, May 29, 2025 at 8:37 AM Amogh Desai < >> > > >> > > > > > amoghdesai....@gmail.com> >> > > >> > > > > > > > > wrote: >> > > >> > > > > > > > > >> > > >> > > > > > > > > > Seems like a promising area to invest in given >> the >> > > >> benefits >> > > >> > > it >> > > >> > > > > can >> > > >> > > > > > > > > provide >> > > >> > > > > > > > > > to >> > > >> > > > > > > > > > the users as mentioned by Shahar and Abhishek. >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > Abhishek also has a promising talk submitted >> which i >> > > am >> > > >> > > looking >> > > >> > > > > > > forward >> > > >> > > > > > > > > to >> > > >> > > > > > > > > > this year at the summit. >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > In any case, this seems to be one of the first of >> > the >> > > >> very >> > > >> > > few >> > > >> > > > > > > > > > implementations of trying >> > > >> > > > > > > > > > to integrate Airflow officially / unofficially >> with >> > an >> > > >> MCP >> > > >> > > > > server. >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > Thanks & Regards, >> > > >> > > > > > > > > > Amogh Desai >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > On Thu, May 29, 2025 at 2:56 AM Aaron Dantley < >> > > >> > > > > > > aarondant...@gmail.com> >> > > >> > > > > > > > > > wrote: >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > > Hey! >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > I also think this is a great idea! >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > Would it be possible to be included in the >> > > development >> > > >> > > > process? >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > Sorry I’m new to this group, but would >> appreciate >> > > any >> > > >> > > > > suggestions >> > > >> > > > > > > on >> > > >> > > > > > > > > how >> > > >> > > > > > > > > > to >> > > >> > > > > > > > > > > contribute to the MCP server development! >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > Regards! >> > > >> > > > > > > > > > > Aaron >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > On Wed, May 28, 2025 at 2:57 PM Avi >> > > >> > > > <a...@astronomer.io.invalid >> > > >> > > > > > >> > > >> > > > > > > > wrote: >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > > Nice to see the idea to incorporate an >> official >> > > MCP >> > > >> > > server >> > > >> > > > > for >> > > >> > > > > > > > > > > > Airflow. It's been really magical to see >> what a >> > > >> simple >> > > >> > > LLM >> > > >> > > > > can >> > > >> > > > > > do >> > > >> > > > > > > > > with >> > > >> > > > > > > > > > an >> > > >> > > > > > > > > > > > Airflow MCP server built just from APIs. >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > A few things that I noticed in my experience: >> > > >> > > > > > > > > > > > - The number of tools that the OpenAPI spec >> > > >> generates >> > > >> > is >> > > >> > > > > quite >> > > >> > > > > > > > huge. >> > > >> > > > > > > > > > Most >> > > >> > > > > > > > > > > > tools (*Claude, VS Code with GitHub Copilot, >> > > Cursor, >> > > >> > > > > Windsurf*) >> > > >> > > > > > > > which >> > > >> > > > > > > > > > > uses >> > > >> > > > > > > > > > > > mcp-client limits it to a number of 100 >> tools. >> > > (*The >> > > >> > > > > read-only >> > > >> > > > > > > mode >> > > >> > > > > > > > > > > creates >> > > >> > > > > > > > > > > > less tools in comparison*.) >> > > >> > > > > > > > > > > > - MCP server are just not tools. There are >> other >> > > >> things >> > > >> > > as >> > > >> > > > > > well, >> > > >> > > > > > > > like >> > > >> > > > > > > > > > > > resources and prompts. Prompts are super >> helpful >> > > in >> > > >> > case >> > > >> > > of >> > > >> > > > > > > > debugging >> > > >> > > > > > > > > > for >> > > >> > > > > > > > > > > > example. It is a way of teaching LLM about >> > > Airflow. >> > > >> > Say I >> > > >> > > > > want >> > > >> > > > > > to >> > > >> > > > > > > > > have >> > > >> > > > > > > > > > a >> > > >> > > > > > > > > > > > failing task investigated. A prompt can be >> > helpful >> > > >> in >> > > >> > > > letting >> > > >> > > > > > LLM >> > > >> > > > > > > > > know >> > > >> > > > > > > > > > a >> > > >> > > > > > > > > > > > step-by-step process of carrying out the >> > > >> investigation. >> > > >> > > > > > > > > > > > - Where do you run the MCP server? I wouldn't >> > want >> > > >> my >> > > >> > > > laptop >> > > >> > > > > to >> > > >> > > > > > > do >> > > >> > > > > > > > > the >> > > >> > > > > > > > > > > > heavy processing, which would want us to go >> for >> > > the >> > > >> SSE >> > > >> > > > > instead >> > > >> > > > > > > of >> > > >> > > > > > > > > > stdio. >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > This is why I chose two different path of >> using >> > > mcp >> > > >> > > server >> > > >> > > > > with >> > > >> > > > > > > > > > airflow, >> > > >> > > > > > > > > > > > which I intend to talk about at the summit. >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > 1. AI-Augmented Airflow - This helped me add >> a >> > > chat >> > > >> > > > interface >> > > >> > > > > > > > inside >> > > >> > > > > > > > > > > > Airflow using a plugin to talk to an Airflow >> > > >> instance >> > > >> > > (read >> > > >> > > > > > only >> > > >> > > > > > > > > mode). >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > 2. Airflow-Powered AI - Experimenting with >> this >> > > has >> > > >> > been >> > > >> > > > > > totally >> > > >> > > > > > > > > > magical, >> > > >> > > > > > > > > > > > how powerful AI can become when it has >> access to >> > > >> > airflow. >> > > >> > > > > > Also, a >> > > >> > > > > > > > > > > directory >> > > >> > > > > > > > > > > > structure to maintain the DAGs, and it can >> write >> > > >> DAGs >> > > >> > on >> > > >> > > > the >> > > >> > > > > > > fly. I >> > > >> > > > > > > > > > > totally >> > > >> > > > > > > > > > > > see a need where LLMs eventually will need a >> > > >> scheduler, >> > > >> > > > > > although >> > > >> > > > > > > a >> > > >> > > > > > > > > > > complete >> > > >> > > > > > > > > > > > airflow just for an LLM might seem a bit >> > overkill >> > > to >> > > >> > the >> > > >> > > > rest >> > > >> > > > > > of >> > > >> > > > > > > > the >> > > >> > > > > > > > > > > > community. >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > I chose to build this on top of open API is >> > > because >> > > >> > that >> > > >> > > > was >> > > >> > > > > > the >> > > >> > > > > > > > only >> > > >> > > > > > > > > > way >> > > >> > > > > > > > > > > > to get proper RBAC enabled. >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > I have so many points to discuss. Would love >> to >> > > hear >> > > >> > from >> > > >> > > > the >> > > >> > > > > > > > > community >> > > >> > > > > > > > > > > and >> > > >> > > > > > > > > > > > then take it forward. >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > Thanks, >> > > >> > > > > > > > > > > > Avi >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > On Wed, May 28, 2025 at 6:32 PM Aritra Basu < >> > > >> > > > > > > > > aritrabasu1...@gmail.com> >> > > >> > > > > > > > > > > > wrote: >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > > I definitely think there's potential to >> > interact >> > > >> with >> > > >> > > an >> > > >> > > > > > > airflow >> > > >> > > > > > > > > MCP >> > > >> > > > > > > > > > > > > server. Though I think I'd be interested to >> > see >> > > >> how >> > > >> > > many >> > > >> > > > > and >> > > >> > > > > > > how >> > > >> > > > > > > > > > > > frequently >> > > >> > > > > > > > > > > > > people are making use of MCP servers in the >> > wild >> > > >> > before >> > > >> > > > > > > investing >> > > >> > > > > > > > > > > effort >> > > >> > > > > > > > > > > > in >> > > >> > > > > > > > > > > > > building and maintaining one for airflow. >> I'm >> > > sure >> > > >> > the >> > > >> > > > data >> > > >> > > > > > is >> > > >> > > > > > > > > > > available >> > > >> > > > > > > > > > > > > out there, just needs finding. >> > > >> > > > > > > > > > > > > -- >> > > >> > > > > > > > > > > > > Regards, >> > > >> > > > > > > > > > > > > Aritra Basu >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > On Wed, 28 May 2025, 11:18 pm Julian >> LaNeve, >> > > >> > > > > > > > > > > > <jul...@astronomer.io.invalid >> > > >> > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > wrote: >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > I think this would be interesting now >> that >> > the >> > > >> > > > Streamable >> > > >> > > > > > > HTTP >> > > >> > > > > > > > > > spec < >> > > >> > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > >> > > >> > > > > > > >> > > >> > > > > > >> > > >> > > > > >> > > >> > > > >> > > >> > > >> > > >> > >> > > >> >> > > >> > >> https://modelcontextprotocol.io/specification/2025-03-26/basic/transports >> > > >> > >> > > >> > > > > > > > > > > > > > is out. I think in theory we could >> publish >> > > this >> > > >> > first >> > > >> > > > as >> > > >> > > > > an >> > > >> > > > > > > > > Airflow >> > > >> > > > > > > > > > > > > > provider that installs a plugin to >> expose an >> > > MCP >> > > >> > > > > endpoint, >> > > >> > > > > > > as a >> > > >> > > > > > > > > > PoC - >> > > >> > > > > > > > > > > > > this >> > > >> > > > > > > > > > > > > > becomes a much nicer experience than a >> local >> > > >> stdio >> > > >> > > one. >> > > >> > > > > > > > > > > > > > -- >> > > >> > > > > > > > > > > > > > Julian LaNeve >> > > >> > > > > > > > > > > > > > CTO >> > > >> > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > Email: jul...@astronomer.io >> > > >> > > > > > > > > > > > > > <mailto:jul...@astronomer.io>Mobile: >> 330 >> > 509 >> > > >> 5792 >> > > >> > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > On May 28, 2025, at 1:25 PM, Shahar >> > Epstein >> > > < >> > > >> > > > > > > > sha...@apache.org >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > > > wrote: >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > Dear community, >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > Following the thread on Slack [1], >> > initiated >> > > >> by >> > > >> > > Jason >> > > >> > > > > > > > Sebastian >> > > >> > > > > > > > > > > > Kusuma, >> > > >> > > > > > > > > > > > > > I'd >> > > >> > > > > > > > > > > > > > > like to start an effort to officially >> > > support >> > > >> MCP >> > > >> > > in >> > > >> > > > > > > > Airflow's >> > > >> > > > > > > > > > > > > codebase. >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > *Some background * >> > > >> > > > > > > > > > > > > > > Model Context Protocol (MCP) is an open >> > > >> standard, >> > > >> > > > > > > open-source >> > > >> > > > > > > > > > > > framework >> > > >> > > > > > > > > > > > > > > that standardizes the way AI models >> like >> > LLM >> > > >> > > > integrate >> > > >> > > > > > and >> > > >> > > > > > > > > share >> > > >> > > > > > > > > > > data >> > > >> > > > > > > > > > > > > > with >> > > >> > > > > > > > > > > > > > > external tools, systems and data >> sources. >> > > >> Think >> > > >> > of >> > > >> > > it >> > > >> > > > > as >> > > >> > > > > > a >> > > >> > > > > > > > > "USB-C >> > > >> > > > > > > > > > > for >> > > >> > > > > > > > > > > > > > AI" - >> > > >> > > > > > > > > > > > > > > a universal connector that simplifies >> and >> > > >> > > > standardizes >> > > >> > > > > AI >> > > >> > > > > > > > > > > > > integrations. A >> > > >> > > > > > > > > > > > > > > notable example of an MCP server is >> > GitHub's >> > > >> > > official >> > > >> > > > > > > > > > > implementation >> > > >> > > > > > > > > > > > > > [3], which >> > > >> > > > > > > > > > > > > > > allows LLMs such as Claude, Copilot, >> and >> > > >> OpenAI >> > > >> > > (or: >> > > >> > > > > "MCP >> > > >> > > > > > > > > > clients") >> > > >> > > > > > > > > > > > to >> > > >> > > > > > > > > > > > > > > fetch pull request details, analyze >> code >> > > >> changes, >> > > >> > > and >> > > >> > > > > > > > generate >> > > >> > > > > > > > > > > review >> > > >> > > > > > > > > > > > > > > summaries. >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > *How could an MCP server be useful in >> > > >> Airflow?* >> > > >> > > > > > > > > > > > > > > Imagine the possibilities when LLMs can >> > > >> > seamlessly >> > > >> > > > > > interact >> > > >> > > > > > > > > with >> > > >> > > > > > > > > > > > > > Airflow’s >> > > >> > > > > > > > > > > > > > > API: triggering DAGs using natural >> > language, >> > > >> > > > retrieving >> > > >> > > > > > DAG >> > > >> > > > > > > > run >> > > >> > > > > > > > > > > > > history, >> > > >> > > > > > > > > > > > > > > enabling smart debugging, and more. >> This >> > > kind >> > > >> of >> > > >> > > > > > > integration >> > > >> > > > > > > > > > opens >> > > >> > > > > > > > > > > > the >> > > >> > > > > > > > > > > > > > door >> > > >> > > > > > > > > > > > > > > to a more intuitive, conversational >> > > interface >> > > >> for >> > > >> > > > > > workflow >> > > >> > > > > > > > > > > > > orchestration. >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > *Why do we need to support it >> officially?* >> > > >> > > > > > > > > > > > > > > Quid pro quo - LLMs become an integral >> > part >> > > of >> > > >> > the >> > > >> > > > > modern >> > > >> > > > > > > > > > > development >> > > >> > > > > > > > > > > > > > > experience, while Airflow evolves into >> the >> > > >> go-to >> > > >> > > for >> > > >> > > > > > > > > > orchestrating >> > > >> > > > > > > > > > > AI >> > > >> > > > > > > > > > > > > > > workflows. By officially supporting it, >> > > we’ll >> > > >> > > enable >> > > >> > > > > > > multiple >> > > >> > > > > > > > > > users >> > > >> > > > > > > > > > > > to >> > > >> > > > > > > > > > > > > > > interact with Airflow through their >> LLMs, >> > > >> > > > streamlining >> > > >> > > > > > > > > automation >> > > >> > > > > > > > > > > and >> > > >> > > > > > > > > > > > > > > improving accessibility across diverse >> > > >> workflows. >> > > >> > > All >> > > >> > > > > of >> > > >> > > > > > > that >> > > >> > > > > > > > > is >> > > >> > > > > > > > > > > > viable >> > > >> > > > > > > > > > > > > > > with relatively small development >> effort >> > > (see >> > > >> > next >> > > >> > > > > > > > paragraph). >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > *How should it be implemented?* >> > > >> > > > > > > > > > > > > > > As of today, there have been several >> > > >> > > implementations >> > > >> > > > of >> > > >> > > > > > MCP >> > > >> > > > > > > > > > servers >> > > >> > > > > > > > > > > > for >> > > >> > > > > > > > > > > > > > > Airflow API, the most visible one [4] >> made >> > > by >> > > >> > > > Abhishek >> > > >> > > > > > > Bhakat >> > > >> > > > > > > > > > from >> > > >> > > > > > > > > > > > > > > Astronomer. >> > > >> > > > > > > > > > > > > > > The efforts of implementing it and >> > > >> maintaining it >> > > >> > > in >> > > >> > > > > our >> > > >> > > > > > > > > codebase >> > > >> > > > > > > > > > > > > > shouldn't >> > > >> > > > > > > > > > > > > > > be too cumbersome (at least in >> theory), as >> > > we >> > > >> > could >> > > >> > > > > > utilize >> > > >> > > > > > > > > > > packages >> > > >> > > > > > > > > > > > > like >> > > >> > > > > > > > > > > > > > > fastmcp to auto-generate the server >> using >> > > the >> > > >> > > > existing >> > > >> > > > > > > > OpenAPI >> > > >> > > > > > > > > > > specs. >> > > >> > > > > > > > > > > > > I'd >> > > >> > > > > > > > > > > > > > > be very happy if Abhishek could share >> his >> > > >> > > experience >> > > >> > > > in >> > > >> > > > > > > this >> > > >> > > > > > > > > > > thread. >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > *Where else could we utilize MCP?* >> > > >> > > > > > > > > > > > > > > Beyond the scope of the public API, I >> > could >> > > >> also >> > > >> > > > > imagine >> > > >> > > > > > > > using >> > > >> > > > > > > > > it >> > > >> > > > > > > > > > > to >> > > >> > > > > > > > > > > > > > > communicate with Breeze. >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > *How do we proceed from here?* >> > > >> > > > > > > > > > > > > > > Feel free to share your thoughts here >> in >> > > this >> > > >> > > > > discussion. >> > > >> > > > > > > > > > > > > > > If there are no objections, I'll be >> happy >> > to >> > > >> > start >> > > >> > > > > > working >> > > >> > > > > > > on >> > > >> > > > > > > > > an >> > > >> > > > > > > > > > > AIP. >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > Sincerely, >> > > >> > > > > > > > > > > > > > > Shahar Epstein >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > *References:* >> > > >> > > > > > > > > > > > > > > [1] Slack discussion, >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > > >> > > >> > > > > >> > > >> > > >> > > >> >> > https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1746121916951569 >> > > >> > > > > > > > > > > > > > > [2] Introducing the model context >> > protocol, >> > > >> > > > > > > > > > > > > > > >> > > >> > > > https://www.anthropic.com/news/model-context-protocol >> > > >> > > > > > > > > > > > > > > [3] GitHub Official MCP server, >> > > >> > > > > > > > > > > > > > >> https://github.com/github/github-mcp-server >> > > >> > > > > > > > > > > > > > > [4] Unofficial MCP Server made by >> Abhishek >> > > >> Hakat, >> > > >> > > > > > > > > > > > > > > >> > > >> > > https://github.com/abhishekbhakat/airflow-mcp-server >> > > >> > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > >> > > >> > > > > > > >> > > >> > > > > > >> > > >> > > > > >> > > >> > > > >> > > >> > > >> > > >> > >> > > >> >> > > > >> > > >> > >> >
