Actually... After your message, Shahar, I woke up today with a much better idea :D.
I figured out that with agentic CLIs - we don't have to do subrepo or submodule..... We can have literally 1 SKILL to copy to "airiflow'` SKILL directory: `Install and manage "Apache Steward" :). No subrepot, No submodule, simply instructions on how to locally copy the "steward" from either released SVN or Github repo (tag or branch), adding .gitignore. This will be even better, and likely much more acceptable for projects that do not want to use submodules. When you want to modify "steward" behaviour, we will have protection so that you cannot modify the gitignored steward. Instead you will add local "modification" of the flow locally (which will not be .gitignored), and when you would like to contribute it back you will have to ask agent to turn your local modification into a proper PR to the repo. It's really magical with agents when you do not have to limit yourself to "common tools" (like submodule) - but you can simply describe your intentions and let the agent execute those :) I will close the PR and will follow that path :) On Sat, May 2, 2026 at 11:02 PM Shahar Epstein <[email protected]> wrote: > Yeah well - I forgot that there's the option of not installing the > submodule if you don't use it, and then there's no problem for most cases > (we could indeed handle the rest, with some AI skill and guidance in the > docs). > In any case it should be better than subrepo, according to your > description. > Let's give it a shot then this way. > > > Shahar > > On Sat, May 2, 2026, 22:43 Jarek Potiuk <[email protected]> wrote: > > > Subrepo has another disadvantage - namely that any "sync" of the subrepo > is > > a full "snapshot" of all changed there - and the files end up as regular > > files in the target repo. Those files are physically committed files in > > Airflow repo, and subrepo is not "official" git feature - it's an > external > > package https://github.com/ingydotnet/git-subrepo. I think subrepo is > grep > > if you want to "vendor-in" part of an external "dependent" repository—one > > that you plan to migrate to the next version, where a submodule is much > > better for "WIP," especially if none of your code actually depends on > > whether the submodule is checked out or not. > > > > Our only dependency on "steward" now is that symlinks to "skills" will be > > missing if the submodule is not updated. > > > > So ... skills in Claude won't work in this case—that's the "default" > stage > > for people who do not use skills. No big deal (because they don't use > them) > > > > On the other hand if some wants to start using skills this is a question > of > > two commands (explained in setup): > > > > > > git submodule update --recursive > > > > Followed by (in Agent) > > > > /setup-apache-steward > > > > And the latter command will add a "post-merge" hook that removes the need > > of manually updating the submodule.Once setup is complete, it should be > > totally transparent. > > > > Of course - this is theory, and with a little practice (it seems to > work), > > there will be some edge cases. But I would rather learn about those edge > > cases and fix them :) > > > > J > > > > > > > > > > > > > > On Sat, May 2, 2026 at 9:25 PM Shahar Epstein <[email protected]> wrote: > > > > > Thanks Jarek for working on it! > > > Confirming a smooth experience with "Steward" - some setup > > > shenanigans, but we figured it out eventually :) > > > > > > Overall the PR looks solid. However, I'm concerned regarding the usage > > > of submodules in Airflow's main repository. Maintaining a submodule > > > over time, as well as removing a submodule from the repo, are not > > > trivial for the average contributor and people could easily get > > > troubled with it. AI can surely solve everything related with a single > > > prompt - but still, I have to be the devil's advocate here. > > > Is it possible to make the transition to subrepo/automation before > > merging? > > > > > > > > > Shahar > > > > > > On Sat, May 2, 2026 at 8:39 PM Jarek Potiuk <[email protected]> wrote: > > > > > > > > Hi everyone, > > > > > > > > I am moving Airflow's "maintainer skill" rules to the shared > "steward" > > > > project, which will serve as the base for the future ASF-wide > > initiative. > > > > For those curious, I have already ported the skills for security > > > workflows, > > > > PR triage, and review to the steward repository: > > > > https://github.com/apache/airflow-steward/tree/main/.claude/skills. > > > > > > > > I propose wiring these skills into Airflow via a Git submodule. While > > we > > > > have had past difficulties with submodules, this approach is > justified > > > here: > > > > > > > > > > > > - Users who do not utilize agentic skills will be unaffected. > > > > - We developed a skill to automate submodule updates, addressing > > > > previous pain points. > > > > - This removes all skill-related code and almost all documentation > > > > (except bootstrapping) from the main Airflow repository. Once we > > > switch, > > > > and make sure all "airflow-specific" things are in Airflow, > further > > > > "steward" work should happen outside of Airflow repo - no more > 1000+ > > > > English markdown PRs in Airflow :). > > > > > > > > > > > > We have a similar setup for our security project, "airflow-s." I am > > > > applying feedback from Shahar; we previously collaborated on some > > > > security-management related work using it, and I recall it being a > > smooth > > > > experience (Shahar, please confirm). > > > > > > > > I would appreciate (positive :)) reviews on this prerequisite PR: > > > > https://github.com/apache/airflow/pull/66283. Once merged, I will > > > follow up > > > > by replacing the current skills with symbolic links to the submodule. > > We > > > > can transition to a subrepo or automated extraction once the "steward > > (to > > > > be renamed)" project stabilizes. > > > > > > > > Best, > > > > Jarek > > > > > > > > On Wed, Apr 29, 2026 at 11:37 AM Jarek Potiuk <[email protected]> > > wrote: > > > > > > > > > Hi everyone, > > > > > > > > > > I would like to provide an update regarding the recent > > > "airflow-steward" > > > > > and "airflow-s" messages you may have seen. I mistakenly > configured a > > > new > > > > > repository to send notifications to the dev@ list, unintentionally > > > > > sharing some early-stage details a bit sooner than I thought. > > > > > > > > > > I have been working at an accelerated pace (pretty much 150% of my > > time > > > > > over the last one-two months) due to an emerging security "crisis" > in > > > the > > > > > industry. New AI models, such as Anthropic’s Mythos (which resulted > > in > > > the > > > > > recently announced cross-industry Project Glasswing [1]), have > become > > > > > exceptionally fast at identifying and chaining security > > > vulnerabilities. > > > > > This has forced a shift in how we handle security, moving from > > reactive > > > > > responses to proactive scanning. We expect an exponential growth in > > > > > high-quality security reports, and we are preparing to catch these > > > issues > > > > > during the development process—before code is even released. > > > > > > > > > > Last month, the ASF announced the Responsible AI Initiative [2], > > > supported > > > > > by Anthropic and Alpha-Omega. I am currently leading a proposal to > > > > > transition our recent work into a new "Apache Steward" PMC, which > is > > > > > expected to be discussed at the next ASF board meeting on May 20th. > > > > > > > > > > Because of the urgency, I have had to make some rapid decisions and > > > move > > > > > faster than our usual PMC pace. My goal is to ensure Apache Steward > > > helps > > > > > the open-source community navigate this transition successfully. > For > > > > > Airflow (and other projects), this starts with self-cleaning of > > > security > > > > > issues first. It very likely means a renewed focus on the "Airflow > > > Beach > > > > > Cleaning" [3] initiative to prune historical baggage and manage > > > > > dependencies more effectively using new agentic tooling (it had > > > previously > > > > > hit some scaling roadblocks - that are now unblocked with Agentic > > AI). > > > > > > > > > > We now have Airflow Steward [4] as the first stage of the > "extracting > > > > > tooling" - this will likely become "Apache Steward" - if all stars > > > align. > > > > > > > > > > I apologize for the hectic communication. I will continue to keep > > > everyone > > > > > updated as things progress. If you have any specific questions, > > please > > > feel > > > > > free to ask here. We can keep that thread to keep everyone updated. > > > > > > > > > > Best regards, > > > > > > > > > > Jarek Potiuk > > > > > > > > > > [1] Project Glasswing https://www.anthropic.com/glasswing > > > > > [2] Responsible AI Initiative of ASF - > > > > > > > > > > > https://news.apache.org/foundation/entry/the-apache-software-foundation-launches-10m-responsible-ai-initiative-with-initial-1-75m-donation > > > > > [3] Airflow Beach Cleaning Keynote from Airflow Summit 2024 - > > > > > https://www.youtube.com/watch?v=f6gfoVJXWEE > > > > > [4] Airflow Steward - https://github.com/apache/airflow-steward > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > >
