Subrepo has another disadvantage - namely that any "sync" of the subrepo is a full "snapshot" of all changed there - and the files end up as regular files in the target repo. Those files are physically committed files in Airflow repo, and subrepo is not "official" git feature - it's an external package https://github.com/ingydotnet/git-subrepo. I think subrepo is grep if you want to "vendor-in" part of an external "dependent" repository—one that you plan to migrate to the next version, where a submodule is much better for "WIP," especially if none of your code actually depends on whether the submodule is checked out or not.
Our only dependency on "steward" now is that symlinks to "skills" will be missing if the submodule is not updated. So ... skills in Claude won't work in this case—that's the "default" stage for people who do not use skills. No big deal (because they don't use them) On the other hand if some wants to start using skills this is a question of two commands (explained in setup): git submodule update --recursive Followed by (in Agent) /setup-apache-steward And the latter command will add a "post-merge" hook that removes the need of manually updating the submodule.Once setup is complete, it should be totally transparent. Of course - this is theory, and with a little practice (it seems to work), there will be some edge cases. But I would rather learn about those edge cases and fix them :) J On Sat, May 2, 2026 at 9:25 PM Shahar Epstein <[email protected]> wrote: > Thanks Jarek for working on it! > Confirming a smooth experience with "Steward" - some setup > shenanigans, but we figured it out eventually :) > > Overall the PR looks solid. However, I'm concerned regarding the usage > of submodules in Airflow's main repository. Maintaining a submodule > over time, as well as removing a submodule from the repo, are not > trivial for the average contributor and people could easily get > troubled with it. AI can surely solve everything related with a single > prompt - but still, I have to be the devil's advocate here. > Is it possible to make the transition to subrepo/automation before merging? > > > Shahar > > On Sat, May 2, 2026 at 8:39 PM Jarek Potiuk <[email protected]> wrote: > > > > Hi everyone, > > > > I am moving Airflow's "maintainer skill" rules to the shared "steward" > > project, which will serve as the base for the future ASF-wide initiative. > > For those curious, I have already ported the skills for security > workflows, > > PR triage, and review to the steward repository: > > https://github.com/apache/airflow-steward/tree/main/.claude/skills. > > > > I propose wiring these skills into Airflow via a Git submodule. While we > > have had past difficulties with submodules, this approach is justified > here: > > > > > > - Users who do not utilize agentic skills will be unaffected. > > - We developed a skill to automate submodule updates, addressing > > previous pain points. > > - This removes all skill-related code and almost all documentation > > (except bootstrapping) from the main Airflow repository. Once we > switch, > > and make sure all "airflow-specific" things are in Airflow, further > > "steward" work should happen outside of Airflow repo - no more 1000+ > > English markdown PRs in Airflow :). > > > > > > We have a similar setup for our security project, "airflow-s." I am > > applying feedback from Shahar; we previously collaborated on some > > security-management related work using it, and I recall it being a smooth > > experience (Shahar, please confirm). > > > > I would appreciate (positive :)) reviews on this prerequisite PR: > > https://github.com/apache/airflow/pull/66283. Once merged, I will > follow up > > by replacing the current skills with symbolic links to the submodule. We > > can transition to a subrepo or automated extraction once the "steward (to > > be renamed)" project stabilizes. > > > > Best, > > Jarek > > > > On Wed, Apr 29, 2026 at 11:37 AM Jarek Potiuk <[email protected]> wrote: > > > > > Hi everyone, > > > > > > I would like to provide an update regarding the recent > "airflow-steward" > > > and "airflow-s" messages you may have seen. I mistakenly configured a > new > > > repository to send notifications to the dev@ list, unintentionally > > > sharing some early-stage details a bit sooner than I thought. > > > > > > I have been working at an accelerated pace (pretty much 150% of my time > > > over the last one-two months) due to an emerging security "crisis" in > the > > > industry. New AI models, such as Anthropic’s Mythos (which resulted in > the > > > recently announced cross-industry Project Glasswing [1]), have become > > > exceptionally fast at identifying and chaining security > vulnerabilities. > > > This has forced a shift in how we handle security, moving from reactive > > > responses to proactive scanning. We expect an exponential growth in > > > high-quality security reports, and we are preparing to catch these > issues > > > during the development process—before code is even released. > > > > > > Last month, the ASF announced the Responsible AI Initiative [2], > supported > > > by Anthropic and Alpha-Omega. I am currently leading a proposal to > > > transition our recent work into a new "Apache Steward" PMC, which is > > > expected to be discussed at the next ASF board meeting on May 20th. > > > > > > Because of the urgency, I have had to make some rapid decisions and > move > > > faster than our usual PMC pace. My goal is to ensure Apache Steward > helps > > > the open-source community navigate this transition successfully. For > > > Airflow (and other projects), this starts with self-cleaning of > security > > > issues first. It very likely means a renewed focus on the "Airflow > Beach > > > Cleaning" [3] initiative to prune historical baggage and manage > > > dependencies more effectively using new agentic tooling (it had > previously > > > hit some scaling roadblocks - that are now unblocked with Agentic AI). > > > > > > We now have Airflow Steward [4] as the first stage of the "extracting > > > tooling" - this will likely become "Apache Steward" - if all stars > align. > > > > > > I apologize for the hectic communication. I will continue to keep > everyone > > > updated as things progress. If you have any specific questions, please > feel > > > free to ask here. We can keep that thread to keep everyone updated. > > > > > > Best regards, > > > > > > Jarek Potiuk > > > > > > [1] Project Glasswing https://www.anthropic.com/glasswing > > > [2] Responsible AI Initiative of ASF - > > > > https://news.apache.org/foundation/entry/the-apache-software-foundation-launches-10m-responsible-ai-initiative-with-initial-1-75m-donation > > > [3] Airflow Beach Cleaning Keynote from Airflow Summit 2024 - > > > https://www.youtube.com/watch?v=f6gfoVJXWEE > > > [4] Airflow Steward - https://github.com/apache/airflow-steward > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
