> I'm not sure if this is a valid approach, but if we decide to keep the package-lock.json, we could include a note in all release blog posts or announcements suggesting that anyone using the source code package run the npm upgrade command to resolve any potential issues in the lock file, since the lock file is intended for development use only and may become stale over time.
I am a bit less concerned for the "npm package" publishing and it's not really our case - in our case we don't publish our package.lock as part of the NPM package and we don't publish it either as part of the .whl files (or at least we should not). What we publish in our .whl files (Which are convenience packages built from sources) is literally the transpiled and minified "dist" javascript code. We do not need, nor use the .lock file in the final .whl package. We do - of course - keep the package.lock in the "source" packages though - both "-sources.tar.gz" which is the official release of ASF, as well as in .sdist python packages that we publish in PyPI - for those who prefer to rebuild airflow from sources (mainly distros). But I am very little concerned about this crowd to be honest. Since those people already have working build environment and actually "run" the build, it's really the matter of proper instructions for them how to "refresh" the package.lock before actual build. That's very simple and straightforward, and I think it's even GOOD to keep those lock files in the sources as the "source of truth at release time" - but ONLY for that - and include the refresh of the lock as an official step in instructions on how to rebuild your packages from sources. What I am really after is how to make easier to "upgrade" those npm dependencies for people who moslty consume the .whl files. J,. On Sat, Nov 1, 2025 at 10:55 AM Bryan Ellis <[email protected]> wrote: > Any thoughts on deleting the package-lock.json during the preparation of > the release package, so that it's not included in the source code package? > > I think keeping the package-lock.json is important in the main and other > long-lived branches for development purposes, but I'm still a bit concerned > about it remaining in the source code package. > > The main concern is that if it stays in the package and contains a pinned > vulnerable sub-dependency, we'd need to address that issue as soon as > possible. Meanwhile, for end-users who would install the package through > the npm registry (the preferred method) would have received the patched > sub-dependency would already have been patched. In this case, the current > published package would have zero vulnerabilities as there was no lock > file. > > Just to make note, for end-users that had already installed the package, > they would need to run npm upgrade or possibly just reinstall the package. > > Preparing a new source code package would then require bumping the version > number. But what's the point of publishing a new version to the npm > registry if it only includes a version bump and no actual code changes? Not > saying this is the case all the time but there are cases where packages are > pretty stable and do not actively receive any code changes to promote a > release. > > I have an idea on how to remove the package-lock.json while keeping it in > the main and long-lived branches, but it would result in an orphaned > commit. Might increase steps for release and probably make automation > harder. > > > In which case our response will be > > "here is the link that explains how you can do it yourself, we never > > release old versions because dependencies that were "latest" at the > moment > > of release have some vulnerabilities > > I'm not sure if this is a valid approach, but if we decide to keep the > package-lock.json, we could include a note in all release blog posts or > announcements suggesting that anyone using the source code package run the > npm upgrade command to resolve any potential issues in the lock file, since > the lock file is intended for development use only and may become stale > over time. > > > On Sat, Nov 1, 2025 at 2:32 AM Jarek Potiuk <[email protected]> wrote: > > > That is a very good set of questions - and I am glad we start discussing > > it, > > > > I had a lot of thoughts about that recently - because of Airflow NPM > > dependencies behaviour when comparing to - for example Python. I think we > > largely figured good aproach for it in Python > > > > *Tl;DR; NPM and lock mechanism is a bit problematic for us and I think > even > > for convenience packages we look for ways to give our users ways to > update > > their dependencies for released convenience packages.* > > > > *First a bit of context - how we do Python (and convenience images) in > > Airflow:* > > > > Let me explain how we do it in Airflow for Python, and how I am thinking > we > > can do something similar for NPM. > > > > For Airflow Python we almost always use lower-bound but non-upper-bound > > dependencies - i.e we specify "pytorch>x.y.a" but we very, very rarely > > (only when we have a good reason do `sqlalchemy>x.y.z,<2` - when we > > **know** we are not compatible with Sqlalchemy 2. This usually happens > when > > our dependencies have a well defined release process, and usually > > semver-like guarantees, and it also means that 1.x has patchlevel > releases > > with security fixes we can depend on. Whenever we upper-bound a > dependency > > we make a comment why, we open an issue describing it and add link to the > > issue to the comment and the issue explains conditions and what is > blocking > > us from removing the upper-binding (and we periodically review and > remove > > those when conditions are met or even run a project to do so - like it > > happens at the very moment with sqlalchemy 2 upgrade: [1] > > > > This means that our users are usually absolutely free to upgrade the > > dependencies (after sufficient testing) - we are **not** blocking them. > > This is a very strong policy we have and the main idea is that when we > > release our software - applying any "future" fixes to CVEs in our > > dependencies might heppen in two ways: > > > > * our future release will include those fixed versions (automatically as > we > > do not upper-bind and we are testing latest versions and highest > > resolutions of our dependencies in canary build 8 times a day to make > sure > > we fix all incompatibilities > > * USER is responsible to upgrade and test older releases and get those > > dependencies upgraded > > > > Even in CRA - we are steward, not manufacturer - so it's manufacturers > > responsibility to "put software on the market without known, exploitable > > vulnerabilities" - so it's on THEM not us to make sure whenever a > > vulnerability is found in one of our deps, to upgrade those. > > > > There are some nuances - of course any **new** version of such > upper-bound > > dependency might break airflow - that's why we produce a "golden set" of > > dependencies that we knew as "working" during the release - and we > publish > > them as "constraints" that we recommend our users to use when they want > to > > install released airflow version in a reproducible way. Then, after they > do > > it, and if they want to upgrade to a new version of dependency that has > > fixed a vulnerability they can do it "on top of that". The first > paragraph > > of our "installing from PyPI" describes this process here [2] > > > > Effectively it means that the users of ours have two choices: > > > > * if they want to upgrade dependencies in released software - in vast > > majority of cases we are not blocking them from that and it's on them to > > make sure it happens > > * or they can wait until we release a new version of Airflow with (pretty > > much automatically) those deps upgraded by us > > > > There are edge-cases of course that are handled individually but those > are > > exceptions rather than a rule even if in Airflow we have in total 750+ > > dependencies - it works nicely with this setup if you ask me. And it not > > only works for "sources" we release but also for convenience packages > > (whl/sdist) that we also publish. > > > > *How is NPM different ?* > > > > Now - NPM is a different beast altogether - because by package.lock and > > minifying the javascript with it, we are effectively blocking our users > > from doing similar things - upgrading the npm deps - without actually > > taking the sources, modifying them and rebuilding the packages. Which > 95%+ > > of our users will never do. I understand (and I am happy with) the fact > > that ASF releases sources and the rest is just "convenience packages" - > yet > > many of our users (including the vast majority of future manufacturers > who > > will release Airflow) are mainly consuming our .whl package. We make them > > reproducible and we also publish them at "https://downloads.apache.org"; > - > > similarly as our sources, so users can easily do binary artifact > > comparison with our signed/checksummed binaries) - but they will have > > really hard time if we tell them - if you want to upgrade NPM > dependencies > > you need to rebuild all packages from the sources and you need to modify > > the package.lock in those sources before. That's mostly a non-go. They > will > > expect ASF to release updated versions with those dependencies upgraded > and > > even more - they will likely **demand** us to release patchlevels for old > > releases (2.7.45, where 3.2.1 is already available) with those > dependencies > > bumped. I do not like it. Our policy for releases is that we release > only > > the latest MINOR branch with patchlevel fixes. Full stop. If a user wants > > to update a dependency in an older version we want THEM to do it (and > > manufacturers will be on a hook for that) - but we do not want us to have > > to do anything other than tell manufacturers how they can do it and send > > them to the page which explains how. And it should be easy for end users > to > > apply it. > > > > I want to figure out if that would be possible with NPM. My initial idea > is > > to produce a separate convenience source package with only things that > npm > > needs and make it possible to build such a "patch" package by anyone and > > add a procedure explaining how you can build and test such a patch with > > latest dependencies. Very similarly as we do with our docker containers. > We > > explain our users already - that when they want to update some of the > > container images they use, we will generally **never** re-build those > > released images, but they are absolutely free to rebuild the containers > > themselves and use latest dependencies from debian available - also they > > can extend such images with updated new version of python packages. We > > have a dedicated chapter about it in our documentation [3] > > > > *Why do I think such a "npm-only rebuild" instructions are needed ?* > > > > Now - maybe such a tooling (to easily rebuild and replace packages for > NPM > > packages already exists ? Or maybe from our side it's just a matter of > > writing some custom package builder script to do so? Such a tooling also > > should include some rudimentary (or even quite comprehensive) regression > > tests that will allow such a user to be reasonably sure that the newly > > released dependencies do not break their build. > > > > I still have not looked at details - but for me having this is quite a > game > > changer that gives you worry-free release mechanism: > > > > * make sure that when you release - you bump deps to latest (fairly easy) > > * when you release a version - freeze the latest "minor" release - and do > > not worry about 3rd-party released in-between - you explicitly explain > that > > you "freeze" the versions (as we do with docker container) > > * then, put it on the users and make it easy so that they can build and > use > > a hassle-free way (replacement of compiled code with NPM) that contains > the > > SAME code we released in that version but with updated NPM dependencies. > > > > This will give us a very powerful leverage to discuss things with > > manufacturers who - undoubtedly - will start banging our doors at the day > > CRA is going to be effective with "Release a new version of Airflow with > > those dependency issues fixed ASAP:". In which case our response will be > > "here is the link that explains how you can do it yourself, we never > > release old versions because dependencies that were "latest" at the > moment > > of release have some vulnerabilities - but here is the instruction on how > > you can do it yourself'. And BTW - We accept any donation to us if you > > want our security team to work even better as currently it's purely > > volunteers and your money as manufacturer would be very helpful to > improve > > our processes". > > > > I hope it might be helpful in the discussion (though probably far too > long > > for anyone to get thus far). > > > > Links referred to: > > > > [1] Issue with sqlalchemy 2 upgrade : > > https://github.com/apache/airflow/issues/56735 > > [2] Installing from PyPI using "golden set" + upgrading to latest Python > > dependencies > > > > > https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html > > [3] What should I do if my security scan shows critical and high > > vulnerabilities in the image? > > > > > https://airflow.apache.org/docs/docker-stack/index.html#what-should-i-do-if-my-security-scan-shows-critical-and-high-vulnerabilities-in-the-image > > > > > > > > > > > > > > On Fri, Oct 31, 2025 at 10:48 AM Piotr P. Karwasz < > > [email protected]> > > wrote: > > > > > Hi Bryan, > > > > > > On 31.10.2025 07:57, Bryan Ellis wrote: > > > > Question: > > > > > > > > If a Dependabot PR appears during the middle of a release vote — and > it > > > > updates a sub-dependency to fix a security issue — **does that > > invalidate > > > > or terminate the vote?** > > > > > > > > > No, only the release manager has the authority to cancel a vote. > > > > > > > What exactly are we voting on? > > > > > > > > 1. **The "release vote package"** — the tarball uploaded to > `dist/dev`, > > > > created using `npm pack`, representing what a consumer would receive > > when > > > > installing the package. > > > > 2. **The Git release tag** — the tag created in the repository, > which a > > > > consumer could also install directly (e.g., `npm install > > > > github:apache/project-repo#1.0.0`). > > > > 3. **The repository clone** — a developer clones the repo, checks out > > the > > > > release tag, and runs `npm install` within the repository. > > > > > > > > > This is consistent across the ASF: votes are cast on the source release > > > package, which must be sufficient for a user to build and test the > > > release [1]. Only the source package constitutes an official act of the > > > Foundation. > > > > > > Compiled artifacts (such as those published to NPM) are provided as a > > > convenience to users, but they are not considered an official ASF > > release. > > > > > > These rules exist because, in compiled languages, it’s difficult to > > > verify that a compiled artifact precisely matches the source. However, > > > thanks to reproducible-build efforts, projects like Log4j also verify > > > the reproducibility of JARs whenever possible. This verification is a > > > prerequisite for allowing releases to be built via GitHub Actions > (i.e., > > > on infrastructure not owned or directly controlled by the release > > manager). > > > > > > ## Vulnerabilities in dependencies > > > > > > If a vulnerability is disclosed in any dependency of the project, I > > > generally follow this rule of thumb: > > > > > > - If you ship the dependency, you effectively own it. You should make a > > > reasonable effort to release versions without “known exploitable > > > vulnerabilities”. > > > - If you do not ship the dependency, it’s sufficient to document its > > > presence. You don’t need to block or reissue a release solely because > > > of it. > > > > > > If your packages include an `npm-shrinkwrap.json` file, I would pause > > > the release to assess the potential impact of the vulnerability. > Without > > > it, you can assume that users will receive the same Dependabot PR and > > > handle the update independently. > > > > > > > > > Piotr > > > > > > [1] https://www.apache.org/legal/release-policy.html#artifacts > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: > > [email protected] > > > For additional commands, e-mail: > > > [email protected] > > > > > > > > >
