Re: How should we handle greenbone-security-assistant?
Quoting Adrian Bunk (2020-12-19 10:17:04) > On Fri, Dec 18, 2020 at 05:12:53PM +0100, Jonas Smedegaard wrote: > > Quoting Adrian Bunk (2020-12-18 15:36:23) > > > On Fri, Dec 18, 2020 at 01:33:33PM +0100, Jonas Smedegaard wrote: > > > > It is indeed not realistic to fit all fast-changing code > > > > projects into Debian. We have made a few fast-paced projects > > > > like Firefox fit, but in my opinion we did that in a problematic > > > > way: By endorsing embedded code copies, which is painful to > > > > maintain. > > > > > > > > I think we should not relax our rules, but (improve our packages > > > > so that we can) tighten our rules to apply more consistently - > > > > e.g. avoid embedded code copies also in Firefox. > > > > > > Embedded code copies are the smallest problem with Firefox, and on > > > that I would actually trust Mozilla to release fixes quickly. > > > > I do trust Mozilla to release fixes quickly - my point was a > > different one: Mozilla and Google and GNOME and KDE each being quick > > to release fixes for libusrsctp or some other embedded library is > > still different from linking with a shared copy. > > Firefox in unstable is mostly using shared libraries, in (old)stable > it is using some static libraries because Firefox wants more recent > versions than are in the distribution. > > The big problem is that Firefox is not security supportable without > upgrading to new upstream versions that are not on the same stable > branch, such software is not suitable for distributions with security > supported stable series like Debian or Ubuntu. Yes, Firefox initially use system-shared libraries and use locally embedded copies only when needed. Similar for Chromium and other packages (to a varying degree of "when needed"). Yes, keeping the application security supportable in a stable environment is the big problem. My point is that we currently address that big problem by effectively encourage locally embedding code copies, as our way of addressing that big problem: Firefox and Chromium are packaged as a single big self-contained thing including its web rendering engine, and can be security-maintained; Epiphany (a.k.a. GNOME Web) and other web browsers are packaged without embedding their web rendering engine, and those (webkitgtk and qtwebengine-opensource-src) loose security support. Firefox is not badly packaged. It works! But it works in a way that does not scale well - and I find it worrisome if big popular projects get preferential treatment in Debian. Possibly they don't. Possibly if 10 or 50 other packages began including local copies of library code to not _need_ to depend on system-shared code at a later stage of their lifecycle in Debian, we would happily accept that. But I highly doubt that, and it feels backwards to me to do it. janus is a fast-paced package. It links with libusrsctp and libsrtp2, and some upgrades require upgrades to those other libraries as well. Should I embed copies of those libraries into src:janus to ease a later upgrade while in stable Debian? Firefox and Chromium and webkitgtk and qtwebengine-opensource-src embed libusrsctp and libsrtp2. It works, and addressed "the big problem", but I think we should not undermine but find ways to embrace Debian Policy [§4.13]: > Some software packages include in their distribution convenience > copies of code from other software packages, generally so that users > compiling from source don’t have to download multiple packages. Debian > packages should not make use of these convenience copies unless the > included package is explicitly intended to be used in this way. If the > included code is already in the Debian archive in the form of a > library, the Debian packaging should ensure that binary packages > reference the libraries already in Debian and the convenience copy is > not used. If the included code is not already in Debian, it should be > packaged separately as a prerequisite if possible. - Jonas [§4.13]: https://www.debian.org/doc/debian-policy/ch-source.html#embedded-code-copies -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
Re: How should we handle greenbone-security-assistant?
On Fri, Dec 18, 2020 at 05:12:53PM +0100, Jonas Smedegaard wrote: > Quoting Adrian Bunk (2020-12-18 15:36:23) > > On Fri, Dec 18, 2020 at 01:33:33PM +0100, Jonas Smedegaard wrote: > > > It is indeed not realistic to fit all fast-changing code projects > > > into Debian. We have made a few fast-paced projects like Firefox > > > fit, but in my opinion we did that in a problematic way: By > > > endorsing embedded code copies, which is painful to maintain. > > > > > > I think we should not relax our rules, but (improve our packages so > > > that we can) tighten our rules to apply more consistently - e.g. > > > avoid embedded code copies also in Firefox. > > > > Embedded code copies are the smallest problem with Firefox, and on > > that I would actually trust Mozilla to release fixes quickly. > > I do trust Mozilla to release fixes quickly - my point was a different > one: Mozilla and Google and GNOME and KDE each being quick to release > fixes for libusrsctp or some other embedded library is still different > from linking with a shared copy. Firefox in unstable is mostly using shared libraries, in (old)stable it is using some static libraries because Firefox wants more recent versions than are in the distribution. The big problem is that Firefox is not security supportable without upgrading to new upstream versions that are not on the same stable branch, such software is not suitable for distributions with security supported stable series like Debian or Ubuntu. > - Jonas cu Adrian
Re: How should we handle greenbone-security-assistant?
On Fri, 2020-12-18 at 14:11 +0200, Adrian Bunk wrote: > On Fri, Dec 18, 2020 at 09:11:42AM +0100, Raphael Hertzog wrote: > > On the top of my head: > > - as a user, I like to have to only know about "apt/dpkg" instead > > of pip/npm/gem/... > > This sounds as if you are making the case for a higher level tool > that calls lower level tools like apt and npm. While there have been several attempts at a universal packaging system (Nix, RPM, Dpkg/apt, pacman...), none have truly come to be the "one true system". Further, simply implementing such a system as an abstraction of several underlying systems would be impossible, from a design perspective. Every packaging system treats installed packages differently. NPM will favor just installing into a local folder: pip has a separate set of installed packages for every python version. Gem permits many versions of the same package to be co-installed: apt would literally explode if you tried that [1]. Almost none of these packaging tools are really intended to install end- user programs. They leave that task up to the distribution. Yes, sure, many of them will try it: but when was the last time you saw an Electron app on the npm registry, or a PyQT app that didn't recommend you install some things separately with apt? Our theoretical system would need to locate such packages and their dependencies, even when such dependencies are on many disparate packaging systems. It would need to manage numerous different views of the list of installed package, some of which are completely inconsistent. Furthermore, look at namespacing. If I call magic-installer imagemagick, do I mean the NPM package, or the apt package? Both? I'm sure you could find an example of a single package name duplicated across all the various managers: imagemagick was just the first to come to mind. And if you say, "well, obviously, every package name would be prefixed with its manager", then why are we even bothering with this exercise, when we could just use the manager? While I do agree that it would be great if things like this could work more smoothly, I have to disagree that the way to do that is through another package manager. The limitations on our current tools are a direct result of their design requirements: I can't see that changing soon. [1]: Trust me: since I tried, I was never able to regrow my eyebrows, and I still find chunks of .deb files all over the house. Thanks for reading, Calum signature.asc Description: This is a digitally signed message part
Re: How should we handle greenbone-security-assistant?
Quoting Michael Stone (2020-12-18 20:42:57) > On Fri, Dec 18, 2020 at 09:11:42AM +0100, Raphael Hertzog wrote: > >And at least anyone that is not installing the latest version can > >have an idea of whether it's important/urgent for them to upgrade or > >not. > > I think the software in question is a good example where there's > basically no value in having an old version packaged. Who the heck > wants to know if something has year old vulnerabilities but doesn't > care about current vulnerabilities? Uhm, can only the newest _code_ show the newest _data_?!? Or do I misunderstand it somehow? - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
Re: How should we handle greenbone-security-assistant?
On Fri, Dec 18, 2020 at 09:11:42AM +0100, Raphael Hertzog wrote: And at least anyone that is not installing the latest version can have an idea of whether it's important/urgent for them to upgrade or not. I think the software in question is a good example where there's basically no value in having an old version packaged. Who the heck wants to know if something has year old vulnerabilities but doesn't care about current vulnerabilities?
Re: How should we handle greenbone-security-assistant?
Quoting Adrian Bunk (2020-12-18 15:36:23) > On Fri, Dec 18, 2020 at 01:33:33PM +0100, Jonas Smedegaard wrote: > > It is indeed not realistic to fit all fast-changing code projects > > into Debian. We have made a few fast-paced projects like Firefox > > fit, but in my opinion we did that in a problematic way: By > > endorsing embedded code copies, which is painful to maintain. > > > > I think we should not relax our rules, but (improve our packages so > > that we can) tighten our rules to apply more consistently - e.g. > > avoid embedded code copies also in Firefox. > > Embedded code copies are the smallest problem with Firefox, and on > that I would actually trust Mozilla to release fixes quickly. I do trust Mozilla to release fixes quickly - my point was a different one: Mozilla and Google and GNOME and KDE each being quick to release fixes for libusrsctp or some other embedded library is still different from linking with a shared copy. - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
Re: How should we handle greenbone-security-assistant?
On Fri, Dec 18, 2020 at 8:59 am, Raphael Hertzog wrote: On Thu, 17 Dec 2020, Pirate Praveen wrote: > - ensurance that we use DFSG free code only > => we can have tool to review licenses of what has been > downloaded during build and embedded in the binary packages Then there would not be any value for Debian with such a scenario as people can do such analysis on any distro/container. It would make debian irrelevant. I don't think so. First the tool is here to help the maintainer do the assertion, it's unlikely to be 100% automated, it will likely point out files to inspect manually and so on. And, as a user, even if the tool exists, I wouldn't want to run it manually, I would continue to rely on Debian for the vetting process. I don't want to have to do this on my own. Even with that tool, if we have to change any modules, we will have to create forks of these modules and publish them and also modify package.json to use these forks. gitlab has to do that many times with many rubygems when upstream is not responsive to pull requests. > => we are doing bad now because many useful things are not packaged > (due to the mismatch between our rules and those not-longer-so-new > ecosystems) and when users have to manually install, the reliability > goes down... This I agree, but this could be achived by a mix of vendoring and individual packages. We can vendor modules that are specific to a single app and package more useful libraries as individual packages. For this to work at scale, you need to work with the upstream ecosystem so that this works out of the box... AFAIK right now adding the required node modules in build-depends will not avoid those modules to be downloaded by the upstream build system and there's no simple flag that you can just add to enable that behaviour. Is that correct ? You will have to create a patch that removes the packaged node modules from package.json See this patch in gitlab, https://salsa.debian.org/ruby-team/gitlab/-/blob/master/debian/patches/0740-use-packaged-modules.patch#L94 Additionally, you will have to tell webpack/rollup to take modules installed by apt https://salsa.debian.org/ruby-team/gitlab/-/blob/master/debian/patches/0740-use-packaged-modules.patch#L35 You may able to also add a plugin to yarn to automate this. If there is a yarn plugin that checks and prefers the apt installed modules, then that would work without having to patch the upstream build files. Ruby does that already with rubygems-integrations package. bundle install --local will take the apt installed modules without any change in upstream build tools. > - possibility to rebuild from source > => we could have some sort of proxy that would store everything > downloaded and let us rebuild an identical package without net access > even if the remote resources disappear Why would anyone need to use debian in such a scenario? I don't know for you but the reasons to use Debian would not be changed by the addition of this mechanism. I know that I use only free software, that all the tools are easy to install, that some sane default configuration has been provided by the maintainer, that further instructions are in README.Debian, etc. All the current trends are making it easy for developers to ship code directly to users. Which encourages more isolation instead of collaboration between projects. It also makes it easy for shipping more proprietary code, duplication of security tracking or lack of it. Debian and other distributions have provided an important buffer between developers and users as we did not necessarily follow the priorities or choices of upstream developers exactly always. This I agree with. And I believe it still stays true even if we accept to vendor large amount of stuff. We need to be doing what is the buzz of the time. Free Software was not a mainstream idea when we started. I don't understand what you are trying to say here. The mainstream idea seems to be isolating every project without any coordination with any other projects, including downstream distributions. The trend is to ship only one configuration (typically as a docker container - some projects don't even support building from source). With distributions like debian, we care for the whole Free Software ecosystem. When we do transitions, we make sure every free software using a library or tool is updated. None of those are mainstream views. Mainstream view is continue using a dependency version till eternity and nothing really breaks once it works on the developers machine. There is even a joke about docker being a way to make sure if it works on the developers system, then lets ship the developers system to everyone. Yes, it is a lot work, but it makes the whole Free Software better by having updated dependencies for all applications. Distributions are also way for users to effec
Re: How should we handle greenbone-security-assistant?
On Fri, Dec 18, 2020 at 01:33:33PM +0100, Jonas Smedegaard wrote: >... > It is indeed not realistic to fit all fast-changing code projects into > Debian. We have made a few fast-paced projects like Firefox fit, but in > my opinion we did that in a problematic way: By endorsing embedded code > copies, which is painful to maintain. > > I think we should not relax our rules, but (improve our packages so that > we can) tighten our rules to apply more consistently - e.g. avoid > embedded code copies also in Firefox. >... Embedded code copies are the smallest problem with Firefox, and on that I would actually trust Mozilla to release fixes quickly. The huge pain with Firefox is that it has a history of needing updated versions of at least 5 different compilers in Debian stable series: https://tracker.debian.org/pkg/gcc-mozilla https://tracker.debian.org/pkg/llvm-toolchain-7 https://tracker.debian.org/pkg/rustc https://tracker.debian.org/pkg/nasm-mozilla https://tracker.debian.org/pkg/nodejs-mozilla IMHO the best option for Firefox would be to stop shipping it in stable, and offer a Flatpak instead. > - Jonas cu Adrian
Re: How should we handle greenbone-security-assistant?
On Fri, Dec 18, 2020 at 09:11:42AM +0100, Raphael Hertzog wrote: > On Thu, 17 Dec 2020, Adrian Bunk wrote: > > > - ease of installation and reliability > > > => we are doing bad now because many useful things are not packaged > > > > What is the value added just by installing things through dpkg instead > > of npm? > > Why are you using Debian if you ask this? I am using Debian on my desktop and Ubuntu on my laptop for having a stable and security-supported environment. > On the top of my head: > - as a user, I like to have to only know about "apt/dpkg" instead > of pip/npm/gem/... This sounds as if you are making the case for a higher level tool that calls lower level tools like apt and npm. > - the Debian maintainer is ensuring some consistency that a random > collection of uptsream installations are not enusring >... Tools like npm can actually give more consistency by pinning all dependencies to exact versions. For security support this is a nightmare, but for consistency it is better. > > > (due to the mismatch between our rules and those not-longer-so-new > > > ecosystems) and when users have to manually install, the reliability > > > goes down... > > > > What reliability do you have with a 3 year old version of software where > > upstream only tells your users to upgrade to the latest versions? > > > > The "changed paradigm" usually includes automatic updates to the latest > > version without any maintainance of older versions. > > Indeed. We should have room for such software that should only be provided > within our rolling distribution and as backports. Such software should not be provided in backports. The point of backports is that our users can cherry-pick software that will be shipped in the next stable release, after upgrading to the next stable release all backports should be upgraded to the version in the new stable where they are (security) supported. bullseye-backports is really supposed to be a subset of what will be in the final bullseye release, software that is not expected to be in the next stable release does not belong there. There is surely a need for PPAs for such software in Debian, but this is a usecase different from backports. > > This is the easy part. > > How do you plan to fix all vulnerable versions? > > By providing the latest and greatest version to all our users. As you > noticed, that kind of software does not mesh well with stable and LTS. >... That kind of software does not mesh well with wanting security fixes. The "changed paradigm" you are pushing includes shipping of vendored dependencies pinned to ancient versions - often without caring about vulnerabilities. As a random example, src:rustc in Debian ships a vendored copy of a 2017 version of src:rust-yaml-rust to which it is pinned and which is missing CVE fixes. I do not know whether they are relevant for rustc or whether this vendored code is used at all, but this is the latest upstream version of rustc and the CVEs are from 2018 and 2019. > Cheers, cu Adrian
Re: How should we handle greenbone-security-assistant?
Quoting Raphael Hertzog (2020-12-18 10:44:10) > On Fri, 18 Dec 2020, Jonas Smedegaard wrote: > > Who is keeping software out of Debian here? > > We, collectively, with the fact that we have not adapted our rules and > tooling. [ sorry I turned it personal! I mis-read your post as starting that ] I agree that some Debian packaging work is pretty time consuming. I agree that our tools can be improved. I disagree that our rules are outdated, and by extension disagree that our tools need adaptations that goes against our current rules. To me, Debian is essentially about curating and combining code projects into a single fully Free, coherent, and long-term maintainable system. We also do other things besides "a single fully Free, coherent, and long-term maintainable system" - some of it within our debian.org domain, some of it nearby in the debian.net domain or elsewhere. Yes, we also do non-free stuff, without releasing it as Debian. Yes, we also use lesser maintainable parts, not released as Debian. It seems you are arguing that Debian should change its rules to fit a World that is not long-term maintainable. I disagree - I think we should acknowledge that not everything can fit into Debian, but that does not mean Debian need to change. In fact, I find it *more* important to insist on the rules governing long-term maintenance when the outside World care less about that. > I'm not saying it's impossible, I'm saying it's not realistic for > volunteer maintainers, and even for the few that have the chance to be > paid for it, they can't justify the expense to do it following the > current rules and policies. I disagree that the issue here is volunteer time or "expenses" - the issue is that we deliberately and consciously want something that not everyone outside Debian wants: We want to be able to release a complete operating system and have it "stable" for several years. It is indeed not realistic to fit all fast-changing code projects into Debian. We have made a few fast-paced projects like Firefox fit, but in my opinion we did that in a problematic way: By endorsing embedded code copies, which is painful to maintain. I think we should not relax our rules, but (improve our packages so that we can) tighten our rules to apply more consistently - e.g. avoid embedded code copies also in Firefox. For fast-paced projects we already have contrib and other approaches, recognizing that not everything fits into Debian but is helpful to keep close. > > Or if you are talking about the Kali helper tool here, then please do > > share more details of that wonderful tool, so we can know if you are > > talking about a way to make the impossible-in-your-view possible while > > still complying with Debian Policy, or the wonder really is in an > > implied relaxing of packaging quality. > > There's no wonderful tool. It's just that we don't have the same > rules. And I don't agree that the "packaging quality" is worse in this > specific case. > > We have certainly made ugly things at times, but in this specific > case, relying on the uptsream build system is certainly not "bad > quality", the tool works and it likely works better when we can ensure > that we use the same set of libraries than upstream compared to the > random versions that happen to be in Debian at any given time. I don't share your certainty about the quality of Node.js dependency management - especially within a surrounding scope of a stable system. Also, I dislike your description of versions in Debian being "random". Technically it is wrong (but that's nitpicking), however my beef is that its seems to me that you really mean to say that the versions in Debian is badly curated: When upstream Node.js developers explicit (paraphrased) instructs to "process this snippet with Babel 7.2.11", the JavaScript team coerces that into "process this snippet with newest Babel available in Debian" - I don't call that "random" nor badly curated, I call that sensibly curated. - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
Re: How should we handle greenbone-security-assistant?
On Fri, 18 Dec 2020, Jonas Smedegaard wrote: > Who is keeping software out of Debian here? We, collectively, with the fact that we have not adapted our rules and tooling. (And yes you certainly did more than me on this front, in particular in the tooling front, I'm not arguing against that) > Me, considering proper packaging of greenbone-security-assistant a > possible task? > > Or you, considering it an impossible task? Please don't make it personal. I did not and I ask you to not do it. I'm not saying it's impossible, I'm saying it's not realistic for volunteer maintainers, and even for the few that have the chance to be paid for it, they can't justify the expense to do it following the current rules and policies. > Or if you are talking about the Kali helper tool here, then please do > share more details of that wonderful tool, so we can know if you are > talking about a way to make the impossible-in-your-view possible while > still complying with Debian Policy, or the wonder really is in an > implied relaxing of packaging quality. There's no wonderful tool. It's just that we don't have the same rules. And I don't agree that the "packaging quality" is worse in this specific case. We have certainly made ugly things at times, but in this specific case, relying on the uptsream build system is certainly not "bad quality", the tool works and it likely works better when we can ensure that we use the same set of libraries than upstream compared to the random versions that happen to be in Debian at any given time. Cheers, -- ⢀⣴⠾⠻⢶⣦⠀ Raphaël Hertzog ⣾⠁⢠⠒⠀⣿⡁ ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/ ⠈⠳⣄ Debian Long Term Support: https://deb.li/LTS signature.asc Description: PGP signature
Re: How should we handle greenbone-security-assistant?
Quoting Raphael Hertzog (2020-12-18 09:05:17) > On Thu, 17 Dec 2020, Jonas Smedegaard wrote: > > > said package was updated in Kali in a matter of hours. > > > > That's great. > > > > I mean - *either* it is great that Kali has a far better tool that > > we might adopt - *or* it is great that Kali exists for those with > > different needs than those of Debian. > > I know that some people are happy when we keep software out of Debian, > but I don't share that view. For me the universal OS ought to to have > a place for any decently maintained free software. > > We might not want to ship it in stable and include it in LTS, but we > should be able to provide the latest version and make it easy to > install to our users. Who is keeping software out of Debian here? Me, considering proper packaging of greenbone-security-assistant a possible task? Or you, considering it an impossible task? Or if you are talking about the Kali helper tool here, then please do share more details of that wonderful tool, so we can know if you are talking about a way to make the impossible-in-your-view possible while still complying with Debian Policy, or the wonder really is in an implied relaxing of packaging quality. - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
Re: How should we handle greenbone-security-assistant?
On Thu, 17 Dec 2020, Adrian Bunk wrote: > > - ease of installation and reliability > > => we are doing bad now because many useful things are not packaged > > What is the value added just by installing things through dpkg instead > of npm? Why are you using Debian if you ask this? On the top of my head: - as a user, I like to have to only know about "apt/dpkg" instead of pip/npm/gem/... - the Debian maintainer is ensuring some consistency that a random collection of uptsream installations are not enusring - simple and consisten upgrade process - etc. > > (due to the mismatch between our rules and those not-longer-so-new > > ecosystems) and when users have to manually install, the reliability > > goes down... > > What reliability do you have with a 3 year old version of software where > upstream only tells your users to upgrade to the latest versions? > > The "changed paradigm" usually includes automatic updates to the latest > version without any maintainance of older versions. Indeed. We should have room for such software that should only be provided within our rolling distribution and as backports. > This is the easy part. > How do you plan to fix all vulnerable versions? By providing the latest and greatest version to all our users. As you noticed, that kind of software does not mesh well with stable and LTS. And at least anyone that is not installing the latest version can have an idea of whether it's important/urgent for them to upgrade or not. Cheers, -- ⢀⣴⠾⠻⢶⣦⠀ Raphaël Hertzog ⣾⠁⢠⠒⠀⣿⡁ ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/ ⠈⠳⣄ Debian Long Term Support: https://deb.li/LTS
Re: How should we handle greenbone-security-assistant?
On Thu, 17 Dec 2020, Jonas Smedegaard wrote: > > said package was updated in Kali in a matter of hours. > > That's great. > > I mean - *either* it is great that Kali has a far better tool that we > might adopt - *or* it is great that Kali exists for those with different > needs than those of Debian. I know that some people are happy when we keep software out of Debian, but I don't share that view. For me the universal OS ought to to have a place for any decently maintained free software. We might not want to ship it in stable and include it in LTS, but we should be able to provide the latest version and make it easy to install to our users. Cheers, -- ⢀⣴⠾⠻⢶⣦⠀ Raphaël Hertzog ⣾⠁⢠⠒⠀⣿⡁ ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/ ⠈⠳⣄ Debian Long Term Support: https://deb.li/LTS signature.asc Description: PGP signature
Re: How should we handle greenbone-security-assistant?
On Thu, 17 Dec 2020, Pirate Praveen wrote: > > - ensurance that we use DFSG free code only > > => we can have tool to review licenses of what has been > > downloaded during build and embedded in the binary packages > > Then there would not be any value for Debian with such a scenario as people > can do such analysis on any distro/container. > > It would make debian irrelevant. I don't think so. First the tool is here to help the maintainer do the assertion, it's unlikely to be 100% automated, it will likely point out files to inspect manually and so on. And, as a user, even if the tool exists, I wouldn't want to run it manually, I would continue to rely on Debian for the vetting process. I don't want to have to do this on my own. > > => we are doing bad now because many useful things are not packaged > > (due to the mismatch between our rules and those not-longer-so-new > > ecosystems) and when users have to manually install, the reliability > > goes down... > > This I agree, but this could be achived by a mix of vendoring and individual > packages. We can vendor modules that are specific to a single app and > package more useful libraries as individual packages. For this to work at scale, you need to work with the upstream ecosystem so that this works out of the box... AFAIK right now adding the required node modules in build-depends will not avoid those modules to be downloaded by the upstream build system and there's no simple flag that you can just add to enable that behaviour. Is that correct ? > > - possibility to rebuild from source > > => we could have some sort of proxy that would store everything > > downloaded and let us rebuild an identical package without net access > > even if the remote resources disappear > > Why would anyone need to use debian in such a scenario? I don't know for you but the reasons to use Debian would not be changed by the addition of this mechanism. I know that I use only free software, that all the tools are easy to install, that some sane default configuration has been provided by the maintainer, that further instructions are in README.Debian, etc. > All the current trends are making it easy for developers to ship code > directly to users. Which encourages more isolation instead of collaboration > between projects. It also makes it easy for shipping more proprietary code, > duplication of security tracking or lack of it. Debian and other > distributions have provided an important buffer between developers and users > as we did not necessarily follow the priorities or choices of upstream > developers exactly always. This I agree with. And I believe it still stays true even if we accept to vendor large amount of stuff. > We need to be doing what is the buzz of the time. Free Software was not a > mainstream idea when we started. I don't understand what you are trying to say here. Cheers, -- ⢀⣴⠾⠻⢶⣦⠀ Raphaël Hertzog ⣾⠁⢠⠒⠀⣿⡁ ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/ ⠈⠳⣄ Debian Long Term Support: https://deb.li/LTS
Re: How should we handle greenbone-security-assistant?
On Thu, Dec 17, 2020 at 02:55:11PM +0100, Raphael Hertzog wrote: >... > By trying to shoehorn node/go modules into Debian packages we are creating > busy work with almost no value. We must go back to what is the value > added by Debian and find ways to continue to provide this value while > accepting the changed paradigm that some applications/ecosystems have > embraced. > > And for me selling points are: >... > - ease of installation and reliability > => we are doing bad now because many useful things are not packaged What is the value added just by installing things through dpkg instead of npm? > (due to the mismatch between our rules and those not-longer-so-new > ecosystems) and when users have to manually install, the reliability > goes down... What reliability do you have with a 3 year old version of software where upstream only tells your users to upgrade to the latest versions? The "changed paradigm" usually includes automatic updates to the latest version without any maintainance of older versions. >... > We must go back to what is the value > added by Debian and find ways to continue to provide this value while > accepting the changed paradigm that some applications/ecosystems have > embraced. >... > - security support > => we need to be able to identify packages that are vulnerable because > they have embedded a problematic version of a node/go module, again we > need tools that analyze what got embedded in the binary package and make > this easy to query >... This is the easy part. How do you plan to fix all vulnerable versions? If the tooling tells you that we have 100 copies of OpenSSL in 70 different versions across the archive, this means you also have to do the actual work of fixing every single one each time there is a new CVE. This is not a purely hypothetical example, I have seen OpenSSL vendored in such "changed paradigm" ecosystems. > Cheers, cu Adrian
Re: How should we handle greenbone-security-assistant?
Quoting Raphael Hertzog (2020-12-17 14:55:11) > On Thu, 17 Dec 2020, Jonas Smedegaard wrote: > > In reality, most Nodejs modules declare too tight versioning for > > their > [...] > > I know this, but I also know that such an analysis is very > time-consuming and needs a good knowledge of the language and of the > upstream package, which I don't have. Ahh, indeed we have no magic tool to do the maintainenance for us. There are tools like npm2deb doing something considered too lousy quality for Debian. Just as we had rpm2deb back in the day when the frustrating thing was that Redhat was far more popular than Debian. > said package was updated in Kali in a matter of hours. That's great. I mean - *either* it is great that Kali has a far better tool that we might adopt - *or* it is great that Kali exists for those with different needs than those of Debian. - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
Re: How should we handle greenbone-security-assistant?
On Thu, Dec 17, 2020 at 2:55 pm, Raphael Hertzog wrote: I know this, but I also know that such an analysis is very time-consuming and needs a good knowledge of the language and of the upstream package, which I don't have. Most of the time Semantic Versioning works (https://semver.org) so you don't really need to know the code. In your original post you seemingly already ruled out proper packaging as a premise, and it seems you continue to use absolutes like "everything" and "never". I find that discouraging - plase consider a less negative tone. Sorry, I don't want to discourage you. But I'm also sure that I will never be able to justify spending days and weeks on packaging (and then maintaining) all those node modules for the benefit of pushing a single package to Debian when said package was updated in Kali in a matter of hours. Though that is not entirely correct. Many of the build tools and libraries I had to package for gitlab are reused by other packages. By trying to shoehorn node/go modules into Debian packages we are creating busy work with almost no value. We must go back to what is the value added by Debian and find ways to continue to provide this value while accepting the changed paradigm that some applications/ecosystems have embraced. And for me selling points are: - ensurance that we use DFSG free code only => we can have tool to review licenses of what has been downloaded during build and embedded in the binary packages Then there would not be any value for Debian with such a scenario as people can do such analysis on any distro/container. It would make debian irrelevant. - ease of installation and reliability => we are doing bad now because many useful things are not packaged (due to the mismatch between our rules and those not-longer-so-new ecosystems) and when users have to manually install, the reliability goes down... This I agree, but this could be achived by a mix of vendoring and individual packages. We can vendor modules that are specific to a single app and package more useful libraries as individual packages. - possibility to rebuild from source => we could have some sort of proxy that would store everything downloaded and let us rebuild an identical package without net access even if the remote resources disappear Why would anyone need to use debian in such a scenario? All the current trends are making it easy for developers to ship code directly to users. Which encourages more isolation instead of collaboration between projects. It also makes it easy for shipping more proprietary code, duplication of security tracking or lack of it. Debian and other distributions have provided an important buffer between developers and users as we did not necessarily follow the priorities or choices of upstream developers exactly always. We need to be doing what is the buzz of the time. Free Software was not a mainstream idea when we started.
Re: How should we handle greenbone-security-assistant?
Hi, On Thu, 17 Dec 2020, Jonas Smedegaard wrote: > > Out of curiosity, I have run your script on the package.json file of > > greenbone-security-assistant and this just confirms that it's not > > realistic to package everything separately: > > https://wiki.debian.org/Javascript/Nodejs/Tasks/gsa > > Nice. Doesn't look like an impossible task to me. It looks like a huge amount of work that does not bring any value compared to the Kali package relying on the upstream build system without any tinkering. > In reality, most Nodejs modules declare too tight versioning for their [...] I know this, but I also know that such an analysis is very time-consuming and needs a good knowledge of the language and of the upstream package, which I don't have. > In your original post you seemingly already ruled out proper packaging > as a premise, and it seems you continue to use absolutes like > "everything" and "never". I find that discouraging - plase consider a > less negative tone. Sorry, I don't want to discourage you. But I'm also sure that I will never be able to justify spending days and weeks on packaging (and then maintaining) all those node modules for the benefit of pushing a single package to Debian when said package was updated in Kali in a matter of hours. By trying to shoehorn node/go modules into Debian packages we are creating busy work with almost no value. We must go back to what is the value added by Debian and find ways to continue to provide this value while accepting the changed paradigm that some applications/ecosystems have embraced. And for me selling points are: - ensurance that we use DFSG free code only => we can have tool to review licenses of what has been downloaded during build and embedded in the binary packages - ease of installation and reliability => we are doing bad now because many useful things are not packaged (due to the mismatch between our rules and those not-longer-so-new ecosystems) and when users have to manually install, the reliability goes down... - possibility to rebuild from source => we could have some sort of proxy that would store everything downloaded and let us rebuild an identical package without net access even if the remote resources disappear - security support => we need to be able to identify packages that are vulnerable because they have embedded a problematic version of a node/go module, again we need tools that analyze what got embedded in the binary package and make this easy to query BTW, that's the kind of infrastructure development that would advance the cause of Debian and that I would be glad to (start to) fund through https://salsa.debian.org/freexian-team/project-funding Cheers, -- ⢀⣴⠾⠻⢶⣦⠀ Raphaël Hertzog ⣾⠁⢠⠒⠀⣿⡁ ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/ ⠈⠳⣄ Debian Long Term Support: https://deb.li/LTS signature.asc Description: PGP signature
Re: How should we handle greenbone-security-assistant?
On Thu, Dec 17, 2020 at 2:19 pm, Raphael Hertzog wrote: Hello, On Thu, 17 Dec 2020, Pirate Praveen wrote: >1/ download all the node modules and add them to the source package, but >then it's just impossible to write a copyright file to document the source >package. That would be the best option though, the yarn.lock file >effectively locks a very specific version of each node module so >even though it's downloaded the net effect is very similar to "vendoring" >as is done by other projects. This will only work for a subset of modules because most modules will not be just source (unlike golang), it will contain prebuilt files. The original source is usually ES6 or typescript usually and you need to build them using babel/rollup/typescript. I don't understand what you mean. Are you trying to say that this won't help much because we will have another DFSG-freeness problem in the sense that what we would embed is not the source and that DFSG requires us to provide the sources? yes. Most modules now ship files generated by tools like babel/rollup/webpack/typescript etc. So we will have to get their preferred source code from their source repos and build them in debian. For example d3-color module ships files generated by rollup, which we rebuild in debian. https://salsa.debian.org/js-team/node-d3-color/-/blob/master/debian/rules#L8 You may still be able to vendor the original source of these modules and only build the final app with tools like webpack, but that would be different from what upstream does (upstream pulls all prebuilt modules from npmjs.com) and some modules may even have other custom build steps/tools. add-node-component from pkg-js-tools makes it easy to vendor original source code. So even if you can't vendor all modules easily in one shot, I think you can still manage a lot of modules that way and package only the smaller subset that you can't vendor. We also have tools to build vendored code. For example, pirates is module vendored in node-babel using their preferred source and built in debian https://salsa.debian.org/js-team/node-babel/-/blob/master/debian/gbp.conf https://salsa.debian.org/js-team/node-babel/-/blob/master/debian/nodejs/pirates/build Older modules were simpler and did not require any build steps. So it was easy to vendor these modules in node-gulp, https://salsa.debian.org/js-team/node-gulp/-/blob/master/debian/gbp.conf I use this option for gitlab, but I want to eventually move it to main once I package all of them. Out of 1700+ node modules, I'm left with 270+ node modules pulled outside of main. I remove already packaged modules from package.json. I appreciate all the efforts that you are doing here but to me it seems like a dead end. Much like the idea of adding another repository to accomodate the ever-growing list of go modules that nobody will ever want to use except as a build-dependency. To me it seems that we must adapt our rules, our expectations and our tooling to cope with this paradigm shift where dependencies are downloaded at build time. Well, its indeed a conflict of different values. It may indeed be a losing fight, but I prefer to still fight this as much as I can. Many developers want the distributions to be just base for containerized apps, but I don't like that vision. It is indeed easier for developers this way, but I don't necessarily think that is the best way for users. I think reproducible builds and the guarantee that every package is built from source by debian and anyone can rebuild easily are still valuable goals from a security perspective. I try to bring in more people to traditional packaging side and if we are able to get more people to realize the value in what we do, we can still manage. But it is no way an easy task.
Re: How should we handle greenbone-security-assistant?
Hello, On Thu, 17 Dec 2020, Pirate Praveen wrote: > >1/ download all the node modules and add them to the source package, but > >then it's just impossible to write a copyright file to document the source > >package. That would be the best option though, the yarn.lock file > >effectively locks a very specific version of each node module so > >even though it's downloaded the net effect is very similar to "vendoring" > >as is done by other projects. > > This will only work for a subset of modules because most modules will > not be just source (unlike golang), it will contain prebuilt files. The > original source is usually ES6 or typescript usually and you need to > build them using babel/rollup/typescript. I don't understand what you mean. Are you trying to say that this won't help much because we will have another DFSG-freeness problem in the sense that what we would embed is not the source and that DFSG requires us to provide the sources? > I use this option for gitlab, but I want to eventually move it to main > once I package all of them. Out of 1700+ node modules, I'm left with > 270+ node modules pulled outside of main. I remove already packaged > modules from package.json. I appreciate all the efforts that you are doing here but to me it seems like a dead end. Much like the idea of adding another repository to accomodate the ever-growing list of go modules that nobody will ever want to use except as a build-dependency. To me it seems that we must adapt our rules, our expectations and our tooling to cope with this paradigm shift where dependencies are downloaded at build time. Cheers, -- ⢀⣴⠾⠻⢶⣦⠀ Raphaël Hertzog ⣾⠁⢠⠒⠀⣿⡁ ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/ ⠈⠳⣄ Debian Long Term Support: https://deb.li/LTS
Re: How should we handle greenbone-security-assistant?
Quoting Raphael Hertzog (2020-12-17 13:16:14) > On Wed, 16 Dec 2020, Jonas Smedegaard wrote: > > 4/ analyze what yarn/npm would do during build, and translate that > > into existing Debian Nodejs packages and actual need for custom > > work. In the JavaScript team we use this page as starting point for > > analyzing large projects: > > https://wiki.debian.org/Javascript/Nodejs/Tasks > > Out of curiosity, I have run your script on the package.json file of > greenbone-security-assistant and this just confirms that it's not > realistic to package everything separately: > https://wiki.debian.org/Javascript/Nodejs/Tasks/gsa Nice. Doesn't look like an impossible task to me. > Even if you package everything, you will never ever have the right > combination of version of the various packages. What is possible to auto-compute is a coarse view of the work needed. In reality, most Nodejs modules declare too tight versioning for their dependencies, and in many cases it is adequate that a module is packaged even if not at the version declared as required. A concrete example is "ansi-styles" which is most likely working just fine in version 4.x. Also, some build-dependencies are for development purposes other than strictly compiling the code - e.g. for benchmarking or producing test coverage reports. A concrete example is "eslint-config-prettier" which is a lint-checker with a specific bias. It is not strictly needed to lint-check code, but a good idea to do so especially if messing with it through patches - but then the more generic non-biased lint-checker eslint can in many cases be used instead. Also, the script fails to detect virtual packages. A concrete example is "@types/jest" provided in virtual package "node-types-jest". Also, the script lists dependencies multiple times. See e.g. "shape" appearing twice (skipping its dependencies at its second entry), and "d3-shape" is listed several times. In your original post you seemingly already ruled out proper packaging as a premise, and it seems you continue to use absolutes like "everything" and "never". I find that discouraging - plase consider a less negative tone. - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
Re: How should we handle greenbone-security-assistant?
Hi, On Wed, 16 Dec 2020, Jonas Smedegaard wrote: > 4/ analyze what yarn/npm would do during build, and translate that into > existing Debian Nodejs packages and actual need for custom work. In the > JavaScript team we use this page as starting point for analyzing large > projects: https://wiki.debian.org/Javascript/Nodejs/Tasks Out of curiosity, I have run your script on the package.json file of greenbone-security-assistant and this just confirms that it's not realistic to package everything separately: https://wiki.debian.org/Javascript/Nodejs/Tasks/gsa Even if you package everything, you will never ever have the right combination of version of the various packages. Cheers, -- ⢀⣴⠾⠻⢶⣦⠀ Raphaël Hertzog ⣾⠁⢠⠒⠀⣿⡁ ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/ ⠈⠳⣄ Debian Long Term Support: https://deb.li/LTS signature.asc Description: PGP signature
Re: How should we handle greenbone-security-assistant?
On 2020, ഡിസംബർ 16 11:57:04 PM IST, Raphael Hertzog wrote: >Hello, > >in the pkg-security team we have greenbone-security-assistant (gsa) >which is a web interface for gvm (former openvas-manager), the >version currently in Debian no longer works with the latest gvm >so we have to update it to the latest upstream release... but the >latest upstream release has significant changes, in particular >it now relies on yarn or npm from the node ecosystem to download >all the node modules that it needs (and there are many of them, >and there's no way that we will package them individually). > >The Debian policy forbids download during the build so we can't >run the upstream build system as is. > >As I see it we have three options: > >1/ download all the node modules and add them to the source package, but >then it's just impossible to write a copyright file to document the source >package. That would be the best option though, the yarn.lock file >effectively locks a very specific version of each node module so >even though it's downloaded the net effect is very similar to "vendoring" >as is done by other projects. This will only work for a subset of modules because most modules will not be just source (unlike golang), it will contain prebuilt files. The original source is usually ES6 or typescript usually and you need to build them using babel/rollup/typescript. Though the build tools situation is much better than a few years ago when I started with diaspora and gitlab. I had to start with packaging these tools first. >2/ disable this download during the package build, move the package >to contrib, and add some code in the postinst to download the required >node modules at package installation time (possibly with a debconf >confirmation prompt and a command that the user can rerun afterwards >to do it later). I use this option for gitlab, but I want to eventually move it to main once I package all of them. Out of 1700+ node modules, I'm left with 270+ node modules pulled outside of main. I remove already packaged modules from package.json. >3/ get rid of greenbone-security-assistant in debian and keep it in kali >only (all the work I do in pkg-security is part of my Kali work), that >would be a regression from the current situation and is something that I'd >like to avoid. We try to contribute back to Debian, but there's only so >much busy-work that I'm willing to do to achieve this goal. > >What option shall we pick? > >Shouldn't we relax some requirements to avoid having to resort to that >kind of hackery? > >Cheers, -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: How should we handle greenbone-security-assistant?
Quoting Raphael Hertzog (2020-12-16 19:27:04) > in the pkg-security team we have greenbone-security-assistant (gsa) > which is a web interface for gvm (former openvas-manager), the version > currently in Debian no longer works with the latest gvm so we have to > update it to the latest upstream release... but the latest upstream > release has significant changes, in particular it now relies on yarn > or npm from the node ecosystem to download all the node modules that > it needs (and there are many of them, and there's no way that we will > package them individually). > > The Debian policy forbids download during the build so we can't run > the upstream build system as is. > > As I see it we have three options: > > 1/ download all the node modules and add them to the source package, > but then it's just impossible to write a copyright file to document > the source package. That would be the best option though, the > yarn.lock file effectively locks a very specific version of each node > module so even though it's downloaded the net effect is very similar > to "vendoring" as is done by other projects. > > 2/ disable this download during the package build, move the package to > contrib, and add some code in the postinst to download the required > node modules at package installation time (possibly with a debconf > confirmation prompt and a command that the user can rerun afterwards > to do it later). > > 3/ get rid of greenbone-security-assistant in debian and keep it in > kali only (all the work I do in pkg-security is part of my Kali work), > that would be a regression from the current situation and is something > that I'd like to avoid. We try to contribute back to Debian, but > there's only so much busy-work that I'm willing to do to achieve this > goal. 4/ analyze what yarn/npm would do during build, and translate that into existing Debian Nodejs packages and actual need for custom work. In the JavaScript team we use this page as starting point for analyzing large projects: https://wiki.debian.org/Javascript/Nodejs/Tasks - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
