Re: security-tracker: A proposal to significantly reduce reported false-positives (no affected-code shipped)
On 31/08/2024 20:07, Samuel Henrique wrote: Hello everyone, I've written another revision of my proposal, this is version 3 of it, the previous ones are on this email thread on debian-security@lists.debian.org. I did get some feedback from the Security Team privately, it wasn't anything confidential, it's just that some members of the team only noticed my proposal after I sent it to the private mailing list, and the biggest part of the feedback was that they wanted some time to think about it. This time I'm cc'ing the team's mailing list as well, so replies will show up here. Not much has changed in this version, but it should be better than the previous one, this is more of a chance to get feedback from the team again. ## A proposal to significantly reduce reported false-positives (no affected-code shipped) - version 3 I would like to propose something which will lower the amount of reported false-positive CVEs to our users by about 20%. ## tl;dr Debian over-reports on numbers of affected CVEs, the main reason is that we don't have a unique way of stating that a CVE does not affect Debian when we don't build the affected package's feature (or hardening blocks exploits). In these cases, we mark the CVEs as affecting our packages because the source-code contains the vulnerable code (the binary package doesn't). This leads to ourselves and our users being required to manually distinguish which CVEs affect them and which don't anytime there's a need to look at the data. It's effectively noise and we end up reporting the binary packages as affected when that's not true (both on the OVAL files and on the security-tracker json file we generate). I propose we mark those cases as not-affected. Alternatively, I mention an option to create a new state to indicate that the resulting package is not affected due to the build options, but that the source-code contains the vulnerability. I also explain why that's not my prefered approach. ## Problem statement The possible outcomes of a CVE assessment in our security-tracker are[0]: | | | | | | We also have the following severity levels [0]: SEVERITY_LEVEL : (unimportant) | (low) | (medium) | (high) "unimportant" being defined as: unimportant: This problem does not affect the Debian binary package, e.g., a vulnerable source file, which is not built, a vulnerable file in doc/foo/examples/, PHP Safe mode bugs, path disclosure (doesn't matter on Debian). All "non-issues in practice" fall also into this category, like issues only "exploitable" if the code in question is setuid root, exploits which only work if someone already has administrative privileges or similar. This severity is also used for vulnerabilities in packages which are not covered by security support. We have a problem in the way we assess CVEs when the generated package is not affected but the source code contains the vulnerability. Our current process is to set "no-dsa" and lower the severity to "unimportant", although it's also possible that in some cases people are making use of "ignored", which represents "won't fix". The result is that "unimportant/no-dsa" CVEs can mean two things: 1) We are affected but we the severity is too low, eg.: packages not covered by security support, the CVE is considered a non-issue by our security-team but we are still affected... 2) We are definitely not affected since we don't build that feature of the software or we have hardening in place which prevents this from being exploited. This leads to our users, who are interested in knowing which CVEs affect their systems, having to check the notes of every CVE on security-tracker to filter-out the false-positives. Besides that, we also struggle with this ourselves, as someone who would like to fix CVEs will have to filter-out these false-positives themselves. Considering the broad usage of Debian (especially on containers), being able to correctly mark these cases as not affecting the binary packages will have a huge impact on all of the industry. I'm not being over-optimistic here, a lot of effort ends up being spent on generating CVE reports and then having to justify why each one is not fixed. Whether the requirements around CVE fixing are right or wrong is a story of its own, but we have the potential to make ourselves and our users' lives easier with this. ## Proposed solution I propose that we start setting CVEs to "not-affected" when the following is true for all officially supported architectures: * We don't ship the affected source package. * We don't build the affected feature. * We have hardening which makes the exploit impossible (only in the cases when
Re: security-tracker: A proposal to significantly reduce reported false-positives (no affected-code shipped)
Hello everyone, I've written another revision of my proposal, this is version 3 of it, the previous ones are on this email thread on debian-security@lists.debian.org. I did get some feedback from the Security Team privately, it wasn't anything confidential, it's just that some members of the team only noticed my proposal after I sent it to the private mailing list, and the biggest part of the feedback was that they wanted some time to think about it. This time I'm cc'ing the team's mailing list as well, so replies will show up here. Not much has changed in this version, but it should be better than the previous one, this is more of a chance to get feedback from the team again. ## A proposal to significantly reduce reported false-positives (no affected-code shipped) - version 3 I would like to propose something which will lower the amount of reported false-positive CVEs to our users by about 20%. ## tl;dr Debian over-reports on numbers of affected CVEs, the main reason is that we don't have a unique way of stating that a CVE does not affect Debian when we don't build the affected package's feature (or hardening blocks exploits). In these cases, we mark the CVEs as affecting our packages because the source-code contains the vulnerable code (the binary package doesn't). This leads to ourselves and our users being required to manually distinguish which CVEs affect them and which don't anytime there's a need to look at the data. It's effectively noise and we end up reporting the binary packages as affected when that's not true (both on the OVAL files and on the security-tracker json file we generate). I propose we mark those cases as not-affected. Alternatively, I mention an option to create a new state to indicate that the resulting package is not affected due to the build options, but that the source-code contains the vulnerability. I also explain why that's not my prefered approach. ## Problem statement The possible outcomes of a CVE assessment in our security-tracker are[0]: > | | | | | | > We also have the following severity levels [0]: > SEVERITY_LEVEL : (unimportant) | (low) | (medium) | (high) "unimportant" being defined as: > unimportant: This problem does not affect the Debian binary package, e.g., a > vulnerable source file, which is not built, a vulnerable file in > doc/foo/examples/, PHP Safe mode bugs, path disclosure (doesn't matter on > Debian). All "non-issues in practice" fall also into this category, like > issues only "exploitable" if the code in question is setuid root, exploits > which only work if someone already has administrative privileges or similar. > This severity is also used for vulnerabilities in packages which are not > covered by security support. We have a problem in the way we assess CVEs when the generated package is not affected but the source code contains the vulnerability. Our current process is to set "no-dsa" and lower the severity to "unimportant", although it's also possible that in some cases people are making use of "ignored", which represents "won't fix". The result is that "unimportant/no-dsa" CVEs can mean two things: 1) We are affected but we the severity is too low, eg.: packages not covered by security support, the CVE is considered a non-issue by our security-team but we are still affected... 2) We are definitely not affected since we don't build that feature of the software or we have hardening in place which prevents this from being exploited. This leads to our users, who are interested in knowing which CVEs affect their systems, having to check the notes of every CVE on security-tracker to filter-out the false-positives. Besides that, we also struggle with this ourselves, as someone who would like to fix CVEs will have to filter-out these false-positives themselves. Considering the broad usage of Debian (especially on containers), being able to correctly mark these cases as not affecting the binary packages will have a huge impact on all of the industry. I'm not being over-optimistic here, a lot of effort ends up being spent on generating CVE reports and then having to justify why each one is not fixed. Whether the requirements around CVE fixing are right or wrong is a story of its own, but we have the potential to make ourselves and our users' lives easier with this. ## Proposed solution I propose that we start setting CVEs to "not-affected" when the following is true for all officially supported architectures: * We don't ship the affected source package. * We don't build the affected feature. * We have hardening which makes the exploit impossible (only in the cases when there's no doubt about it). I
Re: security-tracker: A proposal to significantly reduce reported false-positives (no affected-code shipped)
Hello everyone, Just wondering if the Security team could spend some time availiating my proposal. Feedback from others is always welcomed too, but in order to go ahead I would like to understand where the team stands. Cheers, -- Samuel Henrique
Re: security-tracker: A proposal to significantly reduce reported false-positives (no affected-code shipped)
Hello everyone, I've done some small updates to the proposal, mostly improving readability and making my suggestion more clear. v2 below: I would like to propose something which will lower the amount of reported false-positive CVEs to our users by about 20%. # tl;dr We don't have a unique way of stating that a CVE does not affect us when we don't build the affected package's feature or hardening blocks exploits. This leads to our users being required to manually distinguish which CVEs affect them and which don't. I propose we mark those cases as not-affected. Alternatively, I mention an option to create a new state to indicate that the resulting package is not affected due to the build options. I also explain why that's not my prefered approach. # Problem statement The possible outcomes of a CVE assessment in our security-tracker are[0]: > | | | | | | > We also have the following severity levels [0]: > SEVERITY_LEVEL : (unimportant) | (low) | (medium) | (high) "unimportant" being defined as: > unimportant: This problem does not affect the Debian binary package, e.g., a > vulnerable source file, which is not built, a vulnerable file in > doc/foo/examples/, PHP Safe mode bugs, path disclosure (doesn't matter on > Debian). All "non-issues in practice" fall also into this category, like > issues only "exploitable" if the code in question is setuid root, exploits > which only work if someone already has administrative privileges or similar. > This severity is also used for vulnerabilities in packages which are not > covered by security support. We have a problem in the way we assess CVEs when the generated package is not affected (but the source code contains the vulnerability). Our current process is to set "no-dsa" and lower the severity to "unimportant", although it's also possible that in some cases people are making use of "ignored", which represents "won't fix". The result is that "unimportant/no-dsa" CVEs can mean two things: 1) We are affected but we the severity is too low, eg.: packages not covered by security support, the CVE is considered a non-issue by our security-team but we are still affected... 2) We are definitely not affected since we don't build that feature of the software or we have hardening in place which prevents this from being exploited. This leads to our users, who are interested in knowing which CVEs affect their systems, having to check the notes of every CVE on security-tracker to filter-out the false-positives. # Proposed solution I propose that we start setting CVEs to not-affected also when the following is true for all officially supported architectures: * We don't ship the affected source package. * We don't build the affected feature. * We have hardening which makes the exploit impossible (only in the cases when there's no doubt about it). If we still want to flag the cases where a build with different flags might change that assertion, we can use the "(free text comment)" section of the NOTES[0] to mention it. Effectively this proposal means I would push an MR updating the documentation at [0] and start changing those CVEs to not-affected. I'm not asking for anyone to do the work. # Stats As a way of sampling the impact of this issue, I've done a high-level check on how many sets of affected package-CVE we have in our debian:stable docker image[1]. Out of the 82 affected package/CVE pairs, 15 were clear cases of our packages not being affected. Out of the rest of those, the majority are other cases where we are reporting non-issues, but those require a deeper investigation so I don't want to assume they also fall under this case. So 18% of the reported affected packages are false-positives. Based on what I've seen, I believe this is a fair estimate to extrapolate. I've listed some examples to this issue at [2]. # Alternative solution If using the "free text comment"[0] is not a good enough way of stating that only the source contains the vulnerable code: ## A1) Add a new sub-state "only-source-vulnerable", to be used in addition to "not-affected" ## A2) Add a new mutually exclusive state to the set: "not-affected-build-artifacts" I don't like these approaches because they increase the complexity of our process (a new state is more costly than a free text mention) where there's not a clear benefit/motivation. What's the value in saying the sources carry the vulnerable code? If someone does their own modified build of a package, all bets are off and that's not an official package. It should also be mentioned that identifying cases where only the source-code is vulnerable will never be done perfectly due to how easy it is to miss a bundled library which is not used. For example, rsync bundles zlib and we do not set rsync as affected for all zlib CVEs (rsync does not use the bundled lib), would we like otherwise to be the case? Coming up with a new state is confusing as systems/people reading that might end up parsing it as "affected". S
Re: security-tracker: A proposal to significantly reduce reported false-positives (no affected-code shipped)
* [Wed, Apr 03, 2024 at 11:11:20PM +0100] Samuel Henrique: On the proposed solution I also mention that we can use the "(free text comment)" section to indicate that, while sticking to "not-affected", this would simplify things as no new value is needed. But parsing the cases where only the sources contain the vulnerable code might be a bit harder. Not only it's the parsing harder, but it also is a "lesser" warning than an "affected" status. I'm curious though as to what is the usecase of that, no other Linux distribution specifies the case where only the source carries the vulnerability. My impression is that Debian currently does, even if imperfectly, by marking the package as vulnerable and setting the unimportant bit. What would be the need for this as a user? If this is a need you have, could you clarify it, please? Definitively it isn't a need, I would call it an expectation. I used to recompile a lot of Debian packages, usually for backporting, and I guess I've always assumed that a package marked not-vulnerable would not bring the vulnerability back when, e.g., linked against a previous version of a library. Or, e.g., I would not consider not-vulnerable a package shipping a malicious example script. But I concede that creating a binary-only tag has its own issues. For example, a vulnerability could only affect some architectures, and that means you should now differentiate not only per package name and "form" (source or binary), but also per architecture. Cheers, Gian Piero.
Re: security-tracker: A proposal to significantly reduce reported false-positives (no affected-code shipped)
On Wed, 3 Apr 2024 at 17:04, Gian Piero Carrubba wrote: > > * [Wed, Apr 03, 2024 at 09:21:41AM +0100] Samuel Henrique: > ># Alternative solutions: > >If we really want to distinguish the case when we don't produce any affected > >packages but the source contains the vulnerability (a build with different > >flags might result in an affected package), we can create a new tag to show > >this: not-affected-build-artifacts. > > This. Just marking the CVE as not-affected does not distinguish between > deb and deb-src, that are still part of (and shipped by) Debian. On the proposed solution I also mention that we can use the "(free text comment)" section to indicate that, while sticking to "not-affected", this would simplify things as no new value is needed. But parsing the cases where only the sources contain the vulnerable code might be a bit harder. I'm curious though as to what is the usecase of that, no other Linux distribution specifies the case where only the source carries the vulnerability. What would be the need for this as a user? If this is a need you have, could you clarify it, please? Regards, -- Samuel Henrique
Re: security-tracker: A proposal to significantly reduce reported false-positives (no affected-code shipped)
* [Wed, Apr 03, 2024 at 09:21:41AM +0100] Samuel Henrique: # Alternative solutions: If we really want to distinguish the case when we don't produce any affected packages but the source contains the vulnerability (a build with different flags might result in an affected package), we can create a new tag to show this: not-affected-build-artifacts. This. Just marking the CVE as not-affected does not distinguish between deb and deb-src, that are still part of (and shipped by) Debian. Cheers, Gian Piero.
Fw: security-tracker: A proposal to significantly reduce reported false-positives (no affected-code shipped)
-- Forwarded message -- From: Samuel Henrique <samuel...@debian.org> Date: On Wed, Apr 3, 2024 at 3:21 AM Subject: Fw: security-tracker: A proposal to significantly reduce reported false-positives (no affected-code shipped) To: <debian-security@lists.debian.org> Cc: Hello everyone, I would like to propose something which will lower the amount of reported false-positive CVEs to our users. # tl;dr We don't have a unique way of stating a CVE does not affect us when we don't build the affected package's feature or hardening blocks exploits, this leads to our users being required to manually distinguish which CVEs affect them and which don't. I propose we mark those cases as not-affected, there's an option to create a new state to indicate that the resulting package is not affected due to the build options. # Problem statement The possible outcomes of a CVE assessment in our security-tracker are[0]: > <no-dsa> | <unfixed> | <undetermined> | <not-affected> | <itp> We also have the following severity levels [0]: > SEVERITY_LEVEL : (unimportant) | (low) | (medium) | (high) "unimportant" being defined as: > unimportant: This problem does not affect the Debian binary package, e.g., a > vulnerable source file, which is not built, a vulnerable file in > doc/foo/examples/, PHP Safe mode bugs, path disclosure (doesn't matter on > Debian). All "non-issues in practice" fall also into this category, like > issues only "exploitable" if the code in question is setuid root, exploits > which only work if someone already has administrative privileges or similar. > This severity is also used for vulnerabilities in packages which are not > covered by security support. We have a problem in the way we assess CVEs when the generated package is not affected (but the source code contains the vulnerability). Our current process is to set "no-dsa" and lower the severity to "unimportant". The result is that "unimportant/no-dsa" CVEs can mean two things: 1) We are affected but we the severity is too low, eg.: packages not covered by security support, the CVE is considered a non-issue by our security-team but we are still affected... 2) We are definitely not affected since we don't build that feature of the software or we have hardening in place which prevents this from being exploited. This leads to our users, who are interested in knowing which CVEs affect their systems, having to check the notes of every CVE on security-tracker to filter-out the false-positives. # Proposed solution: I propose that we start setting CVEs to not-affected also when: * We don't ship the affected source package * We don't build the affected feature * We have hardening which makes the exploit impossible (only in the cases when there's no doubt about it). If we still want to flag the cases where a build with different flags might change that, we can use the "(free text comment)" section of the NOTES[0] to mention it. # Some stats: As a way of sampling the impact of this issue, I've done a high-level check on how many sets of affected package-CVE we have in our debian:stable docker image[1]. Out of the 82 affected package/CVE pairs, 15 were clear cases of our packages not being affected. Out of the rest of those, the majority are other cases where we are reporting non-issues, but those require a deeper investigation so I don't want to assume they also fall under this case. So 18% of the reported affected packages are false-positives. Based on what I've seen, I believe this is a fair estimate to extrapolate. I've listed some examples to this issue[2]. # Alternative solutions: If we really want to distinguish the case when we don't produce any affected packages but the source contains the vulnerability (a build with different flags might result in an affected package), we can create a new tag to show this: not-affected-build-artifacts. [0] https://security-team.debian.org/security_tracker.html#summary-of-tracker-syntax [1] $ grype debian:stable [2] https://security-tracker.debian.org/tracker/CVE-2011-3374 [2] https://security-tracker.debian.org/tracker/CVE-2022-0563 [2] https://security-tracker.debian.org/tracker/CVE-2017-18018 [2] https://security-tracker.debian.org/tracker/CVE-2019-19882 [2] https://security-tracker.debian.org/tracker/CVE-2023-28320 Cheers, -- Samuel Henrique <samueloph> signature.asc Description: OpenPGP digital signature
security-tracker: A proposal to significantly reduce reported false-positives (no affected-code shipped)
Hello everyone, I would like to propose something which will lower the amount of reported false-positive CVEs to our users. # tl;dr We don't have a unique way of stating a CVE does not affect us when we don't build the affected package's feature or hardening blocks exploits, this leads to our users being required to manually distinguish which CVEs affect them and which don't. I propose we mark those cases as not-affected, there's an option to create a new state to indicate that the resulting package is not affected due to the build options. # Problem statement The possible outcomes of a CVE assessment in our security-tracker are[0]: > | | | | We also have the following severity levels [0]: > SEVERITY_LEVEL : (unimportant) | (low) | (medium) | (high) "unimportant" being defined as: > unimportant: This problem does not affect the Debian binary package, e.g., a > vulnerable source file, which is not built, a vulnerable file in > doc/foo/examples/, PHP Safe mode bugs, path disclosure (doesn't matter on > Debian). All "non-issues in practice" fall also into this category, like > issues only "exploitable" if the code in question is setuid root, exploits > which only work if someone already has administrative privileges or similar. > This severity is also used for vulnerabilities in packages which are not > covered by security support. We have a problem in the way we assess CVEs when the generated package is not affected (but the source code contains the vulnerability). Our current process is to set "no-dsa" and lower the severity to "unimportant". The result is that "unimportant/no-dsa" CVEs can mean two things: 1) We are affected but we the severity is too low, eg.: packages not covered by security support, the CVE is considered a non-issue by our security-team but we are still affected... 2) We are definitely not affected since we don't build that feature of the software or we have hardening in place which prevents this from being exploited. This leads to our users, who are interested in knowing which CVEs affect their systems, having to check the notes of every CVE on security-tracker to filter-out the false-positives. # Proposed solution: I propose that we start setting CVEs to not-affected also when: * We don't ship the affected source package * We don't build the affected feature * We have hardening which makes the exploit impossible (only in the cases when there's no doubt about it). If we still want to flag the cases where a build with different flags might change that, we can use the "(free text comment)" section of the NOTES[0] to mention it. # Some stats: As a way of sampling the impact of this issue, I've done a high-level check on how many sets of affected package-CVE we have in our debian:stable docker image[1]. Out of the 82 affected package/CVE pairs, 15 were clear cases of our packages not being affected. Out of the rest of those, the majority are other cases where we are reporting non-issues, but those require a deeper investigation so I don't want to assume they also fall under this case. So 18% of the reported affected packages are false-positives. Based on what I've seen, I believe this is a fair estimate to extrapolate. I've listed some examples to this issue[2]. # Alternative solutions: If we really want to distinguish the case when we don't produce any affected packages but the source contains the vulnerability (a build with different flags might result in an affected package), we can create a new tag to show this: not-affected-build-artifacts. [0] https://security-team.debian.org/security_tracker.html#summary-of-tracker-syntax [1] $ grype debian:stable [2] https://security-tracker.debian.org/tracker/CVE-2011-3374 [2] https://security-tracker.debian.org/tracker/CVE-2022-0563 [2] https://security-tracker.debian.org/tracker/CVE-2017-18018 [2] https://security-tracker.debian.org/tracker/CVE-2019-19882 [2] https://security-tracker.debian.org/tracker/CVE-2023-28320 Cheers, -- Samuel Henrique
