Re: pkg audit false negatives

[Roger Marquis]

That leaves just unpackaged base as FreeBSD's remaining audit weakness.


Hi, I am happy that I can reduce your worry factor a bit ;-)

Can you share what the audit weakness is? freebsd-update cron checks
whether or not an update is available and then emails you. If you run
-RELEASE, then that means that either an EN or SA had been released..


Can you run freebsd-update on a -RELEASE system installed and maintained
with buildworld/buildkernel/installkernel/installworld?

Though it's been more than a year since the last time I tested
freebsd-update, on Virtualbox VMs, it resulted in too many bricked
systems to rely on.  That may have changed but it would still be better
to build a packaged base or have reproduceable builds as lighter-weight
solutions to the base audit issue.

Roger
___
freebsd-security@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-security
To unsubscribe, send any mail to "freebsd-security-unsubscr...@freebsd.org"

Re: pkg audit false negatives

[Remko Lodder]

> On 14 Aug 2017, at 05:32, Roger Marquis  wrote:
> 
>> I do not think that holds:
>> 
>> 
>> 17521php -- multiple vulnerabilities
>> 17522
>> 17523  
>> 17524php55
>> 175255.5.38
>> 17526  
>> 
>> This is an entry from svnweb, for php55, which was added in 2016(07-26).
>> 
>> So this entry is there. Thus it did not disappear from VuXML at least.
> 
> You are right Remko.  It looks like there was a policy or at least a
> practice change about a year ago.  Even have an archived email from
> Gerhard Schmidt who first noticed it back in Aug 2016.  My fault for not
> doing sufficient fact rechecking,
> 
> So we are safe from false negatives after all.  Hurray, I can stop
> relying on pkg-version (for this).
> 
> That leaves just unpackaged base as FreeBSD's remaining audit weakness.

Hi, I am happy that I can reduce your worry factor a bit ;-)

Can you share what the audit weakness is? freebsd-update cron checks
whether or not an update is available and then emails you. If you run
-RELEASE, then that means that either an EN or SA had been released..

Cheers
Remko


> 
> Roger
> 
> 
> ___
> freebsd-security@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-security
> To unsubscribe, send any mail to "freebsd-security-unsubscr...@freebsd.org"



signature.asc
Description: Message signed with OpenPGP

Re: pkg audit false negatives

[Roger Marquis]

I do not think that holds:


17521   php -- multiple vulnerabilities
17522   
17523 
17524   php55
17525   5.5.38
17526 

This is an entry from svnweb, for php55, which was added in 2016(07-26).

So this entry is there. Thus it did not disappear from VuXML at least.


You are right Remko.  It looks like there was a policy or at least a
practice change about a year ago.  Even have an archived email from
Gerhard Schmidt who first noticed it back in Aug 2016.  My fault for not
doing sufficient fact rechecking,

So we are safe from false negatives after all.  Hurray, I can stop
relying on pkg-version (for this).

That leaves just unpackaged base as FreeBSD's remaining audit weakness.

Roger


___
freebsd-security@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-security
To unsubscribe, send any mail to "freebsd-security-unsubscr...@freebsd.org"

Re: pkg audit false negatives

[Remko Lodder]

> On 12 Aug 2017, at 02:37, Roger Marquis  wrote:
> 
> On Fri, 11 Aug 2017, Remko Lodder wrote:
> 
>> If an entry is removed from the ports/pkg tree?s and it is also removed
>> from VuXML, then yes, it will no longer get marked in your local
>> installation. That?s a bit of a chicken and egg basically. Although I do
>> not recall that it ever happened that ports that are no longer there, are
>> removed from VuXML as well. (And I follow that since 2004).
>> Do you have a more concrete example that we can dive into to see what is
>> going on/going wrong?
> 
> Should be able to find missing vulxml entries for most anything that has
> been deprecated from the ports tree but most of the ones I've seen are
> for web programming languages, particularly php.

I do not think that holds:


17521   php -- multiple vulnerabilities
17522   
17523 
17524   php55
17525   5.5.38
17526 

This is an entry from svnweb, for php55, which was added in 2016(07-26).

So this entry is there. Thus it did not disappear from VuXML at least.

Can you show such a packet from your local installation(s) and present a
``pkg audit -F`` along side it. I would also like to see a detailed pkg info
from the affected pkg.

Thanks a lot in advance,
Remko

> 
> For example when php5X was dropped it also disappeared from vulxml, with
> no small number of servers still using it.  If those sites depended on
> pkg-audit to tell them they had a vulnerability, well, they were out of
> luck.  There was no warning, no error, no disclaimer, pkg-audit did and
> still does nothing different than it would for a non-vulnerable port or
> package.
> 
> There may be more vulnerabilities in the wild from non-packaged base as
> it is larger but at least people are working on that.  Pkg-audit
> tracking of installed but deprecated ports OTOH, seems to have fallen
> through the cracks.  Even the FreeBSD Foundation and the ports-security
> teams appear to be ignoring this issue.
> 
> Roger Marquis



signature.asc
Description: Message signed with OpenPGP

Re: pkg audit false negatives

[Roger Marquis]

On Fri, 11 Aug 2017, Remko Lodder wrote:


If an entry is removed from the ports/pkg tree?s and it is also removed
from VuXML, then yes, it will no longer get marked in your local
installation. That?s a bit of a chicken and egg basically. Although I do
not recall that it ever happened that ports that are no longer there, are
removed from VuXML as well. (And I follow that since 2004).

Do you have a more concrete example that we can dive into to see what is
going on/going wrong?


Should be able to find missing vulxml entries for most anything that has
been deprecated from the ports tree but most of the ones I've seen are
for web programming languages, particularly php.

For example when php5X was dropped it also disappeared from vulxml, with
no small number of servers still using it.  If those sites depended on
pkg-audit to tell them they had a vulnerability, well, they were out of
luck.  There was no warning, no error, no disclaimer, pkg-audit did and
still does nothing different than it would for a non-vulnerable port or
package.

There may be more vulnerabilities in the wild from non-packaged base as
it is larger but at least people are working on that.  Pkg-audit
tracking of installed but deprecated ports OTOH, seems to have fallen
through the cracks.  Even the FreeBSD Foundation and the ports-security
teams appear to be ignoring this issue.

Roger Marquis
___
freebsd-security@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-security
To unsubscribe, send any mail to "freebsd-security-unsubscr...@freebsd.org"

Re: pkg audit false negatives

[Remko Lodder]

> On 11 Aug 2017, at 23:47, Roger Marquis  wrote:
> 
>> It had been resolved for dovecot (it will now match both variants, since 
>> people might still have
>> the old variant of the port installed) and there is a new paragraph added to 
>> the porters handbook
>> which tells that we need to have a look at the vuxml entries.
> 
> Thanks Remko.

No problemo :)

> 
>> Hope this solves your issue,
> 
> It may for renamed ports/pkgs but doesn't appear to for deprecations.
> Once ports are dropped they do not show up in pkg-audit despite having
> been installed via pkg and/or ports.  That's the false negative that
> appears to still be a problem.

Ports / pkgs that get renamed are now changed and/or added in VuXML as well.
So the old variant and the new variant of the name’s would both be listed in 
pkg audit.

pkg audit parses VuXML, it also does a check on what is locally registered in 
it’s database.

For example if you have a/b installed. And that has a marking in VuXML : 
b
then it would hit on the package you have. If a/b gets removed for some reason, 
and it is still in VuXML
and you have it locally registered. Then it would be still be matched (or 
should).

If an entry is removed from the ports/pkg tree’s and it is also removed from 
VuXML, then yes, it will
no longer get marked in your local installation. That’s a bit of a chicken and 
egg basically. Although
I do not recall that it ever happened that ports that are no longer there, are 
removed from VuXML as
well. (And I follow that since 2004).

Do you have a more concrete example that we can dive into to see what is going 
on/going wrong?

Cheers
Remko


> 
> Roger



signature.asc
Description: Message signed with OpenPGP

Re: pkg audit false negatives

[Roger Marquis]

It had been resolved for dovecot (it will now match both variants, since people 
might still have
the old variant of the port installed) and there is a new paragraph added to 
the porters handbook
which tells that we need to have a look at the vuxml entries.


Thanks Remko.


Hope this solves your issue,


It may for renamed ports/pkgs but doesn't appear to for deprecations.
Once ports are dropped they do not show up in pkg-audit despite having
been installed via pkg and/or ports.  That's the false negative that
appears to still be a problem.

Roger
___
freebsd-security@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-security
To unsubscribe, send any mail to "freebsd-security-unsubscr...@freebsd.org"

Re: pkg audit false negatives

[Remko Lodder]

Hi Roger,

> On 11 Aug 2017, at 17:14, Remko Lodder  wrote:
> 
> Hi Roger,
> 
>> On 11 Aug 2017, at 04:41, Roger Marquis  wrote:
>> 
>> In the past pkg-audit and even pkg-version have not been reliable tools
>> where installed ports or packages have been subsequently discontinued or
>> renamed.  Today, however, I notice that dovecot2 is still showing up in
>> the output of pkg-version despite the port having been renamed to
>> dovecot (without the numeric suffix) several days ago.
> 

It had been resolved for dovecot (it will now match both variants, since people 
might still have
the old variant of the port installed) and there is a new paragraph added to 
the porters handbook
which tells that we need to have a look at the vuxml entries.

Hope this solves your issue,
Remko



signature.asc
Description: Message signed with OpenPGP

Re: pkg audit false negatives

[Remko Lodder]

Hi Roger,

> On 11 Aug 2017, at 04:41, Roger Marquis  wrote:
> 
> In the past pkg-audit and even pkg-version have not been reliable tools
> where installed ports or packages have been subsequently discontinued or
> renamed.  Today, however, I notice that dovecot2 is still showing up in
> the output of pkg-version despite the port having been renamed to
> dovecot (without the numeric suffix) several days ago.

Yes, there is a difference between renaming a port, and renaming the vuxml 
(which is the
database behind pkg audit etc.) entries. The entries are listed as ‘dovecot2-*’ 
there and
when renaming a port these entries should ideally be renamed too.

It seems that that was not under consideration at the name change moment(s).

I’ll try to look into this (starting by prodding the person(s) who did the 
rename) and asking them
to rename the entries in vuxml as well.

> 
> Does this mean there has been a policy change?  If so does it cover
> pkg-audit as well?

There had been no policy change. The application backend is just matching on 
what
was recorded at the moment it was added.

Thanks for the notification though, we should add that to the porters-handbook.

Cheers
REmko

> 
> Roger
> ___
> freebsd-security@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-security
> To unsubscribe, send any mail to "freebsd-security-unsubscr...@freebsd.org"



signature.asc
Description: Message signed with OpenPGP

pkg audit false negatives

[Roger Marquis]

In the past pkg-audit and even pkg-version have not been reliable tools
where installed ports or packages have been subsequently discontinued or
renamed.  Today, however, I notice that dovecot2 is still showing up in
the output of pkg-version despite the port having been renamed to
dovecot (without the numeric suffix) several days ago.

Does this mean there has been a policy change?  If so does it cover
pkg-audit as well?

Roger
___
freebsd-security@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-security
To unsubscribe, send any mail to "freebsd-security-unsubscr...@freebsd.org"

Re: pkg audit false negatives (was: Perl upgrade - 5.20.x vulnerable)

[Mark Felder]

On Tue, Aug 16, 2016, at 11:41, Roger Marquis wrote:
> 
> There's also an issue with older versions (perl 5.1*) no longer showing
> up in the vuln.xml at all.  I've seen perl, php and other critical
> network components still in use because the site depended on 'pkg audit'
> but did not know that expired OR deprecated ports are not audited.
> Apparently this is intentional and by policy.  IMO it is a serious flaw
> in pkg audit's design.
> 

This is hard to keep track of, but I also agree it is a problem we need
to be more conscious of. Unfortunately it adds burden to our
ports-secteam because often times upstream will EoL software and not
state if older versions are actually vulnerable. Short of a PoC or
auditing the source code of something that is EoL / removed from the
ports tree there isn't much we can do but guess. It's entirely possible
something EoL is not actually vulnerable to newer CVEs depending on if
it's in new/refactored code or shared code. Even then it's possible that
the way the code is used in the EoL version it's still not vulnerable.

> A better policy would include expired AND deprecated ports in the output
> of 'pkg audit' for at least a year after they are removed from the ports
> and/or pkg trees.  If a port had no known vulnerability when removed it
> should at least indicate 'no longer audited' in place of 'vulnerable'.
> 

Yes, a solution to create vuxml entries for EoL/removed ports with a
simple statement that they're EoL and could be vulnerable is a sane
approach. This way we aren't incorrectly listing software as vulnerable
to CVEs we haven't validated. I welcome this, but we will need help from
users and the community to let us know when this happens as we aren't
always aware of the EoL schedules of software in the ports tree.


> This is, IMO, one of 3 remaining weaknesses in the otherwise excellent
> freebsd audit framework.  The other two issues have to do with base not
> being packaged (so not really being 'audit'able) and the 'general rule'
> announced on Aug 10 that 'the FreeBSD Security Officer does not announce
> vulnerabilities for which there is no released patch'.  This is
> particularly problematic as there are usually mitigations that do not
> require patches.
> 

I already solved your #2 problem:
https://blog.feld.me/posts/2016/08/monitoring-freebsd-base-system-vulnerabilities-with-pkg-audit/

#3 is being reviewed by secteam/core, so I think we're well on our way
to solving these concerns.


-- 
  Mark Felder
  ports-secteam member
  f...@freebsd.org
___
freebsd-security@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-security
To unsubscribe, send any mail to "freebsd-security-unsubscr...@freebsd.org"

pkg audit false negatives (was: Perl upgrade - 5.20.x vulnerable)

[Roger Marquis]

On 16 Aug 2016, JosC wrote:

In the absence of running 'pkg audit -F', only
the"LOCALBASE/periodic/security/410.pkg-audit script updates the vuxml
file and audit results. Until that happens, or pkg audit -F is run, pkg
will still see an older version.


Thinking with you I now ask myself:
- Would it be a good idea to make this vuxml file update part of the 
Makefile? Then these occurrences won't happen anymore


There's also an issue with older versions (perl 5.1*) no longer showing
up in the vuln.xml at all.  I've seen perl, php and other critical
network components still in use because the site depended on 'pkg audit'
but did not know that expired OR deprecated ports are not audited.
Apparently this is intentional and by policy.  IMO it is a serious flaw
in pkg audit's design.

A better policy would include expired AND deprecated ports in the output
of 'pkg audit' for at least a year after they are removed from the ports
and/or pkg trees.  If a port had no known vulnerability when removed it
should at least indicate 'no longer audited' in place of 'vulnerable'.

This is, IMO, one of 3 remaining weaknesses in the otherwise excellent
freebsd audit framework.  The other two issues have to do with base not
being packaged (so not really being 'audit'able) and the 'general rule'
announced on Aug 10 that 'the FreeBSD Security Officer does not announce
vulnerabilities for which there is no released patch'.  This is
particularly problematic as there are usually mitigations that do not
require patches.

Roger Marquis
___
freebsd-security@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-security
To unsubscribe, send any mail to "freebsd-security-unsubscr...@freebsd.org"