Re: [PSF-Community] Dangerous PyPI packages and PSF

[Nick Coghlan]

On 5 May 2017 at 14:10, Gregory P. Smith  wrote:
> This is not a solvable problem. IMNSHO We should never attempt to implement
> pre screening of packages.
>
> It is a good post-package-upload task for someone to try and do as a
> research project.
>
> Automated code scanning can only find already known things and similar
> signatures (at which point it can have false positives) and we aren't just
> talking about obfuscated source code.  PyPI hosts binary wheels made using
> unreproduceable build processes on untrusted machines created from
> unverifiable inputs.  Scanning services such as Google's
> https://www.virustotal.com/en/about/ exist but I'm not sure that'd be of
> much value to PyPI.

Red Hat's approach to this (https://github.com/fabric8-analytics/)
relies heavily on "popularity within your cohort" as a proxy for
safety. It's far from being a perfect approach (since there's still a
risk of the "bystander effect" coming into play, where everyone
assumes everyone else is handling the security audits), but it at
least gives people a heads up when they're doing something relatively
unusual and hence may want to take more care and treat their potential
dependencies with a bit more suspicion.

Cheers,
Nick.

P.S. Full disclosure: until I switched teams a few months ago, working
on fabric8-analytics (and its precursor projects) was my day job at
Red Hat. As far as I'm aware, the current version still doesn't take
the raw PyPI Big Query download data into account, but it does track
component usage across public GitHub repositories - the benefit of
focusing on the latter is that it gives you co-occurence information
(i.e. "component X is often used in combination with component Y"),
rather than the raw popularity metrics offered by the download numbers
(which can also be heavily skewed by artifact caches, and the lack
thereof, in automated build and test pipelines).

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
PSF-Community mailing list
PSF-Community@python.org
https://mail.python.org/mailman/listinfo/psf-community

Re: [PSF-Community] Dangerous PyPI packages and PSF

[Nick Coghlan]

On 5 May 2017 at 09:41, Bruno Rocha  wrote:
> Hi,
>
> I just read this on reddit[0], a thread asking if PyPI packages are audited
> and somebody pointed the `python-nation`[1] which is a harmful and useless
> module, installing itself and sending the `/etc/passwd` content to external
> endpoint.
>
> The app receiving the data is hosted at http://python-nation.herokuapp.com

This is something that Jacob Kaplan-Moss wrote for a PyCon Australia
security lightning talk a few years back:
https://www.youtube.com/watch?list=PLs4CJRBY5F1KDIN6pv6daYWN_RnFOYvt0=player_detailpage=daVHCUHtOZ4#t=1819s

That talk was prompted by a similar social engineering exercise
carried out in the Ruby community:
http://blog.honeybadger.io/stop-using-rubygemsorg-in-production/

> and as the PSF mission [2] says
>
> The mission of the Python Software Foundation is to promote, protect, and
> advance the Python programming language
>
> I wonder if there are some workgroup at PSF to handle this? and not only the
> specific case of `python-nation` which should be deleted and the user banned
> maybe,

python-nation does not violate PyPI's Terms of Service. However, it
does provide a useful reminder to end users that mistakenly view PyPI
as a restricted app store rather than as an open publication platform
akin to the web itself that "pip install " is
essentially no safer than "curl  | sh" (although
it does offer greater assurances that if you pin your dependencies to
particular versions, future downloads will either get you the same
thing, or else fail outright).

> But also to handle the audit of other packages?

When people and organisations want security audits of open source
software, they either have to do them themselves, pay someone else to
do them on their behalf, or else rely on one of the volunteer-driven
collaborative software auditing projects more commonly known as
"community Linux distributions" (accepting the couple of orders of
magnitude reduction in available components that comes from that last
choice).

Most large organisations will end up relying on some combination of
the three (e.g. it's not uncommon for a RHEL-hosted application to
include commercially audited packages from Red Hat and certified
partners, community audited packages from EPEL, IUS, Fedora COPR,
and/or softwarecollection.org, and team audited packages directly from
PyPI, and we see the same kinds of layered architectures showing up
regardless of which distro or platform people target).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
PSF-Community mailing list
PSF-Community@python.org
https://mail.python.org/mailman/listinfo/psf-community

Re: [PSF-Community] Dangerous PyPI packages and PSF

[Noah Kantrowitz]

> On May 4, 2017, at 4:41 PM, Bruno Rocha  wrote:
> 
> Hi,
> 
> I just read this on reddit[0], a thread asking if PyPI packages are audited 
> and somebody pointed the `python-nation`[1] which is a harmful and useless 
> module, installing itself and sending the `/etc/passwd` content to external 
> endpoint.
> 
> The app receiving the data is hosted at http://python-nation.herokuapp.com
> 
> and as the PSF mission [2] says
> 
> The mission of the Python Software Foundation is to promote, protect, and 
> advance the Python programming language
> 
> I wonder if there are some workgroup at PSF to handle this? and not only the 
> specific case of `python-nation` which should be deleted and the user banned 
> maybe, But also to handle the audit of other packages?
> 
> 
> [0] 
> https://www.reddit.com/r/Python/comments/697da2/does_pypi_review_code_thats_uploaded/
> [1] 
> https://www.reddit.com/r/Python/comments/697da2/does_pypi_review_code_thats_uploaded/dh4uyf8/
> [2] https://www.python.org/psf/mission/

Specifically re: the vector of running code at install time, wheels can help 
with this though I don't think there is a good way to tell pip to ignore 
non-wheel builds. But even then, the whole point is that you're downloading 
code from the internet :) If you want to discuss this further I recommend the 
distutils-sig mailing list.

--Noah




signature.asc
Description: Message signed with OpenPGP
___
PSF-Community mailing list
PSF-Community@python.org
https://mail.python.org/mailman/listinfo/psf-community

Re: [PSF-Community] Dangerous PyPI packages and PSF

[Ryan Birmingham]

I'm not sure what effective package review would look like here. Perhaps we
could establish an entity to screen packages on an opt-in basis, but I
don't know if we have the resources/people for this. Automated code
screening could and probably would miss the python nation example due to
the unorthodox use of compressed instructions.
Does anyone have any ideas?

-Ryan Birmingham

On 4 May 2017 at 20:08, Bruno Rocha  wrote:

> Interesting detail, the mentioned package https://pypi.python.
> org/pypi/python-nation/1.0.1 was created and uploaded by
> Jacob Kaplan Moss, so I guess this is intended to be a POC, to show PyPI
> vulnerabilities or some Infosec experiment.
>
> On Thu, May 4, 2017 at 8:41 PM, Bruno Rocha  wrote:
>
>> Hi,
>>
>> I just read this on reddit[0], a thread asking if PyPI packages are
>> audited and somebody pointed the `python-nation`[1] which is a harmful and
>> useless module, installing itself and sending the `/etc/passwd` content to
>> external endpoint.
>>
>> The app receiving the data is hosted at http://python-nation.heroku
>> app.com
>>
>> and as the PSF mission [2] says
>>
>> The mission of the Python Software Foundation is to promote, protect, and
>> advance the Python programming language
>>
>> I wonder if there are some workgroup at PSF to handle this? and not only
>> the specific case of `python-nation` which should be deleted and the user
>> banned maybe, But also to handle the audit of other packages?
>>
>>
>> [0] https://www.reddit.com/r/Python/comments/697da2/does_pyp
>> i_review_code_thats_uploaded/
>> [1] https://www.reddit.com/r/Python/comments/697da2/does_pyp
>> i_review_code_thats_uploaded/dh4uyf8/
>> [2] https://www.python.org/psf/mission/
>>
>>
>> Cheers,
>>
>> --
>>
>> *Bruno Rocha - @rochacbruno *
>> http://brunorocha.org
>>
>>
>
>
> --
>
> *Bruno Rocha - @rochacbruno *
> http://brunorocha.org
>
>
> ___
> PSF-Community mailing list
> PSF-Community@python.org
> https://mail.python.org/mailman/listinfo/psf-community
>
>
___
PSF-Community mailing list
PSF-Community@python.org
https://mail.python.org/mailman/listinfo/psf-community

Re: [PSF-Community] Dangerous PyPI packages and PSF

[Bruno Rocha]

Interesting detail, the mentioned package
https://pypi.python.org/pypi/python-nation/1.0.1 was created and uploaded
by Jacob Kaplan Moss, so I guess this is intended to be a POC, to show PyPI
vulnerabilities or some Infosec experiment.

On Thu, May 4, 2017 at 8:41 PM, Bruno Rocha  wrote:

> Hi,
>
> I just read this on reddit[0], a thread asking if PyPI packages are
> audited and somebody pointed the `python-nation`[1] which is a harmful and
> useless module, installing itself and sending the `/etc/passwd` content to
> external endpoint.
>
> The app receiving the data is hosted at http://python-nation.herokuapp.com
>
> and as the PSF mission [2] says
>
> The mission of the Python Software Foundation is to promote, protect, and
> advance the Python programming language
>
> I wonder if there are some workgroup at PSF to handle this? and not only
> the specific case of `python-nation` which should be deleted and the user
> banned maybe, But also to handle the audit of other packages?
>
>
> [0] https://www.reddit.com/r/Python/comments/697da2/does_pyp
> i_review_code_thats_uploaded/
> [1] https://www.reddit.com/r/Python/comments/697da2/does_pyp
> i_review_code_thats_uploaded/dh4uyf8/
> [2] https://www.python.org/psf/mission/
>
>
> Cheers,
>
> --
>
> *Bruno Rocha - @rochacbruno *
> http://brunorocha.org
>
>


-- 

*Bruno Rocha - @rochacbruno *
http://brunorocha.org
___
PSF-Community mailing list
PSF-Community@python.org
https://mail.python.org/mailman/listinfo/psf-community

[PSF-Community] Dangerous PyPI packages and PSF

[Bruno Rocha]

Hi,

I just read this on reddit[0], a thread asking if PyPI packages are audited
and somebody pointed the `python-nation`[1] which is a harmful and useless
module, installing itself and sending the `/etc/passwd` content to external
endpoint.

The app receiving the data is hosted at http://python-nation.herokuapp.com

and as the PSF mission [2] says

The mission of the Python Software Foundation is to promote, protect, and
advance the Python programming language

I wonder if there are some workgroup at PSF to handle this? and not only
the specific case of `python-nation` which should be deleted and the user
banned maybe, But also to handle the audit of other packages?


[0] https://www.reddit.com/r/Python/comments/697da2/does_
pypi_review_code_thats_uploaded/
[1] https://www.reddit.com/r/Python/comments/697da2/does_
pypi_review_code_thats_uploaded/dh4uyf8/
[2] https://www.python.org/psf/mission/


Cheers,

-- 

*Bruno Rocha - @rochacbruno *
http://brunorocha.org
___
PSF-Community mailing list
PSF-Community@python.org
https://mail.python.org/mailman/listinfo/psf-community