[issue9216] FIPS support for hashlib

[miss-islington]

miss-islington  added the comment:


New changeset a08b7c3bb0ef9da32400d23b13f78245cd7a9541 by Miss Islington (bot) 
in branch '3.9':
bpo-9216: hashlib usedforsecurity fixes (GH-20258)
https://github.com/python/cpython/commit/a08b7c3bb0ef9da32400d23b13f78245cd7a9541


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[miss-islington]

Change by miss-islington :


--
pull_requests: +19589
pull_request: https://github.com/python/cpython/pull/20320

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[miss-islington]

miss-islington  added the comment:


New changeset 909b5714e1303357868bc5e281c1cf508d5d5a17 by Christian Heimes in 
branch 'master':
bpo-9216: hashlib usedforsecurity fixes (GH-20258)
https://github.com/python/cpython/commit/909b5714e1303357868bc5e281c1cf508d5d5a17


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Christian Heimes]

Change by Christian Heimes :


--
pull_requests: +19544
pull_request: https://github.com/python/cpython/pull/20258

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Christian Heimes]

Christian Heimes  added the comment:

Memo to me: Add whatsnew

--
versions: +Python 3.10, Python 3.9 -Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[STINNER Victor]

STINNER Victor  added the comment:

> _hashlib.get_fips_mode() is not compatible with new FIPS design in OpenSSL 
> 3.0.0:

I suggest to modify the code so the private function becomes unavailable in 
_hashlib on OpenSSL 3.0 and newer. What do you think?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Christian Heimes]

Christian Heimes  added the comment:

_hashlib.get_fips_mode() is not compatible with new FIPS design in OpenSSL 
3.0.0:

  The function calls 'FIPS_mode()' and 'FIPS_mode_set()' are present in OpenSSL 
3.0 but always fail. You should rewrite your application to not use them.

https://wiki.openssl.org/index.php/OpenSSL_3.0#Upgrading_from_the_OpenSSL_2.0_FIPS_Object_Module

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[miss-islington]

miss-islington  added the comment:


New changeset e3dfb9b967c560f4d094092dcae4a16fc9634681 by Victor Stinner in 
branch 'master':
bpo-9216: Expose OpenSSL FIPS_mode() as _hashlib.get_fips_mode() (GH-19703)
https://github.com/python/cpython/commit/e3dfb9b967c560f4d094092dcae4a16fc9634681


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[STINNER Victor]

STINNER Victor  added the comment:

Christian Heimes: "Instead if doing a FIPS test, users and applications should 
perform a feature test and handle the error. The approach is future-proof and 
can also cover crypto policies restriction like minimum key sizes."

Alright, I see. Thanks for your explanation ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Christian Heimes]

Christian Heimes  added the comment:

I'm against exposing the function as hashlib.get_fips_mode() because it is an 
internal implementation detail. I don't want to confuse users or make users 
think that "if hashlib.get_fips_mode()" is sufficient for feature tests. For 
starters there are multiple levels and versions of the FIPS standard like 
FIPS-140-2 and FIPS-140-3.

Instead if doing a FIPS test, users and applications should perform a feature 
test and handle the error. The approach is future-proof and can also cover 
crypto policies restriction like minimum key sizes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[STINNER Victor]

STINNER Victor  added the comment:

Petr Viktorin and Christian Heimes convinced me that it's a bad idea to expose 
OpenSSL FIPS_mode() as a public hashlib.get_fips_mode() function. It is too 
specific to OpenSSL. For example, FIPS_mode() result is a number which is 
specific to OpenSSL. Other crypto libraries are likely to use different values.

Moreover, as I wrote in my previous message, other crypto libraries expose the 
FIPS mode differently. It may not just be a global FIPS mode.

Finally, there are different FIPS modes. For example, Gcrypt has an "Enforced 
FIPS" mode.

So I modified PR 19703 to only expose FIPS_mode() as a private 
_hashlib.get_fips_mode() function. Well, as done in RHEL in fact ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[STINNER Victor]

STINNER Victor  added the comment:

I'm trying to understand how "portable" is it to expose OpenSSL FIPS_mode() as 
hashlib.get_fips_mode() which would return a boolean (True or False).

It seems like FIPS is more complex than that. Other crypto libraries which 
implement FIPS have a different way to expose FIPS mode to the consumer of the 
API:

* NSS seems to have a different API for functions in FIPS mode: 
https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Reference/NSS_cryptographic_module/FIPS_mode_of_operation
* GnuTLS provides gnutls_fips140_mode_enabled() which returns an unsigned 
integer: "return non-zero if true or zero if false"
* Gcrypt doesn't seem to expose a function to know if FIPS is enabled or not. 
It also has an "Enforced FIPS" mode:

  * https://www.gnupg.org/documentation/manuals/gcrypt/Enabling-FIPS-mode.html
  * https://www.gnupg.org/documentation/manuals/gcrypt/Enabling-FIPS-mode.html

* Bouncy Castle has a "FIPS provider": an object should be requested in FIPS 
mode

See also RHEL 8 Security Hardening documentation, "Chapter 3. Using system-wide 
cryptographic policies":
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/using-the-system-wide-cryptographic-policies_security-hardening

For my needs (skip tests which are not relevant in FIPS mode), it seems like 
keeping the function private in _hashlib.get_fips_mode() is enough.

My plan is to use it in as test.support.get_fips_mode() function which would 
return False if _hashlib.get_fips_mode() is missing.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[STINNER Victor]

STINNER Victor  added the comment:

> I'm fine with a used_for_security flag and functions to get/set FIPS state. 
> Something like hashlib.get_fips_mode() is useful for testing.

I proposed PR 19703 to expose OpenSSL FIPS_mode() as hashlib.get_fips_mode().

FIPS support was introduced in version 0.9.7 of OpenSSL and so is available in 
the minimum OpenSSL required to build Python 3.9.

LibreSSL doesn't have FIPS_mode() on purpose. Ted Unangst wrote: "I figured I 
should mention our current libressl policy wrt FIPS mode.  It's gone and it's 
not coming back."
https://marc.info/?l=openbsd-misc=139819485423701=2

My plan is to use hashlib.get_fips_mode() to skip a few tests if the FIPS mode 
is enabled. Simple example: test_crypt.test_methods() checks that 
self.assertEqual(crypt.methods[-1], crypt.METHOD_CRYPT). Except that in FIPS 
mode, METHOD_CRYPT is not available since it's too weak (3DES if I recall 
correctly). I would like to skip this test in FIPS mode.

My colleague Chalampos also plans to add a FIPS enabled buildbot running RHEL8 
to ensure that the Python test suite pass in FIPS mode, and detect regressions 
in FIPS mode.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +19021
pull_request: https://github.com/python/cpython/pull/19703

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[miss-islington]

Change by miss-islington :


--
pull_requests: +18874
pull_request: https://github.com/python/cpython/pull/19524

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[miss-islington]

miss-islington  added the comment:


New changeset 4c0a31fb08407ba043688ad1c21102dd4cb99146 by Miro Hrončok in 
branch 'master':
bpo-9216: Nobody expects the geohashing FIPS inquisition (GH-19520)
https://github.com/python/cpython/commit/4c0a31fb08407ba043688ad1c21102dd4cb99146


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Miro Hrončok]

Change by Miro Hrončok :


--
nosy: +hroncok
nosy_count: 17.0 -> 18.0
pull_requests: +18870
pull_request: https://github.com/python/cpython/pull/19520

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Cheryl Sabella]

Cheryl Sabella  added the comment:

Should this issue be closed as resolved?

--
nosy: +cheryl.sabella

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Christian Heimes]

Christian Heimes  added the comment:

> FYI the multiprocessing module uses hmac.new(authkey, message, 
> 'md5').digest(). I don't think that using MD5 even for HMAC is allowed in 
> FIPS mode.

MD5 us especially not allowed in FIPS mode.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Gregory P. Smith]

Gregory P. Smith  added the comment:


New changeset 7cad53e6b084435a220e6604010f1fa5778bd0b1 by Gregory P. Smith 
(Christian Heimes) in branch 'master':
bpo-9216: Add usedforsecurity to hashlib constructors (GH-16044)
https://github.com/python/cpython/commit/7cad53e6b084435a220e6604010f1fa5778bd0b1


--
nosy: +gregory.p.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Christian Heimes]

Change by Christian Heimes :


--
pull_requests: +15666
pull_request: https://github.com/python/cpython/pull/16044

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[STINNER Victor]

STINNER Victor  added the comment:

FYI the multiprocessing module uses hmac.new(authkey, message, 'md5').digest(). 
I don't think that using MD5 even for HMAC is allowed in FIPS mode.

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Charalampos Stratakis]

Charalampos Stratakis  added the comment:

Providing a simple keyword as a workaround for bypassing the FIPS restrictions, 
could potentially violate the standard, as there is no way from the python side 
to verify if the code in question is used for security purposes or not.

Thus I would close this issue as wontfix. Application distributors who would 
like their code to work under FIPS should preferably use the FIPS approved 
ciphers.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Charalampos Stratakis]

Change by Charalampos Stratakis :


--
pull_requests: +5374

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Charalampos Stratakis]

Charalampos Stratakis added the comment:

@Andrew

This has already been implemented downstream for RHEL and centos.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Andrew DiPrinzio]

Andrew DiPrinzio added the comment:

I have been able to use this flag on RHEL and Centos. Is it possible that this 
has been completed?

--
nosy: +Andrew DiPrinzio

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Charalampos Stratakis]

Changes by Charalampos Stratakis :


--
nosy: +cstratak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Christian Heimes]

Christian Heimes added the comment:

PS: I'm also against a hashlib.fips module in stdlib. FIPS mode is irrelevant 
for majority of users and countries. I neither want to confuse people nor 
introduce more maintenance and documentation burden than necessary. Antoine 
gave another good reason against a fips module, too.

I'm fine with a used_for_security flag and functions to get/set FIPS state. 
Something like hashlib.get_fips_mode() is useful for testing.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Christian Heimes]

Christian Heimes added the comment:

Objection from hashlib maintainer: I will reject a used_for_security flag with 
default of False. I'm slowly moving Python to a secure-by-default policy. 
Therefore used_for_security must be an explicit opt-out.

I'm aware that the policy will require modifications to all software that uses 
MD5. To be honest that's my goal. If you care about FIPS, then any use of MD5 
must be a concious and careful decision. I want developers to move away from 
MD5 and replace it with SipHash24, Blake2 or SHA-2. MD5 should *only* remain 
when backwards incompatibility prevent migration.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Yolanda]

Yolanda added the comment:

@rbtcollins, even if we go with a FIPS aware module, we'd still need to detect 
if md5 was used for security purposes.
If we build a system that detects FIPS enablement, call md5 say ... for 
generating a password, and then the python fips_md5 call is masking it, we'd be 
breaking FIPS rules.
I still see the point of the used_for_security flag. Maybe reverting the flag, 
set used_for_security to False because the normal usage of md5 shall be for 
hashes and non security stuff?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Doug Hellmann]

Doug Hellmann added the comment:

@Antoine - The idea behind introducing some API mechanism is exactly as you 
say, to let the developer say "this use of this algorithm is not related to 
security" to tell FIPS systems to not be pedantic.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Doug Hellmann]

Doug Hellmann added the comment:

@Robert, I thought you were proposing a hashlib.fips module that did not 
include md5() at all. If it does include the function, and the function does 
whatever is needed to disable the "die when using MD5" on a FIPS system, then I 
agree it would work. 

Your point about the FIPS standard changing and needing to include more hash 
types in the future is good.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Yolanda]

Yolanda added the comment:

@rbtcollins, so you mean the apps using it, shall be "fips aware" ? That will 
be the point of your separate module?
So...

if fips_enabled then
use fips.md5
else
use normal.md5

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Antoine Pitrou]

Antoine Pitrou added the comment:

AFAICT from David's patch, there isn't a new argument in all hashlib functions 
but only in the digest constructors.  Someone might want to correct me.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Ian Cordasco]

Ian Cordasco added the comment:

So I see the argument on both sides of this discussion. Having those optional 
arguments for all the functions seems like an obvious blocker. If a submodule 
is a blocker, what if we provide a context-manager to signal this?

--
nosy: +icordasc

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Antoine Pitrou]

Antoine Pitrou added the comment:

The separate module idea is an interesting one, though I wonder if it aligns 
with users' goals.  Perhaps some users simply want to set the 
OPENSSL_FORCE_FIPS_MODE environment variable and then run existing Python code 
with it to ensure that code is FIPS-compliant.  A separate module assumes that 
the developer is the one who makes the decision of running in FIPS compliance 
mode or not.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Robert Collins]

Robert Collins added the comment:

@doug - I don't see how a separate fips module *wouldn't* solve it:
 - code that uses md5 in security contexts wouldn't be able to call it from the 
fips module, which is the needed outcome
 - code that uses md5 and isn't fips compliant would be importing from the 
non-fips module, and thats as auditable as looking for a 
'usedforsecurity=False' flag
 - auditors can assume that code that doesn't use the fips module


And its way less messy: remember we're going to have this flag passed to every 
hashlib invocation from every project in order to *opt out* of the FIPS 
restrictions. Because, over time, FIPS will change, so noone can assume that 
any given function is and will remain FIPS compatible: and this flag is going 
to percolate up into e.g. the HMAC module.

I think thats pretty ugly: want to calculate the sha of a blob to look it up in 
git? sha1sum(file.read(), usedforsecurity=False)

Separately I wonder about the impact on higher layers - are they ready to be 
parameterised by objects, or do they look things up by name - and thus need to 
start accepting this new parameter and passing it down?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Yolanda]

Yolanda added the comment:

I agree with Doug. From my understanding, the intention of the patch is to 
allow the usage of md5 for non-security purposes, without being blocked by FIPS.
Right now, md5() calls running on a FIPS enabled kernel, are blocked without 
discrimination of the usage, that shall be ok for hashing purposes, and more 
performant than other methods. This patch provides the ability to continue 
using md5 just flagging it properly.

--
nosy: +yolanda.robla

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Doug Hellmann]

Doug Hellmann added the comment:

@rbcollins, I don't think providing a hashlib.fips module without md5() solves 
the problem. The idea is to have a way to call md5() in non-secure situations, 
and to signal to the FIPS system that the call is OK. A separate module would 
work if it included an md5() function that always did that signaling. But 
creating a separate module just to wrap one function like that seems like 
overkill, doesn't it?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Robert Collins]

Robert Collins added the comment:

A few thoughts;

usedforsecurity=xxx seems awkward: I wouldn't want, as a user of hashlib, to 
have to put that in literally every use I make of it.

If I understand the situation correctly, the goal is for both linters, and at 
runtime, identification of the intended purpose of a call to md5 - e.g. whether 
there are security implications in its use (as far as FIPS is concerned).

Perhaps having two separate implementations of the interfaces, one general 
purpose and one FIPS would be decent.

e.g. from hashlib.fips import sha1 
etc
etc
and hashlib.fips simply wouldn't contain md5.

Then the md5 thats in hashlib is by definition not FIPS ready and any code 
using it should be fixed.

--
nosy: +rbcollins

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Gregory P. Smith]

Changes by Gregory P. Smith :


--
nosy:  -gregory.p.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Doug Hellmann]

Changes by Doug Hellmann :


--
nosy: +doughellmann

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Christian Heimes]

Changes by Christian Heimes :


--
versions: +Python 3.6, Python 3.7 -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Bohuslav Slavek Kabrda]

Changes by Bohuslav Slavek Kabrda bkab...@redhat.com:


--
nosy: +bkabrda

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9216
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Jan Pokorný]

Changes by Jan Pokorný pokorny_...@seznam.cz:


--
nosy: +jpokorny

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9216
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Christian Heimes]

Christian Heimes added the comment:

It's out of scope for 3.3 but I'd love to see the feature in 3.4.

--
versions:  -Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9216
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Roumen Petrov]

Roumen Petrov added the comment:

Everything in this issue posted until now has to be managed as vendor patch.

--
nosy: +rpetrov

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9216
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Christian Heimes]

Changes by Christian Heimes li...@cheimes.de:


--
nosy: +christian.heimes
versions: +Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9216
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Daniel Holth]

Daniel Holth dho...@fastmail.fm added the comment:

While you are at it, can you edit the docs to put md5() at the bottom of the 
page at the back of the list in a 2-point font and raise a 
DeprecationWarning(This function is totally lame, and it is slower than SHA-3, 
get with the program.) the first time it is used? I don't agree that md5 has a 
legitimate place in systems designed after 1996.

--
nosy: +dholth

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9216
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 While you are at it, can you edit the docs to put md5() at the bottom
 of the page at the back of the list in a 2-point font and raise a
 DeprecationWarning(This function is totally lame, and it is slower
 than SHA-3, get with the program.) the first time it is used?

Please... don't make suggestions unrelated to the issue. Open a new issue 
instead.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9216
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 _Please_ publicise the change a little better? Pretty please!?

This changes haven't been committed in Python, so you probably want to post on 
the Fedora bug tracker instead.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9216
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Luke Carrier]

Luke Carrier h...@lukecarrier.me added the comment:

I've not done enough digging on the issue I'm presently experiencing to draw 
any conclusions make any suggestions, but this change seems to break the 
present distribute module (version 0.6.27). It appears it will likely break a 
great deal of other code too.

I've pasted the relevant output here and attached the full traceback.
  File /usr/lib64/python3.2/hashlib.py, line 112, in __get_openssl_constructor
f(usedforsecurity=False)
TypeError: openssl_md5() takes no keyword arguments

Whilst I agree with the notion behind this change, Fedora's quick actions have 
led to me spending the best part of an hour of the night before ship day 
diagnosing issues caused by undocumented (or at least under-documented) changes 
to code I haven't written or interfaced with. _Please_ publicise the change a 
little better? Pretty please!?

--
nosy: +lukecarrier
Added file: http://bugs.python.org/file26205/virtualenv_distribute

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9216
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

Patch 0002:

- cached_info-error_msg doesn't seem deallocated anywhere?

Patch 0003:

- usedforsecurity is a poor name IMO; make it shorter and/or PEP8-ize it 
(used_for_security)
- the 2-element context array thing is obscure: why not distinct ctx and 
ctx_non_fips members?
- this could fail, e.g. low on memory, or encodings: doesn't it lack an 
error-handling path, then?

Patch 0004:

- openssl_can_enforce_fips(): instead of calling OpenSSL in a subprocess, 
perhaps it's possible to expose a public flag in the hashlib module (e.g. 
hashlib.HAS_FIPS)? or is this info not fetchable programmatically?
- openssl_can_enforce_fips() needs to check the subprocess return code, in case 
another error happened
- run_command_with_fips_enforcement() should use the assert_python_ok() and 
assert_python_failure() functions from Lib/test/script_helper.py

Overall:

- please put back the unconditional tests for the usedforsecurity argument 
(even when FIPS can't be enforced)
- the patches lack docs (Doc/library/hashlib.rst)
- please commit all this as a single commit, not 4 different ones

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9216
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Gregory P. Smith]

Gregory P. Smith g...@krypto.org added the comment:

My summary of our discussion was pretty terse. :)  dmalcolm has more detailed 
TODO list notes that include things like the error cases and .rst documentation.

As for how to commit it, i'd make 0001 its own commit as it is a useful 
refactoring otherwise unrelated to this change.

I'll leave it entirely up to dmalcolm how many commits he wants 0002 onward to 
be.  No need to be picky.

usedforsecurity vs used_for_security, agreed, used_for_security is better.

dmalcolm was going to make an enum to index the two element array, that'd give 
meaningful names instead of 0 or 1.  simply using two named variables would 
also work but it would require the loop in 0003 to be expanded or turned into a 
small static method for the body (not a bad idea) instead.  i'm fine with 
either.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9216
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Gregory P. Smith]

Gregory P. Smith g...@krypto.org added the comment:

quick summary of comments from pycon sprints discussion:

this looks pretty good.  i like the 0001 refactoring cleanup.  a couple things 
to fix in error handling (better messages and some bogus handling in the test). 
dmalcolm has the notes on what to do.

do it and commit away or ask for more review as you see fit.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9216
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Dave Malcolm]

Dave Malcolm dmalc...@redhat.com added the comment:

I've refreshed this patch against the latest version of the code in hg.

In an attempt to make it easier to review, I've split it up into four (so far) 
thematic patches, which apply in sequence.

--
Added file: 
http://bugs.python.org/file23172/0001-Rework-_hashlib-caching-moving-per-hash-cached-data-.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9216
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Dave Malcolm]

Changes by Dave Malcolm dmalc...@redhat.com:


Added file: 
http://bugs.python.org/file23173/0002-Add-error-handling-to-initialization-of-_hashlib.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9216
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Dave Malcolm]

Changes by Dave Malcolm dmalc...@redhat.com:


Added file: 
http://bugs.python.org/file23174/0003-Add-optional-usedforsecurity-argument-in-various-pla.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9216
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Dave Malcolm]

Changes by Dave Malcolm dmalc...@redhat.com:


Added file: 
http://bugs.python.org/file23175/0004-_hashlib-Add-selftest-for-FIPS-mode-and-usedforsecur.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9216
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Dave Malcolm]

Dave Malcolm dmalc...@redhat.com added the comment:

[and yes, I used git to generate the 4 patches; sorry ]

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9216
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Dave Malcolm]

Dave Malcolm dmalc...@redhat.com added the comment:

The cumulative effect of the above patches (to _hashlib) are equivalent to what 
I've applied downstream to python 2 in RHEL 6.0 and Fedora 17 onwards, and 
python 3 in Fedora 17 onwards.

In those environments I've additionally patched hashlib to only use _hashlib, 
rather than falling back on _md5 etc, since otherwise you get confusing error 
messages from hashlib.md5() when it defers to _md5 due to FIPS enforcement.  In 
my downstream builds we can be sure of building against OpenSSL, but this other 
part of the patch seems less appropriate for upstream python, given that 
upstream python tries to be flexible in terms of its dependencies.

Hope this makes sense.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9216
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Antoine Pitrou]

Changes by Antoine Pitrou pit...@free.fr:


--
versions: +Python 3.3 -Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9216
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[R. David Murray]

Changes by R. David Murray rdmur...@bitdance.com:


--
type:  - feature request

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9216
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Dave Malcolm]

Dave Malcolm dmalc...@redhat.com added the comment:

Attached is a patch against the py3k branch which implements this.

I've checked that it builds against openssl-0.9.8o.tar.gz, 
openssl-1.0.0a.tar.gz, and against Fedora 12 and 13's heavily-patched 
openssl-1.0.0. The bulk of my testing has been against Fedora's openssl.

I've added selftests to try to verify the new API.  I try to detect if the 
OpenSSL enforces FIPS, via trying to run openssl md5 as a subprocess, and 
seeing if I can trigger an error.

With FIPS enforcement off, all tests pass when built against 0.9.8o and 1.0.0a 
and F13's 1.0.0, other than those for FIPS enforcement itself, which skip.

With FIPS enforcement on, all tests pass when built against F13's openssl.  (I 
haven't yet figured out how to get the fips selftest to pass for the other 
builds, it's testing checksums against the wrong libcrypto for some reason; see 
caveat below):
$ ./python Lib/test/test_hashlib.py
$ OPENSSL_FORCE_FIPS_MODE=1 ./python Lib/test/test_hashlib.py

For all of the various contexts stored in _hashopenssl.c, we now store two: one 
with the override flag, one without.  This required some reworking of the 
various preprocessor magic in that file, so I've gathered everything related to 
an algorithm into a structure, and moved most of the logic into functions, 
rather than macros.  I'm assuming that these will get inlined under 
optimization, and that the bulk of the time that you're trying to optimize out 
are the EVP lookups and initializations, rather than function call overhead.

How's this looking?

Do I need to add a dummy usedforsecurity arg to all of the non-openssl 
message digest implementations within the tree?


Unfortunately, if fips mode is on, and the fips selftest fails for the openssl 
library, every hash use will fail, both with and without the flag:
  ValueError: error:2D07D06A:FIPS routines:EVP_DigestInit_ex:fips selftest 
failed
and this leads to a crippled hashlib module.  It's not clear to me if there's a 
good way to handle this.  (Having said that, a site that has the technical 
expertise to opt-in to FIPS mode is hopefully able to diagnose this, and fix 
their openssl library)

--
keywords: +patch
stage: needs patch - patch review
Added file: http://bugs.python.org/file17970/py3k-hashlib-fips-issue9216.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9216
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Gregory P. Smith]

Gregory P. Smith g...@krypto.org added the comment:

That sounds fine to me and I do like the usedforsecurity annotation on the API. 
 I'll gladly review any patches.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9216
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Dave Malcolm]

New submission from Dave Malcolm dmalc...@redhat.com:

(taking the liberty of adding gregory.p.smith to the nosy list; hope that's 
OK)

This is a higher-level take on issue 9146.

Some versions of OpenSSL have a FIPS mode that can refuse the use of 
non-certified hashes.

The idea is that FIPS mode should prevent the use of non-certified hashes for 
security uses.  For example, MD5 shouldn't be used for signatures these days 
(see e.g. http://www.kb.cert.org/vuls/id/836068).

However, there are legitimate non-security uses of these hashes.  For example, 
one might use MD5 hashes of objects to places them in bins for later retrieval, 
purely as a speed optimization (e.g. files in directories on a filesystem).

I'm working on a patch to hashlib which would better support this, but it 
involves an API expansion, and I wanted to sound things out first.

The API idea is to introduce a new keyword argument, say usedforsecurity to 
hashlib.new() and to the named hashlib constructors, such as hashlib.md5().  
This would default to True.  If code is using these hashes in FIPS mode, the 
developer needs to override this: usedforsecurity=False to mark the callsite as 
a non-security-sensitive location.  Internally, this would lead to the 
EVP_MD_CTX being initialized with EVP_MD_CTX_FLAG_NON_FIPS_ALLOW.

This way, if you run unaudited code in an environment that cares about FIPS, 
the code will raise exceptions if it uses a non-valid hash, but during code 
audit the callsites can be marked clearly as usedforsecurity=False, and be 
used as before.

In non-FIPS environments, the flag would be ignored.

Am I right in thinking that the _hashlib module should be treated as an 
implementation detail here?  The entry points within _hashlib are likely to 
double, with a pair of pre-initialized contexts, one with the flag, one without.

Does this sound reasonable?  Thanks.

--
components: Library (Lib)
messages: 109808
nosy: dmalcolm, gregory.p.smith
priority: normal
severity: normal
stage: needs patch
status: open
title: FIPS support for hashlib
versions: Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9216
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com