[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Terry J. Reedy added the comment: Phillipe: I was the first to comment, but had no futher involvement with this issue until now. Multiple people, including coredevs, considered the old behavior to be buggy and multiple people, including coredevs, contributed to the fix. You are unlikely to

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Philippe Ombredanne added the comment: Actually this is problematic on multiples counts: 1. the behaviour changes and this is a regression 2. even if that new buggy behaviour was the one to use, it should not give preference to knownfiles ovr init-provided files, but at least take the provide

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Philippe Ombredanne added the comment: The changes introduced by this ticket in https://github.com/python/cpython/commit/9fc720e5e4f772598013ea48a3f0d22b2b6b04fa#r45794801 are problematic. I discovered this from having tests failing when testing on Python 3.7 and up The bug is that calling

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Change by Steve Dower : -- assignee: -> steve.dower resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Steve Dower added the comment: New changeset 2a99fd911ebeecedbb250a05667cd46eca4735b9 by Steve Dower in branch '3.7': bpo-4963: Fix for initialization and non-deterministic behavior issues in mimetypes (GH-14376) https://github.com/python/cpython/commit/2a99fd911ebeecedbb250a05667cd46eca4735

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

David K. Hess added the comment: Thank you Steve! Nice to see this one make it across the finish line. -- ___ Python tracker ___ __

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Steve Dower added the comment: New changeset 25fbe33b92cd938e809839feaa3fda97e6ad0980 by Steve Dower in branch '3.8': bpo-4963: Fix for initialization and non-deterministic behavior issues in mimetypes (GH-14375) https://github.com/python/cpython/commit/25fbe33b92cd938e809839feaa3fda97e6ad09

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Change by Steve Dower : -- pull_requests: +14190 stage: backport needed -> patch review pull_request: https://github.com/python/cpython/pull/14375 ___ Python tracker ___ ___

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Change by Steve Dower : -- pull_requests: +14191 pull_request: https://github.com/python/cpython/pull/14376 ___ Python tracker ___ __

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Steve Dower added the comment: Sorry for the delays! I've merged into master. The backports (3.7, 3.8) are going to need some help, so I'll take a look at them soon unless someone else gets there first. -- stage: needs patch -> backport needed versions: +Python 3.8, Python 3.9 -Pytho

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Steve Dower added the comment: New changeset 9fc720e5e4f772598013ea48a3f0d22b2b6b04fa by Steve Dower (David K. Hess) in branch 'master': bpo-4963: Fix for initialization and non-deterministic behavior issues in mimetypes (GH-3062) https://github.com/python/cpython/commit/9fc720e5e4f772598013

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Change by Abhilash Raj : -- nosy: +maxking ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Christian Heimes added the comment: The PR has a merge conflict. -- nosy: +christian.heimes ___ Python tracker ___ ___ Python-bugs-l

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Change by Sascha Silbe : -- nosy: +sascha_silbe ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.p

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

David K. Hess added the comment: Are there any committers watching this issue that are able to review the PR? https://github.com/python/cpython/pull/3062 It's close to 6 months old now with no action on it. I'm willing to help but doing so and then having the PR gather dust is pretty discoura

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Change by Bert JW Regeer : -- nosy: +X-Istence ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pyt

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

David K. Hess added the comment: FYI, PR opened: https://github.com/python/cpython/pull/3062 -- ___ Python tracker ___ ___ Python-bugs-

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Changes by David K. Hess : -- pull_requests: +3096 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

David K. Hess added the comment: Pushed more commits so here's a branch compare: https://github.com/python/cpython/compare/master...davidkhess:fix-issue-4963 -- ___ Python tracker __

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

David K. Hess added the comment: Ok, I followed @r.david.murray's advice and decided to take a shot at this. First, I noticed that I couldn't reproduce the non-deterministic behavior that I reported above on the latest code (i.e. pre-3.7). After doing some research it appears this was the sequ

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Martin Panter added the comment: I understand hash randomization was added after this bug was opened. Here is a demonstration with “video/mp4”, which only has the extension “.mp4” built in. But my /etc/mime.types file lists “mp4 mp4v mpg4”, so after the second initialization the behaviour chan

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Martin Panter added the comment: I suggest to discuss the non-determinism problem in Issue 1043134 (about determining a canonical extension for each content type). I understood this bug (Issue 4963) is about the behaviour of repeated initialization of the same instance of mimetypes. BTW an or

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Changes by R. David Murray : -- stage: test needed -> needs patch versions: +Python 3.6, Python 3.7 -Python 2.7, Python 3.2, Python 3.3, Python 3.4 ___ Python tracker ___ ___

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

R. David Murray added the comment: @dhess: do you want to work on the OrderedDict + correctly ordered hardcoded lists solution? -- ___ Python tracker ___

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

David Hess added the comment: And the underlying problem causing this: $ python -c 'import mimetypes;print(mimetypes.guess_all_extensions("image/jpeg"))' ['.jpeg', '.jpg', '.jpe'] $ python -c 'import mimetypes;print(mimetypes.guess_all_extensions("image/jpeg"))' ['.jpg', '.jpe', '.jpeg'] $ pyt

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

David Hess added the comment: Concur with @sivert – the result of guess_extension() is non-deterministic between mimetypes module initialization. $ python Python 3.4.3 (default, Nov 17 2016, 01:08:31) [GCC 4.8.4] on linux Type "help", "copyright", "credits" or "license" for more information. >

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Changes by Marcin Szewczyk : -- nosy: +wodny ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Julian Sivertsen added the comment: I bumped into a similar issue with mimetypes.guess_extension on Arch Linux 64-bit in February. The behavior is still present in python 3.4.3. $ python test.py .htm $ python test.py .html $ cat test.py from mimetypes import guess_extension print(guess_extens

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

R. David Murray added the comment: OK, it is great having a test that makes this at least mostly reproducible :) Having reloaded my brain on this thing, I'm thinking that the best solution may be indeed to switch to ordered dicts. If we then reorder the hardcoded lists to be in "preferred" or

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Toshio Kuratomi added the comment: Took a look at this and was able to reproduce it on Fedora Linux 20 and current cpython head. It is somewhat random though. I'm able to get reasonably consistent failures using image/jpeg and iterating the test case about 20 times. Additionally, it looks li

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Wichert Akkerman added the comment: I can reproduce this on Both OSX 10.9 and Ubuntu 12.04: >>> import mimetypes >>> mimetypes.guess_extension('image/jpeg') '.jpe' >>> mimetypes.init() >>> mimetypes.guess_extension('image/jpeg') '.jpeg' The same thing happens for Python 3.4: Python 3.4.0rc3 (d

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

R. David Murray added the comment: I'd forgotten about this issue. I wonder if the dictionary randomization makes the problem worse. -- components: +email nosy: +barry versions: +Python 3.2, Python 3.3, Python 3.4 ___ Python tracker

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

S Arrowsmith added the comment: That solution looks sound to me, in particular documenting the semantics of repeated init() calls! As for the underlying problem, it seems to me that an alternative to copying the existing structures rather than rebuilding them would be to use OrderedDicts. Al

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

R. David Murray added the comment: It must be that the different key order only happens on the one platform because of the quirky nature of dictionary construction. That is, there is *something* on that platform that is changing where things get hashed when the dictionary is recreated. The

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

S Arrowsmith added the comment: I've dug into it -- again -- and my original analysis still holds. Getting consistent guess_extension() results across an explicit init() call depends on dict.items() returning keys in the same order on two different dictionaries (the original, hard-coded types

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

S Arrowsmith added the comment: >>> import mimetypes >>> mimetypes.guess_extension('image/jpeg') /etc/mime.types '.jpe' >>> mimetypes.init() /etc/mime.types >>> mimetypes.guess_extension('image/jpeg') '.jpeg' >>> $ grep jpeg /etc/mime.types image/jpeg jpeg

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

R. David Murray added the comment: S Arrowsmith: can put a print statement into mimetypes.init, find out what files are loading, and paste the image/jpeg lines from each of those files here? That might provide a clue. -- priority: normal -> low stage: -> unit test needed status: clo

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Shashwat Anand added the comment: Can't reproduce. 16:36:36 l0nwlf-MBP:~$ python2.7 Python 2.7rc2+ (trunk:82148M, Jun 22 2010, 10:32:46) [GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import mimetypes >>> mimetyp

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

S Arrowsmith added the comment: Sorry, still there: Python 2.7rc2 (r27rc2:82137, Jun 26 2010, 11:27:59) [GCC 4.3.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import mimetypes >>> mimetypes.guess_extension('image/jpeg') '.jpe' >>> mimetypes.init() >>>

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

R. David Murray added the comment: I can't reproduce this either, and without a reproducer we might as well close it. Antoine it is possible that your fix for #5853 inadvertently fixed this, but I don't feel like untangling the logic of the module enough to figure it out :) So I'm going to

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Antoine Pitrou added the comment: Can't reproduce under Mandriva Linux: >>> import mimetypes >>> print(mimetypes.guess_extension('image/jpeg')) .jpe >>> mimetypes.init() >>> print(mimetypes.guess_extension('image/jpeg')) .jpe The fact that it returns ".jpe" rather than ".jpg", however, could b

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Changes by Terry J. Reedy : -- versions: +Python 2.7 -Python 2.4, Python 2.5 ___ Python tracker ___ ___ Python-bugs-list mailing list U

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

S Arrowsmith added the comment: Ah, yes, forgot to mention this is on Debian 4.0. I doubt you're going to run into it on a Windows system unless you explicitly give init() a mime.types file, looking at the knownfiles list used by default. ___ Python tracker

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Terry J. Reedy added the comment: 3.0, WinXP import mimetypes print(mimetypes.guess_extension('image/jpeg')) mimetypes.init() print(mimetypes.guess_extension('image/jpeg')) gives .jpe .jpe I wonder at this answer since .jpg and occasionally .jpeg is standard in Windows usage, but the doc is unc

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

New submission from S Arrowsmith : Asking mimetypes to reload mime.types can cause guess_extension() to return a different result if multiple extensions are mapped to that mime type: >>> import mimetypes >>> mimetypes.guess_extension('image/jpeg') '.jpe' >>> mimetypes.init() >>> mimetypes.guess_