Re: [Python-Dev] [Speed] speed.python.org
On 6 February 2016 at 04:07, Brett Cannon wrote: > On Thu, 4 Feb 2016 at 05:46 Nick Coghlan wrote: >> Heh, cdecimal utterly demolishing the old pure Python decimal module >> on the telco benchmark means normalising against CPython 3.5 rather >> than 2.7 really isn't very readable :) > > I find viewing the graphs using the horizontal layout is much easier to read > (the bars are a lot thicker and everything zooms in more). That comment was based on the horizontal layout - the telco benchmark runs ~53x faster in Python 3 than it does in Python 2 (without switching to cdecimal), so you end up with all the other benchmarks being squashed into the leftmost couple of grid cells. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Licensing issue (?) for Frozen Python? [was: More optimisation ideas]
On Sat, Feb 6, 2016 at 4:31 PM, Stephen J. Turnbull wrote: > However, the technical problem remains. For example, you mention > Debian. While Debian keeps its source and binary packages very close > to "in sync" on the server, there are several gotchas. For example, > Debian does not restrict itself to packaging patches, it sometimes > breaks your security when it thinks it's smarter than Bruce. So > ... is the corresponding source you're interested in the patched or > unpatched source? Do you know which you get when you install the > source package? Do you know how to get the other? Suppose for > reasons of stability you've "pinned" the binary. Is the corresponding > Debian source package still easily available? Did you think of that > gotcha when you installed the source package, or did you just assume > they were still in sync? I'm sure somebody with the "security > mindset" (eg, Bruce) can think of many more Right, sure. The technical problems are still there. Although I'm fairly confident that Debian's binaries would correspond to Debian's source - but honestly, if I'm looking for sources for anything other than the kernel, I probably want to get the latest from source control, rather than using the somewhat older version shipped in the repos. As to availability, though, most of the big distros (including Debian) keep their sources around for a long time. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Licensing issue (?) for Frozen Python? [was: More optimisation ideas]
Chris Angelico writes: > And even the GPL doesn't require you to distribute the source along > with every copy of the binary. As long as the source is *available*, > it's acceptable to distribute just the binary for convenience. True (and it would apply to frozen Python as long as the source includes the build scripts such as setup.py used to "freeze" Python), but it can be complex (especially for commercial distribution). However, the technical problem remains. For example, you mention Debian. While Debian keeps its source and binary packages very close to "in sync" on the server, there are several gotchas. For example, Debian does not restrict itself to packaging patches, it sometimes breaks your security when it thinks it's smarter than Bruce. So ... is the corresponding source you're interested in the patched or unpatched source? Do you know which you get when you install the source package? Do you know how to get the other? Suppose for reasons of stability you've "pinned" the binary. Is the corresponding Debian source package still easily available? Did you think of that gotcha when you installed the source package, or did you just assume they were still in sync? I'm sure somebody with the "security mindset" (eg, Bruce) can think of many more It's not Python's responsibility to solve these gotchas, of course. Many (eg, do you want patched vs. unpatched) are use-case-dependent anyway. However, many of them do go away (and Python has fulfilled any imaginable responsibility) if we distribute source with the binaries, or arrange that binaries are built from source at installation. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Licensing issue (?) for Frozen Python? [was: More optimisation ideas]
On Sat, Feb 6, 2016 at 3:31 PM, Stephen J. Turnbull wrote: > Of course if *you* want to you can GPL Python (I think that's now > possible, at one time there was a issue with the CNRI license IIRC), > and then licensees of *your* distribution (but not you!) are required > to distribute source. And even the GPL doesn't require you to distribute the source along with every copy of the binary. As long as the source is *available*, it's acceptable to distribute just the binary for convenience. For instance, on my Debian systems, I can say "apt-get install somepackage" to get just the binary, and then "apt-get source somepackage" if I want the corresponding source. IANAL, but I suspect it would be compliant if the same way of obtaining the C source code also gets you the unfrozen stdlib. So yeah, no licensing problem. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Licensing issue (?) for Frozen Python? [was: More optimisation ideas]
Executive summary: There is no licensing issue because Python isn't copyleft. Stick to the pragmatic *technical* issue of how to reliably provide corresponding source to those who want to look at that source (just because that's how we do things in Python). Emile van Sebille writes: > Except for that nasty licensing issue requiring source code. CPython is not now and never has been copyleft. CPython is distributed by the PSF *as* open source with a license that *permits* redistribution of original source and derivatives (including executables), but legally need not *remain* open source downstream. The remaining issue is the PSF's CLA which permits the PSF to relicense/sublicense under any open source license. However it's not clear to me that the PSF is required by the CLA to distribute source! It receives the code under very permissive licenses, and the CLA merely names the contributor's chosen license. I imagine those licenses determine whether the PSF must distribute source. If so, no, not even the PSF is bound (legally) to distribute Python source. Of course if *you* want to you can GPL Python (I think that's now possible, at one time there was a issue with the CNRI license IIRC), and then licensees of *your* distribution (but not you!) are required to distribute source. Of course our trust in the PSF is based on the moral principle of reciprocity: we contribute to the PSF's distribution as open source (according to the CLA) in large part because we expect to receive open source back. But if the PSF ever goes so wrong as to even think of taking advantage of that loophole, we are well and truly hosed anyway. (Among other things, that means a voting majority of the current PSF Board -- many of them core developers -- fell under a bus.) So don't worry about it. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] More optimisation ideas
On Friday, February 5, 2016 11:57 AM, Emile van Sebille wrote: > Aah, 'must' is less restrictive in this context than I expected. When > you combine the two halves the first part might be more accurately > phrased as 'The program must make source code available' rather than > 'must include' which I understood to mean 'ship with'. First, step back and think of this in common sense terms: If being open source required any Python installation to have the .py source to the .pyc or .zip files in the stdlib, surely it would also require any Python installation to have the .c source to the interpreter too. But lots of people have Python without having the .c source. Also, the GPL isn't typical of all open source licenses, it's only typical of _copyleft_ licenses. Permissive licenses, like Python's, are very different. Copyleft licenses are designed to make sure that all derived works are also copylefted; permissive licenses are designed to permit derived works as widely as possible. As the Python license specifically says, "All Python licenses, unlike the GPL, let you distribute a modified version without making your changes open source." Meanwhile, the fact that someone has decided that the Python license qualifies under the Open Source Definition doesn't mean the OSD is the right way to understand it. Read the license itself, or one of the summaries at opensource.org or fsf.org. (And if you still can't figure something out, and it's important to your work, you almost certainly need to ask a lawyer.) So, if you think the first sentence of section 2 of the OSD contradicts the explanation in the rest of the paragraph--well, even if you're right, that doesn't affect Python's license at all. Finally, if you want to see what it takes to actually make all the terms unambiguous both to ordinary human beings and to legal codes, see the GPL FAQ sections on their definitions of "propagate" and "convey". It may take you lots of careful reading to understand it, but when you finally do, it's definitely unambiguous. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] More optimisation ideas
On 2/5/2016 10:38 AM, Brett Cannon wrote: On Fri, 5 Feb 2016 at 10:34 Emile van Sebille mailto:em...@fenx.com>> wrote: >> Except for that nasty licensing issue requiring source code. >> >> Emile > Licensing requires, in the GPL at least, that the *modified* sources be > made *available*, not that they be shipped with the product. Looking at > the Python license, and what tools already do, there is zero need to > ship the source to stay compliant. Hmm, the annotated Open Source Definition explicitly states "The program must include source code" -- how did I misinterpret that? Because you left off the part following: "... and must allow distribution in source code as well as compiled form". This is entirely a discussion of distribution in a compiled form. Aah, 'must' is less restrictive in this context than I expected. When you combine the two halves the first part might be more accurately phrased as 'The program must make source code available' rather than 'must include' which I understood to mean 'ship with'. Emile ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] More optimisation ideas
On 2/5/2016 9:37 AM, Alexander Walters wrote: On 2/5/2016 12:27, Emile van Sebille wrote: On 2/1/2016 9:20 AM, Ethan Furman wrote: On 02/01/2016 08:40 AM, R. David Murray wrote: On the other hand, if the distros go the way Nick has (I think) been advocating, and have a separate 'system python for system scripts' that is independent of the one installed for user use, having the system-only python be frozen and sourceless would actually make sense on a couple of levels. Agreed. Except for that nasty licensing issue requiring source code. Emile Licensing requires, in the GPL at least, that the *modified* sources be made *available*, not that they be shipped with the product. Looking at the Python license, and what tools already do, there is zero need to ship the source to stay compliant. Hmm, the annotated Open Source Definition explicitly states "The program must include source code" -- how did I misinterpret that? Emile http://opensource.org/osd-annotated ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] More optimisation ideas
On Fri, 5 Feb 2016 at 10:34 Emile van Sebille wrote: > On 2/5/2016 9:37 AM, Alexander Walters wrote: > > > > > > On 2/5/2016 12:27, Emile van Sebille wrote: > >> On 2/1/2016 9:20 AM, Ethan Furman wrote: > >>> On 02/01/2016 08:40 AM, R. David Murray wrote: > >> > On the other hand, if the distros go the way Nick has (I think) been > advocating, and have a separate 'system python for system scripts' > that > is independent of the one installed for user use, having the > system-only > python be frozen and sourceless would actually make sense on a > couple of > levels. > >>> > >>> Agreed. > >> > >> Except for that nasty licensing issue requiring source code. > >> > >> Emile > > Licensing requires, in the GPL at least, that the *modified* sources be > > made *available*, not that they be shipped with the product. Looking at > > the Python license, and what tools already do, there is zero need to > > ship the source to stay compliant. > > Hmm, the annotated Open Source Definition explicitly states "The program > must include source code" -- how did I misinterpret that? > Because you left off the part following: "... and must allow distribution in source code as well as compiled form". This is entirely a discussion of distribution in a compiled form. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] speed.python.org
To piggyback on Zach's speed.python.org announcement, we will most likely be kicking off a discussion of redoing the benchmark suite, tweaking the test runner, etc. over on the speed@ ML. Those of us who have been doing perf work lately have found some shortcoming we would like to fix in our benchmarks suite, so if you want to participate in that discussion, please join speed@ by next week. On Wed, 3 Feb 2016 at 22:49 Zachary Ware wrote: > I'm happy to announce that speed.python.org is finally functional! > There's not much there yet, as each benchmark builder has only sent > one result so far (and one of those involved a bit of cheating on my > part), but it's there. > > There are likely to be rough edges that still need smoothing out. > When you find them, please report them at > https://github.com/zware/codespeed/issues or on the sp...@python.org > mailing list. > > Many thanks to Intel for funding the work to get it set up and to > Brett Cannon and Benjamin Peterson for their reviews. > > Happy benchmarking, > -- > Zach > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Speed] speed.python.org
On Thu, 4 Feb 2016 at 05:46 Nick Coghlan wrote: > On 4 February 2016 at 16:48, Zachary Ware > wrote: > > I'm happy to announce that speed.python.org is finally functional! > > There's not much there yet, as each benchmark builder has only sent > > one result so far (and one of those involved a bit of cheating on my > > part), but it's there. > > > > There are likely to be rough edges that still need smoothing out. > > When you find them, please report them at > > https://github.com/zware/codespeed/issues or on the sp...@python.org > > mailing list. > > > > Many thanks to Intel for funding the work to get it set up and to > > Brett Cannon and Benjamin Peterson for their reviews. > > Heh, cdecimal utterly demolishing the old pure Python decimal module > on the telco benchmark means normalising against CPython 3.5 rather > than 2.7 really isn't very readable :) > I find viewing the graphs using the horizontal layout is much easier to read (the bars are a lot thicker and everything zooms in more). ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] More optimisation ideas
On 2/5/2016 12:27, Emile van Sebille wrote: On 2/1/2016 9:20 AM, Ethan Furman wrote: On 02/01/2016 08:40 AM, R. David Murray wrote: On the other hand, if the distros go the way Nick has (I think) been advocating, and have a separate 'system python for system scripts' that is independent of the one installed for user use, having the system-only python be frozen and sourceless would actually make sense on a couple of levels. Agreed. Except for that nasty licensing issue requiring source code. Emile Licensing requires, in the GPL at least, that the *modified* sources be made *available*, not that they be shipped with the product. Looking at the Python license, and what tools already do, there is zero need to ship the source to stay compliant. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] More optimisation ideas
On 2/1/2016 9:20 AM, Ethan Furman wrote: On 02/01/2016 08:40 AM, R. David Murray wrote: On the other hand, if the distros go the way Nick has (I think) been advocating, and have a separate 'system python for system scripts' that is independent of the one installed for user use, having the system-only python be frozen and sourceless would actually make sense on a couple of levels. Agreed. Except for that nasty licensing issue requiring source code. Emile ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] speed.python.org
Big thanks to you, Zachary (and everyone involved)! It's a very good news. Yury On 2016-02-04 1:48 AM, Zachary Ware wrote: I'm happy to announce that speed.python.org is finally functional! There's not much there yet, as each benchmark builder has only sent one result so far (and one of those involved a bit of cheating on my part), but it's there. There are likely to be rough edges that still need smoothing out. When you find them, please report them at https://github.com/zware/codespeed/issues or on the sp...@python.org mailing list. Many thanks to Intel for funding the work to get it set up and to Brett Cannon and Benjamin Peterson for their reviews. Happy benchmarking, ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2016-01-29 - 2016-02-05) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open5413 (+32) closed 32641 (+26) total 38054 (+58) Open issues with patches: 2367 Issues opened (50) == #25660: tabs don't work correctly in python repl http://bugs.python.org/issue25660 reopened by martin.panter #26239: distutils link-objects is not normalized http://bugs.python.org/issue26239 opened by jdfergason #26240: Docstring of the subprocess module should be cleaned up http://bugs.python.org/issue26240 opened by Antony.Lee #26243: zlib.compress level as keyword argument http://bugs.python.org/issue26243 opened by palaviv #26246: Code output toggle button uses removed jQuery method http://bugs.python.org/issue26246 opened by ccwang002 #26247: Document Chrome/Chromium for python2.7 http://bugs.python.org/issue26247 opened by Ismail s #26248: Improve scandir DirEntry docs, especially re symlinks and cach http://bugs.python.org/issue26248 opened by benhoyt #26249: Change PyMem_Malloc to use PyObject_Malloc allocator? http://bugs.python.org/issue26249 opened by haypo #26250: no document for sqlite3.Cursor.connection http://bugs.python.org/issue26250 opened by qinghao #26251: Use "Low-fragmentation Heap" memory allocator on Windows http://bugs.python.org/issue26251 opened by haypo #26252: Add an example to importlib docs on setting up an importer http://bugs.python.org/issue26252 opened by brett.cannon #26253: tarfile in stream mode always set zlib compression level to 9 http://bugs.python.org/issue26253 opened by Patrik Dufresne #26254: ssl should raise an exception when trying to load an unusable http://bugs.python.org/issue26254 opened by abacabadabacaba #26256: Fast decimalisation and conversion to other bases http://bugs.python.org/issue26256 opened by jneb #26257: Eliminate buffer_tests.py http://bugs.python.org/issue26257 opened by martin.panter #26258: readline module for python 3.x on windows http://bugs.python.org/issue26258 opened by Ali Razmjoo #26259: Memleak when repeated calls to asyncio.queue.Queue.get is perf http://bugs.python.org/issue26259 opened by Jonas Brunsgaard #26261: NamedTemporaryFile documentation is vague about the `name` att http://bugs.python.org/issue26261 opened by ztane #26262: Cannot compile with /fp:strict with MSVC http://bugs.python.org/issue26262 opened by zach.ware #26263: Serialize array.array to JSON by default http://bugs.python.org/issue26263 opened by Omer.Katz #26264: keyword module missing async and await keywords http://bugs.python.org/issue26264 opened by tuxtimo #26265: build errors on OS X 10.11 with --enable-universalsdk http://bugs.python.org/issue26265 opened by davidjamesbeck #26266: add classattribute to enum to handle non-Enum attributes http://bugs.python.org/issue26266 opened by ethan.furman #26267: UUID docs should say how to get "standard form" http://bugs.python.org/issue26267 opened by abarnert #26268: Update python.org installers to use OpenSSL 1.0.2f http://bugs.python.org/issue26268 opened by zach.ware #26269: zipfile should call lstat instead of stat if available http://bugs.python.org/issue26269 opened by Patrik Dufresne #26270: Support for read()/write()/select() on asyncio http://bugs.python.org/issue26270 opened by Paulo Costa #26271: freeze.py makefile uses the wrong flags variables http://bugs.python.org/issue26271 opened by Daniel Shaulov #26273: Expose TCP_CONGESTION and TCP_USER_TIMEOUT to the socket modul http://bugs.python.org/issue26273 opened by Omar Sandoval #26275: perf.py: calibrate benchmarks using time, not using a fixed nu http://bugs.python.org/issue26275 opened by haypo #26276: Inconsistent behaviour of PEP 3101 formatting between versions http://bugs.python.org/issue26276 opened by Mark Shannon #26277: Allow zipapp to target modules http://bugs.python.org/issue26277 opened by flying sheep #26278: BaseTransport.close() does not trigger connection_lost() http://bugs.python.org/issue26278 opened by Sümer.Cip #26279: time.strptime does not properly convert out-of-bounds values http://bugs.python.org/issue26279 opened by iaslan #26280: ceval: Optimize [] operation similarly to CPython 2.7 http://bugs.python.org/issue26280 opened by yselivanov #26281: Clear sys.path_importer_cache from importlib.invalidate_caches http://bugs.python.org/issue26281 opened by brett.cannon #26282: Add support for partial keyword arguments in extension functio http://bugs.python.org/issue26282 opened by serhiy.storchaka #26283: zipfile can not handle the path build by os.path.join() http://bugs.python.org/issue26283 opened by å¼µä¼¯èª #26284: FIx telco benchmark http://bugs.python.org/issue26284 opened by skrah #26285: Garbage collection of unused input sections from CPython binar http://bugs.python.org/issue26285
Re: [Python-Dev] Opcode cache in ceval loop
On 05.02.2016 00:06, Matthias Bussonnier wrote: On Feb 4, 2016, at 08:22, Sven R. Kunze wrote: On 04.02.2016 16:57, Matthias Bussonnier wrote: On Feb 3, 2016, at 13:22, Yury Selivanov wrote: An ideal way would be to calculate a hit/miss ratio over time for each cached opcode, but that would be an expensive calculation. Do you mean like a sliding windows ? Otherwise if you just want a let's say 20% miss threshold, you increment by 1 on hit, and decrement by 4 on miss. Division is expensive. I'm not speaking about division here. if you +M / -N the counter will decrease in average only if the hit/miss ratio is below N/(M+N), but you do not need to do the division. Then you deoptimize only if you get < 0. I see but it looks still more complicated. :) On Feb 3, 2016, at 13:37, Sven R. Kunze wrote: On 03.02.2016 22:22, Yury Selivanov wrote: One way of tackling this is to give each optimized opcode a counter for hit/misses. When we have a "hit" we increment that counter, when it's a miss, we decrement it. Within a given range, I suppose. Like: c = min(c+1, 100) Min might be overkill, maybe you can use a or mask, to limit the windows range to 256 consecutive call ? Sure, that is how I would have written it in Python. But I would suggest an AND mask. ;-) Sure, implementation detail I would say. Should not write emails before breakfast... ;-) The other problem, with the mask, is if your increment hit 256 you wrap around back to 0 where it deoptimize (which is not what you want), so you might need to not mask the sign bit and deoptimize only on a certain negative threshold. Does it make sens ? Definitely. I am curious about the actual implementation of this idea. Best, Sven ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] More optimisation ideas
On 5 February 2016 at 15:05, Steven D'Aprano wrote: > (I'm not even sure if this suggestion makes sense, since I'm not really > sure what "freezing" the stdlib entails. Is it documented anywhere?) It's not particularly well documented - most of the docs you'll find are about freeze utilities that don't explain how they work, or the FrozenImporter, which doesn't explain how to *create* a frozen module and link it into your Python executable. Your approach of thinking of a frozen module as a generated .pyc file that has been converted to a builtin module is a pretty good working model, though. (It isn't *entirely* accurate, but the discrepancies are sufficiently arcane that they aren't going to matter in any case that doesn't involve specifically poking around at the import related attributes). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
