Re: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug.
On Thu, May 23, 2019 at 11:52 AM Inada Naoki wrote: > I have only 32GB mem, but AWS has larger memory machine! > Be aware that on larger cloud vendor instances, the backing vCPUs and memory you get allocated might or might not be spread opaquely across multiple sockets and/or NUMA nodes of the backing hardware. Some of them have options where you can allocate raw hardware as well so you can opt to lock your process to run within just one NUMA node and ensure hardware locality for your performance debugging. I'm pointing this out in case you experience any mystery jitters in your benchmark results. -- Joni Orponen ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Last-minute request: please backport bpo-33329 fix to 3.4 and 3.5
On Sun, Mar 10, 2019 at 7:50 AM Larry Hastings wrote: > On 3/4/19 2:29 AM, Joni Orponen wrote: > > On Sat, Mar 2, 2019 at 7:08 AM Larry Hastings wrote: > >> This bug in bpo-33329: >> >> https://bugs.python.org/issue33329 >> >> >> This is also potentially affecting PGO builds of 2.7 on Debian Buster > with GCC. Somehow building with Clang is fine. > > Does the configure time choice of compiler make a difference here for 3.4 > and 3.5? > > I don't know. I only build with the default compiler on my machine, gcc. > (My machine is Linux, 64-bit.) It'd be swell if you tried the RCs with > clang! > > */arry* > I've only risen this as a newer glibc was implicated as a potential source of trouble, I have a lab / 'canary in a coal mine' server on a prerelease OS version (Debian Buster is currently in feature freeze ~ release candidate stage) and I've not noticed anything to have happened with a newer glibc in relation to building Python, but I've not had success building a PGO build of 2.7.15 with the GCC shipped at that time, but building with Clang working fine. I'd thus be inclined to point at digging up relevant changes in GCC than in glibc. Currently 2.7.16 compiles just fine for me on both GCC and Clang. 2.7.15 still does not compile fine for me on GCC. FWIW - both Python 3 RCs from above successfully complete a PGO build for me (and thus implicitly also pass the regression tests) with Clang. They both get stuck with GCC. $ cat /etc/debian_version buster/sid $ aptitude show base-files Package: base-files Version: 10.1 Essential: yes State: installed $ aptitude show libc6 Package: libc6 Version: 2.28-8 State: installed $ aptitude show clang Package: clang Version: 1:7.0-47 State: installed $ clang --version clang version 7.0.1-8 (tags/RELEASE_701/final) Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin $ aptitude show gcc Package: gcc Version: 4:8.3.0-1 State: installed $ gcc --version gcc (Debian 8.3.0-2) 8.3.0 And for anyone wishing to try such things out for themselves, one can simply set the CC environment variable for the autotools provided ./configure script. See ./configure --help for any further details and instructions. -- Joni Orponen ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Last-minute request: please backport bpo-33329 fix to 3.4 and 3.5
On Sat, Mar 2, 2019 at 7:08 AM Larry Hastings wrote: > > This bug in bpo-33329: > > https://bugs.python.org/issue33329 > > was fixed for 3.6+, but it also affects 3.4 and 3.5. The bug is that with > newer versions of glibc--which I'm pretty sure has shipped on all major > Linux distros by now--the test suite may send signals that are invalid > somehow. As a result the test suite... blocks forever? I think? Anyway > the observed resulting behavior is that there are three regression tests in > each branch that seemingly never complete. I started the 3.4 regression > test suite *nine hours ago* and it still claims to be running--and the > 3.5 test suite isn't far behind. Technically, no, it's not a security > bug. But I simply can't ship 3.4 and 3.5 in this sorry state. > This is also potentially affecting PGO builds of 2.7 on Debian Buster with GCC. Somehow building with Clang is fine. Does the configure time choice of compiler make a difference here for 3.4 and 3.5? -- Joni Orponen ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding test.support.safe_rmpath()
On Thu, Feb 14, 2019 at 10:49 AM Ronald Oussoren via Python-Dev < python-dev@python.org> wrote: > I usually use shutil.rmtree for tests that need to create temporary files, > and create a temporary directory for those files (that is, use > tempfile.mkdtemp in setUp() and use shutil.rmtree in tearDown()). That way > I don’t have to adjust house-keeping code when I make changes to test code. > As tempfile provides context managers, should these be used internally for something like this? Provide a decorator which passes in the temp file / directory. -- Joni Orponen ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals
On Fri, Oct 19, 2018 at 9:09 AM Thomas Moreau wrote: > Hello, > > I have been working on the concurent.futures module lately and I think > this optimization should be avoided in the context of python Pools. > > This is an interesting idea, however its implementation will bring many > complicated issues as it breaks the basic paradigm of a Pool: the tasks are > independent and you don't know which worker is going to run which task. > > The function is serialized with each task because of this paradigm. This > ensure that any worker picking the task will be able to perform it > independently from the tasks it has run before, given that it as been > initialized correctly at the beginning. This makes it simple to run each > task. > I would not mind if there would be a subtype of Pool where you can only apply one kind of task to. This is a very common use mode. Though the question there is 'should this live in Python itself'? I'd be fine with a package on PyPi. As the Pool comes with no scheduler, with your idea, you would need a > synchronization step to send the function to all workers before you can > launch your task. But if there is already one worker performing a long > running task, does the Pool wait for it to be done before it sends the > function? If the Pool doesn't wait, how does it ensure that this worker > will be able to get the definition of the function before running it? > Also, the multiprocessing.Pool has some features where a worker can shut > itself down after a given number of tasks or a timeout. How does it ensure > that the new worker will have the definition of the function? > It is unsafe to try such a feature (sending only once an object) anywhere > else than in the initializer which is guaranteed to be run once per worker. > > On the other hand, you mentioned an interesting point being that making > globals available in the workers could be made simpler. A possible solution > would be to add a "globals" argument in the Pool which would instanciate > global variables in the workers. I have no specific idea but on the > implementation of such features but it would be safer as it would be an > initialization feature. > Would this also mean one could use a Pool in a context where threading is used? Currently using threading side effects unpicklables into the globals. Also being able to pass in globals=None would be optimal for a lot of use cases. -- Joni Orponen ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A more flexible task creation
On Thu, Jun 14, 2018 at 9:17 PM Chris Barker via Python-Dev < python-dev@python.org> wrote: > Excuse my ignorance (or maybe it's a vocabulary thing), but I'm trying to > understand the problem here. > Vocabulary-wise 'queue depth' might be a suitable mental aid for what people actually want to limit. The practical issue is most likely something to do with hitting timeouts when trying to queue too much work onto a service. -- Joni Orponen ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Process to remove a Python feature
On Tue, May 8, 2018 at 3:20 PM, Stéphane Blondon <stephane.blon...@gmail.com > wrote: > Le 02/05/2018 à 11:11, Victor Stinner a écrit : > > * Communicate on python-dev, Twitter, Python Insider blog, etc. > > * Collaborate with major Python projects to help them to migrate the > alternative > > I wonder if it would be interesting to have a package available by > pypi.org which would provide the removed features. In your example, the > developers would have to update their source code: > > # giving 'obsolete' as name for this package > platform.linux_distribution() -> obsolete.platform.linux_distribution() If one can import from the future can one can also import from the past? -- Joni Orponen ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OS-X builds for 3.7.0
On Thu, Feb 1, 2018 at 12:18 AM, Chris Barker <chris.bar...@noaa.gov> wrote: > On Wed, Jan 31, 2018 at 3:13 AM, Joni Orponen <j.orpo...@4teamwork.ch> > wrote: > >> On Wed, Jan 31, 2018 at 12:43 AM, Chris Barker - NOAA Federal < >> chris.bar...@noaa.gov> wrote: >> >>> And maybe we could even get rid of the "Framework" builds.. >>>> >>> >>> Please do not. These make life easier for doing things the Apple way for >>> signed sandboxed applications. >>> >>> For the record, are you re-distributing the python.org builds, or >>> re-building yourself? >>> >> >> We are re-building ourselves. >> > > Then it makes no difference to you if the pyton.org binaries are > Framework builds... though maybe you want the configure target available. > And if the official distribution distributes a Framework, the configure target and build for that are also actually maintained, which will keep them likelier usable. Would we not be doing a couple of very specific things, we'd be able to use the distributed Framework as well. It is not unconceivable we could use it as-is in the future or for other projects. -- Joni Orponen ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OS-X builds for 3.7.0
On Wed, Jan 31, 2018 at 9:31 AM, Ray Donnelly <mingw.andr...@gmail.com> wrote: > We see a 1.1 to 1.2 times performance benefit over official releases as > measured using 'python performance'. > > Apart from a static interpreter we also enable LTO and PGO and only build > for 64-bit so I'm not sure how much each bit continues. Our recipe for > python 3.6 can be found at: > Do you metrify LTO and PGO independent of each other as well or only the "enable everything" combo? I've had mixed results with LTO so far, but this is probably hardware / compiler combination specific. -- Joni Orponen ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OS-X builds for 3.7.0
On Wed, Jan 31, 2018 at 12:43 AM, Chris Barker - NOAA Federal < chris.bar...@noaa.gov> wrote: > And maybe we could even get rid of the "Framework" builds.. >> > > Please do not. These make life easier for doing things the Apple way for > signed sandboxed applications. > > Thanks — good to hear there is a good reason for them. I’ve always thought > that Frameworks were designed with other use-casss, and didn’t really help > with Python. > > For the record, are you re-distributing the python.org builds, or > re-building yourself? > We are re-building ourselves. Seems we've cooked up something not too unsimilar to what Anaconda is doing, but less generic and covering less corner cases. -- Joni Orponen ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OS-X builds for 3.7.0
On Tue, Jan 30, 2018 at 6:50 PM, Ray Donnelly <mingw.andr...@gmail.com> wrote: > While we're making such macOS-build requests, any chance of building a > static interpreter too? We've been doing that on the Anaconda > Distribution since the 5.0 release in September and it seems to be > working well. > PyPy is also currently eyeing doing their macOS builds better: https://bitbucket.org/pypy/pypy/issues/2734/establish-a-build-and-release-pipeline-for What do the Anaconda static builds get built on? -- Joni Orponen ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OS-X builds for 3.7.0
On Tue, Jan 30, 2018 at 7:08 PM, Matt Billenstein <m...@vazor.com> wrote: > OSX is in a sad state linking to system libs on the later releases -- maybe > 10.11 and on, not sure of the exact release -- they stopped shipping the > headers for things like ssl and ffi since they don't want 3rd parties > linking > to deprecated versions of those libraries versus, in the case of ssl, their > newer security framework. Recommendation is to bundle what you need if > you're > not using the framework -- something to think about. There are also some practical issues with trying to distribute software using some deprecated Cocoa APIs or weak linked syscalls. The pythonmac-sig thead I linked to earlier has pointers to how to flare those up if one ever needs to distribute Python to a specific macOS version target range while compiling on a newer macOS. It would be nice to do more things the Apple way, including porting to modern runtime feature availability check cascades of the Cocoa APIs and using the Apple provided system Frameworks. This seems like a rather major workload and should be targeting 3.8. I'm willing to participate in that effort. The availability of syscalls across targets when cross-compiling for an older target is a more generic build system problem and I'm not sure if Python should do anything other than just document it being a thing. I'm personally fine patching pyconfig.h after running the configure script for this special case. As suggested on pythonmac-sig, I'd like to see 10.11 get chosen as the macOS to build on as it provides a decent balance between hardware compatibility and being new(er). -- Joni Orponen ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OS-X builds for 3.7.0
On Tue, Jan 30, 2018 at 6:42 PM, Chris Barker <chris.bar...@noaa.gov> wrote: > > And maybe we could even get rid of the "Framework" builds.. > Please do not. These make life easier for doing things the Apple way for signed sandboxed applications. Joining the discussion here from a ~cross-post on pythonmac-sig: https://mail.python.org/pipermail/pythonmac-sig/2018-January/024283.html -- Joni Orponen ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Slipping Python 3.5.5rc1 and 3.4.8rc1 because of a Travis CI issue--can someone make Travis CI happy?
On Mon, Jan 22, 2018 at 11:59 AM, Oleg Broytman <p...@phdru.name> wrote: > On Mon, Jan 22, 2018 at 02:33:01AM -0800, Larry Hastings < > la...@hastings.org> wrote: > > All ... CI tests fail in > > the same way, reporting this error: > > > >The command "pyenv global system 3.5" failed and exited with 1 during > . > >Seems there is a slow workaround (install python 3.5): > > https://github.com/travis-ci/travis-ci/issues/8363#issuecomment-354857845 > > which python3.5 || (pyenv install 3.5.4 && pyenv use system 3.5.4) > There is also https://github.com/praekeltfoundation/travis-pyenv I've found useful when one needs excactness and also to decouple oneself from the Travis side rolling releases of Python. Also caches the Python version for you. See https://github.com/plone/plone.intelligenttext/blob/a71bdc5b485b1562b2e320f5c41a15286f205f98/.travis.yml for a usage example. -- Joni Orponen ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
