ANN: Pandas 0.15.0 released
Hello, We are proud to announce v0.15.0 of pandas, a major release from 0.14.1. This release includes a small number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. This was 4 months of work with 420 commits by 79 authors encompassing 236 issues. We recommend that all users upgrade to this version. *Highlights:* - Drop support for numpy 1.7.0 - The Categorical type was integrated as a first-class pandas type - New scalar type Timedelta, and a new index type TimedeltaIndex - New DataFrame default display for df.info() to include memory usage - New datetimelike properties accessor .dt for Series - Split indexing documentation into Indexing and Selecting Data and MultiIndex / Advanced Indexing - Split out string methods documentation into Working with Text Data - read_csv will now by default ignore blank lines when parsing - API change in using Indexes in set operations - Internal refactoring of the Index class to no longer sub-class ndarray - dropping support for PyTables less than version 3.0.0, and numexpr less than version 2.1 See a full description of Whatsnew for v0.15.0 here: http://pandas.pydata.org/pandas-docs/stable/whatsnew.html *What is it:* *pandas* is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis / manipulation tool available in any language. Documentation: http://pandas.pydata.org/pandas-docs/stable/ Source tarballs, windows binaries are available on PyPI: https://pypi.python.org/pypi/pandas windows binaries are courtesy of Christoph Gohlke and are built on Numpy 1.8 macosx wheels are courtesy of Matthew Brett and are built on Numpy 1.7.1 Please report any issues here: https://github.com/pydata/pandas/issues Thanks The Pandas Development Team Contributors to the 0.15.0 release -- https://mail.python.org/mailman/listinfo/python-announce-list Support the Python Software Foundation: http://www.python.org/psf/donations/
[ANN] iPOPO 0.5.8
iPOPO 0.5.8
===
iPOPO v0.5.8 has been released !
What is iPOPO
=
iPOPO is a Python-based Service-Oriented Component Model (SOCM) based on
Pelix, a dynamic service platform. They are inspired on two popular Java
technologies for the development of long-lived applications: the iPOJO
component model and the OSGi Service Platform.
iPOPO enables to conceive long-running and modular IT services.
It is based on OSGi concepts:
- Bundle: a Python module imported using Pelix and associated to a
context. A bundle has a life-cycle (install, start, updated, stop,
uninstall)
- Service: a Python object registered in a service registry, associated
to a specification and to properties.
- Component: the instance of a class described/manipulated by iPOPO
decorators
Components are bound together by the specification(s) of the service(s)
they provide. The required services are injected into components by iPOPO.
For more information, see
https://ipopo.coderxpress.net/wiki/doku.php?id=ipopo:refcards:concepts
iPOPO is released under the terms of Apache Software License 2.0
What's new in 0.5.8
===
This version adds the immediate_rebind feature to iPOPO, and introduces
new variables in the Shell.
New features:
The @Requires decorator has a new immediate_rebind flag argument,
which indicates iPOPO to not invalidate then re-validate a component if a
service can replace an unbound required one. See
https://github.com/tcalmant/ipopo/issues/25
The shell now supports variables ($var, ${var}, $doted.var, …)
A special variable, $? keeps the last non-None result of shell
commands
Added the run file shell command to execute files as shell commands
You can take a look at the documentation at https://ipopo.coderxpress.net/
iPOPO is available on PyPI: https://pypi.python.org/pypi/iPOPO
Source is available on GitHub: https://github.com/tcalmant/ipopo
Feel free to send feedback on your experience of Pelix/iPOPO, via the
mailing lists:
User list : http://groups.google.com/group/ipopo-users
Development list : http://groups.google.com/group/ipopo-dev
Have fun !
Thomas
--
https://mail.python.org/mailman/listinfo/python-announce-list
Support the Python Software Foundation:
http://www.python.org/psf/donations/
PyPy3 2.4.0 released
= PyPy3 2.4 - Snow White = We're pleased to announce PyPy3 2.4, which contains significant performance enhancements and bug fixes. You can download the PyPy3 2.4.0 release here: http://pypy.org/download.html PyPy3 Highlights Issues reported with our previous release were fixed after reports from users on our new issue tracker at https://bitbucket.org/pypy/pypy/issues or on IRC at #pypy. Here is a summary of the user-facing PyPy3 specific changes: * Better Windows compatibility, e.g. the nt module functions _getfinalpathname _getfileinformation are now supported (the former is required for the popular pathlib library for example) * Various fsencode PEP 383 related fixes to the posix module (readlink, uname, ttyname and ctermid) and improved locale handling * Switched default binary name os POSIX distributions to 'pypy3' (which symlinks to to 'pypy3.2') * Fixed a couple different crashes related to parsing Python 3 source code Further Highlights (shared w/ PyPy2) Benchmarks improved after internal enhancements in string and bytearray handling, and a major rewrite of the GIL handling. This means that external calls are now a lot faster, especially the CFFI ones. It also means better performance in a lot of corner cases with handling strings or bytearrays. The main bugfix is handling of many socket objects in your program which in the long run used to leak memory. We fixed a memory leak in IO in the sandbox_ code We welcomed more than 12 new contributors, and conducted two Google Summer of Code projects, as well as other student projects not directly related to Summer of Code. * Reduced internal copying of bytearray operations * Tweak the internal structure of StringBuilder to speed up large string handling, which becomes advantageous on large programs at the cost of slightly slower small *benchmark* type programs. * Boost performance of thread-local variables in both unjitted and jitted code, this mostly affects errno handling on linux, which makes external calls faster. * Move to a mixed polling and mutex GIL model that make mutlithreaded jitted code run *much* faster * Optimize errno handling in linux (x86 and x86-64 only) * Remove ctypes pythonapi and ctypes.PyDLL, which never worked on PyPy * Classes in the ast module are now distinct from structures used by the compiler, which simplifies and speeds up translation of our source code to the PyPy binary interpreter * Win32 now links statically to zlib, expat, bzip, and openssl-1.0.1i. No more missing DLLs * Many issues were resolved_ since the 2.3.1 release in June .. _`whats-new`: http://doc.pypy.org/en/latest/whatsnew-2.4.0.html .. _resolved: https://bitbucket.org/pypy/pypy/issues?status=resolved .. _sandbox: http://doc.pypy.org/en/latest/sandbox.html We have further improvements on the way: rpython file handling, numpy linalg compatibility, as well as improved GC and many smaller improvements. Please try it out and let us know what you think. We especially welcome success stories, we know you are using PyPy, please tell us about it! Cheers The PyPy Team -- https://mail.python.org/mailman/listinfo/python-announce-list Support the Python Software Foundation: http://www.python.org/psf/donations/
py2exe 0.9.2.2 released
py2exe version 0.9.2.2 released === `py2exe` is a distutils extension which allows to build standalone Windows executable programs (32-bit and 64-bit) from Python scripts; Python 3.3 and later are supported. It can build console executables, windows (GUI) executables, windows services, and DLL/EXE COM servers. (For Python 2.3 - 2.7 the older version 0.6.9 is still available.) Changes in version 0.9.2.2: - Added support for six, cffi, pycparser, openssl. - The cmdline_style options for windows services (py2exe, pywin32, custom) should work again. - Several bugfixes, better error messages. News for py2exe 0.9.2: - brand new implementation, written from scratch for Python 3 - py2exe now supports Python 3.3 and 3.4 - can be installed with 'pip' since py2exe is released in wheel format (in addition to a source package and bdist_wininst installers) - less warnings from the build process thanks to a hooks module which contains knowledge about some often used packages - easy start: executables can now be built with a simple command line script; no need to write a setup-script from scratch. In fact a commented setup-script can now be autogenerated - py2exe does now find Python modules even in zipped eggs Short manual and download at the Python cheeseshop (or simply install or upgrade it with 'pip install -U py2exe'): https://pypi.python.org/pypi/py2exe/0.9.2.2/ Enjoy, Thomas -- https://mail.python.org/mailman/listinfo/python-announce-list Support the Python Software Foundation: http://www.python.org/psf/donations/
Re: OS X Menubar in Tkinter
On 21/10/2014 02:34, Noble Bell wrote: Creating the app with py2app fixed my problem. I'm pleased to see that you have an answer. In return would you please access this list via https://mail.python.org/mailman/listinfo/python-list or read and action this https://wiki.python.org/moin/GoogleGroupsPython to prevent us seeing double line spacing and single line paragraphs, thanks. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Struggling with python-daemon and subprocess module to work together
Hi, I am writing a very basic server side application[0] which get data from a client and create a virtual machine using provided data and also tells client about what's going on during *virt-install* command execution. Previously this basic server is executed using *openvt* but I thought it would be nice to have a daemon process for it and used python-daemon. Issue I am facing is after some time server will stop sending data to client which suppose to happen in #120 and get error message Cannot run interactive console without a controlling TTY . I am not able to figure out issue is occurred due to module or I missed something during daemon initialization. [0] http://fpaste.org/143455/ -- Praveen Kumar http://fedoraproject.org/wiki/User:Kumarpraveen http://fedoraproject.org/ http://kumar-pravin.blogspot.com -- https://mail.python.org/mailman/listinfo/python-list
real-life example LC_CTYPE effects?
Hi, The locale category LC_CTYPE may affect character classification and case conversion. That's the theory. Can you give a practical example where this locale setting matters? Eg.: locale.setlocale(locale.LC_CTYPE, loc) m = re.match(\d+, s, re.I | re.L) So with two different values for loc, and s is identical, there will or won't be a match. Thanks! Albert-Jan -- https://mail.python.org/mailman/listinfo/python-list
Re: Building lists
On 10/20/2014 12:49 PM, Seymore4Head wrote: On Mon, 20 Oct 2014 20:40:18 +0100, MRAB pyt...@mrabarnett.plus.com wrote: snip Do you have to know the number of items the list will have before making it? No, it is not necessary, lists are NOT the same as arrays in other languages. But it IS possible to create an initial list of a specific size: myList = [None] * 50 That creates a 50-element list with each element set to None. (BTW, the indexes are from 0-49, not 0-50.) I have found this occasionally useful, but I'll emphasize, it's only RARELY useful. The .append() method is far more versatile. As to your original problem: my question to you is what is your purpose? 1) To solve this particular problem, using Python. or 2) To explore the usage of lists, applying them to this problem. If your purpose is the first, then I agree with the advice you have already been given here. Dictionaries are a much better fit to this problem. If your purpose is the second, then go ahead and use this for your exploration. But realize that to more experienced Pythonistas this would be a very un-pythonic approach. Even better would be to try multiple approaches -- lists, dictionaries, lists with dictionaries, dictionaries with lists or tuples... And any other combinations you can come up with. This will give you even more experience, and allow you to evaluate the different approaches. And no, I will not give you a ready-made canned answer. For one thing, your description is too vague to effectively do that. Good luck. -=- Larry -=- -- https://mail.python.org/mailman/listinfo/python-list
Re: Building lists
On 10/20/2014 12:49 PM, Seymore4Head wrote: On Mon, 20 Oct 2014 20:40:18 +0100, MRAB pyt...@mrabarnett.plus.com wrote: snip Do you have to know the number of items the list will have before making it? No. Lists are NOT the same as arrays in other languages. But it IS possible to create an initial list of a specific size: myList = [None] * 50 This creates a 50-element list with each element set to None. (BTW, that makes the indexes 0-49, not 0-50.) I have occasionally found this useful, but I emphasize it is only RARELY useful. The .append() method is far more versatile. As to your original problem: my question to you is what is your purpose? 1) To solve this particular problem, using Python. or 2) To explore the usage of lists by applying them to this problem. If your purpose is the first, then I agree with the advice you have already been given here -- dictionaries are a much better fit to this problem. If your purpose is the second, then go ahead, have at it. But realize that to more experienced Pythonistas this approach the very un-pythonic for this problem. It would be even better to try multiple approaches -- lists, dictionaries, lists with dictionaries, dictionaries with lists or tuples... or whatever combination you can come up with. This will give you even more experience and allow you to evaluate these various techniques. And no, I won't give you a ready-made canned answer. For one thing your original description is much too vague. But good luck! -=- Larry -=- -- https://mail.python.org/mailman/listinfo/python-list
Re: When to use assert
On Saturday, November 16, 2013 11:35:50 PM UTC-8, Steven D'Aprano wrote: The question of when to use the assert statement comes up occasionally, usually in response to somebody misusing it, so I thought I'd write a post describing when and why to use assertions, and when not to. For those who aren't aware of it, Python's assert checks a condition, if it is true it does nothing, and if it is false it raises an AssertionError with an optional error message. For example: py x = 23 py assert x 0, x is not zero or negative py assert x%2 == 0, x is not an even number Traceback (most recent call last): File stdin, line 1, in module AssertionError: x is not an even number Many people use asserts as a quick and easy way to raise an exception if an argument is given the wrong value. But this is wrong, dangerously wrong, for two reasons. The first is that AssertionError is usually the wrong error to give when testing function arguments. You wouldn't write code like this: if not isinstance(x, int): raise AssertionError(not an int) you'd raise TypeError instead. assert raises the wrong sort of exception. But, and more dangerously, there's a twist with assert: it can be compiled away and never executed, if you run Python with the -O or -OO optimization flags, and consequently there is no guarantee that assert statements will actually be run. When using assert properly, this is a feature, but when assert is used inappropriately, it leads to code that is completely broken when running with the -O flag. When should use assert? In no particular order, assertions should be used for: * defensive programming; * runtime checks on program logic; * checking contracts (e.g. pre-conditions and post-conditions); * program invariants; and * checked documentation. (It's also acceptable to use assert when testing code, as a sort of quick- and-dirty poor man's unit testing, so long as you accept that the tests simply won't do anything if you run with the -O flag. And I sometimes use assert False in code to mark code branches that haven't been written yet, and I want them to fail. Although raise NotImplementedError is probably better for that, if a little more verbose.) Opinions on assertions vary, because they can be a statement of confidence about the correctness of the code. If you're certain that the code is correct, then assertions are pointless, since they will never fail and you can safely remove them. If you're certain the checks can fail (e.g. when testing input data provided by the user), then you dare not use assert since it may be compiled away and then your checks will be skipped. It's the situations in between those two that are interesting, times when you're certain the code is correct but not *quite* absolutely certain. Perhaps you've missed some odd corner case (we're all only human). In this case an extra runtime check helps reassure you that any errors will be caught as early as possible rather than in distant parts of the code. (This is why assert can be divisive. Since we vary in our confidence about the correctness of code, one person's useful assert is another person's useless runtime test.) Another good use for asserts is checking program invariants. An invariant is some condition which you can rely on to be true unless a bug causes it to become false. If there's a bug, better to find out as early as possible, so we make a test for it, but we don't want to slow the code down with such tests. Hence assert, which can be turned on in development and off in production. An example of an invariant might be, if your function expects a database connection to be open when it starts, and promises that it will still be open when it returns, that's an invariant of the function: def some_function(arg): assert not DB.closed() ... # code goes here assert not DB.closed() return result Assertions also make good checked comments. Instead of writing a comment: # when we reach here, we know that n 2 you can ensure it is checked at runtime by turning it into an assert: assert n 2 Assertions are also a form of defensive programming. You're not protecting against errors in the code as it is now, but protecting against changes which introduce errors later. Ideally, unit tests will pick those up, but let's face it, even when tests exist at all, they're often incomplete. Build-bots can be down and nobody notices for weeks, or people forget to run tests before committing code. Having an internal check is another line of defence against errors sneaking in, especially those which don't noisily fail but cause the code to malfunction and return
Re: Is there an easy way to control indents in Python
On Monday, 20 October 2014 18:56:05 UTC+1, Ian wrote: Rather, I'm saying that where the blank line is should be the start of a new function. There would still be a blank line, just no longer inside the function. Now, maybe you think there should be more blank lines in the above, in which case we'll just have to disagree on that point. Why did you separate the above 2 sequences of thoughts by a blank line? Is the inherent pause in the communication of your thoughts not also applicable to your code? Where we see the pause between thoughts appears to be in a different place. I see them both within the function and between the functions and I assume you see them between the functions only. BTW I'm more than happy to disagree. There is no right or wrong answer unless Guido wants to pronounce on the issue :-J -- https://mail.python.org/mailman/listinfo/python-list
Re: OS X Menubar in Tkinter
On Tuesday, October 21, 2014 12:59:08 AM UTC-5, Mark Lawrence wrote: On 21/10/2014 02:34, Noble Bell wrote: I'm pleased to see that you have an answer. In return would you please access this list via https://mail.python.org/mailman/listinfo/python-list or read and action this https://wiki.python.org/moin/GoogleGroupsPython to prevent us seeing double line spacing and single line paragraphs, thanks. I was not aware of the issue. Sorry. I will correct my posts in the future. Thanks for pointing it out to me. NB -- https://mail.python.org/mailman/listinfo/python-list
Re: real-life example LC_CTYPE effects?
On Mon, Oct 20, 2014, at 16:33, Albert-Jan Roskam wrote: Hi, The locale category LC_CTYPE may affect character classification and case conversion. That's the theory. Can you give a practical example where this locale setting matters? Eg.: locale.setlocale(locale.LC_CTYPE, loc) m = re.match(\d+, s, re.I | re.L) So with two different values for loc, and s is identical, there will or won't be a match. You're generally isolated from this by using unicode strings - there are only a few unicode characters that have different case mappings in different languages. LC_CTYPE was designed in an era of 8-bit character sets. For example, in a Russian locale with KOI8-R character set, C0-DF are all lowercase letters, and E0-FF are all the uppercase equivalent, whereas in an English or other western european locale with ISO-8859-1, C0-DF [except D7] are all uppercase letters, with the lowercase versions in E0-FF [except F7], and in a Hebrew ISO-8859-8 locale only E0-FA are letters and are not uppercase/lowercase. Try setting the locale to tr_TR and matching i against I, for a demonstration of one of the few remaining effects this can have. -- https://mail.python.org/mailman/listinfo/python-list
Re: Building lists
On Tue, 21 Oct 2014 00:11:38 -0700, Larry Hudson org...@yahoo.com wrote: On 10/20/2014 12:49 PM, Seymore4Head wrote: On Mon, 20 Oct 2014 20:40:18 +0100, MRAB pyt...@mrabarnett.plus.com wrote: snip Do you have to know the number of items the list will have before making it? No, it is not necessary, lists are NOT the same as arrays in other languages. But it IS possible to create an initial list of a specific size: myList = [None] * 50 That creates a 50-element list with each element set to None. (BTW, the indexes are from 0-49, not 0-50.) I have found this occasionally useful, but I'll emphasize, it's only RARELY useful. The .append() method is far more versatile. As to your original problem: my question to you is what is your purpose? 1) To solve this particular problem, using Python. or 2) To explore the usage of lists, applying them to this problem. If your purpose is the first, then I agree with the advice you have already been given here. Dictionaries are a much better fit to this problem. If your purpose is the second, then go ahead and use this for your exploration. But realize that to more experienced Pythonistas this would be a very un-pythonic approach. Even better would be to try multiple approaches -- lists, dictionaries, lists with dictionaries, dictionaries with lists or tuples... And any other combinations you can come up with. This will give you even more experience, and allow you to evaluate the different approaches. And no, I will not give you a ready-made canned answer. For one thing, your description is too vague to effectively do that. Good luck. -=- Larry -=- The concept I was asking about was a master list with my example of 1,2,3 as a index for the second and third items. It was suggested to make my task easier. It turns out that it didn't. Thanks for all the suggestions, though. -- https://mail.python.org/mailman/listinfo/python-list
Re: Building lists
On Tue, 21 Oct 2014 00:40:06 -0700, Larry Hudson org...@yahoo.com wrote: On 10/20/2014 12:49 PM, Seymore4Head wrote: On Mon, 20 Oct 2014 20:40:18 +0100, MRAB pyt...@mrabarnett.plus.com wrote: snip Do you have to know the number of items the list will have before making it? No. Lists are NOT the same as arrays in other languages. But it IS possible to create an initial list of a specific size: myList = [None] * 50 This creates a 50-element list with each element set to None. (BTW, that makes the indexes 0-49, not 0-50.) I have occasionally found this useful, but I emphasize it is only RARELY useful. The .append() method is far more versatile. As to your original problem: my question to you is what is your purpose? 1) To solve this particular problem, using Python. or 2) To explore the usage of lists by applying them to this problem. If your purpose is the first, then I agree with the advice you have already been given here -- dictionaries are a much better fit to this problem. If your purpose is the second, then go ahead, have at it. But realize that to more experienced Pythonistas this approach the very un-pythonic for this problem. It would be even better to try multiple approaches -- lists, dictionaries, lists with dictionaries, dictionaries with lists or tuples... or whatever combination you can come up with. This will give you even more experience and allow you to evaluate these various techniques. And no, I won't give you a ready-made canned answer. For one thing your original description is much too vague. But good luck! -=- Larry -=- Also, I had no need to impose a limit on the list, but I asked that question, just in case. Thanks -- https://mail.python.org/mailman/listinfo/python-list
Re: Building lists
On Mon, 20 Oct 2014 21:31:05 -0400, Dennis Lee Bieber wlfr...@ix.netcom.com wrote: On Mon, 20 Oct 2014 17:25:31 -0400, Seymore4Head Seymore4Head@Hotmail.invalid declaimed the following: The thing is I am not really sure what I want. I do know I need more practice to find out. Since I am taking a course now, I can't really ask a direct question and my first example wasn't so good. Unfortunately, that puts your questions at the level of algorithm, not language... Once you know the algorithm and data structures, THEN you can figure out how to map them into Python (or Rexx, or any other language). Thanks -- https://mail.python.org/mailman/listinfo/python-list
Re: Py2App - Could not import Tkinter error from resulting app
On Monday, October 20, 2014 11:07:51 PM UTC-5, Terry Reedy wrote: tkinter imports _tkinter _tkinter connects with tclx.dll and tkx.dll (x is variable) So one possibility is no accessible tcl/tx. I have no idea how py2app is supposed to handle this on your undisclosed system. -- Terry Jan Reedy Thanks for the reply. I am using a Mac. The problem turns out that for some reason when I installed the py2app utility it got installed under python 2.7 instead of python 3, which I am using, and that caused it to create the bundle under 2.x instead of 3.x. The result was it was trying to use python 2.x instead of 3.x syntax. I realized this problem shortly after I posted the question and tried to go back to google groups and delete my post before anyone had seen it. NB -- https://mail.python.org/mailman/listinfo/python-list
Re: Py2App - Could not import Tkinter error from resulting app
On Tue, Oct 21, 2014 at 8:20 AM, Noble Bell nobleb...@gmail.com wrote: I realized this problem shortly after I posted the question and tried to go back to google groups and delete my post before anyone had seen it. In general, that won't work, as lots of people use email ( python-list@python.org) or Usenet (comp.lang.python) to read the group. In fact, Google Groups is very much a latecomer to the party. More importantly, just because you made a mistake doesn't mean you should delete your message. Other people might make the same mistake and learn from your post what the solution is. Skip -- https://mail.python.org/mailman/listinfo/python-list
Re: Is there an easy way to control indents in Python
On Wednesday, October 15, 2014 12:13:51 PM UTC+10, ryguy7272 wrote: I'm just learning Python. One of the best ways to learn any language is to type in the example code by hand. As you type in the code you will make mistakes, you will learn from your mistakes and that will help you learn the language and in the process make you a better programmer. So if you really want to learn the language just type in the code by hand and the problem goes away. -- https://mail.python.org/mailman/listinfo/python-list
string processing - some problems whenever I have to parse a more complex string
Hello together,
currently I have to parse a string in an atomic way. Normally - in this case
too - I have a counter variable to keep the current position inside the string.
So far, I think this is the most flexible solution to do some lookaround's
inside the string if necessary. Subroutines will be feed by the underlying data
and the current position. A subroutine returns a tuple of the new position and
the result. But I would like process subroutines with the same flexibillity
(slicing/lookaround) but without returning the new position every again.
Is there any implementation like C++ StringPiece class? Or something like the
following behavior:
s = StringSlice('abcdef')
s
StringSlice('abcdef') at xxx
s[0]
'a'
s.chop(1) # chop the first item
s[0] # 'b' is the new first item
'b'
s[:2]
'bc'
s.chop(-1) # chop the last item
s[-1]
'e'
s[1:]
'cde'
while s[0] != 'e':
s.chop(1)
s[0]
'e'
s.startswith('e')
True
s.isdigit()
False
Subroutines could chop the number of processed items internally if no error
occours.
Another possibillty will be to chop the current item manually. But I don't know
how efficient this is in case of large strings.
while string:
c = string[0]
# process it ...
string = string[1:]
But this assumes that I have to return rest of the string too and not useful for
my problem covered abrove.
Has anyone some tips what's the best practice to process a complex string,
handling prositions in/and subroutines.
Thanks for all answers...
Cheers
Chris
--
https://mail.python.org/mailman/listinfo/python-list
Pex import problems
Hi all, I'm trying to use Pex (http://pex.readthedocs.org/en/latest/index.html) to include requests in a little script to ping a server from a machine that doesn't come with pip (or much of anything, really) installed. I'm running into problems outputting a pex file that depends on a local script. I've got a command-line file, `client.py`, that can be called with a few command-line options. I can run `pex -r requests -o client.pex -- client.py`, but when I try `./client.pex -h` I get an error about no such file or directory: '-h'. Ok, next attempt: `pex -r requests -o client.pex -e client` -- ImportError: No module named client. I dug around in the code a bit, and from what I can tell, `-e FOO` boils down to `__import__(FOO)`, which I can do, both from the interpreter and from a test script. So what am I missing? The only other option I can think of would be: `pex -r requests -o client.pex` then write a script that calls `client.py` from the pex environment (i.e., `./client.pex client.py -h` or whatever), and bundle the pex file, `client.py`, and the script together. But that seems like a misuse of the tool, at best. Alternatively/additionally: is there any mailing list/help source for pex? It seems like a great project, but I've not been able to find many resources out there, which is why I'm turning to you guys. (Why not just install pip/requests on the target machine? Because this is part of an effort to automate provisioning of a bunch of machines.) (Why not use pants? Because literally all we need is requests, and that seems like overkill.) (Also: anyone who's planning on chewing me out about formatting: I TRIED posting by email to comp.lang.pyt...@googlegroups.com, but it wouldn't let me. Sorry for the extra whitespace.) -- https://mail.python.org/mailman/listinfo/python-list
Re: Pex import problems
On Wed, Oct 22, 2014 at 2:34 AM, Sam Raker sam.ra...@gmail.com wrote: (Also: anyone who's planning on chewing me out about formatting: I TRIED posting by email to comp.lang.pyt...@googlegroups.com, but it wouldn't let me. Sorry for the extra whitespace.) That's because that isn't the mailing list's name. Sign up here: https://mail.python.org/mailman/listinfo/python-list ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Pex import problems
On Tuesday, October 21, 2014 12:05:00 PM UTC-4, Chris Angelico wrote: On Wed, Oct 22, 2014 at 2:34 AM, Sam Raker sam.ra...@gmail.com wrote: (Also: anyone who's planning on chewing me out about formatting: I TRIED posting by email to comp.lang.pyt...@googlegroups.com, but it wouldn't let me. Sorry for the extra whitespace.) That's because that isn't the mailing list's name. Sign up here: https://mail.python.org/mailman/listinfo/python-list ChrisA Thank you for your help. -- https://mail.python.org/mailman/listinfo/python-list
Re: Pex import problems
On Wed, Oct 22, 2014 at 3:36 AM, Sam Raker sam.ra...@gmail.com wrote: That's because that isn't the mailing list's name. Sign up here: https://mail.python.org/mailman/listinfo/python-list ChrisA Thank you for your help. No probs. Sorry I can't help with your main issue, as I'm not at all familiar with pex. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
PyPy3 2.4.0 released
= PyPy3 2.4 - Snow White = We're pleased to announce PyPy3 2.4, which contains significant performance enhancements and bug fixes. You can download the PyPy3 2.4.0 release here: http://pypy.org/download.html PyPy3 Highlights Issues reported with our previous release were fixed after reports from users on our new issue tracker at https://bitbucket.org/pypy/pypy/issues or on IRC at #pypy. Here is a summary of the user-facing PyPy3 specific changes: * Better Windows compatibility, e.g. the nt module functions _getfinalpathname _getfileinformation are now supported (the former is required for the popular pathlib library for example) * Various fsencode PEP 383 related fixes to the posix module (readlink, uname, ttyname and ctermid) and improved locale handling * Switched default binary name os POSIX distributions to 'pypy3' (which symlinks to to 'pypy3.2') * Fixed a couple different crashes related to parsing Python 3 source code Further Highlights (shared w/ PyPy2) Benchmarks improved after internal enhancements in string and bytearray handling, and a major rewrite of the GIL handling. This means that external calls are now a lot faster, especially the CFFI ones. It also means better performance in a lot of corner cases with handling strings or bytearrays. The main bugfix is handling of many socket objects in your program which in the long run used to leak memory. We fixed a memory leak in IO in the sandbox_ code We welcomed more than 12 new contributors, and conducted two Google Summer of Code projects, as well as other student projects not directly related to Summer of Code. * Reduced internal copying of bytearray operations * Tweak the internal structure of StringBuilder to speed up large string handling, which becomes advantageous on large programs at the cost of slightly slower small *benchmark* type programs. * Boost performance of thread-local variables in both unjitted and jitted code, this mostly affects errno handling on linux, which makes external calls faster. * Move to a mixed polling and mutex GIL model that make mutlithreaded jitted code run *much* faster * Optimize errno handling in linux (x86 and x86-64 only) * Remove ctypes pythonapi and ctypes.PyDLL, which never worked on PyPy * Classes in the ast module are now distinct from structures used by the compiler, which simplifies and speeds up translation of our source code to the PyPy binary interpreter * Win32 now links statically to zlib, expat, bzip, and openssl-1.0.1i. No more missing DLLs * Many issues were resolved_ since the 2.3.1 release in June .. _`whats-new`: http://doc.pypy.org/en/latest/whatsnew-2.4.0.html .. _resolved: https://bitbucket.org/pypy/pypy/issues?status=resolved .. _sandbox: http://doc.pypy.org/en/latest/sandbox.html We have further improvements on the way: rpython file handling, numpy linalg compatibility, as well as improved GC and many smaller improvements. Please try it out and let us know what you think. We especially welcome success stories, we know you are using PyPy, please tell us about it! Cheers The PyPy Team -- https://mail.python.org/mailman/listinfo/python-list
Re: Is there an easy way to control indents in Python
On Tue, Oct 21, 2014 at 2:45 AM, Simon Kennedy sffjun...@gmail.com wrote: On Monday, 20 October 2014 18:56:05 UTC+1, Ian wrote: Rather, I'm saying that where the blank line is should be the start of a new function. There would still be a blank line, just no longer inside the function. Now, maybe you think there should be more blank lines in the above, in which case we'll just have to disagree on that point. Why did you separate the above 2 sequences of thoughts by a blank line? Is the inherent pause in the communication of your thoughts not also applicable to your code? Where we see the pause between thoughts appears to be in a different place. I see them both within the function and between the functions and I assume you see them between the functions only. That makes sense. I see two distinct thoughts in the function that I posted: build the graph and search the graph. The blank line separates them. By my view, a function should represent a single, complete thought. Thus, the function should be broken up at the blank line. -- https://mail.python.org/mailman/listinfo/python-list
Re: PyPy3 2.4.0 released
On Wed, Oct 22, 2014 at 4:03 AM, Philip Jenvey pjen...@underboss.org wrote: PyPy3 2.4 - Snow White Interesting choice of name. I flipped through some of the release pages for previous versions and couldn't see a pattern to the names; is there a list somewhere of the code names and why they were selected? I'm sure there's a story behind this! ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Problem with Android Build
Hello, If I have a problem with building Python on an Android device, would this be the list to post it to, or should I post it to python-help or python-dev? Thanks! Cyd -- https://mail.python.org/mailman/listinfo/python-list
Re: Problem with Android Build
On Wed, Oct 22, 2014 at 2:32 AM, Cyd Haselton chasel...@gmail.com wrote: Hello, If I have a problem with building Python on an Android device, would this be the list to post it to, or should I post it to python-help or python-dev? Hi! Start here. If we can't help, python-dev might be the next place to ask, but this is the best first option. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Pex import problems
Hi all, I'm trying to use Pex (http://pex.readthedocs.org/en/latest/index.html) to include requests in a little script to ping a server from a machine that doesn't come with pip (or much of anything, really) installed. I'm running into problems outputting a pex file that depends on a local script. I've got a command-line file, `client.py`, that can be called with a few command-line options. I can run `pex -r requests -o client.pex -- client.py`, but when I try `./client.pex -h` I get an error about no such file or directory: '-h'. Ok, next attempt: `pex -r requests -o client.pex -e client` -- ImportError: No module named client. I dug around in the code a bit, and from what I can tell, `-e FOO` boils down to `__import__(FOO)`, which I can do, both from the interpreter and from a test script. So what am I missing? The only other option I can think of would be: `pex -r requests -o client.pex` then write a script that calls `client.py` from the pex environment (i.e., `./client.pex client.py -h` or whatever), and bundle the pex file, `client.py`, and the script together. But that seems like a misuse of the tool, at best. Alternatively/additionally: is there any mailing list/help source for pex? It seems like a great project, but I've not been able to find many resources out there, which is why I'm turning to you guys. (Why not just install pip/requests on the target machine? Because this is part of an effort to automate provisioning of a bunch of machines.) (Why not use pants? Because literally all we need is requests, and that seems like overkill.) -- https://mail.python.org/mailman/listinfo/python-list
Re: string processing - some problems whenever I have to parse a more complex string
On 10/21/14 10:32 AM, CWr wrote:
Is there any implementation like C++ StringPiece class? Or something like the
following behavior:
s = StringSlice('abcdef')
s
StringSlice('abcdef') at xxx
s[0]
'a'
s.chop(1) # chop the first item
s[0] # 'b' is the new first item
'b'
s[:2]
'bc'
s.chop(-1) # chop the last item
s[-1]
'e'
s[1:]
'cde'
while s[0] != 'e':
s.chop(1)
s[0]
'e'
s.startswith('e')
True
s.isdigit()
False
You could certainly implement a StringSlice object that worked like
this. Store the string, the start and end indexes. Override
__getitem__ to adjust indexes and slice endpoints, implement chop() to
change the start or end indexes. Implementing the string operations
would get messy, likely you would materialize the sliced string to make
them work.
--
Ned Batchelder, http://nedbatchelder.com
--
https://mail.python.org/mailman/listinfo/python-list
Re: Problem with Android Build
On Tue, Oct 21, 2014 at 12:36 PM, Chris Angelico ros...@gmail.com wrote: On Wed, Oct 22, 2014 at 2:32 AM, Cyd Haselton chasel...@gmail.com wrote: Hello, If I have a problem with building Python on an Android device, would this be the list to post it to, or should I post it to python-help or python-dev? Hi! Start here. If we can't help, python-dev might be the next place to ask, but this is the best first option. ChrisA -- https://mail.python.org/mailman/listinfo/python-list Thanks! I'm building Python v2.7.8 on my Android device in a fakechroot environment with GCC 4.8.0 and make fails with the following error Could not import runpy module Running the Makefile command (minus the LD_LIBRARY_PATH prepend) produces the same error, but running the same command minus the -S returns a File ..Lib/os.py, line 117, in module raise ImportError, 'no os specific module found' I've configured with ./configure --prefix=/usr/python --build=arm-linux-androideabi --host=arm-linux-androideabi --target=arm-linux-androideabi --enable-shared and ./configure --prefix=/usr/python with the same results. The two post-configure makefile edits I've added are: LDFLAGS=-Wl,--allow-shlib-undefined which allows make to continue past an 'undefined reference to sincos' and RUNSHARED= LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/Python/src which prevents the fakechroot environment from breaking I've also commented out the initpwd references in config.c...as Android doesn't do passwords per se. Any help would be appreciated! -- https://mail.python.org/mailman/listinfo/python-list
No Error; No Output...Nothing
Hey everyone, I'm trying to run this code.
import os
import pickle
#import urllib2
from urllib.request import urlopen
#import cookielib
import http.cookiejar
import re
import time
import numpy as np
#import pylab as pl
# symbol - USDEUR=X - problem that the server sometimes returns 0.0
def getSpotPrice(symbol):
numberOfAttempts = 0
while numberOfAttempts 10:
url =
'http://download.finance.yahoo.com/d/quotes.csv?s='+symbol+'f=l1e=.cs'
fxrate_pure = urllib2.urlopen(url).read()
fxrate = fxrate_pure.strip()
if fxrate != 0.00:
return fxrate
else:
numberOfAttempts += 1
time.sleep(1)
raise Exception(Unable to obtain market data from Yahoo! ... wrong
ticker???)
# symbol = the yahoo ticker; the expected tickers of the components contain
alphanumerical characters or dot or hyphen; if the yahoo format changes,
nothing is returned
def getConstituentsOfAnIndexFromYahoo(symbol):
url = 'http://finance.yahoo.com/q/cp?s=%s' % symbol
p = re.compile('td class=\yfnc_tabledata1\ba
href=\/q\?s=([A-Z0-9\.\-]*)\')
components = []
pageIndex = 0
finished = False
while not finished:
if pageIndex == 0:
actualUrl = url
else:
actualUrl = url + c= + str(pageIndex)
pageResults = p.findall(urllib2.urlopen(actualUrl).read())
if len(pageResults) == 0:
finished = True
else:
components.extend(pageResults)
pageIndex+=1
return components
# prices = data[:,6] or prices = data[:, title.index(Adj Close)],
pl.num2date(data[:,1]) back dates
# syntax
http://ichart.yahoo.com/table.csv?s={Yahoo.Symbol.[isin]}a={Von.M-1}b={Von.T}c={Von.J}d={Bis.M}e={Bis.T}f={Bis.
J}g=dy=0z=jdsuignore=.csv
def getNumpyHistoricalTimeseries(symbol,fromDate, toDate):
f = urllib2.urlopen('http://ichart.yahoo.com/table.csv?a='+
str(fromDate.month -1) +'c='+ str(fromDate.year) +'b=' + str(fromDate.day) +
'e='+ str(toDate.day) + 'd='+ str(toDate.month-1) +'g=df=' +
str(toDate.year) + 's=' + symbol + 'ignore=.csv')
header = f.readline().strip().split(,)
#return np.loadtxt(f, dtype=np.float, delimiter=,, converters={0:
pl.datestr2num})
I commented out the import pylab as pl because I couldn't get the
matplotlib.pylab import working. So, anyway, I hit F5, and it seems to run,
but it doesn't really do anything. Isn't this either supposed to be
downloading data from the web, or throwing an error so I can troubleshoot, and
try to figure out what's going on? It's hard to troubleshoot, when you don't
get any error. Does this work for others?
Thanks.
--
https://mail.python.org/mailman/listinfo/python-list
Re: No Error; No Output...Nothing
On Wed, Oct 22, 2014 at 8:44 AM, ryguy7272 ryanshu...@gmail.com wrote: So, anyway, I hit F5, and it seems to run, but it doesn't really do anything. It defines a few functions, but nothing ever calls them, so you won't see much out of it :) Try adding some code at the bottom that actually calls one of your functions, providing constants for whatever parameters it needs. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: No Error; No Output...Nothing
On Tue, Oct 21, 2014 at 2:44 PM, ryguy7272 ryanshu...@gmail.com wrote: I commented out the import pylab as pl because I couldn't get the matplotlib.pylab import working. So, anyway, I hit F5, and it seems to run, but it doesn't really do anything. Isn't this either supposed to be downloading data from the web, or throwing an error so I can troubleshoot, and try to figure out what's going on? It's hard to troubleshoot, when you don't get any error. Does this work for others? No where in the code are any of the functions being called, so unless your F5 is bound to something that will call some functions, I would not expect it to do anything. Chris -- https://mail.python.org/mailman/listinfo/python-list
Re: string processing - some problems whenever I have to parse a more complex string
On 10/21/2014 10:32 AM, CWr wrote:
Hello together,
currently I have to parse a string in an atomic way. Normally - in this case
too - I have a counter variable to keep the current position inside the string.
So far, I think this is the most flexible solution to do some lookaround's
inside the string if necessary. Subroutines will be feed by the underlying data
and the current position. A subroutine returns a tuple of the new position and
the result. But I would like process subroutines with the same flexibillity
(slicing/lookaround) but without returning the new position every again.
Is there any implementation like C++ StringPiece class?
I am going to guess that this is a string view class that encapsulates a
piece of an underlying class. Otherwise there is no point.
A view class depends on a primary, independently accessible class for
its data. There are two main categories. A subview gives the primary
class interface to a part of the primary data. Numpy had array subviews
an I presume you are talking about string subviews here. An altview
class gives an alternative interface to the primary data. Dict views
are examples.
If the primary object is mutable, one reason to use a view instead of a
copy is to keep the data for two objects synchronized. This does not
apply to strings.
Another reason is to save memory space. The downside is that the
primary data cannot be erased until *both* objects are deleted.
Moreover, if the primary data is small or the subview data is a small
fraction of the primary data, the memory saving is small. So small
subviews that persist after the primary object may end up costing more
memory than they save. This is one reason Python does not have string
subview. The numpy array view use case is large subarrays of large
arrays that have to persist through a calculation anyway.
Another reason Python lack sequence subviews is that the extra data
needed for a contiguous slice are only the start and stop indexes.
These can easily be manipulated directly without wrapping them in a
class. And anyone who does want a method interface can easily create a
class to their liking.
To answer your question, I tried
https://pypi.python.org/pypi?%3Aaction=searchterm=string+viewsubmit=search
and did not find anything. 'view' matches the generic use of 'view', as
well as 'views', 'viewed', 'viewer', 'review', and 'preview'.
The third answer here
https://stackoverflow.com/questions/10085568/slices-to-immutable-strings-by-reference-and-not-copy
has a StringView class that could be modifed to work on 3.x by removing
the unneeded use of buffer.
Or something like the following behavior:
s = StringSlice('abcdef')
s = 'abcdef'
a, b = 0, len(s) # s start, s end
s
StringSlice('abcdef') at xxx
s[0]
s[a]
'a'
s.chop(1) # chop the first item
s[0] # 'b' is the new first item
a += 1
s[a]
'b'
s[:2]
s[a:a+2]
'bc'
s.chop(-1) # chop the last item
s[-1]
b -= 1
s[b-1]
'e'
s[1:]
s[a+1:b]
'cde'
while s[0] != 'e':
s.chop(1)
s[0]
while s[a] != 'e':
a += 1
s[a]
'e'
s.startswith('e')
s[a:b].startswith('e')
True
s.isdigit()
s[a:b].isdigit()
False
Subroutines could chop the number of processed items internally if no error
occours.
Another possibillty will be to chop the current item manually. But I don't know
how efficient this is in case of large strings.
while string:
c = string[0]
# process it ...
string = string[1:]
This is extremely bad as it replaces the O(n) processing (below) with
O(n*n) processing. In general, the right way to linearly process any
iterable is
for item in iterable:
process(c)
or sometimes
for index, item in enumerate(iterable):
process(index, item)
or even, for sequences, (but not when the first option above suffices)
for index in range(len(sequence)):
process(index, sequence)
--
Terry Jan Reedy
--
https://mail.python.org/mailman/listinfo/python-list
Re: No Error; No Output...Nothing
On 10/21/2014 05:44 PM, ryguy7272 wrote: Hey everyone, I'm trying to run this code. ... I commented out the import pylab as pl because I couldn't get the matplotlib.pylab import working. So, anyway, I hit F5, and it seems to run, but it doesn't really do anything. Isn't this either supposed to be downloading data from the web, or throwing an error so I can troubleshoot, and try to figure out what's going on? It's hard to troubleshoot, when you don't get any error. Does this work for others? Thanks. No, it isn't supposed to be downloading data from the web. You have defined a few functions but you're not actually calling any of them. The script terminates successfully with no output, since there's nothing for it to do after executing your imports and function definitions. Additionally, your attempted use of 'urllib2' inside getConstituentsOfAnIndexFromYahoo will fail. Your import of that module is commented out, since (from your imports) you're presumably using Python 3 and a module with that name no longer exists. Since you have 'from urllib.request import urlopen', you can use the 'urlopen' module without any fully-qualified name. The relevant line in your getConstituentsOfAnIndexFromYahoo function should be 'pageResults = p.findall(urlopen(...'. MMR... -- https://mail.python.org/mailman/listinfo/python-list
Re: No Error; No Output...Nothing
On Tuesday, October 21, 2014 5:44:33 PM UTC-4, ryguy7272 wrote:
Hey everyone, I'm trying to run this code.
import os
import pickle
#import urllib2
from urllib.request import urlopen
#import cookielib
import http.cookiejar
import re
import time
import numpy as np
#import pylab as pl
# symbol - USDEUR=X - problem that the server sometimes returns 0.0
def getSpotPrice(symbol):
numberOfAttempts = 0
while numberOfAttempts 10:
url =
'http://download.finance.yahoo.com/d/quotes.csv?s='+symbol+'f=l1e=.cs'
fxrate_pure = urllib2.urlopen(url).read()
fxrate = fxrate_pure.strip()
if fxrate != 0.00:
return fxrate
else:
numberOfAttempts += 1
time.sleep(1)
raise Exception(Unable to obtain market data from Yahoo! ... wrong
ticker???)
# symbol = the yahoo ticker; the expected tickers of the components contain
alphanumerical characters or dot or hyphen; if the yahoo format changes,
nothing is returned
def getConstituentsOfAnIndexFromYahoo(symbol):
url = 'http://finance.yahoo.com/q/cp?s=%s' % symbol
p = re.compile('td class=\yfnc_tabledata1\ba
href=\/q\?s=([A-Z0-9\.\-]*)\')
components = []
pageIndex = 0
finished = False
while not finished:
if pageIndex == 0:
actualUrl = url
else:
actualUrl = url + c= + str(pageIndex)
pageResults = p.findall(urllib2.urlopen(actualUrl).read())
if len(pageResults) == 0:
finished = True
else:
components.extend(pageResults)
pageIndex+=1
return components
# prices = data[:,6] or prices = data[:, title.index(Adj Close)],
pl.num2date(data[:,1]) back dates
# syntax
http://ichart.yahoo.com/table.csv?s={Yahoo.Symbol.[isin]}a={Von.M-1}b={Von.T}c={Von.J}d={Bis.M}e={Bis.T}f={Bis.
J}g=dy=0z=jdsuignore=.csv
def getNumpyHistoricalTimeseries(symbol,fromDate, toDate):
f = urllib2.urlopen('http://ichart.yahoo.com/table.csv?a='+
str(fromDate.month -1) +'c='+ str(fromDate.year) +'b=' + str(fromDate.day)
+ 'e='+ str(toDate.day) + 'd='+ str(toDate.month-1) +'g=df=' +
str(toDate.year) + 's=' + symbol + 'ignore=.csv')
header = f.readline().strip().split(,)
#return np.loadtxt(f, dtype=np.float, delimiter=,, converters={0:
pl.datestr2num})
I commented out the import pylab as pl because I couldn't get the
matplotlib.pylab import working. So, anyway, I hit F5, and it seems to run,
but it doesn't really do anything. Isn't this either supposed to be
downloading data from the web, or throwing an error so I can troubleshoot,
and try to figure out what's going on? It's hard to troubleshoot, when you
don't get any error. Does this work for others?
Thanks.
OK. Thanks everyone!
--
https://mail.python.org/mailman/listinfo/python-list
Re: No Error; No Output...Nothing
On 21/10/2014 23:15, ryguy7272 wrote: OK. Thanks everyone! I'm pleased to see that you have answers. In return would you please access this list via https://mail.python.org/mailman/listinfo/python-list or read and action this https://wiki.python.org/moin/GoogleGroupsPython to prevent us seeing double line spacing and single line paragraphs, thanks. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: Flush stdin
On Mon, Oct 20, 2014 at 9:41 PM, Marko Rauhamaa ma...@pacujo.net wrote:
Dan Stromberg drsali...@gmail.com:
Often with TCP protocols, line buffered is preferred to character
buffered,
Terminal devices support line buffering on write.
Yes, though that's not the only place it's useful.
Line buffering on read is an illusion created by higher-level libraries.
The low-level read function reads in blocks of bytes.
Actually, doesn't line buffering sometimes exist inside an OS kernel?
stty/termios/termio/sgtty relate here, for *ix examples. Supporting
code: http://stromberg.dnsalias.org/~strombrg/ttype/ It turns on
character-at-a-time I/O in the tty driver via a variety of methods for
portability. I wrote it in C before I took an interest in Python.
Also, here's some supporting documentation:
http://man7.org/linux/man-pages/man3/stdout.3.html - excerpt:
Indeed, normally terminal input is line buffered in the kernel.
But even if line buffering (or even character buffering) were never in
the kernel, calling it an illusion is perhaps going a little far.
It's useful sometimes, irrespective of where it comes from.
Illusion has a bit of an undeserved pejorative connotation.
Also, it's a straightforward way of framing your data, to avoid
getting messed up by Nagle or fragmentation.
Nagle affects the communication between the peer OS kernels and isn't
directly related to anything the application does.
Actually, Nagle can cause two or more small packets to be merged,
which is something an application must be able to deal with, because
they could show up in the receiving application as one or more (but
anyway: fewer) merged recv()'s. That's one reason why something like
http://stromberg.dnsalias.org/~strombrg/bufsock.html can be helpful.
Also, Nagle doesn't
play any role with pipes.
Yes, but pipes aren't the only thing involved in the OP's question.
You simplified the problem down to pipes, but that doesn't really
capture the complete essence of the matter. Nagle is one of the
reasons.
$ bash ./test.sh | strace python3 ./test.py
...
read(0, x, 4096) = 1
read(0, x, 4096) = 1
read(0, x, 4096) = 1
read(0, x, 4096) = 1
read(0, x, 4096) = 1
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 3), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x7f3143bab000
write(1, 120\n, 4120
)= 4
...
This is tremendously inefficient. It demands a context switch for
every character.
Inefficiency isn't an issue when you generate one byte a second.
Of course, but who's doing one byte per second? You and I in our
tests, and perhaps some application developers with remarkably
undemanding I/O. That doesn't really mean we should _recommend_ a
series of os.read(0, 1)'s.
If data
were generated at a brisker pace, read(0, ..., 4096) could get more
bytes at a time. Notice that even if the Python code requests 5 bytes,
CPython requests up to 4096 bytes in a single read.
Not if you use os.read(0, 1), for example, which was what you appeared
to be recommending. os.read(0, 1) (when on a pipe) makes a call into
kernel space via a context switch, once for each os.read(0, 1).
I guess I should add that when you do an os.read(0, 1), and see it
show up in strace, strace is showing kernel-userspace interactions,
not library stuff, and not stuff in an application that sits above
libraries. ltrace shows some of the library stuff, but probably not
all of it - I haven't studied ltrace as much as I have strace.
Just wondering: Are we helping the OP?
--
https://mail.python.org/mailman/listinfo/python-list
Re: Struggling with python-daemon and subprocess module to work together
On Mon, Oct 20, 2014 at 2:16 AM, Praveen Kumar kumarpraveen.nit...@gmail.com wrote: I am writing a very basic server side application[0] which get data from a client and create a virtual machine using provided data and also tells client about what's going on during *virt-install* command execution. Previously this basic server is executed using *openvt* but I thought it would be nice to have a daemon process for it and used python-daemon. Issue I am facing is after some time server will stop sending data to client which suppose to happen in #120 and get error message Cannot run interactive console without a controlling TTY . I am not able to figure out issue is occurred due to module or I missed something during daemon initialization. I believe you either want a pty, to make your interactive application believe it's on a tty, or a redesign. Here's some Python pty code: http://stromberg.dnsalias.org/~strombrg/pypty/ I imagine pexpect might help too, though I tend to favor going directly to a pty. HTH -- https://mail.python.org/mailman/listinfo/python-list
Re: Struggling with python-daemon and subprocess module to work together
Praveen Kumar kumarpraveen.nit...@gmail.com writes: Previously this basic server is executed using *openvt* but I thought it would be nice to have a daemon process for it and used python-daemon. An important difference is that a daemon process has no controlling terminal, by definition. Issue I am facing is after some time server will stop sending data to client which suppose to happen in #120 and get error message Cannot run interactive console without a controlling TTY . So, if the code you're running inside that daemon process needs access to a terminal, you have at least two options: * If the code really needs to talk on an interactive terminal, fake it with a pseudoterminal, and keep that pseudoterminal (its file handle) open. * If, as is probably the case, the code doesn't actually need to talk on an interactive terminal, cut that part of the code out entirely and the rest will run fine inside the daemon. -- \ “I may disagree with what you say, but I will defend to the | `\death your right to mis-attribute this quote to Voltaire.” | _o__) —Avram Grumer, rec.arts.sf.written, 2000-05-30 | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: Flush stdin
On 21Oct2014 16:16, Dan Stromberg drsali...@gmail.com wrote: [...snip...] This is tremendously inefficient. It demands a context switch for every character. Inefficiency isn't an issue when you generate one byte a second. Of course, but who's doing one byte per second? You and I in our tests, and perhaps some application developers with remarkably undemanding I/O. That doesn't really mean we should _recommend_ a series of os.read(0, 1)'s. Indeed not. But there is one glaring exception: the shell's read builtin. Because it can be interspersed in a script between other input-consuming commands, it _must_ read no more than one line, and therefore is has to read in increments of 1 character. Of course, that says nothing about the upstream write() granularity. I now return y'all to your regularly sheduled nit picking. Cheers, Cameron Simpson c...@zip.com.au If it ain't broken, keep playing with it. -- https://mail.python.org/mailman/listinfo/python-list
Re: When to use assert
Anton wrote:
I use ORM and often need to write a function that either takes an id of
the record or already loaded model object. So I end up writing a piece of
code like below:
def do_something(instance):
if isinstance(instance_or_id, int):
instance = Model.get(instance)
assert isinstance(instance, Model)
# Code that assumes that instance is an object of type Model
do_somthing is not a part of public library, though it is a public
function, which can and intended to be used by other programmers; and
assert should never happen if a client uses the function as planned.
I think you mean the assert should never fail.
That seems like a reasonable use for assert, with a proviso below. It's
behaving like a checked comment or a post-condition test: asserting that
Model.get returns a Model instance.
But, the idea of *requiring* Model.get to return a Model instance may be
inadvisable. It goes against duck-typing, and it prevents Model from making
some kinds of implementation changes that might break your post-condition
that get() always returns an instance. For example, it might return a proxy
object instead, and then your assert will fail.
I
wonder if this use-case is controversial to this part:
Many people use asserts as a quick and easy way to raise an exception if
an argument is given the wrong value. But this is wrong, dangerously
wrong, for two reasons. The first is that AssertionError is usually the
wrong error to give when testing function arguments. You wouldn't write
code like this:
if not isinstance(x, int):
raise AssertionError(not an int)
you'd raise TypeError instead. assert raises the wrong sort of
exception.
No, because the nature of the exception depends on the intent of the test
and the audience who is expected to see it. In an ideal world,
AssertionError should never be seen by the end user, or the developer
calling your code (assuming that she obeys the documented requirements of
your code). A failed assert should be considered an internal error, which
the user never sees. Since the failure:
Model.get has stopped returning Model instances
is likely to be an internal problem (oops, I broke the get() method, better
fix that) rather than an expected error, using assert is okay.
What would *not* be okay is something like this:
def do_something(instance_or_id):
if isinstance(instance_or_id, int):
instance = Model.get(instance_or_id)
assert isinstance(instance, Model)
# Code that assumes that instance is an object of type Model
since that fails with (for example) do_something(None): either the assert is
not checked at all, and there will be some mysterious failure deep inside
your code, or the caller will see AssertionError instead of TypeError,
violating user expectations and good design that type errors should raise
TypeError.
A better way to write this might be to have Model.get() responsible for the
error checking, and then just delegate to it:
class Model:
def get(self, obj):
if isinstance(obj, Model):
return obj
elif isinstance(obj, int):
model = Model(do something here)
return model
raise TypeError('expected an int ID or a Model instance')
def do_something(instance_or_id):
instance = Model.get(instance_or_id)
assert isinstance(instance, Model)
# Code that assumes that instance is an object of type Model
That means that the logic for what is acceptable as a Model is all in one
place, namely the Model.get method, and callers don't need to care about
the pre-condition argument is a Model or an integer ID, they only need to
care about the post-condition result of get() is a Model.
--
Steven
--
https://mail.python.org/mailman/listinfo/python-list
Re: When to use assert
On Wed, Oct 22, 2014 at 12:44 PM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: def do_something(instance_or_id): instance = Model.get(instance_or_id) assert isinstance(instance, Model) # Code that assumes that instance is an object of type Model That means that the logic for what is acceptable as a Model is all in one place, namely the Model.get method, and callers don't need to care about the pre-condition argument is a Model or an integer ID, they only need to care about the post-condition result of get() is a Model. And at that point, the assertion is redundant, on par with: a = len(seq) assert isinstance(a, int) because you shouldn't have to assert what's part of a function's guarantee. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Flush stdin
On Sat, 18 Oct 2014 18:42:00 -0700, Dan Stromberg wrote: On Sat, Oct 18, 2014 at 6:34 PM, Dan Stromberg drsali...@gmail.com wrote: Once the nc process actually write()s the data to its standard output (i.e. desriptor 1, not the stdout FILE*) I'm not sure why you're excluding stdout, but even if nc is using filedes 1 instead of FILE * stdout, isn't it kind of irrelevant? On further reflection, isn't it stdio that does the varied buffering, and filedes 1 that's always unbuffered? IOW, the OP might wish nc was using 1, but it probably can't be given what they're seeing. Yes. stdio does buffering. Writing to stdout stores data in a buffer; that data should eventually be written to descriptor 1, although perhaps not until immediately prior to termination. Which is probably the cause of the OP's problem. If it is, using a pseudo-tty would probably fix it. At startup, stdin and stdout are line-buffered if they are associated with a tty and fully-buffered otherwise (file, pipe, ...); stderr is unbuffered. At least, this is the case on Unix and Windows. The exact requirements of the C standard are: As initially opened, the standard error stream is not fully buffered; the standard input and standard output streams are fully buffered if and only if the stream can be determined not to refer to an interactive device. -- https://mail.python.org/mailman/listinfo/python-list
Re: Flush stdin
On Tue, Oct 21, 2014 at 7:49 PM, Nobody nobody@nowhere.invalid wrote: On Sat, 18 Oct 2014 18:42:00 -0700, Dan Stromberg wrote: On Sat, Oct 18, 2014 at 6:34 PM, Dan Stromberg drsali...@gmail.com wrote: Once the nc process actually write()s the data to its standard output (i.e. desriptor 1, not the stdout FILE*) I'm not sure why you're excluding stdout, but even if nc is using filedes 1 instead of FILE * stdout, isn't it kind of irrelevant? On further reflection, isn't it stdio that does the varied buffering, and filedes 1 that's always unbuffered? IOW, the OP might wish nc was using 1, but it probably can't be given what they're seeing. Yes. stdio does buffering. Writing to stdout stores data in a buffer; that data should eventually be written to descriptor 1, although perhaps not until immediately prior to termination. Which is probably the cause of the OP's problem. Huh. And here I thought I demonstrated elsewhere in this thread, that the buffering between nc and python didn't appear to be the problem. 'found it, here it is again: If I run the following in one tty: nc -l localhost 9000 | /tmp/z ...where /tmp/z has just: #!/usr/bin/python3 import sys for line in sys.stdin.buffer: print(line) And then run the following in another tty on the same computer: while read line; do echo $line; sleep 1; done /etc/passwd | nc localhost 9000 ...then everything acts line buffered, or perhaps even character buffered (the two are pretty indistinguishable in this test). What I see is my /etc/passwd file popping out of the nc -l side, one line at a time, each line one second apart. I suppose this suggests that it's the client that's sending TCP data that is buffering. That, or we're using two different versions of netcat (there are at least two available). -- https://mail.python.org/mailman/listinfo/python-list
Matplotlib: getting a figure to show without plt.show()
I'm using Matplotlib to present a control window with clickable buttons, and to plot things in another window when you click buttons, in the style of the code below. I'm ashamed of the stinky way I use first to call plt.show the first time data are plotted but then to call fig.canvas.draw for subsequent data plots. Can someone tell me the right way? If I call plt.show every time, then after about 40 plots I get RuntimeError: maximum recursion depth exceeded while calling a Python object, which makes sense because I'm getting one layer deeper in callbacks with every plot (plt.show doesn't return). But if I call fig.canvas.draw every time, the window for the data plot never appears on my screen. Thanks for any suggestions. ## import matplotlib.pyplot as plt from matplotlib.widgets import Button def callback(event): global n, first fig = plt.figure(2) fig.clear() plt.plot([0,1],[0,n]) n += 1 # (Pretending something changes from one plot to the next.) if first: first = False plt.show() else: fig.canvas.draw() global n, first n = 0 first = True fig = plt.figure(1) quit_button = Button(plt.axes([.1,.3,.4,.2]), Quit) quit_button.on_clicked(lambda x: plt.close(all)) plot_button = Button(plt.axes([.1,.1,.4,.2]), Plot) plot_button.on_clicked(callback) plt.show() ## -- To email me, substitute nowhere-runbox, invalid-com. -- https://mail.python.org/mailman/listinfo/python-list
Re: Flush stdin
Dan Stromberg drsali...@gmail.com: On Mon, Oct 20, 2014 at 9:41 PM, Marko Rauhamaa ma...@pacujo.net wrote: Nagle affects the communication between the peer OS kernels and isn't directly related to anything the application does. Actually, Nagle can cause two or more small packets to be merged, which is something an application must be able to deal with, because they could show up in the receiving application as one or more (but anyway: fewer) merged recv()'s. Packets have barely anything to do with TCP sockets since they provide an octet stream abstraction. Of course, but who's doing one byte per second? You and I in our tests, and perhaps some application developers with remarkably undemanding I/O. That doesn't really mean we should _recommend_ a series of os.read(0, 1)'s. No, here's my statement: if you need to process input as soon as it becomes available, you can't use sys.stdin. Instead, you need to use os.read(). You typically supply os.read() with a buffer of a kilobyte or more. Key is, os.read() returns right away if fewer bytes are available. Marko -- https://mail.python.org/mailman/listinfo/python-list
[issue22676] Creating the string representation of a module is slower
Georg Brandl added the comment: HasAttr would just call GetAttr and discard the exception. @ OP: what objects are you pickling? You can give them a __module__ attribute to save the lookup in sys.modules. -- nosy: +georg.brandl ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22676 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22676] _pickle's whichmodule() is slow
Antoine Pitrou added the comment: There's no doubt that whichmodule() has grown more complex since 3.2. We probably cannot eliminate the O(number of modules) component, but could at least make it faster. -- stage: - needs patch title: Creating the string representation of a module is slower - _pickle's whichmodule() is slow versions: +Python 3.5 -Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22676 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22683] bisect index out of bounds issue
New submission from Paul Ianas:
The precondition for all the bisect functions is implemented like this:
if lo 0:
raise ValueError('lo must be non-negative')
if hi is None:
hi = len(a)
Now, of course, if hi is given, and hi = 2 * len(a), then we get an
IndexError. In case hi 0, we always get 0 as a result (even if the element is
there).
I think it would be better to treat the hi in the precondition in the same way
as the lo parameter: that means, raise a ValueError in case hi has an illegal
value.
Disclaimer: of course, it makes no sense to give an illegal argument to that
function; still, since lo is treated against illegal values, maybe it's better
to do the same for hi.
At the same time, maybe moving the precondition code in a separate function
(which raises a ValueError in case precondition is not met) makes more sense,
for not repeating the same code in all bisect functions.
A small snippet which reproduces this:
from bisect import bisect_left
a = [1, 2, 3, 4]
idx = bisect_left(a, 2, 0, 10) # 10 2 * 4
print(idx)
--
components: Library (Lib)
messages: 229750
nosy: Paul.Ianas
priority: normal
severity: normal
status: open
title: bisect index out of bounds issue
type: enhancement
versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4, Python 3.5
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22683
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22676] _pickle's whichmodule() is slow
Antoine Pitrou added the comment: Note the problem is easily reproduce. For example we can take numpy where some ufuncs (i.e. function-like objects implemented in C) don't have a __module__. $ PYTHONHASHSEED=0 python3.4 -m timeit -s import pickle, numpy; d=numpy.add pickle.dumps(d) 1000 loops, best of 3: 280 usec per loop $ PYTHONHASHSEED=0 python3.4 -m timeit -s import pickle, numpy; d=numpy.diff pickle.dumps(d) 10 loops, best of 3: 2.74 usec per loop We see that pickling numpy.add (which doesn't have a __module__) is 100x slower than numpy.diff (which has a __module__)! Note I'm forcing PYTHONHASHSEED for consistent results, since whichmodule() uses dict iteration. For comparison, 2.7 is fast enough: $ PYTHONHASHSEED=0 python2.7 -m timeit -s import cPickle as pickle, numpy; d=numpy.add pickle.dumps(d) 10 loops, best of 3: 6.12 usec per loop $ PYTHONHASHSEED=0 python2.7 -m timeit -s import cPickle as pickle, numpy; d=numpy.diff pickle.dumps(d) 10 loops, best of 3: 2.35 usec per loop (varying PYTHONHASHSEED didn't produce any slower results) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22676 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22676] _pickle's whichmodule() is slow
Georg Brandl added the comment: Attached patch addresses the repeated __repr__ calling and gives a 4x speedup here. -- keywords: +patch Added file: http://bugs.python.org/file36985/whichmodule_speedup.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22676 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22676] _pickle's whichmodule() is slow
Antoine Pitrou added the comment: Actually, numpy.add takes a different path. Sorry. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22676 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22676] _pickle's whichmodule() is slow
Antoine Pitrou added the comment: So, a proper way to reproduce it is through Ellipsis (which does go through whichmodule()): $ PYTHONHASHSEED=0 python3.4 -m timeit -s import pickle; d=Ellipsis pickle.dumps(d) 1000 loops, best of 3: 201 usec per loop -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22676 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22676] _pickle's whichmodule() is slow
STINNER Victor added the comment: For example we can take numpy where some ufuncs (i.e. function-like objects implemented in C) don't have a __module__. Oh. I was not aware of that. Is there a way to fix numpy to set the __module__ attribute? Maybe we should warn users that serialiaing objects without __module__ is much slower? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22676 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22676] _pickle's whichmodule() is slow
STINNER Victor added the comment: whichmodule_speedup.diff: I proposed once a more generic solution which is much more complex to implement. Lazy formatting of Exception message: in most cases, the message is not used. Replace AttributeError(message) with AttributeError(attr=name), only format when str(exc) is called. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22676 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22676] _pickle's whichmodule() is slow
Antoine Pitrou added the comment: Victor, see https://github.com/numpy/numpy/issues/4952 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22676 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22676] _pickle's whichmodule() is slow
Antoine Pitrou added the comment: Attached patch produces a 8x speedup on Ellipsis. -- Added file: http://bugs.python.org/file36986/whichmodule.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22676 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22676] _pickle's whichmodule() is slow
Georg Brandl added the comment: I didn't have numpy anyway, I tested on a function with __module__ set to None manually. But I see the same speedup for Ellipsis. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22676 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22683] bisect index out of bounds issue
Changes by Antoine Pitrou pit...@free.fr: -- nosy: +rhettinger, tim.peters ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22683 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22660] Review ssl docs for security recommendations
Roundup Robot added the comment: New changeset 6f6e56bb10aa by Antoine Pitrou in branch '2.7': Issue #22660: update various mentions in the ssl module documentation. https://hg.python.org/cpython/rev/6f6e56bb10aa -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22660 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22660] Review ssl docs for security recommendations
Changes by Antoine Pitrou pit...@free.fr: -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22660 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22607] find by dichotomy the failing test
Xavier de Gaye added the comment:
The attached patch adds the '-X' and '-Y' options to the regrtest tool,
allowing to select a range of tests and a range of their subtests. The patch
is missing the test cases for the moment.
Limitation:
Does not work very well with nested subtest (nested subtests are currently only
used by the unittest test suite itself): subtest numbers are flattened across
all the nested subtests.
Here is the sequence of commands that would have allowed to find the subtest
responsible for the crash in test_capi at issue 22588, without modifying any
code:
./python -m test -X [] test_capi # Get the test count: 18
tests.
./python -m test -X range(1,10) -R 23:23 test_capi# pass
./python -m test -X range(10,15) -R 23:23 test_capi # pass
./python -m test -X range(15,17) -R 23:23 test_capi # pass
./python -m test -X [17] -R 23:23 test_capi # pass
./python -m test -X [18] -R 23:23 test_capi # fail
./python -m test -X [18] -Y [] test_capi# Test 18 has 35 subtests.
./python -m test -X [18] -Y range(1,19) -R 23:23 test_capi # fail
./python -m test -X [18] -Y range(1,10) -R 23:23 test_capi # fail
./python -m test -X [18] -Y range(1,6) -R 23:23 test_capi # pass
./python -m test -X [18] -Y range(6,8) -R 23:23 test_capi # pass
./python -m test -X [18] -Y [8] -R 23:23 test_capi # pass
./python -m test -X [18] -Y [9] -R 23:23 test_capi # fail
Output of the last command:
Test# 18: test__testcapi
subTest# 9: internal, {'name': 'test_incref_decref_API'}
--
Added file: http://bugs.python.org/file36987/regrest_XY_options.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22607
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22684] message.as_bytes() produces recursion depth exceeded
New submission from Pas: Please see the attached traceback (or this http://pastebin.com/WYinRGie for fancy colors). It depends on message size, we're trying to send Multipart MIME messages (a PDF attached, that has an image embedded). After editing flask_mail.py to use the fallback ( message().as_string().encode(self.charset or 'utf-8') ) things work again. If anyone could help confirm if this is a bug, or help me understand how I misuse the library, I'd be grateful. Thanks! -- components: email files: py34-email.message.as_bytes-recursion-depth-exceeded.txt messages: 229762 nosy: barry, pas, r.david.murray priority: normal severity: normal status: open title: message.as_bytes() produces recursion depth exceeded type: behavior versions: Python 3.4 Added file: http://bugs.python.org/file36988/py34-email.message.as_bytes-recursion-depth-exceeded.txt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22684 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell
New submission from wabu: using `p = create_subprocess_exec(..., stdout=subprocess.PIPE, limit=...)`, p.stdout has not transport set, so the underlying protocol is unable to pause the reading of the transport, resulting in high memory usage when slowly consuming input from p.stdout, even if the limit parameter is passed. A workaround is to set the transport manually after creating the subprocess: `p.stdout.set_transport(p._transport.get_pipe_transport(1))`, but this should happen inside the create_subprocess call. -- components: asyncio messages: 229763 nosy: gvanrossum, haypo, wabu, yselivanov priority: normal severity: normal status: open title: memory leak: no transport for pipes by create_subprocess_exec/shell versions: Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22684] message.as_bytes() produces recursion depth exceeded
R. David Murray added the comment: It looks like a bug, but I'm not sure why as_bytes would trigger it but not as_string. Can you supply a copy of the message that fails? The as_string version (assuming the content was all ascii) should be enough to reproduce the issue, since it appears to be happening in the header folding step. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22684 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22686] random.randint does not include endpoint
New submission from FH: Upon inspection, random.randint(a,b) does not include the endpoint b, contrary to documentation, and contrary to the argumentation in Issue7009. To see this, run e.g. sum(np.random.randint(0,1,1000)) which will return 0 repeatedly (statistically very unlikely). I tried both on Kubuntu 14.04/python 2.7 and pythoneverwhere.org Within random.py, randint is (in both 2.7 and 3.5) implemented as def randint(self, a, b): Return random integer in range [a, b], including both end points. return self.randrange(a, b+1) which falsely seems to include the endpoint (as randrange excludes the endpoint). However, upon running it does not. -- components: Library (Lib) messages: 229765 nosy: georg.brandl, orsenthil, sciencebuggy priority: normal severity: normal status: open title: random.randint does not include endpoint type: behavior versions: Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22686 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22686] random.randint does not include endpoint
Mark Dickinson added the comment: You seem to be confusing `np.random.randint`, which is a function from NumPy (not part of core Python), with `random.randint` from the standard library. NumPy's np.random.randint does not include the endpoint. Python's does. As far as I can tell, the documentation is correct for both. Python 2.7.8 (default, Oct 15 2014, 22:04:42) [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin Type help, copyright, credits or license for more information. import random sum(random.randint(0, 1) for _ in range(1000)) 501 Closing as 'not a bug'. -- nosy: +mark.dickinson resolution: - not a bug status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22686 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell
STINNER Victor added the comment: p = create_subprocess_exec(..., stdout=subprocess.PIPE, limit=...) I don't see yield from. Do you wait until the coroutine completes? See the documentation: https://docs.python.org/dev/library/asyncio-subprocess.html#subprocess-using-streams Please write a more complete example to demontrate your issue. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22687] horrible performance of textwrap.wrap() with a long word
New submission from Piotr Engelking: Wrapping a paragraph containing a long word takes a lot of time: $ time python3 -c 'import textwrap; textwrap.wrap(a * 2 ** 16)' real3m14.923s user3m14.792s sys 0m0.016s $ A straightforward replacement is 5000 times faster: $ time python3 -c '(.join(x) for x in zip(*[iter(a * 2 ** 16)] * 70))' real0m0.053s user0m0.032s sys 0m0.016s $ Tested on Debian with python3.4 3.4.2-1 and python2.7 2.7.8-10. -- messages: 229768 nosy: inkerman priority: normal severity: normal status: open title: horrible performance of textwrap.wrap() with a long word type: performance versions: Python 2.7, Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22687 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22687] horrible performance of textwrap.wrap() with a long word
Changes by Serhiy Storchaka storch...@gmail.com: -- nosy: +georg.brandl, serhiy.storchaka versions: +Python 3.5 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22687 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22688] Use the subprocess module in the uuid module
New submission from Serhiy Storchaka: There is a proposition (msg229163) to use the subprocess module instead of os.popen() in the uuid module. I hesitate how classify this issue (behavior, security or enhancement) and for which versions target it. May be this is a dependency of issue17293. -- components: Library (Lib) messages: 229769 nosy: haypo, neologix, serhiy.storchaka priority: normal severity: normal stage: needs patch status: open title: Use the subprocess module in the uuid module ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22688 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22687] horrible performance of textwrap.wrap() with a long word
Serhiy Storchaka added the comment: This particular case is related to the behavior of the wordsep_re regular expression in worst case. When text contains long sequence of words characters which is not ended by a hypen, or long sequence of non-word and non-space characters (and in some other cases), computational complexity of this regular expression matching is quadratic. This is a peculiarity of current implementation of regular expression engine. May be it is possible to rewrite the regular expression so that quadratic complexity will gone, but this is not so easy. The workaround -- use break_on_hyphens=False. -- assignee: - serhiy.storchaka priority: normal - low stage: - needs patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22687 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22506] `dir` on Enum subclass doesn't expose parent class attributes
Ram Rachum added the comment: Thanks for the fix Ethan. Will you also push this to the backport on PyPI? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22506 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22688] Use the subprocess module in the uuid module
STINNER Victor added the comment: Did you see my issue which has a patch? http://bugs.python.org/issue22637 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22688 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22506] `dir` on Enum subclass doesn't expose parent class attributes
Ethan Furman added the comment: Thanks for the reminder! ... Done. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22506 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22506] `dir` on Enum subclass doesn't expose parent class attributes
Ram Rachum added the comment: Thanks! Two questions: 1. Is there a GitHub repo for enum34? Couldn't find a link to it on the PyPI page. 2. Aren't 'name' and 'value' already included in `added_behavior`? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22506 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22506] `dir` on Enum subclass doesn't expose parent class attributes
Ram Rachum added the comment: Also, aren't you excluding a lot of important magic methods from that `dir`? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22506 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22688] Use the subprocess module in the uuid module
Serhiy Storchaka added the comment: Oh, I missed it. Thank you for opening the issue and writting the patch. -- resolution: - duplicate stage: needs patch - resolved status: open - closed superseder: - avoid using a shell in uuid: replce os.popen with subprocess.Popen ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22688 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22637] avoid using a shell in uuid: replce os.popen with subprocess.Popen
Serhiy Storchaka added the comment: Besides few nitpicks the patch LGTM. -- nosy: +serhiy.storchaka ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22637 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell
wabu added the comment: Sorry for the confusion, yes i do the yield from. The stdout stream for the process is actually producing data as it should. The subprocess produces a high amount of data (pbzip2), but is only consumed slowly. Normally when the buffer limit is reached for a stream reader, it calls pause_reading on the transport inside the feed_data method (see https://code.google.com/p/tulip/source/browse/asyncio/streams.py#365), but here this is not happening, as the returned reader has no transport set (p.stdout._transport == None). So it fills up all the memory. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell
Guido van Rossum added the comment: Hm... It does look like there's nothing that tells stdout (which is a StreamReader) about its transport. Wabu, could you experiment with a change to asyncio/subprocess.py where SubprocessStreamProtocol.connection_made() calls self.stdout.set_transport(transport) right after creating self.stdout? On Tue, Oct 21, 2014 at 12:55 PM, wabu rep...@bugs.python.org wrote: wabu added the comment: Sorry for the confusion, yes i do the yield from. The stdout stream for the process is actually producing data as it should. The subprocess produces a high amount of data (pbzip2), but is only consumed slowly. Normally when the buffer limit is reached for a stream reader, it calls pause_reading on the transport inside the feed_data method (see https://code.google.com/p/tulip/source/browse/asyncio/streams.py#365), but here this is not happening, as the returned reader has no transport set (p.stdout._transport == None). So it fills up all the memory. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22506] `dir` on Enum subclass doesn't expose parent class attributes
Ethan Furman added the comment: 1. Is there a GitHub repo for enum34? Couldn't find a link to it on the PyPI page. The repo is at: https://bitbucket.org/stoneleaf/enum34 2. Aren't 'name' and 'value' already included in `added_behavior`? They didn't used to be, but they are now. I'll fix that (eventually). Also, aren't you excluding a lot of important magic methods from that `dir`? We decided the dunder methods were not interesting, so declined to include them in the listing. And no, we are not changing our minds on that. ;) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22506 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell
wabu added the comment: Here's a more complete example @coroutine put_data(filename, queue, chunksize=16000): pbzip2 = yield from asyncio.create_subprocess_exec( 'pbzip2', '-cd', filename, stdout=asyncio.subprocess.PIPE, limit=self.chunksize*2) while not pbzip2.stdout.at_eof(): data = yield from pbzip2.stdout.read(chunksize) yield from queue.put(data) adding the workaround after createing the stream fixes the issue: pbzip2.stdout.set_transport(pbzip2._transport.get_pipe_transport(1)) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell
Guido van Rossum added the comment: Can you confirm that this patch fixes the problem (without you needing the workaround in your own code)? -- keywords: +patch Added file: http://bugs.python.org/file36989/fix22685.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22637] avoid using a shell in uuid: replce os.popen with subprocess.Popen
Roundup Robot added the comment: New changeset 8ee63d0bd7b8 by Victor Stinner in branch 'default': Issue #22637: avoid using a shell in uuid https://hg.python.org/cpython/rev/8ee63d0bd7b8 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22637 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell
wabu added the comment: thanks a lot, the fix works! On 21.10.2014 22:16, Guido van Rossum wrote: Guido van Rossum added the comment: Can you confirm that this patch fixes the problem (without you needing the workaround in your own code)? -- keywords: +patch Added file: http://bugs.python.org/file36989/fix22685.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22506] `dir` on Enum subclass doesn't expose parent class attributes
Roundup Robot added the comment: New changeset 9038b63dad52 by Ethan Furman in branch 'default': Issue22506: remove name value from __dir__ as they now show up automatically https://hg.python.org/cpython/rev/9038b63dad52 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22506 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell
Guido van Rossum added the comment: Victor, do you think this needs a unittest? It seems kind of difficult to test for whether memory fills up (the machine may get wedged if it does :-). On Tue, Oct 21, 2014 at 1:39 PM, wabu rep...@bugs.python.org wrote: wabu added the comment: thanks a lot, the fix works! On 21.10.2014 22:16, Guido van Rossum wrote: Guido van Rossum added the comment: Can you confirm that this patch fixes the problem (without you needing the workaround in your own code)? -- keywords: +patch Added file: http://bugs.python.org/file36989/fix22685.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell
STINNER Victor added the comment: Oh ok, now I understood the issue :-) Attached patch should fix it. It implements Guido's suggestion. There is not unit test yet. -- Added file: http://bugs.python.org/file36990/set_transport.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22689] Posix getenv makes no guarantee of lifetime of returned string
New submission from Aidan Hobson Sayers:
Posix says the following on the subject of getenv:
The returned string pointer might be invalidated or the string content might
be overwritten by a subsequent call to getenv()
(http://pubs.opengroup.org/onlinepubs/9699919799/functions/getenv.html)
Unfortunately, in Modules/getpath.c:
static void
calculate_path(void)
{
[...]
char *_rtpypath = Py_GETENV(PYTHONPATH); /* XXX use wide version on
Windows */
wchar_t *rtpypath = NULL;
wchar_t *home = Py_GetPythonHome();
char *_path = getenv(PATH);
So 3 potential getenv calls in quick succession, meaning _rtpypath and home can
become junk before they get used and Python crashes before it can start up (it
becomes unable to find the site module).
Unfortunately it looks like the assumption that getenv pointers will remain
safe forever is used in a few places in python.
Explicit notes on the correct use of getenv:
https://www.securecoding.cert.org/confluence/display/seccode/ENV34-C.+Do+not+store+pointers+returned+by+certain+functions
Someone's apparently seen this before (but didn't report it?) -
http://sourceforge.net/p/edk2/mailman/edk2-devel/thread/66bd57653246d24e9698b0a6509545a86ddb8...@orsmsx109.amr.corp.intel.com/
--
components: Interpreter Core
messages: 229788
nosy: aidanhs
priority: normal
severity: normal
status: open
title: Posix getenv makes no guarantee of lifetime of returned string
type: crash
versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4, Python 3.5, Python 3.6
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22689
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell
STINNER Victor added the comment: test_pause_reading.py: manual test for pause reading. It should pause reading stdout pipe 4 times on UNIX. -- Added file: http://bugs.python.org/file36991/test_pause_reading.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell
STINNER Victor added the comment: Ok, I just missed completly the Guido posted a patch before me... Maybe we wrote the patch at the same time, I didn't refresh my browser :-) Victor, do you think this needs a unittest? It seems kind of difficult to test for whether memory fills up (the machine may get wedged if it does :-). Since the pause/resume thing is very tricky, it would be much better to have a unit test. We can probably mock a lot of things, we just need to check that the stream reader calls transport.pause_reading() when feed_data() is called with too much data. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22689] Posix getenv makes no guarantee of lifetime of returned string
Changes by STINNER Victor victor.stin...@gmail.com: -- nosy: +haypo versions: -Python 3.2, Python 3.3, Python 3.6 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22689 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell
wabu added the comment: On 21.10.2014 22:41, Guido van Rossum wrote: Guido van Rossum added the comment: Victor, do you think this needs a unittest? It seems kind of difficult to test for whether memory fills up (the machine may get wedged if it does :-). You could setup a the subprocess with asyncio.async(asyncio.create_subprocess_exec(...)) and then let the asyncio loop run for a limited time with loop.run_until_complete(asyncio.sleep(.1)), watching carefully for higher memory usage after each sleep. But still it's difficult to create a reliable unit-test with this ... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22687] horrible performance of textwrap.wrap() with a long word
Changes by Ben Roberts bjr.robe...@gmail.com: -- nosy: +roippi ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22687 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22689] Posix getenv makes no guarantee of lifetime of returned string
Aidan Hobson Sayers added the comment: In case it matters, I'm compiling using Emscripten which implements getenv like so: https://github.com/kripken/emscripten/blob/1.25.2/src/library.js#L3323 (I personally think it's a bizarre way to do it, but technically I think it's ok?) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22689 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell
Guido van Rossum added the comment: Victor, your fix is identical to mine except for the variable names. I like your version better. :-) So now it's just about the unittest. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
