[Python-Dev] Adding a "call_once" decorator to functools

[tom]

Hello,
After a great discussion in python-ideas[1][2] it was suggested that I 
cross-post this proposal to python-dev to gather more comments from those who 
don't follow python-ideas.

The proposal is to add a "call_once" decorator to the functools module that, as 
the name suggests, calls a wrapped function once, caching the result and 
returning it with subsequent invocations. The rationale behind this proposal is 
that:
1. Developers are using "lru_cache" to achieve this right now, which is less 
efficient than it could be
2. Special casing "lru_cache" to account for zero arity methods isn't trivial 
and we shouldn't endorse lru_cache as a way of achieving "call_once" semantics 
3. Implementing a thread-safe (or even non-thread safe) "call_once" method is 
non-trivial
4. It complements the lru_cache and cached_property methods currently present 
in functools.

The specifics of the method would be:
1. The wrapped method is guaranteed to only be called once when called for the 
first time by concurrent threads
2. Only functions with no arguments can be wrapped, otherwise an exception is 
thrown
3. There is a C implementation to keep speed parity with lru_cache

I've included a naive implementation below (that doesn't meet any of the 
specifics listed above) to illustrate the general idea of the proposal:

```
def call_once(func):
sentinel = object()  # in case the wrapped method returns None
obj = sentinel
@functools.wraps(func)
def inner():
nonlocal obj, sentinel
if obj is sentinel:
obj = func()
return obj
return inner
```

I'd welcome any feedback on this proposal, and if the response is favourable 
I'd love to attempt to implement it.

1. 
https://mail.python.org/archives/list/python-id...@python.org/thread/5OR3LJO7LOL6SC4OOGKFIVNNH4KADBPG/#5OR3LJO7LOL6SC4OOGKFIVNNH4KADBPG
2. 
https://discuss.python.org/t/reduce-the-overhead-of-functools-lru-cache-for-functions-with-no-parameters/3956
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5CFUCM4W3Z36U3GZ6Q3XBLDEVZLNFS63/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Python 3.7.4, Visual Studio versions for building modules from source

[Kacvinsky, Tom]

HI,

Not sure if this is the right place to ask, but I am trying to build 
pywin32-224 from source for Python 3.7.4.  I think this might
be the right list as this seems to be a generic problem I am having, but I want 
to focus on one particular module.  First, I know
I could get this via 'pip install', but I want to build from source to see what 
it takes in terms of the Windows SDK and Visual Studio
versions for some other work I am doing.

I am using Python 3.7.4, and I have Visual Studio 2017 15.9 (released July of 
this year).

I see this when running 'python setup.y build':


error: Microsoft Visual C++ 14.1 is required. Get it with "Microsoft Visual C++ 
Build Tools": https://visualstudio.microsoft.com/downloads/

I have tried various compilers from that link (VS 2015, VS 2017, and even VS 
2019), but no joy.  I also have the Windows
SDK 8.1 and 10 installed (pywin32 wants the 8.1 SDK)

Does anyone have any ideas as to what I am doing wrong, or if there is some 
weird underlying issue with setuptools and/or
distutils?

Thanks,

Tom
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4V3F6GRM4LBOYKXBIMAM47HCLA5J6XG3/

[Python-Dev] Re: Python 3.7.4, Visual Studio versions for building modules from source

[Kacvinsky, Tom]

> -Original Message-
> From: Steve Dower 
> Sent: Monday, July 22, 2019 11:36 PM
> To: Kacvinsky, Tom ; python-dev@python.org
> Subject: [Python-Dev] Re: Python 3.7.4, Visual Studio versions for building
> modules from source



> This is probably not the best place to ask, though I can understand why you
> ended up here. Please try this command:
> 
>  -m test -v test_distutils -m *get_vc*
> 
> If the tests pass, post to distutils-...@python.org or the Packaging category
> on https://discuss.python.org/
> 
> If the tests fail, post the output into a new bug at https://bugs.python.org/
> along with
> 
> If the tests are skipped, you probably did not install the C++ or "Python
> Native Development" options for Visual Studio. You can run "Visual Studio
> Installer" from Start to change your installed options.

Tests passed once I used the right Visual Studio Command prompt.  I was using 
the
x64 Native toolchain command shell.  Once I switched to the generic command 
shell,
all is well.  Thanks for the lick in the right direction, it got me to thinking 
that even
though the tools are installed, I should try different Visual Studio shells.

Tom
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/55WQKH6XGVJS7EOQT3LGXBJE2LA7LSEW/

[Python-Dev] PEP for libffi + _ctypes

[Kacvinsky, Tom]

I have been comiling Python 3.8 from source and have had a really difficult time
with getting _ctypes to compile.  I see that libffi is no longer distributed 
with the
Python source code, in preference for what is on the system.  I searched for a
PEP that describes the rationale behind this, but my Google fu must be weak.

I have also seen requests that a patch be committed that makes configuring the
use of libffi easier, but as far as I can tell, these have not been committed.  
It is
something I would like to see as I am in a situation where I cannot depend on 
the
system libffi - we support older Linux distributions that don't have libffi - 
an so I
am making a static libffi to be linked in.

Any guidance on this issue would be helpful.

Thanks,

TOm
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PPAN5U3VXQ6GDQFQE6TPGEPWTK7WRJZY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP for libffi + _ctypes

[Kacvinsky, Tom]

> -Original Message-
> From: Victor Stinner 
> Sent: Thursday, October 17, 2019 10:06 AM
> To: Kacvinsky, Tom 
> Cc: python-dev@python.org
> Subject: Re: [Python-Dev] PEP for libffi + _ctypes
> 
> Hi,
> 
> Our bundled copy of libffi has been removed from Python 3.7 by this change
> which should explain the rationale:
> https://bugs.python.org/issue27979

Thanks.

> 
> Not all Python changes need a PEP. For Windows builds, we provide prebuilt
> binaries of our dependencies:
> https://github.com/python/cpython-source-deps/blob/master/README.rst
> 

I am OK with Windows, it is Linux, as below.

> My notes on Python dependencies:
> https://pythondev.readthedocs.io/files.html
> 

Again, thanks for this.

> 
> > we support older Linux distributions that don't have libffi
> 
> I'm curious, which old Linux distributions don't have libffi? Usually, libffi 
> is
> preinstalled on Linux, only the development header files are required (a
> package with a name like "libffi-devel"). Can't you install libffi on these 
> old
> distributions? IMHO libffi installation should not be the Python problem,
> bundling a library copy in Python is causing more issues compared to
> advantages.

RHEL/CentOS 5 come to mind.  We don't want to get into the use case where
we have to have customers install libffi as we try to minimize the hassles our
customers might have to go through for installing and using our product.

Tom
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/A34TOQS5XBMWO5Q7AOPXDFEI3DDXAUBP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP for libffi + _ctypes

[Kacvinsky, Tom]

Hi,

> -Original Message-
> From: Charalampos Stratakis 
> Sent: Thursday, October 17, 2019 11:26 AM
> To: Kacvinsky, Tom 
> Cc: python-dev@python.org
> Subject: Re: [Python-Dev] Re: PEP for libffi + _ctypes
> 



> Well RHEL5 doesn't include libffi in its default repos, however you can find 
> it
> from the EPEL5 repositories.
> 
> Also available directly through the fedora build system [0].
> 
> It seems that you are asking for libffi to continue to be bundled with cpython
> upstream, however if you are compiling python from source, why is it a
> hassle to install an rpm from the EPEL repos? Is it some form of
> installer/scripts/automation?
> 

Internally, this is fine, but we bundle an embedded Python interpreter in our
product, which runs on all kinds of different flavors/versions of Linux.

Suppose I use the EPEL repository for CentOS 5 ,which I just did a few minutes
ago.  Then I dynamically link against libffi.so.5.  Now, _ctypes will depend on
libffi.so.5, but what happens if a customer runs on a distribution/version of
Linux that has only libffi.so.6?  Then _ctypes won't load.  That's no good at 
all,
_ctypes is crucial for some of our code.

The above is why I want to bundle libffi - either statically linked in or 
dynamically.
In the latter case, we would ship the libffi shared library for consistency 
amongst
Linux distributions.

I hope that makes it clearer what my concern is, and the motivation for doing 
what
I want to do.

Regards,

Tom
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LFC6LOAO6PB3HCQ2LDDPDI62NOLADRM2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Fun with Python 3.8 and Qt

[Kacvinsky, Tom]

Today I discovered the this struct

typedef struct{
const char* name;
int basicsize;
int itemsize;
unsigned int flags;
PyType_Slot *slots; /* terminated by slot==0. */
} PyType_Spec;

with "PyTypeSlot *slots" being on line 190 of object.h causes a problem when 
compiled with code that brings in Qt.  Qt has macro definitions of slots.

With a cursory preprocessing of the file I was working with, using the handy 
gcc options -dM -E, I found that
slots was defined to nothing

#define slots

and hence caused problems when object.h was brought into the mix.

I will try to make a simple reproducer tomorrow.  I know this probably could be 
solved by header file inclusion re-ordering,
or in some cases #undef'ing slots before including Python.h, but I also thought 
the Python dev team would like to know
about this issue.

Tom
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5VRZVIMFB5OMX7SVQCXCH7FT5MCTN26J/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] sqlite3 module and thread-safe SQLite library

[Kacvinsky, Tom]

We ran into an issue where having the SQLite library built with 
-DSQLITE_THREADSAFE=0,
but then the sqlite3 module (really, the _sqlite3.so0 crashing in threading 
code.  So I  have
to ask if it is intended that the sqlite3 Python module always  be built with a 
thread safe
SQLite library.

Thanks,

Tom
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PKHFTUQSLXWYOZNX4QMSNCPAF5Y3TL2H/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python2 as 𝑣 → 𝑒

[Kacvinsky, Tom]

> -Original Message-
> From: Mike Miller 
> Sent: Saturday, April 11, 2020 6:17 PM
> To: python-dev@python.org
> Subject: [Python-Dev] Python2 as 𝑣 → 𝑒
> 
> 
> Unless I've read something wrong, it looks like the final Python 2 release
> (2.7.18) should approximate the math constant e:
> 
>  >>> import math
>  >>> math.e
>  2.718281828459045
> 
> Aka:
> 
>  Python as 𝑣 → 𝑒
>  Python ≈ 𝑒
> 
> Would be fun to note that in the announcement/docs somehow.

Of note, metafont already has the distinction of being versioned after the 
digits of e,
just like TeX is versioned after the digits of pi.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NOVIBPPVXFSY7RDGMM3GULQPVDAV3X7U/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Adding a "call_once" decorator to functools

[Tom Forbes]

> Why not? It's a decorator, isn't it? Just make it check for number of 
> arguments at decoration time and return a different object.

It’s not that it’s impossible, but I didn’t think the current implementation 
doesn’t make it easy 
(https://github.com/python/cpython/blob/cecf049673da6a24435acd1a6a3b34472b323c97/Lib/functools.py#L771
 
).
 You’d ideally want to skip creating all these objects and special case 
`user_function` having no parameters, but then you have an issue with 
`cache_info()` being passed `cache_len()`. So maybe it’s simplest to use the 
`cache` dictionary with a single static key, but then you’re not really helping 
much, or avoiding this method altogether, which seemed pretty messy.

The C implementation seemed easier to implement - you could re-use the `cache` 
member 
(https://github.com/python/cpython/blob/cecf049673da6a24435acd1a6a3b34472b323c97/Modules/_functoolsmodule.c#L1192
 
)
 and store the result of the function call, but that also seemed sub-optimal as 
the `root` member doesn’t make much sense to be there.

At least, that was my line of thought. It basically seemed that it would be 
more trouble than it was potentially worth, and it might be better to spend my 
time on `call_once` than special-casing `lru_cache`.

> But presumably we should be making lru_cache thread safe if it isn’t.

lru_cache is indeed thread-safe but it doesn’t guarantee that the wrapped 
method is only called _once_ per unique set of arguments. It apparently just 
ensures that the internal state of the cache is not corrupted by concurrent 
accesses.

> It's unfortunate that cached_property doesn't work at module level

It is indeed, but a solution that works generally in any function defined at 
the module level or not would be good to have.

> On 27 Apr 2020, at 22:55, Steve Dower  wrote:
> 
> On 27Apr2020 2237, t...@tomforb.es wrote:
>> 2. Special casing "lru_cache" to account for zero arity methods isn't 
>> trivial and we shouldn't endorse lru_cache as a way of achieving "call_once" 
>> semantics
> 
> Why not? It's a decorator, isn't it? Just make it check for number of 
> arguments at decoration time and return a different object.
> 
> That way, people can decorate their functions now and get correct behaviour 
> (I assume?) on 3.8 and earlier, and also a performance improvement on 3.9, 
> without having to do any version checking.
> 
> This part could even be written in Python.
> 
>> 3. Implementing a thread-safe (or even non-thread safe) "call_once" method 
>> is non-trivial
> 
> Agree that this is certainly true. But presumably we should be making 
> lru_cache thread safe if it isn't.
> 
>> 4. It complements the lru_cache and cached_property methods currently 
>> present in functools.
> 
> It's unfortunate that cached_property doesn't work at module level (as was 
> pointed out on the other threads - thanks for linking those, BTW).
> 
> Cheers,
> Steve



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YE672NOPVB3AY2VKTL7JWJFLWAYMRNDK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Adding a "call_once" decorator to functools

[Tom Forbes]

Hey Raymond,
Thanks for your input here! A new method wouldn’t be worth adding purely for 
performance reasons then, but there is still an issue around semantics and 
locking.

Should we encourage/document `lru_cache` as the way to do `call_once`? If so, 
then I guess that’s suitable, but people have brought up that it might be hard 
to discover and that it doesn’t actually ensure the function is called once.

The reason I bring this up is that I’ve seen several ad-hoc `call_once` 
implementations recently, and creating one is surprisingly complex for someone 
who’s not that experienced with Python.

So I think there’s room to improve the discoverability of lru_cache as an 
“almost” `call_once` alternative, or room for a dedicated method that might 
re-use bits of the`lru_cache` implementation.

Tom

> On 28 Apr 2020, at 20:51, Raymond Hettinger  
> wrote:
> 
> 
>> t...@tomforb.es wrote:
>> 
>> I would like to suggest adding a simple “once” method to functools. As the 
>> name suggests, this would be a decorator that would call the decorated 
>> function, cache the result and return it with subsequent calls.
> 
> It seems like you would get just about everything you want with one line:
> 
>call_once = lru_cache(maxsize=None)
> 
> which would be used like this:
> 
>   @call_once
>   def welcome():
>   len('hello')
> 
>> Using lru_cache like this works but it’s not as efficient as it could be - 
>> in every case you’re adding lru_cache overhead despite not requiring it.
> 
> 
> You're likely imagining more overhead than there actually is.  Used as shown 
> above, the lru_cache() is astonishingly small and efficient.  Access time is 
> slightly cheaper than writing d[()]  where d={(): some_constant}. The 
> infinite_lru_cache_wrapper() just makes a single dict lookup and returns the 
> value.¹ The lru_cache_make_key() function just increments the empty args 
> tuple and returns it.²   And because it is a C object, calling it will be 
> faster than for a Python function that just returns a constant, "lambda: 
> some_constant()".  This is very, very fast.
> 
> 
> Raymond
> 
> 
> ¹ 
> https://github.com/python/cpython/blob/master/Modules/_functoolsmodule.c#L870
> ² 
> https://github.com/python/cpython/blob/master/Modules/_functoolsmodule.c#L809
> 
> 
> 
> 
> 
> 
>> 
>> Hello,
>> After a great discussion in python-ideas[1][2] it was suggested that I 
>> cross-post this proposal to python-dev to gather more comments from those 
>> who don't follow python-ideas.
>> 
>> The proposal is to add a "call_once" decorator to the functools module that, 
>> as the name suggests, calls a wrapped function once, caching the result and 
>> returning it with subsequent invocations. The rationale behind this proposal 
>> is that:
>> 1. Developers are using "lru_cache" to achieve this right now, which is less 
>> efficient than it could be
>> 2. Special casing "lru_cache" to account for zero arity methods isn't 
>> trivial and we shouldn't endorse lru_cache as a way of achieving "call_once" 
>> semantics 
>> 3. Implementing a thread-safe (or even non-thread safe) "call_once" method 
>> is non-trivial
>> 4. It complements the lru_cache and cached_property methods currently 
>> present in functools.
>> 
>> The specifics of the method would be:
>> 1. The wrapped method is guaranteed to only be called once when called for 
>> the first time by concurrent threads
>> 2. Only functions with no arguments can be wrapped, otherwise an exception 
>> is thrown
>> 3. There is a C implementation to keep speed parity with lru_cache
>> 
>> I've included a naive implementation below (that doesn't meet any of the 
>> specifics listed above) to illustrate the general idea of the proposal:
>> 
>> ```
>> def call_once(func):
>>   sentinel = object()  # in case the wrapped method returns None
>>   obj = sentinel
>>   @functools.wraps(func)
>>   def inner():
>>   nonlocal obj, sentinel
>>   if obj is sentinel:
>>   obj = func()
>>   return obj
>>   return inner
>> ```
>> 
>> I'd welcome any feedback on this proposal, and if the response is favourable 
>> I'd love to attempt to implement it.
>> 
>> 1. 
>> https://mail.python.org/archives/list/python-id...@python.org/thread/5OR3LJO7LOL6SC4OOGKFIVNNH4KADBPG/#5OR3LJO7LOL6SC4OOGKFIVNNH4KADBPG
>> 2. 
>> https://discuss.python.org/t/reduce-the-overhead-of-functools-lru-cache-for-functions-with-no-parameters/3956
>

[Python-Dev] Python 2.7 Won't Build

[Tom Browder]

I am trying to rebujild the 2.7 maintenance branch and get this error
on Ubuntu 10.04.1 LTS:

XXX lineno: 743, opcode: 0
Traceback (most recent call last):
 File "/usr/local/src/python-2.7-maint-svn/Lib/site.py", line 62, in 
   import os
 File "/usr/local/src/python-2.7-maint-svn/Lib/os.py", line 743, in 
   def urandom(n):
SystemError: unknown opcode

I installed it successfully once so I may be getting conflicts, but I
can't figure out why.  There were some similar bugs reported in
previous versions but I didn't see a clear solution.

I have done "make distclean" and "./configure".  I have unset my
PYTHONPATH and LD_LIBRARY_PATH, but python2.7 is my default python.

I guess my next step will be to manually remove the installed python
2.7 unless I hear some other suggestions.

And I will file a bug report soon unless that is inappropriate.

Thanks,

-Tom

Thomas M. Browder, Jr.
Niceville, Florida
USA
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.7 Won't Build

[Tom Browder]

On Thu, Sep 16, 2010 at 13:48, Brett Cannon  wrote:
> Go ahead and file the bug, but chances are that some other installed
> Python is executing the code and picking up the .pyc files which have
> bytecode new to Python 2.7.

But isn't that a problem with the build system?  It seems to me it
should be using all modules from within the build, thus there should
be no such error.

Regards,

-Tom
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.7 Won't Build

[Tom Browder]

On Thu, Sep 16, 2010 at 14:36, Barry Warsaw  wrote:
> On Sep 16, 2010, at 01:41 PM, Tom Browder wrote:
>
>>I am trying to rebujild the 2.7 maintenance branch and get this error
>>on Ubuntu 10.04.1 LTS:
>
> I just tried this on my vanilla 10.04.1 system.  I checked out release27-maint
> ran configure && make.  It built without problem.
>
>>XXX lineno: 743, opcode: 0
>>Traceback (most recent call last):
>> File "/usr/local/src/python-2.7-maint-svn/Lib/site.py", line 62, in
>>  import os
>> File "/usr/local/src/python-2.7-maint-svn/Lib/os.py", line 743, in
>>  def urandom(n):
>>SystemError: unknown opcode
...

> When you say "installed python 2.7" do you mean the one you installed to
> /usr/local from a from-source build, or something else (e.g. a Python 2.7
> package perhaps)?

It was the released source tarball for 2.7, and I get the same error
when I try it from that directory.

-Tom

Thomas M. Browder, Jr.
Niceville, Florida
USA
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.7 Won't Build

[Tom Browder]

I'm attempting to file a bug but keep getting:

An error has occurred

A problem was encountered processing your request. The tracker
maintainers have been notified of the problem.

-Tom

Thomas M. Browder, Jr.
Niceville, Florida
USA
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.7 Won't Build

[Tom Browder]

USAOn Thu, Sep 16, 2010 at 16:36, Victor Stinner
 wrote:
> Le jeudi 16 septembre 2010 23:10:22, Tom Browder a écrit :
>> I'm attempting to file a bug but keep getting:
>
> File another bug about this bug!

I did, and eventually discovered the problem: I tried to "nosy" Barry
as requested by adding his e-mail address, but that causes an error in
the tracker.  After I finally figured that out, I successfully entered
the original bug (and reported it on the "tracker bug").

-Tom

Thomas M. Browder, Jr.
Niceville, Florida
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.7 Won't Build

[Tom Browder]

Continuing on with investigating Python 2.7 build problems, one
problem I just discovered is a different installation on one 64-bit
system (Debian Lenny) versus another (Ubuntu 10.04.1 LTS).

I used gcc-4.5.1 on both systems, with no *PY* environment variables set.

On Debian I got two directories:

  /usr/local/lib64/python2.7
  /usr/local/lib/python2.7

and only the first had the "config" subdirectory.

On Ubuntu I got only

  /usr/local/lib/python2.7

I see that the configure file has some architecture choices
(--with-universal-archs=ARCH) but no explanation about the
consequences.

Can anyone explain the two different "default" installations I got?

It seems to me I should force the Ubuntu-style installation by  the
"--with-universal-archs=64-bit" configure option, and I will try that
on Debian while I await expert help.

Thanks.

-Tom
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.7 Won't Build

[Tom Browder]

On Mon, Sep 20, 2010 at 10:12, Ronald Oussoren  wrote:
>
>
> On 20 Sep, 2010,at 04:31 PM, Tom Browder  wrote:
>
> Continuing on with investigating Python 2.7 build problems, one
> problem I just discovered is a different installation on one 64-bit
> system (Debian Lenny) versus another (Ubuntu 10.04.1 LTS).
>
> I used gcc-4.5.1 on both systems, with no *PY* environment variables set.
>
> On Debian I got two directories:
>
> /usr/local/lib64/python2.7
> /usr/local/lib/python2.7
>
> and only the first had the "config" subdirectory.
>
> On Ubuntu I got only
>
> /usr/local/lib/python2.7

Okay, that leads to an install problem with bazaar on the Debian beast:

Could not find platform dependent libraries 
Consider setting $PYTHONHOME to [:]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site.py", line 549, in 
main()
  File "/usr/local/lib/python2.7/site.py", line 531, in main
known_paths = addusersitepackages(known_paths)
  File "/usr/local/lib/python2.7/site.py", line 264, in addusersitepackages
user_site = getusersitepackages()
  File "/usr/local/lib/python2.7/site.py", line 239, in getusersitepackages
user_base = getuserbase() # this will also set USER_BASE
  File "/usr/local/lib/python2.7/site.py", line 229, in getuserbase
USER_BASE = get_config_var('userbase')
  File "/usr/local/lib/python2.7/sysconfig.py", line 518, in get_config_var
return get_config_vars().get(name)
  File "/usr/local/lib/python2.7/sysconfig.py", line 421, in get_config_vars
_init_posix(_CONFIG_VARS)
  File "/usr/local/lib/python2.7/sysconfig.py", line 275, in _init_posix
raise IOError(msg)
IOError: invalid Python installation: unable to open
/usr/local/lib/python2.7/config/Makefile (No such file or directory)

I tried setting PYTHONHOME to various things but none worked.  The
problem is that the config directory is under /usr/local/lib64 and
none of the documents I have found describes how to handle that
situation.  All the environment variables seem to be for libraries,
modules, and executables, and there is no reference to any lib64.

The pkg-config program looks as if it could be used by such packages
as bazaar but it doesn't look like it's one of the usual installation
tricks.

I'll go to the bazaar mailing list while awaiting an answer here.

Thanks,

-Tom
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.7 Won't Build

[Tom Browder]

On Mon, Sep 20, 2010 at 13:05, Terry Reedy  wrote:
> On 9/20/2010 10:31 AM, Tom Browder wrote:
> ...
>>
>> I see that the configure file has some architecture choices
>> (--with-universal-archs=ARCH) but no explanation about the
>> consequences.
>>
>> Can anyone explain the two different "default" installations I got?
>
> At the moment, this appears to be question about "how do I use (build) a
> current release" rather than a bug report or "how do we develop (improve)
> future releases" question. As such, it seems a bit more appropriate for
> python-list. In any case, there are expert gcc/unix builders and users there
> who might be able to answer your question.

Thanks, I'll do that--docs are pretty sketchy about building from source.

-Tom

Thomas M. Browder, Jr.
Niceville, Florida
USA
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.7 Won't Build

[Tom Browder]

On Mon, Sep 20, 2010 at 13:05, Terry Reedy  wrote:
> On 9/20/2010 10:31 AM, Tom Browder wrote:
> ...
>>
>> I see that the configure file has some architecture choices
>> (--with-universal-archs=ARCH) but no explanation about the
>> consequences.
>>
>> Can anyone explain the two different "default" installations I got?
>
> At the moment, this appears to be question about "how do I use (build) a
> current release" rather than a bug report or "how do we develop (improve)
> future releases" question. As such, it seems a bit more appropriate for
> python-list. In any case, there are expert gcc/unix builders and users there
> who might be able to answer your question.

The more I look into it, though, it seems to be a legitimate developer
question.  The python installation scripts know how to install on a
mixed 32/64 bit system but missed fixing it to look like a more
universal system (if that's the proper word) for programs wanting to
read its installation configuration.

Respectfully,

-Tom
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.7 Won't Build

[Tom Browder]

On Mon, Sep 20, 2010 at 14:28, Antoine Pitrou  wrote:
> On Mon, 20 Sep 2010 09:31:45 -0500
> Tom Browder  wrote:
>>
>> Can anyone explain the two different "default" installations I got?
>>
>> It seems to me I should force the Ubuntu-style installation by  the
>> "--with-universal-archs=64-bit" configure option, and I will try that
>> on Debian while I await expert help.
>
> I think "universal arch" builds only apply under OS X where they
> produce fat binaries.
>
> Under 64-bit Linux, you can compile either a 64-bit executable (the
> default) or a 32-bit executable (by specifying e.g. CC="gcc -m32" to
> the configure script).
>
>
> However, the /usr/local/lib{,64}/python2.7 issue is a bit different,
> since those directories can host architecture independent files
> (such as .py and .pyc files). For example, on my Mandriva
> install, the 64-bit Python executable can import packages from
> both /usr/lib/python2.6/site-packages/
> and /usr/lib64/python2.6/site-packages/.

Thanks, Antoine.

And I think I just found the problem with the installation (it may be
worth a note):  I had  a special configuration file
(/usr/local/share/config.site) for autoconf to force the primary local
libraries to be installed in /usr/local/lib64 (left over from early
64-bit days of this old system).  Removing that file, removing the
/usr/local/lib64/python2.7 directory, and rebuilding and reinstalling
seems to have solved the immediate problem.

Moral of the story: watch out for old cruft in /usr/local when
installing a new distribution.

I apologize for the noise.

However, I'm still investigating the original build problem (gcc trunk
and corrupted byte compile), but it may be related--I'll see.

Regards,

-Tom

Thomas M. Browder, Jr.
Niceville, Florida
USA
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] fatal error callback issue

[Tom Whittock]

I'm writing in regards to http://bugs.python.org/issue1195571

I'm embedding Python in my application and ran into a need for this
functionality. I wrote a similar patch myself, and was about to submit
it. When I searched for similar issues I found that this one has been
available since 2005.

I'd really like to help get this patch approved and integrated into
the python sources. I'm sure many other python embedders have run into
this in the past - it's a legitimate concern for any process which is
not 100% hosted in the Python world.

Thanks.
Tom.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Patching builtin_id to allow for C proxy objects?

[Tom Whittock]

Hi all.

I'm writing a module to proxy C++ objects into Python for a large C++
application. There are hundreds of thousands of C++ objects, some of
which are temporary while others are very long lived.

Currently every time one of these objects is accessed from Python, a
new "myproxy" instance is created. So if I were to access the same
field of an object twice, I would receive two python objects proxying
the same underlying C++ object. This messes up "id" and "is", and is
causing me issues when, for example, I run into circular references
when enoding json or otherwise attempt to determine whether two proxy
objects refer to the same C++ object.

I can't see how to cache the "myproxy" objects instead of returning
new instances - due to the architecture of the C++ application,
there's no weak reference support at all, and the number of objects is
very large.

My current plan would be for me to override the id builtin to return
the underlying C++ object instance pointer instead of the PyObject
instance pointer in the case of the "myproxy" object type, probably
using a new type method slot tp_id or similar. The old behaviour would
be unchanged for all other types, naturally. I'd also need to alter
ceval.c to use builtin_id instead of the raw pointer for comparison
when using PyCmp_IS and PyCmp_IS_NOT. I can see that there could very
well be many other sites throughout the C source where the pointer was
directly compared, and would cause interesting issues for me down the
line. I'm just not sure what else to try.

I'd like to know if I'm being laughably naive or not before I went
about this plan, and whether it'd be worthwhile contributing the patch
back, considering the number of potentially overridden-id-unaware
areas throught the rest of the python source base.

Thanks.
Tom.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Patching builtin_id to allow for C proxy objects?

[Tom Whittock]

Hi Greg thanks for your quick reply.

> Perhaps you could use a WeakValueDictionary to keep a mapping
> from a C++ object address to its Python proxy.

Thank you, I'll implement this and see whether it works out. I'll
certainly be better off if it does. I was avoiding holding weak
references due to perhaps unfounded concerns about increasing the
lifetime and speed/memory impact of certain temporary objects which
are created at very high frequency. I'll test it and see before diving
into messing with id. But now I'm thinking about it again, I can see a
plan for not needing to affect that pathway at all.

Seems I fell into the trap of making things too complicated for myself.

> It also won't solve the problem of keeping the id of the
> proxy for longer than the proxy exists, but that's probably
> not something you should be relying on anyway. The id of
> *any* Python object is only valid while the object lives,
> and if it's still alive you have a real reference somewhere
> that you can use instead of the id for identity testing.

Thanks, yes. I'm actually kind of concerned about the usage of id in
the markers set which the json library uses for circular referencing
checks for exactly this reason. It seems to assume that the objects
lifetime lasts for the duration of the encoding operation. I have no
idea if that's actually the case in my situation, where data members
are property getters producing probably very short lived proxies
generated from C++. I guess I'll find out :)

Thanks again,
Tom.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Patching builtin_id to allow for C proxy objects?

[Tom Whittock]

Hi again.

Just to let you know that Greg's suggestion worked beautifully - I
guess my id idea was just me trying to make life hard for myself.

My concerns over the json modules usage of id seem unjustified, as
circular references are detected now that the weak reference
dictionary is in place.

Thanks for your help, and sorry for bothering dev with something which
was a regular python programming issue after all.

Tom.

On 27 June 2011 13:31, Tom Whittock  wrote:
> Hi Greg thanks for your quick reply.
>
>> Perhaps you could use a WeakValueDictionary to keep a mapping
>> from a C++ object address to its Python proxy.
>
> Thank you, I'll implement this and see whether it works out. I'll
> certainly be better off if it does. I was avoiding holding weak
> references due to perhaps unfounded concerns about increasing the
> lifetime and speed/memory impact of certain temporary objects which
> are created at very high frequency. I'll test it and see before diving
> into messing with id. But now I'm thinking about it again, I can see a
> plan for not needing to affect that pathway at all.
>
> Seems I fell into the trap of making things too complicated for myself.
>
>> It also won't solve the problem of keeping the id of the
>> proxy for longer than the proxy exists, but that's probably
>> not something you should be relying on anyway. The id of
>> *any* Python object is only valid while the object lives,
>> and if it's still alive you have a real reference somewhere
>> that you can use instead of the id for identity testing.
>
> Thanks, yes. I'm actually kind of concerned about the usage of id in
> the markers set which the json library uses for circular referencing
> checks for exactly this reason. It seems to assume that the objects
> lifetime lasts for the duration of the encoding operation. I have no
> idea if that's actually the case in my situation, where data members
> are property getters producing probably very short lived proxies
> generated from C++. I guess I'll find out :)
>
> Thanks again,
> Tom.
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] urllib unicode handling

[Tom Pinckney]

Hi,

While trying to use urllib in python 2.5.1 to HTTP GET content from  
various web sites, I've run into a problem with urllib.quote  
(and .quote_plus): they don't accept unicode strings.


I see that this is an issue that has been discussed before:

see this thread: 
http://mail.python.org/pipermail/python-dev/2006-July/067248.html
especially this post: 
http://mail.python.org/pipermail/python-dev/2006-July/067335.html

While I don't really want to re-open a can of worms, it seems that the  
current implementation of urllib.quote and urllib.quote_plus is  
painfully incompatible with how the web (circa 2008) actually works.  
While the standards may say there is no official way to represent  
unicode strings in URLs, in practice the world uses UTF-8 quite  
heavily. For example, I found the following URLs in Google pretty  
quickly by looking for percent encoded utf-8 encoded accented e's.


http://www.last.fm/music/Jos%C3%A9+Gonz%C3%A1lez
http://en.wikipedia.org/wiki/Joseph_Fouch%C3%A9

http://apps.facebook.com/ilike/artist/Jos%C3%A9+Gonz%C3%A1lez/track/Stay+In+The+Shade?apv=1

While in theory UTF-8 is not a standard, sites like Last.fm, Facebook  
and Wikipedia seem to have embraced it (as have pretty much all other  
major web sites). As with HTML, there is what the standard says and  
what the actual browsers have to accept in order to work in the real  
world.


urllib.urlencode already converts unicode characters to their UTF-8  
representation before percent encoding them. Why not urllib.quote and  
urllib.quote_plus?


Thanks for any thoughts on this,

Tom








___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] urllib unicode handling

[Tom Pinckney]

I may be missing something, but it seems that RFC 3987 (which is about  
IRIs) basically says:


1) IRIs are identical to URIs except they may have unicode characters  
in them

2) IRIs must be converted to URIs before being used in HTTP
3) The way to convert IRIs to URIs is to UTF-8 encode the unicode  
characters in the IRI and then percent encode the resulting octects  
that are unsafe to have in a URI
4) There's some ambiguity over what to do with the hostname portion of  
the URI if it hash one (IDN, replace non-ascii characters with dashes  
etc)


If this is indeed the case, it sounds perfectly legal (according to  
the RFC) and perfectly practical (as required by numerous popular  
websites) to have urllib.quote and urllib.quote_plus do an automatic  
UTF-8 encoding of unicode strings before percent encoding them.


It's not entirely clear to me if people should be calling urllib.quote  
on hostnames and expecting them to be encoded properly if the hostname  
contains non-ascii characters. Perhaps the docs should be clarified on  
this matter?


Similarly, urllib.unquote should precent-decode characters and then  
attempt to convert the resulting octects from utf-8 to unicode. If  
that conversion fails, we can assume the octects should be returned as  
a byte string rather than a unicode string.


On May 7, 2008, at 8:12 AM, Armin Ronacher wrote:


Hi,

Jeroen Ruigrok van der Werven  in-nomine.org> writes:


Would people object if such functionality got added to urllib?
I would ;-)  There are IRIs, just that nobody wrote a useful module  
for that.
There are algorithms in the RFC that can convert URIs to IRIs and  
the other way

round.  IMO that's the way to go.

Regards,
Armin

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/thomaspinckney3%40gmail.com


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] urllib unicode handling

[Tom Pinckney]

Maybe I didn't understand the RFC quite right, but it seemed like how  
to handle hostnames was left as a choice between IDNA encoding the  
hostname or replacing the non-ascii characters with dashes? I guess in  
practice IDNA is the right decision.


Another part I wasn't clear on is whether urllib.quote() understands  
it's working on URIs, arbitrary strings, URLs or what. It seems that  
from the documentation it looks like it's expecting to just work on  
the path component of URLs. If this is so, then it doesn't need to  
understand what to do if the IRI contains a hostname.


Seems like the other somewhat under-specified part of all of this is  
how urllib.unquote() should work. If after percent decoding it sees  
non-ascii octets, should it try to decode them as utf-8 and if that  
fails then leave them as is?


On May 7, 2008, at 11:55 AM, Robert Brewer wrote:


"Martin v. Löwis" wrote:

The proper way to implement this would be IRIs (RFC 3987),
in particular section 3.1. This is not as simple as just
encoding it as UTF-8, as you might have to apply IDNA to
the host part.

Code doing so just hasn't been contributed yet.


But if someone wanted to do so, it's pretty simple:


u'www.\u212bngstr\xf6m.com'.encode("idna")

'www.xn--ngstrm-hua5l.com'


Robert Brewer
[EMAIL PROTECTED]



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] urllib unicode handling

[Tom Pinckney]

I was assuming urllib.quote/unquote would only be called on text  
intended to be used in non-hostname portions of the URIs. I'm not sure  
if this is the actual intent of urllib.quote and perhaps the  
documentation should be updated to specify what precisely it does and  
then peopel can decide what parts of URIs it is appropriate to quote/ 
unquote. I don't believe quote/unquote does anything sensical with  
hostnames today that contain non-printable ascii, so this is no loss  
of existing functionality.


Re your suggestion that IRIs should be a separate module: I guess my  
thought is that urllib out of the box should just work with the way  
websites on the web today actually work. Thus, we should make urllib  
do the utf-8 encode / decode rather than make users switch to a  
different module for certain URLs and another library for other URLs.


Re the specific issue of how urllib.unquote should work: Perhaps there  
could be an optional second argument that specified a content encoding  
to use when decoding escaped characters? I would propose that this  
parameter have a default value of utf-8 since that is what most  
websites seem to do, but if the author knew that the website they were  
using encoded URLs in iso-8559 then they could unquote using that  
scheme.


On May 7, 2008, at 3:10 PM, Martin v. Löwis wrote:

If this is indeed the case, it sounds perfectly legal (according to  
the
RFC) and perfectly practical (as required by numerous popular  
websites)

to have urllib.quote and urllib.quote_plus do an automatic UTF-8
encoding of unicode strings before percent encoding them.


It's probably legal, but I don't understand why you think it's
practical. The DNS lookup then will certainly fail, no?

Regards,
Martin


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Copying cgi.parse_qs() to the urllib.parse module

[Tom Pinckney]

Is there any thought to extending escape to escape / unescape to by  
default handle characters other than <, >, and &? At a minimum it  
should handle arbitrary &xxx; values. Ideally, it would also handle  
common other symbolic names besides < > etc.


HTML from common web sites such as nytimes.com frequently has a  
variety of characters escaped.


Consider the page at 
http://travel.nytimes.com/travel/guides/europe/france/provence-and-the-french-riviera/overview.html

It lists its content type as:
content="text/html; charset=UTF-8"
And contains text like:
There’s the Côte d’
Ideally, we would decode ’ into ’ and ô into ô.
Unfortunately, #146 is really an error -- it's not a utf-8 encoded  
unicode character but really a MS codepage 1252 character for  
apostrophe (apparently may HTML editing systems intermingle unicode  
and codepage 1252 content for apostrophes and a few other common  
characters).
I'm happy to contribute some additional code for these other cases if  
people agree it's useful.




On May 12, 2008, at 10:36 AM, Tony Nelson wrote:


At 11:56 PM -0400 5/10/08, Fred Drake wrote:

On May 10, 2008, at 11:49 PM, Guido van Rossum wrote:

Works for me. The other thing I always use from cgi is escape() --
will that be available somewhere else too?



xml.sax.saxutils.escape() would be an appropriate replacement, though
the location is a little funky.


At least it's right next to the valuable quoteattr().
--

TonyN.:'   
 '  
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/thomaspinckney3%40gmail.com


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Addition of "pyprocessing" module to standard lib.

[Tom Pinckney]

Why not use MPI? It's cross platform, cross language and very widely  
supported already. And there're Python bindings already.


On May 13, 2008, at 8:52 PM, Jesse Noller wrote:


I am looking for any questions, concerns or benchmarks python-dev has
regarding the possible inclusion of the pyprocessing module to the
standard library - preferably in the 2.6 timeline.  In March, I began
working on the PEP for the inclusion of the pyprocessing (processing)
module into the python standard library[1]. The original email to the
stdlib-sig can be found here, it includes a basic overview of the
module:

http://mail.python.org/pipermail/stdlib-sig/2008-March/000129.html

The processing module mirrors/mimics the API of the threading module -
and with simple import/subclassing changes depending on the code,
allows you to leverage multi core machines via an underlying forking
mechanism. The module also supports the sharing of data across groups
of networked machines - a feature obviously not part of the core
threading module, but useful in a distributed environment.

As I am trying to finish up the PEP, I want to see if I can address
any questions or include any other useful data (including benchmarks)
in the PEP prior to publishing it. I am also intending to include
basic benchmarks for both the processing module against the threading
module as a comparison.

-Jesse

[1] Processing page: http://pyprocessing.berlios.de/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/thomaspinckney3%40gmail.com


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Addition of "pyprocessing" module to standard lib.

[Tom Pinckney]

On May 14, 2008, at 12:32 PM, Andrew McNabb wrote:




Think of the processing module as an alternative to the threading
module, not as an alternative to MPI.  In Python, multi-threading  
can be

extremely slow.  The processing module gives you a way to convert from
using multiple threads to using multiple processes.

If it made people feel better, maybe it should be called threading2
instead of multiprocessing.  The word "processing" seems to make  
people

think of parallel processing and clusters, which is missing the point.

Anyway, I would love to see the processing module included in the
standard library.



Is the goal of the pyprocessing module to be exactly drop in  
compatible with threading, Queue and friends? I guess the idea would  
be that if my problem is compute bound I'd use pyprocessing and if it  
was I/O bound I might just use the existing threading library?


Can I write a program using only threading and Queue interfaces for  
inter-thread communication and just change my import statements and  
have my program work? Currently, it looks like the pyprocessing.Queue  
interface is slightly different than than Queue.Queue for example (no  
task_done() etc).


Perhaps a stdlib version of pyprocessing could be simplified down to  
not try to be a cross-machine computing environment and just be a same- 
machine threading replacement? This would make the maintenance easier  
and reduce confusion with users about how they should do cross-machine  
multiprocessing.


By the way, a thread-safe simple dict in the style of Queue would be  
extremely helpful in writing multi-threaded programs, whether using  
the threading or pyprocessing modules.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Python parallel benchmark

[Tom Pinckney]

All the discussion recently about pyprocessing got me interested in  
actually benchmarking Python's multiprocessing performance to see if  
reality matched my expectations around what would scale up and what  
would not. I knew Python threads wouldn't be good for compute bound  
problems, but I was curious to see how well they worked for i/o bound  
problems. The short answer is that for i/o bound problems, python  
threads worked just as well as using multiple operating system  
processes.


I wrote two simple benchmarks, one compute bound and the other i/o  
bound. The compute bound one did a parallel matrix multiply and the i/ 
o bound one read random records from a remote MySQL database. I ran  
each benchmark via python's thread module and via MPI (using mpi4py  
and openmpi and Send()/Recv() for communication). Each test was run  
multiple times and the numbers were consistent between test runs. I  
ran the tests on a dual-core Macbook Pro running OS X 10.5 and the  
included python 2.5.1.


1) Python threads

a) compute bound:

1 thread - 16 seconds
2 threads - 21 seconds

b) i/o bound:

1 thread -- 13 seconds
4 threads -- 10 seconds
8 threads -- 5 seconds
12 threads - 4 seconds

2) MPI

a) compute bound:

1 thread - 17 seconds
2 threads -- 11 seconds

b) i/o bound

1 thread -- 13 seconds
4 threads -- 10 seconds
8 threads -- 6 seconds
12 threads -- 4 seconds
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python parallel benchmark

[Tom Pinckney]

I switched to using numpy for the matrix multiply and while the  
overall time to do the matrix multiply is much faster, there is still  
no speed up from using more than one python thread. If I look at top  
while running 2 or more threads, both cores are being used 100% and  
there is no idle time on the system.


I did a quick google search and didn't find anything conclusive about  
numpy releasing the GIL. The most conclusive and recent reference I  
found was


http://mail.python.org/pipermail/python-list/2007-October/463148.html

I found some other references where people were expressing concern  
over numpy releasing the GIL due to the fact that other C extensions  
could call numpy and unexpectedly have the GIL released on them (or  
something like that).


On May 15, 2008, at 6:43 PM, Nick Coghlan wrote:


Tom Pinckney wrote:
All the discussion recently about pyprocessing got me interested in  
actually benchmarking Python's multiprocessing performance to see  
if reality matched my expectations around what would scale up and  
what would not. I knew Python threads wouldn't be good for compute  
bound problems, but I was curious to see how well they worked for i/ 
o bound problems. The short answer is that for i/o bound problems,  
python threads worked just as well as using multiple operating  
system processes.


Interesting - given that your example compute bound problem happened  
to be a matrix multiply, I'd be curious what the results are when  
using python threads with numpy to do the same thing (my  
understanding is that numpy will usually release the GIL while doing  
serious number-crunching)


Cheers,
Nick.

--
Nick Coghlan   |   [EMAIL PROTECTED]   |   Brisbane, Australia
---
   http://www.boredomandlaziness.org


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python parallel benchmark

[Tom Pinckney]

Interestingly, I think there's something magic going on with  
numpy.dot() on my mac.


If I just run a program without threading--that is just a numpy matrix  
multiply such as:


import numpy
a = numpy.empty((4000,4000))
b = numpy.empty((4000,4000))
c = numpy.dot(a,b)

then I see both cores fully maxed out on my mac. On a dual-core linux  
machine I see only one core maxed out by this program and it runs  
VASTLY slower on the linux box.


It turns out that numpy on Mac's uses Apple's Accelerate.framekwork  
BLAS and LAPACK which in turn is multi-threaded as of OS X 10.4.8.


On May 15, 2008, at 10:55 PM, Greg Ewing wrote:


Tom Pinckney wrote:
If I look at top while running 2 or more threads, both cores are  
being used 100% and there is no idle time on the system.


If you run it with just one thread, does it use up only
one core's worth of CPU?

If so, this suggests that the GIL is being released. If
it wasn't, two threads would still only use one core's worth.

Also, if you have only two cores, using more than two
threads isn't going to gain you anything whatever happens.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/thomaspinckney3%40gmail.com


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python parallel benchmark

[Tom Pinckney]

Here's one example, albeit from a few years ago

http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/1625465

But, I am a numpy novice and so no idea what it actually does in its  
current form.


On May 16, 2008, at 4:17 AM, Hrvoje Nik?i? wrote:


On Thu, 2008-05-15 at 21:02 -0400, Tom Pinckney wrote:

I found some other references where people were expressing concern
over numpy releasing the GIL due to the fact that other C extensions
could call numpy and unexpectedly have the GIL released on them (or
something like that).


Could you please post links to those?  I'm asking because AFAIK that
concern doesn't really stand.  Any (correct) code that releases the  
GIL

is responsible for reacquiring it before calling *any* Python code, in
fact before doing anything that might touch a Python object or its
refcount.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/thomaspinckney3%40gmail.com


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Exception tracebacks

[Tom Mullins]

I do not know where cleanup_traceback (in run.py) is called, or really
anything about the inner workings of python, but there is a problem with
sys.tracebacklimit. If it is set to one or zero, "** IDLE Internal
Exception" is printed in the traceback for any exception, not internal ones.
I think tracebacklimit is applied to the traceback before cleanup_traceback
is called, but should be applied after. I imagine the solution is not that
simple, and you may already be aware of the problem, but thanks anyway.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Csv] skipfinalspace

[Tom Brown]

(Continuing thread started at
http://mail.python.org/pipermail/csv/2008-October/000688.html)
On Sun, Oct 19, 2008 at 16:46, Andrew McNamara
<[EMAIL PROTECTED]>wrote:

> >I downloaded the 2.6 source tar ball, but is it too late for new features
> to
> >get into versions <3?
>
> Yep.
>
> >How would you feel about adding the following tests to
> Lib/test/test_csv.py
> >and getting them to pass?
> >
> >Also http://www.python.org/doc/2.5.2/lib/csv-fmt-params.html says
> >"*skipinitialspace *When True, whitespace immediately following the
> >delimiter is ignored."
> >but my tests show whitespace at the start of any field is ignored,
> including
> >the first field.
>
> I suspect (but I haven't checked) that it means "after the delimiter and
> before any quoted field (or some variation on that).


I agree that whitespace after the delimiter and before any quoted field is
skipped. Also whitespace after the start of the line and before any quoted
field is skipped.


>
>
> All of the "dialect" parameters are there to allow parsing of a specific
> common form of CSV file. Because there is no formal definition of the
> format, the module simply aims to parse (and produce the same result)
> as common applications such as Excel and Access. Changing the behaviour
> in any non-backwards compatible way is sure to get screams of anguish
> from many users. Even when the behaviour appears to be a bug, you can
> be sure people are counting on it working like that.


skipinitialspace defaults to false and by the same logic skipfinalspace
should default to false to preserve compatibility with the csv module in
2.6. On the other hand, the switch to version 3 is as good a time as any to
break backwards compatibility to adopt something that works better for new
users.

Based on my experience parsing several hundred csv generated by many
different people I think it would be nice to at least have a dialect that is
excel + skipinitialspace=True + skipfinalspace=True.


>
> BTW, this discussion probably should move to python-dev.
>
> --
> Andrew McNamara, Senior Developer, Object Craft
> http://www.object-craft.com.au/
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Csv] skipfinalspace

[Tom Brown]

On Mon, Oct 20, 2008 at 00:48, John Machin <[EMAIL PROTECTED]> wrote:

> Tom Brown wrote:
>
>> (Continuing thread started at
>> http://mail.python.org/pipermail/csv/2008-October/000688.html)
>>
>> On Sun, Oct 19, 2008 at 16:46, Andrew McNamara <
>> [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>> wrote:
>>
>> >I downloaded the 2.6 source tar ball, but is it too late for new
>>features to
>> >get into versions <3?
>>
>>Yep.
>>
>> >How would you feel about adding the following tests to
>>Lib/test/test_csv.py
>> >and getting them to pass?
>> >
>> >Also http://www.python.org/doc/2.5.2/lib/csv-fmt-params.html says
>> >"*skipinitialspace *When True, whitespace immediately following the
>> >delimiter is ignored."
>> >but my tests show whitespace at the start of any field is ignored,
>>including
>> >the first field.
>>
>>I suspect (but I haven't checked) that it means "after the delimiter
>> and
>>before any quoted field (or some variation on that).
>>
>> I agree that whitespace after the delimiter and before any quoted field is
>> skipped. Also whitespace after the start of the line and before any quoted
>> field is skipped.
>>
>
> All of the "dialect" parameters are there to allow parsing of a
>> specific
>>common form of CSV file. Because there is no formal definition of the
>>format, the module simply aims to parse (and produce the same result)
>>as common applications such as Excel and Access. Changing the behaviour
>>in any non-backwards compatible way is sure to get screams of anguish
>>from many users. Even when the behaviour appears to be a bug, you can
>>be sure people are counting on it working like that.
>>
>>
>> skipinitialspace defaults to false and by the same logic skipfinalspace
>> should default to false to preserve compatibility with the csv module in
>> 2.6. On the other hand, the switch to version 3 is as good a time as any to
>> break backwards compatibility to adopt something that works better for new
>> users.
>>
>
> Read Andrew's lips: They don't want "better", they want "the same as MS".

okay.


>
>
>  Based on my experience parsing several hundred csv generated by many
>> different people I think it would be nice to at least have a dialect that is
>> excel + skipinitialspace=True + skipfinalspace=True.
>>
>
> Based on my experience extracting data from innumerable csv files (and
> infinite varieties thereof),

Wow, that is a _lot_ of files :-P

spreadsheet files, and database tables, in 99.99% of cases one should
> automatically apply the following transformations to each text field:
>   * strip leading whitespace
>   * strip trailing whitespace
>   * replace embedded runs of whitespace by a single space
> and one needs to ensure that the definition of whitespace includes the
> no-break space (NBSP) character.
>
> As this "space normalisation" is needed for all input sources, the csv
> module is IMHO the wrong place to put it. A string method would be a better
> idea.


I agree that strip() and something like re.sub(r"\s+", " " are handy. If
99.99% percent of csv readers should be applying these fixes to every field
perhaps there should be easy-to-enable option to apply it. Why force almost
everyone to discover they need the transformations and put a line of code
around csv reader?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Should we move to replace re with regex?

[Tom Christiansen]

"M.-A. Lemburg"  wrote
   on Sat, 27 Aug 2011 01:00:31 +0200: 

> The good part is that it's based on the re code, the FUD comes
> from the fact that the new lib is 380kB larger than the old one
> and that's not even counting the generated 500kB of lookup
> tables.

Well, you have to put the property tables somewhere, somehow.
There are various schemes for demand loading them as needed,
but I don't know whether those are used.

--tom
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 340: Breaking out.

[Tom Rothamel]

I have a question/suggestion about PEP 340.

As I read the PEP right now, the code:


while True:

block synchronized(v1):
 if v1.field:
  break

time.sleep(1)


Will never break out of the enclosing while loop. This is because the
break breaks the while loop that the block statement is translated
into, instead of breaking the outer True statement. 

Am I understanding this right, or am I misunderstanding this?

If I am understanding this right, I would suggest allowing some way of
having the iterator call continue or break in the enclosing
context. (Perhaps by enclosing the entire translation of block in a
try-except construct, which catches Stop and Continue exceptions
raised by the generator and re-raises them in the outer context.)

I hope this helps.

-- 
Tom Rothamel --- http://www.rothamel.us/~tom/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (libffi) Re: Copyright issue

[Tom Tromey]

>>>>> "Giovanni" == Giovanni Bajo <[EMAIL PROTECTED]> writes:

Giovanni> This would be a new interpretation of the license. The whole
Giovanni> autotools chain is GPL and it is used on way too many
Giovanni> programs which are not GPL. They're so many I won't even
Giovanni> mention one. Anyway, IANAL, so if you're really concerned
Giovanni> you can mail the FSF and ask clarifications.

No, Martin is correct about this.  The output of autoconf is
explicitly *not* under the GPL, by design.  This is also true for the
m4 macros from automake -- again, an explicit decision.

The problem is, some other projects have not been so careful.

Tom
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (libffi) Re: Copyright issue

[Tom Tromey]

>>>>> "Martin" == Martin v Löwis <[EMAIL PROTECTED]> writes:

Martin> Instead, it means we need a build process for libffi which is
Martin> independent of autoconf (or convince the authors of aclocal.m4 to
Martin> relicense it, but that is likely futile).

Martin> As a matter of fact, Python itself uses autoconf, but without
Martin> aclocal.m4. autoconf generates configure for us, but configure is
Martin> not GPL-licensed, even though it is generated through autoconf:

Martin> # Copyright (C) 2003 Free Software Foundation, Inc.
Martin> # This configure script is free software; the Free Software Foundation
Martin> # gives unlimited permission to copy, distribute and modify it.

libffi's aclocal.m4 is created by the aclocal tool, which stitches it
together from different .m4 files it finds.

For m4 files that are part of the automake distribution, the intent
was explicitly to have the same more liberal permissions that apply to
'configure'.  If you can't find a statement saying this somewhere,
then I believe that to be a bug (I see one at the top of aclocal.m4
FWIW -- this applies to all the automake-shipped m4 files).  I know I
explicitly brought this up with RMS at some point in the distant past
and got the needed permissions to make this so.

However, libffi probably also uses macros from other places -- at
least GCC and libtool.  I don't know the legal status of these.

The GCC macros are probably not really necessary for your situation.
The libffi configury in the GCC tree is set up to build libffi as a
target library.  Most likely, you don't need this.

libffi/acinclude.m4 needs some license clarification.
It isn't clear who owns this code :-(

I think a real fix is probably not out of reach, should you wish to
pursue it.

Tom
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] optparse and unicode

[Tom Cato Amundsen]

optparse


Using unicode strings with non-ascii chars.[1]

I'm working around this by subclassing OptionParser. Below is a
workaround I use in GNU Solfege. Should something like this be
included in python 2.5?

(Please CC me any answer.)

Tom Cato

--- orig/src/mainwin.py
+++ mod/src/mainwin.py
@@ -30,7 +30,13 @@
 
 import optparse
 
-opt_parser = optparse.OptionParser()
+class MyOptionParser(optparse.OptionParser):
+def print_help(self, file=None):
+if file is None:
+file = sys.stdout
+file.write(self.format_help().encode(file.encoding, 'replace'))
+
+opt_parser = MyOptionParser()
 opt_parser.add_option('-v', '--version', action='store_true', dest='version')
 opt_parser.add_option('-w', '--warranty', action='store_true', dest='warranty',
 help=_('Show warranty and copyright.'))

[1]

Traceback (most recent call last):
  File "./solfege.py", line 43, in ?
import src.mainwin
  File "/home/tom/src/solfege-mcnv/src/mainwin.py", line 70, in ?
options, args = opt_parser.parse_args()
  File "/usr/lib/python2.3/optparse.py", line 1129, in parse_args
stop = self._process_args(largs, rargs, values)
  File "/usr/lib/python2.3/optparse.py", line 1169, in _process_args
self._process_long_opt(rargs, values)
  File "/usr/lib/python2.3/optparse.py", line 1244, in _process_long_opt
option.process(opt, value, values, self)
  File "/usr/lib/python2.3/optparse.py", line 611, in process
return self.take_action(
  File "/usr/lib/python2.3/optparse.py", line 632, in take_action
parser.print_help()
  File "/usr/lib/python2.3/optparse.py", line 1370, in print_help
file.write(self.format_help())
UnicodeEncodeError: 'ascii' codec can't encode characters in position 200-202: 
ordinal not in range(128)
-- 
Tom Cato Amundsen <[EMAIL PROTECTED]> http://www.solfege.org/
GNU Solfege - free ear traininghttp://www.gnu.org/software/solfege/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com