[issue45690] Argparse exclusive group inside required exclusive group displays incorrectly
New submission from andrew cooke :
The code below, when invoked with -h, prints:
(.env) [andrew@localhost py]$ python -m tests_sa.argparse_bug -h
usage: argparse_bug.py [-h] (-a A | [-b B | -c C)]
options:
-h, --help show this help message and exit
-a A
-b B
-c C
where the final two characters in the usage line are swapped. It should be
usage: argparse_bug.py [-h] (-a A | [-b B | -c C])
or maybe even
usage: argparse_bug.py [-h] (-a A | (-b B | -c C))
from argparse import ArgumentParser
def main():
parser = ArgumentParser()
outer = parser.add_mutually_exclusive_group(required=True)
outer.add_argument('-a')
inner = outer.add_mutually_exclusive_group()
inner.add_argument('-b')
inner.add_argument('-c')
parser.parse_args()
if __name__ == '__main__':
main()
--
components: Library (Lib)
messages: 405509
nosy: acooke
priority: normal
severity: normal
status: open
title: Argparse exclusive group inside required exclusive group displays
incorrectly
versions: Python 3.10
___
Python tracker
<https://bugs.python.org/issue45690>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Basic misunderstanding on object creation
Hi,
The following code worked on Python 3.2, but no longer works in 3.4. Did
something change, or have I always been doing something dumb?
(I realise the code is pointless as is - it's the simplest example I can give
of a problem I am seeing with more complex code).
class Foo:
... def __new__(cls, *args, **kargs):
... print('new', args, kargs)
... super().__new__(cls, *args, **kargs)
...
class Bar(Foo):
... def __init__(self, a):
... print('init', a)
...
Bar(1)
new (1,) {}
Traceback (most recent call last):
File stdin, line 1, in module
File stdin, line 4, in __new__
TypeError: object() takes no parameters
What I was expecting to happen (and what happens in 3.2) is that the
object.__new__ method passes the argument to the __init__ of the subclass.
Any help appreciated.
Thanks,
Andrew
--
https://mail.python.org/mailman/listinfo/python-list
Re: Basic misunderstanding on object creation
But then nothing will be passed to __init__ on the subclass.
Andrew
class Foo:
... def __new__(cls, *args, **kargs):
... print('new', args, kargs)
... super().__new__(cls)
...
class Bar(Foo):
... def __init__(self, a):
... print('init', a)
...
Bar(1)
new (1,) {}
no init is printed.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Basic misunderstanding on object creation
On Wednesday, 13 May 2015 11:36:12 UTC-3, Thomas Rachel wrote:
Am 13.05.2015 um 15:25 schrieb andrew cooke:
class Foo:
... def __new__(cls, *args, **kargs):
... print('new', args, kargs)
... super().__new__(cls, *args, **kargs)
new (1,) {}
Traceback (most recent call last):
File stdin, line 1, in module
File stdin, line 4, in __new__
TypeError: object() takes no parameters
object's __new__() dosn't take any parameters. So call it without arguments:
class Foo:
def __new__(cls, *args, **kargs):
print('new', args, kargs)
super().__new__(cls)
(at least if we know that we inherit from object. Might be that this one
doesn't work very good with multiple inheritance...)
Thomas
But then nothing will be passed to __init__ on the subclass.
Andrew
--
https://mail.python.org/mailman/listinfo/python-list
Re: Basic misunderstanding on object creation
On Wednesday, 13 May 2015 13:37:23 UTC-3, Terry Reedy wrote:
On 5/13/2015 9:25 AM, andrew cooke wrote:
The following code worked on Python 3.2, but no longer works in 3.4.
Bugfixes break code that depends on buggy behavior. See
https://bugs.python.org/issue1683368
Your code also fails in 2.7.9 if you inherit Foo from object.
The exact error messages changed for 3.4 in
https://bugs.python.org/issue7963
Did something change,
Obviously yes.
thanks, but why does someone on this group always have to be a dick and make
some smart-assed comment like this?
or have I always been doing something dumb?
You were depending on behavior of object that Guido decided was buggy.
I found the tracker issue by looking for 'object' in the Core and
Builtins sections of the changelog one can access from What's New, first
paragraph (using Highlight All in Firefox).
class Foo:
... def __new__(cls, *args, **kargs):
... print('new', args, kargs)
... super().__new__(cls, *args, **kargs)
...
class Bar(Foo):
... def __init__(self, a):
... print('init', a)
...
Bar(1)
new (1,) {}
Traceback (most recent call last):
File stdin, line 1, in module
File stdin, line 4, in __new__
TypeError: object() takes no parameters
What I was expecting to happen (and what happens in 3.2) is that the
object.__new__ method passes the argument to the __init__ of the subclass.
--
Terry Jan Reedy
--
https://mail.python.org/mailman/listinfo/python-list
Re: Basic misunderstanding on object creation
On Wednesday, 13 May 2015 11:56:21 UTC-3, Ian wrote:
On Wed, May 13, 2015 at 8:45 AM, andrew cooke and...@acooke.org wrote:
class Foo:
... def __new__(cls, *args, **kargs):
... print('new', args, kargs)
... super().__new__(cls)
...
class Bar(Foo):
... def __init__(self, a):
... print('init', a)
...
Bar(1)
new (1,) {}
no init is printed.
You're not returning anything from Foo.__new__, so the result of the
constructor is None. None.__init__ does nothing.
ah, you're right, thanks. that was a typo.
more generally, it seems that the error is:
(1) __new__ is called and then __init__ from some other, external code
(2) in 3.2 anything passed to object's __new__ was silently discarded.
the following code works in 3.2 and 3.4:
class Foo:
def __new__(cls, *args, **kargs):
print(new, args, kargs)
return super().__new__(cls)
class Bar(Foo):
def __init__(self, *args, **kargs):
print(init, args, kargs)
Bar(1)
(while my original code didn't).
thanks everyone,
andrew
--
https://mail.python.org/mailman/listinfo/python-list
[issue18422] is_package missing so can't use -m
New submission from andrew cooke: Using python 3.3, if I try to run __main__ I see this error: Traceback (most recent call last): File /usr/local/lib/python3.3/runpy.py, line 140, in _run_module_as_main mod_name, loader, code, fname = _get_module_details(mod_name) File /usr/local/lib/python3.3/runpy.py, line 105, in _get_module_details if loader.is_package(mod_name): AttributeError: 'NamespaceLoader' object has no attribute 'is_package' My directory structure is: tree . . ├── README.md ├── src │ └── simplessl │ ├── ca.py │ ├── __main__.py │ ├── req.py │ └── utils.py └── ssl.iml i assume this is related to http://bugs.python.org/issue18058 but don't understand why this is not a bugfix that can be back-ported. this appears to be a bug to me... -- components: Installation messages: 192799 nosy: acooke priority: normal severity: normal status: open title: is_package missing so can't use -m versions: Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18422 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18422] is_package missing so can't use -m
andrew cooke added the comment: can't see how to edit posts, so adding as a comment, this is what triggers the bug: PYTHONPATH=src python -m simplessl -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18422 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18423] Document limitations on -m
New submission from andrew cooke: Apparently the limited support for -m is standard behaviour - see http://bugs.python.org/issue18422 - but it's not documented at http://docs.python.org/3/using/cmdline.html#cmdoption-m That should say, somewhere, that it only applies to leaf modules and packages. -- assignee: docs@python components: Documentation messages: 192804 nosy: acooke, docs@python priority: normal severity: normal status: open title: Document limitations on -m ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18423 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18422] is_package missing so can't use -m
andrew cooke added the comment: in case anyone else ends up here through google... the problem described here is not related to the linked issue. it was just a missing `__init__.py` in the module (plus sucky error messages). the following works fine: . ├── README.md ├── src │ └── simplessl │ ├── ca.py │ ├── __init__.py │ ├── __main__.py │ ├── req.py │ └── utils.py └── ssl.iml -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18422 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17403] Robotparser fails to parse some robots.txt
andrew cooke added the comment: what is rietveld? and why is this marked as easy? it seems like it involves issues that aren't described well in the spec - it requires some kind of canonical way to describe urls with (and without) parameters to solve completely. -- nosy: +acooke ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17403 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17403] Robotparser fails to parse some robots.txt
andrew cooke added the comment: thanks (only subscribed to this now, so no previous email). my guess is that google are assuming a dumb regexp so http://example.com/foo? in a rule does not match http://example.com/foo and also i realised that http://google.com/robots.txt doesn't contain any url with multiple parameters. so perhaps i was wrong about needing a canonical representation (ie parameter ordering). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17403 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Confused about logger config from within Python (3)
When I use a config file things seem to work (in other projects), but for my current code I hoped to configure logging from Python. I distilled my problem down to the following test, which does not print anything. Please can someone explain why? I was expecting the module's logger to delegate to root, which has the DEBUG level set. from logging import DEBUG, root, getLogger from unittest import TestCase class LoggingTest(TestCase): def test_direct(self): root.setLevel(DEBUG) getLogger(__name__).debug(hello world) Thanks, Andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Confused about logger config from within Python (3)
similarly, if i run the following, i see only done:
from logging import DEBUG, root, getLogger
if __name__ == '__main__':
root.setLevel(DEBUG)
getLogger(__name__).debug(hello world)
print('done')
--
http://mail.python.org/mailman/listinfo/python-list
Re: Confused about logger config from within Python (3)
On Friday, 28 December 2012 21:56:46 UTC-3, Peter Otten wrote: Other revolutionary ideas: read the docs http://docs.python.org/dev/howto/logging.html#logging-basic-tutorial ;) how do you think i knew about the root handler without reading the damn docs you condescending asshole? anyway, thanks for the help. andrew -- http://mail.python.org/mailman/listinfo/python-list
[issue12897] Support for iterators in multiprocessing map
andrew cooke and...@acooke.org added the comment: hi - i'm the original author (may be using a different account). as far as i remember, i raised this because it seemed relevant given the link i gave. if you've looked at the issue and think your approach would work, or that this should be closed, or whatever, that's fine by me. i'm not going to check myself - i can't remember anything about this now (nearly a year later) and it's not my place to worry about your code (no offence - just trying to clarify that i have no skin in this game). -- status: pending - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12897 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12029] Catching virtual subclasses in except clauses
andrew cooke and...@acooke.org added the comment: perhaps it could just work in a simple, consistent way? in my original report i wondered whether there was a significant performance hit. but so far the objections against fixing this seem to be (1) a lawyer could be convinced the current behaviour is consistent with the docs (2) python 3 should remain compatible with python 2 (3) abcmeta is the sucksorz. those don't seem like great arguments against making it just work right, to me. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12029 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12029] Catching virtual subclasses in except clauses
Changes by andrew cooke and...@acooke.org: -- nosy: -acooke ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12029 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
ordering with duck typing in 3.1
hi, please, what am i doing wrong here? the docs say http://docs.python.org/release/3.1.3/library/stdtypes.html#comparisons in general, __lt__() and __eq__() are sufficient, if you want the conventional meanings of the comparison operators but i am seeing assert 2 three E TypeError: unorderable types: int() IntVar() with this test: class IntVar(object): def __init__(self, value=None): if value is not None: value = int(value) self.value = value def setter(self): def wrapper(stream_in, thunk): self.value = thunk() return self.value return wrapper def __int__(self): return self.value def __lt__(self, other): return self.value other def __eq__(self, other): return self.value == other def __hash__(self): return hash(self.value) class DynamicTest(TestCase): def test_lt(self): three = IntVar(3) assert three 4 assert 2 three assert 3 == three so what am i missing? thanks, andrew -- http://mail.python.org/mailman/listinfo/python-list
[issue12897] Support for iterators in multiprocessing map
New submission from andrew cooke and...@acooke.org: http://stackoverflow.com/questions/7306522/combining-itertools-and-multiprocessing/7307078 suggests (and the idea itself seems reasonable) that it would sometimes be useful for multiprocessing to operate correctly (ie lazily) with lazy input (iterables). for example, if the input is large, or perhaps generated by some other process on demand. obviously this complicates matters, given the asynchronous nature of a worker pool, and would mean re-allocating the results list as required. but in principle i suspect it would be possible and might be a useful extension. -- components: Library (Lib) messages: 143511 nosy: acooke priority: normal severity: normal status: open title: Support for iterators in multiprocessing map type: feature request versions: Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12897 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: ABC-registered Exceptions are not caught as subclasses
http://bugs.python.org/issue12029 -- http://mail.python.org/mailman/listinfo/python-list
[issue12029] ABC registration of Exceptions
New submission from andrew cooke and...@acooke.org:
Hi,
In general, registering a class with an ABC is equivalent to making it a
subclass (isinstance and issubclass are patched through ABCMeta). However,
this does not work for exceptions (see example below, where exception is not
caught).
This doesn't seem terribly surprising to me - I imagine that checking would
slow down exception handling - but I couldn't find any documentation (and
posting on c.l.p didn't turn up anything either).
So I thought I would raise it here - perhaps there is a possible fix (my
obscure use case is that I have a backtracking search; backtracking occurs when
a certain exception is encountered; making that exception an ABC and allowing
existing exceptions to be registered with it allows the search to work with
existing code without a wrapper that catches and translates exceptions that
should trigger a backtrack). Or perhaps the docs could be extended. Or
perhaps I've misunderstood something...
Cheers,
Andrew
Python 3.2 (r32:88445, Feb 27 2011, 13:00:05)
[GCC 4.5.0 20100604 [gcc-4_5-branch revision 160292]] on linux2
Type help, copyright, credits or license for more information.
from abc import ABCMeta
class RootException(Exception,metaclass=ABCMeta): pass
...
class MyException(Exception): pass
...
RootException.register(MyException)
try:
... raise MyException
... except RootException:
... print('caught')
...
Traceback (most recent call last):
File stdin, line 2, in module
__main__.MyException
--
components: Interpreter Core
messages: 135521
nosy: acooke
priority: normal
severity: normal
status: open
title: ABC registration of Exceptions
type: behavior
versions: Python 3.2
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12029
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
ABC-registered Exceptions are not caught as subclasses
This isn't hugely surprising, but doesn't seem to be documented. Is it a bug,
or worth raising as one, or have I misunderstood?
Python 3.2 (r32:88445, Feb 27 2011, 13:00:05)
[GCC 4.5.0 20100604 [gcc-4_5-branch revision 160292]] on linux2
Type help, copyright, credits or license for more information.
from abc import ABCMeta
class RootException(Exception,metaclass=ABCMeta): pass
...
class MyException(Exception): pass
...
RootException.register(MyException)
try:
... raise MyException
... except RootException:
... print('caught')
...
Traceback (most recent call last):
File stdin, line 2, in module
__main__.MyException
If you assume that the ABC register class should work likeinheritance (as it
does with issubclass and isinstance then you would, I think, have expected the
exception above to have been caught.
Andrew
--
http://mail.python.org/mailman/listinfo/python-list
Re: My stupidity / strange inconsistency overriding class methods
Thanks for finding that reference in the data model docs! I was about to post a bug report because in PEP 3119 it says otherwise: The primary mechanism proposed here is to allow overloading the built-in functions isinstance() and issubclass(). The overloading works as follows: The call isinstance(x, C) first checks whether C.__instancecheck__ exists, and if so, calls C.__instancecheck__(x) instead of its normal implementation. http://www.python.org/dev/peps/pep-3119/ But that's now what's implemented in Issue http://bugs.python.org/issue1708353 which behaves as you quoted (only on the metaclass). Cheers, Andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: My stupidity / strange inconsistency overriding class methods
I didn't phrase that very well. I do see the point about this being an instance lookup on a class... -- http://mail.python.org/mailman/listinfo/python-list
Re: meteclasses 2.x/3.x compatibility
What I do in Lepl is use two stages. The first calls the type/metaclass
directly and the second subclasses that. This avoids using the sugar that
changes between 2 and 3.
So, for example, in
http://code.google.com/p/lepl/source/browse/src/lepl/matchers/matcher.py#40 I
have
_Matcher = ABCMeta('_Matcher', (object, ), {})
and then
class Matcher(_Matcher):
...
Andrew
--
http://mail.python.org/mailman/listinfo/python-list
Re: Language lib reference in man format ?
(1) Python's docs use Sphinx, which uses restructured text as a markup. You can generate man pages from restructured text using rst2man (which is installed on my computer, probably as part of python/docutils). HOWEVER I imagine it's not going to work very well, if at all, because Sphinx uses lots of fancy extensions. But... (2) Sphinx itself has a builder for man pages. It's described at http://sphinx.pocoo.org/builders.html#sphinx.builders.manpage.ManualPageBuilder So you should be able to configure Sphinx to use that. So you will need to download the docs package, install Sphinx, and then tweak the Sphinx configuration as described at the link above and at http://sphinx.pocoo.org/config.html#confval-man_pages This second approach should work. Andrew -- http://mail.python.org/mailman/listinfo/python-list
My stupidity / strange inconsistency overriding class methods
Hi, I've been staring at this problem, in various forms, all day. Am I missing something obvious, or is there some strange hardwiring of isinstance? This is with Python 3.2. class A(metaclass=ABCMeta): @classmethod def __instancecheck__(cls, instance): return False # no override assert isinstance(A(), A) assert A.__class__.__instancecheck__(A, A()) class B(type): def foo(self): return 42 class C(metaclass=B): @classmethod def foo(cls): return 7 # override assert C().__class__.foo() == 7 It seems to me that the above two cases are inconsistent. ABCMeta declares __instancecheck__ just like B declares foo. Yet C can override foo, but A is unable to override the instance check. Please help! Thanks, Andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: My stupidity / strange inconsistency overriding class methods
Also, there's something strange about the number of arguments (they're not consistent between the two examples - the A to __instancecheck__ should not be needed). Yet it compiles and runs like that. Very confused :o( -- http://mail.python.org/mailman/listinfo/python-list
Re: My stupidity / strange inconsistency overriding class methods
OK, sorry, I see the mistake. I'm confusing __class__ on the instance and on te class (the latter being the metaclass). Sorry again, Andrew -- http://mail.python.org/mailman/listinfo/python-list
Why is __root checked for in OrderedDict?
If you look at the code in http://hg.python.org/cpython/file/6adbf5f3dafb/Lib/collections/__init__.py#l49 the attribute __root is checked for, and only created if missing. Why? I ask because, from what I understand, the __init__ method will only be called when the object is first being created, so __root will always be missing. My only guess is that this allows subclasses to do strange things without breaking the code (and if so, is a nice defensive coding pattern). But I am worried I am missing something. Thanks, Andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Why is __root checked for in OrderedDict?
Is that normal? I mean, OK, it's possible (and yes I forgot it could be called directly), but is there any usual reason to do so? I guess what I'm asking is: if I'm writing library code should I be this careful? (I've written quite a lot of Python code without this ever biting me, but maybe I'm just lazy). For example, would you consider it a bug if someone complained that calling __init__() resulted in unexpected behaviour? Thanks, Andrew -- http://mail.python.org/mailman/listinfo/python-list
Confused about __prepare__
In the code below I use __prepare__ to change the class dictionary so that a
tuple is stored in __setitem__(). Since __getitem__() removes the tuple I
wasn't expecting any problems, but it seems that __init__ is being retrieved
via some other mechanism. Why? Is a copy of the dict being made somewhere?
If so, can I change that?
Thanks,
Andrew
class TupleDict(dict):
'''Stores additional info, but removes it on __getitem__().'''
def __setitem__(self, key, value):
print('setting', key, value)
super(TupleDict, self).__setitem__(key, (value, 'secret'))
def __getitem__(self, key):
value = super(TupleDict, self).__getitem__(key)
print('getting', key, value[0]) # drop secret
return value[0]
class TupleMeta(type):
@classmethod
def __prepare__(metacls, name, bases, **kargs):
print('in prepare')
return TupleDict()
def __new__(cls, name, bases, classdict):
print('in new')
return type.__new__(cls, name, bases, classdict)
class C(metaclass=TupleMeta):
def __init__(self):
self.a = 1
c = C()
in prepare
setting __module__ __main__
setting __init__ function __init__ at 0x7f37cbad40d8
in new
Traceback (most recent call last):
File ..., line 34, in module
c = C() # TypeError: 'tuple' object is not callable
TypeError: 'tuple' object is not callable
--
http://mail.python.org/mailman/listinfo/python-list
Re: Confused about __prepare__
Sorry I should probably have made clear that this is Python 3.2 -- http://mail.python.org/mailman/listinfo/python-list
Re: Confused about __prepare__
Yes, I think you're right, thanks. Makes sense from an efficiency POV. Luckily, it turns out I don't need to do that anyway :o) Cheers, Andrew -- http://mail.python.org/mailman/listinfo/python-list
Replacing *instance* dict
Related to the above, Is there anything wrong with the following code to
replace the *instance* rather than the class dict? It seems very crude, but
appears to work.
Thanks,
Andrew
class TupleSuper:
def __new__(cls):
print('in new')
instance = object.__new__(cls)
instance.__dict__ = TupleDict(instance.__dict__)
return instance
class D(TupleSuper):
def __init__(self):
self.a = 1
if __name__ == '__main__':
d = D()
assert d.a == 1
d.a = 2
assert d.a == 2
d.a = 'three'
assert d.a == 'three'
print('woop')
On Thursday, April 7, 2011 7:31:16 PM UTC-3, andrew cooke wrote:
class TupleDict(dict):
'''Stores additional info, but removes it on __getitem__().'''
def __setitem__(self, key, value):
print('setting', key, value)
super(TupleDict, self).__setitem__(key, (value, 'secret'))
def __getitem__(self, key):
value = super(TupleDict, self).__getitem__(key)
print('getting', key, value[0]) # drop secret
return value[0]
--
http://mail.python.org/mailman/listinfo/python-list
Why is return type in getfullspec().annotations named as return?
This conflicts with any parameter named return. Wouldn't it have been better to use - as the key? Is there any way this can be changed? Andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Why is return type in getfullspec().annotations named as return?
Sorry, ignore that. I just realised that return will be a reserved word, so that can't happen. Andrew -- http://mail.python.org/mailman/listinfo/python-list
python3.2m installed as (additional) binary
Hi, I just downloaded, built and altinstalled Python3.2 on Linux x64. I noticed that in /usr/local/bin I have two identical (says diff) binaries called Python3.2 and Python3.2m. Is this expected? I can find very little reference to them apart from a short discussion in python-dev where someone asks if users will be confused by this and Barry Warsaw replies saying that they won't see them (well, I do!). All I did was the usual ./configure; make; sudo make altinstall So is this a bug? Andrew pl6 Python-3.2: ls -l /usr/local/bin/python3.2* -rwxr-xr-x 2 root root 7368810 2011-02-27 13:03 /usr/local/bin/python3.2 -rwxr-xr-x 2 root root 7368810 2011-02-27 13:03 /usr/local/bin/python3.2m -rwxr-xr-x 1 root root1826 2011-02-27 13:03 /usr/local/bin/python3.2m-config pl6 Python-3.2: diff /usr/local/bin/python3.2 /usr/local/bin/python3.2m pl6 Python-3.2: -- http://mail.python.org/mailman/listinfo/python-list
Re: python3.2m installed as (additional) binary
[Sorry I clicked the wrong button so I think my prev reply went only to Tom] Thanks. Yes, they're hard linked. And the bug report mentions PEP 3149 which says that m means --with-pymalloc was used http://www.python.org/dev/peps/pep-3149/ Cheers, Andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Another Regexp Question
http://bugs.python.org/issue9179 On Jul 5, 9:38 pm, MRAB pyt...@mrabarnett.plus.com wrote: andrew cooke wrote: On Jul 5, 8:56 pm, MRAB pyt...@mrabarnett.plus.com wrote: andrew cooke wrote: What am I missing this time? :o( Nothing. It's a bug. :-( Sweet :o) Thanks - do you want me to raise an issue or will you? You found it. You can have the pleasure. -- http://mail.python.org/mailman/listinfo/python-list
[issue9179] Lookback with group references incorrect (two issues?)
New submission from andrew cooke and...@acooke.org:
from re import compile
# these work as expected
assert compile('(a)b(?=b)(c)').match('abc')
assert not compile('(a)b(?=c)(c)').match('abc')
assert compile('(a)b(?=c)(c)').match('abc')
assert not compile('(a)b(?=b)(c)').match('abc')
# but when you add groups, you get bugs
assert not compile('(?:(a)|(x))b(?=(?(2)x|c))c').match('abc') # matches!
assert not compile('(?:(a)|(x))b(?=(?(2)b|x))c').match('abc')
assert compile('(?:(a)|(x))b(?=(?(2)x|b))c').match('abc') # fails!
assert not compile('(?:(a)|(x))b(?=(?(1)c|x))c').match('abc') # matches!
assert compile('(?:(a)|(x))b(?=(?(1)b|x))c').match('abc') # fails!
# but lookahead works as expected
assert compile('(?:(a)|(x))b(?=(?(2)x|c))c').match('abc')
assert not compile('(?:(a)|(x))b(?=(?(2)c|x))c').match('abc')
assert compile('(?:(a)|(x))b(?=(?(2)x|c))c').match('abc')
assert not compile('(?:(a)|(x))b(?=(?(1)b|x))c').match('abc')
assert compile('(?:(a)|(x))b(?=(?(1)c|x))c').match('abc')
# these are similar but, in my opinion, shouldn't even compile
# (group used before defined)
assert not compile('(a)b(?=(?(2)x|c))(c)').match('abc') # matches!
assert not compile('(a)b(?=(?(2)b|x))(c)').match('abc')
assert not compile('(a)b(?=(?(1)c|x))(c)').match('abc') # matches!
assert compile('(a)b(?=(?(1)b|x))(c)').match('abc') # fails!
assert compile('(a)b(?=(?(2)x|c))(c)').match('abc')
assert not compile('(a)b(?=(?(2)b|x))(c)').match('abc')
assert compile('(a)b(?=(?(1)c|x))(c)').match('abc')
# this is the error we should see above
try:
compile('(a)\\2(b)')
assert False, 'expected error'
except:
pass
--
components: Library (Lib)
messages: 109382
nosy: acooke
priority: normal
severity: normal
status: open
title: Lookback with group references incorrect (two issues?)
type: behavior
versions: Python 2.6
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9179
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9179] Lookback with group references incorrect (two issues?)
andrew cooke and...@acooke.org added the comment: I hope the above is clear enough (you need to stare at the regexps for a time) - basically, lookback with a group conditional is not as expected (it appears to be evaluated as lookahead?). Also, some patterns compile that probably shouldn't. The re package only supports (according to the docs) lookback on expressions whose length is known. So I guess it's also possible that (?(n)pat1|pat2) should always fail that, even when len(pat1) = len(pat2)? Also, the generally excellent unit tests for the re package don't have much coverage for lookback (I am writing my own regexp lib and it passes all the re unit tests but had a similar bug - that's how I found this one...). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9179 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9179] Lookback with group references incorrect (two issues?)
andrew cooke and...@acooke.org added the comment: If it's any help, these are the equivalent tests as I think they should be (you'll need to translate engine(parse(... to compile(...) http://code.google.com/p/rxpy/source/browse/rxpy/src/rxpy/engine/backtrack/_test/engine.py?r=fc52f6959a0cfabdddc6960f47d7380128bb3584#284 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9179 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9179] Lookback with group references incorrect (two issues?)
andrew cooke and...@acooke.org added the comment: I thought someone was working on the re module these days? I thought there I'd seen some issues with patches etc? Anyway, short term, sorry - no patch. Medium/long term, yes it's possible, but please don't rely on it. The simplest way to document it is as you suggest, I think - just extend the qualifier on lookback requiring fixed length to exclude references to groups (it does seem to *bind* groups correctly on lookback, so there's no need to exclude them completely). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9179 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9179] Lookback with group references incorrect (two issues?)
andrew cooke and...@acooke.org added the comment: Ah good point, thanks. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9179 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Another Regexp Question
As ever, I guess it's most likely I've misunderstood something, but in
Python 2.6 lookback seems to actually be lookahead. All the following
tests pass:
from re import compile
assert compile('(a)b(?=(?(2)x|c))(c)').match('abc')
assert not compile('(a)b(?=(?(2)b|x))(c)').match('abc')
assert compile('(a)b(?=(?(1)c|x))(c)').match('abc')
assert compile('(a)b(?=(?(2)x|c))(c)').match('abc')
assert not compile('(a)b(?=(?(2)b|x))(c)').match('abc')
assert compile('(a)b(?=(?(1)c|x))(c)').match('abc')
But it seems to me that the first block should fail, because they
check the match *before* the point in question.
Note that without group references these work as I would expected:
assert compile('(a)b(?=b)(c)').match('abc')
assert not compile('(a)b(?=c)(c)').match('abc')
assert not compile('(a)b(?=b)(c)').match('abc')
assert compile('(a)b(?=c)(c)').match('abc')
in which lookback does indeed lookback (note the asymmetry, while the
first examples were symmetrical).
What am I missing this time? :o(
Thanks,
Andrew
--
http://mail.python.org/mailman/listinfo/python-list
Re: Another Regexp Question
On Jul 5, 8:56 pm, MRAB pyt...@mrabarnett.plus.com wrote: andrew cooke wrote: What am I missing this time? :o( Nothing. It's a bug. :-( Sweet :o) Thanks - do you want me to raise an issue or will you? Cheers, Andrew -- http://mail.python.org/mailman/listinfo/python-list
Help with Regexp, \b
This is a bit embarassing, but I seem to be misunderstanding how \b
works in regexps.
Please can someone explain why the following fails:
from re import compile
p = compile(r'\bword\b')
m = p.match(' word ')
assert m
My understanding is that \b matches a space at the start or end of a
word, and that word is a word - http://docs.python.org/library/re.html
What am I missing here? I suspect I am doing something very stupid.
Thanks,
Andrew
--
http://mail.python.org/mailman/listinfo/python-list
Re: Help with Regexp, \b
On May 29, 11:24 am, Duncan Booth duncan.bo...@invalid.invalid
wrote:
andrew cooke and...@acooke.org wrote:
Please can someone explain why the following fails:
from re import compile
p = compile(r'\bword\b')
m = p.match(' word ')
assert m
[...]
You misunderstand what \b does: it doesn't match a space, it matches a 0
length string on a boundary between a non-word and a word.
[...]
That's what I thought it did... Then I read the docs and confused
empty string with space(!) and convinced myself otherwise. I
think I am going senile.
Thanks very much!
Andrew
--
http://mail.python.org/mailman/listinfo/python-list
Re: Parser
On May 2, 3:54 pm, Andreas Löscher andreas.loesc...@s2005.tu- chemnitz.de wrote: Hi, I am looking for an easy to use parser. I am want to get an overview over parsing and want to try to get some information out of a C-Header file. Which parser would you recommend? Best, Andreas I develop Lepl - http://www.acooke.org/lepl/ I think it's easy to use, and other people have said the documentation is fairly good. It doesn't have a GUI like Antlr, but it is recursive descent rather and completely implemented in Python, which (I think) makes it easier to understand and extend. Cheers, Andrew -- http://mail.python.org/mailman/listinfo/python-list
Ann: Validating Emails and HTTP URLs in Python
Hi, The latest Lepl release includes an implementation of RFC 3696 - the RFC that describes how best to validate email addresses and HTTP URLs. For more information please see http://www.acooke.org/lepl/rfc3696.html Lepl's main page is http://www.acooke.org/lepl Because Lepl compiles to regular expressions wherever possible, the library is quite fast - in testing I was seeing about 1ms needed to validate a URL. Please bear in mind that this is the very first release of this module, so it may have some bugs... If you find any problems contact me and I'll fix them ASAP. Thanks, Andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Ann: Validating Emails and HTTP URLs in Python
FYI, Fourthought's PyXML has a module called uri.py that contains regexes for URL validation. I've over a million URLs (harvested from the Internet) through their code. I can't say I checked each and every result, but I never saw anything that would lead me to believe it was misbehaving. It might be interesting to compare the results of running a large list of URLs through your code and theirs. Good luck Philip It's getting a set of URLs that's the main problem. I've tested it with URL examples in RFC 3696, and with a few extra ones that test particular issues, but when I looked around I couldn't find any public, obvious list of URLs for general testing. Could I use your list? Also, same for emails... Cheers, Andrew Cheers, Andrew -- http://mail.python.org/mailman/listinfo/python-list
Confused by slash/escape in regexp
Is the third case here surprising to anyone else? It doesn't make
sense to me...
Python 2.6.2 (r262:71600, Oct 24 2009, 03:15:21)
[GCC 4.4.1 [gcc-4_4-branch revision 150839]] on linux2
Type help, copyright, credits or license for more information.
from re import compile
p1 = compile('a\x62c')
p1.match('abc')
_sre.SRE_Match object at 0x7f4e8f93d578
p2 = compile('a\\x62c')
p2.match('abc')
_sre.SRE_Match object at 0x7f4e8f93d920
p3 = compile('a\\\x62c')
p3.match('a\\bc')
p3.match('abc')
p3.match('a\\\x62c')
Curious/confused,
Andrew
--
http://mail.python.org/mailman/listinfo/python-list
Re: Confused by slash/escape in regexp
On Apr 11, 8:12 pm, Lie Ryan lie.1...@gmail.com wrote: In the first case, *python* will unescape the string literal '\x62' into letters 'b'. In the second case, python will unescape the double backslash '\\' into a single slash '\' and *regex* will unescape the single-slash-62 into 'b'. In the third case, *python* will unescape double backslash '\\' into single-slash '\' and byte-string-62 '\x62' to letter-b 'b', and regex received it as 'a\bc', which interpreted as a special character to regex: \b Matches the empty string, but only at the start or end of a word. ah, brilliant! yes. thank-you very much! andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Confused by slash/escape in regexp
On Apr 11, 7:18 pm, Paul McGuire pt...@austin.rr.com wrote: [...] So I would say the surprise isn't that case 3 didn't match, but that case 2 matched. Unless I just don't get what you were testing, not being an RE wiz. Case 2 is the regexp engine interpreting escapes that appear as literal strings. It's weird, because what's the point of Python would do it for you anyway, but it seems to be the correct behaviour. Andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Function name unchanged in error message
On Jan 29, 11:22 am, Peter Otten __pete...@web.de wrote: The name is looked up in the code object. As that is immutable you have to make a new one: [details snipped] thanks very much! sorry i didn't reply earlier - been travelling. (also, thanks to any other replies - i'm just reading through at the moment and this is the first one i've got to that will help me solve it, but i don't mean to exclude anything later...!) andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Function name unchanged in error message
On Jan 29, 5:37 pm, Gabriel Genellina gagsl-...@yahoo.com.ar wrote: The decorator module is a very fine addition to anyone's tool set -- but in this case it is enough to use the wraps() function from the functools standard module. ah, thanks! i thought something like this existed in the standard lib, but couldn't find it. andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Function name unchanged in error message
On Jan 29, 11:50 am, exar...@twistedmatrix.com wrote: new.function and new.code will let you construct new objects with different values (and copying over whichever existing attributes you want to preserve). unfortunately new is deprecated and dropped from 3. i can't see how the same functionality is available in the types module for 3 - am i missing something obvious? http://docs.python.org/library/new.html http://docs.python.org/3.1/library/types.html#module-types thanks, andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Function name unchanged in error message
On Jan 30, 7:17 pm, andrew cooke and...@acooke.org wrote: On Jan 29, 5:37 pm, Gabriel Genellina gagsl-...@yahoo.com.ar wrote: The decorator module is a very fine addition to anyone's tool set -- but in this case it is enough to use the wraps() function from the functools standard module. ah, thanks! i thought something like this existed in the standard lib, but couldn't find it. andrew ah, sorry, peter's code uses types, so i assume that's the way to go (i was hoping that there was something a bit simpler - i don't like the fact that the code in peter's code has a fixed list of special names). andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Function name unchanged in error message
On Jan 29, 1:09 pm, Michele Simionato michele.simion...@gmail.com wrote: On Jan 29, 2:30 pm, andrew cooke and...@acooke.org wrote: Is there any way to change the name of the function in an error message? In the example below I'd like the error to refer to bar(), for example (the motivation is related function decorators - I'd like the wrapper function to give the same name) Use the decorator module which does the right thing:http://pypi.python.org/pypi/decorator curiously, decorator doesn't have this issue, because the way it defines decorators uses *args. so the error i gave cannot occur at the level of the decorator - the extra arg is passed to the wrapped function, and so the error message is correct because it is generated by the inner function. i need to look at my code; this might be the simplest solution of all. thanks, andrew -- http://mail.python.org/mailman/listinfo/python-list
Function name unchanged in error message
Is there any way to change the name of the function in an error message? In the example below I'd like the error to refer to bar(), for example (the motivation is related function decorators - I'd like the wrapper function to give the same name) def foo(): ... return 7 ... foo.__name__ = 'bar' foo(123) Traceback (most recent call last): File stdin, line 1, in module TypeError: foo() takes no arguments (1 given) Thanks, Andrew -- http://mail.python.org/mailman/listinfo/python-list
[issue7792] Errors registering non-classes with ABCs
New submission from andrew cooke and...@acooke.org: There are two related issues here. The first is, I think, a simple bug: When I try to register a function as a subclass of an ABC I get the error: TypeError: issubclass() arg 2 must be a class or tuple of classes Looking at the code - http://svn.python.org/view/python/trunk/Lib/abc.py?annotate=72278 - it seems that instead the test at line 99 should have triggered a better error: TypeError(Can only register classes) In other words, line 99 should read: if not isinstance(subclass, type): (currently the code tests the self class, cls, which is pointless). My second issue is that this shouldn't raise an error at all. It should be possible for functions to be registered as subclasses of ABCs. This would simplify some code of mine that uses functions and classes interchangeably, and I can see no real reason why it shouldn't work. From the user's point of view, my library provides a bunch of things that all look the same (they are provided with arguments and do stuff). Whether these are constructors or functions is irrelevant... and yet my code has to handle constructors and functions differently simply because of the errors here. What I suspect I will do is have my own hash table, and forget ABCs, but that is a pity because these functions really do work as constructors (they are factories) and so the idea of them being subclasses of an ABC helps clarify my code. -- components: Library (Lib) messages: 98387 nosy: acooke severity: normal status: open title: Errors registering non-classes with ABCs type: behavior versions: Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7792 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: Comparison of parsers in python?
On Sep 21, 10:59 am, Nobody nob...@nowhere.com wrote: I have a similar question. What I want: a tokeniser generator which can take a lex-style grammar (not necessarily lex syntax, but a set of token specifications defined by REs, BNF, or whatever), generate a DFA, then run the DFA on sequences of bytes. It must allow the syntax to be defined at run-time. What I don't want: anything written by someone who doesn't understand the field (i.e. anything which doesn't use a DFA). lepl will do this, but it's integrated with the rest of the parser (which is recursive descent). for example: float = Token(Float()) word = Token(Word(Lower()) punctuation = ~Token(r'[\.,]') line = (float | word)[:, punctuation] parser = line.string_parser() will generate a lexer with three tokens. here two are specified using lepl's matchers and one using a regexp, but in all three cases they are converted to dfas internally. then a parser is generated that will match a sequence of floats and words, separated by punctuation. spaces are discarded by the lexer by default, but that can be changed through the configuration (which would be passed to the string_parser method). it's also possible to specify everything using matchers and then get lepl to compile as much as possible of the matcher graph to nfas before matching (nfas rather than dfas because they are implemented with a stack to preserve the backtracking abilities of the recursive descent parser they replace). the problem here is that not all matchers can be converted (matchers can contain arbitrary python functions, while my nfa+dfa implementations cannot, and also my compiler isn't very smart), while using tokens explicitly gives you an error if the automatic compilation fails (in which case the simple fix is to just give the regexp). (also, you say sequence of bytes rather than strings - lepl will parse the byte[] type in python3 and even has support for matching binary values). disclaimer: newish library, python 2.6+ only, and while i have quite a few users (or, at least, downloads), i doubt that many use these more advanced features, and everything is pure python with little performance tuning so far. andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Intercepting binding?
On Sep 24, 7:12 am, Carl Banks pavlovevide...@gmail.com wrote: with capture_changed_bindings() as changed: b = 5 c = 4 d = 6 print changed test() Quick and dirty, not robust at all. But you get the idea. Carl Banks brilliant. using the with context is an excellent idea. thanks very much. andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Intercepting binding?
On Sep 24, 5:20 am, Steven D'Aprano ste...@remove.this.cybersource.com.au wrote: Speaking as a user (although not of Andrew's domain specific language), I'd like to say to developers PLEASE PLEASE PLEASE don't try to help me with half-baked unreliable solutions that only work sometimes. There's few things worse than unreliable tools that break just when you've come to rely on them. The context is that I am looking at how best to provide debugging support for a recursive descent parser. If the actions are pure then the whole thing is deterministic, so it might be possible to do something like buddha (a declarative debugger for haskell). I'm also looking for a paper that was linked to somewhere this last month or so on one of the popular sites/blogs about debugging prolog (since the two are closely related - depth first search etc) - I know this is vague, but if it rings a bell with anyone... Anyway, doing this in pure Python makes risks losing the information associated with variable names. I was wondering how I might reduce that. The reason I asked for unreliable half-baked solutions is that I am exploring what might be possible and, in my experience, this group prefers to lecture me on what they think I should do rather than answer the damn question. I was hoping that by explicitly saying that reliability is not important, people might feel more free to give wild ideas that I could learn from and improve on. It's significant, depressing, and not at all surprising that every person who replied to this thread told me, in one way or another, that was I was asking was wrong or impossible or foolhardy. If I want your advice on how to write popular tools, you can be sure I will come and ask you for it. That is not what I asked for here. But since we're all passing round unsolicited advice, here's some from me. Being an anal retentive will get you a long way in programming. Dotting the is and crossing the ts is 99% of what it's all about. I agree. But the last 1% requires a little bit of imagination. You should try it one day. Andrew -- http://mail.python.org/mailman/listinfo/python-list
Intercepting binding?
This is a bit vague, I'm afraid, but is there any way for me to take code like: a = Foo() beta = Bar() and somehow attach the string a to the Foo instance and beta to the Bar instance. At some later point in the program I want to be able to look at the Bar instance and say to the user this was called beta in your routine. The motivation is debugging an embedded domain specific language and the solution has to be cross platform for Python 3+ (bonus points for Python 2 too). Obviously I can parse the code, but I was wondering if there was some other (no doubt terribly hacky) approach. Even some idea of what to google for would be a help... Thanks, Andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Intercepting binding?
For example, I assume it's possible to somehow access the dictionary for the current block, but I can't see how to do this after assignment. If I do it in the Foo constructor, for example, a will not yet be bound. On Sep 23, 8:15 pm, andrew cooke and...@acooke.org wrote: This is a bit vague, I'm afraid, but is there any way for me to take code like: a = Foo() beta = Bar() and somehow attach the string a to the Foo instance and beta to the Bar instance. At some later point in the program I want to be [..] -- http://mail.python.org/mailman/listinfo/python-list
Re: Intercepting binding?
On Sep 23, 8:40 pm, Rhodri James rho...@wildebst.demon.co.uk wrote: eggs[42] = Foo() beans['spam'] = Foo() chips.spam = Foo() spam[eggs.beans['chips']] = Foo() spam.append(Foo()) these are valid points, but in practice the main use (for the restricted application i care about) is si,ple variables, and this is an optional extra to help the user, so it's OK if it only works sometimes. at the moment i'd be happy with any half-baked unreliable solution that is transparent... andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Intercepting binding?
On Sep 23, 10:11 pm, Dave Angel da...@ieee.org wrote: This comes up periodically in this list, and the answer is always something like: you can't get there from here. Well, I'm both flexible and desperate, so this is a possible route (perhaps near enough): import sys class Foo(object): def __rlshift__(self, name): try: raise Exception() except: locals = sys.exc_traceback.tb_frame.f_back.f_locals locals[name] = self if __name__ == '__main__': foo = Foo() 'a' foo print(a) andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Intercepting binding?
for the record, googling for f_back.f_locals reveals a wide variety of similar hacks and also a cleaner way to access the current frame: inspect.currentframe() -- http://mail.python.org/mailman/listinfo/python-list
Re: Comparison of parsers in python?
On Sep 19, 9:34 pm, Peng Yu pengyu...@gmail.com wrote: On Sep 19, 6:05 pm, Robert Kern robert.k...@gmail.com wrote: http://nedbatchelder.com/text/python-parsers.html This is more a less just a list of parsers. I would like some detailed guidelines on which one to choose for various parsing problems. it would be simpler if you described what you want to do - parsers can be used for a lot of problems. also, the parsers do not exist in isolation - you need to worry about whether they are supported, how good the documentation is, etc. and different parsers handle different grammars - see http://wiki.python.org/moin/LanguageParsing - so if you already have a particular grammar then your life is simpler if you choose a parser that matches. these are the three that i know most about - i think all three are currently maintained: for simple parsing problems, i think pyparsing is the most commonly used - http://pyparsing.wikispaces.com/ my own lepl - http://www.acooke.org/lepl/ - tries to combine ease of use with some more advanced features the nltk - http://www.nltk.org/ - is particularly targeted at parsing natural languages and includes a wide variety of tools. but for parsing a large project you might be better interfacing to a compiled parser (lepl has memoisation, so should scale quite well, but it's not something i've looked at in detail yet). andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Comparison of parsers in python?
On Sep 20, 8:11 am, Peng Yu pengyu...@gmail.com wrote: On Sun, Sep 20, 2009 at 6:50 AM, andrew cooke and...@acooke.org wrote: On Sep 19, 9:34 pm, Peng Yu pengyu...@gmail.com wrote: On Sep 19, 6:05 pm, Robert Kern robert.k...@gmail.com wrote: http://nedbatchelder.com/text/python-parsers.html This is more a less just a list of parsers. I would like some detailed guidelines on which one to choose for various parsing problems. it would be simpler if you described what you want to do - parsers can be used for a lot of problems. I have never used any parser. The task at my hand right now is to parse thishttp://genome.ucsc.edu/goldenPath/help/wiggle.html, which is a fairly simple even without any parser package. I think that it is worthwhile for me to learn some parser packages to try to parse this format. So that I may in future parse more complex syntax. Do you have any suggestion what parser I should use for now? pyparsing would work fine for that, and has a broad community of users that will probably be helpful. i am currently working on an extension to lepl that is related, and i may use that format as an example. if so, i'll tell you. but for now, i think pyparsing makes more sense for you. andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Comparison of parsers in python?
On Sep 19, 11:39 pm, TerryP bigboss1...@gmail.com wrote: [...] For flat data, simple unix style rc or dos style ini file will often suffice, and writing a parser is fairly trivial; in fact writing a [...] python already includes parsers for .ini configuration files. [...] The best way to choose a parser, is experiment with several, test (and profile!) them according to the project, then pick the one you like best, out of those that are suitable for the task. Profiling can be very important. profiling is going to show you the constant complexity, but - unless you think hard - it's not going to explain how a parser will scale (how performance changes as the amount of text to be parsed increases). for that, you need to look at the algorithm used, which is usually documented somewhere. there's going to be trade-offs - parsers that handle large texts better could well be more complex and slower on small texts. andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Comparison of parsers in python?
One word of warning - the documentation for that format says at the beginning that it is compressed in some way. I am not sure if that means within some program, or on disk. But most parsers will not be much use with a compressed file - you will need to uncompress it first. -- http://mail.python.org/mailman/listinfo/python-list
Re: Comparison of parsers in python?
The file size of a wig file can be very large (GB). Most tasks on this file format does not need the parser to save all the lines read from the file in the memory to produce the parsing result. I'm wondering if pyparsing is capable of parsing large wig files by keeping only minimum required information in the memory. ok, now you are getting into the kind of detail where you will need to ask the authors of individual packages. lepl is stream oriented and should behave as you want (it will only keep in memory what it needs, and will read data gradually from a file) but (1) it's fairly new and i have not tested the memory use - there may be some unexpected memory leak; (2) it's python 2.6/3 only; (3) parsing line-based formats like this is not yet supported very well (you can do it, but you have to explicitly match the newline character to find the end of line); (4) the community for support is small. so i would suggest asking on the pyparsing list for advice on using that with large data files (you are getting closer to the point where i would recommend lepl - but of course i am biased as i wrote it). andrew ps is there somewhere can download example files? this would be useful for my own testing. thanks. -- http://mail.python.org/mailman/listinfo/python-list
Re: Comparison of parsers in python?
also, parsing large files may be slow. in which case you may be better with a non-python solution (even if you call it from python). your file format is so simple that you may find a lexer is enough for what you want, and they should be stream oriented. have a look at the shlex package that is already in python. will that help? alternatively, perhaps plex - http://www.cosc.canterbury.ac.nz/greg.ewing/python/Plex/ - that is pure python, but greg ewing is a good programmer and he says on that page it is as fast as possible for python, so it is probably going to be quite fast. andrew ps maybe you already know, but a lexer is simpler than a parser in that it doesn't use the context to decide how to treat things. so it can recognise something is a number, or a word, or a quoted string, but not whether it is part of a track definition line or a data value, for example. but in this case the format is so simple that a lexer might do quite a ot of what you want, and would make the remaining plain python program very simple. -- http://mail.python.org/mailman/listinfo/python-list
Re: Comparison of parsers in python?
I don't quite understand this point. If I don't use a parser, since python can read numbers line by line, why I need a lexer package? for the lines of numbers it would make no difference; for the track definition lines it would save you some work. as you said, this is a simple format, so the case for any tool is marginal - i'm just exploring the options. andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Comparison of parsers in python?
So for the track definition, using a lexer package would be better than using regex in python, right? they are similar. a lexer is really just a library that packages regular expressions in a certain way. so you could write your own code and you would really be writing a simple lexer. the advantage of writing your own code is that it will be easier to modify and you will get more experience with regular expressions. the advantage of using a library (a lexer) is that it has already been tested by other people, it is already packaged and so your code will be better structured, and it may have features (perhaps logging, or handling of quoted strings, for example) that will save you some work in the future. andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Comparison of parsers in python?
On Sep 20, 9:12 am, andrew cooke and...@acooke.org wrote: ps is there somewhere can download example files? this would be useful for my own testing. thanks. i replied to a lot of your questions here; any chance you could reply to this one of mine? the wig format looks like it could be a good test for lepl. thanks, andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Comparison of parsers in python?
On Sep 20, 3:16 pm, Peng Yu pengyu...@gmail.com wrote: On Sun, Sep 20, 2009 at 1:35 PM, andrew cooke and...@acooke.org wrote: On Sep 20, 9:12 am, andrew cooke and...@acooke.org wrote: ps is there somewhere can download example files? this would be useful for my own testing. thanks. i replied to a lot of your questions here; any chance you could reply to this one of mine? the wig format looks like it could be a good test for lepl. I missed your question. I have only some files that I only use a subset of the syntax in wig. Here is one example. ah, thanks. i'll see if i can find something on the 'net - i am hoping to test how / whether gigabytes of data can be parsed. andrew -- http://mail.python.org/mailman/listinfo/python-list
[issue3058] Let SimpleXMLRPCServer pass client_address to called functions.
andrew cooke and...@acooke.org added the comment: Came here wondering how best to solve this myself. I already subclass the request handler to do client validation (password etc) and it stuck me that a simpler solution would be to use thread local storage. This avoids having to modify dispatch. The disadvantage is that this uses a single global scope. I don't think it's a better solution than that suggested by samwyse for a final solution, but I thought I'd note it here because it's a simpler alternative for people needing to add a work-around, and I suspect many people aren't aware of the possibility. import _threading_local ; help(_threading_local) If this wouldn't work (maybe I've misunderstood the server threading?) then I'd appreciate someone correcting me. Thanks. -- nosy: +acooke ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3058 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: Frustrated with scopes
On Aug 12, 8:52 am, Dave Angel da...@ieee.org wrote: Supply us with just enough source code to actually try it, give the full error message including traceback, and tell us what you expected to see. Also tell us Python version (sys.version) So far you've done none of these. When I try the following, I get no errors, using Python 2.6.2 class _StreamFactory(object): @staticmethod def __call__(lines, source, join=''.join): class Line(object): __source = source __join = join return Line() fact = _StreamFactory() obj = fact(43, name.txt) print obj Ah! OK, thanks for that. I need to look at this again. I'll post again if necessary, but if it works for you then I clearly don't understand what the issue is myself. Andrew -- http://mail.python.org/mailman/listinfo/python-list
Frustrated with scopes
Is there a way to make this work (currently scope and join are undefined at runtime when the inner class attributes are defined): class _StreamFactory(object): @staticmethod def __call__(lines, source, join=''.join): class Line(object): __source = source __join = join [...] I can get something working by bouncing through global values, but it looks awful and I think it's a source of a bug due too values being redefined. Thanks, Andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Frustrated with scopes
correction: source and join are undefined. Sorry, Andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Parsing Binary Structures; Is there a better way / What is your way?
On Aug 5, 10:46 am, Martin P. Hellwig martin.hell...@dcuktec.org wrote: Hi List, On several occasions I have needed (and build) a parser that reads a binary piece of data with custom structure. For example (bogus one): BE +-+-+-+-+--++ | Version | Command | Instruction | Data Length | Data | Filler | +-+-+-+-+--++ Version: 6 bits Command: 4 bits Instruction: 5 bits Data Length: 5 bits Data: 0-31 bits Filler: filling 0 bits to make the packet dividable by 8 hi, sorry i'm a bit late here, but lepl does exactly this. also, as you asked for in another post, the BitString class allows arbitrary indexing into a sequence of bits. and because it's part of a recursive descent parser you can match anything (lepl will handle - although less efficiently - left recursive and ambiguous grammars). see the example at http://www.acooke.org/lepl/binary.html#matching which constructs an ethernet frame and then parses it, extracting the source and destination addresses. feel free to email me with more questions. disclaimer: this is quite new and i don't know of anyone that actually uses it; it is also Python3 only (because it uses bytes()). andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Suppressing Implicit Chained Exceptions (Python 3.0)
David Bolen wrote: andrew cooke and...@acooke.org writes: However, when printed via format_exc(), this new exception still has the old exception attached via the mechanism described at http://www.python.org/dev/peps/pep-3134/ (this is Python 3.0). If you're in control of the format_exc() call, I think the new chain keyword parameter can disable this and restore the old behavior. If you're not in control of the traceback display, I'm not sure there's an easy way to prevent it, given that displaying chained exceptions is the default mode. Yeah, that's the impression I got too. Just in case it wasn't clear, I'd like the chaining not to exist at all in this case), so that it doesn't appear in format_exc outside my control. Thanks, Andrew -- http://mail.python.org/mailman/listinfo/python-list
Suppressing Implicit Chained Exceptions (Python 3.0)
I have some (library) code where an exception is caught and, since the underlying cause is rather obscure, a different exception is raised that more clearly explains the issue to the caller. However, when printed via format_exc(), this new exception still has the old exception attached via the mechanism described at http://www.python.org/dev/peps/pep-3134/ (this is Python 3.0). Is there any simple way to stop this? It's rather annoying and misleading, as it exposes a lot of internal detail that the caller does not understand or want. This is marked as an open issue in the PEP described above. Thanks, Andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Generic web parser
http://groups.google.com/group/beautifulsoup/browse_thread/thread/d416dd19fdaa43a6 http://jjinux.blogspot.com/2008/10/python-some-notes-on-lxml.html andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: SQL and CSV
Lawrence D'Oliveiro wrote: In message gu0ofm$oj9$0...@news.t-online.com, Peter Otten wrote: While it may not matter here using placeholders instead of manually escaping user-provided values is a good habit to get into. Until you hit things it can't deal with. The post you are replying to was talking about using the SQL library's ? syntax that automatically escapes values. The usual reason this is recommended (if I have understood correctly) is that the library code is much more likely to foil injection attacks. I have seen this mentioned often and assume it is good advice. Can you expand on your comment? I assume you are thinking of how the library might handle some strange class. But aren't the number of types limited by SQL? In which case a thing that can't be handled could presumably be managed by adding an appropriate __str__ or __float__ or whatever? And you would still use the library to give safety with other values. Maybe you could give an example of the kind of problem you're thinking of? Thanks, Andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: SQL and CSV
even if you're not open to injection attacks, you're still less likely to get escaping correct than a puprose written, widely used library. my request for more information was directed to lawrence, who said until you hit things it can't deal with which seemed to be some kind of cryptic argument against parameters. andrew Nick wrote: On May 8, 1:49 pm, andrew cooke and...@acooke.org wrote: Lawrence D'Oliveiro wrote: In message gu0ofm$oj9$0...@news.t-online.com, Peter Otten wrote: While it may not matter here using placeholders instead of manually escaping user-provided values is a good habit to get into. Until you hit things it can't deal with. The post you are replying to was talking about using the SQL library's ? syntax that automatically escapes values. The usual reason this is recommended (if I have understood correctly) is that the library code is much more likely to foil injection attacks. I have seen this mentioned often and assume it is good advice. Can you expand on your comment? I assume you are thinking of how the library might handle some strange class. But aren't the number of types limited by SQL? In which case a thing that can't be handled could presumably be managed by adding an appropriate __str__ or __float__ or whatever? And you would still use the library to give safety with other values. Maybe you could give an example of the kind of problem you're thinking of? Thanks, Andrew Injection attacks aren't an issue, its a local app. It's part of a reconciliation system, where sometimes data is in csv files. If you want the whole csv file, you can use csv module without a problem. In some cases, I need to manipulate the data. The choices are hard code the manipulation, or load the data from a config file. So what I've got is the query in the config and I can process it. Nick -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: The whole story
Paul Hemans wrote: Hi Andrew, The reason I am using mapped objects is that I need to abstract from the database implementation allowing the replication to target a number of different platforms. This will definitely slow things down. have you looked at sqlalchemy's generic sql support? you can construct sql statements using python classes/functions called select, update etc and sqlalchemy automatically makes things work with different database implementations. it's not perfect, but i have the same code working with mysql and oracle, which covers quite a range :o) process a whole pile in memory and then (perhaps every 10,000 - when your memory is about to run out and start paging) flush the session. Under windows how can I tell when memory is about to run out? I guess there is no cross-platform solution to this. i don't know, sorry. Writing external files has all come about from a post titled Memory problems (garbage collection) by Carbon Man which I never got a resolution to. I was trying to execute gc.collect() when a process was complete because I was having huge problems with memory (though the problem still remains). If I stop at import schema There are 2524104 objects processed by gc.collect() you shouldn't need to call gc.collect(). when you write everything out to the database, if you have no other references to the objects, python will clean them up automatically. calling gc.collect() won't make any difference - python's gc already works just fine. if your memory use isn't going down then either (1) you are not writing out/flushing correctly (if you want i can check my code later today and tell you exactly what i do) or you are keeping references to your objects elsewhere (eg in a dictionary in the code you use to construct them). andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Is there a maximum size to a Python program?
not sure i've read all the posts on this, and i don't fully understand the problem, but someone's mentioned sqlalchemy, so here's my experience with that and large updates using mapped objects. 1 - don't commit each object as you modify it. instead, process a whole pile in memory and then (perhaps every 10,000 - when your memory is about to run out and start paging) flush the session. you'll find that the memory use is very predictable - it ramps up as objects are cached and then drops back down to the original level once they're sent to the database. 2 - various sql related commands in sqlalchemy will take lists rather than single values, and process all elements in the list in one chunk. this is much more efficient. using sql directly is faster and uses lest memory than using mapped objects (the nice thing about sqlalchemy is that you can use sql directly when it's the best solution, and mapped objects when they are more useful). these are both kind-of obvious, but that's all i needed to handle fairly large data volumes with sqlalchemy. andrew Carbon Man wrote: I have a program that is generated from a generic process. It's job is to check to see whether records (replicated from another system) exist in a local table, and if it doesn't, to add them. I have 1 of these programs for every table in the database. Everything works well until I do the postcode table. The generated code is 5MB for a system with no current data. Normally the file would not be this big as only the changes are copied over. Python just quits, I have tried stepping through the code in the debugger but it doesn't even start. I am thinking that dynamically generating the programs to run might not be such a good idea. It would be a shame to drop it because the system needs to be generic and it runs from an XML file so the resulting code could be pretty complex, and I am new to Python. The program did generate a pyc so it was able to compile. Thoughts anyone? -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: [OT] large db question about no joins
on the more general point about exactly how to handle large data sets, i found this article interesting - http://highscalability.com/unorthodox-approach-database-design-coming-shard andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Automatically generating arithmetic operations for a subclass
Arnaud Delobelle wrote: I do this: binops = ['add', 'sub', 'mul', 'div', 'radd', 'rsub'] # etc unops = ['neg', 'abs', invert'] # etc binop_meth = def __%s__(self, other): return type(self)(int.__%s__(self, other)) unop_meth = def __%s__(self): return type(self)(int.__%s__(self)) class MyInt(int): for op in binops: exec binop_meth % (op, op) for op in unops: exec unop_meth % (op, op) del op what's the del for? curious, andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Automatically generating arithmetic operations for a subclass
Arnaud Delobelle wrote: andrew cooke and...@acooke.org writes: Arnaud Delobelle wrote: class MyInt(int): for op in binops: exec binop_meth % (op, op) for op in unops: exec unop_meth % (op, op) del op what's the del for? Without it, 'op' would end up as a class attribute. ah! ok, that makes sense, i guess. thanks. (i just tried it out and you're right, of course, but also if binops and unops are empty you get an error, although i guess that's no an issue here). andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Retrieving a specific object from a list?
andrew cooke wrote: [...] but when you need to access instances by more than one value (.bar and .baz) then typically that's a hard problem, and there's a trade-off somewhere. you might find writing a special container that contains two dicts is useful. if so, you might want to use weak references - see weakref module). thinking a bit more about when this has occurred in my own programs, and it tends to be when i've ended up with a monster object/collection that is trying to do too many different things (typically because i am worrying about performance when i shouldn't be). in that case the fix has been to break things up and use simpler collections for each step, converting as necessary between steps. andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Retrieving a specific object from a list?
Jeremiah Dodds wrote: I've been looking over some of my code, and I've found something I do that has a bit of a smell to it. I've searched the group and docs, and haven't found much of anything that solves this particular problem, although I may just not be searching correctly. Anyhow, I find that often I'll have a list of objects of some sort that I want to operate on. Most of the time, I'll want to operate on the entire list, but sometimes I'll want to operate on just one element, or retrieve just one element, and I end up with code something like the following: items = [Foo(), Foo(), ... Foo()] a_item = [x for x in items if x.bar == some_value][0] another_item = [x for x in items if x.baz == some_other_value][0] This doesn't seem correct at all, looping over the entire list to create a list of one element and then pulling that item out. Any advice? [some/all of this may be obvious, but since you asked...] i think the most common solution (at least in my code) is to use a dict and store the data as a map from value to instance. you can still operate on all the instances via .values(), but you also have O(1) access via the key. another approach is to use a sorted list. then you can access things via the bisect module in O(log(n)) time. related to sorted lists, you may be able to reframe the algorithm as something that operates on a queue or stack (if your some_value is to find the next item according to an order that is fixed by earlier processing). in that case you want deque from collections. but when you need to access instances by more than one value (.bar and .baz) then typically that's a hard problem, and there's a trade-off somewhere. you might find writing a special container that contains two dicts is useful. if so, you might want to use weak references - see weakref module). all the above are relatively direct solutions. in my experience this kind of issue often comes from not thinking at a high enough level about the algorithm - even though my third suggestion (deque) sounds rather obscure you may find that once you look at you algorithm more carefully it can be rewritten in that way. i think i've seen this in my own code as i improve at integrating what might be more functional idioms into python in a natural (pythonic) way. andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Re: Why does Python show the whole array?
Peter Otten wrote: John Posner wrote: Given how common string maniuplations are, I guess I'm surprised that Python hasn't yet made contains() into both a string-module function *and* a string-object method. Could you explain why you prefer 'contains(belly, beer)' or 'belly.contains(beer)' over 'beer in belly'? The last form may be a bit harder to find in the documentation, but once a newbie has learned about it he'll find it easy to remember. i don't know why i get involved in this type of discussion, but while i agree that in is the correct answer, i am not sure it's obvious or easy to remember once you know it. perhaps to a newbie with a sweet, innocent and unsullied mind. but for an already corrupted programmer like myself, it's not easy to remember - i keep forgetting it! i think it's because i associate in with iteration, and assume everything else will be method calls or functions (and from earlier discussions here, it's clear some people are even more blinkered, and think everything should be methods) so yeah, in is right, but arguments about what is natural and easy aren't really worth wasting bits over... andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Scrap Posts
are you on the mailing list (python-list@python.org) or reading via google groups? groups is full of junk, but the list is filtered. the (filtered) list is also available via gmane and similar. (disclaimer - i also use spamassasin so it's possible that is cleaning the mail up, but this discussion has happened before, here and on -dev, and people have generally acclaimed the list filtering). andrew Avi wrote: Hey Folks, I love this group and all the awesome and python savvy people who post here. However I also see some dumb posts like 'shoes' or something related to sex :( What can we do about crap like this? Can we clean it up? Or atleast flag some for removal. Moderators? -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
